[1055] Cannot import big files

exterminador · July 19, 2014

Hi there,

I noticed big files (more than 1.x GB) are not being imported to Daminion Server, I think this is since build 1055 but could be wrong.

The exception I get in the Daminion log is:

2014-07-19 22:58:19,478 [sTP SmartThreadPool Thread #18] WARN  PicaJet.Daminion.Service.Utils.GetThumbnailServiceCommandV2 [(null)] - Npgsql.NpgsqlException:
el entero estÃ¡ fuera de rango
Severity: ERROR
Code: 22003
  at Npgsql.NpgsqlState.<ProcessBackendResponses_Ver_3>d__a.MoveNext() in C:\projects\Npgsql2\src\Npgsql\NpgsqlState.cs:line 850
  at Npgsql.ForwardsOnlyDataReader.GetNextResponseObject() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1173
  at Npgsql.ForwardsOnlyDataReader.GetNextRowDescription() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1191
  at Npgsql.ForwardsOnlyDataReader.NextResult() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1377
  at Npgsql.ForwardsOnlyDataReader..ctor(IEnumerable`1 dataEnumeration, CommandBehavior behavior, NpgsqlCommand command, NotificationThreadBlock threadBlock, Boolean synchOnReadError) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1040
  at Npgsql.NpgsqlCommand.GetReader(CommandBehavior cb) in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 609
  at Npgsql.NpgsqlCommand.ExecuteNonQuery() in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 512
  at PicaJet.Daminion.Service.DbCommandWrapper.ExecuteNonQuery() in c:\DaminionAuto\Code\Service.DB\DbCommandWrapper.cs:line 41
  at PicaJet.Daminion.Service.Utils.RenderHelper.UpdateModify(Int64 fileId, DateTime modifyDate, Int64 fileSize, DbConnection connection, UniversalCatalogService service) in c:\DaminionAuto\Code\Service.Utils.Uni\Helper\RenderHelper.cs:line 183
  at PicaJet.Daminion.Service.Utils.RenderHelper.UpdateValuesFromFile(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Helper\RenderHelper.cs:line 136
  at PicaJet.Daminion.Service.Utils.GetThumbnailServiceCommandV2.GenerateCacheFileClass.GenerateCacheFileAndSaveStack(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\GetThumbnailServiceCommandV2.cs:line 382
  at PicaJet.Daminion.Service.Utils.GetThumbnailServiceCommandV2.Treat(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\GetThumbnailServiceCommandV2.cs:line 118

The error I get in the Postgresql log is:

2014-07-19 22:58:19 CEST ERROR:  el entero estÃƒÂ¡ fuera de rango
2014-07-19 22:58:19 CEST SENTENCIA:  update Files set ModifiedTime = ((E'2014-07-19 22:57:20.473736')), PropertiesChangeDate = ((E'2014-07-19 22:57:20.473736')), FileSize = ((2981383848)) where Id = ((45090))

Hope it helps

Thanks!

Murat · July 22, 2014

What's the exact file size that you tried to import?

exterminador · July 23, 2014

What's the exact file size that you tried to import?

Hi there,

I think the one on the error is 2.77 GB (2,979,613,195 bytes), but it seems it will happen with anything above

2147483647 bytes as the Files.FileSize field is an integer. I had no problems to import big files before, so I guess something changed in one of the latest builds.

Regards

Murat · July 24, 2014

The issue will be fixed. Thanks!

exterminador · July 24, 2014

The issue will be fixed. Thanks!

Great! Thanks! Glad to help

Regards

Murat · July 30, 2014

Can you please check it with the 1059 build?

exterminador · July 31, 2014

Can you please check it with the 1059 build?

Thanks Murat, I still get the same error :(

However, it seems the database was not sucessfully updated to build 1059 so probably any update to correct the import issue is still not in my database... I get this error when starting Daminion Server:

It seems it is trying to rename two different indexes to the same name (IDX_Files_FileNameUpper):

2014-07-31 19:39:06,005 [10] DEBUG PicaJet.Daminion.Service.Utils.UniversalCatalogService [(null)] - Program version: Daminion Server 3.3.0 (build 1059) 
2014-07-31 19:39:06,057 [10] DEBUG PicaJet.Daminion.Service.Utils.CheckValidateAndUpdateCommand [(null)] - Start: (1)
2014-07-31 19:39:06,081 [10] DEBUG SQL [(null)] - select Ver from Settings ()
2014-07-31 19:39:06,089 [10] DEBUG SQL [(null)] - select NewRelative from Settings ()
2014-07-31 19:39:06,109 [10] DEBUG SQL [(null)] - ALTER INDEX IF EXISTS FileNameUpper RENAME TO IDX_Files_FileNameUpper; ()
2014-07-31 19:39:06,125 [10] DEBUG SQL [(null)] - ALTER INDEX IF EXISTS RelativePathUpper RENAME TO IDX_Files_FileNameUpper; ()
2014-07-31 19:39:06,136 [10] ERROR PicaJet.Daminion.Service.Utils.CheckValidateAndUpdateCommand [(null)] - Error: (1)
Npgsql.NpgsqlException:
la relaciÃ³n Â«idx_files_filenameupperÂ» ya existe
Severity: ERROR
Code: 42P07
  at Npgsql.NpgsqlState.<ProcessBackendResponses_Ver_3>d__a.MoveNext() in C:\projects\Npgsql2\src\Npgsql\NpgsqlState.cs:line 850
  at Npgsql.ForwardsOnlyDataReader.GetNextResponseObject() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1173
  at Npgsql.ForwardsOnlyDataReader.GetNextRowDescription() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1191
  at Npgsql.ForwardsOnlyDataReader.NextResult() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1377
  at Npgsql.ForwardsOnlyDataReader..ctor(IEnumerable`1 dataEnumeration, CommandBehavior behavior, NpgsqlCommand command, NotificationThreadBlock threadBlock, Boolean synchOnReadError) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1040
  at Npgsql.NpgsqlCommand.GetReader(CommandBehavior cb) in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 609
  at Npgsql.NpgsqlCommand.ExecuteNonQuery() in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 512
  at PicaJet.Daminion.Service.DbCommandWrapper.ExecuteNonQuery() in c:\DaminionAuto\Code\Service.DB\DbCommandWrapper.cs:line 41
  at PicaJet.Daminion.Service.Utils.UpdateDatabaseVersion33.RenameIndexesForSpeedUpImport(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\UpdateDatabaseVersion33.cs:line 36
  at PicaJet.Daminion.Service.Utils.UpdateDatabaseVersion33.DoIt(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\UpdateDatabaseVersion33.cs:line 15
  at PicaJet.Daminion.Service.DbCommandExecutor.ExecWithTransaction(Execute execDelegate, IDatabaseEnvironment databaseEnvironment, DbConnection connection, Boolean processErrorHandler) in c:\DaminionAuto\Code\Service.DB\DbCommandExecutor.cs:line 70
  at PicaJet.Daminion.Service.DbCommandExecutor.ExecWithTransaction(Execute execDelegate, IDatabaseEnvironment databaseEnvironment, Boolean processErrorHandler) in c:\DaminionAuto\Code\Service.DB\DbCommandExecutor.cs:line 23
  at PicaJet.Daminion.Service.DbCommandExecutor.ExecWithTransaction(Execute execDelegate, IDatabaseEnvironment databaseEnvironment) in c:\DaminionAuto\Code\Service.DB\DbCommandExecutor.cs:line 14
  at PicaJet.Daminion.Service.Utils.UpdateDatabaseSuperclass.Process(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\UpdateDatabaseSuperclass.cs:line 48
  at PicaJet.Daminion.Service.Utils.CheckValidateAndUpdateCommand.Start(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\CheckValidateAndUpdateCommand.cs:line 213
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Start() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 109
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Process() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 50
2014-07-31 19:39:06,155 [10] DEBUG PicaJet.Daminion.Service.Utils.InitWatchedFoldersManagerCommand [(null)] - Start: (2)

Thanks!

Murat · July 31, 2014

Thanks for the bug-report. In the next version it will be fixed automatically.

Murat · August 1, 2014

Please check the new 1060 build.

exterminador · August 5, 2014

Please check the new 1060 build.

Thanks Murat!

I've updated to the 1061 build and can import big files now, however I get another (I think) related error when tagging files:

2014-08-05 11:07:18,711 [sTP SmartThreadPool Thread #11] ERROR PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand [(null)] - Error in ProcessMediaItem
System.InvalidCastException: Specified cast is not valid.
  at Npgsql.NpgsqlDataReader.GetInt32(Int32 i) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 293
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Map(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 36
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Get(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 187
  at PicaJet.Daminion.Service.DbCommandHelper.GetUniqueValue[T](IDbCommand command, SelectDelegate selectDelegate) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 157
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](IDbCommand command, IMapper mapper, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 223
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](DbConnection connection, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 247
  at PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand.ProcessMediaItem(MediaItemLocalEntity mediaItem, DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\SynchronizeMediaItemCommand.cs:line 96
2014-08-05 11:07:18,712 [sTP SmartThreadPool Thread #11] DEBUG SQL [(null)] - update Synchronize set Processed = @Processed where ID = @Id; (@Id=2598098, @Processed=True)
2014-08-05 11:07:18,713 [sTP SmartThreadPool Thread #11] ERROR PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand [(null)] - Error: (2993) - 556232
System.InvalidCastException: Specified cast is not valid.
  at Npgsql.NpgsqlDataReader.GetInt32(Int32 i) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 293
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Map(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 36
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Get(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 187
  at PicaJet.Daminion.Service.DbCommandHelper.GetUniqueValue[T](IDbCommand command, SelectDelegate selectDelegate) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 157
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](IDbCommand command, IMapper mapper, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 223
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](DbConnection connection, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 247
  at PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand.ProcessMediaItem(MediaItemLocalEntity mediaItem, DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\SynchronizeMediaItemCommand.cs:line 130
  at PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand.Start(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\SynchronizeMediaItemCommand.cs:line 45
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Start() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 109
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Process() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 50
2014-08-05 11:07:18,713 [sTP SmartThreadPool Thread #8] DEBUG PicaJet.Daminion.Service.Utils.GetNextSynchronizeTaskCommand [(null)] - Start: (2995)

Thanks again

exterminador · August 6, 2014

Please check the new 1060 build.

Thanks Murat! With build 1062 the synching issue is gone :)

Regards!

Sign In

[1055] Cannot import big files

Recommended Posts

exterminador

Link to comment

Share on other sites

Murat

Link to comment

Share on other sites

exterminador

Link to comment

Share on other sites

Murat

Link to comment

Share on other sites

exterminador

Link to comment

Share on other sites

Murat

Link to comment

Share on other sites

exterminador

Link to comment

Share on other sites

Murat

Link to comment

Share on other sites

Murat

Link to comment

Share on other sites

exterminador

Link to comment

Share on other sites

exterminador

Link to comment

Share on other sites

Join the conversation

Browse

Activity