[1055] Cannot import big files

[exterminador]

Hi there,

 

I noticed big files (more than 1.x GB) are not being imported to Daminion Server, I think this is since build 1055 but could be wrong.

 

The exception I get in the Daminion log is:

 

2014-07-19 22:58:19,478 [sTP SmartThreadPool Thread #18] WARN  PicaJet.Daminion.Service.Utils.GetThumbnailServiceCommandV2 [(null)] - Npgsql.NpgsqlException:
el entero está fuera de rango
Severity: ERROR
Code: 22003
  at Npgsql.NpgsqlState.<ProcessBackendResponses_Ver_3>d__a.MoveNext() in C:\projects\Npgsql2\src\Npgsql\NpgsqlState.cs:line 850
  at Npgsql.ForwardsOnlyDataReader.GetNextResponseObject() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1173
  at Npgsql.ForwardsOnlyDataReader.GetNextRowDescription() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1191
  at Npgsql.ForwardsOnlyDataReader.NextResult() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1377
  at Npgsql.ForwardsOnlyDataReader..ctor(IEnumerable`1 dataEnumeration, CommandBehavior behavior, NpgsqlCommand command, NotificationThreadBlock threadBlock, Boolean synchOnReadError) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1040
  at Npgsql.NpgsqlCommand.GetReader(CommandBehavior cb) in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 609
  at Npgsql.NpgsqlCommand.ExecuteNonQuery() in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 512
  at PicaJet.Daminion.Service.DbCommandWrapper.ExecuteNonQuery() in c:\DaminionAuto\Code\Service.DB\DbCommandWrapper.cs:line 41
  at PicaJet.Daminion.Service.Utils.RenderHelper.UpdateModify(Int64 fileId, DateTime modifyDate, Int64 fileSize, DbConnection connection, UniversalCatalogService service) in c:\DaminionAuto\Code\Service.Utils.Uni\Helper\RenderHelper.cs:line 183
  at PicaJet.Daminion.Service.Utils.RenderHelper.UpdateValuesFromFile(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Helper\RenderHelper.cs:line 136
  at PicaJet.Daminion.Service.Utils.GetThumbnailServiceCommandV2.GenerateCacheFileClass.GenerateCacheFileAndSaveStack(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\GetThumbnailServiceCommandV2.cs:line 382
  at PicaJet.Daminion.Service.Utils.GetThumbnailServiceCommandV2.Treat(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\GetThumbnailServiceCommandV2.cs:line 118

 

The error I get in the Postgresql log is:

 

2014-07-19 22:58:19 CEST ERROR:  el entero está fuera de rango
2014-07-19 22:58:19 CEST SENTENCIA:  update Files set ModifiedTime = ((E'2014-07-19 22:57:20.473736')), PropertiesChangeDate = ((E'2014-07-19 22:57:20.473736')), FileSize = ((2981383848)) where Id = ((45090))

 

Hope it helps

 

Thanks!

[Murat]

What's the exact file size that you tried to import?

[exterminador]

What's the exact file size that you tried to import?

 

Hi there,

 

I think the one on the error is 2.77 GB (2,979,613,195 bytes), but it seems it will happen with anything above

2147483647 bytes as the Files.FileSize field is an integer. I had no problems to import big files before, so I guess something changed in one of the latest builds.

 

Regards

[Murat]

The issue will be fixed. Thanks!

[exterminador]

The issue will be fixed. Thanks!

 

Great! Thanks! Glad to help

 

Regards

[Murat]

Can you please check it with the 1059 build?

[exterminador]

Can you please check it with the 1059 build?

 

Thanks Murat, I still get the same error :(

However, it seems the database was not sucessfully updated to build 1059 so probably any update to correct the import issue is still not in my database... I get this error when starting Daminion Server:

 

It seems it is trying to rename two different indexes to the same name (IDX_Files_FileNameUpper):

 

2014-07-31 19:39:06,005 [10] DEBUG PicaJet.Daminion.Service.Utils.UniversalCatalogService [(null)] - Program version: Daminion Server 3.3.0 (build 1059) 
2014-07-31 19:39:06,057 [10] DEBUG PicaJet.Daminion.Service.Utils.CheckValidateAndUpdateCommand [(null)] - Start: (1)
2014-07-31 19:39:06,081 [10] DEBUG SQL [(null)] - select Ver from Settings ()
2014-07-31 19:39:06,089 [10] DEBUG SQL [(null)] - select NewRelative from Settings ()
2014-07-31 19:39:06,109 [10] DEBUG SQL [(null)] - ALTER INDEX IF EXISTS FileNameUpper RENAME TO IDX_Files_FileNameUpper; ()
2014-07-31 19:39:06,125 [10] DEBUG SQL [(null)] - ALTER INDEX IF EXISTS RelativePathUpper RENAME TO IDX_Files_FileNameUpper; ()
2014-07-31 19:39:06,136 [10] ERROR PicaJet.Daminion.Service.Utils.CheckValidateAndUpdateCommand [(null)] - Error: (1)
Npgsql.NpgsqlException:
la relación «idx_files_filenameupper» ya existe
Severity: ERROR
Code: 42P07
  at Npgsql.NpgsqlState.<ProcessBackendResponses_Ver_3>d__a.MoveNext() in C:\projects\Npgsql2\src\Npgsql\NpgsqlState.cs:line 850
  at Npgsql.ForwardsOnlyDataReader.GetNextResponseObject() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1173
  at Npgsql.ForwardsOnlyDataReader.GetNextRowDescription() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1191
  at Npgsql.ForwardsOnlyDataReader.NextResult() in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1377
  at Npgsql.ForwardsOnlyDataReader..ctor(IEnumerable`1 dataEnumeration, CommandBehavior behavior, NpgsqlCommand command, NotificationThreadBlock threadBlock, Boolean synchOnReadError) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 1040
  at Npgsql.NpgsqlCommand.GetReader(CommandBehavior cb) in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 609
  at Npgsql.NpgsqlCommand.ExecuteNonQuery() in C:\projects\Npgsql2\src\Npgsql\NpgsqlCommand.cs:line 512
  at PicaJet.Daminion.Service.DbCommandWrapper.ExecuteNonQuery() in c:\DaminionAuto\Code\Service.DB\DbCommandWrapper.cs:line 41
  at PicaJet.Daminion.Service.Utils.UpdateDatabaseVersion33.RenameIndexesForSpeedUpImport(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\UpdateDatabaseVersion33.cs:line 36
  at PicaJet.Daminion.Service.Utils.UpdateDatabaseVersion33.DoIt(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\UpdateDatabaseVersion33.cs:line 15
  at PicaJet.Daminion.Service.DbCommandExecutor.ExecWithTransaction(Execute execDelegate, IDatabaseEnvironment databaseEnvironment, DbConnection connection, Boolean processErrorHandler) in c:\DaminionAuto\Code\Service.DB\DbCommandExecutor.cs:line 70
  at PicaJet.Daminion.Service.DbCommandExecutor.ExecWithTransaction(Execute execDelegate, IDatabaseEnvironment databaseEnvironment, Boolean processErrorHandler) in c:\DaminionAuto\Code\Service.DB\DbCommandExecutor.cs:line 23
  at PicaJet.Daminion.Service.DbCommandExecutor.ExecWithTransaction(Execute execDelegate, IDatabaseEnvironment databaseEnvironment) in c:\DaminionAuto\Code\Service.DB\DbCommandExecutor.cs:line 14
  at PicaJet.Daminion.Service.Utils.UpdateDatabaseSuperclass.Process(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\UpdateDatabaseSuperclass.cs:line 48
  at PicaJet.Daminion.Service.Utils.CheckValidateAndUpdateCommand.Start(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\CheckValidateAndUpdateCommand.cs:line 213
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Start() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 109
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Process() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 50
2014-07-31 19:39:06,155 [10] DEBUG PicaJet.Daminion.Service.Utils.InitWatchedFoldersManagerCommand [(null)] - Start: (2)

 

Thanks!

[Murat]

Thanks for the bug-report. In the next version it will be fixed automatically.

[Murat]

Please check the new 1060 build.

[exterminador]

Please check the new 1060 build.

 

Thanks Murat!

 

I've updated to the 1061 build and can import big files now, however I get another (I think) related error when tagging files:

 

2014-08-05 11:07:18,711 [sTP SmartThreadPool Thread #11] ERROR PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand [(null)] - Error in ProcessMediaItem
System.InvalidCastException: Specified cast is not valid.
  at Npgsql.NpgsqlDataReader.GetInt32(Int32 i) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 293
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Map(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 36
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Get(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 187
  at PicaJet.Daminion.Service.DbCommandHelper.GetUniqueValue[T](IDbCommand command, SelectDelegate selectDelegate) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 157
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](IDbCommand command, IMapper mapper, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 223
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](DbConnection connection, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 247
  at PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand.ProcessMediaItem(MediaItemLocalEntity mediaItem, DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\SynchronizeMediaItemCommand.cs:line 96
2014-08-05 11:07:18,712 [sTP SmartThreadPool Thread #11] DEBUG SQL [(null)] - update Synchronize set Processed = @Processed where ID = @Id; (@Id=2598098, @Processed=True)
2014-08-05 11:07:18,713 [sTP SmartThreadPool Thread #11] ERROR PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand [(null)] - Error: (2993) - 556232
System.InvalidCastException: Specified cast is not valid.
  at Npgsql.NpgsqlDataReader.GetInt32(Int32 i) in C:\projects\Npgsql2\src\Npgsql\NpgsqlDataReader.cs:line 293
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Map(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 36
  at PicaJet.Daminion.Service.Utils.Mapper.FileMapper.Get(DbDataReader reader) in c:\DaminionAuto\Code\Service.Utils.Uni\Mapper\FileMapper.cs:line 187
  at PicaJet.Daminion.Service.DbCommandHelper.GetUniqueValue[T](IDbCommand command, SelectDelegate selectDelegate) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 157
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](IDbCommand command, IMapper mapper, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 223
  at PicaJet.Daminion.Service.DbCommandHelper.GetById[T](DbConnection connection, Int64 id) in c:\DaminionAuto\Code\Service.DB\DbCommandHelper.cs:line 247
  at PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand.ProcessMediaItem(MediaItemLocalEntity mediaItem, DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\SynchronizeMediaItemCommand.cs:line 130
  at PicaJet.Daminion.Service.Utils.SynchronizeMediaItemCommand.Start(DbConnection connection) in c:\DaminionAuto\Code\Service.Utils.Uni\Command\SynchronizeMediaItemCommand.cs:line 45
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Start() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 109
  at PicaJet.Daminion.Service.Utils.DatabaseCommand.Process() in c:\DaminionAuto\Code\Service.Utils.Uni\DatabaseCommand.cs:line 50
2014-08-05 11:07:18,713 [sTP SmartThreadPool Thread #8] DEBUG PicaJet.Daminion.Service.Utils.GetNextSynchronizeTaskCommand [(null)] - Start: (2995)

 

 

Thanks again

[exterminador]

Please check the new 1060 build.

 

Thanks Murat! With build 1062 the synching issue is gone :)

 

Regards!