Are there any explanations of catalog size difference?

[SergeS]

I am in process of migrating from old Daminion Server 5 (and from Win8.1) to latest Daminion Server 7 (and to Win10).
My catalog is relatively big, close to 200К photos, all tags and keywords are written into raw and jpg files, video files are just few. Testing two approaches of Daminion catalog migration (both on just installed Win10 and Daminion Server 7):
1) Creating new catalog by adding all photos;
2) Restoring previously backup'd catalog from old Daminion, and enforcing Daminion to generate thumbnails (were not backup'd-restored).

Both approaches are working (although not fast 🙂) but difference in size is dramatically huge - for first approach catalog takes almost all my 500GB SSD (only 8GB left!), second approach makes catalog significantly less than 100GB. Did not see any differences yet in functionality. Accelerated preview for large images is enabled, limit max JPEG preview size is 100 Megapixels, Medium size JPEG previews for web-client is off, Video preview also turned off, - for both cases.

Are there any explanations for such difference in size?
Which way to migrate is better? I would like to use first approach, but that size makes me confusing and thinking to buy new SSD 🙂 ...

[Uwe]

Hello,

what is the catalog for you? The postgreSQL database or the thumbnails too? Which postgreSQL version do you use?

This is the status in my system: The biggest catalogs are for photos (140K) and audio (155K). I use the same catalogs since many years.

Regards, Uwe

[lintujuh]

I don't know if this is valid, as you have been writing the tags in the raws. At least, if you are not writing the tags in raws and changing e.g. creation date/time will write the change into the xmp file, but when Daminion imports the file, the raw has precedence.

[SergeS]

3 hours ago, Uwe said:

Hello,

what is the catalog for you? The postgreSQL database or the thumbnails too? Which postgreSQL version do you use?

I meant the whole disk space taken by Daminion files so both, database and thumbnails.
I have tried with brand new Danimion server installation so postgreSQL version is the same as integrated in Daminion 7 server installation. Cannot check now, far from PC.

[Uwe]

Hello,

I wrote a small Python program to check the following: are there thumbnails (e.g. a\a8\aa8c4876-d0e5-44c1-aaf9-e0bcec26ecf1.dat) in the thumbnail folders but not an corresponding/existing record of this thumbnail in the table "thumbnailstack". I found many!!! Where do they come from? This seems to be the reason for the difference between an catalog update ( all thumbnails remain on disk - also the "dead ones" and the creation of a new catalog and thumbnails. 

Question to Daminion:

Is it "ok" to read the files on the disk in the thumbnail folders (the thumbnail main folder, the midsize and preview folders) and then to check if there is a corresponding record in the table "thumbnailstack". If no such record is found in the table "thumbnailstack" - the thumbnail file in the folder is a "dead" file and has to/can be deleted? The file name (GUID) of the thumbnail is the same in the thumbnail main folder and the folders preview and midsize?

This is the current situation in my system:

Regards, Uwe

 

Edited June 14, 2021 by Uwe
screenshot inserted

[Uwe]

Hello,
The reason for the "dead" thumbnails in my system could be that I may have used the same thumbnail folder for another catalogue (test catalogue) for a short time in the past before I assigned the test catalogue its own folder. I have now deleted the "dead" thumbnails (small Python program).
Regards, Uwe

[SergeS]

On 6/17/2021 at 12:52 PM, Uwe said:

Hello,
The reason for the "dead" thumbnails in my system could be that I may have used the same thumbnail folder for another catalogue (test catalogue) for a short time in the past before I assigned the test catalogue its own folder. I have now deleted the "dead" thumbnails (small Python program).
Regards, Uwe

How to find and delete these "dead" thumbnails? Can you share your small Python program (with instruction)? :-)

[Uwe]

Hello SergeS,

here is the small program. I'm not a programmer and didn't spend any time and thoughts neither to implement a GUI or code to work with a command line nor Elements of object-oriented programming are used - quick and dirty - sorry.

What is to do? Please check the code. There are comments marked with "# *** ....". It should explain what you have to adjust in the following line(s)depending on the settings in your system. E.g. in my system the thumbnails of the photos are on an internal SSD drive in the folders:

h:\Daminion\thumbnail\Foto\ - this is the variable root_dir
h:\Daminion\thumbnail\Foto\midsize\ - this is the variable root_dir_midsize
h:\Daminion\thumbnail\Foto\preview\ - this is the variable root_dir_preview

You can call the program by using the batch file. There is a start and end message in the Windows Terminal but the result is in the log file that you can adjust in the code of the program.

You have to remove the comment in the line 101 (os.remove(filename)) to delete the "dead" files. I suggest to call the program first to check if there are "dead" thumbnails and how many are there. Then remove the comment and start the program once more to delete the "dead" thumbnails. A third call of the program should not find any "dead" thumbnail.

Good luck,
Uwe

 

Check_thumbnail_exists_in_Daminion_catalog.zip

[SergeS]

23 hours ago, Uwe said:

Hello SergeS,

here is the small program. I'm not a programmer and didn't spend any time and thoughts neither to implement a GUI or code to work with a command line nor Elements of object-oriented programming are used - quick and dirty - sorry.

Thank you, will try. But later, I am organizing new PC now, there is no python installed yet, and it is not highest priority. 

[WilfriedB]

On 6/12/2021 at 9:14 PM, SergeS said:

Are there any explanations for such difference in size?

Interesting question!

Under Daminion Home server 6.8 (2344), I have 219.6k media items in my catalog. Not sure, whether Daminion counts 1.000 or 1.024 as 1k. If the latter is true, it would be ~224.870 items.

According to Windows properties, my thumbnail folder contains 224.767 files and using Uwe's approach, I found 222.543 thumbnails thereof 707 orphans (=37 MB), that is 221.836 'living' thumbnails.

The total size on disk for the thumbnail folder is 12,1 GB. This is much less than both of Serge's results. So, it is a valid question, why so much space is used in your case. I tweaked Uwe's program a little to accumulate the total size of the orphans. I might possibly help to watch the time stamp of the file creation date and to verify if there other files not having a .dat extension.