exterminador Posted June 26, 2013 Report Share Posted June 26, 2013 Hi there, I wanted to share my import performance statistics. It would be great if you can share yours! :) To get them, you can follow the steps bellow (it may look like a lot but it is actually pretty straight forward ): Start pgAdmin III from your windows start menu Under the Object browser panel, double click on your PostgreSQL instance: Enter your postgres user account's password (usually 'postgres') Expand the tree at the left to: Databases -> Daminion and hit the 'SQL' button on the toolbar: Copy and paste the code bellow into the editor and hit the green 'Run' button: The query will run and show results at the bottom panel: To create the report file, click on File -> Quick report... Select the options you want (I suggest unchecking the "Include the SQL in the report" field), enter the file name and hit the OK button And you are done! Then just add a reply with your computer/server characteristics and attach your report :) Hope it helps! I'm sure that can be interesting information for the Daminion's team to improve import performance over the time. select i.id, i.importtime as ImportTime, count(*) as NumberOfFiles, max(f.addeddate) - min(f.addeddate) as ElapsedTime, extract(epoch from (max(f.addeddate) - min(f.addeddate))) as Seconds, case when extract(epoch from (max(f.addeddate) - min(f.addeddate))) > 0 then count(*) / extract(epoch from (max(f.addeddate) - min(f.addeddate))) else 0 end as FilesPerSecond, ( select string_agg(distinct mf.mediaformat_name,',') from mediaitems m inner join mediaformat_table mt on m.id_mediaformat = mt.id inner join ( select cast('558c5cd6-4f28-4ff5-8929-7d1afbc8dcf2' as text) as mediaformat_guid,cast('MPEG1-2' as text) as mediaformat_name union select cast('11cc4740-f040-4b56-aa78-12684d9af8b4' as text) as mediaformat_guid,cast('MKV' as text) as mediaformat_name union select cast('e7259a8f-7829-496e-8f97-46f7355c30b6' as text) as mediaformat_guid,cast('MOV' as text) as mediaformat_name union select cast('a8c6b16a-5357-4c39-ac51-c4f5d1c826f7' as text) as mediaformat_guid,cast('TIFF' as text) as mediaformat_name union select cast('bc260bfe-9378-4282-b94b-adc800901635' as text) as mediaformat_guid,cast('MPEG 4' as text) as mediaformat_name union select cast('79f9644f-a917-47cd-980a-5f96d2d53677' as text) as mediaformat_guid,cast('AVI-Divx' as text) as mediaformat_name union select cast('a8c51c19-30da-4e7f-b37f-b0aa94a5eda4' as text) as mediaformat_guid,cast('DVR-MS' as text) as mediaformat_name union select cast('bdd7e5ff-8740-436b-bd6d-28be2cfe3409' as text) as mediaformat_guid,cast('JPEG' as text) as mediaformat_name union select cast('8e98be26-9a23-443c-899b-d764ce125523' as text) as mediaformat_guid,cast('GIF' as text) as mediaformat_name union select cast('d9b380a3-6c48-4da0-a246-93f4c1cd8655' as text) as mediaformat_guid,cast('PNG' as text) as mediaformat_name union select cast('d760ba69-8549-4b28-b610-3273c7f0c041' as text) as mediaformat_guid,cast('FLV' as text) as mediaformat_name union select cast('e640b0c0-73c0-4c77-8694-af3cb00b9d86' as text) as mediaformat_guid,cast('BDAV' as text) as mediaformat_name union select cast('605f71bd-15ec-4d25-8015-d48b870e7c09' as text) as mediaformat_guid,cast('BMP' as text) as mediaformat_name union select cast('9f6cb6a4-a0dd-4dc9-b701-ba11090e7969' as text) as mediaformat_guid,cast('Windows Media' as text) as mediaformat_name union select cast('1f1317c3-055d-4371-9dd3-7b88f4571284' as text) as mediaformat_guid,cast('3GP' as text) as mediaformat_name union select cast('8179db75-5fed-4848-b3d0-1592dd09784d' as text) as mediaformat_guid,cast('WebM' as text) as mediaformat_name ) mf on mt.value = mf.mediaformat_guid where m.id_importsession = i.id ) as ImportedFileFormats from importsession i inner join mediaitems mi on i.id = mi.id_importsession inner join files f on mi.id_file = f.id group by i.id, i.importtime order by i.importtime Regards. 1 Quote Link to comment Share on other sites More sharing options...
exterminador Posted June 26, 2013 Author Report Share Posted June 26, 2013 Now my results :) Server specs Intel Core i7-3770S CPU @ 3.10GHz 4 GB RAM Windows 8 Pro 64-bit Storage Media files stored on 3TB Western Digital Green drives (SATA 3.0Gb/s, 5400 RPM) OS and Daminion Thumbnail files stored on an SSD disk (Kingston SSDNow V Series 128GB) PostgreSQL database files stored on a 320GB Seagate Momentus 3.5" drive (SATA 3.0Gb/s, 5400 RPM) Import Statistics Daminion Import Statistics 20130626.html NOTE: Thanks a lot to Murat and Valery for considering my suggestion to modify the file.addeddate field behavior which made this statistics possible :) Regards. Quote Link to comment Share on other sites More sharing options...
Uwe Posted June 26, 2013 Report Share Posted June 26, 2013 Hi Exterminator, where do I get the GUID for the RAW files (CR2, NEF, DNG...), music (MP3, FLAC...) to extend the selection. Regards, Uwe Quote Link to comment Share on other sites More sharing options...
exterminador Posted June 26, 2013 Author Report Share Posted June 26, 2013 Hi Exterminator, where do I get the GUID for the RAW files (CR2, NEF, DNG...), music (MP3, FLAC...) to extend the selection. Regards, Uwe Hi Uwe, Couldn't find a table with the labels for the file formats. But you can associate them with this query (could be not as exact as having the label but I think it will work for most scenarios): select mediaformatui, substring(lower(name) from '\.[a-zA-Z0-9]{2,5}$') as fileextension, count(*) from mediaitems group by mediaformatui, substring(lower(name) from '\.[a-zA-Z0-9]{2,5}$') order by substring(lower(name) from '\.[a-zA-Z0-9]{2,5}$') Regards. 1 Quote Link to comment Share on other sites More sharing options...
Albert Posted June 27, 2013 Report Share Posted June 27, 2013 Here's mine, Server: i5 650 @ 3.2 GHz, Windows 7 64 bit, 12 GB Ram O/S and thumbnails on same drive; Seagate 500GB 7200 RPM Photos on WD Black 750 GB drive. daminion Import Report.html Quote Link to comment Share on other sites More sharing options...
Uwe Posted June 27, 2013 Report Share Posted June 27, 2013 Hello, here are my results: Daminion_833_Import_Statistics_Uwe_20130627.html total number of files in catalog: 280.000 Server: Intel® Core i7-3770 CPU @3,40GHz 3,40GHz Win7-Ultimate 64bit RAM: 16GByte Storage, LAN: Media files on NAS (StudioRaid 12TB, RAID6) 1GBit LAN OS, Daminion Thumbnails, PostgreSQL on a SSD OCZ-Vertex2 120GByte) You can find the media format GUID and the assigned media format in the log file. Search for "@MediaFormatUi". To get a comparable result, we have to use the same test files and the same number of files to see if the import speed depends on that. The file size should be typical (but what is typical) for this media format. E.g. to test the PDF import I used 2 files > 350MByte and 8 files from 600KB to 30MB. Total 870MByte. To copy these files from NAS to SSD with TotalCommander: 10sec, Daminion Import time: 98sec. Regards, Uwe Quote Link to comment Share on other sites More sharing options...
petrovich Posted June 27, 2013 Report Share Posted June 27, 2013 where do I get the GUID for the RAW files (CR2, NEF, DNG...), music (MP3, FLAC...) to extend the selection. <fileFormats> <fileFormat guid="bdd7e5ff-8740-436b-bd6d-28be2cfe3409" name="JPEG" supportedExtensions=".jpe.jpg,.jpeg,"/> <fileFormat guid="605f71bd-15ec-4d25-8015-d48b870e7c09" name="BMP" supportedExtensions=".bmp"/> <fileFormat guid="a8c6b16a-5357-4c39-ac51-c4f5d1c826f7" name="TIFF" supportedExtensions=".tif.tiff,"/> <fileFormat guid="8e98be26-9a23-443c-899b-d764ce125523" name="GIF" supportedExtensions=".gif"/> <fileFormat guid="8508a8ad-6113-40ea-af3d-64e3dd2ac33c" name="PSD" supportedExtensions=".psd"/> <fileFormat guid="3c2ed4f9-ee9c-4670-b5c0-87e80b24e62f" name="ICO" supportedExtensions=".ico"/> <fileFormat guid="2e23aa4e-430e-4929-8f93-4923155b408a" name="JPEG 2000" supportedExtensions=".jp2.j2k,.jpc,"/> <fileFormat guid="6cd16bad-6c77-4ac2-b264-156a4a29f0b4" name="HD Photo" supportedExtensions=".hdp.wdp,.wmp,"/> <fileFormat guid="10e6e3f5-6257-42be-b605-f1e8bd64d750" name="PCX" supportedExtensions=".pcx"/> <fileFormat guid="2543e77c-f028-48c6-b265-ee72b64e3836" name="DCX" supportedExtensions=".dcx"/> <fileFormat guid="d9b380a3-6c48-4da0-a246-93f4c1cd8655" name="PNG" supportedExtensions=".png"/> <fileFormat guid="08019035-4eae-4c87-953f-e2705dd012ac" name="TGA" supportedExtensions=".tga.targa,.vda,.icb,.vst,.pix,"/> <fileFormat guid="41055649-67a8-467f-ab47-0a61640a493b" name="PXM - Portable Pixmap" supportedExtensions=".ppm.pgm,.pbm,"/> <fileFormat guid="bec8c067-ac97-4650-b39c-59354fd07c69" name="MPO" supportedExtensions=".mpo"/> <fileFormat guid="e67c67d5-772b-485d-8308-08d70fb65f35" name="CUR - Cursor" supportedExtensions=".cur"/> <fileFormat guid="73e1d28f-79d6-4e1d-89df-2c45df18c85b" name="ANI - Animated Cursor" supportedExtensions=".ani"/> <fileFormat guid="4beb7032-602e-4761-8bae-bb9d5c7d4711" name="WMF" supportedExtensions=".wmf"/> <fileFormat guid="f5b77eba-c79c-43af-8b60-eb75b5234c76" name="EMF" supportedExtensions=".emf"/> <fileFormat guid="46f4cd0d-a85c-4bef-a281-fc1d610235c7" name="PS - PostScript" supportedExtensions=".ps"/> <fileFormat guid="4c4a02b8-94c4-47d3-9726-3e412cc2fa0d" name="EPS - Encapsulated PostScript" supportedExtensions=".eps"/> <fileFormat guid="9a1c5cbc-550d-45ca-9131-926f2fbef9b0" name="CR2 - Canon RAW" supportedExtensions=".cr2"/> <fileFormat guid="c64f7672-8b52-47a4-9ecb-32f419a27fc3" name="CRW - Canon RAW" supportedExtensions=".crw"/> <fileFormat guid="1125ecc9-8554-45ff-a384-23f4e05db0c2" name="NEF - Nikon RAW" supportedExtensions=".nef"/> <fileFormat guid="5267618e-e885-4494-b3c2-ac9ac0b4275f" name="NRW - Nikon RAW" supportedExtensions=".nrw"/> <fileFormat guid="843895c2-9201-4c82-a16e-cd659649e2b7" name="DNG - Adobe Digital Negative" supportedExtensions=".dng"/> <fileFormat guid="72cd331c-5e29-4f9f-80f4-7272662a3236" name="IIQ - Phase One RAW" supportedExtensions=".iiq"/> <fileFormat guid="43ab136c-ea17-42dd-9193-f6d84acc959a" name="MRW - Konica-Minolta RAW" supportedExtensions=".mrw"/> <fileFormat guid="e69bcbf9-3edd-43a2-86d3-5eb750b8dbf0" name="ORF - Olympus RAW" supportedExtensions=".orf"/> <fileFormat guid="8a941277-2336-419e-b375-c378d950e587" name="PEF - Pentax RAW" supportedExtensions=".pef"/> <fileFormat guid="202e8877-0a96-40ae-8fc6-42b5e2949c7b" name="CS1 - Sinar RAW" supportedExtensions=".cs1"/> <fileFormat guid="b03e0695-7afa-41d9-bdb0-9dfccfa64c02" name="AW" supportedExtensions=".aw"/> <fileFormat guid="e9b5cc60-e4da-49b3-9a34-6ed0056d0e99" name="BAY - Casio RAW" supportedExtensions=".bay"/> <fileFormat guid="9f6ae1c9-8707-4c29-9deb-48a9e5db91fd" name="BMQ - Nucore" supportedExtensions=".bmq"/> <fileFormat guid="bf719ef2-412d-4c55-965e-f88fe7053c6d" name="ARW - Sony Alpha RAW" supportedExtensions=".arw"/> <fileFormat guid="a2ee4fbb-7135-4a6a-b959-23aae5677077" name="CAM - Casio RAW" supportedExtensions=".cam"/> <fileFormat guid="a4c70724-eadc-4d83-92cf-ce25c5afdd83" name="DC2 - Kodak RAW" supportedExtensions=".dc2"/> <fileFormat guid="4fc46217-317c-4135-9239-991fc1318dab" name="DCR - Kodak RAW" supportedExtensions=".dcr"/> <fileFormat guid="ac4ff27d-4a31-4f80-a232-e0df0c0102ab" name="RAW - Panasonic RAW" supportedExtensions=".raw"/> <fileFormat guid="95d27b90-271c-4a2f-a767-8083abadd47b" name="RW2 - Panasonic RAW" supportedExtensions=".rw2"/> <fileFormat guid="53d036cd-33e4-454a-ad74-bf39908b3dd3" name="RWL - Leica RAW" supportedExtensions=".rwl"/> <fileFormat guid="2de92b02-c53e-4278-8349-73d458511ebd" name="RAF - FujiFilm RAW" supportedExtensions=".raf"/> <fileFormat guid="781bc0e9-d608-4c9b-b93e-7cc13b729ff8" name="SRF - Sony RAW" supportedExtensions=".srf"/> <fileFormat guid="1fd4fc88-74b3-43b6-866c-0762d8433148" name="X3F - Sigma RAW" supportedExtensions=".x3f"/> <fileFormat guid="f1433c1f-a8d9-4f90-8204-72a34ddbf137" name="KDC - Kodak RAW" supportedExtensions=".kdc"/> <fileFormat guid="34ca4486-befb-416d-98f0-587e528e7d79" name="MOS - Creo Leaf Mosaic" supportedExtensions=".mos"/> <fileFormat guid="f35976bb-c6eb-41d5-ae46-135ebc06d768" name="K25 - Kodak RAW" supportedExtensions=".k25"/> <fileFormat guid="bb94d3d8-1dde-47b8-8477-955fb58a2d6f" name="SR2 - Sony RAW" supportedExtensions=".sr2"/> <fileFormat guid="fbf6e3fe-a534-4551-ae22-2f3086230e57" name="ERF - Epson RAW" supportedExtensions=".erf"/> <fileFormat guid="f675d370-0a3e-485a-8597-237fac82a82e" name="MEF - Mamiya RAW" supportedExtensions=".mef"/> <fileFormat guid="460a4888-07dd-4c93-8584-2ad68703f49f" name="3FR - Hasselblad RAW" supportedExtensions=".3fr"/> <fileFormat guid="21c613da-13cb-460e-83f6-427e04d91ecc" name="FFF - Hasselblad RAW" supportedExtensions=".fff"/> <fileFormat guid="58215b91-956b-4a14-ae84-cc43eeee34c3" name="ADTS AAC" supportedExtensions=".aac"/> <fileFormat guid="0f5e7074-0366-46e2-8821-b790f7f20c60" name="AMR" supportedExtensions=".amr"/> <fileFormat guid="abdc6637-d1c8-4a4c-9e5d-d50b70ffb8ab" name="AIFF" supportedExtensions=".aif"/> <fileFormat guid="e061967b-145c-458c-b62d-7ec2454f83c9" name="APE - Monkey's Audio" supportedExtensions=".ape"/> <fileFormat guid="1cd771c9-9f98-4ec7-991f-4a8a92b7096b" name="Windows Media Audio" supportedExtensions=".wma"/> <fileFormat guid="a39b27ed-2a02-4e6d-ac41-3194ed0dd76a" name="Flac" supportedExtensions=".flac"/> <fileFormat guid="c97a45ef-d792-4db3-8137-b8d3e68608c7" name="MusePack" supportedExtensions=".mpc.mp+,.mpp,"/> <fileFormat guid="377f3a30-5be3-423f-8955-1b7453920c06" name="MPEG Audio" supportedExtensions=".m2a.mp2,.mp1,"/> <fileFormat guid="3f6a3ba5-a1b1-4bcd-b0e8-ef8084858db5" name="MP3" supportedExtensions=".mp3"/> <fileFormat guid="8f6a759f-06dc-40bf-8569-0096798a4465" name="OGG" supportedExtensions=".ogg.oga,.ogv,"/> <fileFormat guid="0367b81d-bb2d-462b-8f4b-4dfb64c862dc" name="WAV" supportedExtensions=".wav"/> <fileFormat guid="087f3c05-89d2-4447-9d25-eabfd42485f1" name="WavPack" supportedExtensions=".wv"/> <fileFormat guid="e7259a8f-7829-496e-8f97-46f7355c30b6" name="MOV - QuickTime Movie" supportedExtensions=".mov.moov,.movie,.qt,"/> <fileFormat guid="9f6cb6a4-a0dd-4dc9-b701-ba11090e7969" name="Windows Media" supportedExtensions=".wmv.asf,"/> <fileFormat guid="79f9644f-a917-47cd-980a-5f96d2d53677" name="AVI, DivX" supportedExtensions=".avi.divx,"/> <fileFormat guid="1f1317c3-055d-4371-9dd3-7b88f4571284" name="3GP" supportedExtensions=".3gp.3g2,"/> <fileFormat guid="d760ba69-8549-4b28-b610-3273c7f0c041" name="FLV - Flash Video" supportedExtensions=".flv"/> <fileFormat guid="bc260bfe-9378-4282-b94b-adc800901635" name="MPEG 4" supportedExtensions=".m4a.m4b,.m4v,.m4p,.mp4,"/> <fileFormat guid="558c5cd6-4f28-4ff5-8929-7d1afbc8dcf2" name="MPEG 1-2" supportedExtensions=".mpg.mpeg,.mpe,.mpv2,.m2v,"/> <fileFormat guid="11cc4740-f040-4b56-aa78-12684d9af8b4" name="MKV - Matroska" supportedExtensions=".mkv"/> <fileFormat guid="8179db75-5fed-4848-b3d0-1592dd09784d" name="WebM" supportedExtensions=".webm"/> <fileFormat guid="e640b0c0-73c0-4c77-8694-af3cb00b9d86" name="BDAV" supportedExtensions=".mts.m2ts,.m2t,"/> <fileFormat guid="a8c51c19-30da-4e7f-b37f-b0aa94a5eda4" name="DVR-MS" supportedExtensions=".dvr-ms"/> </fileFormats> Quote Link to comment Share on other sites More sharing options...
Murat Posted June 28, 2013 Report Share Posted June 28, 2013 It should be helpful it total number of files can be displayed to the end of the each import (or at the begin of the import). Because the import speed is depended from this value. Quote Link to comment Share on other sites More sharing options...
Uwe Posted June 28, 2013 Report Share Posted June 28, 2013 Hi Murat, which number of files do you mean? The total number of files in the catalog before the import? The number of files imported in one Import Session ID is the column "numberoffiles". Regards, Uwe Quote Link to comment Share on other sites More sharing options...
Murat Posted June 28, 2013 Report Share Posted June 28, 2013 Hi Murat, which number of files do you mean? The total number of files in the catalog before the import? The number of files imported in one Import Session ID is the column "numberoffiles". Regards, Uwe The first one: The total number of files in the catalog before the import. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.