Jump to content

Daminion Server Import Statistics


exterminador
 Share

Recommended Posts

Hi there,

 

I wanted to share my import performance statistics. It would be great if you can share yours! :)

 

 

To get them, you can follow the steps bellow (it may look like a lot but it is actually pretty straight forward :yes3: ):

 

  • Start pgAdmin III from your windows start menu
  • Under the Object browser panel, double click on your PostgreSQL instance:
    post-97-0-26082400-1372233054_thumb.png
  • Enter your postgres user account's password (usually 'postgres')
  • Expand the tree at the left to: Databases -> Daminion and hit the 'SQL' button on the toolbar:
    post-97-0-08766000-1372233263_thumb.png
  • Copy and paste the code bellow into the editor and hit the green 'Run' button:
    post-97-0-04083200-1372233445_thumb.png
  • The query will run and show results at the bottom panel:
    post-97-0-52640400-1372233741_thumb.png
  • To create the report file, click on File -> Quick report...
  • Select the options you want (I suggest unchecking the "Include the SQL in the report" field), enter the file name and hit the OK button
    post-97-0-87209300-1372233924_thumb.png
  • And you are done! Then just add a reply with your computer/server characteristics and attach your report :)

 

Hope it helps! I'm sure that can be interesting information for the Daminion's team to improve import performance over the time.

 

select	i.id,
	i.importtime as ImportTime,
	count(*) as NumberOfFiles,
	max(f.addeddate) - min(f.addeddate) as ElapsedTime,
	extract(epoch from (max(f.addeddate) - min(f.addeddate))) as Seconds,
	case when  extract(epoch from (max(f.addeddate) - min(f.addeddate))) > 0 then
		count(*) / extract(epoch from (max(f.addeddate) - min(f.addeddate)))
	else
		0
	end as FilesPerSecond,
	(
		select	string_agg(distinct mf.mediaformat_name,',')
		from	mediaitems m inner join
				mediaformat_table mt on
					m.id_mediaformat = mt.id inner join
				(
					select	cast('558c5cd6-4f28-4ff5-8929-7d1afbc8dcf2' as text) as mediaformat_guid,cast('MPEG1-2' as text) as mediaformat_name
					union
					select	cast('11cc4740-f040-4b56-aa78-12684d9af8b4' as text) as mediaformat_guid,cast('MKV' as text) as mediaformat_name
					union
					select	cast('e7259a8f-7829-496e-8f97-46f7355c30b6' as text) as mediaformat_guid,cast('MOV' as text) as mediaformat_name
					union
					select	cast('a8c6b16a-5357-4c39-ac51-c4f5d1c826f7' as text) as mediaformat_guid,cast('TIFF' as text) as mediaformat_name
					union
					select	cast('bc260bfe-9378-4282-b94b-adc800901635' as text) as mediaformat_guid,cast('MPEG 4' as text) as mediaformat_name
					union
					select	cast('79f9644f-a917-47cd-980a-5f96d2d53677' as text) as mediaformat_guid,cast('AVI-Divx' as text) as mediaformat_name
					union
					select	cast('a8c51c19-30da-4e7f-b37f-b0aa94a5eda4' as text) as mediaformat_guid,cast('DVR-MS' as text) as mediaformat_name
					union
					select	cast('bdd7e5ff-8740-436b-bd6d-28be2cfe3409' as text) as mediaformat_guid,cast('JPEG' as text) as mediaformat_name
					union
					select	cast('8e98be26-9a23-443c-899b-d764ce125523' as text) as mediaformat_guid,cast('GIF' as text) as mediaformat_name
					union
					select	cast('d9b380a3-6c48-4da0-a246-93f4c1cd8655' as text) as mediaformat_guid,cast('PNG' as text) as mediaformat_name
					union
					select	cast('d760ba69-8549-4b28-b610-3273c7f0c041' as text) as mediaformat_guid,cast('FLV' as text) as mediaformat_name
					union
					select	cast('e640b0c0-73c0-4c77-8694-af3cb00b9d86' as text) as mediaformat_guid,cast('BDAV' as text) as mediaformat_name
					union
					select	cast('605f71bd-15ec-4d25-8015-d48b870e7c09' as text) as mediaformat_guid,cast('BMP' as text) as mediaformat_name
					union
					select	cast('9f6cb6a4-a0dd-4dc9-b701-ba11090e7969' as text) as mediaformat_guid,cast('Windows Media' as text) as mediaformat_name
					union
					select	cast('1f1317c3-055d-4371-9dd3-7b88f4571284' as text) as mediaformat_guid,cast('3GP' as text) as mediaformat_name
					union
					select	cast('8179db75-5fed-4848-b3d0-1592dd09784d' as text) as mediaformat_guid,cast('WebM' as text) as mediaformat_name
				) mf on
					mt.value = mf.mediaformat_guid
		where	m.id_importsession = i.id
	) as ImportedFileFormats
from	importsession i inner join
	mediaitems mi on
		i.id = mi.id_importsession inner join
	files f on
		mi.id_file = f.id
group by	i.id,
		i.importtime
order by i.importtime

 

 

Regards.

  • Upvote 1
Link to comment
Share on other sites

Now my results :)

 

Server specs

Intel Core i7-3770S CPU @ 3.10GHz

4 GB RAM

Windows 8 Pro 64-bit

 

Storage

Media files stored on 3TB Western Digital Green drives (SATA 3.0Gb/s, 5400 RPM)

OS and Daminion Thumbnail files stored on an SSD disk (Kingston SSDNow V Series 128GB)

PostgreSQL database files stored on a 320GB Seagate Momentus 3.5" drive (SATA 3.0Gb/s, 5400 RPM)

 

Import Statistics

Daminion Import Statistics 20130626.html

 

NOTE: Thanks a lot to Murat and Valery for considering my suggestion to modify the file.addeddate field behavior which made this statistics possible :)

 

Regards.

Link to comment
Share on other sites

Hi Exterminator,

 

where do I get the GUID for the RAW files (CR2, NEF, DNG...), music (MP3, FLAC...) to extend the selection.

 

Regards, Uwe

 

Hi Uwe,

 

Couldn't find a table with the labels for the file formats. But you can associate them with this query (could be not as exact as having the label but I think it will work for most scenarios):

 

select	mediaformatui,
	substring(lower(name) from '\.[a-zA-Z0-9]{2,5}$') as fileextension,
	count(*) 
from mediaitems 
group by mediaformatui,
	substring(lower(name) from '\.[a-zA-Z0-9]{2,5}$')
order by substring(lower(name) from '\.[a-zA-Z0-9]{2,5}$')

 

Regards.

  • Upvote 1
Link to comment
Share on other sites

Hello,

 

here are my results:

Daminion_833_Import_Statistics_Uwe_20130627.html

 

total number of files in catalog: 280.000

 

Server:

Intel® Core™ i7-3770 CPU @3,40GHz 3,40GHz

Win7-Ultimate 64bit

RAM: 16GByte

 

Storage, LAN:

Media files on NAS (StudioRaid 12TB, RAID6)

1GBit LAN

OS, Daminion Thumbnails, PostgreSQL on a SSD OCZ-Vertex2 120GByte)

 

You can find the media format GUID and the assigned media format in the log file. Search for "@MediaFormatUi".

 

To get a comparable result, we have to use the same test files and the same number of files to see if the import speed depends on that. The file size should be typical (but what is typical) for this media format. E.g. to test the PDF import I used 2 files > 350MByte and 8 files from 600KB to 30MB. Total 870MByte. To copy these files from NAS to SSD with TotalCommander: 10sec, Daminion Import time: 98sec.

 

Regards, Uwe

Link to comment
Share on other sites

where do I get the GUID for the RAW files (CR2, NEF, DNG...), music (MP3, FLAC...) to extend the selection.

 

<fileFormats>
<fileFormat guid="bdd7e5ff-8740-436b-bd6d-28be2cfe3409" name="JPEG" supportedExtensions=".jpe.jpg,.jpeg,"/>
<fileFormat guid="605f71bd-15ec-4d25-8015-d48b870e7c09" name="BMP" supportedExtensions=".bmp"/>
<fileFormat guid="a8c6b16a-5357-4c39-ac51-c4f5d1c826f7" name="TIFF" supportedExtensions=".tif.tiff,"/>
<fileFormat guid="8e98be26-9a23-443c-899b-d764ce125523" name="GIF" supportedExtensions=".gif"/>
<fileFormat guid="8508a8ad-6113-40ea-af3d-64e3dd2ac33c" name="PSD" supportedExtensions=".psd"/>
<fileFormat guid="3c2ed4f9-ee9c-4670-b5c0-87e80b24e62f" name="ICO" supportedExtensions=".ico"/>
<fileFormat guid="2e23aa4e-430e-4929-8f93-4923155b408a" name="JPEG 2000" supportedExtensions=".jp2.j2k,.jpc,"/>
<fileFormat guid="6cd16bad-6c77-4ac2-b264-156a4a29f0b4" name="HD Photo" supportedExtensions=".hdp.wdp,.wmp,"/>
<fileFormat guid="10e6e3f5-6257-42be-b605-f1e8bd64d750" name="PCX" supportedExtensions=".pcx"/>
<fileFormat guid="2543e77c-f028-48c6-b265-ee72b64e3836" name="DCX" supportedExtensions=".dcx"/>
<fileFormat guid="d9b380a3-6c48-4da0-a246-93f4c1cd8655" name="PNG" supportedExtensions=".png"/>
<fileFormat guid="08019035-4eae-4c87-953f-e2705dd012ac" name="TGA" supportedExtensions=".tga.targa,.vda,.icb,.vst,.pix,"/>
<fileFormat guid="41055649-67a8-467f-ab47-0a61640a493b" name="PXM - Portable Pixmap" supportedExtensions=".ppm.pgm,.pbm,"/>
<fileFormat guid="bec8c067-ac97-4650-b39c-59354fd07c69" name="MPO" supportedExtensions=".mpo"/>
<fileFormat guid="e67c67d5-772b-485d-8308-08d70fb65f35" name="CUR - Cursor" supportedExtensions=".cur"/>
<fileFormat guid="73e1d28f-79d6-4e1d-89df-2c45df18c85b" name="ANI - Animated Cursor" supportedExtensions=".ani"/>
<fileFormat guid="4beb7032-602e-4761-8bae-bb9d5c7d4711" name="WMF" supportedExtensions=".wmf"/>
<fileFormat guid="f5b77eba-c79c-43af-8b60-eb75b5234c76" name="EMF" supportedExtensions=".emf"/>
<fileFormat guid="46f4cd0d-a85c-4bef-a281-fc1d610235c7" name="PS - PostScript" supportedExtensions=".ps"/>
<fileFormat guid="4c4a02b8-94c4-47d3-9726-3e412cc2fa0d" name="EPS - Encapsulated PostScript" supportedExtensions=".eps"/>
<fileFormat guid="9a1c5cbc-550d-45ca-9131-926f2fbef9b0" name="CR2 - Canon RAW" supportedExtensions=".cr2"/>
<fileFormat guid="c64f7672-8b52-47a4-9ecb-32f419a27fc3" name="CRW - Canon RAW" supportedExtensions=".crw"/>
<fileFormat guid="1125ecc9-8554-45ff-a384-23f4e05db0c2" name="NEF - Nikon RAW" supportedExtensions=".nef"/>
<fileFormat guid="5267618e-e885-4494-b3c2-ac9ac0b4275f" name="NRW - Nikon RAW" supportedExtensions=".nrw"/>
<fileFormat guid="843895c2-9201-4c82-a16e-cd659649e2b7" name="DNG - Adobe Digital Negative" supportedExtensions=".dng"/>
<fileFormat guid="72cd331c-5e29-4f9f-80f4-7272662a3236" name="IIQ - Phase One RAW" supportedExtensions=".iiq"/>
<fileFormat guid="43ab136c-ea17-42dd-9193-f6d84acc959a" name="MRW - Konica-Minolta RAW" supportedExtensions=".mrw"/>
<fileFormat guid="e69bcbf9-3edd-43a2-86d3-5eb750b8dbf0" name="ORF - Olympus RAW" supportedExtensions=".orf"/>
<fileFormat guid="8a941277-2336-419e-b375-c378d950e587" name="PEF - Pentax RAW" supportedExtensions=".pef"/>
<fileFormat guid="202e8877-0a96-40ae-8fc6-42b5e2949c7b" name="CS1 - Sinar RAW" supportedExtensions=".cs1"/>
<fileFormat guid="b03e0695-7afa-41d9-bdb0-9dfccfa64c02" name="AW" supportedExtensions=".aw"/>
<fileFormat guid="e9b5cc60-e4da-49b3-9a34-6ed0056d0e99" name="BAY - Casio RAW" supportedExtensions=".bay"/>
<fileFormat guid="9f6ae1c9-8707-4c29-9deb-48a9e5db91fd" name="BMQ - Nucore" supportedExtensions=".bmq"/>
<fileFormat guid="bf719ef2-412d-4c55-965e-f88fe7053c6d" name="ARW - Sony Alpha RAW" supportedExtensions=".arw"/>
<fileFormat guid="a2ee4fbb-7135-4a6a-b959-23aae5677077" name="CAM - Casio RAW" supportedExtensions=".cam"/>
<fileFormat guid="a4c70724-eadc-4d83-92cf-ce25c5afdd83" name="DC2 - Kodak RAW" supportedExtensions=".dc2"/>
<fileFormat guid="4fc46217-317c-4135-9239-991fc1318dab" name="DCR - Kodak RAW" supportedExtensions=".dcr"/>
<fileFormat guid="ac4ff27d-4a31-4f80-a232-e0df0c0102ab" name="RAW - Panasonic RAW" supportedExtensions=".raw"/>
<fileFormat guid="95d27b90-271c-4a2f-a767-8083abadd47b" name="RW2 - Panasonic RAW" supportedExtensions=".rw2"/>
<fileFormat guid="53d036cd-33e4-454a-ad74-bf39908b3dd3" name="RWL - Leica RAW" supportedExtensions=".rwl"/>
<fileFormat guid="2de92b02-c53e-4278-8349-73d458511ebd" name="RAF - FujiFilm RAW" supportedExtensions=".raf"/>
<fileFormat guid="781bc0e9-d608-4c9b-b93e-7cc13b729ff8" name="SRF - Sony RAW" supportedExtensions=".srf"/>
<fileFormat guid="1fd4fc88-74b3-43b6-866c-0762d8433148" name="X3F - Sigma RAW" supportedExtensions=".x3f"/>
<fileFormat guid="f1433c1f-a8d9-4f90-8204-72a34ddbf137" name="KDC - Kodak RAW" supportedExtensions=".kdc"/>
<fileFormat guid="34ca4486-befb-416d-98f0-587e528e7d79" name="MOS - Creo Leaf Mosaic" supportedExtensions=".mos"/>
<fileFormat guid="f35976bb-c6eb-41d5-ae46-135ebc06d768" name="K25 - Kodak RAW" supportedExtensions=".k25"/>
<fileFormat guid="bb94d3d8-1dde-47b8-8477-955fb58a2d6f" name="SR2 - Sony RAW" supportedExtensions=".sr2"/>
<fileFormat guid="fbf6e3fe-a534-4551-ae22-2f3086230e57" name="ERF - Epson RAW" supportedExtensions=".erf"/>
<fileFormat guid="f675d370-0a3e-485a-8597-237fac82a82e" name="MEF - Mamiya RAW" supportedExtensions=".mef"/>
<fileFormat guid="460a4888-07dd-4c93-8584-2ad68703f49f" name="3FR - Hasselblad RAW" supportedExtensions=".3fr"/>
<fileFormat guid="21c613da-13cb-460e-83f6-427e04d91ecc" name="FFF - Hasselblad RAW" supportedExtensions=".fff"/>
<fileFormat guid="58215b91-956b-4a14-ae84-cc43eeee34c3" name="ADTS AAC" supportedExtensions=".aac"/>
<fileFormat guid="0f5e7074-0366-46e2-8821-b790f7f20c60" name="AMR" supportedExtensions=".amr"/>
<fileFormat guid="abdc6637-d1c8-4a4c-9e5d-d50b70ffb8ab" name="AIFF" supportedExtensions=".aif"/>
<fileFormat guid="e061967b-145c-458c-b62d-7ec2454f83c9" name="APE - Monkey's Audio" supportedExtensions=".ape"/>
<fileFormat guid="1cd771c9-9f98-4ec7-991f-4a8a92b7096b" name="Windows Media Audio" supportedExtensions=".wma"/>
<fileFormat guid="a39b27ed-2a02-4e6d-ac41-3194ed0dd76a" name="Flac" supportedExtensions=".flac"/>
<fileFormat guid="c97a45ef-d792-4db3-8137-b8d3e68608c7" name="MusePack" supportedExtensions=".mpc.mp+,.mpp,"/>
<fileFormat guid="377f3a30-5be3-423f-8955-1b7453920c06" name="MPEG Audio" supportedExtensions=".m2a.mp2,.mp1,"/>
<fileFormat guid="3f6a3ba5-a1b1-4bcd-b0e8-ef8084858db5" name="MP3" supportedExtensions=".mp3"/>
<fileFormat guid="8f6a759f-06dc-40bf-8569-0096798a4465" name="OGG" supportedExtensions=".ogg.oga,.ogv,"/>
<fileFormat guid="0367b81d-bb2d-462b-8f4b-4dfb64c862dc" name="WAV" supportedExtensions=".wav"/>
<fileFormat guid="087f3c05-89d2-4447-9d25-eabfd42485f1" name="WavPack" supportedExtensions=".wv"/>
<fileFormat guid="e7259a8f-7829-496e-8f97-46f7355c30b6" name="MOV - QuickTime Movie" supportedExtensions=".mov.moov,.movie,.qt,"/>
<fileFormat guid="9f6cb6a4-a0dd-4dc9-b701-ba11090e7969" name="Windows Media" supportedExtensions=".wmv.asf,"/>
<fileFormat guid="79f9644f-a917-47cd-980a-5f96d2d53677" name="AVI, DivX" supportedExtensions=".avi.divx,"/>
<fileFormat guid="1f1317c3-055d-4371-9dd3-7b88f4571284" name="3GP" supportedExtensions=".3gp.3g2,"/>
<fileFormat guid="d760ba69-8549-4b28-b610-3273c7f0c041" name="FLV - Flash Video" supportedExtensions=".flv"/>
<fileFormat guid="bc260bfe-9378-4282-b94b-adc800901635" name="MPEG 4" supportedExtensions=".m4a.m4b,.m4v,.m4p,.mp4,"/>
<fileFormat guid="558c5cd6-4f28-4ff5-8929-7d1afbc8dcf2" name="MPEG 1-2" supportedExtensions=".mpg.mpeg,.mpe,.mpv2,.m2v,"/>
<fileFormat guid="11cc4740-f040-4b56-aa78-12684d9af8b4" name="MKV - Matroska" supportedExtensions=".mkv"/>
<fileFormat guid="8179db75-5fed-4848-b3d0-1592dd09784d" name="WebM" supportedExtensions=".webm"/>
<fileFormat guid="e640b0c0-73c0-4c77-8694-af3cb00b9d86" name="BDAV" supportedExtensions=".mts.m2ts,.m2t,"/>
<fileFormat guid="a8c51c19-30da-4e7f-b37f-b0aa94a5eda4" name="DVR-MS" supportedExtensions=".dvr-ms"/>
</fileFormats>

Link to comment
Share on other sites

Hi Murat,

which number of files do you mean? The total number of files in the catalog before the import? The number of files imported in one Import Session ID is the column "numberoffiles".

Regards, Uwe

 

The first one: The total number of files in the catalog before the import.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...