"Copy to Folder" Performance/Speed Suggestions?

[moorecokhodge]

I'm looking for ideas on how to speed up the "Copy to Folder" process for large batches of PSD files. Has anyone found any settings in Daminion or otherwise that help to speed up this process?

 

On its own this process is fairly quick, but I am looking at using "Copy to Folder" to output between 12,000-20,000 images to JPEG (100% quality, 100% size, 300DPI, no metadata), so any speed savings will help.

 

I am currently exporting about 12,000 PSD files from my catalog, with a target folder that resides on the same HDD as my catalog. These source files range in size from 2MB to 800MB. I currently have an estimated speed and completion time of: "2 files/min, time left: 2 days, 20 hours".

 

RAM doesn't seem to be maxing out and neither does CPU. The Daminion server process is hovering around 15% max RAM and 20% max CPU.

 

Possible options that I've come up with (but not tried) so far:

1) Set the "Copy to Folder" target as a large flash drive plugged into the server

2) Change Daminion Server process priority to "Above Normal"

3) Experiment with alternate output formats (PNG/TIFF/etc) and, if faster, then using a 3rd party tool to batch convert to JPEG

 

Any input is greatly appreciated!

 

Server information:

Second generation Intel i5 3.1 GHZ

8GB RAM

Integrated Intel® HD Graphics 2000

Windows Server 2008

Used in "Scheme A" configuration (NAS drives wired on same gigabit internal network as server)

Almost nothing else running

[Murat]

Are your PSD files CMYK or RGB? We plan to improve the performance of rendering CMYK files in the next Daminion version.

 

1) Set the "Copy to Folder" target as a large flash drive plugged into the server

 

This will not help because the Daminion Client downloads the resulted JPEG file (after transformation on the Server) and then save it in the destination folder.

 

Not a good way, but it also possible to import these RAW images into the local catalog on your local PC, and open it by 4-5 different instances of the Daminion Clients. And then launch the export processes for different set of folders. (ugly but should work)

 

We also think about the Linux version of the Daminion and adding support for multi-thread image processing operations, but this is in the future plans only.

[moorecokhodge]

Are your PSD files CMYK or RGB? We plan to improve the performance of rendering CMYK files in the next Daminion version.

 

Ah, that is probably a big part of the issue then. As we do a lot of print work with our files, almost 99% of our catalog is in CMYK mode. We'll definitely look forward to the upgrade, in that case!

 

 

This will not help because the Daminion Client downloads the resulted JPEG file (after transformation on the Server) and then save it in the destination folder.

 

If I plug the flash drive into my client machine and change the target folder to that drive, I assume that would speed it up somewhat? As the client would only be downloading the converted JPEG and writing it to the flash drive instead of downloading it and writing it back to already-busy NAS.

[Uwe]

Hi khodge,

 

I made a short test with 200PSD (RGB) files. Between 1MB - 200MB. Also schema A. Destination folder is a separate HD, 1GBit LAN,16GByte, core i7-3770 3,4GHz, Win7-64bit

The average speed was 60files/min.

 

I don't know if the performance adjustments have an impact. I use the settings as recommended:

https://daminion.net/user-forum/index.php?/topic/1161-daminion-server-performance/

 

Regards, Uwe

[moorecokhodge]

Hi khodge,

 

I made a short test with 200PSD (RGB) files. Between 1MB - 200MB. Also schema A. Destination folder is a separate HD, 1GBit LAN,16GByte, core i7-3770 3,4GHz, Win7-64bit

The average speed was 60files/min.

 

I don't know if the performance adjustments have an impact. I use the settings as recommended:

https://daminion.net/user-forum/index.php?/topic/1161-daminion-server-performance/

 

Regards, Uwe

 

Thank you for testing this! I have made the same adjustments as are in the link you sent.

 

Based on Murat's response, I am thinking that this is issue has to do primarily with our use of CMYK files. The conversion process for those is currently longer than the conversion process for RGB, it seems.

 

I'm still going to tinker and see if I can eek out a few minor improvements that hopefully, over the course of 20,000 exports, will add up.

[Murat]

I don't know if the performance adjustments have an impact. I use the settings as recommended:

https://daminion.net/user-forum/index.php?/topic/1161-daminion-server-performance/

 

The bottleneck in the Kollin's case is the Image processing and probably network bandwidth. The above topic is addressed for improving performance of the PostgreSQL queries which have less impact in this case.

 

If I plug the flash drive into my client machine and change the target folder to that drive, I assume that would speed it up somewhat? As the client would only be downloading the converted JPEG and writing it to the flash drive instead of downloading it and writing it back to already-busy NAS.

 

Yep, slightly. But if have the gigabit ethernet and Usb 2.0 flash card the speed will be comparable. However it depends from how large your PSD files are.

 

Can you please share with me one of your big CMYK PSD files?

[moorecokhodge]

Yep, slightly. But if have the gigabit ethernet and Usb 2.0 flash card the speed will be comparable. However it depends from how large your PSD files are.

 

I do have some USB 3.0 flash drives sitting around so I'll give that a try shortly. Even speeding it up by 3 seconds per file would save me around 16 hours total in the end!

 

 

Can you please share with me one of your big CMYK PSD files?

 

Sure thing, I've uploaded two large CMYK PSDs and two smaller CMYK PSDs files to Dropbox. I'm PMing you with a link now.