Mardon Posted January 5, 2013 Report Share Posted January 5, 2013 I am unable to add more than 10,000 or so files during a single import. Adding less than 10,000 files to a catalogue seems to work fine but every time I try to add more than this, the program crashes. It doesn't crash at precisely 10,000 but around that number. I have about 800,000 images to add to my catalog so this import problem is a serious issue for me. Has anyone been able to successfully add more than10,000 items to a catalog without a crash? Here is a link to a PDF file that shows the details of my system, the error messages and the Bug Report. This PDF file covers several crashes as seen in the table on page one of the file: http://www.JustPhotos.ca/post/DaminionBugReport_v0.9.8_711-03-Jan-1012.pdf (File is also attached to this post.) BTW, I tried everything I could think of that might stop this issue. I used both drag-and-drop and the menu item "Add Files" to initiate the import. Neither worked. I shut down all non-essential OS processes (including my anti-virus) and this made no difference either. Then I tried closing all the 'extra' Windows within Daminion. This didn't help either. I hope that a solution can be found. I love what I see in this product but this is a serious problem for me. Daminion Bug Report v0.9.8_711 03-Jan-1012.pdf Quote Link to comment Share on other sites More sharing options...
Russell Posted January 6, 2013 Report Share Posted January 6, 2013 Wish I had any idea to help you... maybe ship me that incredibly nice computer and I'll give it a go? :-) Russell Quote Link to comment Share on other sites More sharing options...
Mammut Posted January 6, 2013 Report Share Posted January 6, 2013 Hi, I imported 29000 files at once last time without a problem with build 711. But how many files in your Catalog right now? I'm using maximum 100-150000 files now in 1 Catalog (I have a lot of files so I'm using more Catalogs), there could be bugs above that number because of timeout. If you're using too many files right now, maybe there could be the problem. Or there could be a problem with a specific file. If you launch Daminion with -log_enable parameter, it will log the import operation. Regards, mammut Quote Link to comment Share on other sites More sharing options...
Mardon Posted January 6, 2013 Author Report Share Posted January 6, 2013 *snip* If you launch Daminion with -log_enable parameter, it will log the import operation. Regards, mammut Mammut, I have never been able to add more than 10,000 files during a single import regardless of the size of the existing file. The crash occurs even with a brand new file. The largest catalog that I now have contains just over 100,000 files but it has taken forever to get it that large because of all the crashes. I also don't think it is a corrupt file causing the problem because I have tried several times to import files that I had never tried to import before. I wondered if it might be a bad hard drive but I've tried adding files from two separate hard drives during two separate imports and Daminion still crashed. I have also noticed that the "Rescan Folders" option causes Daminion to crash in apparently the same way as when adding a large number of files and produces a similar looking error report. As you suggested, I launched Daminion with the -log_enable parameter on the command line. I tried to eliminate as many variables as possible by creating a new empty catalog and attempting to add approximately27,000 files that I had never attempted to add to a Daminion catalog before. Almost all the files were image files (a few were WAV files). Many were Canon CR2 files with XMP sidecars. The process crashed after adding only 2184 files. Eleven Daminion Log files were created. The file I have attached to this post contains all the information from those eleven files. For ease in posting, I concatenated the text into a single file going from the earliest event to the latest. (The file wouldn't upload with the post so here is a link to it: http://www.JustPhoto...rAddAttempt.pdf) If there are any other tests you want me to run, just ask. I'd really like to get Daminion working correctly for me. Additional Comment: On page 337 of the concatenated log file that I posted, I just notice the following line: 2013-01-06 11:52:56,040 [sTP SmartThreadPool Thread #1] WARNPicaJet.Daminion.Service.Utils.ImportMediaItemFromFileCommand [(null)]ImportFile error: System.OutOfMemoryException: Out of memory. After that, the "Out of memory." Message occurs 15 more times. It seems to me like this must be directly related to the crash. Hope this lead helps. Quote Link to comment Share on other sites More sharing options...
Mardon Posted January 6, 2013 Author Report Share Posted January 6, 2013 *snip* maybe ship me that incredibly nice computer and I'll give it a go? :-) Russell Thanks Russell, I bought my computer over the web from http://www.adkvideoediting.com/ They shipped it to Newfoundland. I like it so much I even created a YouTube testimonial video: Quote Link to comment Share on other sites More sharing options...
Murat Posted January 6, 2013 Report Share Posted January 6, 2013 Hi Mardon, Thanks a lot for the comprehensive tests. We plan to test Daminion for memory leaks on the middle of the next week. It will be very helpful if you can create 20k copies of the same image from the Windows \ Libraries \ Pictures folder. This helps to exlude dependency of memory leaks from a certain media format. Btw, did you try to import images to Daminion Server? Quote Link to comment Share on other sites More sharing options...
Mardon Posted January 7, 2013 Author Report Share Posted January 7, 2013 Hi Mardon, Thanks a lot for the comprehensive tests. We plan to test Daminion for memory leaks on the middle of the next week. It will be very helpful if you can create 20k copies of the same image from the Windows \ Libraries \ Pictures folder. This helps to exlude dependency of memory leaks from a certain media format. Btw, did you try to import images to Daminion Server? I did as you suggested and created 20,000 copies of the Windows7 system image file "\Windows\Web\Wallpaper\Windows\img0.jpg". This image is 1920x1200 and 627 KB. I sequentially numbered the copies I created from"TestImage00001.jpg" to "TestImage20000.jpg" and put them in the folder F:\TestImageFiles". I then created a new Daminion catalog and added the 20,000 files to the new catalog. The import went smoothly. The program did not crash and all 20,000 jpeg image files were added. My previous imports that crashed always contained lots of CR2 files, many with XMP sidecars, as well as some large PSD files. The total size of the files that I imported successfully in this test was just under 12 GB whereas the attempted imports that crashed were always over 50GB, some over 100GB I did not try importing into Daminion Server. I thought about it but didn't really want to have another server running on my PC (I already have a web server running). If it would be helpful to you, I guess I could try temporarily installing the server and see what happens. Let me know if you want that done or if there are any other tests you would like me to perform. If my problem is caused by a memory leak (which seems possible to me), do you think that your testing will catch it or would you have to import the same type and number of files as I am doing in order to have the leak show up in your testing? Quote Link to comment Share on other sites More sharing options...
Mammut Posted January 7, 2013 Report Share Posted January 7, 2013 I think... try to import only those cr2 files (first with xmp sidecars). I didn't have any problems with hq psd files yet. Regards, mammut Quote Link to comment Share on other sites More sharing options...
Murat Posted January 7, 2013 Report Share Posted January 7, 2013 If my problem is caused by a memory leak (which seems possible to me), do you think that your testing will catch it or would you have to import the same type and number of files as I am doing in order to have the leak show up in your testing? Thank you for the tests! It might be due to memory leaks on certain file formats or might depends from the metadata inside the XMP files. Could you please let me know what media formats did you import except CR2, JPEG and WAV? Can you send me a few sample files of each media format, including .xmp file? Quote Link to comment Share on other sites More sharing options...
Mardon Posted January 8, 2013 Author Report Share Posted January 8, 2013 Thank you for the tests! It might be due to memory leaks on certain file formats or might depends from the metadata inside the XMP files. Could you please let me know what media formats did you imported except CR2, JPEG and WAV? Can you send me a few sample files of each media format, including .xmp file? I created 12,000 copies of a CR2 file (12.7 MB) and its XMPsidecar. I then created a new catalog and started an import of the 12,000 image files (with sidecars). Daminion crashed after 6,529 images had been added to the catalog. I then deleted all of the the XMP files, created a new Daminion catalog and began another import of the same 12,000 CR2 files (without sidecars). 6,378 images were added to the catalog before Daminion crashed. T he two files used inthis test are here: http://www.JustPhoto...CR2XMP00001.CR2 http://JustPhotos.ca...CR2XMP00001.xmp I created 12,000 copies of a large jpeg file (6.14 MB) and import it into a new catalogue without any problem. I am in the process of doing two other tests and will post the results as soon as they are done. I created 12,000 copies of a CRW file (2.9 MB) and will try to import those into a new catalog. Here is a link to the image file I am using: http://www.JustPhoto...estCRW00001.Crw I have also created 12,000 copies of a DNG file (7.3 MB) and will try to import that into a new catalog Here is a link to the image file I am using: http://www.JustPhotos.ca Quote Link to comment Share on other sites More sharing options...
Murat Posted January 8, 2013 Report Share Posted January 8, 2013 Thank you, Mardon. I appreciate your assistance! I'll check the import issues now... Quote Link to comment Share on other sites More sharing options...
Mardon Posted January 8, 2013 Author Report Share Posted January 8, 2013 It just occurred to me that my web server may not handle the CR2 and XMP file types. In case you couldn't get the files from there, here are two links to the files in my public Dropbox folder. I know these links will work. http://dl.dropbox.co...CR2XMP00001.CR2 http://dl.dropbox.co...CR2XMP00001.xmp I can also confirm that both the CRW and DNG imports went fine. No problem there. As a final test, I am now running an import on 14,000 copies of a 16.4MB jpeg file. Up until this test, the CR2 files were the largest that I had tried to import. I am doing these very large jpeg files just to make sure file size isn't the issue. I will report here again once that import concludes. Quote Link to comment Share on other sites More sharing options...
Mardon Posted January 8, 2013 Author Report Share Posted January 8, 2013 The import of 14,000 copies of a 16.4 MB jpeg image went fine. Only the CR2 files have caused Daminion to crash during my testing. Let me know if there is anything more I can do. Hope this problem is able to be fixed soon. :) Quote Link to comment Share on other sites More sharing options...
Mardon Posted January 8, 2013 Author Report Share Posted January 8, 2013 *snip* Btw, did you try to import images to Daminion Server? I decieded to install the server and see if that worked. It did not. After installing the server, I tried to add to the shared catalog the same 12,000 copies of the CR2 file (without sidecars) that I used for the Daminion client-only test. Below is the error mesage that I got reporting the failure. The process did not crash like it did with the client-only but it didn't import all the files. I do not know why this shows 11,695 files imported and 648 failures when the total number selected for adding was only 12,000. Quote Link to comment Share on other sites More sharing options...
Murat Posted January 8, 2013 Report Share Posted January 8, 2013 I was able to reproduce the problem with importing with 20k CR2 files using your sample image. We'll perform a few tests tomorrow and try to resolve the issue soon. Quote Link to comment Share on other sites More sharing options...
Murat Posted January 10, 2013 Report Share Posted January 10, 2013 The problem is due to a memory leak in the Image Processing Library that we use. I've contacted to the library's developer and reported about this issue. Once the update version will be ready we'll include it into new Daminion update. BTW with minor optimizations in 715 build the number of images per one session might be higher. Quote Link to comment Share on other sites More sharing options...
Murat Posted January 15, 2013 Report Share Posted January 15, 2013 Good news. The problem will be fixed in the next Daminion build... Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.