679 Posibility to turn-off folder counters?

[exterminador]

Hello,

 

I started using Daminion at early stages but stopped using it because the more pictures I added the performance went down at the point it was not usable. I have a huge images library, about 800,000 files).

 

But I always had in my mind comming back to Daminion :) I really like the way it works and some new features as custom metadata are a plus for me.

 

The server hosting the files (and Daminion server) is a low power server (so performance is not the best), files are stored in a 5400 rpm drive and OS is running on an SSD.

 

As you can see, the server will not offer too much on the performance side. With Daminion with about 400,000 files I'm getting slow response times at displaying pictures, I changed the number of pictures shown to 500 but sometimes thumbnails take too long to display or don't display at all. The folder tree takes also a long time to change from ?/? to the actual number of files, so I'm guessing those queries are taking a long time (probably because of the size of the database) and that CPU time cannot be used to query/get/display thumbnails faster. So I have a couple of questions I hope you can help me with:

 

- Is there a way to hide the files count from the Folder's tree?

- Are counts dinamically calculated or stored in pre-calculated tables?

- Are all the files count queried even if the category is not opened in the tree? for instance, I mostly use the Folders tree opened but have the Rating, Keywords and Flag collapsed, are the counts for this items also queried when those are closed?

- It seems Daminion only start loading the thumnails for the items showing on the pane, wich is a great approach when the environment is fast enough to queryDB/read thumbnails/network/display, but in this case it takes a long time to show the thumbnails, so I'm thinking, would it be possible to have a setting to switch between this behavior and one that loads all the thumbnails in the page (i.e. 500 or 1000)? That way, even in slow environments, user can get a better response time when scrolling down trough the pane.

 

 

Thanks in advance!

[Russell]

The server hosting the files (and Daminion server) is a low power server (so performance is not the best), files are stored in a 5400 rpm drive and OS is running on an SSD.

 

Hi Exterminator,

 

Murat is probably best to answer your questions, but I have one for you:

 

Why not run Daminion server on a more powerful machine - and use your existing "server" as a file store. That's what I'm doing - run Daminion server on your more powerful computer (and store the database on that machine also)... then backup your database to your server regularly. (Murat, I'd love to be able to "schedule this" within your software - plan to use Windows Scheduler to accomplish it - if I can figure out how to turn off the server by command and restart it when backup complete)

 

Let me know how it works if you change your server location, Exterminator.

 

Russell

[Murat]

Exterminador, thanks for the good questions! As you can see, we keep an approach of

consequential development, and progressing step by step.

 

The closest goal is to implement a minimal core functionality. A few features are left in our roadmap to complete this stage:

- Folder Management (90% completed, will be available in the Daminion 0.9.8)

- Restricting access to Daminion Server by user roles

- Publish documents on Mac, Mobile devices, and Web. The prototype is ready, please let me know if anyone would like to check it in action.

- Tag Presets. Will be available in Daminion Downloader, Properties Panel and in the Copy/Paste tags feature.

 

Optimization of the performance is the next logical stage.

 

> Is there a way to hide the files count from the Folder's tree?

 

Interesting idea. Cause it might improve the overall performance when the statistic info will be switched off. Added to the feature list (waiting for other votes).

 

 

> Are counts dynamically calculated or stored in pre-calculated tables?

 

Dynamically. Because the content on the server might be changed by other users. But this is a subject for future optimizations.

 

 

I am waiting for a more comprehensive responses for this and other questions from Alexey and Valery.

 

Could you please send me a dump of your database. I'd like to check the performance issues.

 

And could you please tell me your server PC configuration? RAM, CPU, Windows Version, etc... May be, as Russell said, moving Daminion Server to a more powerful configuration will helps with the performance issues.

 

 

Russell, please find the instructions and scripts to automate backup of your shared database on this server backup guide.

[Mammut]

Interesting idea. Cause it might improve the overall performance when the statistic info will be switched off. Added to the feature list (waiting for other votes).

 

I vote anything which improves the performance of Daminion, because I have a lot of files, too. So +1.

 

Dynamically. Because the content on the server might be changed by other users. But this is a subject for future optimizations.

 

Maybe there could be a setting for this, too. If you know that a specific folder or tag will not change, you don't need dynamic data there.

 

Regards,

mammut

[Uwe]

Hello,

 

I also vote for everything that improves the performance of the server.

Regarding the statistics etc. I would not recommend to make the customizing more complex regarding the performance. Sure one can buffer information and use the buffer to display the data compared to the dynamic calculation of the information. Daminion should not shift hardware issues into the software complexity until the usage of the hardware ressources are not on the very end.

I would like to have a powerful load balancing of the server and database activities in the background. That means: as much as possible use of the available memory (RAM), as much as possible parallel use of the CPU kernels, as much as possible parallel processes of the database, parallelized activities of the server when ever possible. Maybe split a request for mass tagging or rescan folders in portions to start parallel tasks. E.g. today I changed the author information of about 10.000 photos but only one PostgreSQL process is activ. When you rescan folders - this is a serial process and not parallelized. Why not use parallel processes? Distribute the database and the previews on different disks to avoid a bottle neck in the read/write performance? Improve the performance regarding network access. 64-bit Version in the future? And not to forget: We should install the server on the most powerful hardware.

From my point of view the client gets the data from the server when the client has been lunched. These data are calculated dynamically in the moment the client is logged on the server. And this is done for each client in the moment of logon. Correct?

Once the clients are online the changes are not so much, except the mass activities. And this belongs to the performance issues above.

Now we can have only one shared catalog. And this is a huge one. Does it help when we can create more than one shared catalog? I would like to have it as soon as possible. One for the photos (maybe more than one - splitted by years, content, archived etc), one for the Videos, one for the documents, one for the music etc. Then I can adjust the customizing of the columns, the tags etc. as I can do now with the local catalogs.

Does this help to improve the performance of the server?

Some of performance issues are already known by Murat and on his list. I've accepted to wait for the roll-out of the Daminion Priority List described by Murat and then to come back immediately to the performance issues.

 

Regards, Uwe

[Murat]

> Is there a way to hide the files count from the Folder's tree?

 

We can add this option.

 

> Are all the files count queried even if the category is not opened in the tree? for instance, I mostly use the Folders tree opened but have the Rating, Keywords and Flag collapsed, are the counts for this items also queried when those are closed?

 

Statistic info is calculating for expanded and visible tags only. It will not be calculated for collapsed tags. It will even not be calculated for tags in the bottom of the expanded list that are not visible at the moment (until you'll make them visible by scrolling).

 

> It seems Daminion only start loading the thumnails for the items showing on the pane, which is a great approach when the environment is fast enough to queryDB/read thumbnails/network/display, but in this case it takes a long time to show the thumbnails, so I'm thinking, would it be possible to have a setting to switch between this behavior and one that loads all the thumbnails in the page (i.e. 500 or 1000)? That way, even in slow environments, user can get a better response time when scrolling down trough the pane.

 

Added to the feature list and waiting for extra feedbacks and votes. This might be reflected in the server performance on a large number of connections.

[Murat]

Now we can have only one shared catalog. And this is a huge one. Does it help when we can create more than one shared catalog? I would like to have it as soon as possible. One for the photos (maybe more than one - splitted by years, content, archived etc), one for the Videos, one for the documents, one for the music etc. Then I can adjust the customizing of the columns, the tags etc. as I can do now with the local catalogs.

 

Added an extra vote for this feature.

[exterminador]

Hi Exterminator,

 

Murat is probably best to answer your questions, but I have one for you:

 

Why not run Daminion server on a more powerful machine - and use your existing "server" as a file store. That's what I'm doing - run Daminion server on your more powerful computer (and store the database on that machine also)... then backup your database to your server regularly. (Murat, I'd love to be able to "schedule this" within your software - plan to use Windows Scheduler to accomplish it - if I can figure out how to turn off the server by command and restart it when backup complete)

 

Let me know how it works if you change your server location, Exterminator.

 

Russell

 

 

Hi Russell,

 

Thanks for your suggestion. I actually did it, I put Daminion Server and its db on a more powerful computer. Performance improved a lot, still not getting the results I would like but that one is also not the fastest computer. I will ellaborate more on my findings but wanted to thank you first for your suggestion

 

Regards

[exterminador]

Thanks for your reply Murat, and sorry it did take a long time for me to reply.

 

- Publish documents on Mac, Mobile devices, and Web. The prototype is ready, please let me know if anyone would like to check it in action.

 

Please! I would like to see it in action! :) I wondered if there would be a way to access content from iOS, I'm glad it will!

 

And could you please tell me your server PC configuration? RAM, CPU, Windows Version, etc... May be, as Russell said, moving Daminion Server to a more powerful configuration will helps with the performance issues.

 

It did helped moving the server to a more powerful computer. Config of both is:

 

Server: Intel Atom D2700 (2.13 GHz), 4 GB RAM, Windows 7 32-bit, OS running on a SSD, 5400 rpms hard disks for storage. It consumes less than 150 Watts on load which is great if you want to have it 27/7 on.

HTPC (new Daminion Server): Intel Core 2 Duo (2.66 GHz), 4 GB RAM, Windows 8 Pro 32-bit, OS running on a SSD

 

If you need any other detail please let me know.

 

 

Thanks again

[exterminador]

Hello Uwe,

 

Thanks for your comments,

 

Regarding the statistics etc. I would not recommend to make the customizing more complex regarding the performance. Sure one can buffer information and use the buffer to display the data compared to the dynamic calculation of the information. Daminion should not shift hardware issues into the software complexity until the usage of the hardware ressources are not on the very end.

 

While I'm with you on looking for ways to improve performance by pushing the hardware at its best (parallel computing, split data into several disks, etc.), I will have to disagree on this statement. Why not to do more with less if you can optimize software? ;)

 

I know this can lead into a more philosophical conversation but I think power consumption is quickly becoming an important parameter when choosing computing solutions at both, home and business environments and every step done to reduce power consumption/computing power consumption can lead to see more and more "environment friendly software".

 

I was reviewing the latest Intel CPU's with 22nm technology (in the thoughts of getting a better computer for Daminion), but those are using about 80 Watts at iddle, only the cpu, that's about the consumption average of my whole Atom server...

 

I would like to have a powerful load balancing of the server and database activities in the background. That means: as much as possible use of the available memory (RAM), as much as possible parallel use of the CPU kernels, as much as possible parallel processes of the database, parallelized activities of the server when ever possible. Maybe split a request for mass tagging or rescan folders in portions to start parallel tasks. E.g. today I changed the author information of about 10.000 photos but only one PostgreSQL process is activ. When you rescan folders - this is a serial process and not parallelized. Why not use parallel processes? Distribute the database and the previews on different disks to avoid a bottle neck in the read/write performance? Improve the performance regarding network access. 64-bit Version in the future? And not to forget: We should install the server on the most powerful hardware.

 

I'm also researching on ways to do clustering with Postgre, will share if I find something that can help, do you know of any solution that can help to take advantage of the computing power of various pcs with Postgre?

 

Thanks again

[Murat]

There are different aspects with Daminion Server that requires different approaches (mainly from our (developers) side):

 

- Speed of document importing

- Speed of image viewing

- How fast are queries to catalogs

- Speed of metadata syncing (or metadata writing)

- Speed of loading catalogs

- Speed of catalog creation, exporting, etc...

 

And most of the above points can (will) be improved in the future Daminion versions.