File exif data incomplete or wrong

[pgrondin]

I have been busy tagging photos, testing search/filter capabilities etc.

 

Recently, I noticed that when viewing metadata using either Nikon View NX or Exiftool, that I have quite a few directories where the tags in the photos do not match the tags shown in Daminion.

 

In several of these directories the tags actually written to file are the same for all photos, and are not generally appropriate.

 

In several directories, some of the tags are correct, and some are either incomplete, or totally inappropriate.

 

I need help to figure out how this is happening.

 

There are four possibilities that come to my mind:

 

--1-- I have had several occasions where Daminion crashed, and asked whether to reopen the catalog when I re-started. I selected yes. The catalog always appears to be OK after a restart Unfortunately, I have not been documenting what I was doing at these times, which directories was I working in, etc. Do you have any recommendations on what I should be checking / documenting after a crash? Is there a set of files I should copy to a safe location before I restart?

 

--2-- There may be some way in which the operator is at fault (Me). The one area that comes to mind here is the SAVE button at the bottom of the "Properties" panel. Most tagging that is applied using the "Catalog Tags" panel appears to just update in the background. When I select a group of photos, and set the "Place" tags using the "Properties" panel I have been using the SAVE button. Does this cause the other tags to get overwritten? What is the appropriate use for the SAVE button?

 

--3-- There may be a problem created when I shut Daminion and/or my computer down before Daminion finishes background updating all of the photo files. There are many occasions where I apply tags to thousands of photos, get tired, and shut down. This update queue problem may also be impacted by the shutdowns mentioned in --1--

 

--4-- There is some other bug in the way Daminion updates tags written to file.

 

If there is anything I can do to help isolate this problem, please feel free to ask.

 

At this point, the tags in the Daminion catalog appear to be OK, but I need to be able to have confidence that the tags I assign are actually being written correctly to file.

 

Anyone have any ideas?

 

Thanks

 

Peter

[franky68]

Hello Peter,

 

thanks for the info - this is pretty scary!

 

From how I read your post the problems are limited to the tags (keywords etc.) you assigned from Daminion and that are written back into your files (exported) is this correct or does this occur with imported values (apperture, shutter etc) as well?

 

If it happens with exported tags #1 seems pretty logical. If Daminion crashes while its still syncing the changes would be in the database but not in the file. Former versions did not have an indicator for syncing so you might would not even notice if Daminion crashed during the sync process. I dont know how the queue is treated but I am sure Murat can clarify.

 

I generally would prefer a "unclean" Tag as a file property in the database which tags files that are not yet synced back.

 

Your statement that this seems to happen by directory seems to indicate that this happened only to cirtain work sessions. So the crash theory is not too unlikely.

[Murat]

...

In several directories, some of the tags are correct, and some are either incomplete, or totally inappropriate.

...

 

I need to know more info about the difference between Tags in the Catalog and Metadata in the File. In this case I can give you more precise answer.

 

But anyway I guess the problem is occured due to the termination (for some reasons) of the syncing process.

Daminion do not starts the unfinished sync process after the program restart. A state of the sync queue is saved into the catalog and after the next user action (a tag assigment) the sync thread will be launched again.

 

BTW on the 0.8.6 version we added a feature to manually write the Tags to Metadata.

 

Franky, we don't remove/touch the major part of the EXIF info when updating the metadata. Most of the EXIF technical fields are readonly for Daminion, including the exposure and camera/lens info.

[franky68]

Daminion do not starts the unfinished sync process after the program restart. A state of the sync queue is saved into the catalog and after the next user action (a tag assigment) the sync thread will be launched again.

 

Sorry, this is a little misunderstandable... Does Daminion resume the sync queue or does it not?

 

Edit: It seems Peter understood better and Daminion continues with the queue once the sync process is started by a recent change?

 

Franky, we don't remove/touch the major part of the EXIF info when updating the metadata. Most of the EXIF technical fields are readonly for Daminion, including the exposure and camera/lens info.

 

Oh, I am sorry. I did not mean Daminion was writing to technical fields but I wondered if (for whatever reason) they have not been read correctly so the Daminion data differs from the file data. Peter was not very clear about this (or I didn't read properly) thats why I asked.

 

As you already said, it would be good if he could give a few more details.

[pgrondin]

I thought I sent a version of this message off two or three hours ago. I got called away, not sure I sent it. Hope this is not duplicated.

 

I installed 0.8.6 today and ran a couple of tests.

 

I selected a directory where the Daminion catalog tags were correct, but the NEF Keyword data was completely wrong. I tried the "Item/Actions/Write Keyword data to file". All of the tags were written correctly. Great! The "Sync" indicator showed no activity. It would be nice to have feedback that the write is in progress / complete.

 

I selected 5 photos in another directory where most of the Keyword data in the NEF files was correct, but some of the photos had incorrect keywords. I applied a new tag to the photos. The Sync indicator came to life, but if I noticed correctly showed 7 ?? photos being updated. The keyword I had applied completely replaced the keyword data that had been in the NEF file, replacing the previously incorrect data. Great !!

 

But: The update write also changed the Place data to a completely incorrect location. (Wrong City, and State, and it erased the Location field) Country was correct. OUCH!! The Place data in the Daminion catalog still shows the correct data.

 

I am considering the possibility that my Daminion catalog is corrupted in some way.

 

I am considering the following procedure:

- backing up my working photo directories, and Daminion catalog,

- Using the new "Write Keyword data to file" one directory at a time in the "working set" .

[EDIT]

- Save the Catalog again using a new name or directory.

[Edit]

- Create a new catalog

- Add the NEF files into the new catalog.

 

Does this look like it may work to fix the inconsistencies problem?

 

As I have time over the next few days, I will try to document the exact inconsistencies I am having and try to see if I can get repeatable errors.

 

Murat, If I understand correctly;

When I shut the program down before it completes a sync write, the program saves it's sync status. When the program is re-started, syncing does not start automatically but waits till after a tag is changed.

 

If there is a program crash, would the sync status still be available, or would it not have been saved?

 

Thanks Murat, Franky86

[pgrondin]

I decided to proceed with writing metadata to file.

 

Writing metdata to file seems to work flawlessly so far. I have done about 10 directories, 1400 photos. I have to wait for it to complete writing; no multitasking here. It replaces all the expected fields: Keywords, Place, Description, Title, Categories.

 

As I am going through the directories I occasionally add Keywords to various photos. When I do this, it generally works as expected. It adds or removes the selected keyword, leaving all other fields unchanged. Where The NEF file has data in other fields that does not match the Daminion catalog, the “Sync” leaves these fields alone. The word “Sync” for me has more of a “make consistent” implication to it, but the behaviour is correct in my view. It should only apply the changes made. ( A consistency check feature would be nice at some point though.)

 

The following appears to be a repeatable bug – to replicate on my machine, and catalog using 0.8.6 :

 

-1- Write metadata to disk for a directory. Note that on my system each directory is for a specific combination of Country, and City.ie c:PhotosCountryCity.

-2- Select one or more photos in another directory

-3- Verify that the Place data in the NEF file is correct for these photos

-4- Apply a keyword to the files selected in 2 above.

-3- After “Sync” the Place data in the NEF file has been changed to match the State and City for the previous directory for which I wrote Metadata to disk. Where the previous directory had multiple “Locations”, the location written to the NEF file by the “Sync” is the last Location.

 

I think I'm on to something. If a group variable (Keyword, Category, Place, Title etc) is not changed by the current operator action, it carries it's previous value forward and is written to file when something else is changed.

 

Lets try to test that.

 

-1-Assign a Title to a photo.

-2- Save it.

-3- Check that the metadata is correct

-4- Select some other photo

-5- Check that the metadata is correct. Either it has a title, or the title is blank.

-6- Add a Keyword to the photo

-7- Check to see if the title for the previous photo has been carried forward.

 

IT HAS !!

 

I have currently gone through a sequence where I changed one Group at a time on different photos. The variables changed were:

 

Photo-----------Group/Variable-----New contents

1 ------------- Location---------- Erwhon

2 ------------- Title------------- Courtyard of the baths

3 ------------- Category---------- Textures

4 ------------- Keyword----------- Curiosities

 

After “Sync” the "Photo 4" NEF now shows all four variables changed, so that in its NEF File:

Location ______ = ______ Erwhon

Title _________ = ______ Courtyard of the baths

Category ______ = ______ Textures

Keyword _______ = ______ Curiosities (Plus what tags were there before)

 

Both “Sync” from changes made in the Catalog tags panel, and changes “Saved” from the Properties panel appear to function the same way.

 

This behaviour is consistent with the discrepancies I have written about in previous postings on this topic. For example thousands of photos have a "Title" in the NEF which I intended for one photo.

 

Enough for now; got a few other things to do tonight.

[Murat]

Murat, If I understand correctly;

When I shut the program down before it completes a sync write, the program saves it's sync status. When the program is re-started, syncing does not start automatically but waits till after a tag is changed.

 

If there is a program crash, would the sync status still be available, or would it not have been saved?

 

Yes, exactly. We'll improve this feature on the next program version:

- The Syncing Thread State (Paused or Active) will be restored after the program restart.

[franky68]

...this actually is one of the (many) reasons why I believe a "unclean flag" is better than a queue. If (for whatever reason) not all files are synced, its pretty easy to display a warning on files that are out of sync - without having to look them up in the queue table.

 

Also this would eliminate most of the effort to maintain a queue.

[Murat]

...this actually is one of the (many) reasons why I believe a "unclean flag" is better than a queue. If (for whatever reason) not all files are synced, its pretty easy to display a warning on files that are out of sync - without having to look them up in the queue table.

 

Also this would eliminate most of the effort to maintain a queue.

 

Franky, to be honest I hate "Settings" as an UI component. This is why Daminion still doesn't have any settings option. We'll try avoid this as long as possible ;)

 

PG36, I appreciate for the assistance with discovering the bug with syncing tags with metadata! The bug has been fixed. A new update will be available tomorrow.

 

The problem is related to RAW files only - there was a small issue with caching ExifTool data that we use to write tags to the Camera RAW files. Other formats should be handled correctly.

[pgrondin]

Thanks Murat. We really appreciate the fast turnaround your team delivers.

[franky68]

Hello Murat,

 

Franky, to be honest I hate "Settings" as an UI component. This is why Daminion still doesn't have any settings option. We'll try avoid this as long as possible ;)

 

I guess this belongs to the other thread (disabling autosync via preferences)? However: Most users love the settings part as it gives them the feeling of beeing able to configure an application just they way they want it to - and in a limited way they are even right. At least its better to have cirtain options in the "settings" than to blow up the UI by too many often unneeded stuff.

 

But sure - just go on avoiding it as long as you can. As long as they are comming at all we will be fine ;)

 

The problem is related to RAW files only - there was a small issue with caching ExifTool data that we use to write tags to the Camera RAW files. Other formats should be handled correctly.

 

I agree with Peter - the lightning speed of your reactions to critical bugs is amazing! Great work.

 

...wasn't there a thumbs-up emoticon supposed to come?

[Murat]

Please update your Daminion to 0.8.6.414 version

 

We not released a new build yesterday, because I waited until we attach a progress to Updating Metadata feature, plus we fixed a few minor bugs.

 

...wasn't there a thumbs-up emoticon supposed to come?

Here

[pgrondin]

I ran a quick test on the new version. Seems to be working perfectly.

 

Great Job!!