Monitor progress of Generate AI labels queue?

[k1w]

Is there any way to monitor the progress of generating AI labels? I'm attempting to generate labels for a lot of unlabeled photos (now that the "search for unassigned" bug is fixed), but it seems like a bunch of photos are not getting labeled, even after several hours. It's difficult to tell whether they aren't getting queued, whether the queue is stalled, or whether they were not processed successfully.

When I look at the cvis.txt log file, I see a lot of entries like:

DEBUG 2020-03-19 13:03:35,751 CVis.OrchestratorActor [29] - NextTaskMessage 
INFO  2020-03-19 13:03:35,760 CVis.ImageTask [29] - not found media items for processing 
DEBUG 2020-03-19 13:03:35,762 CVis.OrchestratorActor [29] - WorkDoneMessage 
DEBUG 2020-03-19 13:04:20,775 CVis.OrchestratorActor [29] - NextTaskMessage 
INFO  2020-03-19 13:04:20,791 CVis.ImageTask [29] - not found media items for processing 
DEBUG 2020-03-19 13:04:20,796 CVis.OrchestratorActor [29] - WorkDoneMessage 
DEBUG 2020-03-19 13:05:05,808 CVis.OrchestratorActor [33] - NextTaskMessage 
INFO  2020-03-19 13:05:05,818 CVis.ImageTask [33] - not found media items for processing 
DEBUG 2020-03-19 13:05:05,821 CVis.OrchestratorActor [33] - WorkDoneMessage 
DEBUG 2020-03-19 13:05:50,825 CVis.OrchestratorActor [34] - NextTaskMessage 
INFO  2020-03-19 13:05:50,840 CVis.ImageTask [34] - not found media items for processing 
DEBUG 2020-03-19 13:05:50,843 CVis.OrchestratorActor [34] - WorkDoneMessage 
DEBUG 2020-03-19 13:06:35,848 CVis.OrchestratorActor [35] - NextTaskMessage 
INFO  2020-03-19 13:06:35,859 CVis.ImageTask [35] - not found media items for processing 
DEBUG 2020-03-19 13:06:35,862 CVis.OrchestratorActor [35] - WorkDoneMessage 

When I look at my Google Cloud Vision API metrics, I see 89 requests in the past hour, but none in the past several minutes, and that's after marking 2000 photos in Daminion for labeling.

BTW, in the cvis.txt log, when it does find a file, it seems like it's logging the location of it's thumbnail, not the original photo location. Is there any way to easily determine a actual photo file that has been processed?  Example is: 

DEBUG 2020-03-19 13:31:33,598 CVis.OrchestratorActor [63] - NextTaskMessage 
DEBUG 2020-03-19 13:31:33,606 CVis.ImageTask [63] - 2\4e\24e1df6a-5b7b-482d-84f9-b603eca9b9c5.dat: 42880, 17164 
DEBUG 2020-03-19 13:31:33,610 CVis.ImageTask [63] - file exitst: E:\Users\Daminion\AppData\Roaming\Daminion Server\Thumbnails\2\4e\24e1df6a-5b7b-482d-84f9-b603eca9b9c5.dat 
INFO  2020-03-19 13:31:33,614 CVis.ImageTask [63] - start GetBuffer 
INFO  2020-03-19 13:31:33,624 CVis.ImageTask [63] - end GetBuffer: 42880 
DEBUG 2020-03-19 13:31:33,627 CVis.ImageTask [63] - get image from file 
DEBUG 2020-03-19 13:31:33,651 CVis.ImageTask [63] - start DetectLabels 
DEBUG 2020-03-19 13:31:34,333 CVis.ImageTask [63] - end DetectLabels 
INFO  2020-03-19 13:31:34,337 CVis.ImageTask [63] - [ { "mid": "/m/0838f", "description": "Water", "score": 0.9684016, "topicality": 0.9684016 }, { "mid": "/m/06q40", "description": "Smoke", "score": 0.891961336, "topicality": 0.891961336 }, { "mid": "/m/015s2f", "description": "Water resources", "score": 0.870399833, "topicality": 0.870399833 }, { "mid": "/m/07pw27b", "description": "Atmospheric phenomenon", "score": 0.8569148, "topicality": 0.8569148 }, { "mid": "/m/08t9c_", "description": "Grass", "score": 0.821845353, "topicality": 0.821845353 }, { "mid": "/m/049_3v", "description": "Yard", "score": 0.77845, "topicality": 0.77845 }, { "mid": "/m/025s3q0", "description": "Landscape", "score": 0.7564218, "topicality": 0.7564218 }, { "mid": "/m/0hrcj2p", "description": "Backyard", "score": 0.7451672, "topicality": 0.7451672 }, { "mid": "/m/07j7r", "description": "Tree", "score": 0.686921835, "topicality": 0.686921835 }, { "mid": "/m/0gmc028", "description": "Spray", "score": 0.677393854, "topicality": 0.677393854 } ] 
DEBUG 2020-03-19 13:31:34,342 CVis.ImageTask [63] - start UpdateCloudVisionTable 
DEBUG 2020-03-19 13:31:34,346 CVis.ImageTask [63] - end UpdateCloudVisionTable 
DEBUG 2020-03-19 13:31:34,350 CVis.ImageTask [63] - process label: { "mid": "/m/0838f", "description": "Water", "score": 0.9684016, "topicality": 0.9684016 } 
DEBUG 2020-03-19 13:31:34,354 CVis.ImageTask [63] - labels tag value id: 25 
DEBUG 2020-03-19 13:31:34,357 CVis.ImageTask [63] - process label: { "mid": "/m/06q40", "description": "Smoke", "score": 0.891961336, "topicality": 0.891961336 } 
DEBUG 2020-03-19 13:31:34,361 CVis.ImageTask [63] - labels tag value id: 1494 
DEBUG 2020-03-19 13:31:34,366 CVis.ImageTask [63] - process label: { "mid": "/m/015s2f", "description": "Water resources", "score": 0.870399833, "topicality": 0.870399833 } 
DEBUG 2020-03-19 13:31:34,372 CVis.ImageTask [63] - labels tag value id: 420 
DEBUG 2020-03-19 13:31:34,376 CVis.ImageTask [63] - process label: { "mid": "/m/07pw27b", "description": "Atmospheric phenomenon", "score": 0.8569148, "topicality": 0.8569148 } 
DEBUG 2020-03-19 13:31:34,381 CVis.ImageTask [63] - labels tag value id: 270 
DEBUG 2020-03-19 13:31:34,387 CVis.ImageTask [63] - process label: { "mid": "/m/08t9c_", "description": "Grass", "score": 0.821845353, "topicality": 0.821845353 } 
DEBUG 2020-03-19 13:31:34,393 CVis.ImageTask [63] - labels tag value id: 115 
DEBUG 2020-03-19 13:31:34,398 CVis.ImageTask [63] - process label: { "mid": "/m/049_3v", "description": "Yard", "score": 0.77845, "topicality": 0.77845 } 
DEBUG 2020-03-19 13:31:34,404 CVis.ImageTask [63] - labels tag value id: 2 
DEBUG 2020-03-19 13:31:34,408 CVis.ImageTask [63] - process label: { "mid": "/m/025s3q0", "description": "Landscape", "score": 0.7564218, "topicality": 0.7564218 } 
DEBUG 2020-03-19 13:31:34,412 CVis.ImageTask [63] - labels tag value id: 116 
DEBUG 2020-03-19 13:31:34,418 CVis.ImageTask [63] - process label: { "mid": "/m/0hrcj2p", "description": "Backyard", "score": 0.7451672, "topicality": 0.7451672 } 
DEBUG 2020-03-19 13:31:34,424 CVis.ImageTask [63] - labels tag value id: 106 
DEBUG 2020-03-19 13:31:34,430 CVis.ImageTask [63] - process label: { "mid": "/m/07j7r", "description": "Tree", "score": 0.686921835, "topicality": 0.686921835 } 
DEBUG 2020-03-19 13:31:34,434 CVis.ImageTask [63] - labels tag value id: 55 
DEBUG 2020-03-19 13:31:34,440 CVis.ImageTask [63] - process label: { "mid": "/m/0gmc028", "description": "Spray", "score": 0.677393854, "topicality": 0.677393854 } 
DEBUG 2020-03-19 13:31:34,444 CVis.ImageTask [63] - labels tag value id: 2925 
DEBUG 2020-03-19 13:31:34,451 CVis.OrchestratorActor [63] - NextTaskMessage 

 

[WilfriedB]

14 hours ago, k1w said:

it seems like a bunch of photos are not getting labeled

I noticed this too,and I'm glad not being the only one experiencing this issue 😎, but it seems those are less than 1% of all requests

Just today, I worked with Valery to correct and catch the issue. From 90 items, I identified, he was able to get 78 tagged. Still 12 "refused" being tagged.

14 hours ago, k1w said:

Is there any way to monitor the progress of generating AI labels?

If you are able, to use SQL, there two ways:

1. To see the number of requests for AI Labels this month (up to this second) to avoid exceeding the monthly free 1,000 requests, you can use. However, there is no guarantee, this number is exactly the same as Google's billing. So far, it was correct for me, but I just noticed, Google was counting 3 more than Daminion. Possibly a result of Valery's actions on the database today or two days ago.

select COUNT(*) as CloudVision_requests_this_month from cloudvision c where enddate >= date_trunc('month', CURRENT_DATE); 

2. To see more details, including the item ID:

select c.id_mediaitem as ID, c.requesttime as requested, c.startdate as started, c.enddate as ended, c.error as error, c."value" as results from
    cloudvision c where enddate >= date_trunc('month', CURRENT_DATE)
order by started;

[WilfriedB]

3 hours ago, k1w said:

Is there any way to easily determine a actual photo file that has been processed?

As I said above, the second SQL statement shows you the item IDs, which you can easily enter in to Quick Search field, in order to find all information the item(s). Make sure to  select 'Any Words' 'Item Id', deselect all other options and enter multiple IDs separated by a space.

[k1w]

Thanks WilfriedB. I use SQL in my day job, so trying your suggestions out shouldn't be a problem.  Much appreciated!

[WilfriedB]

On 3/19/2020 at 6:36 PM, k1w said:

but it seems like a bunch of photos are not getting labeled

It seems to me, the following SQL statement reveals the IDs for all items which - for what reason, I don't know - did not get labeled and even when you select them again and choose Action -> Generate Labels, nothing happens:

select * from cloudvision where value = '[ ]'

The bracket space bracket ('[ ]'), is what I found in my database for all - or at least many - of those problematic items.

[k1w]

I am still have issues with some photos with no AI tags not able to be updated. I think that WilfriedB's SQL query with cloudvision.value = '[ ]' is likely the culprit. What I'm not sure of is the best way to manually update the rows in that table for Daminion to recognize that they need to be regenerated?

Should I set column 'value' to NULL, or delete the problematic rows from table cloudvision entirely? And if I do that, are there other tables that need to be updated to keep everything in sync, or is just updated cloudvision enough?

(FYI, I am quite comfortable with writing SQL)