Why are my photos being deleted

[gregeeh]

Hi all,

Here is the script I run daily of my NAS to update Google Photos.

#!/bin/bash
# Purpose = Sync Google Photos

VENV="/root/gphotos/bin"
TARGET="/gphotos/storage"
APP=$VENV/gphotos-sync
FLAGS="--skip-video --omit-album-date --do-delete --flush-index --progress"
LOGFILE=$TARGET/gphotos_full.log

$VENV/python $APP $TARGET $FLAGS $@ >> $LOGFILE --logfile /tmp 2>&1

Here is the relevant part of the gphotos_full.log

Note all the deletions at the end of the log.

The deletions did occur but the links to them in albums have not been removed. The issue is these photos are still on Google Photos and have never been deleted.

Why are these photos being deleted?

TIA

EDIT:
The following scheduled sync downloaded the files that were deleted.

Log is here.

[gilesknap]

There is a long standing bug with the Google API which is that it fails to deliver all the results from a search. It starts sending empty responses before it should and if you keep asking for new pages then eventually more results appear.

I hold little hope that they will ever fix this see

• https://issuetracker.google.com/issues/151032572
• https://issuetracker.google.com/issues/129250413

gphotos-sync tries to get around it by retrying the next page multiple times but I guess that its not always enough.

I've just looked in the code (for the first time in quite a while) and it looks like it will always ask for another page if there is a next page token in the response - and even if the response has zero content. The relevant lines are here.

gphotos-sync/src/gphotos_sync/GooglePhotosIndex.py

Lines 209 to 219 in 1516059

	next_page = items_json.get("nextPageToken")
	if next_page:
	items_json = self.search_media(
	page_token=next_page,
	start_date=start_date,
	end_date=self.end_date,
	do_video=self.include_video,
	favourites=self.favourites,
	)
	else:
	break

If you are interested in capturing more info on this then if you could send a log of it happening with loglevel=trace then we could see what Google API is sending. But if I'm right about the cause then I don't know there is a fix other that to stop doing a full rescan (but that does mean keeping backups files you have deleted from the library)

[gregeeh]

Thanks for your reply.

I've run the app several times this morning and it has not deleted any photos, but it also has not downloaded the photos it incorrectly deleted the last time it was run.

It seems it deletes the same photos when the deletion occurs, which seems a little odd. Also the deletions don't delete the links in the albums folders, which I would have thought it would.

Presently it is not downloading the incorrectly deleting photos and has not deleted any more photos that should not have been deleted.

I have attached the logs from one of this mornings runs.

gphotos-logs.zip

[gilesknap]

Because you are doing flush-index the whole library will be indexed each time. Therefore I would expect the deleted photos to come back if the issue was transient. I've seen this kind of thing before and blamed the Google Photos API - and if you look at the tickets I linked above a Google engineer admits that similar issues are seen but then there have been no further comments for years.

If you look in the trace file you posted above do you see any mention of your missing files names?

[gregeeh]

Therefore I would expect the deleted photos to come back

Me too but they are not.

If you look in the trace file you posted above do you see any mention of your missing files names?

No, they are not there.

Also why when deleting photos, incorrectly, why are the links to them in the albums not deleted too?

[gilesknap]

If your files are not in trace then that is because Microsoft API is not sending them. Since you are using --flush-index that means that we have no filters and are asking for all photos, thus this seems to be a bug at the server end. Perhaps you could try not using flush index but setting a start date before the missing files - that way we are not asking for many many pages from one API call.

Regarding the links that is a bug in gphotos sync perhaps it is related to the --omit-date feature. Could you also try running without that?