Connectors Remain Stuck in 'Deleting' State; Unable to Remove from UI

[quareth]

After scheduling several connectors for deletion, they became stuck in the "Deleting" state. Subsequently, all indexing processes began to fail. Despite waiting for an extended period with no changes, I rebooted the server and rebuilt the Docker image, which resolved the indexing issues. I can now delete newly indexed connectors without problems. However, the older connectors remain stuck in the "Deleting" state. I'm uncertain whether this is an issue with the UI or if these connectors are still present in the database. Is there a way to remove these stuck connectors directly from the UI?

All of them web connectors.

[shametim]

maybe the 'background' container is no longer running?

[quareth]

Background container looks fine. When I start Run Complete Re-indexing, Connectors start to do indexing again and looks like able to do indexing. However, their state never changes from "Deleting", which UI not allowing new Schedule for Deletion because it is currently Deleting. But this Delete state never changes and Connector never deleted.

[zarlor]

Just wanted to add (having posted this on the Discord without checking here, sorry) that I've been running into the same issue and the background container seems to otherwise work fine. In fact adding a new file connector, while slow, does seem to get indexed eventually but I think the deleting connectors cause some major delays if one of them seems to get stuck or the like

[zarlor]

Potential duplicate, I might add, of #1202 and #1275, so it seems to be a consistent problem that others are running into as well.

[acidgeneration]

Same issue with google drive connector, cant delete it

[quareth]

If for some reason the Document set sync cannot be completed, it cannot be removed manually later. Mine remained in sync due to disk fullness and this state does not change even if I delete the Connector after solving the disk problem. I am writing here thinking that the reason for this problem may be the same.

[Weves]

hey @acidgeneration / @quareth we put in a couple improvements to the syncing reliability. If you pull in the latest do you still run into this problem?

[mmanolakeris]

Running with the latest and still having this issue with both of the Confluence connectors

Edit:
Background container logs in docker also seem to be fine, apart from the consistent spam of:

Document set '2' is already syncing. Skipping.

2024-05-24 21:19:14 [2024-05-24 19:19:13,982: INFO/MainProcess] Task check_for_document_sets_sync_task[de75cce7-bb1f-49e7-a2c7-9322685fb531] succeeded in 0.013694385008420795s: None
2024-05-24 21:19:14 05/24/2024 07:19:13 PM celery.py 195 : Document set '2' is already syncing. Skipping.
2024-05-24 21:19:18 [2024-05-24 19:19:18,056: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task)
2024-05-24 21:19:19 [2024-05-24 19:19:18,985: INFO/MainProcess] Task check_for_document_sets_sync_task[77149f64-d292-4676-ae97-e9efaba4ab6c] received
2024-05-24 21:19:19 [2024-05-24 19:19:18,988: INFO/MainProcess] Document set '2' is already syncing. Skipping.
2024-05-24 21:19:19 [2024-05-24 19:19:18,998: INFO/MainProcess] Task check_for_document_sets_sync_task[77149f64-d292-4676-ae97-e9efaba4ab6c] succeeded in 0.012558657996123657s: None
2024-05-24 21:19:19 05/24/2024 07:19:18 PM celery.py 195 : Document set '2' is already syncing. Skipping.

The document set was unrelated to these connectors that are being deleted.

Edit 2: Both connectors were, however, connected to another document set which got deleted when I triggered the connectors to delete.
After rebuilding the image I was able to remove one of them, but now the remaining connector throws this:

background-1 | 05/24/2024 07:41:51 PM connector_deletion.py 119 : Cleaning up Document Sets for CC Pair with ID: '13'
background-1 | 05/24/2024 07:41:51 PM celery.py 89 : Failed to run connector_deletion due to Cannot delete CC pair while it is attached to a document set that is syncing. Please wait for the document set to finish syncing, and then try again.
background-1 | Traceback (most recent call last):
background-1 | File "/app/danswer/background/celery/celery.py", line 83, in cleanup_connector_credential_pair_task
background-1 | return delete_connector_credential_pair(
background-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
background-1 | File "/app/danswer/background/connector_deletion.py", line 183, in delete_connector_credential_pair
background-1 | cleanup_synced_entities(cc_pair, db_session)
background-1 | File "/app/danswer/background/connector_deletion.py", line 121, in cleanup_synced_entities
background-1 | mark_cc_pair__document_set_relationships_to_be_deleted__no_commit(
background-1 | File "/app/danswer/db/document_set.py", line 294, in mark_cc_pair__document_set_relationships_to_be_deleted__no_commit
background-1 | raise ValueError(
background-1 | ValueError: Cannot delete CC pair while it is attached to a document set that is syncing. Please wait for the document set to finish syncing, and then try again.

I also tried navigating to:
http://localhost:3005/admin/documents/sets/13
but I get: