Regression: Temporary git unavailability uninstalls applications

[siretart]

In my homelab, I've encoutered ocassional removal of argocd managed applications after upgrading to argocd 2.11. Upon further investigation, this always happens when my git server becomes temporarily unavailable.

I'm using https://github.com/argoproj-labs/argocd-autopilot to manage my homelab. In that setup, argocd-autopilot sets up a repository that defines an ApplicationSet that comes with two git generators:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  creationTimestamp: null
  name: homelab
  namespace: argocd
spec:
  ignoreApplicationDifferences:
    - jsonPointers:
        - /spec/syncPolicy
  generators:
  - git:
      files:
      - path: apps/**/homelab/config.json
      repoURL: https://gitea.redacted.io/redacted/argocd-autopilot.git
      requeueAfterSeconds: 20
      revision: ""
      template:
        metadata: {}
        spec:
          destination: {}
          project: ""
          source:
            repoURL: ""
  - git:
      files:
      - path: apps/**/homelab/config_dir.json
      repoURL: https://gitea.redacted.io/redacted/argocd-autopilot.git
      requeueAfterSeconds: 20
      revision: ""
      template:
        metadata: {}
        spec:
          destination: {}
          project: ""
          source:
            directory:
              exclude: '{{ exclude }}'
              include: '{{ include }}'
              jsonnet: {}
              recurse: true
            repoURL: ""
  syncPolicy:
    # Determines whether the controller will delete Applications when an ApplicationSet is deleted.
    preserveResourcesOnDeletion: true
  template:

After inspecting the code around https://github.com/similark/argo-cd/blob/997f5c7da304efa209a32668d768966d75ee8c2e/applicationset/controllers/applicationset_controller.go#L128-L141 a git blame brought me to #17062 which I believe is causing this regression. Basically, the thinking in that PR is "if the number of generated projects is more than one, then it is save to take the resulting list as good.

In the scenario that happens quite regulariliy, this isn't the case when argocd is unable to contact the the git repository for one, but not both git generators.

Version

argocd-server: v2.11.0+d3f33c0

Logs

2024-05-07 20:23:45.656	time="2024-05-08T00:23:45Z" level=info msg="end reconcile" applicationset=argocd/homelab requeueAfter=20s
2024-05-07 20:23:45.606	time="2024-05-08T00:23:45Z" level=info msg="generated 10 applications" applicationset=argocd/homelab generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:apps/**/homelab/config_dir.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:&ApplicationSourceDirectory{Recurse:true,Jsonnet:ApplicationSourceJsonnet{ExtVars:[]JsonnetVar{},TLAs:[]JsonnetVar{},Libs:[],},Exclude:{{ exclude }},Include:{{ include }},},Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
2024-05-07 20:23:45.596	time="2024-05-08T00:23:45Z" level=info msg="generated 8 applications" applicationset=argocd/homelab generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:apps/**/homelab/config.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
2024-05-07 20:23:44.635	time="2024-05-08T00:23:44Z" level=info msg="end reconcile" applicationset=argocd/cluster-resources requeueAfter=20s
2024-05-07 20:23:44.610	time="2024-05-08T00:23:44Z" level=info msg="generated 1 applications" applicationset=argocd/cluster-resources generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:bootstrap/cluster-resources/*.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
2024-05-07 20:23:44.581	time="2024-05-08T00:23:44Z" level=info msg="end reconcile" applicationset=argocd/cluster-resources requeueAfter=20s
2024-05-07 20:23:44.558	time="2024-05-08T00:23:44Z" level=info msg="generated 1 applications" applicationset=argocd/cluster-resources generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:bootstrap/cluster-resources/*.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
2024-05-07 20:23:44.549	time="2024-05-08T00:23:44Z" level=info msg="end reconcile" applicationset=argocd/cluster-resources requeueAfter=20s
2024-05-07 20:23:44.518	time="2024-05-08T00:23:44Z" level=info msg="generated 1 applications" applicationset=argocd/cluster-resources generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:bootstrap/cluster-resources/*.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
2024-05-07 20:23:44.487	time="2024-05-08T00:23:44Z" level=error msg="error generating application from params" applicationset=argocd/cluster-resources error="error generating params from git: error retrieving Git files: rpc error: code = Internal desc = unable to resolve git revision : Get \"https://gitea.redacted.io/redacted/argocd-autopilot.git/info/refs?service=git-upload-pack\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:bootstrap/cluster-resources/*.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
2024-05-07 20:23:44.486	time="2024-05-08T00:23:44Z" level=error msg="error generating params" error="error generating params from git: error retrieving Git files: rpc error: code = Internal desc = unable to resolve git revision : Get \"https://gitea.redacted.io/redacted/argocd-autopilot.git/info/refs?service=git-upload-pack\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" generator="&{0xc00095b780}"
2024-05-07 20:23:10.167	time="2024-05-08T00:23:10Z" level=info msg="end reconcile" applicationset=argocd/homelab requeueAfter=20s
2024-05-07 20:23:10.115	time="2024-05-08T00:23:10Z" level=info msg="generated 10 applications" applicationset=argocd/homelab generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:apps/**/homelab/config_dir.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:&ApplicationSourceDirectory{Recurse:true,Jsonnet:ApplicationSourceJsonnet{ExtVars:[]JsonnetVar{},TLAs:[]JsonnetVar{},Libs:[],},Exclude:{{ exclude }},Include:{{ include }},},Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
2024-05-07 20:23:10.103	time="2024-05-08T00:23:10Z" level=info msg="generated 8 applications" applicationset=argocd/homelab generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:apps/**/homelab/config.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"

[siretart]

cc: @todaywasawesome @similark PTAL

[todaywasawesome]

Likely related to #14318