Empty objects vs. null objects cause repeated syncs

[pgier]

Summary

When a config option is not set in the a kube manifest (e.g. statefulset) ArgoCD continually tries to re-sync the object because it sees a diff between an empty and undefined object.

Motivation

I have a statefulset that does not define the resources section. ArgoCD continually tries to sync the object because Kubernetes automatically injects an empty object.

Proposal

The diff calculator should ignore the difference between an empty and null object in a kubernetes resource field.

Related

#15554

[blakeromano]

Is this with Server Side Diff on or off? I would imagine looking at https://argo-cd.readthedocs.io/en/stable/user-guide/diff-strategies/#server-side-diff may help with scenarios like this. I don't believe there's plans to make the default Server Side Diff until a 3.x release but this may be something you want to take a look at...

[evanrich]

Is this with Server Side Diff on or off? I would imagine looking at https://argo-cd.readthedocs.io/en/stable/user-guide/diff-strategies/#server-side-diff may help with scenarios like this. I don't believe there's plans to make the default Server Side Diff until a 3.x release but this may be something you want to take a look at...

i just tried enabling that and it didn't seem to make a difference on my environment

[evanrich]

FWIW, this works for me for the "resources: {}" diff

      ignoreDifferences:
      - group: apps
        kind: Deployment
        jqPathExpressions:
        - '.spec.template.spec.containers[].resources'
      - group: apps
        kind: StatefulSet
        jqPathExpressions:
        - '.spec.template.spec.containers[].resources'

[pgier]

Is this with Server Side Diff on or off?

I haven't set any options for server side diff in the application or the main configmap, so it should be off.
On the application I only have these sync options set.

        syncOptions:
        - CreateNamespace=true
        - ServerSideApply=true

@evanrich Thanks, I was hoping I wouldn't have to ignore resources for the cases where there are real differences, but I can probably use that for now.

[pgier]

This is very strange, I have multiple clusters with basically the same configuration.
AWS EKS (same region)
Kube version: 1.28.8
ArgoCD version: v2.10.7+b060053

Same application (external-secrets) deployed to both clusters.
One of the clusters shows a diff on the empty vs. null resources and continually tries to resync.
The other cluster shows the same difference if you look at the live manifest vs. desired manifest, but does not display anything on the diff tab and does not try to re-sync.

I verified that on the cluster which is not showing a diff and therefore not syncing has resources: {} in the live manifest and nothing defined for resources in the desired manifest. So for some reason this cluster is successfully ignoring that difference even though I didn't configure any ignoreDifferences.

So same application running in the same version of Argo and Kubernetes seems to sometimes ignore this difference and sometimes not ignore it.

[batazor]

Noticed this problem with version 2.11 quite a lot

Adding this to ignoreDifferences doesn't seem like a good idea to me

[javydekoning]

This has suddenly emerged in my k3s cluster as well.

Kubernetes version: v1.30.0+k3s1
Argo: v2.11.2