[kube-prometheus-stack] Uses ServiceMonitors for kube-system components instead of PodMonitors

[PryceA]

Is your feature request related to a problem ?

The way that scrape configurations are created for things like kubeControllerManager and coreDns uses a ServiceMonitor. This of course requires that a service be created for these which in-turn requires a change in the kube-system namespace. If instead a PodMonitor is used then the deployment can be limited to only the namespace specified.

This is desirable to limit the scope of changes made to a single namespace.

Describe the solution you'd like.

Ideally a deployment would not make any changes outside the specified namesapce (eg. monitoring) and certainly not in kube-system.
By using PodMonitor instead of ServiceMonitor there is less dependency on what is deployed in kube-system.

Describe alternatives you've considered.

Disable these scrape configs and re-implement them as PodMonitors in our values.yaml.

Additional context.

Examples of services made just so ServiceMonitor can be used:

helm-charts/charts/kube-prometheus-stack/templates/exporters/kube-etcd/endpoints.yaml

Line 10 in 67e796b

namespace: kube-system

helm-charts/charts/kube-prometheus-stack/templates/exporters/kube-proxy/service.yaml

Line 10 in 67e796b

namespace: kube-system

helm-charts/charts/kube-prometheus-stack/templates/exporters/kube-dns/service.yaml

Line 10 in 67e796b

namespace: kube-system

helm-charts/charts/kube-prometheus-stack/templates/exporters/core-dns/service.yaml

Line 10 in 67e796b

namespace: kube-system

[PryceA]

One challenge I ran into when re-implementing them in PodMonitors was that the job label shifted to include the namespace it was deployed to. For example monitoring/coredns instead of just coredns. Not a big deal on its own, but there are lots of built-in dashboards that include a job label filter. So there are some subtle differences between ServiceMonitor and PodMonitor.