Add Scaler that Read Metrics From Current Custom Metrics Adapter

[imroc]

Metric Type and Kubernetes Metric API

There are several metric types when defining HPA: Resource , Pods, Object and External:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: test
spec:
  metrics:
    - type: Pods
       pods:
       ...

• Resource type uses Resource Metrics API(v1beta1.metrics.k8s.io).
• Pods type and Object type uses Custom Metrics API(v1beta1.custom.metrics.k8s.io).
• External type uses External Metrics API(v1beta1.external.metrics.k8s.io).

KEDA occupies the External Metric API, and provide cpu and memory triggers which read metrics from current Resource Metrics API adapter, but no trigger read metrics from current Custom Metrics API adapter.

Scenario: Migrate HPA from Pods or Object metric type

Some cloud vendors provide rich metrics for HPA by default. For example, Tencent TKE provide a lot of HPA metrics for users: https://www.tencentcloud.com/document/product/457/34025

And these metrics are based on Custom Metrics API, which means it has a default adapter of Custom Metrics API, users can define HPA like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test
  minReplicas: 1
  maxReplicas: 100
  metrics:
    - pods:
        metric:
          name: k8s_pod_rate_cpu_core_used_limit
        target:
          averageValue: "80"
          type: AverageValue
      type: Pods
    - pods:
        metric:
          name: k8s_pod_rate_mem_usage_limit
        target:
          averageValue: "80"
          type: AverageValue
      type: Pods
    - pods:
        metric:
          name: k8s_pod_rate_gpu_used_request
        target:
          averageValue: "60"
          type: AverageValue
      type: Pods

But if users want to use KEDA to add some triggers to the same workload, they need to delete the previously defined HPA because KEDA and HPA cannot be used together, and KEDA didn't provide a trigger that can read metrics from current Custom Mtrics API, so this prevents users from migrating to KEDA.

Proposal: Add Scaler that Read Metrics From Current Custom Metrics API Adapter

The Pods and Object type metric have multiple levels of definitions, and metadata is a map[string]string. It is not possible to directly move existing HPA pods and object metric definitions to metadata. We need to consider how to design it.

[imroc]

Maybe it's better to allow keep existing metrics spec when resue an existing HPA?

For example, add scaledobject.keda.sh/keep-existing-hpa-metrics-spec annotation to ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    scaledobject.keda.sh/keep-existing-hpa-metrics-spec: "true"
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      name: test

Then KEDA only add extra external metric spec to HPA metrics list, keep existing HPA metrics spec in the HPA metrics list.

[imroc]

Maybe it's better to allow keep existing metrics spec when resue an existing HPA?

This is not friendly to GitOps, for example, use ArgoCD to manage YAMLs, and it both include HPA and ScaledObject which reuses HPA, the KEDA will change exsiting HPA's spec, and ArgoCD found that HPA is been changed, then it change it back to original defination.

[imroc]

I think I found the best solution, add a field to horizontalPodAutoscalerConfig, let's say it metrics, we can paste HPA'sspec.metrics to ScaledObject's spec.advanced.horizontalPodAutoscalerConfig.metrics:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: test
  namespace: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test
  pollingInterval: 15
  minReplicaCount: 1
  maxReplicaCount: 100
  advanced:
    horizontalPodAutoscalerConfig:
      metrics:
        - pods:
            metric:
              name: k8s_pod_rate_cpu_core_used_limit
            target:
              averageValue: "80"
              type: AverageValue
          type: Pods
        - pods:
            metric:
              name: k8s_pod_rate_mem_usage_limit
            target:
              averageValue: "80"
              type: AverageValue
          type: Pods
        - pods:
            metric:
              name: k8s_pod_rate_gpu_used_request
            target:
              averageValue: "60"
              type: AverageValue
          type: Pods
  triggers:
    - type: cron
      metadata:
        timezone: Asia/Shanghai
        start: 30 9 * * *
        end: 30 10 * * *
        desiredReplicas: "10"

And KEDA then populate the metrics spec to the HPA that been managed to KEDA, just like keep the behaviour field.

This should be a small but very useful change.