Unable to create Scaled Object with Oauthbearer SASL type with Confluent Cloud cluster

[acartag7]

Report

We are trying to setup Kafka Scaled object with our confluent cloud dedicated cluster using SASL/OAUTHBEARER authentication and we are getting authentication failures.

"error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed during authentication due to invalid credentials with SASL mechanism OAUTHBEARER"}

I've tried following the documentation here, but seems like there is a problem with the extensions part as during my tests, I saw the following:

If I set the oauthExtensions: invalid=nothing I get the usual logicalCluster is missing a cluster_id

"error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed: 1 extensions are invalid! They are: logicalCluster: CLUSTER_ID_MISSING_OR_EMPTY"}

Now if I set up oauthExtensions: extension_identityPoolId=pool-ebYj without the cluster id I get the authentication failed message (the same if I put the the pool id and cluster id as in the manifests below):

"error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed during authentication due to invalid credentials with SASL mechanism OAUTHBEARER"}

I know its not the credentials as when I input incorrect credentials I get the following message in the operator directly from microsoft Entra ID:

"error": "error creating kafka client: kafka: client has run out of available brokers to talk to: oauth2: \"unauthorized_client\" \"XXXXXXXXX: Application with identifier 'YYYYY-YYYY-YYYY-YYYY-YYYYYYYYYxxx' was not found in the directory 'XXX AG'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant. Trace ID: xxxx-xxxx-xxx-xxxx Correlation ID: xxxx-xxxx-xxx-xxxx Timestamp: 2024-04-30 16:00:34Z\" \"https://login.microsoftonline.com/error?code=700016\""}

The Scaledobject using sasl plaintext and the api keys is working without issues but we can't use this auth method in our setup.

I think this issue hasn't been reported before, any ideas on what I could try?

Expected Behavior

The Kafka scaler is active, in ready status and the deployments scale properly.

Actual Behavior

Authentication Fails with sasl/oauthbearer.

Steps to Reproduce the Problem

To test this you will need a kafka cluster and you should authenticate with SASL/OAUTHBEARER
Deploy the manifests below:

kind: Secret
apiVersion: v1
metadata:
  name: keda-secrets-0004
stringData:
  username: superusername
  password: superpassword
  oauthTokenEndpointUri: https://login.microsoftonline.com/tenant_id/oauth2/token
  scopes: "superusername/.default"
  oauthExtensions: extension_logicalCluster=lkc-x8ff65,extension_identityPoolId=pool-xyzs
type: Opaque
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-authentication-0004
spec:
  secretTargetRef:
    - parameter: username
      name: keda-secrets-0004
      key: username
    - parameter: password
      name: keda-secrets-0004
      key: password
    - parameter: oauthTokenEndpointUri
      name: keda-secrets-0004
      key: oauthTokenEndpointUri
    - parameter: scopes
      name: keda-secrets-0004
      key: scopes
    - parameter: oauthExtensions
      name: keda-secrets-0004
      key: oauthExtensions
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scaler-0004
spec:
  minReplicaCount: 0
  maxReplicaCount: 3
  scaleTargetRef:
    name: kafka-inspector-keda-test
    apiVersion: apps/v1
    kind: Deployment
  pollingInterval:  60
  triggers:
    - type: kafka
      authenticationRef:
        name: trigger-authentication-0004
      metadata:
        bootstrapServers: lkc-x8ff65.privdomxxxxx.eu-central-1.aws.confluent.cloud:9092
        topic: test-topic
        consumerGroup: app.myconsumer-dev
        lagThreshold: '50'
        activationLagThreshold: '0'
        offsetResetPolicy: latest
        sasl: "oauthbearer"
        tls: "enable"
---
# create simple nginx deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kakfa-inspector-keda-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kakfa-inspector-keda-test
  template:
    metadata:
      labels:
        app: kakfa-inspector-keda-test
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Logs from KEDA operator

2024-04-30T15:13:57Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scaler-0004","namespace":"default"}, "namespace": "default", "name": "scaler-0004", "reconcileID": "16bc803b-9af0-4f8a-8685-268e32428f04"}
2024-04-30T15:13:57Z    INFO    Creating a new HPA      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scaler-0004","namespace":"default"}, "namespace": "default", "name": "scaler-0004", "reconcileID": "16bc803b-9af0-4f8a-8685-268e32428f04", "HPA.Namespace": "default", "HPA.Name": "keda-hpa-scaler-0004"}
2024-04-30T15:14:19Z    ERROR   scale_handler   error resolving auth params     {"type": "ScaledObject", "namespace": "default", "name": "scaler-0004", "scalerIndex": 0, "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed during authentication due to invalid credentials with SASL mechanism OAUTHBEARER"}
2024-04-30T15:14:19Z    ERROR   Error getting scalers   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scaler-0004","namespace":"default"}, "namespace": "default", "name": "scaler-0004", "reconcileID": "16bc803b-9af0-4f8a-8685-268e32428f04", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed during authentication due to invalid credentials with SASL mechanism OAUTHBEARER"}
2024-04-30T15:14:19Z    ERROR   Failed to create new HPA resource       {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scaler-0004","namespace":"default"}, "namespace": "default", "name": "scaler-0004", "reconcileID": "16bc803b-9af0-4f8a-8685-268e32428f04", "HPA.Namespace": "default", "HPA.Name": "keda-hpa-scaler-0004", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed during authentication due to invalid credentials with SASL mechanism OAUTHBEARER"}
2024-04-30T15:14:19Z    ERROR   failed to ensure HPA is correctly created for ScaledObject      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scaler-0004","namespace":"default"}, "namespace": "default", "name": "scaler-0004", "reconcileID": "16bc803b-9af0-4f8a-8685-268e32428f04", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed during authentication due to invalid credentials with SASL mechanism OAUTHBEARER"}
2024-04-30T15:14:19Z    ERROR   Reconciler error        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scaler-0004","namespace":"default"}, "namespace": "default", "name": "scaler-0004", "reconcileID": "16bc803b-9af0-4f8a-8685-268e32428f04", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: kafka server: SASL Authentication failed: Authentication failed during authentication due to invalid credentials with SASL mechanism OAUTHBEARER"}

KEDA Version

2.14.0

Kubernetes Version

1.27

Platform

Any

Scaler Details

Kafka

Anything else?

No response

[JorTurFer]

@dttung2905 @zroubalik , you are the kafka experts 😄

[fabiodellanna]

Hi all, I experienced the same issue reported by @acartag7
I'm not able to autoscale a cluster with oauthbearer authentication. No news so far?

[dttung2905]

sorry I missed this. Let me take a look into this in the next few days.Probably need to create a local strimzi kafka cluster with SASL/OAUTHBEARER authentication first 😆