[kube-prometheus-stack] Cannot access Grafana dashboard

[mohammed-bachrouch]

I am setting up a Grafana dashboard and it is not showing up from the browser.

I have been facing an error with my Grafana setup.
I currently have a Prometheus service with Grafana (kube-prometheus-stack helm chart) running on an EKS cluster inside a VPC. I set up the values for the helm chart as such:

grafana:
  adminPassword: '${grafana_admin_password}'
  assertNoLeakedSecrets: false
  ingress:
    enabled: true
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/certificate-arn: '${certificate_arn}'
    hosts: ['grafana-${region}-${env}.${domain_name}']
  grafana.ini:
    server:
      root_url: 'https://grafana-${region}-${env}.${domain_name}'

Then, I create a Route53 record pointing to the created load balancer.

When trying to access this host, the browser's network hangs for a long time, then errors with a connection refused.

Here is the Terraform for my helm release:

resource "helm_release" "prometheus" {
  name       = "kube-prometheus-stack"
  namespace  = var.namespace
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  version    = "58.1.3"

  values = [templatefile("${path.module}/file/values.tftpl", {
    prefix      = var.prefix,
    env         = var.env,
    region      = var.region,
    domain_name = var.domain_name,
    grafana_admin_password = var.password,
  })]
}

Upon checking the associated ALB in AWS, I see this error: Unhealthy: Health checks failed with these codes: [302].

What am I missing here?

• I looked everywhere online to find a solution, and tried replicating those solutions, with not much help. For example, I tried adding pathType: ImplementationSpecific and serve_from_sub_path: true and domain: grafana-${region}-${env}.${domain_name} but nothing fixed the issue.
• I tried using localhost. That did not fix it either.

Any help is appreciated.

[mohammed-bachrouch]

Any updates on this?

[antoinecampbell]

@mohammed-bachrouch You may need to modify the health check used by your ALB to include 302 as a valid status code.
https://stackoverflow.com/a/49389860/1510175

[mohammed-bachrouch]

Thanks for this @antoinecampbell. It turns out that the issue was coming from network issue in my office.