You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to configure prometheus 2.45.3 to scrape metrics made available from watchtower from the /v1/metrics endpoint. I ran into a dead end when prometheus keeps getting a 401 Unauthorized HTTP error.
Steps to reproduce
Run prometheus and watchtower from docker-compose
In the Targets section of the prometheus UI, observe the watchtower target to fail with a 401 HTTP error
Expected behavior
No authorization error, metrics are scraped.
Screenshots
No response
Environment
Platform: Synology DSM420+
Architecture: Linux
Docker Version: 20.10.23, build 876964a
Prometheus version: 2.45.3
Your logs
watchtower | time="2024-02-28T16:34:01Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
watchtower | time="2024-02-28T16:34:02Z" level=debug msg="Making sure everything is sane before starting"
watchtower | time="2024-02-28T16:34:02Z" level=debug msg="Retrieving running and restarting containers"
watchtower | time="2024-02-28T16:34:02Z" level=debug msg="There are no additional watchtower containers"
watchtower | time="2024-02-28T16:34:02Z" level=info msg="Watchtower 1.7.1"
watchtower | time="2024-02-28T16:34:02Z" level=info msg="Using no notifications"
watchtower | time="2024-02-28T16:34:02Z" level=info msg="Checking all containers (except explicitly disabled with label)"
watchtower | time="2024-02-28T16:34:02Z" level=info msg="Scheduling first run: 2024-02-29 16:34:02 +0000 UTC"
watchtower | time="2024-02-28T16:34:02Z" level=info msg="Note that the first check will be performed in 23 hours, 59 minutes, 59 seconds"
watchtower | time="2024-02-28T16:34:02Z" level=warning msg="Trace level enabled: log will include sensitive information as credentials and tokens"
Additional context
I used the guide from https://containrrr.dev/watchtower/metrics/ to configure both watchtower and prometheus. However, it seems that bearer_token: demo_token is no longer supported in the latest LTS version of prometheus (2.45). I couldn't find any documentation of it. What is documented instead is:
(Note that I'm using and actual long random alphanum string, not demo_token.)
I checked what prometheus actually sends as header with this configuration and it is authorization: Bearer demo_token. So that seem correct. I used wget from the prometheus container to simulate a request to watchtower with the same header:
and get the data back successfully. So I don't understand where this authorization error comes from. I tried to increase logging to trace level but I'm not seeing anything after start up.
Is there anything else I can try to see why the authorization fails? It would be great if I could disable the authorization althogether. Both prometheus and watchtower run on the same internal docker network, so the 8080 port doesn't even have to be exposed to the internet.
Here's how I'm running watchtower from docker-compose:
Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏
I'm facing exactly the same issue. I fiddled in the same way, without any success.
I'm not on a NAS, but on a Debian-based server using Docker 26.0.1 and Prometheus 2.48.0, but with the same results.
Did you find any way around that to make it work in the end?
Describe the bug
I am trying to configure
prometheus
2.45.3 to scrape metrics made available fromwatchtower
from the/v1/metrics
endpoint. I ran into a dead end whenprometheus
keeps getting a 401 Unauthorized HTTP error.Steps to reproduce
prometheus
andwatchtower
from docker-composewatchtower
target to fail with a 401 HTTP errorExpected behavior
No authorization error, metrics are scraped.
Screenshots
No response
Environment
Your logs
Additional context
I used the guide from https://containrrr.dev/watchtower/metrics/ to configure both
watchtower
andprometheus
. However, it seems thatbearer_token: demo_token
is no longer supported in the latest LTS version ofprometheus
(2.45). I couldn't find any documentation of it. What is documented instead is:(Note that I'm using and actual long random alphanum string, not
demo_token
.)I checked what
prometheus
actually sends as header with this configuration and it isauthorization: Bearer demo_token
. So that seem correct. I usedwget
from theprometheus
container to simulate a request towatchtower
with the same header:and get the data back successfully. So I don't understand where this authorization error comes from. I tried to increase logging to trace level but I'm not seeing anything after start up.
Is there anything else I can try to see why the authorization fails? It would be great if I could disable the authorization althogether. Both
prometheus
andwatchtower
run on the same internal docker network, so the 8080 port doesn't even have to be exposed to the internet.Here's how I'm running
watchtower
from docker-compose:And here is the
prometheus
config:The text was updated successfully, but these errors were encountered: