New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachprod: prometheus file_sd_config has missing labels #124319

Closed

nameisbhaskar opened this issue May 17, 2024 · 1 comment · Fixed by #124598

Assignees

Labels

C-bug

Contributor

nameisbhaskar commented May 17, 2024 •

edited

Previously when the gce_sd_configs was used, few labels were read from the gce metadata using the config below:

source_labels: [__meta_gce_private_ip]
separator: ;
regex: (.*)
target_label: instance
replacement: ${1}:9100
action: replace
source_labels: [__meta_gce_instance_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}-sys
action: replace
source_labels: [__meta_gce_instance_name]
separator: ;
regex: (.*)
target_label: host
replacement: ${1}
action: replace
source_labels: [__meta_gce_label_cluster]
separator: ;
regex: (.*)
target_label: cluster
replacement: ${1}
action: replace
source_labels: [__meta_gce_tags]
separator: ;
regex: (.*)
target_label: instance_tags
replacement: ${1}
action: replace
source_labels: [__meta_gce_label_test_name]
separator: ;
regex: (.*)
target_label: test_name
replacement: ${1}
action: replace
source_labels: [__meta_gce_label_test_run_id]
separator: ;
regex: (.*)
target_label: test_run_id
replacement: ${1}
action: replace

Now, as we are using file_sd_configs, these configs has to be fed to prometheus in a different way.

The approach can be to add the labels directly to the file using the pormhelpers

Jira issue: CRDB-38823

nameisbhaskar added the C-bug label

nameisbhaskar self-assigned this

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom labels for prometheus

ee4026d

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the ame values are set from roachprod now.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom labels for prometheus

19a58b7

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the ame values are set from roachprod now.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

9d7b0c8

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

0e4acd1

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

70f9d4d

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

efbe3b2

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

46fc5aa

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

777e87e

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

202a5d7

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

bd0abbe

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

87e86c1

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none

nameisbhaskar mentioned this issue

roachprod: add custom tags for prometheus config #124409

Merged

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

cb19272

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

37acdbf

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

2ff83c7

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

97d0125

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

7784dac

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

a2bf4e5

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

d7a6f95

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

ebad0c2

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

69b6585

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

0fc302a

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

b69253d

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

6c75397

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

ad0d260

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

17f03b8

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: add custom tags for prometheus config

9e30f6f

few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

0dc719f

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

craig bot pushed a commit that referenced this issue


          Merge #124409

d146ecf

124409: roachprod: add custom tags for prometheus config r=srosenberg a=nameisbhaskar

few of the labels are missing in prometheus with the change from "gce_sd_configs" to "file_sd_configs". This is because these were extracted from the VM metadata which is not available now. So, the same values are set from roachprod now.

Informs: #124319
Epic: none

Co-authored-by: Bhaskarjyoti Bora <bhaskar.bora@cockroachlabs.com>

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client to for modify labels

d90d136

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar mentioned this issue

roachprod: changes in promhelper client for modify labels #124460

Closed

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

e8b1b11

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

be97a53

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

571e452

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

4de7986

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

e41e003

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

87646e9

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

2787ec1

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: changes in promhelper client for modify labels

When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none

Contributor Author

nameisbhaskar commented May 23, 2024

The issue is fixed with the previous commit as we set the test tags at start and we always restart the clusters before running a new test.

nameisbhaskar closed this as completed

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: condition to support cockroach-ephemeral

a93aff0

only the clusters in cockroach-ephemeral should be selected
for scrapping. So, adding a condition for the same.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar mentioned this issue

roachprod: condition to support cockroach-ephemeral #124598

Merged

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: condition to support cockroach-ephemeral

d6728f3

only the clusters in cockroach-ephemeral should be selected
for scrapping. So, adding a condition for the same.

Fixes: cockroachdb#124319
Epic: none

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue


          roachprod: fix access to cloud storage

cc2c63b

The access to the cloud storage for accessing the secrets
is failing from teamcity. This change should fix the same.

Fixes: cockroachdb#124319
Epic: none

craig bot pushed a commit that referenced this issue


          Merge #124598

46b6fab

124598: roachprod: condition to support cockroach-ephemeral r=herkolategan a=nameisbhaskar

only the clusters in cockroach-ephemeral should be selected for scrapping. So, adding a condition for the same.

Fixes: #124319
Epic: none

Co-authored-by: Bhaskarjyoti Bora <bhaskar.bora@cockroachlabs.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment