roachprod: changes in promhelper client for modify labels

[nameisbhaskar]

When a test is run, a new label has to be added to prometheus. To do this, we need to first get the current config, add the new label and update the config. This change does that. Post this change, the label update will be integrated.

The change also needs a change in httputil client GET function to support headers being passed.

Fixes: #124319
Epic: none

[cockroach-teamcity]

This change is 

[DarrylWong]

Can you explain to me why we need to update the config this way? By the time we call cluster.Start the first time, we should already the test name and the run ID, is there a reason we can't add it all in one shot?

[nameisbhaskar]

Can you explain to me why we need to update the config this way? By the time we call cluster.Start the first time, we should already the test name and the run ID, is there a reason we can't add it all in one shot?

If a cluster is reused, do we always stop the nodes and then add the tags and start? If that is the case, we should be good without this change.

[DarrylWong]

Yes, that's correct.

If we reuse a cluster, we call c.WipeForReuse.

cockroach/pkg/cmd/roachtest/test_runner.go

Lines 636 to 637 in 99d7687

	// We found a test to run on this cluster. Wipe the cluster.
	if err := c.WipeForReuse(ctx, l, testToRun.spec.Cluster); err != nil {

This eventually calls c.Stop

cockroach/pkg/roachprod/install/cluster_synced.go

Lines 547 to 549 in 6918532

	func (c *SyncedCluster) Wipe(ctx context.Context, l *logger.Logger, preserveCerts bool) error {
	display := fmt.Sprintf("%s: wiping", c.Name)
	if err := c.Stop(ctx, l, 9, true /* wait */, 0 /* maxWait */, ""); err != nil {

Same with removing labels. We call this at the end of runTest

cockroach/pkg/cmd/roachtest/test_runner.go

Lines 1031 to 1033 in 99d7687

	defer func() {
	t.end = timeutil.Now()
	if err := c.removeLabels([]string{VmLabelTestName}); err != nil {

[nameisbhaskar]

Thats great! Then we should be good without this change. I will close this PR! Thanks for pointing out!!