TraceId is not propagating in microservices using micrometer and zipkin

[pSinghDelaplex]

I'm generating request from service 1 to service 2 , traceId and spanId are getting generated in the logs for service 1 but when calls service 2 REST API, traceId and spanId get changed. TraceId should be constant for the complete request and response call but it is not happening it also getting changed. I'm running my zipkin server locally and using WebClient for sending request for service calls.

Microservice 1

build.gradle

    id "org.springframework.boot" version "3.1.9"
    springCloudVersion = "2022.0.5"
    springBootAdminVersion = "3.2.2"
    
    implementation "org.springframework.boot:spring-boot-starter-actuator"
    implementation "org.springframework.boot:spring-boot-starter-integration"
    implementation "org.springframework.boot:spring-boot-starter-log4j2"
    implementation "org.springframework.boot:spring-boot-starter-validation"
    implementation "org.springframework.boot:spring-boot-starter-web"
    implementation "org.springframework.boot:spring-boot-starter-webflux"
    implementation "org.springframework.integration:spring-integration-http"
    implementation "org.springframework.cloud:spring-cloud-config-client"
    implementation "org.springframework.integration:spring-integration-ip:6.2.2"
    **implementation 'io.micrometer:context-propagation:latest.integration'
    implementation 'io.micrometer:micrometer-tracing-bridge-brave'
    implementation 'io.zipkin.reporter2:zipkin-reporter-brave'**

application.yml

management:
  endpoints.web.exposure.include: "*"
  endpoint.health.show-details: always
  tracing:
    enabled: true
    sampling:
      probability: 1.0
zipkin:
  tracing:
    endpoint: http://localhost:9411
logging:
  config: classpath:log4j2-spring.yml
  file.name: service1.log    

SpringBootApplication.java

  import brave.sampler.Sampler;
    @Bean
  public Sampler defaultSampler()
  {
    return Sampler.ALWAYS_SAMPLE;
  }

log4j2-spring.yml

    RollingFile:
      - name: RollingFile_Appender
        fileName: ${log-path}/${sys:LOG_FILE}
        filePattern: "${log-path}/${sys:LOG_FILE}.%d{yyyy-MM-dd}T%d{HH-mm-ss}.%i.log"
        PatternLayout:
          pattern: "[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%X{traceId},%X{spanId}] [%-5level] [%5T] [%-25logger{1}]: %message%n"

---

Microservice 2

build.gradle

    id "org.springframework.boot" version "3.2.5"
    springCloudVersion = "2023.0.1"
    springBootAdminVersion = "3.2.2"
    
    implementation "org.springframework.boot:spring-boot-starter-actuator"
    implementation "org.springframework.boot:spring-boot-starter-integration"
    implementation "org.springframework.boot:spring-boot-starter-log4j2"
    implementation "org.springframework.boot:spring-boot-starter-validation"
    implementation "org.springframework.boot:spring-boot-starter-web"
    implementation "org.springframework.boot:spring-boot-starter-webflux"
    implementation "org.springframework.integration:spring-integration-http"
    implementation "org.springframework.cloud:spring-cloud-config-client"
    implementation "org.springframework.integration:spring-integration-ip:6.2.2"
    **implementation 'io.micrometer:context-propagation:latest.integration'
    implementation 'io.micrometer:micrometer-tracing-bridge-brave'
    implementation 'io.zipkin.reporter2:zipkin-reporter-brave'**

application.yml

management:
  endpoints.web.exposure.include: "*"
  endpoint.health.show-details: always
  tracing:
    enabled: true
    sampling:
      probability: 1.0
zipkin:
  tracing:
    endpoint: http://localhost:9411
logging:
  config: classpath:log4j2-spring.yml
  file.name: service2.log    

SpringBootApplication.java

  import brave.sampler.Sampler;
    @Bean
  public Sampler defaultSampler()
  {
    return Sampler.ALWAYS_SAMPLE;
  }

log4j2-spring.yml

    RollingFile:
      - name: RollingFile_Appender
        fileName: ${log-path}/${sys:LOG_FILE}
        filePattern: "${log-path}/${sys:LOG_FILE}.%d{yyyy-MM-dd}T%d{HH-mm-ss}.%i.log"
        PatternLayout:
          pattern: "[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%X{traceId},%X{spanId}] [%-5level] [%5T] [%-25logger{1}]: %message%n"

[wilkinsona]

Thanks for the report. Unfortunately, it doesn't include enough information for us to be able to diagnose the problem. For example, you haven't shown how the requests are being made (all we know is that you're using WebClient) or what's receiving them (are you using Spring MVC, Spring WebFlux, something else).

If you would like us to spend some more time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.

[pSinghDelaplex]

Hi @wilkinsona

I've created a own myWebClient class to define the WebClient . Calling this get method to calling the service 2 api's. I don't have issues with inter service call request and response. Only having issue with the same traceId is not propagating from service 1 to service2. Let me know if there is any missing dependencies or yml properties or any other related configurations is there.

  public ServiceResponse<T> get(String baseURL, String apiEndpoint, Map<String, Object> params) {
    ServiceResponse<T> response = (ServiceResponse)this.getWebClient(baseURL).get().uri(apiEndpoint, params).retrieve().bodyToMono(ServiceResponse.class).retryWhen(this.retries()).block();
    return response;
  }

import io.netty.channel.ChannelOption;
import io.netty.handler.timeout.ReadTimeoutHandler;
import io.netty.handler.timeout.WriteTimeoutHandler;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.stereotype.Component;
import org.springframework.web.reactive.function.client.ExchangeStrategies;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.client.HttpClient;

@Component
public class MyWebClient {

  protected WebClient getWebClient(String url) {
    return WebClient.builder().exchangeStrategies(ExchangeStrategies.builder().codecs((configurer) -> {
      configurer.defaultCodecs().maxInMemorySize(Integer.parseInt(this.webClientProperties.getInMemorySize()));
    }).build()).clientConnector(new ReactorClientHttpConnector(this.getHttpClient())).baseUrl(url).build();
    
  }

  private HttpClient getHttpClient()
  {
    return HttpClient.create()
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, Integer.parseInt(webClientProperties.getConnectionTimeout()))
        .doOnConnected(conn -> conn
            .addHandlerLast(new ReadTimeoutHandler(Integer.parseInt(webClientProperties.getReadTimeout())))
            .addHandlerLast(new WriteTimeoutHandler(Integer.parseInt(webClientProperties.getWriteTimeout()))));
  }

[wilkinsona]

There's still a lot that's missing, I'm afraid. For example, I can't easily tell what propagation format service 1 is producing and what propagation format(s) service 2 is consuming. I can also see that you're creating some components yourself which means that you'll lose Spring Boot's corresponding auto-configuration.

As I said above, if you would like us to spend some more time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue. If you can't provide us with such a sample then I'm afraid that we'll probably have to close this issue as it is not actionable in its current form.

[himanshunp]

@pSinghDelaplex You have to use WebClient.builder as autowired bean otherwise observability won't work. so your method should be like following:
@Bean("myWebClient") public WebClient getWebClient(WebClient.builder builder, String url) { return builder.exchangeStrategies...
let me know if this helps

[pSinghDelaplex]

Hi @himanshunp It helped me out. Thankful for your suggestion