微服务全链路跟踪：jaeger集成istio，并兼容uber-trace-id与b3

微服务全链路跟踪：grpc集成zipkin

微服务全链路跟踪：grpc集成jaeger

微服务全链路跟踪：springcloud集成jaeger

微服务全链路跟踪：jaeger集成hystrix

微服务全链路跟踪：jaeger增加tag参数

公司有自己的一套基于k8s的paas系统，并且集成了istio，这里主要是想讲解下springcloud服务如何集成istio

jaeger跨进程传递

在基于HTTP协议的分布式调用中，通常会使用HTTP Header来传递SpanContext的内容。
常见的Wire Protocol包含Zipkin使用的b3 HTTP header，Jaeger使用的uber-trace-id HTTP Header,LightStep使用的"x-ot-span-context" 等。
Istio1.0支持b3 header和x-ot-span-context header,可以和Zipkin,Jaeger及LightStep对接；istio1.4以上支持uber-trace-id。
注：请参考github官方说明：https://github.com/istio/istio/issues/12400

uber-trace-id

图中可以看到其中traceId、spanId等字段都拼接到一个key中了

b3

istio的b3头详情可以参考：https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-request-id
b3并没有将字段都拼接，二十单个传递，下面是案例：

X-B3-TraceId:427fde2dc7edb084
X-B3-ParentSpanId:427fde2dc7edb084
X-B3-SpanId:827e270489aafbd7
X-B3-Sampled:1

变更jaeger传输为b3

如果需要集成istio的jaeger，则需要将传输方式修改为b3
微服务全链路跟踪：springcloud集成jaeger该章中已经描述了如何集成jaeger，这里只需要修改一个配置enable-b3-propagation，如下

opentracing:jaeger:enable-b3-propagation: true// 默认为falseudp-sender:host: localhostport: 6831remote-reporter:flush-interval: 1000max-queue-size: 5000log-spans: trueprobabilistic-sampler:sampling-rate: 1

这样springboot服务就可以与istio中的jaeger信息串起来形成完整全链路。

兼容uber-trace-id与b3

现在遇到了另外一个问题，公司已经很早就搭建了一套全链路jaeger，并且已经接入了大部分系统，采用的是默认的header传输，即：uber-trace-id
而下游有很多paas内部系统都是非java的不方便接入jaeger，只是注入了istio，并自动注入了jaeger-agent，这里使用的是b3头传输，这就导致了部分链路上下游无法串联起来。而如果需要统一传输方式暂不现实，首先如果都改成b3，则需要上游很多已接入的系统修改配置为b3，如果是都改成uber-trace-id，istio当前版本不支持，如果需要升级istio，则需要升级kubernetes，风险比较大，所以这里根据实际情况先采用集成两种头，即上游都是uber-trace-id，到中间层服务时手动注入b3相关头如下。

grpc注入b3头

这里需要使用grpc的拦截器

import com.google.common.collect.ImmutableMap;
import io.grpc.*;
import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.propagation.Format;
import io.opentracing.propagation.TextMap;
import lombok.extern.slf4j.Slf4j;import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.Set;
import java.util.concurrent.TimeUnit;/** * An intercepter that applies tracing via OpenTracing to all client requests. */
@Slf4j
public class ClientTracingInterceptor implements ClientInterceptor {private final Tracer tracer;private final OperationNameConstructor operationNameConstructor;private final boolean streaming;private final boolean verbose;private final Set<ClientRequestAttribute> tracedAttributes;private final ActiveSpanSource activeSpanSource;private final  Metadata.Key<String> b3TraceIdKey = Metadata.Key.of("X-B3-TraceId", Metadata.ASCII_STRING_MARSHALLER);private final  Metadata.Key<String> b3SpanIdKey = Metadata.Key.of("X-B3-SpanId", Metadata.ASCII_STRING_MARSHALLER);private final  Metadata.Key<String> b3ParentSpanIdKey = Metadata.Key.of("X-B3-ParentSpanId", Metadata.ASCII_STRING_MARSHALLER);private final  Metadata.Key<String> b3SampledKey = Metadata.Key.of("X-B3-Sampled", Metadata.ASCII_STRING_MARSHALLER);/*** @param*/public ClientTracingInterceptor(Tracer tracer) {this.tracer=tracer;this.operationNameConstructor = OperationNameConstructor.DEFAULT;this.streaming = false;this.verbose = false;this.tracedAttributes = new HashSet<ClientRequestAttribute>();this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT;}private ClientTracingInterceptor(Tracer tracer, OperationNameConstructor operationNameConstructor, boolean streaming,boolean verbose, Set<ClientRequestAttribute> tracedAttributes, ActiveSpanSource activeSpanSource) {this.tracer = tracer;this.operationNameConstructor = operationNameConstructor;this.streaming = streaming;this.verbose = verbose;this.tracedAttributes = tracedAttributes;this.activeSpanSource = activeSpanSource;}/*** Use this intercepter to trace all requests made by this client channel.* @param channel to be traced* @return intercepted channel*/ public Channel intercept(Channel channel) {return ClientInterceptors.intercept(channel, this);}@Overridepublic <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {final String operationName = operationNameConstructor.constructOperationName(method);Span activeSpan = this.activeSpanSource.getActiveSpan();final Span span = createSpanFromParent(activeSpan, operationName);for (ClientRequestAttribute attr : this.tracedAttributes) {switch (attr) {case ALL_CALL_OPTIONS:span.setTag("grpc.call_options", callOptions.toString());break;case AUTHORITY:if (callOptions.getAuthority() == null) {span.setTag("grpc.authority", "null");} else {span.setTag("grpc.authority", callOptions.getAuthority());                        }break;case COMPRESSOR:if (callOptions.getCompressor() == null) {span.setTag("grpc.compressor", "null");} else {span.setTag("grpc.compressor", callOptions.getCompressor());}break;case DEADLINE:if (callOptions.getDeadline() == null) {span.setTag("grpc.deadline_millis", "null");} else {span.setTag("grpc.deadline_millis", callOptions.getDeadline().timeRemaining(TimeUnit.MILLISECONDS));}break;case METHOD_NAME:span.setTag("grpc.method_name", method.getFullMethodName());break;case METHOD_TYPE:if (method.getType() == null) {span.setTag("grpc.method_type", "null");} else {span.setTag("grpc.method_type", method.getType().toString());}break;case HEADERS:break;}}return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {@Overridepublic void start(Listener<RespT> responseListener, Metadata headers) {if (verbose) {span.log("Started call");}if (tracedAttributes.contains(ClientRequestAttribute.HEADERS)) {span.setTag("grpc.headers", headers.toString()); }tracer.inject(span.context(), Format.Builtin.HTTP_HEADERS, new TextMap() {@Overridepublic void put(String key, String value) {log.info("jaeger key:{},value:{}",key,value);Metadata.Key<String> headerKey = Metadata.Key.of(key, Metadata.ASCII_STRING_MARSHALLER);headers.put(headerKey, value);String[] mm=value.split("%3A");if("uber-trace-id".equals(key)&&mm.length==4){headers.put(b3TraceIdKey,mm[0]);log.info("jaeger traceId:{}",mm[0]);headers.put(b3SpanIdKey,mm[1]);headers.put(b3ParentSpanIdKey,mm[2]);headers.put(b3SampledKey,mm[3]);}}@Overridepublic Iterator<Entry<String, String>> iterator() {throw new UnsupportedOperationException("TextMapInjectAdapter should only be used with Tracer.inject()");}});Listener<RespT> tracingResponseListener = new ForwardingClientCallListener.SimpleForwardingClientCallListener<RespT>(responseListener) {@Overridepublic void onHeaders(Metadata headers) {if (verbose) { span.log(ImmutableMap.of("Response headers received", headers.toString())); }delegate().onHeaders(headers);}@Overridepublic void onMessage(RespT message) {if (streaming || verbose) { span.log("Response received"); }delegate().onMessage(message);}@Override public void onClose(Status status, Metadata trailers) {if (verbose) { if (status.getCode().value() == 0) { span.log("Call closed"); }else { span.log(ImmutableMap.of("Call failed", status.getDescription())); }}span.finish();delegate().onClose(status, trailers);}};delegate().start(tracingResponseListener, headers);}@Override public void cancel(@Nullable String message, @Nullable Throwable cause) {String errorMessage;if (message == null) {errorMessage = "Error";} else {errorMessage = message;}if (cause == null) {span.log(errorMessage);} else {span.log(ImmutableMap.of(errorMessage, cause.getMessage()));}delegate().cancel(message, cause);}@Overridepublic void halfClose() {if (streaming) { span.log("Finished sending messages"); }delegate().halfClose();}@Overridepublic void sendMessage(ReqT message) {if (streaming || verbose) { span.log("Message sent"); }delegate().sendMessage(message);}};}private Span createSpanFromParent(Span parentSpan, String operationName) {if (parentSpan == null) {return tracer.buildSpan(operationName).startManual();} else {return tracer.buildSpan(operationName).asChildOf(parentSpan).startManual();}}/*** Builds the configuration of a ClientTracingInterceptor.*/public static class Builder {private Tracer tracer;private OperationNameConstructor operationNameConstructor;private boolean streaming;private boolean verbose;private Set<ClientRequestAttribute> tracedAttributes;private ActiveSpanSource activeSpanSource;  /*** @param tracer to use for this intercepter* Creates a Builder with default configuration*/public Builder(Tracer tracer) {this.tracer = tracer;this.operationNameConstructor = OperationNameConstructor.DEFAULT;this.streaming = false;this.verbose = false;this.tracedAttributes = new HashSet<ClientRequestAttribute>();this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT;} /*** @param operationNameConstructor to name all spans created by this intercepter* @return this Builder with configured operation name*/public Builder withOperationName(OperationNameConstructor operationNameConstructor) {this.operationNameConstructor = operationNameConstructor;return this;} /*** Logs streaming events to client spans.* @return this Builder configured to log streaming events*/public Builder withStreaming() {this.streaming = true;return this;}/*** @param tracedAttributes to set as tags on client spans*  created by this intercepter* @return this Builder configured to trace attributes*/public Builder withTracedAttributes(ClientRequestAttribute... tracedAttributes) {this.tracedAttributes = new HashSet<ClientRequestAttribute>(Arrays.asList(tracedAttributes));return this;}/*** Logs all request life-cycle events to client spans.* @return this Builder configured to be verbose*/public Builder withVerbosity() {this.verbose = true;return this;}/*** @param activeSpanSource that provides a method of getting the *  active span before the client call* @return this Builder configured to start client span as children *  of the span returned by activeSpanSource.getActiveSpan()*/public Builder withActiveSpanSource(ActiveSpanSource activeSpanSource) {this.activeSpanSource = activeSpanSource;return this;}/*** @return a ClientTracingInterceptor with this Builder's configuration*/public ClientTracingInterceptor build() {return new ClientTracingInterceptor(this.tracer, this.operationNameConstructor, this.streaming, this.verbose, this.tracedAttributes, this.activeSpanSource);}}public enum ClientRequestAttribute {METHOD_TYPE,METHOD_NAME,DEADLINE,COMPRESSOR,AUTHORITY,ALL_CALL_OPTIONS,HEADERS}
}

主要改动点在header那一块

feign注入b3头

@Configuration
public class FeignConfig implements RequestInterceptor {@Autowiredprivate final Tracer tracer;@Overridepublic void apply(RequestTemplate requestTemplate) {ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();if(attributes!=null&&attributes.getRequest()!=null){HttpServletRequest request = attributes.getRequest();JaegerSpanContext context=(JaegerSpanContext) tracer.activeSpan().context();requestTemplate.header("X-B3-TraceId",String.valueOf(context.getTraceId()));requestTemplate.header("X-B3-SpanId", String.valueOf(context.getSpanId()));requestTemplate.header("X-B3-ParentSpanId", String.valueOf(context.getParentId()));requestTemplate.header("X-B3-Sampled", context.isSampled()?"1":"0");}}@BeanLogger.Level feignLoggerLevel() {return Logger.Level.FULL;}}

经过手动注入后，就可以实现上下游串起来，暂时达到目标，后面的方案是统一传输方式，慢慢升级。