Add ability to view traces inside tilt

[a13x5]

To improve observability it will be good to have integrations with popular tracing solutions like Jaeger and Zipkin. Right now I see it something like this: New tab in tilt's UI called traces, which can be something like recent grafana tracing viewer. Ability to go throughout all traces is not necessary, better have clickable trace IDs inside app's logs. Traces can be fetched from tracing agent inside k8s cluster via REST API calls. We can fetch only traces that appears in logs or all traces from the agent. Tracing agent must be separate for a stand (for example it can be spun up with help of tilt extensions). Ability to share traces with snapshots is also important, because traces can add a lot more context to help with debugging.

[wu-victor]

Hi @Alex-Sizov : Thanks for the great idea! Which use case is more important to you?

• See historical usage data of Tilt and the resources and it's performance (e.g. traces like you mentioned) for myself, as an individual developer using Tilt.
• See historical data of other users on my team, using the same Tiltfile and repo and resources.
• See historical data aggregated across multiple/all users on my team.
• See aggregated traces/data.
• See an individual trace.
• Something else.

[a13x5]

@victorwuky Thank you for a quick reply! I was thinking more about seeing an individual trace, but with an ability to share it across a team. I'll try to describe our use case and maybe it will be more clear for you. So the use case is follows: As a developer I make experimetal changes in my code. When I test it and find an error i can see the trace ID in a logs or in an erroneous response. To see a trace I need to go to a separate UI (like Jaeger UI) and search it there. If I want to share a trace I need to give an access to my Jaeger instance to my team or (in a worst case) share screenshots. As you can see we have two major issues with tilt stands + tracing:

• switching between apps(UIs)
• troubles with sharing traces

Second one was partially resolved by using same Jaeger instance across all developers, but this approach is kinda bad, especially if some cyclic trace appears and that instance ultimately became just a dump.

[wu-victor]

Great thanks @Alex-Sizov . Makes sense.

[ebenoist]

@Alex-Sizov We actually use a combination of jaeger and traefik with Tilt to enable a lot of what you're looking for. I'd be happy to put up a post on how we manage it. Sharing traces can be a bit harder, but Jaeger's JSON import/export function works pretty well.

[asheeshv]

@Alex-Sizov - Would you be able to share the details of how you are able to get jaeger and traefik working with Tilt? I'm looking at how to integrate tracing in my local dev cluster, but haven't been able to do so yet. Any examples would definitely be useful. Thanks, Asheesh

[ebenoist]

docker-compose.yml

  jaeger:
    image: jaegertracing/all-in-one:1
    ports:
      - "5775"
      - "6831"
      - "6832"
      - "5778"
      - "16686"
      - "14268"
      - "14250"
      - "9411"
    environment:
      COLLECTOR_ZIPKIN_HTTP_PORT: 9411
    labels:
      - "traefik.http.routers.jaeger.rule=Host(`jaeger.devo.test`)"
      - "traefik.http.routers.jaeger.tls=true"
      - "traefik.http.services.jaeger.loadbalancer.server.port=16686"
  traefik:
    image: traefik:v2.6
    ports:
      - "443:443"
      - "80:80"
      - "8080:8080"
    environment:
      - LOG_LEVEL=INFO
      - ES7_ENCODED_CREDENTIALS
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - "${PWD}/traefik/:/etc/traefik/"
      - "${PWD}/certs/:/etc/certs/"

traefik.yml

entryPoints:
  web:
    address: ":80"

  websecure:
    address: ":443"

accessLog: {}

serversTransport:
  insecureSkipVerify: true

log:
  level: INFO

tracing:
  serviceName: devo
  jaeger:
    samplingServerURL: http://jaeger:5778/sampling
    localAgentHostPort: 127.0.0.1:6831
    gen128Bit: true
    traceContextHeaderName: "x-request-id"
    collector:
      endpoint: http://jaeger:14268/api/traces?format=jaeger.thrift