Distributed tracing

In a distributed system, a slow request is rarely a slow server. Tracing shows which of the services on the path consumed the time, and which one failed first.

The standard. Request tracing across distributed applications with X-Ray or OpenTelemetry, from instrumentation through context propagation to analysis.

What tracing provides

• End-to-end visibility. A single trace follows a request across API Gateway, Lambda, ECS, RDS, caches and external calls.
• Root-cause analysis. The failing service and the failing span are visible without guesswork.
• Performance attribution. Time is attributed to a specific hop, dependency or query.
• Dependency mapping. The real call graph, derived from traffic, rather than the diagram someone drew.

Instrumentation

AWS-native services

Some services are instrumented by configuration alone:

Service	How
Lambda	Enable active tracing in the function configuration
API Gateway	Enable X-Ray tracing on the stage
Step Functions	Enable tracing on the state machine
SQS / SNS	Trace context propagation enabled where supported

Applications

For EC2, ECS and EKS, the application (or a sidecar) runs the X-Ray daemon and the SDK:

const AWSXRay = require('aws-xray-sdk');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));
const http = AWSXRay.captureHTTPs(require('http'));

app.use(AWSXRay.express.openSegment('example-service'));

app.get('/items', async (req, res) => {
  const subsegment = AWSXRay.getSegment().addNewSubsegment('dynamodb-query');
  try {
    const result = await queryItems(req.query);
    res.json(result);
  } finally {
    subsegment.close();
  }
});

app.use(AWSXRay.express.closeSegment());

OpenTelemetry

Where the customer prefers vendor-neutral instrumentation, OpenTelemetry exports to X-Ray or to a third-party backend:

# OpenTelemetry Collector — X-Ray exporter
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  awsxray:
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]

Trace context propagation

A trace is only end-to-end if the context travels with the request.

Boundary	Mechanism
HTTP	X-Amzn-Trace-Id or W3C traceparent header
Queue	Message attribute carrying the trace header
Event	Event detail field, or traceparent in the envelope
Database	Application-level: record the trace id alongside slow queries
Third party	Pass the header if the vendor supports it; otherwise note the boundary

Where context is lost, the trace stops. The first step in diagnosing an incomplete trace is finding the boundary where propagation was not implemented.

Sampling

Full tracing is expensive at volume. Use a sampling strategy that keeps the traces you need:

Strategy	Use
Fixed-rate sampling	Predictable volume reduction
Reservoir sampling	Even coverage without bursts
Rate-limited sampling	Cap traces per second
Adaptive sampling	Adjust to traffic automatically
Tail-based sampling	Keep all errors and slow traces, sample the rest

The default is a moderate fixed rate, plus retention of every error trace. An error with no trace is the most expensive kind of missing data.

Using traces

Diagnosing latency

1. Open the trace from the alert or the slow-request log entry.
2. Identify the longest span.
3. Distinguish time spent in the service from time waiting on a dependency.
4. Check whether the pattern is uniform or concentrated in one instance.

Diagnosing errors

1. Filter traces by status code or exception.
2. Compare failing and succeeding traces for the same endpoint.
3. Look for the first failing span, not the propagated error at the edge.

Capacity planning

Aggregate span durations by dependency to see which resource saturates first as traffic grows. That is the constraint to address, ahead of the symptom.

Service map

X-Ray builds a service map from observed traffic. Review it monthly:

• Dependencies with high error rates or latency.
• Services with no incoming traffic that should have some.
• Unexpected edges — a service calling another it should not.
• Single points of failure in the call graph.

Evidence

• Trace configuration on Lambda, API Gateway and application runtime.
• A sample trace across at least three services.
• Sampling configuration and rationale.
• Service map export.
• A worked example: an incident or slow request diagnosed from a trace.

Related standards

• Monitoring
• Monitoring metrics
• Runbooks