Logs, metrics and trace correlation for a Java service in a container

Problem

Logs, metrics and traces are three different tools, and they only help in an incident when they can be tied together. The common failure is not the lack of data. It is that the data is unstructured, expensive, or impossible to connect to a single request. Containers add a convention on top: write to standard output, and let the platform collect it. Everything else is about keeping that stream useful and cheap.

Options and trade-offs

• Plain-text logs written to files: easy at first, but hard to parse, and the files fill the container's disk.
• Structured JSON logs on standard output: parseable, collected by the platform, and the right default.
• Tracing everything: great detail, but the CPU and storage bill grows with traffic. Sampling keeps the cost predictable.

My recommendation

Log structured JSON to standard output with a small, fixed set of fields. Put a correlation ID on every request at the edge and carry it through the logs. Keep metric labels low cardinality, and sample traces. Alert on user-facing symptoms, not on every raw number.

Worked example

A servlet filter that makes sure every request has a correlation ID, puts it in the logging context, and returns it to the caller.

import jakarta.servlet.*;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.slf4j.MDC;

import java.io.IOException;
import java.util.UUID;

public class CorrelationFilter implements Filter {
    private static final String HEADER = "X-Correlation-Id";

    @Override
    public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
            throws IOException, ServletException {
        String id = ((HttpServletRequest) req).getHeader(HEADER);
        if (id == null || id.isBlank() || id.length() > 64) {
            id = UUID.randomUUID().toString();
        }
        MDC.put("correlation_id", id);
        ((HttpServletResponse) res).setHeader(HEADER, id);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.remove("correlation_id");
        }
    }
}

The Logback configuration that writes each event as one JSON line to standard output, including the correlation ID from the MDC. It uses the open-source Logstash encoder.

<configuration>
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
  </appender>
  <root level="INFO">
    <appender-ref ref="STDOUT"/>
  </root>
</configuration>

Then log with parameters, so nothing is formatted when the level is off:

log.info("request_finished route={} status={} durationMs={}", route, status, elapsedMs);
log.warn("upstream_failed upstream={} reason={}", "inventory", "timeout");

Checklist

1. Log JSON to standard output, with level, timestamp, component, message and correlation ID.
2. Create the correlation ID at the edge, accept a sane inbound one, and cap its length.
3. Keep metric labels to a small set such as route class and outcome, never per user.
4. Sample traces, and raise the rate temporarily during an incident.
5. Alert on error rate and latency, and link the alert to a dashboard or runbook.

Where this doesn't apply

A single small service may not need distributed tracing at all. Structured logs with a correlation ID will already answer most questions.

One honest limitation

Observability has a cost. Every extra log line and span uses CPU and memory, and during an incident, noisy logging can make the incident worse.

Sources

• SLF4J Manual: Mapped Diagnostic Context (MDC)
• Logback Documentation: Appenders
• OpenTelemetry Documentation: Sampling
• Docker Docs: View container logs