The Hidden Price of "One More Log": Observability Costs on the JVM

Where the cost comes from

• Formatting. Building a message string you never emit is wasted work.
• Allocation. Varargs arrays, boxed values, and message objects add garbage.
• I/O and locking. A synchronous appender can block request threads on disk or on a shared lock.
• Volume. Ten log lines per request at high traffic is a data pipeline you have to pay for.
• Cardinality. Metric and span attributes with unbounded values (user IDs, request IDs as labels) explode storage and backend cost.

Logging guardrails

// Bad: concatenation always runs
log.debug("Fetched profile " + id + " in " + elapsedMs + "ms");

// Good: SLF4J formats only if DEBUG is enabled
log.debug("Fetched profile {} in {}ms", id, elapsedMs);

if (log.isDebugEnabled()) {
    log.debug("State: {}", expensiveDump(state));
}

String snapshot;
synchronized (lock) {
    snapshot = state.summary();      // cheap copy
}
log.info("Rebalanced: {}", snapshot);  // logging outside the lock

Tracing guardrails

import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.samplers.Sampler;

SdkTracerProvider provider = SdkTracerProvider.builder()
        .setSampler(Sampler.parentBased(Sampler.traceIdRatioBased(0.05)))   // ~5% of new traces
        .build();

• Keep span attributes low-cardinality. Put high-cardinality identifiers in logs, not metric labels.
• Avoid creating spans in tight loops. Instrument the request and the major dependencies.
• Cap attribute value sizes.

Measure the overhead

import org.openjdk.jmh.annotations.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(java.util.concurrent.TimeUnit.NANOSECONDS)
public class LoggingBenchmark {

    private static final Logger log = LoggerFactory.getLogger(LoggingBenchmark.class);   // level = INFO
    private final String id = "user-12345";
    private final long elapsed = 42;

    @Benchmark
    public void concatenation() {
        log.debug("Fetched profile " + id + " in " + elapsed + "ms");
    }

    @Benchmark
    public void parameterized() {
        log.debug("Fetched profile {} in {}ms", id, elapsed);
    }
}

Failure modes

• High-cardinality labels that overload the metrics backend.
• Log amplification, where one failure triggers retries that each log at ERROR.
• Synchronous appenders blocking request threads.
• Tracing everything in staging, then discovering production cannot afford it.

A starter checklist

1. Parameterized logging everywhere, with guards for expensive arguments.
2. No logging inside locks or tight loops.
3. Async appender with a bounded queue.
4. Parent-based ratio sampling, with the ratio justified by numbers.
5. Attribute and label cardinality reviewed.
6. A benchmark or load test whenever you add instrumentation to a hot path.

Takeaways

• Observability has a runtime cost that shows up in the tail.
• Sample traces, bound log queues, and keep cardinality low.
• Benchmark instrumentation like any other hot-path code.