Debugging a container that gives you nothing: exit codes, thread dumps and ephemeral tools

Problem

A container fails and all you get is an exit code and perhaps a log line. The tempting response is to add a shell, curl, a debugger and a package manager to the image so that next time you can look around. That is the wrong trade. Production images should be small and boring, and your tools should arrive only when needed and leave no trace.

Options and trade-offs

• Bake debugging tools into the image: convenient, but it enlarges the attack surface and the image.
• Add an admin endpoint: useful for version and uptime, but it is one more thing to secure.
• Use the platform: logs, exit codes, signals, and a separate short-lived container that shares the target's network.

My recommendation

Start with the exit code and the last logs. Use signals the JVM already understands, such as a thread dump on SIGQUIT, before reaching for tools. Make crashes leave evidence, and attach any tooling from outside.

Worked example

The first look should cost nothing. Exit code 137 usually means the process was killed, often for memory, and 143 means it was asked to stop:

docker ps -a --filter name=api
docker inspect --format '{{.State.ExitCode}} oom={{.State.OOMKilled}}' api
docker logs --since 15m api

The JVM prints a full thread dump to standard output when it receives SIGQUIT, so you need no tools inside the container:

docker kill --signal=SIGQUIT api
docker logs --tail 200 api

Make a crash leave something behind. These flags write a heap dump on an out-of-memory error to a mounted volume:

services:
  api:
    image: demo/api:1.4.2
    environment:
      JAVA_TOOL_OPTIONS: >-
        -XX:+HeapDumpOnOutOfMemoryError
        -XX:HeapDumpPath=/dumps
        -XX:+ExitOnOutOfMemoryError
    volumes:
      - ./dumps:/dumps

And have the application log one structured line when it dies, so the exit code has context:

public class Main {
    public static void main(String[] args) {
        Thread.setDefaultUncaughtExceptionHandler((thread, error) -> {
            System.err.printf("{\"level\":\"FATAL\",\"thread\":\"%s\",\"error\":\"%s\"}%n",
                    thread.getName(), error.getClass().getName());
            error.printStackTrace();
        });
        Runtime.getRuntime().addShutdownHook(new Thread(() ->
                System.err.println("{\"level\":\"INFO\",\"msg\":\"shutdown\"}")));
        // start the application
    }
}

When you need to poke around the network or a volume, use a throwaway container that shares the target's network namespace:

docker run --rm -it --network=container:api -v api-data:/data:ro alpine sh
# inside: wget -qO- http://localhost:8080/health

Checklist

1. Read the exit code, OOMKilled flag and recent logs before changing anything.
2. Use SIGQUIT for a thread dump instead of installing tools.
3. Configure heap dumps on out-of-memory errors, with a size cap and a clean-up plan.
4. Log one structured fatal line with no secrets.
5. Attach tools from a separate short-lived container and remove it afterwards.

Where this doesn't apply

If the failure is inside a native library or the container never starts, thread dumps will not help. You will need the platform's events and the host's logs.

One honest limitation

An outside container can see the network and volumes but not the process's memory. Heap dumps are sensitive too, since they can contain secrets, so store them carefully and delete them quickly.

Sources

• Docker Docs: docker container logs
• Docker Docs: docker container kill (signals)
• Oracle Java Documentation: Troubleshooting guide (thread dumps and heap dumps)