Restart-safe stream jobs in containers: checkpoints and idempotent writes

Problem

Stream jobs are different from stateless APIs. They carry progress, they read from an offset, and they have to pick up cleanly after a restart. In a container platform, restarts are routine, not exceptional. If a restart changes your output, the job does not really own its state. Two ideas fix most of this: store progress outside the process, and make every write safe to repeat.

Options and trade-offs

• Keep progress in memory: fast, and all progress is lost on restart.
• Checkpoint after every record: safest, but the overhead can dominate.
• Checkpoint on a bounded interval and make writes idempotent: a crash replays a little work, and the replay changes nothing.

My recommendation

Write idempotent upserts keyed by a stable record ID, store the checkpoint atomically, and on startup resume from the last checkpoint. Treat exactly-once as a goal you approach by combining at-least-once delivery with idempotent effects.

Worked example

The sketch below has no dependencies. It simulates a crash in the middle of a run, then restarts from the checkpoint. The replayed records are written again, but the output does not change.

import java.io.IOException;
import java.nio.file.*;
import java.util.*;

public class IdempotentJob {

    record Event(String id, long offset, String value) { }

    static final Path CHECKPOINT = Path.of("checkpoint.txt");
    static final Map<String, String> sink = new TreeMap<>();   // upsert by id

    static long readCheckpoint() throws IOException {
        return Files.exists(CHECKPOINT) ? Long.parseLong(Files.readString(CHECKPOINT).strip()) : 0L;
    }

    static void writeCheckpoint(long offset) throws IOException {
        Path tmp = Path.of("checkpoint.tmp");
        Files.writeString(tmp, Long.toString(offset));
        Files.move(tmp, CHECKPOINT, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
    }

    static void run(List<Event> source, long crashAfterOffset) throws IOException {
        long start = readCheckpoint();
        int sinceCheckpoint = 0;
        for (Event e : source) {
            if (e.offset() < start) continue;
            sink.put(e.id(), e.value());                 // idempotent: same key, same value
            if (++sinceCheckpoint == 5) {
                writeCheckpoint(e.offset() + 1);
                sinceCheckpoint = 0;
            }
            if (e.offset() == crashAfterOffset) {
                throw new IllegalStateException("simulated crash at offset " + e.offset());
            }
        }
        writeCheckpoint(source.size());
    }

    public static void main(String[] args) throws IOException {
        Files.deleteIfExists(CHECKPOINT);
        List<Event> source = new ArrayList<>();
        for (int i = 0; i < 20; i++) source.add(new Event("evt-" + i, i, "value-" + i));

        try {
            run(source, 7);                              // crashes mid-run
        } catch (IllegalStateException crash) {
            System.out.println(crash.getMessage() + ", resuming from " + readCheckpoint());
        }
        run(source, -1);                                 // restart: replays 5..7, then finishes
        System.out.println("records in sink = " + sink.size());   // 20, not 23
    }
}

In a container, the checkpoint file would live on a volume or, better, in an external store, so it survives the container being replaced. The same idea applies when the source is a message broker with committed offsets and the sink supports upserts or transactions.

Checklist

1. Persist progress outside the container.
2. Write the checkpoint atomically, so a crash never leaves it half-written.
3. Key writes by a stable ID, and make repeating them harmless.
4. On startup, read the checkpoint and resume, and log what you recovered without logging payloads.
5. Pace catch-up after an outage with bounded batches and concurrency.
6. Test by killing the process mid-run and comparing the output with an uninterrupted run.

Where this doesn't apply

This sketch is single-threaded with an in-memory sink, to show the mechanics. Real jobs also need partitioning, ordering guarantees per key, and a sink that is itself durable.

One honest limitation

Idempotency limits what a write can be. Operations like incrementing a counter are not safe to repeat, so they need a composite key, a version, or a transactional sink to be handled correctly.

Sources

• Apache Kafka Documentation: Message delivery semantics
• Oracle Java Documentation: java.nio.file.Files.move (atomic move)
• Docker Docs: Volumes