Starting a local data pipeline in the right order, on a laptop that stays usable

Problem

Everyone has had the local stack that mostly works. The database is still initialising when the API connects, a worker grabs a port something else expected, and a few minutes later the fans are screaming because three containers decided they own every core. None of this is exotic. It comes from three habits: relying on start order instead of readiness, letting ports be whatever they turn out to be, and leaving resource limits at their defaults. All three are cheap to fix in a Compose file.

Options and trade-offs

• Fixed sleeps in an entrypoint: easy to write, wrong on a slow day and wasteful on a fast one.
• Wait-for scripts: they work, but you maintain a script per dependency and each one hides a different definition of ready.
• Healthchecks plus depends_on with condition service_healthy: Compose does the waiting, and readiness is defined once, next to the service it describes.

My recommendation

I keep a small default stack (database and API) and put the heavy pipeline workers behind a profile, so day-to-day work does not pay for ETL. Every service gets a real healthcheck, downstream services wait on health rather than on container start, ports are bound to 127.0.0.1 and written down, and each service has a CPU and memory ceiling.

Worked example

The example below starts a Postgres database and an API by default. The worker only starts when you ask for the etl profile, and it will not start until the database reports healthy.

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: local-only-password
    ports:
      - "127.0.0.1:5433:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 10
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M

  api:
    build: .
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      DB_URL: jdbc:postgresql://db:5432/postgres
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M

  etl-worker:
    build: ./worker
    profiles: ["etl"]
    environment:
      DB_URL: jdbc:postgresql://db:5432/postgres
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: 256M

Start the base stack and wait until everything is healthy, then add the pipeline when you need it:

docker compose up -d --wait
docker compose --profile etl up -d
docker compose ps
docker stats --no-stream

The last command is the quick check that the limits are actually applied. If a container sits at its memory ceiling, that is information, not a bug in the cap.

Checklist

1. Put a real readiness check on every service that others depend on.
2. Use condition: service_healthy instead of sleeps or start-order assumptions.
3. Bind ports explicitly, to 127.0.0.1, and keep them in a small documented range.
4. Set CPU and memory limits on every service, and make workers back off or fail with a clear message when they hit them.
5. Mark throwaway volumes clearly and prune them on a schedule.

Where this doesn't apply

This is a local-development pattern. Profiles are a convenience that can drift from how the real system is orchestrated, so do not treat a passing local stack as proof that production wiring is right.

One honest limitation

Healthchecks add startup time, and a heavy check makes it worse. Keep them to a cheap probe, and remember that a limit set too low turns a slow job into an out-of-memory kill.

Sources

• Docker Docs: Control startup and shutdown order in Compose
• Docker Docs: Using profiles with Compose
• Docker Docs: Compose deploy specification (resources)