Skip to content

DuckHaven — Operator Runbook

Operational procedures for the single control-plane box plus its agents. For how the pieces fit together, see Architecture.

1. Bring up the control plane

The control plane is one docker compose stack (deploy/docker-compose.yml) of ten services: postgres, objectstore, objectstore-bootstrap, polaris-bootstrap, polaris, api, agent, and the otel-collector/tempo/grafana observability trio. The api service publishes port 8000 directly on the host. See Install for a first-time walkthrough.

  1. (Optional) create deploy/.env. Defaults work — SECRET_KEY is generated on first boot and persisted under /var/duckhaven/secrets in the api_data volume. POSTGRES_PASSWORD is not generated; it falls back to the literal duckhaven, so set it here on anything but a private box you trust. Set values in .env only if you need to override them (e.g. pinning a release tag):
DUCKHAVEN_IMAGE_TAG=v1.2.3

Images are published to ghcr.io/tamasmrtn/duckhaven-{api,agent} by .github/workflows/build.yml: :latest on every main push, :v1.2.3 / :v1.2 / :v1 on git tags. Built for linux/amd64 and linux/arm64. 2. Start the stack: make compose-up. Migrations apply automatically. 3. Read the one-shot setup token: docker compose -f deploy/docker-compose.yml cp api:/var/duckhaven/setup_token ./setup_token && cat ./setup_token. 4. Open http://<host>:8000 and create the first admin from the setup screen using the token. 5. The API publishes port 8000 on the host. It speaks plain HTTP, so keep it on a private network — a Tailscale/WireGuard tunnel, or a TLS reverse proxy if it must be reachable more widely.

2. Register additional agents

Agents dial home with a one-time bootstrap token; the control plane never initiates connections.

For each agent host:

  1. In the admin UI, mint a bootstrap token from Compute → Generate bootstrap (single-use, 24 h).
  2. On the agent host, build/pull the agent image and set its .env:
CONTROL_PLANE_URL=ws://<control-plane-tailscale>:8000/agents/connect
BOOTSTRAP_TOKEN=<token-from-step-1>
# Operator ceilings (non-overridable by per-query requests):
MAX_TIMEOUT_S=600
RESULT_RETENTION_HOURS=24
  1. Start the agent (python -m agent.main). It exchanges the bootstrap token for a long-lived agent_session credential, advertises its capabilities, and holds the WebSocket open.
  2. Confirm both agents show green with fresh "last ping" in Admin → Agents (capabilities re-advertise on every heartbeat).

Repeat so at least two agents are registered (e.g. one S3-capable, one local).

3. Exercise the engine selector under load (manual)

  1. Open the worksheet; the engine picker lists both agents with their backend compatibility tags (✓ / ✗).
  2. Run several queries, switching the selected agent per worksheet. Confirm:
  3. Each query is dispatched to the chosen agent (Compute shows query counts per agent; History records agent/user/duration/rows).
  4. Picking an agent that lacks the workspace backend's extension fails fast with an inline "missing <ext> extension" error (server-side check, re-advertise on every heartbeat).
  5. A query that exceeds its timeout is interrupted on the agent and reported as failed (status "timeout"), not left running.
  6. Result range-reads are proxied with the agent's session bearer; a result that has aged past RESULT_RETENTION_HOURS is swept on the agent and the query is re-runnable from saved SQL.

4. Backups & disaster recovery

Schedule nightly Postgres backups

scripts/pg-backup.sh dumps the DuckHaven app state and the Polaris metastore.

# Point backups at a SECOND disk / NAS mount, not the data disk:
sudo cp deploy/systemd/duckhaven-backup.{service,timer} /etc/systemd/system/
# Edit WorkingDirectory and DUCKHAVEN_BACKUP_DIR in the .service first.
sudo systemctl enable --now duckhaven-backup.timer
systemctl list-timers duckhaven-backup.timer   # verify next run

DUCKHAVEN_BACKUP_DIR overrides the default /var/duckhaven/backups.

Restore

gunzip -c <backup>.sql.gz | docker compose -f deploy/docker-compose.yml \
    exec -T postgres psql -U duckhaven duckhaven

Data DR by backend kind

  • s3 / adls_gen2: delegated to the cloud provider's durability.
  • object_store (the bundled store): no off-box DR — the web UI shows a DR banner for these backends. Ensure an independent backup of the bundled bucket.

5. Tailscale outage

Where Tailscale is the only network path, if it is down the platform is unreachable. Document the agents' and control plane's static Tailscale IPs so operators can confirm reachability; agents auto-reconnect (5 s backoff) once the tailnet recovers.

6. Query queueing & concurrency

Each agent runs queries under admission control so it never oversubscribes its memory and gets OOM-killed: instead of picking up every dispatched query, it admits queries up to a memory budget and queues the rest (FIFO). When a running query finishes, the oldest queued query starts.

How capacity is split

The agent's budget is effective memory × (1 − headroom) (cgroup limit when set, else host RAM; headroom defaults to 10%, MEMORY_HEADROOM_FRACTION). Budget and slices are binary units throughout — a 4 GiB cgroup limit at the default headroom yields a 3.6 GiB budget, and that is the value passed to DuckDB's memory_limit as GiB. (DuckDB reads a bare GB suffix as 10⁹, so mixing the two silently shrinks every slice by ~7%.) There are two ways to size each query's slice of that budget:

  • auto (the default) — before running a query the agent runs EXPLAIN and estimates its peak memory from the optimizer's plan: the cardinality of blocking operators (joins, group-bys, sorts) times the row width, with a safety multiplier. The estimate snaps to a "T-shirt" bucket of the budget, so cheap queries reserve a small slice and pack in while heavy ones reserve more and queue when the agent is busy. Unestimable queries (DDL/DML, multi-statement, or an EXPLAIN failure/timeout) fall back to a default bucket. Estimation is best-effort under a short timeout and never delays or drops a query.
  • Static slot ladders — the budget is divided into a fixed weighted slot ladder; a new query takes the largest free slot, so the first running query gets the most memory and later ones get less.
Profile Weights Slots Share of budget
auto — — per-query, from the EXPLAIN estimate
single [1] 1 one query gets 100%
equal_2 [1,1] 2 50% / 50%
decaying_2 [2,1] 2 67% / 33%
decaying_3 [3,2,1] 3 50% / 33% / 17%

The weights divide memory only. Every query gets the agent's full core count under every profile, auto included: the container's CPU quota already caps the agent as a whole, so narrowing one query's thread count slows that query down without leaving anything spare for the others.

Default is auto (MAX_CONCURRENCY_PROFILE); the static ladders remain selectable as fallbacks. The queue holds up to MAX_QUEUE_DEPTH (default 100) queries; beyond that a query fails with queue full. QUEUED_TIMEOUT_S (default 0 = off) fails a query that waits too long with queued timeout. Whatever the mode, the agent enforces Σ running memory_limit ≤ budget, so it never oversubscribes (DuckDB fixes a session's memory at start and cannot resize a running query). Under a static ladder a lone query uses its slot's share, not the whole box (only single gives one query the full budget); under auto a query reserves the bucket its estimate maps to, up to the full budget.

Tuning the auto estimator

Variable Default Description
ESTIMATE_SAFETY_MULTIPLIER 1.5 Multiplies the raw EXPLAIN estimate to absorb under-estimation.
ESTIMATE_FLOOR_BYTES 64 MiB Minimum reservation, so a tiny estimate still gets a usable slice.
ESTIMATE_CEILING_FRACTION 1.0 Caps a reservation at this fraction of the budget.
EXPLAIN_TIMEOUT_S 2.0 Time budget for the pre-run EXPLAIN; on timeout the query uses the fallback bucket.
ESTIMATE_FALLBACK_BUCKET M Bucket used when a query is unestimable (DDL/DML, multi-statement, EXPLAIN error/timeout).

Changing the profile from the worksheet

Run this DuckHaven control command (its own statement) in a worksheet:

SET duckhaven_concurrency = 'auto';   -- auto | single | equal_2 | decaying_2 | decaying_3
RESET duckhaven_concurrency;           -- back to the default (auto)
  • It applies to the agent currently selected in that worksheet and is agent-global: it changes concurrency for every user's queries on that agent (like ALTER WAREHOUSE), not just your session.
  • It takes effect for future admissions; already-running queries keep their slot. The setting is held in memory and resets to the default on agent restart.
  • It is recorded in History like any query.

Monitoring

Compute → (pick an agent) → Monitoring shows live Running queries and Queued queries counters, and a Peak query count chart over the last 1–24 hours. A persistently non-zero queued band means the agent is saturated: raise the slot count (e.g. switch to decaying_3) only if per-query memory still suffices, or add another agent. The Failures & rejections chart on the same page separates saturation (queue_full, queued_timeout) from per-query problems like out_of_memory. See Monitoring.

7. Query profiles

After a query finishes, the agent captures DuckDB's per-operator execution profile and ships it (KB-sized) to the control plane, where it is stored on the query and served from GET /api/queries/{query_id}/profile. There are two ways to view it:

  • Worksheet → Profile tab — an inline summary + collapsible operator tree for a quick glance at the query you just ran, with an Open full profile link to the dedicated page.
  • Dedicated profile page (/{ws}/queries/{id}) — reached by clicking any row in History. It shows an interactive operator graph (result on top, scans at the bottom; data flows up) where clicking a node opens its detail.

Both surface:

  • a summary strip — latency, CPU time, rows returned, result size, peak memory, the reserved memory + CPU the query ran under, spill to disk, and bytes read/written (so a spill is read against what was reserved);
  • per-operator metrics — rows scanned → produced, bytes, a time-share bar, and the operator's EXTRA_INFO (join conditions, filters, group keys);
  • inefficiency highlights computed from the profile — spilled queries (worth a larger reservation or less intermediate data), scan blow-ups (a scan reading far more rows than the query returns), bad cardinality estimates (actual far from the optimizer's EXPLAIN estimate), and time hotspots. The dedicated page also ranks the most expensive operators and lists the detected issues in a diagnostics panel, each linking to the offending node.

Profiling is on by default and best-effort: a capture failure yields no profile rather than failing the query, and DDL/DML carry no profile (a no-profile state is shown). Set PROFILING_ENABLED=false on the agent to disable it.

8. Recover a stuck or failed catalog storage migration

A catalog storage migration is driven by the leader-elected migration runner; all its state lives in Postgres (catalog_migrations, catalog_migration_tables, catalog_migration_events), so a restart resumes an in-flight migration from its last completed table.

Inspect. Open Admin → Migrations, select the catalog, and read the live log, or query the tables directly:

SELECT id, status, tables_done, tables_total, error FROM catalog_migrations ORDER BY created_at DESC;
  • Stuck in copying/verifying. Confirm the migration runner is enabled (MIGRATION_RUNNER_ENABLED=true) on at least one healthy replica and that the target backend is reachable (re-run its health check on the storage admin page). The runner retries transient storage/Polaris errors; a genuinely unreachable backend ends the migration as failed.
  • failed. The catalog is untouched on its original backend (the pointer only changes at the atomic cutover). Read the error / log, fix the cause, and start a new migration. The failed run's shadow catalog is cleaned up best-effort; if an orphaned <name>__m<hex> Polaris catalog remains, drop it manually.
  • Cancel. Cancelling before cutover tears the shadow copy down and leaves the catalog on its original backend.
  • Reverse a completed migration. The old data is retained for MIGRATION_RETENTION_DAYS after cutover — start a new migration back to the original backend within that window.

9. Debugging a running container (no shell in production images)

The api and agent images run on Chainguard's distroless base: no shell, no package manager, no coreutils in the running container. docker compose exec api sh (or bash) no longer works. What still works unchanged: docker logs, docker compose logs, docker inspect, docker stats, docker top, and both services' healthchecks.

  • Read a file out of the container. Use docker compose cp <service>:<path> <local-path> (or plain docker cp) — this reads via the daemon's container-archive API and doesn't execute anything inside the container. This is the direct replacement for the old exec ... cat pattern (see §1 step 3).
  • One-off shell access, preferred: docker debug <container>. Docker Desktop's ephemeral debug-toolbox sidecar attaches a shell into a running container's namespaces without modifying the image, and works against distroless by design. Verify it's available on your actual host — it's historically been a Docker Desktop feature, not bundled with a plain Linux dockerd.
  • One-off shell access, fallback: run the -dev tag. Temporarily run the image's -dev build (e.g. cgr.dev/chainguard/python:latest-dev-based) for the affected service — it has a full shell, coreutils, and pip, at the cost of not being the exact production image.
  • Everything else (logs, health, resource usage, process list) needs no workaround — those all operate from outside the container via the Docker API, not by execing into it.

10. "Could not connect to server" from the object store under concurrency

A burst of concurrent queries fails with DuckDB errors naming the object store:

IO Error: Could not connect to server error for HTTP GET to
'http://objectstore:9000/warehouse/.../metadata/....avro'

Check the object store last, not first. This error usually means the agent ran out of outbound TCP ports, not that storage is down. The signature is distinctive:

  • The store answers fine from elsewhere — curl http://<store>:9000/health/ready from the host returns 200 throughout, and its own logs record nothing.
  • Failures are all-or-nothing per burst rather than scattered, and the first burst after an idle period succeeds while later ones fail.
  • It clears on its own after roughly a minute with no intervention.
  • In the worst case the agent container restarts, because the IO exception can escape on a thread with nothing to catch it and abort the process.

Confirm it by counting sockets in the agent's network namespace:

docker exec <agent-container> python3 -c "
n=0
for f in ('/proc/net/tcp','/proc/net/tcp6'):
    for line in open(f).read().splitlines()[1:]:
        if line.split()[3]=='06': n+=1
print('TIME_WAIT:', n)
print('port range:', open('/proc/sys/net/ipv4/ip_local_port_range').read().strip())"

A TIME_WAIT count at or near the width of the port range (the default 32768–60999 gives 28,232) is the diagnosis: every port is spent and connect() is returning EADDRNOTAVAIL.

Fix. Agents reuse object-store connections (httpfs_connection_caching) — see Connection reuse — which holds the socket count in the hundreds. If you hit this, you are running an agent image built before that landed, or a custom image that opens its own DuckDB connections without it. Rebuild or enable the setting; widening the port range only moves the cliff.