leasework

Distributed job scheduler on Kafka, Redis and Postgres.

Clients submit jobs: run now, run later, retry on failure. Workers execute them under a fenced lease, so a job that is delivered twice still takes effect once. Operators get metrics, an SLO and alerts that say whether the system is keeping its promise.

Status: Phase 1, walking skeleton. A job can be submitted over HTTP, is written to Postgres with its outbox row in one transaction, is relayed to Kafka, and is consumed and marked succeeded by a worker. The handler itself is a no-op. There is no lease, no fence, no delayed execution, no retry and no dead-letter queue yet, so a redelivered job really would execute twice. Nothing below is claimed as working until its phase lands. See docs/plan.md for the sequence.

---

The idea in one paragraph

Workers die mid-job. Processes freeze and come back believing no time has passed. Brokers restart. The guarantee this project makes is that every job runs, and no job takes effect twice, and that both halves are provable, not asserted.

Getting there means pushing every failure into "this might happen twice" and never into "this might not happen at all", then making "twice" harmless. Two mechanisms do that work:

A transactional outbox. The job row and a "publish this" row are written in one Postgres transaction. A relay loop reads the outbox and produces to Kafka. The alternative (insert, then publish as a separate step) loses a job silently if the process dies in between, which is exactly the failure this project claims to prevent.

A fenced lease. A worker claims a job for a bounded time and heartbeats to hold it. If it dies, the claim expires on its own with nothing to clean up. But a TTL alone is not enough: a worker that is merely stalled can wake up after its lease expired, still believing it owns the job, and execute it a second time. So each claim also carries a monotonically increasing fence token, and the terminal write is conditioned on it:

UPDATE jobs SET status = $3 WHERE id = $1 AND fence = $2

The stale worker is not stopped; it cannot reliably be. Its write simply matches zero rows.

This is not exactly-once. It is at-least-once delivery, at-most-once side effects for fenced handlers, effectively-once outcomes.

---

Architecture

Three long-running Go programs (api, scheduler, worker) and a one-shot migrate, with Postgres as the only source of truth, Kafka for delivery, and Prometheus and Grafana for monitoring. Redis (leases and fencing) and Kubernetes arrive in later phases.

docs/architecture.md has the diagram, how a job flows end to end, and which parts are built versus planned. It is updated whenever a phase lands.

---

Running it

Requires Docker. A local Go toolchain is optional: the Go targets (build, test, lint, fmt, tidy) use go when it is on your PATH and otherwise run inside a golang:1.25 container. If you install Go, use 1.25 to match CI and the Dockerfile. A newer release lets in dependencies that need it, which build locally and then fail in CI. go test -race needs cgo, so a local toolchain also needs a C compiler on PATH.

make up

make topics

make smoke

make smoke is the real check: it submits a job, then polls until the API reports it succeeded, and fails with the last observed status if it never does.

Service	URL
API	http://localhost:8080
Prometheus	http://localhost:9090
Grafana	http://localhost:3000

make help lists every target. make down stops the stack; make clean also drops volumes.

Schema changes are applied by a one-shot migrate service that runs to completion before api, worker or scheduler start. make up runs it automatically, and make migrate runs it on its own. Migrations deliberately do not run at API startup: schema creation needs exactly one owner, or several replicas booting together race each other and the long-running services race the schema.

Each binary serves /healthz, /readyz and /metrics: api on 8080, worker on 9101, scheduler on 9102. Readiness is not cosmetic. A process reports ready only while every dependency it needs is reachable, and flips back when one is not, so a database or broker outage is visible on /readyz instead of a process sitting there claiming health.

API

POST /jobs
{"type": "noop", "payload": {}}

202 Accepted
{"job_id": "16ef6018-98ee-4cfe-a418-cf9da06a5b22", "status": "pending"}

202 rather than 201, because the work has been accepted for later execution, not performed.

GET /jobs/{id}

200 OK
{"id": "...", "type": "noop", "payload": {}, "status": "succeeded", ...}

Idempotency-Key, listing with pagination and cancellation arrive in Phase 2.

Configuration

Environment variables, all prefixed LEASEWORK_:

Variable	Default
LEASEWORK_LOG_LEVEL	info
LEASEWORK_OPS_ADDR	per service: :8080 / :9101 / :9102
LEASEWORK_POSTGRES_DSN	local Postgres
LEASEWORK_REDIS_ADDR	localhost:6379
LEASEWORK_KAFKA_BROKERS	localhost:19092,localhost:29092,localhost:39092
LEASEWORK_SHUTDOWN_TIMEOUT	20s

---

Documentation

• docs/architecture.md: the diagram, the job flow, and what is built versus planned.
• docs/design-decisions.md: what was decided, what it beat, and what it costs.
• docs/plan.md: the phase sequence, from scaffold to Kubernetes autoscaling.
• docs/failure-modes.md: every point a worker can die, what recovers it, and what the handler must guarantee for itself.

Known gaps

Listed deliberately. Naming them is what makes the rest credible.

• Single scheduler replica, no leader election.
• Single Redis instance: no Sentinel, no Cluster.
• No Kafka transactions / EOS.
• No RBAC beyond API keys.
• kind is not a real cluster: no CNI, storage or upgrade experience.
• No recurring schedules.