diff --git a/SPEC.md b/SPEC.md index 2c4870a..8b4b1fb 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,4 +1,4 @@ -# Work Queue API — Specification +# Work Queue API — Specification (Python Rewrite) ## Overview @@ -6,32 +6,43 @@ A lightweight internal API that tracks the full lifecycle of work items across T --- -## Database Schema (SQLite) +## Tech Stack + +- **Language:** Python 3.12+ +- **Package manager:** uv +- **Web framework:** FastAPI (or Flask if preferred) +- **Database:** PostgreSQL +- **Docs:** MkDocs +- **Container:** Docker, pushed to `git.danhenry.dev/thelab/work-queue-api` + +--- + +## Database Schema (PostgreSQL) ### Table: `projects` | Column | Type | Notes | |---|---|---| -| id | TEXT PRIMARY KEY | UUID | +| id | UUID PRIMARY KEY | | | name | TEXT NOT NULL | Human-readable project name | | external_ref | TEXT | Todoist project ID, GitHub repo, etc. (optional) | -| created_at | TEXT | ISO8601 | -| updated_at | TEXT | ISO8601 | +| created_at | TIMESTAMPTZ | ISO8601 | +| updated_at | TIMESTAMPTZ | ISO8601 | ### Table: `work_items` | Column | Type | Notes | |---|---|---| -| id | TEXT PRIMARY KEY | UUID | -| project_id | TEXT FK | References projects.id (optional) | +| id | UUID PRIMARY KEY | | +| project_id | UUID FK | References projects.id (optional) | | type | TEXT NOT NULL | e.g. `code_review`, `bug_fix`, `infra_setup`, `gitea_issue` | | description | TEXT NOT NULL | Human-readable summary | -| payload | TEXT | JSON blob with type-specific fields | +| payload | JSONB | Type-specific fields | | priority | INTEGER | 1-5, 1=highest. Default 3 | | status | TEXT NOT NULL | See Status Lifecycle below | | assigned_agent | TEXT | e.g. `steve-w`. NULL until dispatched | | created_by | TEXT | e.g. `marcus-a`, `gitea-watcher`, `bms-ticket-workflow` | -| created_at | TEXT | ISO8601 | -| updated_at | TEXT | ISO8601 | -| completed_at | TEXT | ISO8601, set when status → completed/failed/cancelled | +| created_at | TIMESTAMPTZ | ISO8601 | +| updated_at | TIMESTAMPTZ | ISO8601 | +| completed_at | TIMESTAMPTZ | ISO8601, set when status → completed/failed/cancelled | | outcome | TEXT | `success`, `failed`, `cancelled`, NULL | | notes | TEXT | Agent-added notes, URLs, context | @@ -54,11 +65,11 @@ queued → dispatched → in_progress → completed ### Table: `dispatch_log` | Column | Type | Notes | |---|---|---| -| id | TEXT PRIMARY KEY | UUID | -| work_item_id | TEXT FK | References work_items.id | -| dispatched_at | TEXT | ISO8601 | +| id | UUID PRIMARY KEY | | +| work_item_id | UUID FK | References work_items.id | +| dispatched_at | TIMESTAMPTZ | ISO8601 | | agent | TEXT | Which agent it was dispatched to | -| completed_at | TEXT | ISO8601, when status reached terminal state | +| completed_at | TIMESTAMPTZ | ISO8601, when status reached terminal state | | outcome | TEXT | success, failed, cancelled | ### Constraints @@ -71,136 +82,41 @@ queued → dispatched → in_progress → completed ### Projects -**`POST /projects`** -```json -// Request -{ "name": "Shopping List API", "external_ref": "todoist:123" } - -// Response 201 -{ "id": "uuid", "name": "Shopping List API", "external_ref": "todoist:123", "created_at": "...", "updated_at": "..." } -``` - -**`GET /projects`** — list all projects -**`GET /projects/:id`** — get project +**`POST /projects`** — create project +**`GET /projects`** — list all +**`GET /projects/:id`** — get one **`PATCH /projects/:id`** — update name or external_ref ---- - ### Work Items -**`POST /work`** -```json -// Request -{ - "project_id": "uuid", // optional - "type": "code_review", - "description": "Review PR #3 in shopping-list-api", - "payload": { "pr": 3, "repo": "shopping-list-api" }, - "priority": 2, - "assigned_agent": "steve-w" -} - -// Response 201 -{ "id": "uuid", "status": "queued", "created_at": "...", ... } -``` - -Note: `assigned_agent` is accepted on POST but item is still created as `queued`. Marcus must call `PATCH /work/:id` with `status=dispatched` to actually dispatch. (This allows Marcus to set up all fields before pulling the trigger.) - -**`GET /work`** — list work items. Supports filters: -- `?status=queued` — pending work -- `?status=in_progress` — active work -- `?status=blocked` — needs intervention -- `?agent=steve-w&status=queued` — Steve's pending queue -- `?project_id=uuid` — items in a project -- `?since=ISO8601` — created after timestamp - -Sort order: `priority ASC, created_at ASC` (unless `since` is used, then `created_at ASC`) - -**`GET /work/:id`** — get single work item with dispatch history - -**`PATCH /work/:id`** -```json -// Request — one or more fields -{ - "status": "in_progress", // queued→dispatched→in_progress, or in_progress→blocked - "outcome": "success", // set when moving to completed/failed/cancelled - "notes": "Reviewed and approved, merged to main", - "assigned_agent": "steve-w" // required to move from queued→dispatched -} - -// Response 200 — updated work item -``` - -Special transitions: -- `queued → dispatched` requires `assigned_agent` to be set -- `dispatched → in_progress` is reserved for Marcus (or a heartbeat safety net) to confirm agent picked it up -- `in_progress → completed` or `in_progress → failed` requires `outcome` -- `blocked` should include a `notes` explanation - -**`DELETE /work/:id`** — alias for `PATCH /work/:id` with `status=cancelled`. Returns 204. - ---- +**`POST /work`** — create item (status=`queued` on create) +**`GET /work`** — list with filters: ?status=, ?agent=, ?project_id=, ?since= +**Sort:** priority ASC, created_at ASC +**`GET /work/:id`** — single item with dispatch history +**`PATCH /work/:id`** — update status, outcome, notes, assigned_agent +**`DELETE /work/:id`** — cancel (status=cancelled). Returns 204. ### Monitoring (for Marcus heartbeat) **`GET /work?status=in_progress`** — what's being worked on right now **`GET /work?status=blocked`** — items that need intervention **`GET /work?status=failed`** — items that need review -**`GET /work?status=completed&since=`** — completed since last check (for notifications) +**`GET /work?status=completed&since=`** — completed since last check --- ## Dispatch Flow -### Marcus dispatches to Steve: -1. `POST /work` with Steve's queue item → status=`queued` -2. `PATCH /work/:id` with `status=dispatched`, `assigned_agent=steve-w` -3. (optional) Immediately `PATCH /work/:id` with `status=in_progress` to mark Steve has picked it up - -### Steve picks up his queue: -``` -GET /work?agent=steve-w&status=dispatched -→ for each item: PATCH /work/:id with status=in_progress -→ work the task -→ PATCH /work/:id with status=completed, outcome=success, notes=... -``` - -### Marcus heartbeat checks: -``` -GET /work?status=blocked → alert Daniel if anything new -GET /work?status=failed → alert Daniel if anything new -GET /work?status=completed&since= → notify Daniel of completions -GET /work?status=in_progress → detect stale items (>30min → flag blocked) -``` +1. Marcus POSTs /work → status=`queued` +2. Marcus PATCHs /work/:id with status=`dispatched`, assigned_agent=steve-w +3. (optional) Immediately PATCH to status=`in_progress` +4. Steve polls GET /work?agent=steve-w&status=dispatched, PATCHes to in_progress, works, PATCHes completed --- ## Stale Task Detection -Marcus's heartbeat checks `in_progress` items on every run. If any item has `updated_at` older than 30 minutes and no `notes` update, Marcus marks it `blocked` with a note about the timeout and alerts Daniel. - ---- - -## Skills / Integration - -### Marcus (main dispatcher) -A `work-queue` skill for Marcus: thin wrapper around HTTP calls to the API. Methods: -- `work_add(type, description, payload, agent, project_id?)` → POST /work -- `work_dispatch(work_item_id, agent)` → PATCH status=dispatched+in_progress -- `work_update(work_item_id, status, outcome?, notes?)` → PATCH -- `work_list(status?, agent?, project_id?)` → GET /work -- `work_stale_check()` → poll in_progress, timeout stale items - -### Steve's Agent -Steve's agent uses a polling loop: -``` -every N minutes: - GET /work?agent=steve-w&status=dispatched - for each item: - PATCH /work/:id with status=in_progress - do work - PATCH /work/:id with status=completed, outcome=success, notes=result -``` +Marcus's heartbeat checks `in_progress` items. If any item has `updated_at` older than 30 minutes, Marcus marks it `blocked` and alerts Daniel. --- @@ -210,62 +126,107 @@ every N minutes: work-queue-api/ ├── SPEC.md ├── Dockerfile -├── docker-compose.yml ← includes PostgreSQL (production) / SQLite dev option -├── ci.yml ← GitHub Actions: build + push container -├── internal/ -│ ├── api/ -│ │ ├── server.go -│ │ ├── handlers_work.go -│ │ ├── handlers_projects.go -│ │ └── middleware.go -│ ├── db/ -│ │ ├── migrations/ -│ │ │ └── 001_initial.sql -│ │ └── sqlite.go -│ └── model/ -│ └── models.go -└── README.md +├── docker-compose.yml ← must include postgres container +├── .github/ +│ └── workflows/ +│ └── ci.yml ← build + push to git.danhenry.dev/thelab/work-queue-api:latest +├── pyproject.toml / uv project files +├── mkdocs.yml ← MkDocs configuration +├── docs/ +│ └── index.md ← Docker Compose example + usage docs +├── app/ +│ ├── __init__.py +│ ├── main.py ← FastAPI app entry +│ ├── config.py ← Settings (DATABASE_URL, PORT, etc.) +│ ├── models.py ← Pydantic models +│ ├── db.py ← PostgreSQL connection +│ ├── routers/ +│ │ ├── __init__.py +│ │ ├── projects.py +│ │ └── work.py +│ └── migrations/ +│ └── 001_initial.sql +└── tests/ + └── test_api.py ``` --- ## CI/CD -GitHub Actions workflow (`ci.yml`): -1. Build Docker container on push to `main` +GitHub Actions (`ci.yml`): +1. Build Docker image on push to `main` 2. Push to `git.danhenry.dev/thelab/work-queue-api:latest` -3. Tag with git short SHA +3. Tag with short SHA + +Docker registry: `git.danhenry.dev` +Secrets available in thelab org: `DOCKER_REGISTRY`, `DOCKER_USERNAME`, `DOCKER_PASSWORD` --- -## Docker Compose Example +## Docker Compose Example (must be in docs AND in docker-compose.yml) ```yaml version: '3.8' services: - work-queue-api: + api: image: git.danhenry.dev/thelab/work-queue-api:latest ports: - "8080:8080" environment: - - DATABASE_URL=/data/work_queue.db + - DATABASE_URL=postgresql://postgres:password@db:5432/work_queue - PORT=8080 - volumes: - - ./data:/data + depends_on: + db: + condition: service_healthy restart: unless-stopped healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 3 + + db: + image: postgres:16-alpine + environment: + - POSTGRES_PASSWORD=password + - POSTGRES_DB=work_queue + volumes: + - ./data/postgres:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 10s + timeout: 5s + retries: 5 ``` --- -## Notes for Implementation +## Docs (MkDocs) -- Use Go with a lightweight router (chi or gin) -- Use `net/http` with SQLite via `mattn/go-sqlite3` -- No auth required (internal network only) -- `assigned_agent` uniqueness on `in_progress` should be enforced in application logic (SQLite lacks proper constraint for cross-row conditions) -- `dispatch_log` table is append-only; used for audit trail and staleness detection +`mkdocs.yml` with: +- `site_name: Work Queue API` +- `repo_url: https://git.danhenry.dev/thelab/work-queue-api` +- Nav structure: Getting Started, API Reference, Docker Compose + +`docs/index.md` must include: +- Overview +- Docker Compose full example (the postgres version) +- Quick start +- API endpoint reference +- Status lifecycle diagram + +--- + +## Skills to build after API is done + +For Marcus (main dispatcher): +- `work-queue` skill: thin wrapper around HTTP calls + - `work_add(type, description, payload, agent, project_id?)` → POST /work + - `work_dispatch(work_item_id, agent)` → PATCH status=dispatched+in_progress + - `work_update(work_item_id, status, outcome?, notes?)` → PATCH + - `work_list(status?, agent?, project_id?)` → GET /work + - `work_stale_check()` → poll in_progress, timeout stale items + +For Steve's agent: +- Polling skill: every N minutes, GET /work?agent=steve-w&status=dispatched, pick up items