thelab/work-queue-api
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite SPEC.md for Python/PostgreSQL rebuild
All checks were successful
ci / test-and-build (push) Successful in 3m56s
Details

Browse Source
This commit is contained in:
Marcus A.
2026-04-11 14:17:02 -05:00
parent f0275c41d0
commit 7420adb7aa
1 changed files with 115 additions and 154 deletions
Download Patch File Download Diff File Expand all files Collapse all files

269
SPEC.md
View File

@@ -1,4 +1,4 @@
# Work Queue API — Specification # Work Queue API — Specification (Python Rewrite)
## Overview ## Overview
@@ -6,32 +6,43 @@ A lightweight internal API that tracks the full lifecycle of work items across T
--- ---
## Database Schema (SQLite) ## Tech Stack
- **Language:** Python 3.12+
- **Package manager:** uv
- **Web framework:** FastAPI (or Flask if preferred)
- **Database:** PostgreSQL
- **Docs:** MkDocs
- **Container:** Docker, pushed to `git.danhenry.dev/thelab/work-queue-api`
---
## Database Schema (PostgreSQL)
### Table: `projects` ### Table: `projects`
| Column | Type | Notes | | Column | Type | Notes |
|---|---|---| |---|---|---|
| id | TEXT PRIMARY KEY | UUID | | id | UUID PRIMARY KEY | |
| name | TEXT NOT NULL | Human-readable project name | | name | TEXT NOT NULL | Human-readable project name |
| external_ref | TEXT | Todoist project ID, GitHub repo, etc. (optional) | | external_ref | TEXT | Todoist project ID, GitHub repo, etc. (optional) |
| created_at | TEXT | ISO8601 | | created_at | TIMESTAMPTZ | ISO8601 |
| updated_at | TEXT | ISO8601 | | updated_at | TIMESTAMPTZ | ISO8601 |
### Table: `work_items` ### Table: `work_items`
| Column | Type | Notes | | Column | Type | Notes |
|---|---|---| |---|---|---|
| id | TEXT PRIMARY KEY | UUID | | id | UUID PRIMARY KEY | |
| project_id | TEXT FK | References projects.id (optional) | | project_id | UUID FK | References projects.id (optional) |
| type | TEXT NOT NULL | e.g. `code_review`, `bug_fix`, `infra_setup`, `gitea_issue` | | type | TEXT NOT NULL | e.g. `code_review`, `bug_fix`, `infra_setup`, `gitea_issue` |
| description | TEXT NOT NULL | Human-readable summary | | description | TEXT NOT NULL | Human-readable summary |
| payload | TEXT | JSON blob with type-specific fields | | payload | JSONB | Type-specific fields |
| priority | INTEGER | 1-5, 1=highest. Default 3 | | priority | INTEGER | 1-5, 1=highest. Default 3 |
| status | TEXT NOT NULL | See Status Lifecycle below | | status | TEXT NOT NULL | See Status Lifecycle below |
| assigned_agent | TEXT | e.g. `steve-w`. NULL until dispatched | | assigned_agent | TEXT | e.g. `steve-w`. NULL until dispatched |
| created_by | TEXT | e.g. `marcus-a`, `gitea-watcher`, `bms-ticket-workflow` | | created_by | TEXT | e.g. `marcus-a`, `gitea-watcher`, `bms-ticket-workflow` |
| created_at | TEXT | ISO8601 | | created_at | TIMESTAMPTZ | ISO8601 |
| updated_at | TEXT | ISO8601 | | updated_at | TIMESTAMPTZ | ISO8601 |
| completed_at | TEXT | ISO8601, set when status → completed/failed/cancelled | | completed_at | TIMESTAMPTZ | ISO8601, set when status → completed/failed/cancelled |
| outcome | TEXT | `success`, `failed`, `cancelled`, NULL | | outcome | TEXT | `success`, `failed`, `cancelled`, NULL |
| notes | TEXT | Agent-added notes, URLs, context | | notes | TEXT | Agent-added notes, URLs, context |
@@ -54,11 +65,11 @@ queued → dispatched → in_progress → completed
### Table: `dispatch_log` ### Table: `dispatch_log`
| Column | Type | Notes | | Column | Type | Notes |
|---|---|---| |---|---|---|
| id | TEXT PRIMARY KEY | UUID | | id | UUID PRIMARY KEY | |
| work_item_id | TEXT FK | References work_items.id | | work_item_id | UUID FK | References work_items.id |
| dispatched_at | TEXT | ISO8601 | | dispatched_at | TIMESTAMPTZ | ISO8601 |
| agent | TEXT | Which agent it was dispatched to | | agent | TEXT | Which agent it was dispatched to |
| completed_at | TEXT | ISO8601, when status reached terminal state | | completed_at | TIMESTAMPTZ | ISO8601, when status reached terminal state |
| outcome | TEXT | success, failed, cancelled | | outcome | TEXT | success, failed, cancelled |
### Constraints ### Constraints
@@ -71,136 +82,41 @@ queued → dispatched → in_progress → completed
### Projects ### Projects
**`POST /projects`** **`POST /projects`** — create project
```json **`GET /projects`** — list all
// Request **`GET /projects/:id`** — get one
{ "name": "Shopping List API", "external_ref": "todoist:123" }
// Response 201
{ "id": "uuid", "name": "Shopping List API", "external_ref": "todoist:123", "created_at": "...", "updated_at": "..." }
```
**`GET /projects`** — list all projects
**`GET /projects/:id`** — get project
**`PATCH /projects/:id`** — update name or external_ref **`PATCH /projects/:id`** — update name or external_ref
---
### Work Items ### Work Items
**`POST /work`** **`POST /work`** — create item (status=`queued` on create)
```json **`GET /work`** — list with filters: ?status=, ?agent=, ?project_id=, ?since=
// Request **Sort:** priority ASC, created_at ASC
{ **`GET /work/:id`** — single item with dispatch history
"project_id": "uuid", // optional **`PATCH /work/:id`** — update status, outcome, notes, assigned_agent
"type": "code_review", **`DELETE /work/:id`** — cancel (status=cancelled). Returns 204.
"description": "Review PR #3 in shopping-list-api",
"payload": { "pr": 3, "repo": "shopping-list-api" },
"priority": 2,
"assigned_agent": "steve-w"
}
// Response 201
{ "id": "uuid", "status": "queued", "created_at": "...", ... }
```
Note: `assigned_agent` is accepted on POST but item is still created as `queued`. Marcus must call `PATCH /work/:id` with `status=dispatched` to actually dispatch. (This allows Marcus to set up all fields before pulling the trigger.)
**`GET /work`** — list work items. Supports filters:
- `?status=queued` — pending work
- `?status=in_progress` — active work
- `?status=blocked` — needs intervention
- `?agent=steve-w&status=queued` — Steve's pending queue
- `?project_id=uuid` — items in a project
- `?since=ISO8601` — created after timestamp
Sort order: `priority ASC, created_at ASC` (unless `since` is used, then `created_at ASC`)
**`GET /work/:id`** — get single work item with dispatch history
**`PATCH /work/:id`**
```json
// Request — one or more fields
{
"status": "in_progress", // queued→dispatched→in_progress, or in_progress→blocked
"outcome": "success", // set when moving to completed/failed/cancelled
"notes": "Reviewed and approved, merged to main",
"assigned_agent": "steve-w" // required to move from queued→dispatched
}
// Response 200 — updated work item
```
Special transitions:
- `queued → dispatched` requires `assigned_agent` to be set
- `dispatched → in_progress` is reserved for Marcus (or a heartbeat safety net) to confirm agent picked it up
- `in_progress → completed` or `in_progress → failed` requires `outcome`
- `blocked` should include a `notes` explanation
**`DELETE /work/:id`** — alias for `PATCH /work/:id` with `status=cancelled`. Returns 204.
---
### Monitoring (for Marcus heartbeat) ### Monitoring (for Marcus heartbeat)
**`GET /work?status=in_progress`** — what's being worked on right now **`GET /work?status=in_progress`** — what's being worked on right now
**`GET /work?status=blocked`** — items that need intervention **`GET /work?status=blocked`** — items that need intervention
**`GET /work?status=failed`** — items that need review **`GET /work?status=failed`** — items that need review
**`GET /work?status=completed&since=<ts>`** — completed since last check (for notifications) **`GET /work?status=completed&since=<ts>`** — completed since last check
--- ---
## Dispatch Flow ## Dispatch Flow
### Marcus dispatches to Steve: 1. Marcus POSTs /work → status=`queued`
1. `POST /work` with Steve's queue item → status=`queued` 2. Marcus PATCHs /work/:id with status=`dispatched`, assigned_agent=steve-w
2. `PATCH /work/:id` with `status=dispatched`, `assigned_agent=steve-w` 3. (optional) Immediately PATCH to status=`in_progress`
3. (optional) Immediately `PATCH /work/:id` with `status=in_progress` to mark Steve has picked it up 4. Steve polls GET /work?agent=steve-w&status=dispatched, PATCHes to in_progress, works, PATCHes completed
### Steve picks up his queue:
```
GET /work?agent=steve-w&status=dispatched
→ for each item: PATCH /work/:id with status=in_progress
→ work the task
→ PATCH /work/:id with status=completed, outcome=success, notes=...
```
### Marcus heartbeat checks:
```
GET /work?status=blocked → alert Daniel if anything new
GET /work?status=failed → alert Daniel if anything new
GET /work?status=completed&since=<ts> → notify Daniel of completions
GET /work?status=in_progress → detect stale items (>30min → flag blocked)
```
--- ---
## Stale Task Detection ## Stale Task Detection
Marcus's heartbeat checks `in_progress` items on every run. If any item has `updated_at` older than 30 minutes and no `notes` update, Marcus marks it `blocked` with a note about the timeout and alerts Daniel. Marcus's heartbeat checks `in_progress` items. If any item has `updated_at` older than 30 minutes, Marcus marks it `blocked` and alerts Daniel.
---
## Skills / Integration
### Marcus (main dispatcher)
A `work-queue` skill for Marcus: thin wrapper around HTTP calls to the API. Methods:
- `work_add(type, description, payload, agent, project_id?)` → POST /work
- `work_dispatch(work_item_id, agent)` → PATCH status=dispatched+in_progress
- `work_update(work_item_id, status, outcome?, notes?)` → PATCH
- `work_list(status?, agent?, project_id?)` → GET /work
- `work_stale_check()` → poll in_progress, timeout stale items
### Steve's Agent
Steve's agent uses a polling loop:
```
every N minutes:
GET /work?agent=steve-w&status=dispatched
for each item:
PATCH /work/:id with status=in_progress
do work
PATCH /work/:id with status=completed, outcome=success, notes=result
```
--- ---
@@ -210,62 +126,107 @@ every N minutes:
work-queue-api/ work-queue-api/
├── SPEC.md ├── SPEC.md
├── Dockerfile ├── Dockerfile
├── docker-compose.yml ← includes PostgreSQL (production) / SQLite dev option ├── docker-compose.yml ← must include postgres container
├── ci.yml ← GitHub Actions: build + push container ├── .github/
├── internal/ │ └── workflows/
│ ├── api/ │ └── ci.yml ← build + push to git.danhenry.dev/thelab/work-queue-api:latest
│ │ ├── server.go ├── pyproject.toml / uv project files
│ │ ├── handlers_work.go ├── mkdocs.yml ← MkDocs configuration
│ │ ├── handlers_projects.go ├── docs/
│ │ └── middleware.go │ └── index.md ← Docker Compose example + usage docs
│ ├── db/ ├── app/
│ │ ├── migrations/ │ ├── __init__.py
│ │ │ └── 001_initial.sql │ ├── main.py ← FastAPI app entry
│ │ └── sqlite.go │ ├── config.py ← Settings (DATABASE_URL, PORT, etc.)
│ └── model/ │ ├── models.py ← Pydantic models
│ └── models.go │ ├── db.py ← PostgreSQL connection
└── README.md │ ├── routers/
│ │ ├── __init__.py
│ │ ├── projects.py
│ │ └── work.py
│ └── migrations/
│ └── 001_initial.sql
└── tests/
└── test_api.py
``` ```
--- ---
## CI/CD ## CI/CD
GitHub Actions workflow (`ci.yml`): GitHub Actions (`ci.yml`):
1. Build Docker container on push to `main` 1. Build Docker image on push to `main`
2. Push to `git.danhenry.dev/thelab/work-queue-api:latest` 2. Push to `git.danhenry.dev/thelab/work-queue-api:latest`
3. Tag with git short SHA 3. Tag with short SHA
Docker registry: `git.danhenry.dev`
Secrets available in thelab org: `DOCKER_REGISTRY`, `DOCKER_USERNAME`, `DOCKER_PASSWORD`
--- ---
## Docker Compose Example ## Docker Compose Example (must be in docs AND in docker-compose.yml)
```yaml ```yaml
version: '3.8' version: '3.8'
services: services:
work-queue-api: api:
image: git.danhenry.dev/thelab/work-queue-api:latest image: git.danhenry.dev/thelab/work-queue-api:latest
ports: ports:
- "8080:8080" - "8080:8080"
environment: environment:
- DATABASE_URL=/data/work_queue.db - DATABASE_URL=postgresql://postgres:password@db:5432/work_queue
- PORT=8080 - PORT=8080
volumes: depends_on:
- ./data:/data db:
condition: service_healthy
restart: unless-stopped restart: unless-stopped
healthcheck: healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"] test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
db:
image: postgres:16-alpine
environment:
- POSTGRES_PASSWORD=password
- POSTGRES_DB=work_queue
volumes:
- ./data/postgres:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 10s
timeout: 5s
retries: 5
``` ```
--- ---
## Notes for Implementation ## Docs (MkDocs)
- Use Go with a lightweight router (chi or gin) `mkdocs.yml` with:
- Use `net/http` with SQLite via `mattn/go-sqlite3` - `site_name: Work Queue API`
- No auth required (internal network only) - `repo_url: https://git.danhenry.dev/thelab/work-queue-api`
- `assigned_agent` uniqueness on `in_progress` should be enforced in application logic (SQLite lacks proper constraint for cross-row conditions) - Nav structure: Getting Started, API Reference, Docker Compose
- `dispatch_log` table is append-only; used for audit trail and staleness detection
`docs/index.md` must include:
- Overview
- Docker Compose full example (the postgres version)
- Quick start
- API endpoint reference
- Status lifecycle diagram
---
## Skills to build after API is done
For Marcus (main dispatcher):
- `work-queue` skill: thin wrapper around HTTP calls
- `work_add(type, description, payload, agent, project_id?)` → POST /work
- `work_dispatch(work_item_id, agent)` → PATCH status=dispatched+in_progress
- `work_update(work_item_id, status, outcome?, notes?)` → PATCH
- `work_list(status?, agent?, project_id?)` → GET /work
- `work_stale_check()` → poll in_progress, timeout stale items
For Steve's agent:
- Polling skill: every N minutes, GET /work?agent=steve-w&status=dispatched, pick up items