Developer manual

For senior engineers taking a fork of the openshelf_os codebase. The goal is to get the stack running, understand the architecture, and extend it. Read time: 25–35 minutes.

What this is

An Openshelf device is a robot arm in a cabinet that stores and retrieves vials by barcode. Each lab device runs the Openshelf OS stack: a FastAPI backend (api, port 8000) and a SvelteKit frontend (dev_ui, port 8080) in one docker compose project.

The API is the contract. Robot motion, inventory, calibration, diagnostics, and printing are all exposed through /api/* endpoints; the Local API reference is the maintained reference and is generated from the code. The UI is one consumer of that API; lab software can talk to the API directly.

Forking this repo means owning three Python packages in a clean dependency chain: low-level device drivers, domain and orchestration logic, and the HTTP service. This manual maps them, shows how to run the stack in every mode, and walks through which files to touch for the common extensions.

Repository layout

openshelf_os/
  dev_ui/                    SvelteKit frontend (local and device UI)
  openshelf_hardware/        Python uv workspace
    packages/hardware/           low-level device drivers
    packages/openshelf_os/       domain + orchestration (robot, storage, config)
    packages/openshelf_os_server/ FastAPI service (routers, core, database)
    data/                      live runtime state (sqlite, config, mode files)
  scripts/                   start/stop lifecycle + hardware debug helpers
  docker-compose.yml           base stack (api, dev_ui)
  docker-compose.dev.yml       development overrides (source hot reload)
  docker-compose.hardware.yml  USB/device passthrough

dev_ui/ talks to the API over HTTP and consumes its SSE event streams. Everything else lives inside the api container: openshelf_os_server is the HTTP surface, openshelf_os is the robot and storage domain, and hardware is driver code only. data/ is the live source of truth for local state; delete or edit it carefully.

Running the stack

Prerequisites: docker with compose v2, uv, and node for frontend checks.

First-time local development start:

./scripts/start --mode development --build

Normal local development start:

./scripts/start --mode development

Stop the stack with ./scripts/stop. Always start and stop through scripts/start and scripts/stop rather than raw docker compose commands, so the mode tracking in data/.mode stays in sync. On a machine with connected robot, camera, scanner, or printer devices, pass --hardware (and stop first when restarting: ./scripts/stop then ./scripts/start -d -h), otherwise the API container can restart without the hardware compose overlay and lose device passthrough.

Every checkout of this repo shares one compose project name (openshelf_os), so starting a second checkout replaces the containers of the first. Set OPSHELF_COMPOSE_PROJECT=<name> per worktree when two stacks have to coexist.

scripts/start enables the hardware overlay on its own when the selected mode's config sets robot.enabled or robot_app.enabled to true, never in mock mode, and takes --force-recreate alongside -d, -h, -b, and -p.

Operating modes

The mode is selected only through scripts/start and tracked in data/.mode; the API exposes it read-only. A plain ./scripts/start with no mode file starts production, while an API process launched without scripts/start (no data/.mode, no OPERATING_MODE) falls back to development.

Modes use persistent mode-specific state variants (base.development.db, config.development.json, ...). The first start in development or demo copies the existing production state without modifying it, so returning to production restores the original workspace. Mock creates a fresh base.mock.db (empty inventory, seeded users and roles) and never copies production database or item state. It may still copy config.json to config.mock.json, so configuration values travel into mock. Sentry follows should_enable_sentry, which is true for demo and production and false for mock and development, so a demo run does send events to team telemetry. When demo state has to be recovered from production, the .demo_workspace_seeded marker is deleted and the next demo start re-runs the demo seed, replacing demo data.

Mock is the default environment for API and runtime work without a device:

• The virtual cabinet is 80 deterministic cells: front module then back, four shelves per side, ten cells per shelf. Occupancy comes from sqlite items and item_locations plus the reserved homes of configured storage_bins rows.
• Commands run through one deterministic FIFO worker with a 30-second simulated execution time and a 32 queued/running limit; a full queue returns 429 with Retry-After: 1. The same 32-command limit and the same seven-day request_id window guard the real path in openshelf_os_server/core/command_lifecycle.py.
• An optional top-level request_id makes a request idempotent for seven days: an exact replay returns the original command, reuse with a different payload returns 409. Completed command rows are pruned after seven days, so history and idempotency end together.
• Commands interrupted by a restart are persisted as failed, never replayed.
• Only store_item and retrieve_item are accepted in mock; any other command type returns 400. A store enrolls the item at admission, so a command that fails in execution still leaves the item in the database, and a store with no supplied weight persists a deterministic one.
• POST /api/print answers printed=true without producing a label. It is not anonymous: it needs printer:print or a relay token, and the barcode stays printable ASCII within 255 characters.
• The normal developer account is seeded, but authentication remains required. Outside disposable local tests, keep JWT_SECRET and AUDIT_HMAC_KEY stable by pinning them in the checkout .env (append, never overwrite) with a unique AUDIT_HMAC_KEY_ID.

Useful variants:

./scripts/start --mode mock --detached
./scripts/start --mode development --hardware
./scripts/start --mode development --pull   # devices: pull prebuilt images from the team registry

Quick checks: the UI at http://localhost:8080, API health at http://localhost:8000/api/health, and the seeded developer login [email protected] with PIN 1111. Container logs are capped at 10 MiB x 5 files; docker logs stops at the first truncated line, so after an unclean shutdown read the container JSON log directly.

Quality gates

cd openshelf_hardware
uv sync --all-packages      # bare `uv sync` prunes workspace member dependencies
uv run pytest
uv run ruff check packages
uv run mypy packages/openshelf_os_server/src

cd dev_ui
npm install
npm run check
npm run build

The backend suite is order-dependent by default only because collection order is stable; before landing test changes, run it shuffled at least once:

cd openshelf_hardware && uv run pytest --random-order

Both commands read the operating mode from openshelf_hardware/data/.mode, because importing the app resolves runtime state at import time. A leftover mock mode file makes robot tests take the mock paths, so unrelated tests that post a home command fail with 400 on a clean tree. Use a throwaway data dir whenever the run itself is the evidence:

cd openshelf_hardware && OPENSHELF_DATA_DIR=$(mktemp -d) uv run pytest --random-order

Tests take their identity from openshelf_os_server/tests/conftest.py, which seeds [email protected] with PIN 1234. Its autouse cleanup clears the bin, item, and location tables only for modules named test_storage*, and the audit, user, and role tables only for modules named test_audit*, before and after each test. A new module that writes those tables leaks rows into the shuffled run until it is added there. openshelf_os/__init__.py configures Sentry at import time, so a test that wants Sentry off must set SENTRY_DSN_HARDWARE before that package is first imported.

Focused runs take a package path directly, for example uv run pytest packages/openshelf_os_server/tests/test_storage_endpoints.py.

Nothing runs these gates for you: the only workflow in .github/workflows/ dispatches a release build, so every command above is a human step before a merge.

Async tests mix pytest-asyncio (configured in auto mode by openshelf_os_server/pytest.ini alone, while the other two packages set no asyncio_mode) with AnyIO. A test marked pytest.mark.anyio must take its background-task fixtures from @pytest.fixture, not @pytest_asyncio.fixture, or the worker attaches to a different event loop in the full suite.

Architecture

• hardware is driver code only: SharedModbusController for the RS-485 bus, Motor, Servo (the gripper), LaserSensor, ToFSensorBus, UvcCamera/ToolCamera, the label printer, the scale, the MotionSafetyInterlock (curtain), DoorRelease, and the GPIO layer (I2C expanders with a Raspberry Pi fallback, plus a fake implementation for tests). It knows no domain logic.
• openshelf_os is the domain: robot/ (core orchestration, the FIFO worker, pydantic command models, status types), the storage grid and cabinet, the config loader with schema versioning, the laser calibration suite, the I/O box, serial port mapping, and operating-mode handling.
• openshelf_os_server is the service: the FastAPI app (main.py), the routers, the audit and correlation middleware, core/ long-lived services, the SQLAlchemy/Alembic database layer, and the OpenAPI generator, whose JSON is a build-time artifact rather than a committed file.

Startup sequence

Startup resolves configuration before the main.py app lifespan, then initializes the services in this order:

1. Resolve the mode's runtime paths (config file, database file, calibration directory) and migrate the active config before loading it through settings.get_root_config.
2. Run Alembic migrations and create any missing tables.
3. Seed default roles, role permissions, access levels, and the developer user (plus the configured development auto-user, when auth.dev_mode is on).
4. Ensure onboarding state and apply the mode transition.
5. Recover interrupted robot commands (queued/running become failed).
6. Sync storage bin identities from config and recover interrupted bin workflows.
7. Start the mock robot runtime (mock mode) or the background robot initialization task (real hardware), then the camera managers, I/O box sizer, and crash video recorder.
8. The app is ready; /api/health reports the mode and robot readiness (status, mode, robot_status, robot_phase) and answers 503 when initialization failed. It reports no database, camera, or other dependency checks.

Robot initialization on real hardware is a tracked, multi-phase process. Robot core connects the motor, scale, and laser buses, then GPIO, then Tool.configure, which configures the gripper servo and the five axis motors in the order y_axis, x_axis, z_axis, stage, rotate, initializes the laser, and only then homes: z_axis and stage first, then rotate, then x_axis and y_axis. IOBox.configure runs afterwards and homes door last, before the cabinet (openshelf_os/tool.py configure and _home_sequence, openshelf_os/robot/core.py). Progress is tracked per component in core/init_progress.py through the phases pending, connecting, configuring, homing, ready, failed, and surfaced through /api/health and the status stream. The app serves requests while that runs, so an initializing phase is not a failed start. The overall phase turns degraded when a component registered as degrading fails after the tracker already reached ready, and returns to ready when every degrading component is ready again. A machine without a limit switch must have limit_switches_enabled: false on that motor entry, or homing fails because it verifies the home limit input.

Config

Runtime configuration is config.<mode>.json, loaded and validated through pydantic models. The top-level schema_version is the persisted schema: absent means 1, the pre-versioning layout that device setup writes, and the current version is config.schema.CURRENT_SCHEMA_VERSION (5). The server automatically migrates the active config before loading it; each other mode's config migrates when that mode next starts. config_migrations/apply.py runs the full migration chain, validates the result, preserves the original bytes in an exclusively created <filename>.schema-v<N>.bak (where N is the source version), and replaces the config atomically. A newer schema, failed migration, or conflicting backup stops startup. Preserve and reconcile an existing backup before retrying. Device scripts and other offline entry points still need scripts/migrate_config.py (preview first, --write with the application stopped). Never fix a startup error by changing only schema_version — the old field names would then lose their meaning. Version 5 renames the left/right scan config keys and bin home sides to front/back.

Database and state

One sqlite database per mode (base.db in production, base.development.db, base.mock.db, ...), managed with SQLAlchemy and migrated with Alembic. The tables cover users/roles/access levels, the HMAC-chained audit_logs, items and item_locations, robot_commands, storage bins and workflows, item_sizer_measurements, telemetry, and storage_cells. The storage grid itself is a JSON file under data/ (storage.json and mode variants) describing modules, shelves, and cells.

Current side values are front_module and back_module, with F/B in demo location strings. The database side rename is also applied after startup baseline stamping, so item_locations and storage_bins are updated even when Alembic skips the historical revision chain. Demo mapping readers, including the demo seed script, migrate old L/R locations in place through demo_side_rename.py, preserving a .demo-side-rename.bak copy. Saved laser runs map historical side names in memory when loaded; their files stay intact.

Live status and results fan out over the SSE endpoints /api/robot/status and /api/robot/events from in-process queues. /api/telemetry is an ordinary JSON GET, not a stream. Camera video is a separate transport: MJPEG over HTTP plus a WebSocket at /api/camera/ws/{camera_id}, and both take the token as a token query parameter because EventSource and WebSocket clients cannot set an Authorization header.

Frontend conventions

The shared client in dev_ui/src/lib/api.ts fetches the storage grid to infer whether the device uses left/right or front/back names. lib/side.ts maps both vocabularies to stable side indices (0 and 1); UI state uses canonical front/back tokens, while API wrappers translate outbound sides to the device vocabulary. Load the grid before side-specific requests, compare received side strings with sideIndex(), and use the shared label helpers. Laser snapshot normalization in the client also maps legacy per-side field names.

+layout.svelte owns the shared robot status stream, NeedsHomeAlert, SafetyAckAlert, and ToastStack. Homing and acknowledgement prompts follow server status; the status page also shows the required actions and blocking sources. ErrorMessage and StatusMessage post to lib/toasts.ts: errors stay until dismissed, status messages expire after ten seconds, and identical visible messages are deduplicated. Reuse these components for action feedback.

The status page reads identity and alias from /api/system/info and mode from /api/system/mode. The public info endpoint also exposes nullable release_sha and release_branch: RELEASE_SHA/RELEASE_BRANCH override /etc/opshelf/release; the alias comes from /etc/opshelf/device-alias. Readiness still comes from health and robot status.

Calibration views live under lib/components/calibration/. CellCalibration.svelte places cell selection beside motion controls, with explicit calibrate and save-and-test actions. StorageLayoutEditor.svelte owns bulk deletion and coordinate inference for newly added cells; those are client workflows over existing grid endpoints. Bulk deletion can partially succeed and reloads the grid afterwards. New cell coordinates are inferred from the origin or calibrated neighbours when possible, then persisted with a separate grid update.

A command's life

This is the reference path for anything you add: a store_item command from client to persisted result.

1. CorrelationIdMiddleware adopts or generates X-Correlation-ID and echoes it on the response. The audit middleware reads that header from the request instead, so only a client-supplied id ever lands in an audit row; a generated one reaches logs through the logging context, not through the audit table. Both wrap every router, but the audit middleware records nothing before the handler runs and not everything after it: OPTIONS, /api/health, /api/robot/debug/motors/probe, and /api/calibration/laser/stream are skipped, every response of 400 or above is written, successful reads on an allowlist are skipped, and /api/storage/items/lookup counts as a read. Rows are kept for 120 days.
2. The router validates the payload: the item exists, the caller's access level covers it, and the location rules hold. retrieve validates the requested location and access before queueing, then removes the item location on completion.
3. The command is persisted as queued; a full queue (32 active) returns 429, and a request_id seen within seven days replays the stored command instead of queueing.
4. The FIFO worker dequeues and hands the command to the robot core.
5. The robot core orchestrates the motion: homing, moves along the tool axes, gripper action, scale and laser confirmations, with the shared MotionSafetyInterlock gating motion for curtain, cabinet-door, and weight-change interruptions. Status snapshots publish continuously.
6. The outcome (succeeded / failed) is persisted and broadcast on the event stream. In mock mode the same lifecycle runs against one deterministic worker with a 30-second simulated execution.
7. A motion fault passes through Tool._handle_fault, which attaches a fault_context (per axis motor snapshot, phase, coordinates, and any previous fault context) to the exception, and the worker reports it with that context plus crash-recorder footage. The machine does not recover on its own; the log line asks for manual intervention.

Barcodes never travel in a URL path (uvicorn unquotes %2F before routing, so a barcode containing / could not be addressed); item and print endpoints take the barcode in the POST body.

Weight-change interruptions

openshelf_os/robot/weight_guard.py samples the scale during Y-axis motion, including the startup clearance jog before the worker exists. It compares readings against the last idle weight and blocks motion on a sustained change. The guard is enabled by default when the scale and Y motor are configured; settings live under robot.io.scale. Defaults are motion_guard_delta_grams: 10, motion_guard_min_samples: 2, motion_guard_arm_delay_s: 0.05, and motion_guard_ack_grace_s: 1.0. Sampling uses Scale.read_weight_fast() with one Modbus transaction, no retries, and a short timeout. A busy scale bus skips a sample; repeated read failures suspend sampling with backoff, so inspect the source's error and failed fields when diagnosing coverage. Trip records, including recent samples and operation context, are written to the relative data/weight_faults/ directory after the motion block is requested.

The worker reports weight_change in robot_status.safety.sources with clears_on_ack. Clients should use the server's can_acknowledge flag: a weight trip can be acknowledged while its source is still active, because acknowledgement resets the baseline. A physically active curtain or cabinet-door source still makes POST /api/robot/safety/acknowledge return 409 with active_sources. Acknowledgement does not home the robot. The engineering component override endpoint also accepts weight_change, under the same robot:control permission as the other component bypasses.

At the driver layer, jogs now count as motion for monitoring. A relative jog interrupted by the safety guard fails instead of reporting completion or repeating its full distance; preserve this behavior when extending motor commands.

The hardware layer

• Five tool-axis motors and the gripper Servo share the RS-485 motor bus, and door is a sixth Motor built in openshelf_os/iobox.py. The default bus config prefers stable udev symlinks defined in openshelf_os/serial_ports.py: /dev/openshelf-rs485-ch1 (motor), ch2 (scale), ch3 (laser), ch4 (spare), each with legacy port fallbacks. The baudrate is a config value (robot.bus.<bus>.baudrate): the motor and laser buses run 115200 and the scale bus 9600, in the code defaults and in the config delivered to devices alike. Without the udev rule, generic /dev/ttyACM* names can belong to scanners instead.
• The barcode scanner uses a serial by-id path; override with OPENSHELF_BARCODE_PORT for scanners that do not appear at the built-in path. Do not default to /dev/ttyACM0 on waveshare systems — it can be RS-485 channel 1.
• GPIO (door, limit switches, curtain) runs on I2C expanders (PCA9535 / PCF8575) with a Raspberry Pi fallback and a fake implementation for tests.
• Limit switches are optional per machine: limit_switches_enabled on a motor entry defaults to true; set it to false on motors without a switch (a machine can have y_axis/z_axis fitted and stage/x_axis/door not).
• Hardware debugging helpers: scripts/laser.py (laser tray scan, single-cell rescan) and scripts/z_stall_probe.py (shelf-plate probe, z delta model) run while the long-running api container is stopped, otherwise the robot serial ports stay locked by the app server, in a one-off hardware-enabled compose container. That run carries its own -v <repo>/scripts:/app/scripts:ro bind mount because the dev overlay does not mount scripts/. The four scripts/y_*_diagnostic.py scripts have scripts/run_y_*.sh safety wrappers that require an explicit restraint and e-stop confirmation. Per-register Modbus reads log at DEBUG on the hardware.controller.controller logger and are silenced by default; set MODBUS_TRACE=true to restore them for bus debugging.

Mock mode never initializes any of this; the mock robot and cabinet are deterministic software stand-ins in openshelf_os_server/core/mock_robot.py, and the printer stand-in is the mock branch of _print_payload in the printer router. Do not add hardware fallbacks to either.

Config and state, in practice

openshelf_hardware/data/ holds everything: data/.mode and data/.last_start_mode (both written by scripts/start), data/.last_mode and data/.workspace.<mode>.json (written by the API to record the mode transition and what was copied versus adopted), data/audit-hmac-key.<mode> (a locally generated signing key), the per-mode sqlite database and config files, the storage grid files, calibration recordings, and generated state. The server migrates the selected mode's config at startup. To preview or pre-migrate all mode configs on a containerized device, obtain the matching released api image and run the offline migration runner with the application stopped:

docker compose -f docker-compose.yml run --rm --no-deps --pull never --entrypoint sh \
  -v "$PWD/scripts:/app/scripts:ro" api -lc \
  'cd /app/openshelf_hardware && uv run --no-sync python ../scripts/migrate_config.py data'

Repeat with --write added once the preview is correct, then restart through the repo-managed flow preserving the selected mode and hardware overlay.

Adding a config migration:

1. Add v006_<description>.py under openshelf_os/config_migrations/; its migrate(config) accepts the raw previous-schema dict and returns the transformed dict plus the list of changed field paths.
2. Register it under the destination version in runner.MIGRATIONS.
3. Bump config.schema.CURRENT_SCHEMA_VERSION.
4. Never modify a released migration — append a new step. Test value preservation, conflicting keys, optional fields, and the whole chain, and update generated config defaults, fixtures, and the device config template.

Adding a database migration: a new Alembic revision under openshelf_os_server/alembic/versions/; migrations run automatically at startup, so the next scripts/start applies it. The startup runner looks for openshelf_hardware/alembic.ini and, when it is absent, logs alembic_ini_missing and falls back to a bare Alembic config, so the checked-in packages/openshelf_os_server/alembic.ini steers manual alembic commands and not startup. A failed migration does not stop the app: the lifespan logs continuing_without_migrations, then Base.metadata.create_all creates only the missing tables, so a broken revision leaves a silently un-migrated database. Read the startup log after the first boot with a new revision.

Outside the containers

The API entry point is uv run start from openshelf_hardware/ (uvicorn on 0.0.0.0:8000), and the dev image runs it under debugpy listening on 0.0.0.0:5678, so a local debugger attaches without a rebuild. Every runtime path resolves from the environment, which is how a second checkout or a scratch workspace stays isolated.

Auth and audit

• Users authenticate with email and PIN and receive a JWT (Argon2-hashed PIN, short-lived access token plus refresh token); the developer user is seeded at startup and marked is_system. That system user is not a login shortcut: a relay service token resolves to the first is_system admin user, normally the developer user, while the development auto-user resolves to its own separately seeded is_system account named by auth.dev_auto_user. The audit surface is not anonymous; GET /api/audit/logs and the verify endpoints require an ordinary user JWT carrying audit:read.
• Roles grant permissions as resource:action strings with wildcard support (*, storage:*, *:read), seeded from DEFAULT_ROLES in main.py; access levels (three of them, Level 1 to Level 3) gate which items a user may touch. Every access check happens in a router dependency or the core layer, and the outcome is audited.
• Audit: audited calls write rows to audit_logs, chained with HMAC signatures, after the response. Auditing is on by default, and openshelf_os_server/middleware/audit.py holds the exclusion lists (ignored methods, always-excluded paths, successful reads), so a new endpoint needs a deliberate answer there only when it must stay out of the log. Every audit HMAC key has an id: v2 is what the device .env ships with, v2-<mode> is what the API generates for itself per mode; a locally pinned key must use another id, and an id must never be reused with a different key. Send X-Correlation-ID to associate API activity with audit records; GET /api/audit/logs filters by exact correlation_id.
• Cross-origin calls need no per-endpoint work: any origin may call the API, and no response advertises credentialed access, because authentication travels in an Authorization header or a token query parameter rather than a cookie. Pinning OPENSHELF_CORS_ORIGINS narrows the policy to the listed origins and enables credentialed handling for them (utils/cors_policy.py).
• The integration relay authenticates to a device with a unique opaque bearer token instead of a user email and PIN: give the raw token only to the relay, and start the OS container with its lowercase SHA-256 digest in OPSHELF_RELAY_SERVICE_TOKEN_SHA256. The token is accepted only by the relay-facing item endpoints, command submission and polling, robot status/events streams, and printing — not by users, roles, access levels, audit, calibration, or system configuration.

Extending Openshelf OS

The audit and correlation middleware apply to every router automatically; permission checks are explicit, through require_permissions("<resource>:<action>"), or require_relay_or_permissions(...) on a relay-facing route, from routers/auth_deps.py, so a new endpoint that touches items must repeat the access-level check of the endpoints it resembles. The contract checks also gate documentation: every operation needs a summary, a docstring description that is not the summary repeated verbatim, and tags that already exist in the document, which means tags declared in documentation.TAGS (openshelf_os_server/openapi_checks.py). require_permissions matches only the permissions stored on the caller's role, with *:read and *:* wildcards, so a brand-new resource:action string returns 403 for everyone until it is added to a role, through the roles API or DEFAULT_ROLES in main.py.

After every change:

1. Run the focused tests for the touched package, then the full shuffled suite (OPENSHELF_DATA_DIR=$(mktemp -d) uv run pytest --random-order), plus uv run ruff check packages and uv run mypy packages/openshelf_os_server/src.
2. For API surface changes, validate the contract from openshelf_hardware: uv run openapi --check regenerates the document from the live routes and validates it in memory, and uv run pytest packages/openshelf_os_server/tests/test_openapi.py compares it with the served routes. There is nothing to commit: uv run openapi --output <path> writes the JSON on demand, the generated file is gitignored, and the docs site regenerates it during its own build. Keep endpoint prose in route docstrings and field semantics on the models.
3. For frontend changes, cd dev_ui && npm run check && npm run build.
4. On a development machine, source-only changes hot-reload through the bind mounts; dependency or dockerfile changes need ./scripts/start --mode development --build.

Cloud relay portal

Customers of the cloud relay sign in at https://api.opshelf.com/settings with a one-time email link. From there they copy or rotate the API key for their organization and see the virtual machines assigned to them. Those customers, keys, and assignments live in the relay database. Device containers and ports still come from the relay config.json. Creating a customer or assigning a machine is a relay-admin command on the relay, described in the relay repository README.

Where to go next

• The Local API reference — the maintained endpoint contract, generated from this codebase.
• README.md in this repository — development workflow, mock mode, quick checks, hardware notes, and the config schema migration workflow.