Developer manual
Last updated
This manual is for senior engineers who take a fork of the openshelf_os codebase. It shows
how to run the stack, how the code is organized, and which files to change for common
extensions. Reading time is 25 to 35 minutes.
What Openshelf OS is
An Openshelf device is a cabinet with a gantry robot that stores and retrieves vials by
barcode. Each device runs the Openshelf OS stack as one Docker Compose project with two
services: a FastAPI backend (api, port 8000) and a SvelteKit frontend (dev_ui, port 8080).
The API is the contract. It exposes robot motion, inventory, calibration, diagnostics, and
printing through /api/* endpoints. The Local API reference
is generated from the code and is the maintained endpoint reference. The UI is one client of
the API, and lab software can call the API directly.
A fork owns three Python packages. The HTTP service depends on the domain and orchestration package, and the domain package depends on the low-level device drivers.
Repository layout
┌─────────────────────────────────────────────────────────────────┐ │ openshelf_os checkout │ │ │ │ │ │ ┌────────────────────────────┐ │ │ │ docker compose project │ │ │ │ │ │ │ │ │ │ │ │ ┌────────────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ dev_ui │ │ │ │ │ │ SvelteKit · port 8080 │ │ │ │ │ │ │ │ │ │ │ └────────HTTP┬/api───────┘ │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ ┌────────────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ api │ │ │ │ │ │ FastAPI · port 8000 │ │ │ │ │ │ │ │ │ │ │ └────────────┬───────────┘ │ │ │ │ │ │ │ │ └──────────────┼─────────────┘ │ │ ┌──────────────┼─────────────┐ │ │ │ openshelf_hardware │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌────────────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ openshelf_os_server ├─┼─────────────────┐ │ │ │ │ FastAPI service │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────┬───────────┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ▼ │ ▼ │ │ │ ┌────────────────────────┐ │ ╭──────────────────────────────╮ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ openshelf_os ├─┼►│ data/ │ │ │ │ │ domain + orchestration │ │ │ sqlite + config + mode files │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────┬───────────┘ │ ╰──────────────────────────────╯ │ │ │ │ │ │ │ │ ▼ │ │ │ │ ┌────────────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ hardware │ │ │ │ │ │ device drivers │ │ │ │ │ │ │ │ │ │ │ └────────────────────────┘ │ │ │ │ │ │ │ └────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────┘
dev_ui/ calls the API over HTTP and reads its SSE event streams. All other code runs inside
the api container. openshelf_os_server is the HTTP service, openshelf_os is the robot
and storage domain, and hardware contains only driver code.
data/ is the authoritative copy of local state, and the running stack reads and writes it.
An edit or a deletion there changes the device.
Run the stack
You need Docker with Compose v2, uv, and Node.js for the frontend checks.
Start and stop the stack only through scripts/start and scripts/stop. Raw
docker compose commands leave the mode file data/.mode out of sync.
To start local development for the first time, build the images:
After the first build, start without --build:
To stop the stack, run ./scripts/stop.
On a machine with a connected robot, camera, scanner, or printer, pass --hardware. Without
it, the API container can restart without the hardware compose overlay and lose device
passthrough. To restart with hardware, stop the stack first:
scripts/start adds the hardware overlay by itself when the selected mode's config sets
robot.enabled or robot_app.enabled to true. It never adds the overlay in mock mode. The
short flags -d, -h, -b, -p, and -m stand for --detached, --hardware, --build,
--pull, and --mode. The script also takes --force-recreate.
Other common starts:
Every checkout uses the same compose project name, openshelf_os. If you start a second
checkout, it replaces the containers of the first. To run two stacks side by side, set
OPSHELF_COMPOSE_PROJECT=<name> in each worktree.
Check that the stack is up
Open the UI at http://localhost:8080 and the health check at
http://localhost:8000/api/health.
The developer system account is [email protected]. Its PIN defaults to 1111 only in
development and mock. A development or mock UI that you open directly needs no sign-in. The
UI served through the relay signs in with the opshelf_developer_pin cookie and never asks
for the PIN.
Each container keeps at most five log files of 10 MiB each. docker logs stops at the first
truncated line, so after an unclean shutdown, read the container's JSON log file directly.
Operating modes
scripts/start is the only way to select a mode. It records the mode in data/.mode, and
the API exposes the mode read-only. If no mode file exists, a plain ./scripts/start starts
production. An API process started without scripts/start, with neither data/.mode nor
OPERATING_MODE set, falls back to development.
productionis the deployed device mode. It uses slim images with no dev tooling.developmentbind-mounts the threepackages/*/src/<package>trees anddev_ui/src, and the container reloads on.pychanges. Source-only changes need no rebuild. Dependency and Dockerfile changes need--build. The mounts do not includescripts/ortests/.demois a demo workspace. Robot storage is limited to the barcode-to-cell mappings indata/demo.json.mockis the workbench without hardware. It runs the real FastAPI service, authorization rules, and sqlite persistence. It does not initialize the robot, camera, scanner, sizer, printer, CUPS, or Sentry.
Each mode keeps its own persistent state files, such as base.development.db and
config.development.json. The first start in development or demo copies the production
state and leaves the original unchanged. Switching back to production restores the original
workspace.
Mock starts from a fresh base.mock.db with an empty inventory and seeded users and roles.
It never copies the production database or item state. It can still copy config.json to
config.mock.json, so configuration values carry over into mock.
should_enable_sentry decides whether Sentry runs. It is true for demo and production and
false for mock and development, so a demo run sends events to team telemetry.
If the .demo_workspace_seeded marker is missing, the next demo start re-runs the demo seed
and replaces the demo data. Delete the marker to recover demo state from production.
How mock mode behaves
Use mock mode for API and runtime work when you have no device.
- The virtual cabinet has 80 fixed cells. The front module comes first, then the back module,
with four shelves per side and ten cells per shelf. Occupancy comes from the sqlite
itemsanditem_locationstables plus the reserved homes of the configuredstorage_binsrows. - One deterministic FIFO worker runs the commands, and each takes a simulated 30 seconds. At
most 32 commands can be queued or running. A full queue returns
429withRetry-After: 1. - An optional top-level
request_idmakes a request idempotent for seven days. An exact replay returns the original command. A reusedrequest_idwith a different payload returns409. Completed command rows are pruned after seven days, so history and idempotency end together. - Real hardware has the same 32-command limit and the same seven-day
request_idwindow, both inopenshelf_os_server/core/command_lifecycle.py. - A restart marks interrupted commands as failed and never replays them.
- Mock accepts only
store_itemandretrieve_item. Any other command type returns400. - A store enrolls the item when the server admits the command. If the command then fails during execution, the item stays in the database. A store without a weight saves a deterministic weight.
POST /api/printreturnsprinted=trueand produces no label. It still requires theprinter:printpermission or a relay token. The barcode must be printable ASCII of at most 255 characters.- The developer account is seeded. In development and mock, a request without a token runs as that auto-user. Production and demo still require authentication.
Outside disposable local tests, keep JWT_SECRET and AUDIT_HMAC_KEY stable. Pin both in
the checkout .env with a unique AUDIT_HMAC_KEY_ID. Append to .env, and never overwrite
it.
Run the quality gates
No CI runs these checks. The only workflow in .github/workflows/ dispatches a release
build, so run every command below yourself before you merge.
The backend suite runs in a fixed order by default, and a fixed order can hide order dependencies. Before you land test changes, run the suite in random order at least once.
The tests also read the operating mode from openshelf_hardware/data/.mode, because importing
the app resolves runtime state at import time. A leftover mock mode file sends robot tests
down the mock paths. On a clean tree, unrelated tests that post a home command then fail
with 400.
This command shuffles the suite and uses a throwaway data directory. Use it whenever the test run is your evidence:
To run one module, pass its path, for example
uv run pytest packages/openshelf_os_server/tests/test_storage_endpoints.py.
openshelf_os_server/tests/conftest.py sets the test identity. It seeds [email protected]
with PIN 1234. Its autouse cleanup runs before and after each test, but only in two groups
of modules. It clears the bin, item, and location tables for modules named test_storage*,
and the audit, user, and role tables for modules named test_audit*. A new module that writes
those tables leaks rows into the shuffled run until you add it to that cleanup.
openshelf_os/__init__.py configures Sentry at import time. To turn Sentry off in a test,
set SENTRY_DSN_HARDWARE before the first import of that package.
Async tests use both pytest-asyncio and AnyIO. Only openshelf_os_server/pytest.ini sets
pytest-asyncio to auto mode. The other two packages set no asyncio_mode. A test marked
pytest.mark.anyio must take its background-task fixtures from @pytest.fixture, not from
@pytest_asyncio.fixture. Otherwise the worker attaches to a different event loop in the full
suite.
Packages and their responsibilities
┌─────────────────────────────────────────────────────┐
│ │
│ dev_ui │
│ SvelteKit views: storage, sizer, scale, │
│ camera, scanner, printer, motors, power, wifi │
│ │
└──────────────────────────┬──────────────────────────┘
│
│
┌────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ │ openshelf_os_server │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ routers/ │ │
│ │ auth · users · roles · access_levels · audit │ │
│ │ robot · storage · cabinet · calibration · chemicals │ │
│ │ camera · iobox_sizer · printer · report · telemetry │ │
│ │ onboarding · system · wifi · demo │ │
│ │ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼───────────────────────────────────────────────────────────▼─────────────────────────────────────────────────────────────────────▼ │
│ ┌─────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────┐ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ core/ │ │ │ │
│ │ middleware/ │ ┌─┤ long-lived services: robot runtime, mock robot, ├─┐ │ openapi.py │ │
│ │ audit (HMAC rows) · correlation (X-Correlation-ID) │ │ │ command lifecycle, permissions, laser auto-calibration, │ │ │ generator (checks live in openapi_checks.py) │ │
│ │ │ │ │ camera managers, crash recorder, bin storage │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ └─────────────────────────────────────────────────────┘ │ └────────────────────────────┬────────────────────────────┘ │ └────────────────────────────────────────────────────────────────────────┘ │
│ │ │ │ │
│ │ │ │ │
│ │ │ │ │
│ ┌────────────────────────────┘ │ └──────────────────────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────┐ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ database/ │ │ robot/ │ │ storage grid + cabinet │ │
│ │ SQLAlchemy models + Alembic migrations │ │ core · worker · commands · status │ │ │ │
│ │ │ │ │ │ │ │
│ └─────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────┘ └────────────────────────────────────────────────────────────────────────┘ │
│ │ │
└────────────────────────────▼───────────────────────────────────────────────────────────▼─────────────────────────────────────────────────────────────────────▼─────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────┐ ┌────────────────────────────────────────────────────────────────────────┐
┌─┼─────────────────────────────────────────────────────┼───┼─────────────────────────────────────────────────────────┼───┼────────────────────────────────────────────────────────────────────────┼─┐
│ │ config loader + schema + migrations │ │ laser calibration (scanpeanglef_zsstall) │ │ iobox · tool · serial ports · barcode scanner service · operating mode │ │
│ │ │ │ │ │ │ │
│ └─────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────┘ └────────────────────────────────────┬───────────────────────────────────┘ │
│ │ │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────┐ │
│ hardware │ │
│ │ │
│ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │ │ │
│ │ │ │ │
│ │ motors (Modbus) · laser · scale · TOF · cameras │◄┼────────────────────────────────────────────────────────────────────────────────────────────────────┘
│ │ printer · GPIO · curtain interlock │ │
│ │ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────┘ hardwarecontains only driver code and no domain logic. It providesSharedModbusControllerfor the RS-485 bus,Motor,Servofor the gripper,LaserSensor,ToFSensorBus,UvcCamera,ToolCamera, the label printer, the scale,MotionSafetyInterlockfor the curtain,DoorRelease, and the GPIO layer. The GPIO layer uses I2C expanders, falls back to Raspberry Pi GPIO, and has afakeimplementation for tests.openshelf_osis the domain package. It containsrobot/with the core orchestration, the FIFO worker, the pydantic command models, and the status types. It also contains the storage grid and cabinet, the config loader with schema versioning, the laser calibration suite, the I/O box, serial port mapping, and operating-mode handling.openshelf_os_serveris the service package. It contains the FastAPI app inmain.py, the routers, the audit and correlation middleware, the long-lived services incore/, the SQLAlchemy and Alembic database layer, and the OpenAPI generator. The generator's JSON output is a build artifact, not a committed file.
Startup order
Startup resolves configuration before the app lifespan in main.py begins. It then
initializes the services in this order:
- Resolve the mode's runtime paths: the config file, the database file, and the calibration
directory. Migrate the active config, then load it through
settings.get_root_config. - Run the Alembic migrations and create any missing tables.
- Seed the default roles, role permissions, access levels, and the developer user. When
auth.dev_modeis on, also seed the configured development auto-user. - Ensure the onboarding state and apply the mode transition.
- Recover interrupted robot commands. Queued and running commands become failed.
- Sync the storage bin identities from config and recover interrupted bin workflows.
- Start the mock robot runtime in mock mode, or the background robot initialization task on real hardware. Then start the camera managers, the I/O box sizer, and the crash video recorder.
- The app is ready.
/api/healthreportsstatus,mode,robot_status, androbot_phase, and returns503when initialization failed. It does not check the database, the cameras, or any other dependency.
On real hardware, robot initialization runs in tracked phases while the app already serves
requests. An initializing phase therefore does not mean that the start failed. Robot core
brings up the hardware in this order:
- The motor, scale, and laser buses.
- GPIO.
Tool.configure, which configures the gripper servo and the five axis motors in the ordery_axis,x_axis,z_axis,stage,rotate. It then initializes the laser and homes the axes:z_axisandstagefirst, thenrotate, thenx_axisandy_axis.IOBox.configure, which homesdoorlast.- The cabinet.
The code is in openshelf_os/tool.py (configure and _home_sequence) and
openshelf_os/robot/core.py.
core/init_progress.py tracks each component through the phases pending, connecting,
configuring, homing, ready, and failed. /api/health and the status stream report
that progress. The overall phase becomes degraded when a component registered as degrading
fails after the tracker has reached ready. The phase returns to ready when every degrading
component is ready again.
Homing checks the home limit input. If a motor has no limit switch, set
limit_switches_enabled: false on its motor entry, or homing fails.
Config schema
Runtime configuration lives in config.<mode>.json. Pydantic models load and validate it.
The top-level schema_version records the persisted schema. A missing value means version 1,
the layout from before versioning, which device setup still writes. The current version is
config.schema.CURRENT_SCHEMA_VERSION, which is 8.
The server migrates the active config before it loads it. The config of every other mode
migrates the next time that mode starts. config_migrations/apply.py runs the full chain of
migrations and validates the result. It then saves the original bytes in a new
<filename>.schema-v<N>.bak file, where N is the source version, and replaces the config
atomically. The backup file must not exist yet, so an existing backup is a conflict.
Startup stops when the config has a newer schema, when a migration fails, or when a backup conflicts. Keep the existing backup and reconcile it before you retry.
Device scripts and other offline entry points do not migrate configs. Run
scripts/migrate_config.py for them. Preview first, then run it with --write while the
application is stopped.
Never fix a startup error by changing only schema_version. The old field names would then
lose their meaning.
These migrations change behavior that you can see on a device:
- Version 5 renames the left and right scan config keys and bin home sides to front and back.
- Version 6 adds a separate IO pose for the multi-item tray. It seeds the pose's XY position
from
PICKUP_POSITION_Land its stage and Z extensions from the previous shared IO values. Existing motion stays the same until someone recalibrates the pose.
Versions 7 and 8 only drop keys that nothing reads any more: the z stall probe keys, and the keys that chose between stepped and flying laser scans.
Database and state
Each mode has one sqlite database: base.db in production, base.development.db,
base.mock.db, and so on. SQLAlchemy manages the database, and Alembic migrates it. The
tables hold:
- users, roles, and access levels
- the HMAC-chained
audit_logs itemsanditem_locationsrobot_commands- storage bins and their workflows
item_sizer_measurements,telemetry, andstorage_cells
The storage grid is not in the database. It is a JSON file under data/, storage.json or
one of its mode variants, that describes the modules, shelves, and cells.
The current side values are front_module and back_module. Demo location strings use F
and B. Older data used left and right, and three mechanisms rename it:
- The database rename also runs after startup stamps the Alembic baseline. It updates
item_locationsandstorage_binseven when Alembic skips the historical revision chain. - Demo mapping readers, including the demo seed script, rewrite old
LandRlocations in place throughdemo_side_rename.py. They keep a.demo-side-rename.bakcopy. - Saved laser runs map old side names in memory when they load. Their files stay unchanged.
Two SSE endpoints stream live status and results from in-process queues:
/api/robot/status and /api/robot/events. /api/telemetry is an ordinary JSON GET, not a
stream.
Camera video uses separate transports: MJPEG over HTTP, and a WebSocket at
/api/camera/ws/{camera_id}. Both take the token as a token query parameter, because
EventSource and WebSocket clients cannot set an Authorization header.
Frontend conventions
The shared client in dev_ui/src/lib/api.ts fetches the storage grid to find out which side
names the device uses, left and right or front and back. lib/side.ts maps both
vocabularies to the stable side indices 0 and 1. UI state uses the canonical front and back
tokens, and the API wrappers translate outbound sides to the device's vocabulary. The
client's laser snapshot normalization also maps the legacy per-side field names. In
side-specific code:
- Load the grid before you send side-specific requests.
- Compare received side strings with
sideIndex(). - Use the shared label helpers.
+layout.svelte owns the shared robot status stream, NeedsHomeAlert, SafetyAckAlert, and
ToastStack. The homing and acknowledgement prompts follow the server status.
NeedsHomeAlert offers one action per affected motor and sends queued homing requests one at
a time, because the server rejects concurrent motion. The status page also shows the required
actions and the sources that block motion.
Use ErrorMessage and StatusMessage for action feedback. Both post to lib/toasts.ts.
Errors stay until the user dismisses them. Status messages expire after ten seconds. Identical
visible messages appear once.
The status page reads the device identity and alias from /api/system/info and the mode from
/api/system/mode. Readiness still comes from health and robot status. The public info
endpoint also returns the nullable fields release_sha and release_branch. Their values come
from /etc/opshelf/release, and the RELEASE_SHA and RELEASE_BRANCH environment variables
override that file. The alias comes from /etc/opshelf/device-alias.
Calibration views live in lib/components/calibration/:
CellCalibration.svelteplaces cell selection next to the motion controls, with explicit calibrate and save-and-test actions.StorageLayoutEditor.svelteowns bulk deletion and the coordinate inference for new cells. Both are client workflows over the existing grid endpoints. Bulk deletion can partly succeed, and the editor reloads the grid afterwards. The editor infers new cell coordinates from the origin or from calibrated neighbours when it can, then saves them with a separate grid update.IOBoxCalibration.sveltecalibrates the low and high pickup points, which share one Z and stage extension. It also calibrates the multi-item tray pose, which has its own XY, Z, and stage values and its own test cycle.
How a command runs
Use this path as the reference for anything you add. It follows a store_item command from
the client to the persisted result.
┌────────┐ ┌───────────────────────────┐ ┌───────────────────┐ ┌────────┐ ┌─────────────────────┐ ┌────────────┐ ┌──────────┐ ┌────────────┐
│ client │ │ middleware (auth + audit) │ │ router /api/robot │ │ sqlite │ │ robot worker (FIFO) │ │ robot core │ │ hardware │ │ SSE stream │
└────┬───┘ └─────────────┬─────────────┘ └─────────┬─────────┘ └────┬───┘ └──────────┬──────────┘ └──────┬─────┘ └─────┬────┘ └──────┬─────┘
│ │ │ │ │ │ │ │
│ POST /api/robot/commands │ │ │ │ │ │ │
│─────────────────────────────▶ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ ├───┐ │ │ │ │ │ │
│ │ │ validate token, check permissions │ │ │ │ │
│ ◀───┘ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ ├───┐ │ │ │ │ │
│ │ │ │ validate item, access, location rules │ │ │ │
│ │ ◀───┘ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ persist command (queued) │ │ │ │ │
│ │ │─────────────────────────────▶ │ │ │ │
│ │ │ │ │ │ │ │
│ command id │ │ │ │ │ │
◀╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌│ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ write HMAC audit row after the response │ │ │ │ │
│ │─────────────────────────────────────────────────────────▶ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ dequeue next command │ │ │ │
│ │ │ ◀─────────────────────────│ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ execute store_item │ │ │
│ │ │ │ │───────────────────────▶ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ home, move, gripper, scale/laser confirm │ │
│ │ │ │ │ │─────────────────────────────────────────────▶ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ status snapshots + confirmations │ │
│ │ │ │ │ ◀╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌│ │
│ │ │ │ │ │ │ │
│ │ │ │ mark succeeded │ │ │
│ │ │ ◀─────────────────────────────────────────────────│ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ publish event │ │
│ │ │ │ │ │────────────────────────────────────────────────────────────▶
│ │ │ │ │ │ │ │
│ │ │ │ robot events stream │ │ │ │
◀╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌│
│ │ │ │ │ │ │ │
│ GET /api/robot/commands/{id} │ │ │ │ │ │
│─────────────────────────────────────────────────────────▶ │ │ │ │ │
│ │ │ │ │ │ │ │
│ persisted lifecycle + result │ │ │ │ │ │
◀╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌│ │ │ │ │ │
│ │ │ │ │ │ │ │
┌────┴───┐ ┌─────────────┴─────────────┐ ┌─────────┴─────────┐ ┌────┴───┐ ┌──────────┴──────────┐ ┌──────┴─────┐ ┌─────┴────┐ ┌──────┴─────┐
│ client │ │ middleware (auth + audit) │ │ router /api/robot │ │ sqlite │ │ robot worker (FIFO) │ │ robot core │ │ hardware │ │ SSE stream │
└────────┘ └───────────────────────────┘ └───────────────────┘ └────────┘ └─────────────────────┘ └────────────┘ └──────────┘ └────────────┘ CorrelationIdMiddlewaretakesX-Correlation-IDfrom the request, or generates one, and echoes it on the response. The audit middleware reads the header from the request, so only a client-supplied id reaches an audit row. A generated id reaches the logs through the logging context, not through the audit table.- The router validates the payload. The item must exist, the caller's access level must cover it, and the location rules must hold. For a retrieve, the router checks the requested location and access before it queues the command. The item location is removed when the retrieve completes.
- The router saves the command as
queued. If 32 commands are already active, it returns429. If the samerequest_idarrived within the last seven days, it returns the stored command and queues nothing. - The FIFO worker dequeues the command and passes it to the robot core.
- The robot core runs the motion: homing, moves along the tool axes, the gripper action, and
the scale and laser confirmations. The shared
MotionSafetyInterlockblocks motion when the curtain, the cabinet door, or a weight change interrupts it. The robot core publishes status snapshots throughout. - The outcome,
succeededorfailed, is saved and broadcast on the event stream. In mock mode, the same lifecycle runs on one deterministic worker with a simulated execution time of 30 seconds. - On a motion fault,
Tool._handle_faultattaches afault_contextto the exception. The context holds a snapshot of each axis motor, the phase, the coordinates, and any earlier fault context. The worker reports the fault with that context and the crash recorder footage. The machine does not recover by itself, and the log line asks for manual intervention.
Both middlewares wrap every router. The audit middleware writes its row after the handler runs, and it skips some requests:
- It never writes
OPTIONSrequests,/api/health,/api/robot/debug/motors/probe, or/api/calibration/laser/stream. - It always writes a response with status 400 or higher.
- It skips successful reads on an allowlist.
/api/storage/items/lookupcounts as a read.
Audit rows are kept for 120 days.
Barcodes never appear in a URL path. Uvicorn decodes %2F before routing, so a path could
not address a barcode that contains /. The item and print endpoints take the barcode in the
POST body.
Weight-change interruptions
openshelf_os/robot/weight_guard.py samples the scale during Y-axis motion. It also covers
the startup clearance jog, which runs before the worker exists. The guard compares each
reading with the last idle weight and blocks motion when a change persists.
The guard is on by default when the scale and the Y motor are configured. Its settings live
under robot.io.scale. The defaults are:
motion_guard_delta_grams: 10motion_guard_min_samples: 2motion_guard_arm_delay_s: 0.05motion_guard_ack_grace_s: 1.0
Each sample calls Scale.read_weight_fast(), which makes one Modbus transaction with no
retries and a short timeout. If the scale bus is busy, the guard skips that sample. After
repeated read failures, the guard suspends sampling and backs off. To diagnose gaps in
coverage, inspect the error and failed fields of the guard's safety source. After the guard
requests the motion block, it writes a trip record to the relative directory
data/weight_faults/. The record holds the recent samples and the operation context.
The worker reports weight_change in robot_status.safety.sources, with clears_on_ack.
Base the client's acknowledge action on the server's can_acknowledge flag. A weight trip can
be acknowledged while its source is still active, because acknowledgement resets the
baseline. An active curtain or cabinet-door source still makes
POST /api/robot/safety/acknowledge return 409 with active_sources. Acknowledgement does
not home the robot.
The engineering endpoint for component overrides also accepts weight_change. It requires
the same robot:control permission as the other component bypasses.
At the driver layer, monitoring treats jogs as motion. If the safety guard interrupts a relative jog, the jog fails. It does not report completion, and it does not repeat its full distance. Keep this behavior when you extend the motor commands.
Hardware layer
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────┐
│ serial + I2C buses hardware package │ │
│ │ │
│ │ │
│ ┌────────────────────────────────────────────┐ ┌──────────────────┐ ┌────────────────────┐ ┌──────────────────────────────────────────────────────┐ ┌────────────────┐ ┌─────────────────┐ │ ┌─────────────────────────────────────────┐ ┌──────────────────────┐ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ RS-485 ch1 │ ┌─┤ RS-485 ch2 │ ┌─┤ RS-485 ch3 │ ┌─┤ TOF bus │ ┌─┤ I2C / rpi GPIO │ │ barcode scanner │ │ │ cameras (UVC, tool, overhead, overview) │ │ label printer (CUPS) │ │
│ │ motor bus · 115200 │ │ │ scale bus · 9600 │ │ │ laser bus · 115200 │ │ │ │ │ │ │ │ serial by-id │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ └──────────────────────┬─────────────────────┘ │ └──────────────────┘ │ └────────────────────┘ │ └──────────────────────────────────────────────────────┘ │ └────────────────┘ └────────┬────────┘ │ └────────────────────┬────────────────────┘ └───────────┬──────────┘ │
│ │ │ │ │ │ │ │ │ │ │
├────────────────────────┼───────────────────────┼──────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────┼──────────┘ │ │ │
│ │ │ │ │ │ │ │ │ │
│ ├───────────────────────┴───────────┬──────────┴────────────┬───────────┴──────────────────────────────────────────────────────────┘ │ │ │ │
│ │ │ │ │ │ │ │
│ ▼ ▼ ▼ │ │ │ │
│ ┌────────────────────────────────────────────┐ ┌──────────────────┐ ┌────────────────────┐ ┌──────────────────────────────────────────────────────┐ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │
│ │ SharedModbusController │ │ TOF sensors ├─┐ │ GPIO manager ├─┬►│ robot core / worker · iobox · tool · barcode service │◄────────────────────────────────┴─────────────────────────────────┴────────────────────────────────────┘ │
│ │ │ │ │ │ │ │ │ │ │ │
│ └────────────────────────────────────────────┘ └──────────────────┘ │ └────────────────────┘ │ └──────────────────────────────────────────────────────┘ │
│ │ │ │ ▲ │
│ ▼───────────────────────┬───────────▼──────────┼────────────▼───────────┼─────────────────────────────▼ │
│ ┌────────────────────────────────────────────┐ │ ┌──────────────────┐ │ ┌────────────────────┐ │ ┌───────────────────────────┴──────────────────────────┐ │
│ │ │ │ │ │ │ │ │ │ │ │ │
│ │ Motor and Servo classes │ │ │ │ │ │ │ │ │ │ │
│ │ instances are built in Tool and IOBox ├─┘ │ scale ├─┘ │ laser sensor ├─┘ │ curtain interlock │ │
│ │ x · y · z · stage · rotate · door, one bus │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ └────────────────────────────────────────────┘ └──────────────────┘ └────────────────────┘ └──────────────────────────────────────────────────────┘ │
│ │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
- The five tool-axis motors and the gripper
Servoshare the RS-485 motor bus.dooris a sixthMotoron the same bus, built inopenshelf_os/iobox.py. - The default bus config prefers the stable udev symlinks defined in
openshelf_os/serial_ports.py:/dev/openshelf-rs485-ch1for the motors,ch2for the scale,ch3for the laser, andch4as a spare. Each has legacy port fallbacks. Without the udev rule, generic/dev/ttyACM*names can belong to scanners instead. - The baud rate is the config value
robot.bus.<bus>.baudrate. The motor and laser buses run at 115200 and the scale bus at 9600. The code defaults and the config shipped to devices agree. - The barcode scanner uses a serial by-id path. For a scanner that does not appear at the
built-in path, set
OPENSHELF_BARCODE_PORT. Do not default to/dev/ttyACM0on Waveshare systems, where that name can be RS-485 channel 1. - GPIO for the door, the limit switches, and the curtain runs on I2C expanders (PCA9535 or PCF8575). It falls back to Raspberry Pi GPIO and has a fake implementation for tests.
- Limit switches are optional per motor.
limit_switches_enabledon a motor entry defaults totrue. Set it tofalseon a motor without a switch. For example, a machine can have switches ony_axisandz_axisbut not onstage,x_axis, ordoor. - Per-register Modbus reads log at DEBUG on the
hardware.controller.controllerlogger, and that logger is silent by default. To see the reads when you debug a bus, setMODBUS_TRACE=true.
Mock mode initializes none of this hardware. The mock robot and cabinet are deterministic
software stand-ins in openshelf_os_server/core/mock_robot.py. The printer stand-in is the
mock branch of _print_payload in the printer router. Do not add hardware fallbacks to
either.
Config and state files
openshelf_hardware/data/ holds all runtime state:
.modeand.last_start_mode, whichscripts/startwrites..last_modeand.workspace.<mode>.json, which the API writes. They record the mode transition and which state the API copied or adopted.audit-hmac-key.<mode>, a locally generated signing key.- The sqlite database and the config file of each mode.
- The storage grid files.
- Calibration recordings and other generated state.
Migrate every mode's config offline
The server migrates only the selected mode's config at startup. To preview or pre-migrate the
configs of all modes on a containerized device, get the matching released api image, stop
the application, and run the offline migration runner:
When the preview is correct, run the same command with --write added after data. Then
restart through scripts/start with the same mode and the hardware overlay.
Add a config migration
Never modify a released migration. Append a new step instead.
- Add
v009_<description>.pyunderopenshelf_os/config_migrations/. Itsmigrate(config)function takes the raw dict of the previous schema. It returns the transformed dict and the list of changed field paths. - Register the step under its destination version in
runner.MIGRATIONS. - Bump
config.schema.CURRENT_SCHEMA_VERSION. - Test value preservation, conflicting keys, optional fields, and the whole chain.
- Update the generated config defaults, the fixtures, and the device config template.
Add a database migration
Add an Alembic revision under openshelf_os_server/alembic/versions/. Migrations run at
startup, so the next scripts/start applies the revision.
After the first boot with a new revision, read the startup log. A failed migration does not
stop the app. The lifespan logs continuing_without_migrations, and
Base.metadata.create_all then creates only the missing tables. A broken revision therefore
leaves an unmigrated database, and nothing else reports it.
The startup runner looks for openshelf_hardware/alembic.ini. If that file is absent, the
runner logs alembic_ini_missing and falls back to a bare Alembic config. The checked-in
packages/openshelf_os_server/alembic.ini therefore applies to manual alembic commands, not
to startup.
Run the API outside the containers
From openshelf_hardware/, uv run start starts the API with uvicorn on 0.0.0.0:8000. The
dev image runs the API under debugpy, which listens on 0.0.0.0:5678, so you can attach a
local debugger without a rebuild.
Every runtime path resolves from the environment. These variables keep a second checkout or a scratch workspace isolated:
OPENSHELF_DATA_DIRis the data directory. Config, database, calibration, and generated audit keys resolve against it.OPENSHELF_CONFIG_PATHis the config file inside the data directory.scripts/startrejects host paths outside the data directory.DATABASE_URLis a full SQLAlchemy URL. It overrides the per-mode sqlite file.OPERATING_MODE,DEPLOYMENT_ENVIRONMENT, andSENTRY_ENVIRONMENTare the mode and environment labels.scripts/startsets them.JWT_SECRET,AUDIT_HMAC_KEY, andAUDIT_HMAC_KEY_IDare the signing material. The checkout.envdelivers them.OPSHELF_RELAY_SERVICE_TOKEN_SHA256is the lowercase SHA-256 digest of the relay bearer token.OPSHELF_COMPOSE_PROJECTis the compose project name for this checkout.OPENSHELF_BARCODE_PORT,OPENSHELF_BARCODE_BAUDRATE, andOPENSHELF_DISABLE_BARCODEset the scanner path and speed, or bypass the scanner.OPENSHELF_PRINTER_DEVICEis the printer device path. The default is/dev/usb/lp0.LOG_LEVELsets the log level.MODBUS_TRACEturns on per-register Modbus traces.UVICORN_RELOADturns on reload inside the container.OPENSHELF_CORS_ORIGINSis a comma-separated list of exact origins. Without it, the API accepts any origin and never advertises credentialed access.CAS_COMMON_CHEMISTRY_API_KEYis an optional key for Common Chemistry lookups. Set it only in the server environment.VITE_API_BASE_URLis the API base URL fordev_ui. Without it, the client uses<origin>/<id>/apiwhen the page path starts with an 8-hex-digit device id orsessionStorage.deviceIdholds one. Otherwise it useshttp://<hostname>:8000/api. The logic is indev_ui/src/lib/api.ts.
The crash recorder has its own variables:
| Variable | Effect | Default |
|---|---|---|
CRASH_RECORDER_ENABLED |
turns the recorder on or off | |
CRASH_RECORDER_LOOKBACK_SECONDS |
recording window | 15 s |
CRASH_RECORDER_FPS |
frame rate | 3 fps |
CRASH_RECORDER_MAX_WIDTH |
maximum frame width | 640 px |
CRASH_RECORDER_MAX_CAMERA_BUFFER_MIB |
camera buffer cap | 16 MiB |
CRASH_RECORDER_MAX_ATTACHMENT_MIB |
cap per attachment file | 4 MiB |
CRASH_RECORDER_MAX_TOTAL_ATTACHMENT_MIB |
cap for all attachments | 14 MiB |
Auth and audit
Users and login
Each managed user has a unique email, a four-digit PIN hashed with Argon2, a role, an
optional access level, an active flag, and an optional unique login barcode. /api/users
excludes system users. Deleting a user deactivates the account and keeps the row. Creating,
editing, and deactivating users each require the matching users:* permission.
Operator login takes the PIN plus one identifier: an email, an unambiguous full name, an assigned barcode, or a short-lived signed scan token. Login returns an access token and a refresh token.
The unauthenticated lock-screen endpoints show user cards only when there are at most three
active non-system users. Larger installations get identifier entry instead. /api/auth/scans
streams tokenized scanner events and never exposes raw barcode values. After a failed login,
each credential backs off exponentially, up to 60 seconds.
Printing a user's login barcode requires both users:write and printer:print. The operation
rotates the assignment before it prints and never returns the barcode in the response. If
printing fails, the new assignment stays, and the caller must retry the whole rotate-and-print
operation.
System accounts
Startup seeds the developer user and marks it is_system. A relay service token resolves to
the first is_system admin user, normally the developer user. The unauthenticated development
fallback resolves to the active is_system account named by auth.dev_auto_user, normally
the same developer user.
The audit endpoints are not anonymous. GET /api/audit/logs and the verify endpoints require
an ordinary user JWT with audit:read.
Permissions and access levels
Roles grant permissions as resource:action strings. Wildcards such as *, storage:*, and
*:read are allowed. DEFAULT_ROLES in main.py seeds the roles. Three access levels,
Level 1 to Level 3, control which items a user may touch. Every access check runs in a
router dependency or in the core layer, and the audit log records the outcome.
Audit log
Audited calls write rows to audit_logs after the response, and HMAC signatures chain the
rows. Auditing is on by default. openshelf_os_server/middleware/audit.py holds the exclusion
lists for ignored methods, always-excluded paths, and successful reads. A new endpoint needs
an entry there only if it must stay out of the log.
Every audit HMAC key has an id. The device .env ships with v2, and the API generates
v2-<mode> for the keys it creates per mode. A locally pinned key must use a different id.
Never reuse an id with a different key.
To link API activity to audit records, send X-Correlation-ID. GET /api/audit/logs filters
by exact correlation_id.
Cross-origin calls
Cross-origin calls need no per-endpoint work. Any origin may call the API. No response
advertises credentialed access, because authentication travels in an Authorization header
or a token query parameter, not in a cookie. If you set OPENSHELF_CORS_ORIGINS, the policy
accepts only the listed origins and allows credentialed requests from them. The policy is in
utils/cors_policy.py.
Relay service token
The integration relay authenticates to a device with a unique opaque bearer token instead of
a user email and PIN. Give the raw token only to the relay. Start the OS container with the
token's lowercase SHA-256 digest in OPSHELF_RELAY_SERVICE_TOKEN_SHA256.
The token works only on the relay-facing endpoints: the item endpoints, command submission and polling, the robot status and events streams, and printing. It does not work on users, roles, access levels, audit, calibration, or system configuration.
Extend Openshelf OS
Add an API endpoint
The audit and correlation middleware apply to every new router without extra work. Permissions do not.
- Create
openshelf_os_server/routers/<name>.pyand register it inmain.pywithinclude_router. - Define pydantic request and response models. Type the 2xx response as a model, because an empty object schema fails the contract check.
- Add an auth dependency from
routers/auth_deps.py. Userequire_permissions("<resource>:<action>"), orrequire_relay_or_permissions(...)on a relay-facing route. - Grant the permission string to a role, through the roles API or
DEFAULT_ROLESinmain.py.require_permissionsmatches only the permissions stored on the caller's role, with the*:readand*:*wildcards. A newresource:actionstring returns403for everyone until a role has it. - If the endpoint touches items, repeat the access-level check of the endpoints it resembles. No dependency applies that check for you.
- Document the errors with
responses=response_docs(errors=(...)). - Give the route a
summaryand a docstring that adds to the summary instead of repeating it. Use only tags declared indocumentation.TAGS. The contract checks inopenshelf_os_server/openapi_checks.pyenforce both rules. - If the route must stay out of the audit log, add it to the exclusion lists in
middleware/audit.py. - Add tests under
openshelf_os_server/tests/.
Add a robot command
Change these files:
openshelf_os/robot/commands.pyfor the pydantic command modelopenshelf_os/robot/core.pyfor the executionopenshelf_os/robot/worker.pyfor the queueingopenshelf_os_server/core/mock_robot.pyfor the mock behavioropenshelf_os_server/routers/robot.pyfor the endpoint
Add a hardware device
Put the driver in hardware/src/hardware/<device>.py, with no domain logic. Integrate it in
openshelf_os through the I/O box, the tool, or the robot core, and add a config key.
The config does not define the axes. Tool builds five hard-coded motors and IOBox builds
door. robot/core.py keeps the hard-coded lists _INIT_MOTOR_COMPONENTS and POWER_RAILS.
An extra axis needs an entry in all four places.
Add a config key
Change the models and the loader in openshelf_os/config/. If the schema changes, add a
migration step and bump CURRENT_SCHEMA_VERSION, as described in "Add a config migration".
Add a database table or column
Change openshelf_os_server/database/models.py and add an Alembic revision.
Add a UI view
- Create
dev_ui/src/routes/<view>/. - Add an entry to the
linkslist indev_ui/src/routes/+layout.svelte. Without that entry, no link leads to the view. - Add one typed wrapper per endpoint to the shared client in
dev_ui/src/lib/api.ts. - Use
dev_ui/src/lib/sse.tsfor streams.
A permission-restricted view can look healthy during development, because every call runs as
the developer account. Direct development and mock requests resolve to the developer
auto-user. Behind the relay, the client logs in as the developer account with the
opshelf_developer_pin cookie.
Verify every change
- Run the focused tests for the touched package. Then run the full shuffled suite with
OPENSHELF_DATA_DIR=$(mktemp -d) uv run pytest --random-order, and runuv run ruff check packagesanduv run mypy packages/openshelf_os_server/src. - For API changes, validate the contract from
openshelf_hardware.uv run openapi --checkregenerates the document from the live routes and validates it in memory.uv run pytest packages/openshelf_os_server/tests/test_openapi.pycompares the document with the served routes. - For frontend changes, run
cd dev_ui && npm run check && npm run build. - On a development machine, source-only changes reload through the bind mounts. Dependency
and Dockerfile changes need
./scripts/start --mode development --build.
The OpenAPI document is not committed. uv run openapi --output <path> writes the JSON on
demand, the generated file is gitignored, and the docs site regenerates it in its own build.
Keep endpoint prose in route docstrings and field meanings on the models.
Cloud relay portal
Customers of the cloud relay sign in at https://console.opshelf.com with a one-time email
link. Anyone with an email address on the customer's registered domain can sign in. In the
console, customers copy or rotate their organization's API key and see the virtual machines
assigned to them.
The relay database stores the customers, keys, and assignments. Device containers and ports
still come from the relay config.json. To create a customer or assign a machine, run a
relay-admin command on the relay. The README of the relay repository describes the
commands.
Accounts on the staff domain get an organization switcher next to Sign out. The staff
domain is opshelf.com by default, and relay.staff_domain in the relay config changes it.
Choosing another organization re-scopes only the current browser session. The console then
shows the other organization's API key and robots. Any robot interface opened for the previous
organization ends on its next request. The session keeps its own sign-in address, which the
console shows as Signed in. The relay logs every switch.
Where to go next
- The Local API reference is the maintained endpoint
contract, generated from the
openshelf_oscode. README.mdin theopenshelf_osrepository covers the development workflow, mock mode, quick checks, hardware notes, and the config schema migration workflow.