Status: working end to end, 2026-08-07. Five instances booted concurrently on this machine; a custom-assembled FreeBSD distro boots, self-configures and accepts SSH.
PARKVPS exists to replace the proxied-web "fake browser" behind RAVIO's PARK scene with real machines: spin up operating systems, tab between them, drive them from the cockpit. This document is the authoritative description of what was built and what was measured.
vpsd/vps.py the instance supervisor (Python stdlib only, no root)
vpsd/api.py HTTP API, serial-console broker, VNC WebSocket proxy
vpsd/fetch_base.sh download + SHA512-verify + expand a FreeBSD base image
vpsd/web/ the fleet UI, the terminal, and the framebuffer viewer
var/ images, per-instance overlays, seeds, sockets, logs (gitignored)The distro moved out. T&R — Travel & RRABBIT — is everything that used to be distro/, and is now its own repo: <https://github.com/c-u-l8er/travel-and-rrabbit>. It stopped being a detail of this hypervisor and became a thing in its own right.
The seam was already here: vpsd runs machines, the distro makes the thing they run, and neither imports the other. Sections 3, 4 and the FreeBSD-specific findings below are kept because they are the record of how these images came to exist and of what this supervisor had to accommodate — but the authoritative copy of the distro's own documentation now lives in that repo's README.
Measured on this box: 24 cores, AMD-V, /dev/kvm mode 0666, 550 GB free.
No root anywhere in the normal path. /dev/kvm is world read-write here, so guests run as the calling user. Three consequences follow and they are the main design constraints:
No bridged/TAP networking (needs root). Instances use user-mode SLIRP with
explicit host port forwards, recorded per instance. They are reachable from the host and not yet from each other. QEMU's socket netdev is present and is the unprivileged route to VM-to-VM networking when that is wanted.
No libvirt. It is not installed and installing it needs sudo. One QEMU
process per instance, supervised by pid + QMP socket, with all state on disk as JSON — which is also what makes the RAVIO bridge trivial: reading the fleet is reading a directory.
UEFI via OVMF. Firmware code is shared read-only; only the variable store
is per-instance.
Instances are qcow2 overlays on one shared golden image, so spawning costs kilobytes rather than gigabytes.
python3 vpsd/vps.py doctor # check host prerequisites
python3 vpsd/vps.py create NAME --start
python3 vpsd/vps.py list
python3 vpsd/vps.py console NAME # serial console, Ctrl-] detaches
python3 vpsd/vps.py ssh NAME -- uname -a
python3 vpsd/vps.py reimage NAME --start # reset disk to golden + fresh seed
python3 vpsd/vps.py stop|destroy NAMEThese are the things that were wrong in the obvious approach. All verified on 15.1-RELEASE-p2.
FreeBSD 15 ships pkgbase. The base system is ~509 packages in a FreeBSD-base repo. Building a root filesystem is pkg -r <dir> install — no source tree, no buildworld, no release tarballs. This is what makes the distro a days-long job rather than a months-long one, and it is the single most important fact in this document.
`pkg -r` resolves everything against the target root, including trust. An empty directory has no signing keys, so pkg fails with "Error opening the trusted directory", which reads like a network problem and is not. Copy /usr/share/keys/pkgbase-15 and /etc/pkg/*.conf into the staged root first.
cloud-init is not installed. nuageinit is. FreeBSD 15's cloud images are handled by /usr/libexec/nuageinit, a Lua reimplementation in base that supports a subset of cloud-config. Two traps:
runcmd entries must be plain strings. The ["sh", "-c", "..."] list
form that real cloud-init accepts kills nuageinit with "attempt to concatenate a table value", and the crash aborts the rest of the user-data. One list-form runcmd silently costs you every key below it, which presents as packages: being ignored rather than as any error.
growpart, resize_rootfs, final_message and disable_root are not
implemented.
The only place the crash is reported is /var/log/nuageinit.log in the guest.
nuageinit is `KEYWORD: firstboot`. It runs exactly once, on the first boot of a given disk, and has none of cloud-init's "re-run when the instance-id changes" behaviour. Regenerating the seed and rebooting is silently a no-op. Worse, the rc script appends to nuageinit.log, so after a reboot you are reading the first boot's stack trace and diagnosing a crash that is not happening any more. This is why reimage exists and why reseed carries a warning.
`KEYWORD: firstboot` scripts are gated on `/firstboot` existing. An assembled image must create that file. Without it the image boots perfectly and silently ignores its seed drive, so no SSH key is ever installed and the instance is unreachable.
Stock cloud images re-patch themselves on every first boot. /etc/rc.conf ships firstboot_pkg_upgrade_enable=YES with firstboot_pkg_upgrade_repos="FreeBSD-base". For a fleet this is a ~512 MiB download per instance, forever. Removing it is most of the size win in §5.
`rm -rf` cannot delete a staged root. FreeBSD sets the system-immutable (schg) flag on parts of base including /sbin/init and /var/empty. Clear it with chflags -R noschg first, or the next build layers on top of a half-deleted tree.
Four base packages look optional and are not. FreeBSD-runtime alone gives you a kernel that mounts root and then fails:
| missing | symptom |
|---|---|
FreeBSD-rc | no /etc/rc at all → straight to Enter full pathname of shell |
FreeBSD-ufs | fsck_ufs not found → ERROR: ABORTING BOOT, single-user |
FreeBSD-geom | gpart missing → growfs cannot expand the partition |
FreeBSD-utilities | no awk, which the rc scripts use constantly |
Plus FreeBSD-pam, without which sshd cannot authenticate anyone. Note the ssh package is FreeBSD-ssh, not FreeBSD-openssh.
../travel-and-rrabbit/tandr.conf is the distro — package set, enabled services, identity, image geometry. build-image.sh is mechanism and should not need editing to produce a different flavour. An operator ../travel-and-rrabbit/overlay/ tree is copied over the staged root last, so local changes never require editing the script.
The assembler is the NomadBSD approach updated for pkgbase: seed trust → install base packages → install ports packages → write configuration → makefs an ESP and a UFS root → mkimg a GPT image.
Choices worth stating:
Root is found by GPT label, never device name. vtbd0 under virtio,
ada0 under SATA, nvd0 under NVMe — a device-name fstab makes an image that only boots on the hypervisor it was built on.
Serial console is primary (comconsole,vidconsole + boot_multicons).
The image is built to be run headless by a supervisor; a guest that only talks to a framebuffer cannot be driven or logged.
No `resolv.conf` is baked in — it would pin every instance to the build
host's nameserver.
`firstboot_pkg_upgrade_enable` is deliberately not set. Patching is a
build-time job done once, not a per-instance tax paid forever.
Absent on purpose: FreeBSD-tests (~200 MB of test suites), all -dbg
packages, all lib32 packages.
Build it:
../travel-and-rrabbit/make.sh builder| stock FreeBSD 15.1 cloud image | T&R 0.1 | |
|---|---|---|
| golden image on disk | 2.51 GiB | 363 MiB |
| packages installed | 504 | 50 |
| per-instance overlay after boot | 2,999 MB | 51 MB |
| first-boot network download | ~512 MiB | none |
Five instances ran concurrently (1 × 2560 MB, 2 × 1024 MB, 2 × 768 MB) on a host with ~5 GB free. growfs correctly expanded the 3 GB image to fill a 9.6 GB overlay with no help from the seed.
The per-instance figure is the one that matters for PARK: at 51 MB and 768 MB of RAM, a dozen tabs is a disk rounding error and a RAM question only.
Done and verified: a machine is a tab. Opening ⬢ MACHINES in PARK's rail lists the fleet; clicking a running instance opens its serial console as a tab in the same strip the web reader uses, with a live root shell inside.
vpsd/api.py (port 8905, stdlib, localhost)| route | |
|---|---|
GET /api/fleet | instances + available golden images |
GET /api/instance/<n> | one instance |
POST /api/instance/<n>/{start,stop,reimage,destroy} | lifecycle |
POST /api/create | new instance |
GET /api/console/<n>/stream | serial output, SSE, base64 frames |
POST /api/console/<n>/input | keystrokes in |
GET /console/<n> | the terminal page, built to be iframed |
GET / | a standalone fleet UI |
Deliberate choices:
The console is a broker, not a connection per viewer. A QEMU
-chardev socket accepts exactly one client, so a second viewer would simply hang. The broker holds the socket and fans out; it connects lazily and lets go when the last viewer leaves, which keeps vps.py console (socat, same socket) working whenever the web console is closed.
Scrollback is primed from QEMU's own logfile, so a viewer sees the boot it
was not present for.
Frames are base64. Serial output is arbitrary bytes: a chunk boundary can
land mid-UTF-8, and a raw newline would terminate the SSE frame.
The terminal is a fixed 80×24, scaled by CSS. A serial line has no in-band
resize, so a terminal that reflowed to the pane would wrap where the guest does not.
Mutations require an `x-parkvps` header, which a cross-origin form cannot
send, so it forces a preflight that the origin allowlist refuses. Binding to localhost is not protection — localhost is reachable from any page the operator happens to be browsing.
world/index.html only: a #tabvps button and #vpslist in the rail, newVpsTab(), a vps branch in tabTitle, and vps carried through tabsSave/tabsRestore. The tab machinery is otherwise untouched — select, close, and the frame kept alive behind the active one all already did the right thing for anything that is a page.
Two things that are not obvious:
A machine tab cannot be restored from its URL. tabsRestore runs every
saved address through proxyForUrl, which rewrites the console's absolute address to /ext/127.0.0.1:8905/... and serves it through the bridge — at which point the page loads, looks right, and its console stream points at the wrong origin. The name is the durable identity; the address is derived from it on the way back in.
The console must not go through `/ext/` at all: that proxy is a GET-only
reader, and a terminal needs to POST keystrokes. Proxying it would produce a terminal that displays perfectly and cannot be typed into.
The rail manages the fleet fully: selecting a machine opens an action strip (console · desktop · start/stop · reimage · destroy) and ＋ NEW MACHINE creates one, with a "give it a screen" checkbox that maps to graphics=std. Selection is keyed by name, not index, or the strip would jump to a different machine whenever the 4-second poll re-sorted the list. destroy and reimage confirm; in-flight actions disable their buttons, because destroy is not a thing to fire twice on a double click. desktop is offered only for a machine that has a display adapter — otherwise it would open a tab whose only content is an apology.
Done: a FreeBSD desktop in the browser, via GET /desktop/<name>. icewm on X.org, keyboard and mouse verified reaching the guest.
Four things had to be true, and the first was not:
`-nodefaults` means no display adapter. Every instance had an open VNC
port with no framebuffer behind it, which accepts a connection and then shows nothing forever — indistinguishable from a broken viewer. Instances now carry a graphics field (none | std | virtio); none is still the default, because a server guest gains nothing from a second console.
`VGA` (bochs std), not virtio-vga. It is what UEFI's framebuffer and
Xorg's scfb driver both understand with no DRM kernel module — the difference between a desktop that comes up on a stock GENERIC kernel and one that needs drm-kmod matched to it.
A WebSocket→TCP proxy, since websockify is not installed. RFC 6455
framing is ~80 lines in api.py (ws_frame/ws_read, both unit-tested) rather than a new system dependency. The browser does not apply CORS to WebSockets, so the Origin check there is the only thing between a hostile page and every guest's framebuffer.
noVNC must come from git, not npm. The npm package @novnc/novnc ships
only lib/, which is the babel-compiled CommonJS build (require/ exports) and cannot be loaded as an ES module in a browser. The ES modules are only in the GitHub tree, under core/. Vendored with its licence and a hash in PROVENANCE.json.
The terminal and the framebuffer make the same call for the same reason: scale to fit, never reflow/resize. A serial line has no in-band resize, and a plain VGA has no guest agent to receive a resize request, so a viewer that claimed to change the guest's geometry would be lying.
../travel-and-rrabbit/tandr-desktop.conf is the graphical flavour — a separate config, not a flag, because the whole point of the server image is that an instance costs 51 MB. It autologins root on ttyv0 only (via its own Pc-autologin gettytab entry, so the serial console keeps a normal login) and starts X from .profile without `exec` — an exec would end the session on failure, getty would autologin and retry, and the only symptom of the resulting boot loop would be a flickering black screen.
It costs 4.92 GB against the server image's 364 MB. Most of that is /usr/local, because xorg-minimal pulls Mesa which on FreeBSD pulls LLVM for software rasterisation; Firefox and the fonts add ~450 MB on top. Not yet optimised.
Fonts. xorg-minimal installs the font libraries — freetype, fontconfig, libXft — and zero font files. fc-list returns 0 on a fresh image. The damage is far worse than missing text:
icewm draws menus, titles and the taskbar as blank strips, icons only;
icewm also stops moving windows. Press, drag, release moved a window by
exactly its decoration offset (+4, +24) and no further, at any drag distance. That reads as "click and drag is broken" and sent me through noVNC, the WebSocket proxy, the pointer device, X input and XTEST before the cause turned out to be a missing font;
fluxbox refuses to start at all: Failed to load default value 'fixed'.
font-misc-misc provides the core fixed a WM falls back to; dejavu is what fontconfig resolves sans-serif to. Both, plus core FontPath entries in xorg.conf.d, plus a firstboot fc-cache. Verified after the fix: a drag of (+400, +190) moved the window exactly (+400, +190).
A USB tablet. VNC speaks absolute pointer positions. With only the default PS/2 mouse, QEMU converts them to relative motion and the guest cursor drifts away from where you actually clicked — press, move and release land in three inconsistent places, so every drag fails even once the WM is healthy. Adding qemu-xhci + usb-tablet made it exact: a pointer driven to a client position mapping to guest (721,485) reported x:720 y:484 via xdotool getmouselocation.
The isolating measurement worth reusing: xev on its own window (which gets an implicit grab on button press) counted 6 MotionNotify events, all with state 0x100, during a drag. That proved X, libinput, the tablet, the proxy and noVNC were all correct and the fault was above them, in the WM.
Once the canvas has focus, noVNC forwards every keystroke to the guest — Escape included, which is what RAVIO uses for "back to the road". So from inside a machine tab no key reaches the page around it. Both the console and the desktop view listen in the capture phase (or the chord is swallowed before they see it) for Ctrl+Alt+Shift+Esc, blur the guest, take focus, and postMessage a release to the parent, which RAVIO validates by origin. There is also a visible ⎋ RELEASE button, because a chord nobody can discover is not an escape hatch.
Not Esc+CapsLock, which was the suggestion: CapsLock is a lock, not a modifier, so "Escape while CapsLock is on" silently steals Escape from vi for as long as the light is on and the browser cannot show you that it is. RELEASE_ON_CAPSLOCK in desktop.html turns it on anyway — one line.
Guest audio is wired to the host through an intel-hda + hda-duplex pair on a detected -audiodev. Two things to know:
It comes out of the host's speakers, not the browser. VNC has no audio
channel and neither does a serial console. A machine's sound arrives the way a machine on your desk would.
QEMU's audio drivers are separate packages. A stock Arch qemu-base
offers only none, spice and wav, and naming a driver that is not installed makes QEMU refuse to start — so the backend is probed from -audiodev help at run time and simply omitted when there is none. Install qemu-audio-pipewire and restart an instance; nothing here needs editing. vps.py doctor reports which backend it found, or the command to get one.
The desktop uses RAVIO's own palette, read out of world/index.html rather than eyeballed: --acc #F2C14E amber, --cool #2DE2E6 cyan, --rim #C8862A bronze, --warn #FF5A3C, on #03040a with #f3ead4 text. Amber is the focused window, the way the lit gauge is the one currently reading.
Everything lives in ../travel-and-rrabbit/overlay-desktop/, named by OVERLAY_DIR in the flavour config so none of it can land in the server image, which has no X to theme.
Wallpaper is banner.jpg from the c-u-l8er.link header — the checkerboard
planet with the roads — fill-cropped from its native 1254×1254 square to the guest's 1280×800 and taken down 14% in brightness so window text reads over it. Set by icewm, with xwallpaper in .xinitrc as a fallback because icewm's image support depends on how it was built and a black desktop looks broken rather than unconfigured.
The panel is vertical, down the left, which icewm cannot do — its taskbar
is horizontal only. So icewm's taskbar is off and tint2 provides the panel while icewm stays the window manager.
Colours are in a THEME, not in `preferences`. A loaded theme's colours
override the preferences file, so setting them there does nothing and reads as "icewm ignored my config". They live in themes/parkbsd/default.theme, and preferences says so rather than duplicating them.
Two gotchas worth keeping:
**panel_size is width height for a horizontal tint2 panel and SWAPPED for
a vertical one.** A 62 px full-height column is 100% 62, not 62 100%. Writing it the intuitive way produces a full-width bar 62 px tall, which looks exactly like the vertical keyword being ignored — and isn't: tint2 read the orientation, swapped the pair, and did what it was told.
Built and working. DISPLAY_MANAGER="sddm" in the flavour config; leaving it empty goes back to autologin-into-startx, and build-image.sh sets up exactly one of the two, never both — they would fight over ttyv0 and the symptom is a screen flickering between a greeter and a shell.
The greeter is a QML theme of our own (themes/parkbsd/Main.qml): the road behind, an amber T&R over the machine's own hostname, cyan-focused USER and KEY fields, and an IGNITION button. Deliberately QtQuick only — no QtQuick.Controls, no SddmComponents. A greeter whose imports fail shows a blank screen with no way in, and the only trace is a log inside a machine you now cannot log into.
Cost: 127 packages, ~966 MiB (qt6-base, qt6-declarative, consolekit2), which is why ROOT_SIZE is 9g and the image is 5.16 GB.
Three things bit, all silent:
`QtVersion=6` in `metadata.desktop` is load-bearing. Without it SDDM 0.21
assumes the Qt5 greeter, looks for /usr/local/bin/sddm-greeter, does not find it (the package ships sddm-greeter-qt6) and falls back to its default theme. It looks exactly like the theme name being wrong in sddm.conf. The one line that says otherwise is in /var/log/sddm.log.
The icewm package ships three xsession entries of its own —
icewm-session.desktop, icewm.desktop, xinitrc.desktop — and a greeter picks among them by sort order. SDDM launched icewm-session directly, so the tint2 panel never started: a desktop that comes up looking almost right, missing only the part that makes it ours. The build now removes them, because an image that ships one desktop should offer one session.
A greeter needs somebody to greet. Root is refused for graphical login and
the nuageinit user has a locked password, so the image carries a real desktop user — driver / driver, set in the flavour config.
lightdm + lightdm-gtk-greeter remains the ~1.7 MiB alternative if the Qt6 weight is ever unwelcome.
The `driver` password is in the config file, in the clear. It is defensible only because these guests are reachable through a unix socket on one machine, and root already has an empty password on a
secureconsole. Change it before anything here listens on a real address.
Firefox is in the image and renders real sites, video included. Lighter swaps that exist here: netsurf (4.6 MB, own engine, weak JS), dillo (1.7 MB, no JS), badwolf/luakit (WebKitGTK). One line in PORT_PKGS.
Unexplained: a freshly booted instance's Firefox opened one of the operator's own sites on first launch. The image carries no Mozilla profile, no distribution/policies.json and no browser.startup.homepage pref — all three checked and absent — so this is recorded as an open question rather than explained away.
An instrument that ruins what it measures. status() probed readiness by
reading sshd's banner and closing, which sshd logs at error level, which FreeBSD's syslog sends to /dev/console. Polling /api/fleet every 3s flooded every guest's console at exactly the poll rate. Measured 690 bytes per 20s before, 0 per 25s after caching readiness per pid.
Replaying a recording into a live terminal is not display-only. xterm.js
answers any query it is fed, and the answer goes to the guest as input. A ESC[6n recorded during boot got answered minutes later, arrived at a shell prompt that never asked, and was executed (-sh: 80R: not found). Backlog is now stripped of CSI…n / CSI…c / OSC colour queries; live output is not, or a guest asking something now would never get a reply.
A broker must notice its guest leaving. The reader thread exited on EOF
but left the socket set, so the next keystroke after a reimage wrote to a dead fd and took the request handler down with no response at all.
Independently, the distro stopped probing the terminal: FreeBSD's /root/.profile runs resizewin -z on serial lines, whose reply arrives after the probe has timed out. Replaced with stty rows 24 cols 80, which is the answer it would have got. Verified: stty size → 24 80.
Instances cannot talk to each other. SLIRP isolates them. QEMU's socket
netdev is the unprivileged fix and is unimplemented.
No PULSE Loop Manifest yet. Per the repo convention every product declares
its loop topology in this spec. PARKVPS has no honest one to declare yet — it is not a loop, it is a substrate — and writing one before it is true would be fabrication.
No snapshot/restore. qcow2 internal snapshots are free and unwired.
The desktop image is 4.48 GB, ~4.1 GB of it Mesa/LLVM pulled in by
xorg-minimal for software rasterisation. A narrower package list (X server + scfb without full Mesa) is very likely to cut this hard, and is untried.
`graphics` cannot be changed on an existing instance. It is fixed at
create time; there is no vps.py set.
`vpsd` has no test suite. Everything above was verified by running it,
which is not the same thing. strip_queries is the only unit-tested piece.
T&R's root has an empty password and `ttyu0` is marked `secure`, so
anyone who can reach the console socket is root without authenticating. That is fine while every instance is reachable only through a unix socket on this machine, and is exactly wrong the moment one is exposed. It is a decision, not an oversight — but it is a decision that has to be revisited before anything here listens on a real address.
Console input is one HTTP POST per keystroke batch. Fine on localhost;
the reason terminal size negotiation could not complete in time. A WebSocket would fix the latency and is not built.
RAVIO's changes are uncommitted on purpose. That tree had ~3,200
uncommitted lines from other work when this landed, and committing would have swept up someone else's in-progress changes.
The name `T&R` is a placeholder. Rename it in tandr.conf and the
image, hostname, MOTD and artifact name all follow.
The golden image is built from `quarterly`, not latest.
First build attempt: No CMAKE_C_COMPILER could be found. The T&R image is pkgbase and minimal, and ships no toolchain at all — deliberately. This is the same constraint ic32 already lives under: it is C, it is compiled on the FreeBSD builder guest, and the image ships a binary.
So the addon follows that pattern rather than fighting it, and the split is:
| built on the builder | shipped in the image |
|---|---|
| clang, cmake, ninja, greenfield source | libproxy-encoding.so + the .node addons |
gstreamer1, gstreamer1-plugins, -plugins-gl, -plugins-x264, mesa-dri, node24 |
Which is better for the download than it sounds: the runtime set is what costs image size, and no user ever needs the 1 GB of LLVM.
Note the pkg repo here is FreeBSD's quarterly branch and carries no FreeBSD-clang (pkgbase sets live in a separate FreeBSD-base repo). The ports compilers available are llvm19…llvm22 and gcc15…gcc17; llvm19-lite is the smallest thing that can do this job.