Alex Coulombe Presents / field notes / 2026-10-07 to 2026-10-13
From a photo
to a little world.
A garden that answers a wish. A memory rebuilt as toys. Four Unreal prototypes, a fleet of AI collaborators, and the difference between a successful log line and something you can actually feel.
01 / The brief meets the prototype
“We don't even see any interactivity.”
Alex, reviewing the first teasers, 2026-10-07
The Surreal Minds work for the SensAI brief explored small spatial experiences for PICO: import a memory, tend an impossible garden, grow a tabletop world, or care for a little creature. The brief asked us to distinguish AI used to build an app from AI that responds inside the app.
That distinction mattered. Memory Portal began with authored destinations. Living Tabletop and Creature Home had deterministic play and care rules. Attractive footage did not establish photo understanding, a conversational pet, or generated game rules.
“none of the teasers show enough of the actual experience. We don't even see any interactivity”
All three first teasers were rejected. The next useful artifact was a real action-and-result recording. In the PICO emulator, scripted controller input made Miso react, grew plants, and advanced a recipe. It also exposed a Tabletop trigger that never registered. Finding that before a wearer test was progress too.
PICO emulator window · scripted input · no runtime AI
The first recorded interaction loop
02 / Input → AI → visible result
Give the garden a sentence.
On 2026-10-08, Memory Portal could read an actual image file through a studio-PC vision service. Three synthetic fixtures became different destinations, palettes, light and captions. Cancel, undo and reset preserved control over the result.
Conservatory gained a typed-wish path: a companion page sends a sentence, Gemma 3 4B returns a constrained recipe, and Unreal validates it again before changing the garden. “Add a dragon” is refused. The model can choose supported care actions; it cannot invent arbitrary executable behavior.
Follow the purple wish
- 1. Input: “paint the leaves purple”
- 2. LAN AI: a validated
#7A3FA0tint recipe, 783 ms in this take - 3. Result: the emulator's native leaf render changes color
The video also shows rain, a dragon refusal, sun and bloom, and an autumn-color request. It preserves the original prototype staging so the change is easy to inspect.
PICO emulator native renders · LAN Gemma 3 4B
Five wishes through the real model and garden
03 / The headset changes the brief
The wait became part of the world.
By the next Quest session, both photo captioning and voice wishes could run on the headset. The first Memory in-app read failed at 75 seconds, despite a roughly nine-second standalone command-line test. Sharing CPU time with Unreal changed the problem.
“we should have SOMETHING that happens while you wait. also there's blocks in the middle of my hands”
V13 changed inference thread pools and stopped busy polling. A later Quest read completed correctly in 20.0 seconds; another v14b run took 26.4. Real progress stages and moving motes made the wait legible. Controller stand-in bars were hidden when a hand was tracked.
Voice exposed another gap: the Quest heard “Make all the leaves pink,” produced the correct recipe, and logged its application. Alex could not tell that it had listened, and saw no leaf change. V10 now has larger listening, transcript and done states and a stronger color response, verified on desktop. Headset acceptance is still open.
Quest 3 headset · on-device caption + LAN PC depth
The photo-depth experiment Alex redirected
04 / A charming interpretation
Keep the memory. Change the material.
“This should be a charming interpretation, not a simple tech demo of photo to 3D conversion.”
Alex, after wearing the v14 photo-depth build, 2026-10-12
V15 replaces the photo surface with a kit of 15 primitive kinds. Cloud vision returns object boxes, relative depth, scale and color; Unreal builds palms, cabins, pines, clouds and boats with staggered bounce, twist and assembly cues. The geometry is authored and bounded. The model interprets the composition.
Six real photo requests, including a mirrored beach, passed the object contract. Mirroring moved the palms and sun to opposite sides, a useful check that layout came from the image. Two native desktop sequences exercised import, assembly, undo, a second photo, cancellation and reset. These desktop captures preceded the October 13 Quest wearer test. Alex confirmed that v15 worked, then requested less starter scenery and fewer hallucinated objects.
05 / Same image, three routes
Where should the thinking happen?
2026-10-13 / Memory Portal v16
One photo. Switch the model. Keep the comparison.
“ok it's working. it's pretty good... but there's some weird hallucinations or it's not removing enough of the starter content. also can't I test cloud or local AI too ?”
Alex, wearing v15 on Quest 3
V16 clears the authored starter props when the toy stage appears; undo brings them back. It limits the scene to eight objects and requires caption support, independent detector evidence and overlap checks. A convincing sentence is no longer enough to place an object.
The controls now switch between cloud, studio PC and local captioning. Compare rebuilds the photo already in the frame through the next route. Each result keeps its backend and measured caption, grounding and assembly stages visible.
Cloud. Gemini proposes the composition through a PC bridge. Independent detectors validate its boxes before Unreal builds the toys.
LAN PC. Gemma 3 4B describes the photo on an RTX 4090 PC. OWLv2 and a selective Grounding DINO fallback find objects on an RTX 5090 PC. The photo stays on the studio network.
On-device path. SmolVLM supplies a caption. Simple rules place primitives, explicitly labeled “caption-only / approximate.” Desktop footage of this code path does not establish headset latency.
Unreal desktop native samples / v16 / identical photo
The same beach through three app backends
This take includes a cold detector start: cloud took 73.4 seconds through assembly, of which 57.6 seconds loaded local detector libraries/models and 11.0 seconds ran the cloud vision request. LAN then used warm detectors and took 12.2 seconds. The local caption path took 6.7 seconds on the desktop. These are measured stages in this run, not a fair warm-start race or Quest timing claims.
The conservative filter has a cost: it can omit real detail. In the five-fixture evaluation, cloud garden flowers failed independent box matching; the LAN garden kept larger detected flower groups. An autumn caption incorrectly named a moon, but the detector check prevented a moon from appearing. These are synthetic test images, not a general accuracy benchmark.
Earlier backend study, October 9–12
Below, the same project-generated seaside image goes through each backend. These are visualizations of recorded outputs, with the resulting sky, water and accent palettes underneath. They compare what each route returned. They are not a matched three-backend app-render or headset-video test.
On-device
Quest 3 CLI · SmolVLM-500M
9.095 s CLI wall time
“This image is a digital art piece.”
A generic caption. Deterministic pixel rules completed the scene: water, dusk, Lantern Tide. Swatches are the resulting scene palette, not model-predicted object colors.
LAN PC
Host request · Gemma 3 4B on Archie
7.408 s client total
“Sunset on a tropical beach”
The model returned a structured Lantern Tide / dusk scene, a title, palette and motion. 6.875 s was model inference; this recheck ran while the PCs served other work.
Cloud
Host result visualization · Gemini Flash
11.890 s bridge total
“Sunset over the beach with stylized palms”
The reply added object kinds, boxes, scale and approximate relative depth. The boxes above visualize actual detections. These support toy assembly; they are not segmentation or metric depth.
Timing scopes differ: the on-device card is the October 9 Quest CLI benchmark, not the later app. LAN is a client total; cloud is a bridge total. The CLI's raw caption trailed off; its displayed sentence and palette are the validated downstream result. The cloud study recorded the resolved model as Gemini 3.8 Flash, behind a rolling alias.
Earlier measured interaction costs
| Backend | Observed latency | What it means |
|---|---|---|
| On the headset | Memory caption: 20.0–26.4 s in the Quest app. Conservatory voice: 10.9–12.4 s after recording (speech 1.4–2.8 s + model about 9.5 s). | SmolVLM-500M caption; Whisper tiny.en + Qwen3-0.6B wishes. Offline with bundled models. No learned on-device depth. Correct wish recipes did not establish a visible leaf-color change. |
| Studio PC over LAN | Same three Memory fixtures: 3.9–7.4 s captions and 0.85–1.07 s depth; Conservatory wishes: 0.53–0.78 s typed, 1.07 s warm voice | Gemma 3 4B on Archie through the Fort bridge; Depth Anything V2 Small on Fort CPU. No internet needed after caching, but both PCs and the network must be available. Device tests used USB port forwarding; untethered Wi-Fi is unproven here. |
| Remote cloud API | Same three images: 11.9–20.0 s; matching purple wish: 2.68 s (typed text, no speech recognition). | Server-held Gemini 3.8 Flash produced the strongest grounded toy-object layouts. Its relative object depth is not a dense depth map, and the key never enters either app. |
The LAN warm-voice measurement is a synthetic “rain and flowers” clip: 0.224 s speech recognition + 0.842 s model. It is a different input from the on-device pink/blue wishes and the typed cloud purple request. Recording time is additional. These are observed runs, not service guarantees.
On-device keeps the model and photo local. LAN needs the studio machines; cloud sends the image through the bridge to the provider and requires internet. LAN and local have no per-call provider fee; cloud is metered. Electricity and hardware cost were not measured. No provider key is shipped in the app.
For the next demo, LAN is the practical choice for responsive wishes and captions. Cloud earned a place by returning richer object layouts. On-device remains meaningful for privacy and offline operation, with a much more visible wait. The fast LAN depth model does not by itself make a charming scene.
06 / Tested without a headset
Save the wearer for the questions only a wearer can answer.
Claude and Codex worked across parallel app, inference, capture and review lanes. Shared evidence and the fleet bus connected those lanes; named source branches and a host build lease kept them from competing for the same build. The useful measure is what they verified before asking Alex to put a headset on.
3 / 3 photos
LAN scene contracts passed for beach, snowy pines and garden. Native emulator sequences exercised photo import, cancel, undo and reset.
5 wishes
The emulator garden handled rain, purple leaves, a dragon refusal, sun/bloom and autumn tint. Unsupported actions preserved the garden.
8 gesture routes
Conservatory desktop tests dispatched all eight gestures into care actions. This verified the mapping, not real hand recognition.
Forage: find what we already own ↗
Our asset scout searches the owned library by intent. The diary workflow checks it before acquiring new props. This iteration ultimately used a consistent procedural toy kit; we are not claiming these palms came from a Forage import.
UOXR + xrsim: exercise the shared foundation ↗
Memory Portal derives from the Ultimate OpenXR Template; the other apps reuse its hand bridge on the vr58 baseline. In separate upstream xrsim QA, a native menu-click fixture passed 18 checks and tracking visuals passed 12. That protects shared input behavior, but is not four-app physical acceptance.
PICO emulator + Windows Graphics Capture
Install the real Android package, drive an app action, record the emulator window. Foreground-checked SendInput reached Creature's controller actions. Native Unreal Shot captures supplied app pixels when the emulator compositor put a black panel over Memory.
adb, logcat and native Shots
Package, signature, cooked-content, startup, focus and result checks caught failures before a wearer. Logcat and pulled Unreal logs locate the event; native Shots verify its pixels. Not every app sends its own log to logcat, and application pixels do not prove final compositor output.
Fleet bus: explicit handoffs
Claude and Codex shared concrete artifact paths, test results and ownership. A queued message did not count as accepted work. Parallel development helped cover apps and tooling while shared host builds remained serialized.
Video QA Workbench: inspect the evidence
The first six teaser exports were decoded and analyzed across all 5,400 frames. Freeze, motion and sharpness advisories gave review points; no black spans were detected. Alex still rejected their missing interactivity. Automated video checks support creative judgment; they do not supply it.
Progress in the other little worlds
Give every action a readable payoff.
Living Tabletop and Creature Home remain authored gameplay, with no runtime AI in these captures. Better framing, lighting, contact shadows, response animation and controls make the interaction easier to understand.
07 / Lessons along the way
The failures that changed the next test.
An input API can exist and still be unusable.
PICO emulator 0.13 gRPC hand input and screenshot requests crashed the emulator. PostMessage keys had no effect. Foreground-checked SendInput was the working Creature route; later keyboard regressions made one-action smoke checks essential before another take.
An OpenXR instance is not an XR session.
In the adjacent Joby360 diagnostic, a normal launch reached the runtime but created no swapchains. The -vr launch argument advanced it; bStartInVR was the Library-launch fix. The emulator predicted the startup failure before the physical test.
Frame synthesis can activate while capture goes blank.
Related PICO emulator tests reported AppSW on while captures stayed flat. Switching frame synthesis off restored pixels. Memory's black panel persisted with synthesis off, so it was a different unresolved capture defect. Native Shots proved app rendering, not headset composition.
A CLI benchmark is not an app budget.
The tiny vision model managed about nine seconds on Quest's command line, then failed at 75 seconds inside Unreal. Thread-pool changes brought an app run to 20 seconds. Measure while rendering and interacting; do not advertise the isolated benchmark as the experience.
Applied is not visible.
Conservatory logged the correct tint, but Alex saw no leaf change. The repair must pass a same-pose native before/after and a wearer check. Likewise, a valid caption was not enough when Alex expected geometry that evoked the photo.
Pretty source assets are not enough.
Memory and Creature instanced props fell back to the default material because an instancing usage flag was missing. Native captures exposed it. Cooked-content checks then caught a new Conservatory material absent from an otherwise successful package.