Why don't pixel-based RFMs run well in realtime?

Because they tokenize every pixel of every frame. Attention cost scales quadratically, sensor noise leaks into the policy, and control loops cannot close at 30 FPS on edge hardware. The compute bill grows with resolution, not with task complexity.

How does geometric priming change the picture?

Geometric priming trains the foundation on 3D data with depth so the model learns geometry and physics as invariants — surfaces, edges, poses, velocities. Downstream focused action models consume that compact spatial state instead of raw frames, staying fast at 30 FPS and transferring cleanly from simulation to hardware.

←Back

Guide · Foundations

Robotics
Foundation Models.

Q: What is a robotics foundation model?

A robotics foundation model is a large, pretrained model — usually a vision-language-action (VLA) model — that maps raw sensory input to robot actions across many tasks. It borrows the LLM playbook: pretrain once on massive multi-embodiment data, then adapt with lightweight fine-tuning or prompting.

The field is racing to build a foundation model for robots the way GPT was a foundation for language. Most attempts tokenize every pixel. This guide explains what an RFM actually is, where pixel-first designs strain, and why a spatial foundation — primed on geometry — is the more efficient path.

01What Is a Robotics Foundation Model?

A robotics foundation model (RFM) is a large, pretrained model that generalizes across tasks, embodiments, and environments. Most of today's RFMs are vision-language-action (VLA) models: they ingest images plus a natural-language instruction and emit robot actions. Google's RT-2, Open X-Embodiment's π-family, and industrial efforts from Covariant, Physical Intelligence, and Figure all live in this shape.

The promise is the LLM playbook applied to control: pretrain once at scale, then adapt with a small amount of task data. The bet is that a shared prior across billions of frames will unlock the same emergent generalization we saw in language.

02The Pixel Assumption

Nearly every mainstream RFM treats the physical world the way an LLM treats text — as an undifferentiated stream of tokens. Every pixel of every frame is tokenized, attended to, and paid for. That works well for description and low-rate reasoning, but it makes realtime state expensive: attention is quadratic in tokens, and a 224×224 camera at 30 FPS is a firehose.

The result is a foundation that generalizes broadly at low frequency, but struggles to close a control loop at 30 FPS on the edge — the regime real robots operate in.

03Pixel-First vs. Spatial-First

Dimension	Pixel-First RFM	Spatial Foundation
Pretraining substrate	Web-scale video + teleop pixels	3D data with depth (geometric priming)
Native representation	Pixel + language tokens	Position, orientation, velocity in 3D
Compute per frame	Scales with resolution	Compact state, edge-friendly
Sim-to-real behavior	Sensitive to lighting, shaders, texture	State is identical in sim and real
Data efficiency	Needs millions of demonstrations	Minutes-to-hours of task data
Best fit	Open-vocabulary description, low-rate policies	Realtime manipulation, closed-loop control

04Geometric Priming as a Better Prior

Language is a symbolic system whose structure has to be learned from data. Physical geometry isn't — it's already there, in the depth channel. A spatial foundation is primed on 3D data so it learns the rules of geometry and physics once, at the model level, and outputs invariants: surfaces, edges, poses, velocities.

MoreSight is that kind of foundation. GINN — the geometric neural network at its core — extracts spatial state directly from depth-rich 3D data, and downstream focused action models consume the state, not the pixels. The pretraining bill sits with the foundation; the deployed policy stays small.

05Where Each Foundation Wins

Reach for a pixel-first RFM when — you need open-vocabulary description, semantic reasoning, or coarse policies that tolerate a few Hz of control.
Reach for a spatial foundation when — a controller has to close its loop at 30 FPS: manipulation, mobile navigation, tele-op, precise assembly.
Combine them when — semantics need to attach to 3D state — 'pick the red mug 40 cm away, tilted 15°.' Let the VLA name; let the spatial model act.

06What to Watch Next

The RFM conversation is shifting from "bigger VLA" to "better prior." Expect a hybrid stack to win: a geometry-native foundation that produces compact spatial state, a compact language model that binds intent, and focused action models specialized per embodiment. The parts that scale are the ones that stay separable.

07Frequently Asked

Is an RFM the same as a VLA?

Most current RFMs are VLAs, but the terms aren't identical. RFM describes the role — a pretrained foundation for robotics — while VLA describes an architecture that fuses vision, language, and action tokens end-to-end.

Do RFMs replace focused action models?

No. A foundation supplies the shared prior; focused action models supply the last-mile skill. The foundation shouldn't ship on the robot — the compact, specialized policy should.

Why is depth such a big deal?

Depth turns perception from a 2D inference problem into a 3D measurement problem. It collapses the sim-to-real gap and gives downstream controllers a state they can act on without re-deriving geometry every frame.

08Further Reading

For the broader picture, read Spatial Intelligence vs. Computer Vision and Spatial AI for Robotics, or return to the MORE, Inc. overview.

←Back to MORE AI

RoboticsFoundation Models.