01What Each Model Is
RT-2 (Google DeepMind, 2023) took a large vision-language model — PaLI-X or PaLM-E — and co-fine-tuned it on robot trajectories, encoding actions as text tokens. Its headline result was emergent semantic generalization: "move the object to the German flag" worked without that specific instruction in training.
OpenVLA (Stanford, 2024) is a 7B open-source VLA built on the Prismatic VLM stack and trained on the Open X-Embodiment dataset. It set the reference baseline because the weights, tokenizer, and finetuning recipe are public — the first VLA that a lab could actually reproduce and adapt.
02Architecture Side by Side
| Dimension | OpenVLA | RT-2 |
|---|
| Parameters | 7B | 55B (PaLI-X) / 12B (PaLM-E variant) |
| Backbone | Prismatic VLM (Llama-2 7B + DINOv2/SigLIP) | PaLI-X or PaLM-E co-fine-tuned |
| Training data | Open X-Embodiment (~970k episodes) | Robot trajectories + web-scale VQA |
| Action tokenization | 256 discrete bins per DoF, appended vocab | String-encoded action tokens |
| License | Open weights, code, recipes | Closed; API-only in published work |
| Reported rate | ~3–6 Hz single GPU | Cloud-TPU inference; not edge-deployable |
| Best at | Reproducible research, community finetunes | Emergent semantic generalization |
03Inference Speed
RT-2's published control loop depends on cloud TPU inference; its 55B backbone is not deployable at edge rates without distillation. OpenVLA at 7B is reproducible on a single high-end GPU, but even there it lands around 3–6 Hz — an order of magnitude below the 30 FPS that contact-rich manipulation demands.
The bottleneck is the same in both: pixel and language tokens run through a large transformer before any action can be emitted. Compact VLAs (SmalVLA, TinyVLA, π0-FAST) attack the latency by shrinking the model. Geometry-primed policies attack it by shrinking the input — 3D state is orders of magnitude smaller than pixel-token streams.
04Sim-to-Real Reliability
Both models tokenize RGB, so both inherit the substrate's weaknesses: lighting, shader, and camera-pose shifts degrade performance in ways that don't show up in training loss. RT-2 leans on web-scale VLM priors to smooth over the gap; OpenVLA leans on the breadth of the Open X-Embodiment dataset. Neither closes it — a policy that saw 40k mugs in RGB still fails on the 40,001st mug under different lighting.
A geometry-primed foundation like MoreSight™ sidesteps the gap entirely: 6DoF pose and velocity are identical in sim and real. The action model consumes state, not pixels, so the lighting change never reaches it.
05Where MoreSight Fits
MoreSight™ inverts the VLA pipeline. GINN — the Geometric Invariance Neural Network — extracts position, orientation, and velocity from depth at 30 FPS. A focused action model then closes the loop on compact state instead of pixel tokens. The result is realtime, precise control with minutes of demonstration data rather than millions of teleop episodes.
The two families aren't rivals so much as complements. A VLA like OpenVLA or RT-2 can name intent — "pick up the red mug" — while a geometry-primed policy supplies the state to act on. Pixel tokens for semantics; geometry for physics.
06Which to Reach For
| If you need | Pick |
|---|
| Reproduce results or finetune on your own robot | OpenVLA — you can actually run it. |
| Push the ceiling on open-vocabulary reasoning | RT-2-class model — if you have TPU access. |
| 30 FPS closed-loop control on real hardware | Neither. Reach for a geometry-primed policy. |
| Bind language intent to precise manipulation | VLA for intent + geometric state for the loop. |
07Further Reading
For the broader landscape, read Vision-Language-Action Models and Robotics Foundation Models. For the geometry-first case in depth, see Spatial AI for Robotics, or return to the MoreAI overview.
