HOYALABA PERSONAL RESEARCH NOTEBOOK

Paper Readings / 2026-09-21 / 9 MIN

Reading GR00T N1: connecting the islands of robot data

A reading of GR00T N1, from its two-system architecture to the data pipeline that connects different robot embodiments.

UPDATED / 2026-10-20

Paper Figures 1 and 3 are reproduced from the cited paper for commentary. Copyright remains with the original authors or rights holders.

Today I am reading GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. I came to this after reading Diffusion Policy, so my first question was simple: what changes when an action-generation policy becomes part of a robot foundation model?

This is an English adaptation of my Korean reading notes. The scope is the N1 paper: its introduction, architecture, training data, evaluation, and limitations. The GR00T project continues beyond N1; I return briefly to the later versions at the end.

The Korean post also links an accompanying GR00T video.

1. The problem: data islands

A robot's body is part of its learning problem. Different robots have different joints, action dimensions, sensors, and camera placements. A demonstration collected on one platform cannot simply be treated as a demonstration on another.

The paper calls attention to these separated pools of experience. Robot data is already expensive; splitting it by embodiment makes each pool smaller still. I found this a useful starting point because it changes what we should look for in the model. The question is also how to make more experience usable together.

2. Two systems, one policy

GR00T N1 separates vision-language understanding, called System 2, from action generation, called System 1. System 2 processes the image and instruction. System 1 uses that representation to generate robot actions. The modules are coupled and trained together.

The names borrow Daniel Kahneman's distinction in Thinking, Fast and Slow: System 1 is fast, automatic, and intuitive; System 2 is slower and more deliberate. Understanding “this is a cup, and the instruction says to move it right” is the vision-language module's role. Producing the joint commands to grasp it is the action module's role. This is an architectural analogy, not a claim that the network reproduces human cognition.

System 2: a representation for control

The paper uses NVIDIA Eagle-2 as its vision-language backbone. Of GR00T-N1-2B's 2.2 billion parameters, 1.34 billion belong to the VLM. Images are encoded at 224×224224\times224 resolution and reduced through pixel shuffle to 64 image tokens per frame.

One choice caught my attention: it uses the 12th language-model layer, rather than the final layer. The paper reports that intermediate features gave both faster inference and higher downstream policy success.

I still find that choice less intuitive than it first sounds. A feature useful for producing text need not be the feature that best supports a downstream motor policy. The practical question is the policy's behavior and computation time, not which layer feels most complete.

System 1: generating an action chunk

The action module uses a diffusion transformer, VθV_\theta, with alternating cross-attention and self-attention blocks. The paper relates its structure to Flamingo and VIMA. It predicts an action chunk:

At=[at,at+1,…,at+H−1],H=16.A_t=[a_t,a_{t+1},\dots,a_{t+H-1}],\qquad H=16.

So the connection to Diffusion Policy is easy to see: the output is a short trajectory rather than a single isolated command. The training objective is flow matching. The paper writes Equation (1) as:

Lfm(θ)=Eτ[∥Vθ(ϕt,Atτ,qt)−(ϵ−At)∥2],Atτ=τAt+(1−τ)ϵ.\mathcal{L}_{fm}(\theta)=\mathbb{E}_\tau\left[\left\|V_\theta(\phi_t,A_t^\tau,q_t)-(\epsilon-A_t)\right\|^2\right], \qquad A_t^\tau=\tau A_t+(1-\tau)\epsilon.

Here ϕt\phi_t is the vision-language representation and qtq_t is the robot's proprioceptive state. At τ=0\tau=0, the interpolated action is pure noise; at τ=1\tau=1, it is the clean action chunk. The network learns a vector field along the line joining noise and actions. There is a sign detail to watch: with this interpolation, the direction from noise to actions is At−ϵA_t-\epsilon, whereas the printed training target is ϵ−At\epsilon-A_t. The sampling update must use a consistent sign convention; these two vectors point in opposite directions.

The paper reports that four inference steps worked well across embodiments, and sampling a 16-action chunk took 63.9 ms on an L40 GPU using bf16.

Why the two systems run at different rates

The paper describes System 2 at 10 Hz and motor action generation at 120 Hz. A robot does not necessarily need to reinterpret the full instruction every time it issues another motor command. The scene and instruction often change more slowly than the joint commands.

This reminded me of Diffusion Policy's observation and action-execution horizons: keep the current scene representation while executing a short sequence, then observe and replan. The rates refer to different parts of the pipeline. The 63.9 ms above measures generation of an entire chunk; it does not mean a fresh full chunk is inferred every 1/1201/120 second.

Paper Figure 3 (NVIDIA, 2025): the GR00T N1 architecture. Eagle-2, with its language component frozen, encodes the image and instruction; embodiment-specific state and action encoders feed DiT blocks that cross-attend to those features; an action decoder produces the action chunk.
PAPER FIG. 3 (NVIDIA, 2025) · the GR00T N1 architecture. Eagle-2, with its language component frozen, encodes the image and instruction; embodiment-specific state and action encoders feed DiT blocks that cross-attend to those features; an action decoder produces the action chunk.

3. A pyramid of training data

This was the most interesting part of the paper for me. Its training material includes real robot demonstrations, human video, and synthetic trajectories. Each source contributes something different, and each comes with a different gap to bridge.

Paper Figure 1 (NVIDIA, 2025): the data pyramid for robot foundation model training. Web data and human videos at the bottom, synthetic data in the middle, real-world robot data at the top. Quantity decreases and embodiment-specificity increases toward the top.
PAPER FIG. 1 (NVIDIA, 2025) · the data pyramid for robot foundation model training. Web data and human videos at the bottom, synthetic data in the middle, real-world robot data at the top. Quantity decreases and embodiment-specificity increases toward the top.

Human video and latent actions

Human video from datasets such as Ego4D and EPIC-KITCHENS does not contain the robot's joint commands. The paper trains a VQ-VAE whose encoder takes two frames, xtx_t and xt+Hx_{t+H}, and compresses the change between them into a latent action ztz_t. The decoder uses that latent and the first frame to reconstruct the later frame.

After training, the encoder acts as an inverse dynamics model: it infers what happened between the frames. The policy uses the continuous embedding before quantization as a latent-action label, with the same flow-matching objective but a separate embodiment identity. The representation provides supervision from visual change; it should not be confused with measured robot controls.

That distinction matters. Calling both things “actions” makes the pipeline sound simpler than it is. One is an observed command in a particular body; the other is a learned description of what happened between images.

Neural trajectories: from 88 hours to 827 hours

The paper fine-tunes video-generation models on 88 hours of real teleoperation data. Given an existing starting frame and a new language prompt, those models generate 827 hours of video, roughly ten times the original duration.

A multimodal LLM judges eight sampled frames from each video to check whether it follows the instruction. Filtering and re-captioning help manage quality. Generating this dataset required approximately 1.5 days on 3,600 L40 GPUs. That scale made me stop for a moment.

The generated videos still need action labels. Besides latent actions, the pipeline can use a separate inverse dynamics model trained on real robot data to predict robot-action labels. A plausible-looking video is not, by itself, a physically valid control trajectory.

Simulation trajectories: nine months in eleven hours

The other synthetic source is DexMimicGen. It splits a small set of human demonstrations into object-centric subtasks, transforms each segment to a new object arrangement while preserving the relative end-effector pose, and replays it in simulation. Only successful demonstrations are retained.

Across pre-training and post-training data, the paper reports 780,000 simulated trajectories, equivalent to about nine continuous months of human demonstration, generated in 11 hours. The pre-training subset contains 540,000 demonstrations: 10,000 for each of 54 source/target receptacle combinations. These are related totals with different scopes, not two counts of the same dataset.

Sharing a model across bodies

Small, embodiment-specific MLPs convert states and actions of different dimensions into a shared embedding size; an embodiment-specific decoder maps the output back to the appropriate action space. This lets the large model share learned structure while smaller components account for differences between robot bodies.

Training in stages

The paper distinguishes three settings:

  1. Pre-training: mix real robot data, human video, and synthetic data across embodiments using the flow-matching objective.
  2. Post-training: fine-tune for one target robot. The language component of the vision-language backbone stays frozen; the rest of the model can adapt.
  3. Post-training with neural trajectories: mix real and generated trajectories at a 1
    sampling ratio
    to supplement scarce target-task data.

In my notes, I think of this as learning broadly, adapting to a body, and then testing how much generated experience can help. Pre-training used up to 1,024 H100 GPUs and approximately 50,000 H100 GPU-hours.

4. What the evaluation tells me

The simulation benchmarks are RoboCasa Kitchen (24 tasks), the DexMimicGen Cross-Embodiment Suite (9 tasks), and GR-1 Tabletop (24 tasks). On the real Fourier GR-1 humanoid, the paper evaluates 13 tasks covering pick-and-place, articulated objects, industrial objects, and bimanual coordination. The baselines are BC-Transformer and the Diffusion Policy we read across the previous two posts.

For the simulation comparison with 100 demonstrations per task, Table 2 reports:

Benchmark GR00T-N1-2B Diffusion Policy
RoboCasa 32.1% 25.6%
DexMimicGen 66.5% 56.1%
GR-1 Tabletop 50.0% 32.7%

For the real robot, Table 3 reports:

Target-task demonstration data GR00T-N1-2B Diffusion Policy
Full data 76.8% 46.4%
10% of data 42.6% 10.2%

The last two rows impressed me most. With only 10% of the target-task data, GR00T-N1-2B is just 3.8 percentage points below Diffusion Policy trained on all of it. This is a comparison of target-task data efficiency: GR00T also benefits from broad pre-training, so the models have not seen equal total training data.

The neural-trajectory ablation also reports gains. On RoboCasa, adding generated trajectories improves average success by 4.2, 8.8, and 6.8 percentage points with 30, 100, and 300 demonstrations per task, respectively. Across eight real-robot tasks, the average gain is 5.8 percentage points. Generated data helps in these settings, although these numbers do not show a monotonic “less data means a larger gain” relationship.

I read each result together with its task set, training data, and evaluation conditions. A benchmark result is not a guarantee on an arbitrary robot. The numerical tables and settings are in Section 4 of the paper.

5. Limitations

N1 focuses on relatively short tabletop manipulation. Walking while manipulating objects, extended tasks, and broader whole-body behavior require more work. Synthetic experience also inherits limitations from its generator or simulator.

I appreciated that the paper leaves these boundaries visible. “Generalist” is a direction for the model, not evidence that every kind of robot behavior has already been solved.

My takeaway

Reading this after Diffusion Policy helped me see the continuity. Generating a sequence of actions remains central. What expands is the conditioning information, the range of bodies, and the effort spent bringing different kinds of experience into the training process.

That is the question I want to carry into the next reading: when a robot foundation model improves, how much comes from the policy architecture, and how much comes from the experience it can now learn from?

After N1

The Korean notes also point ahead to GR00T N1.5, its Eagle-2.5 backbone, and the GR00T-Dreams data-generation pipeline. The N1.5 project page is the follow-up reference. Later versions, including the N1.6 update mentioned in the Korean post and the N2/DreamZero direction, deserve their own reading rather than being folded into N1's experiments.