Lately I've been working on picking up scissors and knives with the SO-ARM101. With a little data collected, I wanted to fine-tune SmolVLA, try it on the robot, and compare inference time on my RTX 5070 Ti and Mac mini M6.
This is a light development log: a small training run, a check that the whole pipeline works, and some inference timings. The Mac's model name is retained as reported in my original notes; the measurements below are for PyTorch's MPS backend. MPS runs on the Mac's GPU, as described in the PyTorch 2.11 documentation.
Why SmolVLA?
My criteria for a robot policy were simple: Can I run it locally, and can I fine-tune it on my own data?
At roughly 450 million parameters, SmolVLA is on the smaller side of the VLA family and can run on consumer GPUs and Macs. It is integrated into LeRobot, so I could start fine-tuning from lerobot/smolvla_base. The published results also looked promising to me. Among the models I could realistically run, it felt like a good fit.
With a much larger model, getting inference to run on the Mac would itself become a project, before even thinking about training. I wanted the Mac option because mounting my heavy desktop underneath the system I'm building didn't seem practical.
A small training run
The point of this run was to check that data collection → training → inference worked from end to end. This was a proof of concept, so I used just 50 demonstrations.
| Item | Training setup |
|---|---|
| Data | 50 teleoperated episodes, 42,561 frames total |
| Recording rate | 30 fps |
| Episode length | Roughly 20–50 seconds |
| Cameras | top covering the workbench, and wrist on the arm |
| Image resolution | 640 × 480 for each camera |
| Task instruction | Grasp the scissors |
| Starting checkpoint | lerobot/smolvla_base |
| Batch size | 48 |
| Training steps | 13,000 |
| Training device | RTX 5070 Ti, 16 GB |
| Total training time | 3 hours 45 minutes |
| Time per step | About 1.04 seconds |
| GPU memory used | About 11.4 GB |
For comparison, batch size 32 took about 0.7 seconds per step and used 8.1 GB of GPU memory. I ran the training during the day: leaving it overnight sounded convenient until I considered trying to sleep next to the fans. Haha.
Of the roughly 450 million parameters reported for this setup, about 100 million were trainable. I froze the vision-language part and trained the action expert.
Loss started at 0.397 and had largely settled around 0.039 by about 10,000 steps. With this dataset, more training didn't seem likely to buy much in terms of loss.
On the real robot, the result was a mixture of successes and failures. With just 50 episodes, I took that as evidence that the pipeline was connected, rather than a finished policy.
So how long does inference take?
I ran the same trained model on the RTX 5070 Ti and the Mac mini. Here are the measurement conditions before looking at the numbers.
| Item | Benchmark setup |
|---|---|
| Checkpoint | The 13,000-step checkpoint from the training run above |
| Precision | float32 |
| Denoising steps | 10 |
| Output per inference | One chunk of 50 actions |
| Chunk duration | About 1.67 seconds at 30 fps |
| Input | Two 640 × 480 images and six joint values |
| Input source | Synthetic inputs, with no robot connected for the timing test |
| Warm-up | 3 runs |
| Measured runs | 20 runs after warm-up |
| Timing scope | Preprocessing through postprocessing |
| Software | The same LeRobot commit on both machines; PyTorch 2.11 |
| Device | Standard inference, median (p95) | RTC enabled, median (p95) | Reported memory |
|---|---|---|---|
| RTX 5070 Ti (CUDA) | 187 ms (192 ms) | 259 ms (261 ms) | 1.3 / 1.8 GB |
| Mac mini M6 (MPS, as reported) | 404 ms (430 ms) | 443 ms (450 ms) | 1.7 / 2.5 GB |
The memory pairs are preserved as recorded in the original table; the notes did not specify what each member of the pair measures. They should not be read as an independently verified comparison of identical memory counters.
These timings cover generating an action chunk under the conditions above. They are not a measurement of per-action control-loop latency or a task success rate. The original log also does not include the exact LeRobot commit hash, the RTC settings, or the timing code, so this is a record of my run rather than a fully reproducible benchmark.
Trying it on the robot
Watch the SO-ARM101 proof-of-concept inference video.
The video is from an actual inference run on the RTX 5070 Ti. In that run, I saw the robot distinguish between the knife and scissors, pick the target up, and successfully return it to its original position. That was encouraging! It is an example from the experiment, alongside the failures mentioned earlier, rather than a measured success rate over repeated trials.
Next, I want to collect a more substantial dataset, change the end effector to a parallel gripper, and make dedicated storage for the knives and scissors.
For the broader hardware questions behind this run, see Can my hardware run my robot policy?. The camera-mount post covers the wrist-camera hardware.
I'll be back with the next round. Keep going, everyone!
