RoboCasa#

RoboCasa is the kitchen-scale, long-horizon manipulation environment. In RPent it is driven by the RLDX-1 VLA policy, served over HTTP RPC by default (matching LIBERO); a pickle-framed socket transport is also supported. See robots/robocasa/vla_server.py and robots/robocasa/robot_spec.py for the wire/transport selection.

Note

The public Target50 protocol is frozen in robots/robocasa/eval/target50.json. RPent uses ordinary single-task rpent --robot robocasa commands for its 340 cells.

Runtime flow#

RoboCasa365 uses a PandaOmron mobile manipulator and the frozen RLDX-1 policy. The integration is planner-agnostic: API planners, Claude Code and Codex use the same RoboCasa toolkit. See Agentic Planner for credentials and backend configuration; users supply credentials outside the repository.

rpent CLI -> task-memory sync -> environment and VLA servers
          -> planner toolkit -> final environment state.success

Unless external endpoints are supplied, RPent starts an environment server and a VLA server for each run. The planner selects primitives using the live task language and observations; RLDX-1 executes manipulation skills. Only the environment’s own _check_success() result, surfaced as state.success, determines evaluation success. The public protocol uses ordinary single-cell commands, not an included batch launcher. See Harness VLA for the Harness VLA overview.

Installation#

RLDX-1 requires Python 3.10. Create a dedicated environment and install the complete RoboCasa365 stack with .[robocasa]:

uv venv --python 3.10
source .venv/bin/activate

First install a matching CUDA-enabled PyTorch and torchvision pair using the PyTorch installation selector for your GPU, driver and Python version. Run the selected command in this environment (use uv pip in place of pip). The RLDX dependency requires Torch >= 2.7 and torchvision >= 0.22; choose a mutually compatible pair, not two independent versions. Then install RPent:

uv pip install -e ".[robocasa]" \
   --constraint robots/robocasa/eval/target50-constraints.txt
uv pip check

The RoboCasa-specific constraints file pins the compatibility-sensitive package versions validated for Target50 reproduction without narrowing RPent’s shared LIBERO or RoboTwin dependencies. The robocasa extra tracks the maintained rpent branches of RoboCasa, RLDX and Robosuite for both ordinary runs and Target50. The manifest records these branches, not frozen source commits. RoboCasa’s rpent branch declares the distribution name rpent-robocasa365; do not co-install the rlinf-robocasa365 distribution, which provides the same import package. No second source-install step is required. Branches can advance, so record the resolved Git commits and installed versions with each evaluation:

uv pip freeze > installed-requirements.txt

Keep this environment record with the experiment artifacts. Installing the same branch later is not a guarantee of identical source code. The checkpoint, backbone support resources and task-memory snapshots remain fixed below. The constraints do not pin Torch, torchvision or a CUDA backend. Installation retains a compatible installed pair; dependency conflicts must be resolved before running. The manifest’s reference_accelerator records the previously used Torch 2.7.0 / torchvision 0.22.0 / CUDA 12.6 combination as provenance only, not an installation requirement. Record your actual versions with your results and run the component checks below; other combinations are not presumed to have identical numerical results. Package mirrors are optional user configuration, not part of the evaluation protocol.

Note

flash-attn is optional; RLDX-1 uses PyTorch SDPA when it is absent. If installing it, follow the FlashAttention installation guidance and select a build compatible with your Python, Torch, CUDA and GPU. This guide does not prescribe a machine-specific wheel.

Post-install setup

Download the kitchen assets (~10 GB) outside site-packages so they survive reinstalls. Target50 does not use RoboCasa dataset or teleop macros, so skip the optional private-macros setup:

robocasa-download-assets --assets-path ~/.robocasa/assets --no-macros -y

It prints the environment variable to export afterwards; add it to the shell that launches rpent:

export ROBOCASA_ASSETS_PATH=~/.robocasa/assets

The external root requires the six downloaded collections and the bundled static scene, arena and fixture files. The corrected installer supplements the latter without replacing different existing content. Re-run with --skip-existing to verify successful download inventories; a nonempty directory alone is not a complete installation. Keep official attribution files and finish interrupted downloads before starting experiments.

Resource publication is atomic: interrupted copies do not leave half-written final files. To repair conflicting resources left by an earlier installer, rerun the same command with explicit overwrite permission:

robocasa-download-assets --assets-path ~/.robocasa/assets --no-macros --overwrite -y

--overwrite takes precedence over --skip-existing and replaces only resource files in the installation scope, not unrelated files or whole directories. Without it, different existing content is preserved. Atomic no-overwrite publication requires hard-link support on the destination filesystem. Temporary files left by a killed process do not block retries.

New collections require space for the ZIP and one unpacked copy: staging is published without copying the payload again. Existing installations need additional space during replacement. --skip-existing avoids downloading and comparing completed collections; bundled static files are checked separately.

Navigation camera

The robocasa extra installs the rpent branch of RLinf/robosuite, which provides the Omron base’s fixed navview camera. Its composed MuJoCo name is mobilebase0_navview. Navigation RGB-D and world-map rendering validate the camera when they first request it, and report an error if it is missing. No manual site-packages XML patch is required. Target50 uses this same maintained branch; record the resolved revision with the environment information above.

RLDX-1 checkpoint

The --vla-model-path flag on the run commands below expects a local path to the RLDX-1-FT-RC365 checkpoint (the RoboCasa365 fine-tune). Download it from HuggingFace:

hf download RLWRLD/RLDX-1-FT-RC365 \
   --revision 587e9ecdcc5e7184fcc17f58713908edff5af041 \
   --local-dir ./checkpoints/rldx-1-ft-rc365

If the download is slow, use the HF mirror:

HF_ENDPOINT=https://hf-mirror.com hf download RLWRLD/RLDX-1-FT-RC365 \
   --revision 587e9ecdcc5e7184fcc17f58713908edff5af041 \
   --local-dir ./checkpoints/rldx-1-ft-rc365

RLDX-1 backbone support files

The FT checkpoint contains the weights but also references RLWRLD/RLDX-1-VLM for architecture, processor and tokenizer metadata. Target50 freezes revision 4b9f870d1287e0d38d7eb1445e6d8c60afe66dd7: 15 non-weight files, about 16.4 MB including documentation and images. Download these into the same cache used when launching RPent:

export HF_HOME="$PWD/.cache/huggingface"
export HF_HUB_CACHE="$HF_HOME/hub"
hf download RLWRLD/RLDX-1-VLM \
   --revision 4b9f870d1287e0d38d7eb1445e6d8c60afe66dd7 \
   --include "*.json" "*.txt" "*.jinja" "*.md" "*.png" ".gitattributes" \
   --exclude "*.safetensors.index.json"

No additional base weights are required. Keep these cache variables in the launch shell; do not shadow them with an empty TRANSFORMERS_CACHE. The RoboCasa VLA worker automatically uses this same support revision for both ordinary and Target50 runs, including separately started RPent VLA servers. There is no extra revision flag or manual cache-ref edit. The pin applies only to backbone metadata, not the weights selected by --vla-model-path. Model and asset licenses apply separately from RPent’s code license.

Task memory

Select automatic synchronization with --memory-profile hf (the default). Before every such ordinary run, RPent’s shared memory manager synchronizes the robocasa/** subtree from the RLinf/RPent-memory dataset into memory/robocasa. An online ordinary run therefore requires no separate memory download. The current task may use only these task-matched files under results/:

memory/robocasa/results/<Task>_s0.json
memory/robocasa/results/recipe_<Task>_s0.jsonl
memory/robocasa/results/<Task>.md  # optional

The final published corpus contains 43 audit JSON files, 43 recipe JSONL files, and 25 task Markdown files, for 111 files in total and no global memory. The JSON/JSONL pair contains reviewed seed-0 evidence. The optional Markdown file contains task-specific exploration memory and may summarize multiple attempts; all 16 Composite-Seen and 9 Composite-Unseen tasks provide one. The prompt requires the planner to read every current-task file that exists before acting. RPent makes those files available through read_text_file but does not inject their contents into the prompt.

RoboCasa never asks the planner to use global memory or another task’s memory. Seven Composite-Unseen tasks have no task memory and remain in the evaluation: HeatKebabSandwich, PanTransfer, PortionHotDogs, SeparateFreezerRack, WaffleReheat, WashFruitColander, and WeighIngredients. They continue from live observations. Memory is strategy evidence; historical coordinates, poses, pixels, and subtask prompts must not replace current localization or the live task language.

Ordinary runs synchronize Hugging Face main. Formal Target50 runs use the immutable memory snapshot 551fc3157b3e56b40a3d3a3b4c7ff81721ebe89b:

hf download RLinf/RPent-memory \
   --repo-type dataset \
   --revision 551fc3157b3e56b40a3d3a3b4c7ff81721ebe89b \
   --include "robocasa/**" \
   --local-dir ./target50-memory

Select the local profile and pass the directory containing the frozen results corpus:

rpent --robot robocasa --task-name OpenDrawer --seed 1 \
      --vla-model-path ./checkpoints/rldx-1-ft-rc365 \
      --planner claude_code --model claude-opus-4-8 \
      --memory-profile local \
      --memory-dir ./target50-memory/robocasa

Harness VLA Target50 reproduction protocol#

robots/robocasa/eval/target50.json is the canonical manifest for reproducing Harness VLA on RoboCasa Target50. It freezes the target environment split, HF resource revisions, memory scope, task and seed matrix, cell time limits, success source, and retry policy. Its protocol ID is robocasa-harness-vla-v1. Source dependencies follow the recorded rpent branches and must be recorded at their resolved revisions for each run:

RoboCasa Target50 matrix#

Split

Tasks

Seeds per task

Cell timeout

Cells

Atomic

18

1–10

1800 s

180

Composite-Seen

16

1–5

3600 s

80

Composite-Unseen

16

1–5

3600 s

80

Total

50

340

The tasks split into three groups:

  • Atomic (18) — single-primitive articulation and pick-place tasks: CloseBlenderLid, CloseFridge, CloseToasterOvenDoor, CoffeeSetupMug, NavigateKitchen, OpenCabinet, OpenDrawer, OpenStandMixerHead, PickPlaceCounterToCabinet, PickPlaceCounterToStove, PickPlaceDrawerToCounter, PickPlaceSinkToCounter, PickPlaceToasterToCounter, SlideDishwasherRack, TurnOffStove, TurnOnElectricKettle, TurnOnMicrowave, TurnOnSinkFaucet.

  • Composite seen (16) — multi-step tasks on kitchen layouts seen during training: ScrubCuttingBoard, StackBowlsCabinet, WashLettuce, RinseSinkBasin, PreSoakPan, StirVegetables, LoadDishwasher, SteamInMicrowave, SetUpCuttingStation, GetToastedBread, DeliverStraw, KettleBoiling, PrepareCoffee, StoreLeftoversInBowl, SearingMeat, PackIdenticalLunches.

  • Composite unseen (16) — multi-step tasks on layouts not seen during training (generalization eval): ArrangeBreadBasket, ArrangeTea, BreadSelection, CategorizeCondiments, CuttingToolSelection, GarnishPancake, GatherTableware, HeatKebabSandwich, MakeIceLemonade, PanTransfer, PortionHotDogs, RecycleBottlesByType, SeparateFreezerRack, WaffleReheat, WashFruitColander, WeighIngredients.

Pass any of these to --task-name. The full RoboCasa catalog is larger; see the RoboCasa upstream.

Running a task#

HTTP RPC endpoints whose hostname is 127.0.0.1 or localhost are reached directly, whether RPent starts the worker or the user supplies the endpoint. Every other hostname and IP uses the standard proxy environment. Codex applies the same two-host exception only to its child process for the local MCP connection. Leave HTTP_PROXY and HTTPS_PROXY unchanged when Hugging Face, a remote planner, or another remote service requires them; the default runtime does not require a shell-wide NO_PROXY setup.

If a user-supplied local service uses another hostname or IP and should be reached directly, add that exact value to the user’s existing NO_PROXY and no_proxy configuration.

The RoboCasa CLI flags are registered by robots/robocasa/__init__ and are visible under rpent --robot robocasa --help:

rpent --robot robocasa \
      --task-name OpenDrawer \
      --split target \
      --seed 1 \
      --vla-model-path /path/to/rldx \
      --planner claude_code \
      --model claude-opus-4-8

RoboCasa does not select a planner implementation; any planner supported by RPent can run this robot. See Agentic Planner for configuration.

For Target50, first download the fixed resources above, then invoke one ordinary command for each manifest cell. The Codex reference profile is gpt-5.5, xhigh, and max_turns=100; RoboCasa itself remains planner-agnostic. For the scene identity, use the ordinary --seed argument and do not set RLDX_RESET_SEED. Ordinary RoboCasa uses max_chunks=70; Target50 alone overrides it to 40. Freeze the Target50 RLDX execution values first:

export RLDX_MAX_CHUNKS=40
export RLDX_SETTLE_PATIENCE=999
export RLDX_ACTION_STEPS_PER_CHUNK=8
unset RLDX_RESET_SEED

The first OpenDrawer Atomic cell is:

rpent --robot robocasa \
      --task-name OpenDrawer --split target --seed 1 \
      --vla-model-path ./checkpoints/rldx-1-ft-rc365 --cuda-device 0 \
      --planner codex --model gpt-5.5 --reasoning-effort xhigh \
      --max-turns 100 --planner-timeout-s 1800 \
      --memory-profile local \
      --memory-dir ./target50-memory/robocasa \
      --output-dir ./runs/target50/atomic/OpenDrawer_s1

Use --planner-timeout-s 3600 for either composite split. Execute Atomic, Composite-Seen, and Composite-Unseen in that order. A cell succeeds only when the final recorded environment state has state.success=true; the planner’s finish(status=...) argument is not an evaluation label. Valid task failures and planner timeouts are not retried. Retry an infrastructure failure only when no valid environment result was produced for that cell.

Every completed command atomically writes <output-dir>/result.json using the final environment state.success. The record includes the effective protocol values but omits provider errors and credentials. Once all cells are present under <results-root>/<manifest-split>/<Task>_s<seed>/result.json, validate the fixed denominator and print the task-weighted score with:

python -m robots.robocasa.eval.validate_target50 ./runs/target50

Note

Use --env-endpoint / --vla-endpoint to point at already-running servers ([protocol://]host:port); when omitted, RPent spawns the env and VLA daemons in-process and writes their logs to <output_dir>/env_server.log and <output_dir>/vla_server.log.

Published Target50 results#

The published Codex reproduction contains all 340 cells and reports the following task-level aggregates:

Codex Target50 reproduction#

Split

Successful cells

Success rate

Harness VLA reference

Atomic

163/180

90.56%

165/180 (91.67%)

Composite-Seen

49/80

61.25%

45/80 (56.25%)

Composite-Unseen

12/80

15.00%

11/80 (13.75%)

Overall (task-weighted)

N/A

57.00%

55.40%

The complete per-task table contains the success count and accuracy for every task. The published record is task-level aggregate data; it does not include per-seed traces, raw trajectories, or failure classifications and therefore is not a per-cell audit artifact.

Environment smoke tests#

After installing RoboCasa and its assets, run the opt-in environment smoke suite to check simulator installation and interfaces. It requires no planner credentials or VLA checkpoint:

uv pip install pytest pytest-timeout
RPENT_RUN_ROBOCASA_INTEGRATION=1 \
   pytest tests/integration_tests/robots/robocasa/test_target50_runtime_smoke.py -v

The four cases cover OpenDrawer, NavigateKitchen, and PickPlaceCounterToCabinet at seed 1, plus mobile-camera movement. Task checks verify construction/reset, 12D actions, operation cameras, navigation RGB-D/world map, the success predicate, and clean close. The camera check verifies pose and image changes after eight base steps. These real-simulator tests require a working GPU/EGL setup and are separate from offline CPU CI; skipped tests are not passes.

Troubleshooting#

Run the environment smoke tests first. After downloading all four resources, use the existing RoboCasa E2E component test to verify VLA worker startup, HTTP RPC and first inference. Install .[test] if needed, select one available GPU and use a fresh output directory:

CUDA_VISIBLE_DEVICES=0 \
RLDX_MODEL_PATH="$PWD/checkpoints/rldx-1-ft-rc365" \
RPENT_E2E_OUTPUT_DIR="$PWD/e2e-robocasa" \
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 NO_ALBUMENTATIONS_UPDATE=1 \
MUJOCO_GL=egl python -m pytest -q \
   tests/e2e_tests/robocasa/test_components.py::test_rldx_component --timeout=300

These checks do not run a planner or create benchmark results. Skipped tests are not passes. Offline variables apply only to the check; ordinary HF memory sync needs network access. Keep remote planner proxies unchanged. The lightweight protocol tests still validate all 50 tasks and the fixed 340-cell denominator; full benchmark execution is a separate procedure.

  • For slow package downloads, use UV_HTTP_TIMEOUT=600 and put caches and temporary files on a sufficiently large filesystem. Retry the pinned HF download; apparent shard size is not a completeness check. Do not disable TLS.

  • A read-only asset failure needs the corrected RoboCasa dependency, not writable canonical assets. Transformed XML uses the temporary directory.

  • For an RLDX offline cache miss, check the support snapshot and cache variables above. NO_ALBUMENTATIONS_UPDATE=1 disables only an import-time version check, not image processing. Keep the existing image-geometry fallback.

  • Test the selected Torch/CUDA build with a GPU operation and EGL render, not just the driver’s version display. Use a build compatible with the host.

  • For shared read-only installations, set NUMBA_CACHE_DIR to a writable per-user directory instead of making package code writable.

  • If navigation RGB-D or world-map rendering reports a missing mobilebase0_navview, reinstall .[robocasa] to refresh the RLinf/robosuite rpent branch. Do not patch installed XML files manually.

  • If read_text_file reports a missing current-task result, check the memory/robocasa/results/ corpus or the selected local directory. RPent does not fall back to another task’s memory. Markdown is optional; Atomic tasks have no published <Task>.md.

  • Environment and VLA startup failures are recorded in <output_dir>/env_server.log and <output_dir>/vla_server.log; also inspect <output_dir>/run.log for the run-level error.

  • Only the exact 127.0.0.1 and localhost hostnames bypass HTTP proxies automatically. Other hostnames and IPs use the standard proxy environment; add the exact host to NO_PROXY and no_proxy only when it should be reached directly.

Toolkit design vs. LIBERO#

The RoboCasa toolkit exposes the same shape of tools as LIBERO (a primitive call, a state view, a finish), with two RoboCasa-specific aspects:

  • Env-side helpers. Grasp checks and action assembly need the live simulator env, so they live in env_server as RPCs. The agent-side skill holds both clients: the env client for render/step, the model client for RLDX-1 inference. See Add a New Robot for the rationale.

  • Observation shape. RLDX-1 sees 3 camera video tensors (1, T, H, W, 3) stacked over history T, plus state.* and annotation.* fields. The session id is not part of the observation — it is managed automatically by the RPC framework: RpcClient generates a private rpc_ + uuid hex session id, wait_for_ready registers it with the server on connect; the server tracks each session’s idle time and a background sweep thread reaps sessions idle longer than the timeout (default 3600s), and the client sends session.close via atexit on process exit. Business code (rldx_skill / vla_client) never sees the session id directly; the server injects it into predict / reset_session to isolate per-client RLDX memory/RTC policy state.