LingBot-VLA v2 Examples¶

This example runs the official LingBot-VLA v2 6B base checkpoint through TeleFuser. It accepts a RobotWin observation and returns a normalized 50 x 55 canonical action chunk through direct Python inference or the native structured service.

Model Source¶

Model	HuggingFace	ModelScope	Purpose
LingBot-VLA v2 6B base	robbyant/lingbot-vla-v2-6b	Robbyant/lingbot-vla-v2-6b	Vision-language-action policy checkpoint supplied as local shards
Qwen3-VL-4B-Instruct	Qwen/Qwen3-VL-4B-Instruct	Qwen/Qwen3-VL-4B-Instruct	Backbone configuration and processor

The parity reference uses Robbyant/lingbot-vla-v2 at commit be27333c9b5f2663b0ec33f069dd7dfd67fa32b5.

Feature Support¶

Feature	Support	Notes
Official 6B base checkpoint	Supported	Local sharded safetensors checkpoint
RobotWin preprocessing	Supported	Three RGB cameras, task text, and a raw 14-dimensional state
Canonical action output	Supported	Normally `50 x 55` normalized actions
BF16 inference	Supported	Default path; strict 38-tensor upstream parity passed
CUDA Graph	Supported	Dynamic eager prefix with an opt-in fixed-shape action-denoising graph
Quantization	Partial	Profile-specific release status; see Configuration and Performance
Native server API	Supported	Asynchronous structured task API and `TFClient`
RoboTwin policy protocol	Supported	Standalone persistent MessagePack WebSocket service
Request replicas	Supported	One complete policy copy per GPU
Single-policy FSDP, TP, or PP	Unsupported	The integration does not split one policy across GPUs
RoboTwin action mapping	Supported	Unnormalizes canonical output to absolute-position `50 x 14` chunks

Requirements¶

GPU: one H100 80 GB was used for parity, performance, quantization, and service validation.
Software: Python 3.10.12, PyTorch 2.11.0+cu130, CUDA 13.0, Transformers 5.14.1, and Triton 3.6.0.
Quantization: TorchAO 0.17.0 and bitsandbytes 0.48.0 are pinned; tf-kernel FP8 needs a compatible SM90 wheel.
Input assets: three RGB camera images, a non-empty instruction, and a finite 14-dimensional RobotWin state.

Install TeleFuser after installing the PyTorch build that matches the target CUDA runtime:

python3.10 -m venv .venv-vla
source .venv-vla/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e ".[dev]"

Model Directory¶

${TF_MODEL_ZOO_PATH}/
|-- lingbot/
|   `-- lingbot-vla-v2-6b/
|       |-- model.safetensors.index.json
|       `-- model-*.safetensors
`-- Qwen3-VL-4B-Instruct/

export TF_MODEL_ZOO_PATH=/path/to/model_zoo

The VLA directory must contain every shard referenced by model.safetensors.index.json.

Quick Start¶

mkdir -p work_dirs/lingbot_vla_v2
.venv-vla/bin/python examples/lingbot_vla_v2/lingbot_vla_v2_inference.py \
  --model-root "$TF_MODEL_ZOO_PATH/lingbot/lingbot-vla-v2-6b" \
  --qwen3vl-root "$TF_MODEL_ZOO_PATH/Qwen3-VL-4B-Instruct" \
  --camera-high /data/cam_high.png \
  --camera-left-wrist /data/cam_left_wrist.png \
  --camera-right-wrist /data/cam_right_wrist.png \
  --task "pick up the red block" \
  --state-json '[0,0,0,0,0,0,0,0,0,0,0,0,0,0]' \
  --seed 7 \
  --output work_dirs/lingbot_vla_v2/action_chunk.npz

The output NPZ contains the normalized canonical action chunk, shape metadata, checkpoint variant, and policy verification status.

Examples¶

RobotWin Action Inference¶

`lingbot_vla_v2_inference.py`¶

This is the smallest in-process entry point. The input processor loads images in high, left-wrist, right-wrist order, applies the bundled bounds_99_woclip statistics, and maps the raw 14-dimensional state into the 55-dimensional canonical space.

python examples/lingbot_vla_v2/lingbot_vla_v2_inference.py \
  --model-root "$TF_MODEL_ZOO_PATH/lingbot/lingbot-vla-v2-6b" \
  --qwen3vl-root "$TF_MODEL_ZOO_PATH/Qwen3-VL-4B-Instruct" \
  --camera-high /path/to/cam_high.png \
  --camera-left-wrist /path/to/cam_left_wrist.png \
  --camera-right-wrist /path/to/cam_right_wrist.png \
  --task "pick up the red block" \
  --state-json '[0,0,0,0,0,0,0,0,0,0,0,0,0,0]' \
  --output work_dirs/lingbot_vla_v2/action_chunk.npz

Key options:

Option	Default	Description
`--device`	`cuda`	Inference device
`--seed`	None	Optional deterministic noise seed
`--cuda-graph`	Disabled	Keep the dynamic prefix eager and graph the fixed-shape denoising loop
`--quantization`	None	Select an optional quantization profile
`--output`	`canonical_action_chunk.npz`	Output NPZ path

The returned action chunk normally has horizon 50 and action dimension 55. The official base checkpoint intentionally returns policy_verified=False and verification_status="unverified_official_6b_base".

Configuration¶

CUDA Graph¶

--cuda-graph keeps vision-language prefix encoding and 36-layer KV-cache construction eager for every request, then lazily captures all 10 fixed-shape action-denoising steps. Different instructions and language padding masks therefore reuse the same denoising graph without being tied to the warmup instruction.

The denoising graph remains specialized to its tensor shapes, dtypes, and device. Standard preprocessing keeps these layouts fixed at batch 1, language length 72, and action shape 1 x 50 x 55. Prefix and graph execution are serialized per policy instance. Close the pipeline to release graph buffers.

CUDA Graph cannot be combined with torch.compile or online quantization other than fused-fp8-graph. Invalid combinations fail before model loading.

Online Quantization¶

BF16 remains the default and the only profile covered by strict upstream parity. Quantization is applied in memory and does not modify checkpoint files.

CLI value	Backend	Scope	Validation status
`fused-fp8-graph`	Native scaled GEMM and Triton	Repeated denoising Linear and routed-MoE weights	H100 functional/AIPerf validated; release gate failed exact HTTP replay
`torchao-fp8`	TorchAO	492 selected Qwen/action-expert Linear layers	H100 functional/AIPerf validated; release gate failed exact HTTP replay
`bnb-nf4`	bitsandbytes	Same 492 Linear-layer manifest, NF4 weights and BF16 compute	H100 functional/AIPerf validated; release gate failed exact HTTP replay
`tf-kernel-fp8`	TeleFuser tf-kernel	Per-token activation and per-output-channel weight FP8	Code/unit tested; hardware unverified

Use the direct example with one of the following variants:

# Fused FP8 requires CUDA Graph.
--quantization fused-fp8-graph --cuda-graph

# Online capacity profiles without CUDA Graph.
--quantization torchao-fp8
--quantization bnb-nf4
--quantization tf-kernel-fp8

The tf-kernel path requires an SM90 wheel built for the exact PyTorch/CUDA ABI. It remains "code support, hardware unverified" until that real-model run succeeds on a compatible installation.

The complete H100 release suite passed BF16 eager and BF16 Graph. The three runnable quantized profiles passed direct/HTTP numerical thresholds, AIPerf, dynamic-instruction, fault, and shutdown checks, but did not produce bit-exact HTTP replays. They therefore remain code-supported capacity profiles rather than release-validated profiles.

The public loader accepts the same options:

from telefuser.pipelines.lingbot_vla_v2.runtime import get_lingbot_vla_v2_pipeline

pipeline = get_lingbot_vla_v2_pipeline(
    "/path/to/lingbot-vla-v2-6b",
    "/path/to/Qwen3-VL-4B-Instruct",
    device="cuda:0",
    quantization="torchao-fp8",
)

Compare a deterministic quantized capture with the corresponding TeleFuser BF16 capture:

.venv-vla/bin/python tools/validation/compare_lingbot_vla_v2_quantization.py \
  --reference work_dirs/vla_quantization/bf16_seed7.npz \
  --candidate work_dirs/vla_quantization/torchao_seed7.npz \
  --candidate-replay work_dirs/vla_quantization/torchao_replay_seed7.npz \
  --min-cosine 0.995 --max-relative-l2 0.10 --max-abs 0.5 \
  --require-exact-replay \
  --output work_dirs/vla_quantization/bf16_vs_torchao.json

Serving¶

Start the native structured service:

TF_MODEL_ZOO_PATH=/path/to/model_zoo \
  .venv-vla/bin/telefuser serve \
  examples/lingbot_vla_v2/lingbot_vla_v2_native_service.py \
  --task vla_action --parallelism 1 --host 127.0.0.1 --port 18080

Submit POST /v1/tasks/structured with task="vla_action", the instruction, 14-dimensional state, three Base64 camera fields, and an optional seed. Poll GET /v1/tasks/{task_id}/status. Each encoded camera is limited to 10 MiB and 16,777,216 decoded pixels.

from telefuser.client import TFClient

client = TFClient("http://127.0.0.1:18080")
result = client.predict_vla_actions(
    instruction="pick up the red block",
    state=[0.0] * 14,
    camera_high_path="/data/cam_high.png",
    camera_left_wrist_path="/data/cam_left_wrist.png",
    camera_right_wrist_path="/data/cam_right_wrist.png",
    seed=7,
)

Use request-level replicas when multiple GPUs are available:

CUDA_VISIBLE_DEVICES=0,1 TF_MODEL_ZOO_PATH=/path/to/model_zoo \
  .venv-vla/bin/telefuser serve \
  examples/lingbot_vla_v2/lingbot_vla_v2_native_service.py \
  --task vla_action --parallelism 2 --num-replicas 2 --port 18080

This creates one complete policy per GPU; it does not enable tensor or pipeline parallelism within a policy.

RoboTwin Policy Server¶

The standalone policy server implements the persistent MessagePack WebSocket protocol used by the upstream WebsocketClientPolicy. It is isolated from telefuser serve: no TeleFuser API routes, service schemas, or other model integrations are changed.

Install the protocol dependency in the TeleFuser inference environment:

.venv-vla/bin/python -m pip install -r examples/lingbot_vla_v2/requirements-robotwin.txt

Start one resident policy process:

.venv-vla/bin/python examples/lingbot_vla_v2/lingbot_vla_v2_robotwin_server.py \
  --model-root "$TF_MODEL_ZOO_PATH/lingbot/lingbot-vla-v2-6b" \
  --qwen3vl-root "$TF_MODEL_ZOO_PATH/Qwen3-VL-4B-Instruct" \
  --device cuda:0 --host 0.0.0.0 --port 9330 --use-length 50

On NVIDIA H100, this dedicated entrypoint disables cuDNN SDPA before model warmup because the current PyTorch/cuDNN combination cannot build a valid vision-attention execution plan. Flash, memory-efficient, and math SDPA remain enabled. The override is process-local and is not applied to other TeleFuser pipelines.

The server exposes GET /healthz and the policy WebSocket at /. On connection it sends a MessagePack metadata frame, including the explicit 16 MiB request limit, then accepts multiple binary MessagePack requests on the same connection. This matches the upstream client contract:

from deploy.websocket_client_policy import WebsocketClientPolicy

policy = WebsocketClientPolicy(host="127.0.0.1", port=9330)
policy.reset("robotwin")
result = policy.infer(
    {
        "observation.images.cam_high": camera_high,
        "observation.images.cam_left_wrist": camera_left_wrist,
        "observation.images.cam_right_wrist": camera_right_wrist,
        "observation.state": state,
        "task": instruction,
    }
)
actions = result["action"]  # float32 [50, 14] when --use-length=50

The initial metadata frame describes protocol version 1.0, absolute_qpos action semantics, float32 dtype, horizon, dimension, and the exact dual-arm joint order. Inference requests may include an integer seed plus request_id and episode_id; the response echoes them and reports decode, lock-wait, pipeline, action-mapping, and adapter timings. Existing clients may omit all three request fields.

The endpoint also advertises an additive, latest-wins action scheduler. A client that overlaps simulation and inference should send a monotonically increasing sequence_id within each episode_id, plus a positive request_ttl_ms. The server has one GPU worker, retains at most one pending request per connection/episode, and accepts new observations while inference is running. A newer observation replaces queued work; because an in-flight CUDA call cannot be cancelled, its result is discarded after completion when it has become stale. Successful responses use scheduler_status="completed". Responses with superseded, expired, stale_sequence, or overloaded contain action=None and a structured error; clients must never execute those responses.

request_ttl_ms starts when the H100 server receives the request. Do not compare monotonic timestamps between the H100 and RTX machines. The RTX client should separately enforce its round-trip deadline and hold the current joint positions when no fresh action is available.

Validate this direct endpoint before a simulator is available. This sends reset and repeated inference requests to the resident model, validates the returned [H, 14] action contract, and optionally verifies exact fixed-seed replay:

.venv-vla/bin/python -m tools.validation.validate_lingbot_vla_v2_robotwin_ws \
  --host 127.0.0.1 --port 9330 \
  --image examples/data/lingbot_world_fast/image.jpg \
  --max-image-edge 640 \
  --task "pick up the object" --seed 7 --requests 10 \
  --output work_dirs/robotwin_ws_validation/smoke.json

The validator preserves aspect ratio and downsizes only images whose longest edge exceeds --max-image-edge, then checks the encoded MessagePack request against the limit advertised by the server before sending it. This keeps the large repository sample representative of normal RoboTwin camera payloads. Add --require-exact-replay only when validating a runtime profile that promises bitwise determinism; BF16 H100 inference is validated with numerical tolerances rather than identical action hashes.

Exercise overlapping submissions and stale-action rejection without a simulator:

.venv-vla/bin/python -m tools.validation.validate_lingbot_vla_v2_robotwin_ws \
  --host 127.0.0.1 --port 9330 \
  --image examples/data/lingbot_world_fast/image.jpg \
  --task "pick up the object" --seed 7 --requests 3 \
  --request-ttl-ms 5000 --overlap-requests \
  --output work_dirs/robotwin_ws_validation/overlap.json

This mode sends all observations before receiving responses, requires the newest request to return an action, and requires at least one older request to be reported as superseded.

Each request runs the existing pipeline, converts normalized canonical 50 x 55 output through the bundled RoboTwin profile, and returns absolute-position actions in raw RoboTwin order. --use-length may truncate the returned chunk; start with 50 for upstream-equivalent open-loop execution. The adapter accepts episode reset messages but deliberately rejects runtime checkpoint switching.

For split-machine deployment, run the model endpoint and the repository-owned XPolicyLab proxy on the H100 inference host. The proxy does not load a second model; it translates XPolicyLab observations to the direct TeleFuser protocol:

cd /data/RoboTwin
bash XPolicyLab/policy/TeleFuser_LingBot_VLA/setup_eval_policy_server.sh \
  RoboTwin lift_pot remote_base arx_x5 joint 0 0 \
  /data/RoboTwin/.venv 19000 0.0.0.0 \
  127.0.0.1 9330

On the remote RTX/Vulkan workstation, use the standard RoboTwin evaluation client and point it at the proxy. No TeleFuser files or model weights are required on that workstation:

cd /data/RoboTwin
bash scripts/eval_policy.sh \
  --bench_name RoboTwin \
  --task_name lift_pot \
  --env_cfg_type arx_x5 \
  --policy_name TeleFuser_LingBot_VLA \
  --host INFERENCE_HOST --port 19000 --protocol ws \
  --eval_batch false --root_dir /data/RoboTwin --device_id 0 \
  --additional_info ckpt_name=remote_base,action_type=joint \
  --seed 0 --task_config demo_clean --test_num 1

The current XPolicyLab proxy calls infer() synchronously, so it remains compatible but does not yet overlap action execution with inference. Full overlap requires an incremental RTX-side change: execute chunk N while submitting a newer observation for chunk N+1, keep only the newest completed chunk in an atomic action buffer, and apply the same sequence/deadline checks before execution. That simulator-side change is outside this repository and is not required for the no-simulation server validation above.

Keep ports 9330 and 19000 on a trusted private network or an SSH/VPN tunnel. These WebSocket endpoints do not provide authentication or transport encryption. The direct validator covers preprocessing, inference, mapping, and the inner WebSocket contract; only the RTX smoke episode can additionally establish XPolicyLab translation and one real SAPIEN simulation step.

The base checkpoint remains marked unverified_official_6b_base. This endpoint establishes preprocessing, inference, action mapping, transport, and simulator execution continuity; it does not establish RoboTwin task success without an embodiment-validated checkpoint.

Validation¶

The repository includes strict upstream parity, runtime, quantization, structured-service, fault, and AIPerf validators under tools/validation/ and benchmarks/telefuser_aiperf/.

Compare previously captured upstream and TeleFuser artifacts:

.venv-vla/bin/python tools/validation/run_lingbot_vla_v2_parity.py \
  --reference work_dirs/vla_parity/upstream_seed7.npz \
  --candidate work_dirs/vla_parity/telefuser_seed7.npz \
  --profile strict --output work_dirs/vla_parity/strict_report.json

Validate a running structured service or run a bounded soak:

.venv-vla/bin/python tools/validation/validate_lingbot_vla_v2_structured_service.py \
  --base-url http://127.0.0.1:18080 \
  --image examples/data/lingbot_world_fast/image.jpg \
  --quantization-profile bf16 --warmup 1 --requests 20 --concurrency 1 \
  --output work_dirs/vla_service_validation/smoke_20.json

# Replace --requests 20 with --duration-seconds 3600 for a one-hour run.

Run the repository-owned AIPerf workload with:

bash scripts/setup_aiperf.sh
bash benchmarks/telefuser_aiperf/scripts/run_vla_structured_bench.sh

Run the complete real-GPU release suite after installing AIPerf. It executes every runtime profile in an isolated process, compares direct and HTTP actions, changes instruction layout under CUDA Graph, runs bounded load and fault checks, verifies shutdown/restart, and records full checkpoint, processor, software, CUDA, and GPU identity:

.venv-vla/bin/python tools/validation/run_lingbot_vla_v2_release_suite.py suite \
  --model-root "$TF_MODEL_ZOO_PATH/lingbot/lingbot-vla-v2-6b" \
  --qwen3vl-root "$TF_MODEL_ZOO_PATH/Qwen3-VL-4B-Instruct" \
  --image examples/data/lingbot_world_fast/image.jpg \
  --gpu-index 0 \
  --output-dir work_dirs/lingbot_vla_v2_release

Use --profiles bf16-eager,bf16-graph for an intermediate run. Such a partial run is useful for development but is not a complete quantization support-matrix release result.

These checks establish framework parity and serving contracts, not physical robot task success.

Troubleshooting¶

CUDA Graph Or Quantization Is Rejected¶

Use CUDA Graph only with BF16 or fused-fp8-graph; the fused profile must include --cuda-graph. Other online quantization profiles run without CUDA Graph and require CUDA.

tf-kernel FP8 Cannot Load¶

Install a tf-kernel wheel built for the visible GPU architecture and exact PyTorch/CUDA ABI, or use BF16, TorchAO FP8, or BNB NF4. Do not promote tf-kernel FP8 to supported status based only on unit tests.

Notes¶

Canonical normalized actions are not physical robot commands. Deployment requires de-normalization, embodiment mapping, control frequency, limits, safety policy, feedback, and emergency-stop behavior.
unverified_official_6b_base remains intentional until an embodiment checkpoint and task-success evaluation are available.
Quantized profiles require separate numerical, performance, and robot task-success acceptance.
Generated captures and benchmark reports belong under the Git-ignored work_dirs/ directory.