Dexteritythatacross glove,across human,across robot.

One representation across sources, people, and robots

Current Supported Devices

Three core components, one hand-motion pipeline

The public interface is organized around three focused tools. Each stage has one job and a stable output for the next stage.

From clone to your first replay

Run all commands from the repository root.

Prerequisites

Linux, Git, uv, Python 3.11 or 3.12, and a desktop session for visualization. A supported glove and Dora are required only for collection and live --rt modes.

Installation

$ git clone <repository-url>
$ cd GHR
$ uv sync

Use uv run for the commands below; activating the environment is optional.

Configure one active tuple

Edit the repository-root ghr.env to select the source, person, hand model, optimizer, side, and robot:

GLOVE_SOURCE=manus
GLOVE_PERSON=<person-id>
GLOVE_MODELING=3-2-1-thumbCMCUp
OPTIMIZER=autodiff
HAND_SIDE=right
ROBOT=wuji2

For a one-off run, override a value inline: HAND_SIDE=left uv run caliber.

Fetch the active demo data

Download the captures, calibration results, and robot assets selected by ghr.env. The command also handles datasets published under the legacy calibration directory name.

$ uv run demo-data fetch

Run the first pipeline

With a fetched or collected calibration session:

# Calibrate and inspect.
$ uv run caliber
$ uv run viz

# Replay, export, and retarget.
$ uv run tracker
$ uv run tracker --save
$ uv run retargeter

Calibrate a glove to a human hand

Caliber turns static EMF pose captures into a verified, person-specific 21-point skeleton used by runtime Tracker IK.

Calibration input

Each glove adapter provides raw_device_data with shape (2, 6, 7): five fingertip EMF RX poses and one independent EMF TX/palm reference row for each side, stored as xyz + wxyz.

1. Collect the static pose set

Set GLOVE_SOURCE, GLOVE_PERSON, and HAND_SIDE first, then launch the collection UI:

$ uv run collect

# Optional one-run overrides:
$ uv run collect <person-id> --source manus --hand left

Match HAND_SIDE to the powered glove. After the static set, record one continuous motion for runtime validation.

2. Tune the fingertip sensor offset

If a pinch remains open, adjust the affected finger’s local sensor x offset, save it, and rerun Caliber. Offsets are specific to the source, person, hand, and glove fit.

$ uv run define_tip2rx
$ uv run caliber

3. Run the single-stage solve

Caliber runs one full-hand joint solve with analytic thumb-CMC coupling.

$ uv run caliber

4. Verify before runtime use

Inspect every pose and confirm that predicted RX poses align with the measurements. Then replay continuous motion to check pinch closure and joint stability.

$ uv run viz
$ uv run tracker

5. Use the result

# Replay and export the human hand.
$ uv run tracker
$ uv run tracker --save

# Retarget a recording or run the composed pipeline.
$ uv run retargeter
$ uv run teleoperater

# Live glove.
$ uv run tracker --rt
$ uv run teleoperater --rt

Add a New Robot

Add left/right kinematic assets, declare a robot profile, register its code, and validate a saved hand motion.

1. Add self-contained URDF and MJCF assets

The pipeline uses Pinocchio FK from URDF and Mink/MuJoCo from MJCF, so both representations are required for every supported side.

2. Declare the per-side hand profile

Create left/right YAML files under retarget/config/retargeter/robot/<name>/. Define the robot code, finger frames, fingertip sites, coupled joints, palm origin, and hand anchors. Shared solver settings remain in retarget/config/retargeter/common.yaml.

• Declare fingers in anatomical order: thumb, index, middle, ring, pinky.
• Use model frame and joint names, not qpos indices.
• Register the robot in ROBOT_CONFIGS and VALID_ROBOT.

3. Define origin, fingertips, and anchors

Run the geometry editor once per side and save the palm origin, fingertip sites, and hand anchors.

$ uv run define_robot example left
$ uv run define_robot example right

4. Validate with a saved human-hand motion

$ uv run retargeter path/to/recording.pkl --robot example --hand left
$ uv run retargeter path/to/recording.pkl --robot example --hand right

Verify model loading, joint limits, anchor alignment, stable motion, and pinch closure on both sides before adding a hardware driver.

Add a New Glove

A new glove is a producer for one normalized EMF pose format. Once the adapter satisfies that contract, collection, Caliber, visualization, and runtime Tracker IK remain source-agnostic.

1. Implement the one input contract

Every source emits raw_device_data with shape (2, 6, 7).

2. Separate hardware I/O from axis conversion

hardware/glove/collect/<source>_node.py
hardware/glove/dataflows/collect_<source>.yml
hardware/glove/dataflows/realtime_<source>.yml
calib/caliber/utils/<source>_adapt.py

• Keep SDK connections and Dora output in hardware/.
• Normalize metres, axes, sensor order, and wxyz quaternions in the adapter.
• Register the source in calib/caliber/utils/glove_sources.py.

Measure the fingertip-to-RX offsets for the new source before calibration:

$ GLOVE_SOURCE=<source> HAND_SIDE=left uv run define_tip2rx

3. Validate end to end

# Capture the complete static set from the new adapter.
$ GLOVE_SOURCE=<source> uv run collect --source <source> --hand left

# Calibration and inspection use the saved capture.
$ GLOVE_SOURCE=<source> uv run caliber
$ GLOVE_SOURCE=<source> uv run viz

# Only after offline validation: stream the live glove.
$ GLOVE_SOURCE=<source> uv run tracker --rt

Check shape, units, handedness, sensor loss, flat/pinch poses, and continuous motion before enabling live use.

VR and video-to-robot

Retargeter accepts metric 21-point human-hand motion from gloves, VR tracking, video, or another pose producer.

Input paths

MediaPipe-21 HDF5 replay

$ uv run retargeter --hdf5 path/to/recording.h5

The HDF5 adapter selects the active HAND_SIDE, uses the active ROBOT, opens the online MuJoCo UI at 30 FPS, loops until the window closes, holds the latest valid pose across invalid detections, and does not save a trajectory.

Live Meta Quest → robot retargeting

Start the XRoboToolkit PC Service and Quest client, then run one hand or both hands:

# One hand: ROBOT and HAND_SIDE come from ghr.env.
$ uv run teleoperater quest

# Override the robot and side for one run.
$ uv run teleoperater quest --robot wuji2 --hand right

# Track left and right together.
$ uv run teleoperater quest --bimanual

Use --robot, --hand, --bimanual, and --filter-alpha as needed. Quest mode renders robot targets but does not command physical hardware.

21-point input contract

Each frame is a finite (21, 3) array in metres using MediaPipe landmark order. Producers may provide world-oriented raw points or wrist-centred points in the project frame: x forward, y left, z up. Source adapters own scale, handedness, coordinate conversion, confidence filtering, and timing.

Dexterous manipulation demos

Recorded examples of GHR teleoperation across bimanual grasping, assembly, stacking, and in-hand manipulation.

Policy

Real-robot policy evidence is presented separately from teleoperated references so execution and demonstration remain unambiguous.

Command map

Primary workflow commands use uv run.

Artifacts and paths

The centralized helpers derive every path from the active tuple. Do not reconstruct a result path by guessing a nearby person or model.

data/caliber/data/
└── <source>/<person>/<hand>/
    ├── all_poses.npz
    ├── <pose>.npz
    └── motion_<timestamp>.npz

data/caliber/results/
└── <source>/<modeling>/<optimizer>/<person>/<hand>/
    ├── params.json
    ├── calibration_result.json
    ├── loss_history.json
    ├── bone_roll_correction.json
    └── viz_data.json

Troubleshooting

Contact Wei Pan