Payana did not start as a generic labeling tool - it grew out of building datasets that teach robots to act. The platform's original mission is converting first-person (egocentric) video of humans performing real tasks into robot-training data: what the person saw, what their hands did, which objects they touched, and how the task decomposed into steps. That heritage is why Payana exports robot-learning formats natively and why its video tooling is unusually deep for an annotation service.
Imitation learning and vision-language-action (VLA) models have made human demonstration data the scarcest resource in robotics. Robot fleets are expensive; human video is cheap and abundant - if it can be annotated richly enough to train on. That is precisely the gap Payana fills.
What an embodied-AI dataset contains
| Signal | What it teaches the robot | How Payana produces it |
|---|---|---|
| Object detections & masks | What is in the scene and exactly where | AI-assisted boxes and per-frame instance masks on manipulated objects |
| Hand trajectories | How the end-effector should move | hand keypoints per frame, converted to wrist position + orientation + gripper-state vectors |
| Action segmentation | Task structure: steps, order, success | episode timelines segmented into task cycles with per-step labels |
| Language annotations | Instruction-following (VLA training) | natural-language task and step descriptions aligned to timestamps |
| Success / failure labels | Outcome supervision | per-episode and per-step success tags with off-task filtering |
Egocentric video, handled properly
Head-mounted and chest-mounted camera footage is its own discipline: the camera moves with the wearer, hands enter and leave the frame constantly, and the interesting objects are frequently occluded by the very hands manipulating them. Payana's tracking was hardened on exactly this footage - occlusion-aware tracks that survive a hand passing in front of the target object, per-frame masks that stay aligned through rapid head motion, and automatic filtering of off-task segments (the tea break in the middle of the assembly demonstration) so the training data contains the task and only the task.
Robot-native export formats
This is the capability that separates Payana from general labeling services: datasets export directly to LeRobot v3 and RLDS / Open X-Embodiment - the formats the open robot-learning ecosystem trains on - alongside Parquet and JSONL for custom pipelines. Episodes carry synchronized video, per-frame observation and action vectors (wrist position, orientation, gripper state), language instructions, and success labels. Your research stack loads the delivery like any public benchmark dataset; no conversion scripts, no schema archaeology.
From raw demonstration to dataset, end to end
A robotics engagement typically runs: (1) you record humans performing the target task - we advise on camera placement and capture protocol; (2) footage is uploaded and automatically segmented into task episodes; (3) AI-assisted annotation adds object masks, hand trajectories, and step labels, with human review throughout; (4) off-task and failed segments are flagged and excluded or kept as negatives, per your training recipe; (5) the dataset exports to LeRobot/RLDS with versioned splits. The result is demonstration data at video-collection cost rather than robot-teleoperation cost.
Why human video beats teleoperation for scale
The dominant way to collect robot demonstrations today is teleoperation: a human drives the robot through the task, and the robot logs its own observations and actions. It produces clean data - and it does not scale. Teleoperation requires the robot, the rig, a trained operator, and real time; every hour of data costs an hour of an expensive, contended physical system. Human first-person video inverts the economics: any worker wearing a camera generates demonstration data as a side effect of doing their actual job, in the real environment, at the real pace, with all the variation robot policies need and lab collections lack. The catch is that raw human video is not training data - it has no action labels, no episode boundaries, no success signals. Payana's pipeline is the conversion layer: it supplies the hand-derived action vectors, the task segmentation, and the outcome labels that turn cheap video into the same tensor structure a teleoperated dataset provides. Teams use both - a small teleoperated set for grounding, a large human-video set for coverage - and Payana makes the second one affordable.
Episode curation: the unglamorous half of dataset quality
Policy training is exquisitely sensitive to what is in the dataset, not just how it is labeled. A long recording session contains the task - and also the interruptions: a phone check, a chat with a colleague, a tea break mid-assembly. Payana's curation layer segments long recordings into task episodes automatically, detects off-task activity by comparing each segment against the task's own visual signature (no hard-coded list of "distractions" - the anchor is your task), and flags outlier episodes across the collection. Reviewers confirm or override every exclusion. The result is a dataset where every episode is actually a demonstration of the task - which is worth more than any amount of annotation polish applied to contaminated data.
Quality metrics that predict policy performance
Robotics datasets need different QA than detection datasets. Alongside standard label review, we report the statistics that correlate with whether a policy will train: episode-length distribution (policies struggle when demonstrations vary wildly in duration), action-vector continuity (jumps in the wrist trajectory indicate hand-tracking dropouts, which we re-annotate), per-step label coverage, and success-label balance. Because the export is a standard format, these checks run on the exact tensors your trainer will read - the dataset is validated in its consumable form, not in an internal representation that might export differently.
Where this is going: video as the robot-data substrate
The frontier of robot learning is converging on a simple bet: the cheapest path to general manipulation runs through massive human-video pretraining, refined by comparatively small robot-collected datasets. Every major open robot-learning effort now mixes human video into training. That makes the capability Payana was built around - converting ordinary task video into aligned, action-labeled, format-compliant episodes - the supply chain for embodied AI, and it is the reason the rest of the platform (the tracking depth, the per-frame masks, the occlusion handling) exists at the quality level it does: robot data demanded it first, and every other industry on this page inherits it.
What a delivered episode actually contains
Concretely, a Payana robot-learning delivery is a set of episodes, each carrying: the synchronized video stream; a per-frame observation.state and action vector in the 7-DoF convention (wrist position xyz, wrist direction xyz, gripper open/close) derived from tracked hand pose; step-level segmentation with natural-language descriptions ("pick the bottle", "place on scale"); object annotations for the manipulated items where the training recipe uses them; and episode-level metadata - task name, success label, duration, capture source. In LeRobot v3 that arrives as the standard parquet-and-video layout your dataloader already reads; in RLDS it follows the Open X-Embodiment episode convention, making the dataset mixable with the public corpora most VLA training runs already consume. The practical consequence: the day the delivery lands, it trains - the integration cost that kills most custom robotics data projects is zero by construction.
Mixing your robot's data with human video
Most teams arrive with some robot-collected data already - teleoperated episodes, scripted rollouts - and the real question is how human video joins it. The format answer is above (same episode structure, same conventions); the schema answer is co-design: we align task names, step vocabularies, and success criteria with your existing collection so the mixed dataset is coherent, and we tag provenance (human video vs. robot) on every episode so training recipes can weight the sources differently - the standard approach in current robot-learning practice. Some clients also route their robot footage through Payana for the parts robots cannot label about themselves: semantic object masks, failure-mode tags, and language descriptions. For teams searching for a robot learning dataset service, imitation learning data collection, egocentric video annotation, or LeRobot / RLDS dataset preparation, this section is the platform's origin story rather than an add-on - the entire pipeline exists because robot data demanded it.
Capture protocol: the cheapest quality lever
Every robotics dataset's ceiling is set before annotation begins, at recording time - so protocol advice is part of the engagement, not an upsell. The rules are simple and violating them is expensive: mount the camera so both hands and the workspace stay in frame through the full task cycle (chest or head mount chosen per task geometry); keep task cycles distinct rather than blending activities, so episode segmentation has clean boundaries to find; vary what the policy must be robust to (object positions, lighting, operators) and hold constant what it should not learn as signal; and record failures deliberately - a dataset of only successes teaches a policy nothing about recovery. We review the first day of recordings before the client collects the rest, because catching a framing problem after one hour of footage costs one hour; catching it after a month costs the month. This loop - protocol, pilot recordings, review, then scale - is the same pilot discipline as every other domain on this page, applied to the domain where Payana itself began.
LeRobot v3 and RLDS (Open X-Embodiment convention) natively, plus Parquet and JSONL for custom training stacks. Standard vision formats (COCO, YOLO) are also available from the same projects.
Yes - hand keypoints are estimated per frame and converted to end-effector-style action vectors (wrist position and direction plus gripper open/close), the standard recipe for human-video pretraining of manipulation policies.
Any stable first-person camera works - head-mounted action cameras are the common choice. We advise on mounting, field of view, and capture protocol before recording starts so the footage is trainable, not just watchable.
Yes - episodes and steps carry natural-language descriptions aligned to timestamps, suitable for instruction-conditioned policies and vision-language-action training.