Robotics, embodied AI and AR teams use it to teach models how hands, tools and objects interact in the real world.
Capturing it at scale takes far more than handing out cameras. Training-ready footage depends on five things done well:
- The right capture rig, matched to what the model must learn.
- Scripted, diverse tasks across people, places and conditions.
- Quality checks at capture time, before bad footage reaches annotation.
- Consent and privacy by design, for participants and bystanders.
- Annotation built for action, not just object labels.
Miss any one of these and the dataset either fails in training or has to be re-shot. This guide walks through each stage, drawing on how Aya Data runs first-person capture and annotation programmes for physical AI teams.
Why are AI teams investing in first-person video data now?
First-person video shows a model of the world the way a robot or wearable device will see it in deployment. A robot’s onboard camera sits where the robot stands, so footage shot from a fixed corner camera teaches the wrong viewpoint.
Three trends are driving demand:
- Vision-language-action (VLA) models. These models take an image and an instruction, then output a physical action. They learn manipulation by watching human demonstrations from the doer’s perspective.
- Learning from demonstration. Recording a person folding laundry or assembling a part is far cheaper and more varied than collecting the same data with robot teleoperation.
- Smart glasses and AR assistants. Devices that guide a technician or cook need models trained on exactly what the wearer sees.
Public datasets show both the promise and the gap. Ego4D, built by Meta and university partners, gathered roughly 3,000 hours of everyday first-person video. Ego-Exo4D paired wearable cameras with synchronised external cameras across 740 participants, 13 cities and 1,286 hours of video. These datasets are valuable for research, but commercial teams usually need footage of their own tasks, environments and edge cases, under licences that allow commercial training.
That is why most teams now commission custom first-person datasets. For background on the format itself, see our complete guide to egocentric video for AI training, and for when to add external camera views, read Egocentric vs Exocentric: What Each View Teaches a Model.
How do you choose the right capture setup?
Start from what the model must learn, then pick the rig. A model learning fine hand manipulation needs different hardware from one learning to navigate a warehouse.
| Decision | Options | What it means for the model |
| Mount position | Head, chest, or wrist | Head mounts follow gaze and attention. Chest mounts give steadier footage of both hands. Wrist mounts capture close grasp detail. |
| Field of view | Standard or wide-angle | Wide lenses keep hands in frame during reaching, but add distortion that must be calibrated. |
| Frame rate | 30 fps or 60 fps and above | Fast manipulation, like knot-tying or tool use, needs higher frame rates to avoid motion blur. |
| Resolution | 1080p to 4K | Higher resolution helps with small objects and text, but raises storage and annotation cost. |
| Extra sensors | Depth, IMU, eye tracking, audio | Depth gives 3D hand and object positions. IMU records head and body motion. Eye tracking shows where attention goes. Audio supports narration. |
| Paired views | Egocentric only, or ego plus exocentric | Adding external cameras gives a stable global frame and helps models generalise across viewpoints. |
Sensor sync is where many programmes fail. When depth, IMU and video streams drift out of time, the labels no longer line up with the action. Every multi-sensor rig needs a shared clock or timestamp alignment, plus a calibration routine run at the start of each session.
Aya Data’s capture teams configure rigs per project and handle sensor fusion end to end, from synchronising 2D camera feeds with 3D and LiDAR data to delivering aligned, labelled streams. For more on depth sensing, see our Ultimate Guide to LiDAR.
How should first-person recording tasks be designed?
A task script turns a vague goal like “kitchen activities” into a repeatable recording plan. Without one, contributors record whatever is convenient, and the dataset ends up full of near-duplicates.
A good task script defines:
- The activity and its steps, such as “make tea: fill kettle, boil, add bag, pour, remove bag.”
- Start and stop points, so every clip begins and ends at a consistent moment.
- Required variations, such as left-and right-handed execution, different tools, or interrupted tasks.
- Target volume, set in hours or episodes per task, environment and contributor.
- Failure and recovery cases, like dropping an object and picking it up. Models learn a lot from mistakes.
Why diversity decides whether the model generalises
A model trained on one type of kitchen struggles in another. Real-world generalisation needs variety across:
- Environments: homes, workshops, farms, warehouses, clinics and outdoor sites.
- People: hand sizes, skin tones, ages, and handedness.
- Objects and tools: local brands, materials and layouts, not just one standard set.
- Conditions: daylight, low light, mixed lighting, clutter and noise.
This is where many datasets fall short. Much of today’s public first-person footage skews towards Western homes and labs. Aya Data runs field capture across Africa, Europe and Asia, giving clients access to environments, objects and working practices that most datasets miss. Our data acquisition teams have already captured field data in demanding conditions, from drone surveys over 6,000 hectares of palm plantations to on-the-ground agricultural monitoring.
What makes first-person footage unusable, and how do you prevent it?
Most first-person footage that fails does so for predictable reasons, and nearly all of them can be caught at capture time. Catching them in the field costs minutes. Catching them after annotation costs a re-shoot.
| Failure | Why it happens | How to prevent it |
| Hands out of frame | Camera angle too high, or a narrow lens | Test framing per contributor; use chest mounts or wider lenses for close work |
| Motion blur | Fast head movement, low frame rate, dim light | Raise frame rate, add lighting, coach contributors on steady movement |
| Occluded objects | Hands or body block the action | Add an exocentric camera for occlusion-heavy tasks |
| Sensor drift | Unsynchronised clocks between streams | Calibrate and verify sync at the start of every session |
| Missing metadata | Task ID, location or device not logged | Auto-tag clips at capture with a structured metadata schema |
| Off-script recording | Unclear instructions | Short training plus a reference video for each task |
A three-gate quality process
Aya Data runs quality control in three gates, so problems stop early:
- On-site checks. Field leads review sample clips during each session for framing, blur and audio.
- Ingest validation. Automated checks flag corrupt files, sync drift, missing metadata and clips outside length limits.
- Pre-annotation review. A reviewer confirms each clip matches its task script before it enters annotation.
The same discipline carries into annotation, where we measure inter-annotator agreement on every delivery. If you’re evaluating vendors, our guide to data annotation SLAs to ask your vendor lists the quality guarantees worth putting in writing.
How do you handle consent and privacy in first-person video?
A wearable camera records everything in its view, not just the task. That makes consent and privacy part of the capture design, not a legal step added afterwards.
A sound first-person collection programme covers four areas:
- Informed participant consent. Contributors know what is recorded, how footage will be used, who can access it and how long it is kept.
- Bystander protection. Recording avoids people who have not consented, and faces, screens and documents are blurred where they appear.
- Location controls. Clear rules on when recording must stop, such as in bathrooms or near sensitive paperwork.
- Secure handling. Encrypted transfer and storage, role-based access, and deletion on schedule.
Buyers increasingly ask vendors to prove these controls, because poorly consented footage creates legal and reputational risk for the company that trains on it. Aya Data operates under ISO, GDPR, HIPAA and SOC aligned processes. We’ve set out our full approach in Egocentric Data Collection: Protocol, Consent, Privacy.
What annotation makes first-person video trainable?
Object detection tells a model what is in the frame. Physical AI also needs to know what is happening, in what order, and why. First-person annotation therefore layers several label types onto the same footage:
| Annotation layer | What it captures | Used for |
| Action segmentation | Start and end of each step in a task | Task planning, step recognition |
| Hand-object interaction | Which hand touches which object, and how (grasp type, contact) | Manipulation and grasp learning |
| Hand pose and keypoints | Finger and wrist positions frame by frame | Dexterous robot hands, gesture models |
| Object tracking | Objects followed across frames despite camera motion | Scene understanding, state tracking |
| Natural-language narration | Plain descriptions of each action, at one or more levels of detail | VLA and video-language models |
| 3D labels | Cuboids and segmentation on depth or LiDAR data | Spatial reasoning, navigation |
Consistency matters more than volume. Two annotators often disagree on exactly when “pick up the cup” starts. Clear guidelines, worked examples and measured inter-annotator agreement turn those judgement calls into consistent labels.
Aya Data’s video annotation teams handle frame-by-frame temporal labelling, hand-object interaction and multi-level narration. Our human-in-the-loop services add expert review where accuracy is critical. We’ve applied this at scale, including annotating 5,000 hours of surveillance footage for action detection and labelling complex pedestrian scenes for Glidance’s navigation aid for blind and low-vision users. To see where AI-assisted labelling still needs people, read How AI Image Labeling Works.
What should you ask a first-person video data vendor?
The right vendor can show capture capability, not just annotation capacity. Use these ten questions to compare providers:
- Capture or annotation only? Can they plan and run field capture, or do they only label footage you supply?
- Rig flexibility. Which mounts, cameras and sensors can they deploy, and how do they calibrate and sync them?
- Environment and contributor diversity. How many locations, countries and contributor profiles can they cover?
- Throughput. How many hours of usable footage can they deliver per week, and how fast can they scale?
- Usable-footage rate. What share of raw footage passes quality checks, and how is it measured?
- Consent and privacy. Can they show consent forms, bystander policies, anonymisation steps and retention rules?
- Annotation depth. Do they support action segmentation, hand-object interaction, narration and 3D labels on the same footage?
- Quality metrics. Do they measure inter-annotator agreement on every delivery, and will they commit to it in an SLA?
- Security and compliance. Which certifications do they hold, and can they share audit reports?
- Pilot terms. Will they run a small pilot to validate the protocol before full commitment?
A vendor who answers all ten with specifics, and backs them with real projects they have delivered on, is ready for a production programme.
Why physical AI teams choose Aya Data for first-person video
Aya Data delivers the full first-person data pipeline in one engagement: capture in the field, quality control, annotation to specification, and delivery in training-ready formats. That is the core of our physical AI data services.
- Capture and annotation under one roof. Our teams plan protocols, configure rigs, run field sessions and label the results, so nothing is lost between vendors.
- Diversity most datasets lack. Operations across Africa, Europe and Asia give access to environments, objects, people and working practices underrepresented in public data.
- Sensor fusion expertise. We synchronise first-person video with depth, LiDAR and external cameras for robotics and embodied AI programmes.
- Measured quality. Inter-annotator agreement is measured on every delivery, backed by ISO, GDPR, HIPAA and SOC aligned processes.
- Proven at scale. Clients and partners including Labelbox, V7, Glidance and Cydar Medical trust us with complex, high-precision data. Browse our case studies.
- Built for the next stage. The same teams support RLHF and model evaluation once your first models are trained.
Start with a free pilot
The fastest way to de-risk a first-person data programme is to test the protocol small. Aya Data offers a free pilot to validate your capture setup, task scripts and annotation guidelines before full collection.
Ready to build your first-person video dataset? Talk to our physical AI team about your tasks, environments and timelines.
Frequently Asked Questions
What is first-person video data for AI?
First-person video data is footage recorded from the viewpoint of the person performing a task, using a head, chest or wrist-mounted camera. It teaches robotics, embodied AI and AR models how hands, tools and objects interact, from the same perspective a robot or wearable device has in deployment.
Is first-person video the same as egocentric video?
Yes. “First-person video,” “egocentric video” and “POV video” describe the same thing: footage captured from the actor’s own viewpoint. “Egocentric” is the common term in research, while “first-person” and “POV” are more common in industry.
How much first-person video does a robotics model need?
It depends on the task’s complexity and variety. Narrow tasks in one environment may need tens of hours. General-purpose manipulation models are trained for thousands of hours across many people and places. A pilot helps estimate the volume your use case needs.
Can’t we just use public egocentric datasets?
Public datasets like Ego4D and Ego-Exo4D are useful for research and pre-training. But they rarely cover your specific tasks, environments and edge cases, and their licences may restrict commercial use. Most production teams combine public data with custom-captured footage.
What annotation does first-person video need?
Common layers include action segmentation, hand-object interaction, hand pose keypoints, object tracking, natural-language narration and 3D labels on depth data. The right mix depends on whether the model is learning recognition, manipulation or planning.
How does Aya Data capture first-person video?
Aya Data designs the protocol and task scripts, configures and calibrates capture rigs, runs consented field sessions across Africa, Europe and Asia, checks quality at each stage, and annotates footage to your specification. Programmes typically start with a free pilot.
Written by

Head, Data Annotation
Akhil Singh is the Head of Business Unit, Annotation, at Aya Data and a seasoned AI and GTM leader. With over a decade of experience scaling B2B sales and operations across AI, data, and SaaS, he combines commercial discipline with deep domain expertise to drive growth across computer vision, GenAI, and Medical AI
