Back to Details

hz-unity-passthrough-camera-access

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Meta Quest Passthrough Camera Access (PCA)

Meta Quest Passthrough Camera Access (PCA)

Building on top of the Android Camera2 API, Passthrough Camera Access provides access to the forward-facing RGB cameras on Quest 3 and Quest 3S for the purpose of supporting Computer Vision and Machine Learning. This API is distinct from the Media Projection API, which supports casting from the user's POV (including the UI) and should be used if your purpose is to represent what the user is seeing. Use the Passthrough Camera API to add application-specific computer-vision capabilities that extend the understanding of the user's environment and actions beyond what is provided by the Quest Scene API.
基于Android Camera2 API，透视摄像头访问（PCA）可让开发者访问Quest 3和Quest 3S上的前置RGB摄像头，以支持计算机视觉和机器学习任务。本API与Media Projection API不同，后者支持捕获用户视角（包括UI），如果你的需求是呈现用户所见内容，应使用该API。而透视摄像头API则用于添加特定于应用的计算机视觉功能，以扩展对用户环境和行为的理解，这是Quest Scene API所无法提供的。

Use Cases

使用场景

This API provides an unobstructed view using the forward-facing RGB cameras and can be integrated with ML/CV pipelines. Common use cases:
  1. Specially-trained ML/CV models that identify specific objects the application can interact with (e.g. fitness equipment like dumbbells, industrial equipment like audio mixers).
  2. ML/CV assistants or guides for experiences. For instance, a museum tour where looking at a painting surfaces information about its history, artist, and style using off-device LLMs.
  3. Feedback for training or special-interest applications — after asking the user to perform a task, the app can detect whether it was done correctly by interpreting changes to the environment (e.g. writing on a whiteboard).
  4. Design improvements. By interpreting the lighting or using the camera image to modify a texture, much more realistic design effects can be achieved.
本API通过前置RGB摄像头提供无遮挡视图，可与ML/CV管道集成。常见使用场景包括：
  1. 经过专门训练的ML/CV模型，识别应用可交互的特定对象（如哑铃等健身器材、调音台等工业设备）。
  2. 用于体验场景的ML/CV助手或向导。例如，在博物馆导览中，当用户看向一幅画作时，通过离线LLM展示其历史、艺术家和风格相关信息。
  3. 培训或特殊兴趣应用的反馈机制——在要求用户完成某项任务后，应用可通过解读环境变化（如在白板上书写）来检测任务是否正确完成。
  4. 设计效果优化。通过解读光线或使用摄像头图像修改纹理，可实现更逼真的设计效果。

General Prerequisites

通用前提条件

  1. Horizon OS v74 or later.
  2. Quest 3 or Quest 3S (Quest Pro and earlier are not supported).
  3. Either permission 
    android.permission.CAMERA
    or 
    horizonos.permission.HEADSET_CAMERA
    . 
    CAMERA
    grants access to both the passthrough and avatar cameras; 
    HEADSET_CAMERA
    grants access only to the passthrough camera.
  4. The Passthrough feature must be enabled to access the Passthrough Camera API.
  1. Horizon OS v74或更高版本。
  2. Quest 3或Quest 3S（不支持Quest Pro及更早机型）。
  3. 需具备
    android.permission.CAMERA
    或
    horizonos.permission.HEADSET_CAMERA
    权限。
    CAMERA
    权限可同时访问透视摄像头和虚拟形象摄像头；
    HEADSET_CAMERA
    权限仅允许访问透视摄像头。
  4. 必须启用透视功能才能访问透视摄像头API。

Passthrough Camera Using Android Camera2

使用Android Camera2实现透视摄像头访问

Camera Access on Quest is implemented on top of Android's Camera2 API within Horizon OS. Starting from Horizon OS v74, Camera2 and its Unity extension are available on Quest headsets (Horizon OS v83 added an Unreal extension). On Quest 3 and Quest 3S developers have access to the left and right cameras on the face of the HMD.
The Android Camera2 API provides:
  • Image capture: capture camera data for advanced processing.
  • Camera metadata: query the API for hardware capabilities and configuration information.
  • Multi-camera support: access and control multiple cameras.
Quest设备上的摄像头访问功能基于Horizon OS中的Android Camera2 API实现。从Horizon OS v74开始，Quest头显支持Camera2及其Unity扩展（Horizon OS v83新增了Unreal扩展）。在Quest 3和Quest 3S上，开发者可访问头显正面的左右摄像头。
Android Camera2 API提供以下功能：
  • 图像捕获：捕获摄像头数据以进行高级处理。
  • 摄像头元数据：查询API获取硬件能力和配置信息。
  • 多摄像头支持：访问和控制多个摄像头。

Known Issues

已知问题

  1. After the first install of an app using the Camera API in v74 after flashing a new OS, the RGB channels might flip causing color distortion (e.g. flesh tones appear bluish). Reboot the device to fix.
  2. Passthrough Camera API is not supported in XR Simulator.
  3. The passthrough camera texture captures a rectangular area (1280×960) smaller than what the user sees. The 1280×1280 resolution added in v83 expands the vertical field of view, but still does not cover the entire passthrough view.
  4. Restrictions applied by a parent to restrict Teen and Youth accounts (e.g. accessing the Passthrough Camera API) are not applied if the application is installed through the Meta Quest Developer Hub.
  1. 在刷入新系统后，首次安装使用v74版本Camera API的应用时，RGB通道可能会翻转，导致颜色失真（如肤色呈现蓝色调）。重启设备即可修复。
  2. 透视摄像头API不支持XR模拟器。
  3. 透视摄像头纹理捕获的是一个矩形区域（1280×960），小于用户实际视野。v83版本新增的1280×1280分辨率扩大了垂直视野，但仍无法覆盖整个透视视图。
  4. 如果应用是通过Meta Quest开发者中心安装的，家长针对青少年账户设置的限制（如访问透视摄像头API）将不会生效。

Best Practices

最佳实践

  1. Ensure user privacy. Camera image data is considered Device User Data and is covered by the Developer Data Use Policy. Adhere to this policy fully.
  2. Avoid costly processing on device. On-device processing of camera images can enable compelling use cases, but keep experiences comfortable by maintaining a high framerate.
  1. 保障用户隐私：摄像头图像数据属于设备用户数据，受开发者数据使用政策约束。请严格遵守该政策。
  2. 避免设备端高成本处理：设备端摄像头图像处理可实现引人注目的使用场景，但需保持高帧率以确保体验流畅舒适。

References

参考资料

Detailed, task-specific guidance for projecting and placing PCA imagery in world space lives under references/:
  • references/principal-point-offset.md — Correctly project the camera image onto a flat world-space surface (frustum-slice quad, FOV visualizer, image-plane overlay) using the optical axis as the plane normal. Read this when corner rays from 
    ViewportPointToRay
    produce a non-rectangular quad, a camera-aligned quad drifts off-center, or 
    2 * d * tan(fov / 2)
    gives the wrong size on a device whose camera principal point is offset even though it worked on Quest 3.
  • references/detection-bounding-boxes.md — Draw precise 3D bounding boxes around real-world objects detected by an ML model. Read this when placing 2D ML detections (YOLO, SSD, DETR, custom CNN) into world space, building a camera-facing billboard quad from a 2D detection, or wiring 
    PassthroughCameraAccess
    + 
    EnvironmentRaycastManager
    for object localization.
关于在世界空间中投影和放置PCA图像的详细任务指导，请参阅references/目录：
  • references/principal-point-offset.md —— 使用光轴作为平面法线，将摄像头图像正确投影到平面世界空间表面（视锥体切片四边形、FOV可视化器、图像平面叠加层）。当
    ViewportPointToRay
    生成的边角射线产生非矩形四边形、摄像头对齐的四边形偏离中心，或者
    2 * d * tan(fov / 2)
    在主点偏移的设备上计算尺寸错误（尽管在Quest 3上正常工作）时，请阅读本文档。
  • references/detection-bounding-boxes.md —— 在ML模型检测到的现实世界对象周围绘制精确的3D边界框。当你需要将2D ML检测结果（YOLO、SSD、DETR、自定义CNN）转换到世界空间、从2D检测结果构建面向摄像头的广告牌四边形，或是将
    PassthroughCameraAccess
    与
    EnvironmentRaycastManager
    结合用于对象定位时，请阅读本文档。