hz-unity-face-tracking
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseUnity Face Tracking for ARKit-Rigged Models (Meta Movement SDK)
适用于ARKit绑定模型的Unity面部追踪方案(Meta Movement SDK)
End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Quest. Uses the public extension hook — no , ships to 3P.
OVRCustomFaceOVR_INTERNAL_CODEThe model's blendshape names must follow the ARKit naming convention (camelCase, / suffixes, e.g. , , ). An optional prefix like is automatically stripped.
_L_ReyeBlink_LjawOpenmouthSmile_RblendShape2.eyeBlink_L本教程提供端到端方案,让采用标准52个ARKit blendshape绑定的头部/面部模型,能够根据Quest佩戴者的面部表情实现动画。使用公开的扩展钩子——无需,可交付给第三方。
OVRCustomFaceOVR_INTERNAL_CODE模型的blendshape名称必须遵循ARKit命名规范(驼峰式命名,后缀为/,例如、、)。类似的可选前缀会被自动去除。
_L_ReyeBlink_LjawOpenmouthSmile_RblendShape2.eyeBlink_LWhen to use
适用场景
- User has an FBX/GLB/mesh with ARKit-named blendshapes and wants it driven by Quest face tracking.
- User asks: "animate this head with my face", "drive these blendshapes from face tracking", "use Movement SDK A2E with my model", "wire ARKit shapes to Quest".
- Target device: Quest Pro, Quest 3, Quest 3S (Quest 2 is no-op — no face cameras).
- 用户拥有带有ARKit命名blendshape的FBX/GLB/网格模型,希望通过Quest面部追踪驱动其动画。
- 用户提出如下需求:“用我的面部表情驱动这个头部模型”、“通过面部追踪驱动这些blendshape”、“将Movement SDK A2E与我的模型结合使用”、“将ARKit形状与Quest关联”。
- 目标设备:Quest Pro、Quest 3、Quest 3S(Quest 2无面部摄像头,无法实现该功能)。
Prerequisites checklist
前置条件清单
- Quest face tracking-capable headset (Pro / 3 / 3S).
- Packages in :
Packages/manifest.json- (Meta XR Core — provides
com.meta.xr.sdk.core,OVRFaceExpressions)OVRCustomFace - (Meta Movement SDK — A2E + retargeting helpers)
com.meta.xr.sdk.movement
- (verify via Project Settings → Meta XR):
Assets/Oculus/OculusProjectConfig.asset- (Supported) or
faceTrackingSupport: 1(Required)2 - if the rig has gaze
eyeTrackingSupport: 1
- Android manifest () permissions:
Assets/Plugins/Android/AndroidManifest.xml<uses-feature android:name="oculus.software.face_tracking" android:required="false" /><uses-permission android:name="com.oculus.permission.FACE_TRACKING" />- (required for A2E)
<uses-permission android:name="android.permission.RECORD_AUDIO" /> - For eye gaze: +
oculus.software.eye_trackingcom.oculus.permission.EYE_TRACKING
- OVRCameraRig in the scene with including
OVRManager.FaceTrackingDataSources(A2E) — if you skip Audio, mouth motion is visual-only.Audio
After any change to OculusProjectConfig, call to regenerate the manifest.
meta_update_android_manifest- 支持面部追踪的Quest头显(Pro/3/3S)。
- 中的依赖包:
Packages/manifest.json- (Meta XR Core——提供
com.meta.xr.sdk.core、OVRFaceExpressions)OVRCustomFace - (Meta Movement SDK——包含A2E和重定向工具)
com.meta.xr.sdk.movement
- (可通过Project Settings → Meta XR验证):
Assets/Oculus/OculusProjectConfig.asset- (支持)或
faceTrackingSupport: 1(必需)2 - 如果模型包含视线追踪,需设置
eyeTrackingSupport: 1
- Android清单文件()权限:
Assets/Plugins/Android/AndroidManifest.xml<uses-feature android:name="oculus.software.face_tracking" android:required="false" /><uses-permission android:name="com.oculus.permission.FACE_TRACKING" />- (A2E必需)
<uses-permission android:name="android.permission.RECORD_AUDIO" /> - 视线追踪需添加:+
oculus.software.eye_trackingcom.oculus.permission.EYE_TRACKING
- 场景中的OVRCameraRig需在中包含
OVRManager.FaceTrackingDataSources(A2E)——若跳过Audio,嘴部动作仅为视觉驱动。Audio
修改OculusProjectConfig后,调用重新生成清单文件。
meta_update_android_manifestApproach (high level)
整体流程(概览)
- Drop the script (in
ARKitOVRCustomFace) into the project.references/ARKitOVRCustomFace.cs - Add an component to the OVRCameraRig (or anywhere in the scene).
OVRFaceExpressions - On the GameObject that has the model's , add
SkinnedMeshRenderer. Adding the component triggersARKitOVRCustomFace, which auto-populates the blendshape→FaceExpression mapping by scanning the mesh's blendshape names.Reset() - Wire the component's field to the
FaceExpressionsinstance.OVRFaceExpressions - For eye gaze: drop in (see Eye tracking below).
ARKitEyeGazeBlendshapeDriver - Press Play with Meta XR Link, or build APK and side-load. On first launch, accept the Face Tracking / Eye Tracking / Microphone permission prompts.
That's the whole flow. Details below.
- 将****脚本(位于
ARKitOVRCustomFace)导入项目。references/ARKitOVRCustomFace.cs - 为OVRCameraRig(或场景中任意对象)添加组件。
OVRFaceExpressions - 在带有模型的GameObject上添加
SkinnedMeshRenderer组件。添加组件时会触发ARKitOVRCustomFace方法,通过扫描网格的blendshape名称自动填充blendshape→FaceExpression映射关系。Reset() - 将组件的字段关联到
FaceExpressions实例。OVRFaceExpressions - 如需视线追踪:导入****(详见下方视线追踪部分)。
ARKitEyeGazeBlendshapeDriver - 通过Meta XR Link运行Play模式,或构建APK并侧载到设备。首次启动时,接受面部追踪/视线追踪/麦克风权限请求。
以上就是完整流程,详细说明如下。
Eye tracking (DO THIS — face expressions alone don't work)
视线追踪(必须配置——仅面部表情无法实现完整功能)
Don't rely on for eye gaze. Those fields are derived from face-camera visuals, not the dedicated eye tracker. On Quest Pro they're often zero or noisy even when face tracking is otherwise working. The right API is , which taps the eye tracker directly.
OVRFaceExpressions.EyesLook*OVREyeGazeFor ARKit rigs with blendshapes (no eye bones), this skill ships :
eyeLook*ARKitEyeGazeBlendshapeDriver- Reads gaze rotation from two components (one per eye,
OVREyeGaze).TrackingMode = HeadSpace - Computes rotation relative to a head reference (e.g. ).
CenterEyeAnchor - Decomposes pitch → , yaw →
eyeLookUp/Down_{L,R}(with the ARKit "_In = toward nose" convention).eyeLookIn/Out_{L,R} - Writes weights in , so it overrides whatever
LateUpdatewrote inARKitOVRCustomFace.Update
For rigs with eye bones (no blendshapes), skip the blendshape decomposition and just parent to each eye bone with — the component will rotate the bone directly.
eyeLook*OVREyeGazeApplyRotation = true请勿依赖实现视线追踪。这些字段由面部摄像头视觉数据推导而来,而非专用的眼球追踪器。在Quest Pro上,即便面部追踪正常工作,这些字段的值也常为0或存在噪声。正确的API是,它直接调用眼球追踪器。
OVRFaceExpressions.EyesLook*OVREyeGaze对于带有 blendshape(无眼球骨骼)的ARKit绑定模型,本方案提供****:
eyeLook*ARKitEyeGazeBlendshapeDriver- 从两个组件(双眼各一个,
OVREyeGaze)读取视线旋转数据。TrackingMode = HeadSpace - 计算相对于头部参考对象(例如)的旋转角度。
CenterEyeAnchor - 将旋转分解为俯仰角→,偏航角→
eyeLookUp/Down_{L,R}(遵循ARKit“_In = 朝向鼻子”的约定)。eyeLookIn/Out_{L,R} - 在中写入权重,覆盖
LateUpdate在ARKitOVRCustomFace中写入的值。Update
对于带有眼球骨骼(无 blendshape)的模型,无需分解blendshape,只需将作为子对象添加到每个眼球骨骼,并设置——组件会直接旋转骨骼。
eyeLook*OVREyeGazeApplyRotation = trueEye gaze setup (blendshape rigs)
视线追踪设置(Blendshape绑定模型)
- Create two empty GameObjects under (or your head transform):
CenterEyeAnchor,LeftEyeGaze.RightEyeGaze - Add to each. Set
OVREyeGaze/Eye = Left,Right,TrackingMode = HeadSpace,ApplyPosition = false,ApplyRotation = true.ConfidenceThreshold = 0.5 - Add to the head's SkinnedMeshRenderer GameObject (alongside
ARKitEyeGazeBlendshapeDriver). WireARKitOVRCustomFace,leftEye, andrightEye(=referenceFrame).CenterEyeAnchor - Tweak (default 30°) and
maxAngleDeg(default 0.4) to taste.smoothing
- 在(或你的头部变换对象)下创建两个空GameObject:
CenterEyeAnchor、LeftEyeGaze。RightEyeGaze - 为每个对象添加组件。设置
OVREyeGaze/Eye = Left,Right,TrackingMode = HeadSpace,ApplyPosition = false,ApplyRotation = true。ConfidenceThreshold = 0.5 - 为头部的SkinnedMeshRenderer GameObject添加组件(与
ARKitEyeGazeBlendshapeDriver共存)。关联ARKitOVRCustomFace、leftEye和rightEye(即referenceFrame)。CenterEyeAnchor - 根据需求调整(默认30°)和
maxAngleDeg(默认0.4)参数。smoothing
Step-by-step
分步指南
1. Install the script
1. 安装脚本
Copy into . It defines the public, 3P-shippable ARKit ↔ OVR FaceExpression table and a method that scans , strips any prefix before the last , lowercases, and matches against the table. Unmatched mesh blendshapes are set to (sentinel — skipped at runtime).
references/ARKitOVRCustomFace.csAssets/Scripts/ARKitOVRCustomFace.csMapBlendshapes()SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i).OVRFaceExpressions.FaceExpression.Max将复制到。该脚本定义了公开的、可交付给第三方的ARKit ↔ OVR FaceExpression映射表,以及方法——该方法会扫描,去除最后一个之前的所有前缀,转换为小写后与映射表匹配。未匹配的网格blendshape会被设置为(标记值——运行时会跳过)。
references/ARKitOVRCustomFace.csAssets/Scripts/ARKitOVRCustomFace.csMapBlendshapes()SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i).OVRFaceExpressions.FaceExpression.Max2. Enable Movement SDK + permissions
2. 启用Movement SDK及权限
Project settings:
Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported (if needed)Then regenerate manifest:
- Unity MCP:
meta_update_android_manifest - Or Editor menu: Meta → Tools → Update AndroidManifest.xml
项目设置:
Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported(如需视线追踪)然后重新生成清单文件:
- Unity MCP:执行
meta_update_android_manifest - 或通过编辑器菜单:Meta → Tools → Update AndroidManifest.xml
3. Scene setup
3. 场景设置
Scene Hierarchy
├── OVRCameraRig (from meta_add_camerarig)
│ └── (add) OVRFaceExpressions component
└── YourHeadModel
└── ...SkinnedMeshRenderer GO...
├── SkinnedMeshRenderer (existing)
└── (add) ARKitOVRCustomFace component
└── FaceExpressions = the OVRFaceExpressions ref
└── retargetingType = Custom (set automatically by base when overriding)
└── Mappings[] = auto-filled on Reset()
└── BlendShapeStrengthMultiplier = 100 (default; OVR weights are 0–1, mesh wants 0–100)If the head has multiple s (e.g. separate teeth/tongue meshes), add to each one.
SkinnedMeshRendererARKitOVRCustomFace场景层级
├── OVRCameraRig (来自meta_add_camerarig)
│ └── (添加) OVRFaceExpressions 组件
└── YourHeadModel
└── ...带有SkinnedMeshRenderer的GameObject...
├── SkinnedMeshRenderer (已存在)
└── (添加) ARKitOVRCustomFace 组件
└── FaceExpressions = 关联OVRFaceExpressions实例
└── retargetingType = Custom (重写时由基类自动设置)
└── Mappings[] = 重置时自动填充
└── BlendShapeStrengthMultiplier = 100 (默认值;OVR权重范围0–1,网格权重范围0–100)如果头部模型包含多个(例如牙齿/舌头为独立网格),需为每个添加组件。
SkinnedMeshRendererSkinnedMeshRendererARKitOVRCustomFace4. OVRManager — enable Audio as a data source (A2E)
4. OVRManager——启用Audio作为数据源(A2E)
On 's OVRManager component:
OVRCameraRig- Face Tracking Data Sources → check Visual AND Audio (Audio = A2E; produces mouth shapes from microphone when the visual face cameras can't see something — talking, occlusion, etc.).
在的OVRManager组件中:
OVRCameraRig- Face Tracking Data Sources → 勾选Visual和Audio(Audio即A2E;当面部摄像头无法捕捉到动作时——如说话、遮挡等,通过麦克风生成嘴部形状)。
5. Trigger the mapping if the component already existed
5. 若组件已存在,触发映射操作
MapBlendshapes()Reset()OnValidate()Mappingscsharp
go.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();From Unity MCP , the cleanest invocation (the OVR/MSDK types aren't referenced in the MCP dynamic assembly, so use SendMessage to avoid reflection):
Unity_RunCommandcsharp
GameObject.Find("YourHeadModel")
.GetComponentInChildren<SkinnedMeshRenderer>().gameObject
.SendMessage("MapBlendshapes", SendMessageOptions.RequireReceiver);MapBlendshapes()Reset()MappingsOnValidate()csharp
go.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();在Unity MCP 中,最简洁的调用方式(OVR/MSDK类型未在MCP动态程序集中引用,因此使用SendMessage避免反射):
Unity_RunCommandcsharp
GameObject.Find("YourHeadModel")
.GetComponentInChildren<SkinnedMeshRenderer>().gameObject
.SendMessage("MapBlendshapes", SendMessageOptions.RequireReceiver);6. Verify
6. 验证
- In Editor, with Meta XR Link / Quest Link, enter Play mode and make faces. The model should mirror them.
- On device, build APK, side-load, grant Face Tracking + Microphone permissions on first launch.
- Check the mapping at edit time: inspect the component.
ARKitOVRCustomFaceshould equalMappings.Length. The Console log fromSkinnedMeshRenderer.sharedMesh.blendShapeCountreportsMapBlendshapes()— X should be 50 (or 52 if the FBX has all of them).mapped X/N blendshapes
- 编辑器中:通过Meta XR Link / Quest Link进入Play模式,做出面部表情,模型应同步镜像表情。
- 设备上:构建APK并侧载,首次启动时授予面部追踪+麦克风权限。
- 编辑时检查映射:查看组件。
ARKitOVRCustomFace应等于Mappings.Length。SkinnedMeshRenderer.sharedMesh.blendShapeCount的控制台日志会显示MapBlendshapes()——X应为50(若FBX包含全部52个则为52)。mapped X/N blendshapes
Troubleshooting
故障排查
| Symptom | Cause / fix |
|---|---|
| Mesh blendshape names don't follow ARKit convention. Verify with |
| Face is frozen | |
| Mouth doesn't move when speaking | A2E disabled. Enable Audio under |
| Eyes don't blink | Mesh's eyelid shapes aren't named |
| Eyes don't move (look around) | You're relying on |
| Eye gaze in wrong direction | Note the deliberate ARKit ↔ OVR swap: ARKit's |
| Weights look half-strength or clamped | |
| Base method is |
| Mesh changed since mapping was generated. Re-run |
| 症状 | 原因/修复方案 |
|---|---|
控制台显示 | 网格blendshape名称不符合ARKit规范。通过 |
| 面部模型静止不动 | 未关联 |
| 说话时嘴部无动作 | A2E未启用。在 |
| 眼睛不眨眼 | 模型的眼睑blendshape名称不是 |
| 眼睛无法转动(环顾四周) | 依赖 |
| 视线方向错误 | 注意ARKit与OVR的语义故意互换:ARKit的 |
| 权重强度减半或被限制 | |
编译错误 | 基类方法在其他程序集中为 |
启动时断言 | 映射生成后网格已更改。通过组件上下文菜单重新运行 |
ARKit ↔ OVR FaceExpression mapping (reference)
ARKit ↔ OVR FaceExpression映射参考
The 52 ARKit shapes don't 1:1 a FACS-based OVR enum. Notable choices baked into :
ARKitTable- Centrally-named ARKit shapes that have L+R OVR pairs (,
browInnerUp,cheekPuff,mouthFunnel,mouthPucker,mouthRollLower) pick only the L-side OVR expression. If your model has obviously asymmetric mouth/brow when the user uses these expressions and you want true symmetric drive, subclass and additively sum L+R in a custommouthRollUpper(useMapBlendshapes-style per-blendshape sum). For most heads the L-only choice is fine because the face is roughly symmetric and the asymmetry is below visual threshold.SlothARKitFaceDriver - are deliberately swapped per side relative to OVR's left/right semantics (see Troubleshooting).
eyeLookIn/Out - →
mouthClose(closest FACS analogue).LipsToward - →
tongueOut(requires HorizonOS ≥ 65 + tongue tracking; otherwise stays at 0).TongueOut
52个ARKit形状与基于FACS的OVR枚举并非一一对应。中包含以下关键映射规则:
ARKitTable- ARKit的中央命名形状对应OVR的左右成对形状(、
browInnerUp、cheekPuff、mouthFunnel、mouthPucker、mouthRollLower)仅选择左侧OVR表情。若用户做出这些表情时模型的嘴部/眉毛明显不对称,且需要真正的对称驱动,可创建子类并在自定义mouthRollUpper中累加左右侧值(参考MapBlendshapes的逐blendshape累加方式)。对于大多数头部模型,仅使用左侧的选择已足够,因为面部大致对称,不对称性低于视觉阈值。SlothARKitFaceDriver - ****的左右侧语义与OVR故意互换(详见故障排查)。
eyeLookIn/Out - →
mouthClose(最接近的FACS对应项)。LipsToward - →
tongueOut(要求HorizonOS ≥ 65 + 舌头追踪;否则保持为0)。TongueOut
Files in this skill
本方案包含的文件
- — this file
SKILL.md - — drop-in
references/ARKitOVRCustomFace.cssubclassOVRCustomFace - — OVREyeGaze →
references/ARKitEyeGazeBlendshapeDriver.csblendshape driver (LateUpdate)eyeLook*
- — 本文档
SKILL.md - — 可直接导入的
references/ARKitOVRCustomFace.cs子类OVRCustomFace - — OVREyeGaze →
references/ARKitEyeGazeBlendshapeDriver.csblendshape驱动脚本(LateUpdate)eyeLook*
Why this approach (vs. alternatives)
本方案的优势(对比其他方案)
- Custom driver that reads
MonoBehaviourand writes blendshape weights directly: works, but doesn't integrate with MSDK's correctives, eye constraints, or future retargeting upgrades. Use only if you can't extendOVRFaceExpressions[expr].OVRCustomFace - +
OVRCustomFace: clean, but guarded behindRetargetingType.ARKitBlendshapesin the public Oculus Integration package — not 3P-shippable.#if OVR_INTERNAL_CODE - +
OVRCustomFace+RetargetingType.Customoverride (this skill): fully public API, shippable, and gets all base-class behavior (data validity gating, weight scaling, mesh assertion).GetCustomBlendShapeNameAndExpressionPairs
- 自定义MonoBehaviour驱动脚本:读取并直接写入blendshape权重——可行,但无法集成MSDK的修正、眼球约束或未来的重定向升级。仅当无法扩展
OVRFaceExpressions[expr]时使用。OVRCustomFace - +
OVRCustomFace:简洁,但在公开Oculus Integration包中被RetargetingType.ARKitBlendshapes保护——无法交付给第三方。#if OVR_INTERNAL_CODE - +
OVRCustomFace+ 重写RetargetingType.Custom(本方案):完全基于公开API,可交付,且具备所有基类行为(数据有效性校验、权重缩放、网格断言)。GetCustomBlendShapeNameAndExpressionPairs