Unity Face Tracking for ARKit-Rigged Models (Meta Movement SDK)

End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Quest. Uses the public

OVRCustomFace

extension hook — no

OVR_INTERNAL_CODE

, ships to 3P.

The model's blendshape names must follow the ARKit naming convention (camelCase,

_L

_R

suffixes, e.g.

eyeBlink_L

jawOpen

mouthSmile_R

). An optional prefix like

blendShape2.eyeBlink_L

is automatically stripped.

When to use

User has an FBX/GLB/mesh with ARKit-named blendshapes and wants it driven by Quest face tracking.
User asks: "animate this head with my face", "drive these blendshapes from face tracking", "use Movement SDK A2E with my model", "wire ARKit shapes to Quest".
Target device: Quest Pro, Quest 3, Quest 3S (Quest 2 is no-op — no face cameras).

Prerequisites checklist

Quest face tracking-capable headset (Pro / 3 / 3S).

Packages in

Packages/manifest.json

com.meta.xr.sdk.core

(Meta XR Core — provides

OVRFaceExpressions

OVRCustomFace

)

```
com.meta.xr.sdk.movement
```
(Meta Movement SDK — A2E + retargeting helpers)

Assets/Oculus/OculusProjectConfig.asset
(verify via Project Settings → Meta XR):
- ```
faceTrackingSupport: 1
```
  (Supported) or
```
2
```
  (Required)
- ```
eyeTrackingSupport: 1
```
  if the rig has gaze

Android manifest (

Assets/Plugins/Android/AndroidManifest.xml

) permissions:

<uses-feature android:name="oculus.software.face_tracking" android:required="false" />

<uses-permission android:name="com.oculus.permission.FACE_TRACKING" />

<uses-permission android:name="android.permission.RECORD_AUDIO" />

(required for A2E)

For eye gaze:

oculus.software.eye_tracking

com.oculus.permission.EYE_TRACKING

OVRCameraRig in the scene with
```
OVRManager.FaceTrackingDataSources
```
including
```
Audio
```
(A2E) — if you skip Audio, mouth motion is visual-only.

After any change to OculusProjectConfig, call

meta_update_android_manifest

to regenerate the manifest.

Approach (high level)

Drop the ARKitOVRCustomFace
script (in

references/ARKitOVRCustomFace.cs

) into the project.

Add an
```
OVRFaceExpressions
```
component to the OVRCameraRig (or anywhere in the scene).
On the GameObject that has the model's
```
SkinnedMeshRenderer
```
, add
```
ARKitOVRCustomFace
```
. Adding the component triggers
```
Reset()
```
, which auto-populates the blendshape→FaceExpression mapping by scanning the mesh's blendshape names.
Wire the component's
```
FaceExpressions
```
field to the
```
OVRFaceExpressions
```
instance.
For eye gaze: drop ARKitEyeGazeBlendshapeDriver
in (see Eye tracking below).
Press Play with Meta XR Link, or build APK and side-load. On first launch, accept the Face Tracking / Eye Tracking / Microphone permission prompts.

That's the whole flow. Details below.

Eye tracking (DO THIS — face expressions alone don't work)

Don't rely on
OVRFaceExpressions.EyesLook*
for eye gaze. Those fields are derived from face-camera visuals, not the dedicated eye tracker. On Quest Pro they're often zero or noisy even when face tracking is otherwise working. The right API is

OVREyeGaze

, which taps the eye tracker directly.

For ARKit rigs with

eyeLook*

blendshapes (no eye bones), this skill ships ARKitEyeGazeBlendshapeDriver
:

Reads gaze rotation from two
```
OVREyeGaze
```
components (one per eye,
```
TrackingMode = HeadSpace
```
).
Computes rotation relative to a head reference (e.g.
```
CenterEyeAnchor
```
).
Decomposes pitch →
```
eyeLookUp/Down_{L,R}
```
, yaw →
```
eyeLookIn/Out_{L,R}
```
(with the ARKit "_In = toward nose" convention).
Writes weights in
```
LateUpdate
```
, so it overrides whatever
```
ARKitOVRCustomFace
```
wrote in
```
Update
```
.

For rigs with eye bones (no

eyeLook*

blendshapes), skip the blendshape decomposition and just parent

OVREyeGaze

to each eye bone with

ApplyRotation = true

— the component will rotate the bone directly.

Eye gaze setup (blendshape rigs)

Create two empty GameObjects under
```
CenterEyeAnchor
```
(or your head transform):
```
LeftEyeGaze
```
,
```
RightEyeGaze
```
.

Add

OVREyeGaze

to each. Set

Eye = Left

Right

TrackingMode = HeadSpace

ApplyPosition = false

ApplyRotation = true

ConfidenceThreshold = 0.5

Add

ARKitEyeGazeBlendshapeDriver

to the head's SkinnedMeshRenderer GameObject (alongside

ARKitOVRCustomFace

). Wire

leftEye

rightEye

, and

referenceFrame

CenterEyeAnchor

Tweak
```
maxAngleDeg
```
(default 30°) and
```
smoothing
```
(default 0.4) to taste.

Step-by-step

1. Install the script

Copy

references/ARKitOVRCustomFace.cs

into

Assets/Scripts/ARKitOVRCustomFace.cs

. It defines the public, 3P-shippable ARKit ↔ OVR FaceExpression table and a

MapBlendshapes()

method that scans

SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i)

, strips any prefix before the last

, lowercases, and matches against the table. Unmatched mesh blendshapes are set to

OVRFaceExpressions.FaceExpression.Max

(sentinel — skipped at runtime).

2. Enable Movement SDK + permissions

Project settings:

Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported (if needed)

Then regenerate manifest:

Unity MCP:
```
meta_update_android_manifest
```
Or Editor menu: Meta → Tools → Update AndroidManifest.xml

3. Scene setup

Scene Hierarchy
├── OVRCameraRig                         (from meta_add_camerarig)
│   └── (add) OVRFaceExpressions         component
└── YourHeadModel
    └── ...SkinnedMeshRenderer GO...
        ├── SkinnedMeshRenderer          (existing)
        └── (add) ARKitOVRCustomFace     component
             └── FaceExpressions = the OVRFaceExpressions ref
             └── retargetingType = Custom (set automatically by base when overriding)
             └── Mappings[] = auto-filled on Reset()
             └── BlendShapeStrengthMultiplier = 100 (default; OVR weights are 0–1, mesh wants 0–100)

If the head has multiple

SkinnedMeshRenderer

s (e.g. separate teeth/tongue meshes), add

ARKitOVRCustomFace

to each one.

4. OVRManager — enable Audio as a data source (A2E)

OVRCameraRig

's OVRManager component:

Face Tracking Data Sources → check Visual AND Audio (Audio = A2E; produces mouth shapes from microphone when the visual face cameras can't see something — talking, occlusion, etc.).

5. Trigger the mapping if the component already existed

MapBlendshapes()

runs automatically when the component is added (

Reset()

) and from

OnValidate()

when

Mappings

is empty. If you need to remap manually (e.g. after re-importing the FBX), use the component's inspector context menu → Map Blendshapes, or from a script:

csharp

go.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();

From Unity MCP

Unity_RunCommand

, the cleanest invocation (the OVR/MSDK types aren't referenced in the MCP dynamic assembly, so use SendMessage to avoid reflection):

csharp

GameObject.Find("YourHeadModel")
    .GetComponentInChildren<SkinnedMeshRenderer>().gameObject
    .SendMessage("MapBlendshapes", SendMessageOptions.RequireReceiver);

6. Verify

In Editor, with Meta XR Link / Quest Link, enter Play mode and make faces. The model should mirror them.
On device, build APK, side-load, grant Face Tracking + Microphone permissions on first launch.
Check the mapping at edit time: inspect the
```
ARKitOVRCustomFace
```
component.
```
Mappings.Length
```
should equal
```
SkinnedMeshRenderer.sharedMesh.blendShapeCount
```
. The Console log from
```
MapBlendshapes()
```
reports
```
mapped X/N blendshapes
```
— X should be 50 (or 52 if the FBX has all of them).

Troubleshooting

Symptom	Cause / fix
`mapped 0/N` in Console	Mesh blendshape names don't follow ARKit convention. Verify with `mesh.GetBlendShapeName(i)` . Names must match e.g. `eyeBlink_L` , `jawOpen` — case-insensitive, prefix before last `.` is stripped.
Face is frozen	`OVRFaceExpressions` not assigned, or scene has no `OVRCameraRig` / `OVRManager` with face tracking enabled. Check `OVRFaceExpressions.FaceTrackingEnabled` and `ValidExpressions` at runtime.
Mouth doesn't move when speaking	A2E disabled. Enable Audio under `OVRManager → Face Tracking Data Sources` , and ensure `RECORD_AUDIO` permission is granted on device.
Eyes don't blink	Mesh's eyelid shapes aren't named `eyeBlink_L` / `_R` . Either rename, or add a custom row to `ARKitTable` .
Eyes don't move (look around)	You're relying on `OVRFaceExpressions.EyesLook*` instead of `OVREyeGaze` — switch to `ARKitEyeGazeBlendshapeDriver` .
Eye gaze in wrong direction	Note the deliberate ARKit ↔ OVR swap: ARKit's `eyeLookIn_L` (eye looking nose-ward, i.e. right) maps to OVR `EyesLookRightL` . The table already does this — don't "fix" it.
Weights look half-strength or clamped	`BlendShapeStrengthMultiplier` defaults to 100 because Unity blendshapes are 0–100 while OVR is 0–1. Don't lower this unless intentional.
`cannot change access modifiers when overriding` compile error	Base method is `protected internal` in another assembly. The override must use `protected` (not `protected internal` ) — already correct in the supplied script.
`_mappings out of sync with shared mesh` assertion at Start	Mesh changed since mapping was generated. Re-run `MapBlendshapes()` via the component context menu.

ARKit ↔ OVR FaceExpression mapping (reference)

The 52 ARKit shapes don't 1:1 a FACS-based OVR enum. Notable choices baked into

ARKitTable

Centrally-named ARKit shapes that have L+R OVR pairs (
```
browInnerUp
```
,
```
cheekPuff
```
,
```
mouthFunnel
```
,
```
mouthPucker
```
,
```
mouthRollLower
```
,
```
mouthRollUpper
```
) pick only the L-side OVR expression. If your model has obviously asymmetric mouth/brow when the user uses these expressions and you want true symmetric drive, subclass and additively sum L+R in a custom
```
MapBlendshapes
```
(use
```
SlothARKitFaceDriver
```
-style per-blendshape sum). For most heads the L-only choice is fine because the face is roughly symmetric and the asymmetry is below visual threshold.
eyeLookIn/Out
are deliberately swapped per side relative to OVR's left/right semantics (see Troubleshooting).
mouthClose
→
LipsToward
(closest FACS analogue).
tongueOut
→
TongueOut
(requires HorizonOS ≥ 65 + tongue tracking; otherwise stays at 0).

Files in this skill

```
SKILL.md
```
— this file

references/ARKitOVRCustomFace.cs

— drop-in

OVRCustomFace

subclass

references/ARKitEyeGazeBlendshapeDriver.cs

— OVREyeGaze →

eyeLook*

blendshape driver (LateUpdate)

Why this approach (vs. alternatives)

Custom
MonoBehaviour
driver that reads
```
OVRFaceExpressions[expr]
```
and writes blendshape weights directly: works, but doesn't integrate with MSDK's correctives, eye constraints, or future retargeting upgrades. Use only if you can't extend
```
OVRCustomFace
```
.
OVRCustomFace
+
RetargetingType.ARKitBlendshapes
: clean, but guarded behind
```
#if OVR_INTERNAL_CODE
```
in the public Oculus Integration package — not 3P-shippable.
OVRCustomFace
+
RetargetingType.Custom
+
GetCustomBlendShapeNameAndExpressionPairs
override (this skill): fully public API, shippable, and gets all base-class behavior (data validity gating, weight scaling, mesh assertion).

hz-unity-face-tracking

NPX Install

Tags

SKILL.md Content