realitykit-ar

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

RealityKit + ARKit

Build AR experiences on iOS using RealityKit for rendering and ARKit for world tracking. Covers

RealityView

, entity management, raycasting, scene understanding, and gesture-based interactions. Targets Swift 6.2 / iOS 26+.

在iOS上使用RealityKit进行渲染、ARKit进行世界追踪，构建AR体验。内容涵盖

RealityView

、实体管理、射线投射、场景理解以及基于手势的交互。目标环境为Swift 6.2 / iOS 26+。

Setup

设置

Project Configuration

项目配置

Add
```
NSCameraUsageDescription
```
to Info.plist
For iOS, RealityKit uses the device camera by default via
```
RealityViewCameraContent
```
(iOS 18+, macOS 15+)
No additional capabilities required for basic AR on iOS

在Info.plist中添加
```
NSCameraUsageDescription
```
在iOS上，RealityKit默认通过
```
RealityViewCameraContent
```
使用设备摄像头（iOS 18+、macOS 15+）
iOS基础AR功能无需额外配置权限

Device Requirements

设备要求

AR features require devices with an A9 chip or later. Always verify support before presenting AR UI.

swift

import ARKit

guard ARWorldTrackingConfiguration.isSupported else {
    showUnsupportedDeviceMessage()
    return
}

AR功能需要搭载A9芯片及以上的设备。在展示AR界面前务必验证设备支持性。

swift

import ARKit

guard ARWorldTrackingConfiguration.isSupported else {
    showUnsupportedDeviceMessage()
    return
}

Key Types

核心类型

Type	Platform	Role
`RealityView`	iOS 18+, visionOS 1+	SwiftUI view that hosts RealityKit content
`RealityViewCameraContent`	iOS 18+, macOS 15+	Content displayed through the device camera
`Entity`	All	Base class for all scene objects
`ModelEntity`	All	Entity with a visible 3D model
`AnchorEntity`	All	Tethers entities to a real-world anchor

类型	平台	作用
`RealityView`	iOS 18+, visionOS 1+	承载RealityKit内容的SwiftUI视图
`RealityViewCameraContent`	iOS 18+, macOS 15+	通过设备摄像头显示的内容
`Entity`	全平台	所有场景对象的基类
`ModelEntity`	全平台	带有可见3D模型的实体
`AnchorEntity`	全平台	将实体绑定到现实世界锚点的实体

RealityView Basics

RealityView 基础

RealityView

is the SwiftUI entry point for RealityKit. On iOS, it provides

RealityViewCameraContent

which renders through the device camera for AR.

swift

import SwiftUI
import RealityKit

struct ARExperienceView: View {
    var body: some View {
        RealityView { content in
            // content is RealityViewCameraContent on iOS
            let sphere = ModelEntity(
                mesh: .generateSphere(radius: 0.05),
                materials: [SimpleMaterial(
                    color: .blue,
                    isMetallic: true
                )]
            )
            sphere.position = [0, 0, -0.5]  // 50cm in front of camera
            content.add(sphere)
        }
    }
}

RealityView

是RealityKit的SwiftUI入口。在iOS上，它通过

RealityViewCameraContent

实现基于设备摄像头的AR渲染。

swift

import SwiftUI
import RealityKit

struct ARExperienceView: View {
    var body: some View {
        RealityView { content in
            // iOS上content为RealityViewCameraContent
            let sphere = ModelEntity(
                mesh: .generateSphere(radius: 0.05),
                materials: [SimpleMaterial(
                    color: .blue,
                    isMetallic: true
                )]
            )
            sphere.position = [0, 0, -0.5]  // 位于摄像头前方50cm处
            content.add(sphere)
        }
    }
}

Make and Update Pattern

创建与更新模式

Use the

update

closure to respond to SwiftUI state changes:

swift

struct PlacementView: View {
    @State private var modelColor: UIColor = .red

    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(
                    color: .red,
                    isMetallic: false
                )]
            )
            box.name = "colorBox"
            box.position = [0, 0, -0.5]
            content.add(box)
        } update: { content in
            if let box = content.entities.first(
                where: { $0.name == "colorBox" }
            ) as? ModelEntity {
                box.model?.materials = [SimpleMaterial(
                    color: modelColor,
                    isMetallic: false
                )]
            }
        }

        Button("Change Color") {
            modelColor = modelColor == .red ? .green : .red
        }
    }
}

使用

update

闭响应用户界面状态变化：

swift

struct PlacementView: View {
    @State private var modelColor: UIColor = .red

    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(
                    color: .red,
                    isMetallic: false
                )]
            )
            box.name = "colorBox"
            box.position = [0, 0, -0.5]
            content.add(box)
        } update: { content in
            if let box = content.entities.first(
                where: { $0.name == "colorBox" }
            ) as? ModelEntity {
                box.model?.materials = [SimpleMaterial(
                    color: modelColor,
                    isMetallic: false
                )]
            }
        }

        Button("更改颜色") {
            modelColor = modelColor == .red ? .green : .red
        }
    }
}

Loading and Creating Entities

加载与创建实体

Loading from USDZ Files

从USDZ文件加载

Load 3D models asynchronously to avoid blocking the main thread:

swift

RealityView { content in
    if let robot = try? await ModelEntity(named: "robot") {
        robot.position = [0, -0.2, -0.8]
        robot.scale = [0.01, 0.01, 0.01]
        content.add(robot)
    }
}

异步加载3D模型以避免阻塞主线程：

swift

RealityView { content in
    if let robot = try? await ModelEntity(named: "robot") {
        robot.position = [0, -0.2, -0.8]
        robot.scale = [0.01, 0.01, 0.01]
        content.add(robot)
    }
}

Programmatic Mesh Generation

程序化生成网格

swift

// Box
let box = ModelEntity(
    mesh: .generateBox(size: [0.1, 0.2, 0.1], cornerRadius: 0.005),
    materials: [SimpleMaterial(color: .gray, isMetallic: true)]
)

// Sphere
let sphere = ModelEntity(
    mesh: .generateSphere(radius: 0.05),
    materials: [SimpleMaterial(color: .blue, roughness: 0.2, isMetallic: true)]
)

// Plane
let plane = ModelEntity(
    mesh: .generatePlane(width: 0.3, depth: 0.3),
    materials: [SimpleMaterial(color: .green, isMetallic: false)]
)

swift

// 立方体
let box = ModelEntity(
    mesh: .generateBox(size: [0.1, 0.2, 0.1], cornerRadius: 0.005),
    materials: [SimpleMaterial(color: .gray, isMetallic: true)]
)

// 球体
let sphere = ModelEntity(
    mesh: .generateSphere(radius: 0.05),
    materials: [SimpleMaterial(color: .blue, roughness: 0.2, isMetallic: true)]
)

// 平面
let plane = ModelEntity(
    mesh: .generatePlane(width: 0.3, depth: 0.3),
    materials: [SimpleMaterial(color: .green, isMetallic: false)]
)

Adding Components

添加组件

Entities use an ECS (Entity Component System) architecture. Add components to give entities behavior:

swift

let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)

// Make it respond to physics
box.components.set(PhysicsBodyComponent(
    massProperties: .default,
    material: .default,
    mode: .dynamic
))

// Add collision shape for interaction
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))

// Enable input targeting for gestures
box.components.set(InputTargetComponent())

实体采用ECS（实体组件系统）架构。通过添加组件为实体赋予行为：

swift

let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)

// 启用物理特性
box.components.set(PhysicsBodyComponent(
    massProperties: .default,
    material: .default,
    mode: .dynamic
))

// 添加碰撞形状以支持交互
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))

// 启用输入目标以支持手势
box.components.set(InputTargetComponent())

Anchoring and Placement

锚定与放置

AnchorEntity

Use

AnchorEntity

to anchor content to detected surfaces or world positions:

swift

RealityView { content in
    // Anchor to a horizontal surface
    let floorAnchor = AnchorEntity(.plane(
        .horizontal,
        classification: .floor,
        minimumBounds: [0.2, 0.2]
    ))

    let model = ModelEntity(
        mesh: .generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .orange, isMetallic: false)]
    )
    floorAnchor.addChild(model)
    content.add(floorAnchor)
}

使用

AnchorEntity

将内容锚定到检测到的表面或世界位置：

swift

RealityView { content in
    // 锚定到水平表面
    let floorAnchor = AnchorEntity(.plane(
        .horizontal,
        classification: .floor,
        minimumBounds: [0.2, 0.2]
    ))

    let model = ModelEntity(
        mesh: .generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .orange, isMetallic: false)]
    )
    floorAnchor.addChild(model)
    content.add(floorAnchor)
}

Anchor Targets

锚定目标

Target	Description
`.plane(.horizontal, ...)`	Horizontal surfaces (floors, tables)
`.plane(.vertical, ...)`	Vertical surfaces (walls)
`.plane(.any, ...)`	Any detected plane
`.world(transform:)`	Fixed world-space position

目标	描述
`.plane(.horizontal, ...)`	水平表面（地面、桌面）
`.plane(.vertical, ...)`	垂直表面（墙面）
`.plane(.any, ...)`	任意检测到的平面
`.world(transform:)`	固定的世界空间位置

Raycasting

射线投射

Use

RealityViewCameraContent

to convert between SwiftUI view coordinates and RealityKit world space. Pair with

SpatialTapGesture

to place objects where the user taps on a detected surface.

使用

RealityViewCameraContent

在SwiftUI视图坐标与RealityKit世界空间之间进行转换。结合

SpatialTapGesture

，可在用户点击检测到的表面处放置对象。

Gestures and Interaction

手势与交互

Drag Gesture on Entities

实体拖拽手势

swift

struct DraggableARView: View {
    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: true)]
            )
            box.position = [0, 0, -0.5]
            box.components.set(CollisionComponent(
                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
            ))
            box.components.set(InputTargetComponent())
            box.name = "draggable"
            content.add(box)
        }
        .gesture(
            DragGesture()
                .targetedToAnyEntity()
                .onChanged { value in
                    let entity = value.entity
                    guard let parent = entity.parent else { return }
                    entity.position = value.convert(
                        value.location3D,
                        from: .local,
                        to: parent
                    )
                }
        )
    }
}

swift

struct DraggableARView: View {
    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: true)]
            )
            box.position = [0, 0, -0.5]
            box.components.set(CollisionComponent(
                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
            ))
            box.components.set(InputTargetComponent())
            box.name = "draggable"
            content.add(box)
        }
        .gesture(
            DragGesture()
                .targetedToAnyEntity()
                .onChanged { value in
                    let entity = value.entity
                    guard let parent = entity.parent else { return }
                    entity.position = value.convert(
                        value.location3D,
                        from: .local,
                        to: parent
                    )
                }
        )
    }
}

Tap to Select

点击选择

swift

.gesture(
    SpatialTapGesture()
        .targetedToAnyEntity()
        .onEnded { value in
            let tappedEntity = value.entity
            highlightEntity(tappedEntity)
        }
)

swift

.gesture(
    SpatialTapGesture()
        .targetedToAnyEntity()
        .onEnded { value in
            let tappedEntity = value.entity
            highlightEntity(tappedEntity)
        }
)

Scene Understanding

场景理解

Per-Frame Updates

逐帧更新

Subscribe to scene update events for continuous processing:

swift

RealityView { content in
    let entity = ModelEntity(
        mesh: .generateSphere(radius: 0.05),
        materials: [SimpleMaterial(color: .yellow, isMetallic: false)]
    )
    entity.position = [0, 0, -0.5]
    content.add(entity)

    _ = content.subscribe(to: SceneEvents.Update.self) { event in
        let time = Float(event.deltaTime)
        entity.position.y += sin(Float(Date().timeIntervalSince1970)) * time * 0.1
    }
}

订阅场景更新事件以进行持续处理：

swift

RealityView { content in
    let entity = ModelEntity(
        mesh: .generateSphere(radius: 0.05),
        materials: [SimpleMaterial(color: .yellow, isMetallic: false)]
    )
    entity.position = [0, 0, -0.5]
    content.add(entity)

    _ = content.subscribe(to: SceneEvents.Update.self) { event in
        let time = Float(event.deltaTime)
        entity.position.y += sin(Float(Date().timeIntervalSince1970)) * time * 0.1
    }
}

visionOS Note

visionOS 说明

On visionOS, ARKit provides a different API surface with

ARKitSession

WorldTrackingProvider

, and

PlaneDetectionProvider

. These visionOS-specific types are not available on iOS. On iOS, RealityKit handles world tracking automatically through

RealityViewCameraContent

在visionOS上，ARKit提供了不同的API，包括

ARKitSession

、

WorldTrackingProvider

和

PlaneDetectionProvider

。这些visionOS专属类型在iOS上不可用。在iOS上，RealityKit通过

RealityViewCameraContent

自动处理世界追踪。

Common Mistakes

常见错误

DON'T: Skip AR capability checks

错误做法：跳过AR兼容性检查

Not all devices support AR. Showing a black camera view with no feedback confuses users.

swift

// WRONG -- no device check
struct MyARView: View {
    var body: some View {
        RealityView { content in
            // Fails silently on unsupported devices
        }
    }
}

// CORRECT -- check support and show fallback
struct MyARView: View {
    var body: some View {
        if ARWorldTrackingConfiguration.isSupported {
            RealityView { content in
                // AR content
            }
        } else {
            ContentUnavailableView(
                "AR Not Supported",
                systemImage: "arkit",
                description: Text("This device does not support AR.")
            )
        }
    }
}

并非所有设备都支持AR。直接显示黑屏且无提示会使用户困惑。

swift

// 错误示例 -- 未做设备检查
struct MyARView: View {
    var body: some View {
        RealityView { content in
            // 在不支持的设备上会静默失败
        }
    }
}

// 正确示例 -- 检查支持性并提供回退界面
struct MyARView: View {
    var body: some View {
        if ARWorldTrackingConfiguration.isSupported {
            RealityView { content in
                // AR内容
            }
        } else {
            ContentUnavailableView(
                "AR不支持",
                systemImage: "arkit",
                description: Text("该设备不支持AR功能。")
            )
        }
    }
}

DON'T: Load heavy models synchronously

错误做法：同步加载大型模型

Loading large USDZ files on the main thread causes frame drops and hangs. The

make

closure of

RealityView

async

-- use it.

swift

// WRONG -- synchronous load blocks the main thread
RealityView { content in
    let model = try! Entity.load(named: "large-scene")
    content.add(model)
}

// CORRECT -- async load
RealityView { content in
    if let model = try? await ModelEntity(named: "large-scene") {
        content.add(model)
    }
}

在主线程加载大型USDZ文件会导致帧丢卡顿。

RealityView

的

make

闭包支持

async

，请使用异步加载。

swift

// 错误示例 -- 同步加载阻塞主线程
RealityView { content in
    let model = try! Entity.load(named: "large-scene")
    content.add(model)
}

// 正确示例 -- 异步加载
RealityView { content in
    if let model = try? await ModelEntity(named: "large-scene") {
        content.add(model)
    }
}

DON'T: Forget collision and input target components for interactive entities

错误做法：交互式实体缺少碰撞与输入组件

Gestures only work on entities that have both

CollisionComponent

and

InputTargetComponent

. Without them, taps and drags pass through.

swift

// WRONG -- entity ignores gestures
let box = ModelEntity(mesh: .generateBox(size: 0.1))
content.add(box)

// CORRECT -- add collision and input components
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))
box.components.set(InputTargetComponent())
content.add(box)

只有同时添加

CollisionComponent

和

InputTargetComponent

的实体才能响应手势。缺少这些组件时，点击和拖拽操作会无效。

swift

// 错误示例 -- 实体无法响应手势
let box = ModelEntity(mesh: .generateBox(size: 0.1))
content.add(box)

// 正确示例 -- 添加碰撞与输入组件
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))
box.components.set(InputTargetComponent())
content.add(box)

DON'T: Create new entities in the update closure

错误做法：在update闭包中创建新实体

The

update

closure runs on every SwiftUI state change. Creating entities there duplicates content on each render pass.

swift

// WRONG -- duplicates entities on every state change
RealityView { content in
    // empty
} update: { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    content.add(sphere)  // Added again on every update
}

// CORRECT -- create in make, modify in update
RealityView { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    sphere.name = "mySphere"
    content.add(sphere)
} update: { content in
    if let sphere = content.entities.first(
        where: { $0.name == "mySphere" }
    ) as? ModelEntity {
        // Modify existing entity
        sphere.position.y = newYPosition
    }
}

update

闭包会在每次SwiftUI状态变化时执行。在此处创建实体会导致每次渲染都重复添加内容。

swift

// 错误示例 -- 状态变化时重复创建实体
RealityView { content in
    // 空实现
} update: { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    content.add(sphere)  // 每次更新都会重复添加
}

// 正确示例 -- 在make中创建，在update中修改
RealityView { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    sphere.name = "mySphere"
    content.add(sphere)
} update: { content in
    if let sphere = content.entities.first(
        where: { $0.name == "mySphere" }
    ) as? ModelEntity {
        // 修改现有实体
        sphere.position.y = newYPosition
    }
}

DON'T: Ignore camera permission

错误做法：忽略相机权限

RealityKit on iOS needs camera access. If the user denies permission, the view shows a black screen with no explanation.

swift

// WRONG -- no permission handling
RealityView { content in
    // Black screen if camera denied
}

// CORRECT -- check and request permission
struct ARContainerView: View {
    @State private var cameraAuthorized = false

    var body: some View {
        Group {
            if cameraAuthorized {
                RealityView { content in
                    // AR content
                }
            } else {
                ContentUnavailableView(
                    "Camera Access Required",
                    systemImage: "camera.fill",
                    description: Text("Enable camera in Settings to use AR.")
                )
            }
        }
        .task {
            let status = AVCaptureDevice.authorizationStatus(for: .video)
            if status == .authorized {
                cameraAuthorized = true
            } else if status == .notDetermined {
                cameraAuthorized = await AVCaptureDevice
                    .requestAccess(for: .video)
            }
        }
    }
}

iOS上的RealityKit需要相机访问权限。如果用户拒绝权限，界面会显示黑屏且无说明。

swift

// 错误示例 -- 未处理权限
RealityView { content in
    // 相机权限被拒绝时显示黑屏
}

// 正确示例 -- 检查并请求权限
struct ARContainerView: View {
    @State private var cameraAuthorized = false

    var body: some View {
        Group {
            if cameraAuthorized {
                RealityView { content in
                    // AR内容
                }
            } else {
                ContentUnavailableView(
                    "需要相机权限",
                    systemImage: "camera.fill",
                    description: Text("请在设置中启用相机权限以使用AR功能。")
                )
            }
        }
        .task {
            let status = AVCaptureDevice.authorizationStatus(for: .video)
            if status == .authorized {
                cameraAuthorized = true
            } else if status == .notDetermined {
                cameraAuthorized = await AVCaptureDevice
                    .requestAccess(for: .video)
            }
        }
    }
}

Review Checklist

检查清单

```
NSCameraUsageDescription
```
set in Info.plist
AR device capability checked before presenting AR views
Camera permission requested and denial handled with a fallback UI
3D models loaded asynchronously in the
```
make
```
closure
Entities created in
```
make
```
, modified in
```
update
```
(not created in
```
update
```
)
Interactive entities have both
```
CollisionComponent
```
and
```
InputTargetComponent
```
Collision shapes match the visual size of the entity
```
SceneEvents.Update
```
subscriptions used for per-frame logic (not SwiftUI timers)
Large scenes use
```
ModelEntity(named:)
```
async loading, not
```
Entity.load(named:)
```
Anchor entities target appropriate surface types for the use case
Entity names set for lookup in the
```
update
```
closure

Info.plist中已设置
```
NSCameraUsageDescription
```
展示AR界面之前已检查设备AR支持性
已处理相机权限，拒绝时提供回退UI
3D模型已在
```
make
```
闭包中异步加载
实体在
```
make
```
中创建，仅在
```
update
```
中修改（不在
```
update
```
中创建）
交互式实体已同时添加
```
CollisionComponent
```
和
```
InputTargetComponent
```
碰撞形状与实体视觉尺寸匹配
使用
```
SceneEvents.Update
```
订阅实现逐帧逻辑（而非SwiftUI定时器）
大型场景使用
```
ModelEntity(named:)
```
异步加载，而非
```
Entity.load(named:)
```
锚定实体已根据使用场景选择合适的表面类型
已为实体设置名称，以便在
```
update
```
闭包中查找

References

参考资料

Extended patterns (physics, animations, lighting, ECS):
```
references/realitykit-patterns.md
```
RealityKit framework
RealityView
RealityViewCameraContent
Entity
ModelEntity
AnchorEntity
ARKit framework
ARKit in iOS
ARWorldTrackingConfiguration
Loading entities from a file

扩展模式（物理、动画、光照、ECS）：
```
references/realitykit-patterns.md
```
RealityKit框架
RealityView
RealityViewCameraContent
Entity
ModelEntity
AnchorEntity
ARKit框架
iOS中的ARKit
ARWorldTrackingConfiguration
从文件加载实体

realitykit-ar

Original

Translation

RealityKit + ARKit

RealityKit + ARKit

Contents

目录

Setup

设置

Project Configuration

项目配置

Device Requirements

设备要求

Key Types

核心类型

RealityView Basics

RealityView 基础

Make and Update Pattern

创建与更新模式

Loading and Creating Entities

加载与创建实体

Loading from USDZ Files

从USDZ文件加载

Programmatic Mesh Generation

程序化生成网格

Adding Components

添加组件

Anchoring and Placement

锚定与放置

AnchorEntity

AnchorEntity

Anchor Targets

锚定目标

Raycasting

射线投射

Gestures and Interaction

手势与交互

Drag Gesture on Entities

实体拖拽手势

Tap to Select

点击选择

Scene Understanding

场景理解

Per-Frame Updates

逐帧更新

visionOS Note

visionOS 说明

Common Mistakes

常见错误

DON'T: Skip AR capability checks

错误做法：跳过AR兼容性检查

DON'T: Load heavy models synchronously

错误做法：同步加载大型模型

DON'T: Forget collision and input target components for interactive entities

错误做法：交互式实体缺少碰撞与输入组件

DON'T: Create new entities in the update closure

错误做法：在update闭包中创建新实体

DON'T: Ignore camera permission

错误做法：忽略相机权限

Review Checklist

检查清单

References

参考资料