axiom-realitykit

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

RealityKit Development Guide

RealityKit 开发指南

Purpose: Build 3D content, AR experiences, and spatial computing apps using RealityKit's Entity-Component-System architecture iOS Version: iOS 13+ (base), iOS 18+ (RealityView on iOS), visionOS 1.0+ Xcode: Xcode 15+
用途:使用RealityKit的实体-组件-系统(ECS)架构构建3D内容、AR体验和空间计算应用 iOS版本:iOS 13+(基础功能),iOS 18+(iOS端支持RealityView),visionOS 1.0+ Xcode:Xcode 15+

When to Use This Skill

何时使用该技能

Use this skill when:
  • Building any 3D experience (AR, games, visualization, spatial computing)
  • Creating SwiftUI apps with 3D content (RealityView, Model3D)
  • Implementing AR with anchors (world, image, face, body tracking)
  • Working with Entity-Component-System (ECS) architecture
  • Setting up physics, collisions, or spatial interactions
  • Building multiplayer or shared AR experiences
  • Migrating from SceneKit to RealityKit
  • Targeting visionOS
Do NOT use this skill for:
  • SceneKit maintenance (use
    axiom-scenekit
    )
  • 2D games (use
    axiom-spritekit
    )
  • Metal shader programming (use
    axiom-metal-migration-ref
    )
  • Pure GPU compute (use Metal directly)

在以下场景中使用本技能:
  • 构建任何3D体验(AR、游戏、可视化、空间计算)
  • 创建包含3D内容的SwiftUI应用(RealityView、Model3D)
  • 实现带锚点的AR功能(平面、图像、面部、身体追踪)
  • 基于实体-组件-系统(ECS)架构进行开发
  • 设置物理系统、碰撞检测或空间交互
  • 构建多人协作或共享AR体验
  • 从SceneKit迁移到RealityKit
  • 开发visionOS应用
请勿在以下场景使用本技能:
  • SceneKit维护(使用
    axiom-scenekit
  • 2D游戏开发(使用
    axiom-spritekit
  • Metal着色器编程(使用
    axiom-metal-migration-ref
  • 纯GPU计算(直接使用Metal)

1. Mental Model: ECS vs Scene Graph

1. 思维模型:ECS vs 场景图

Scene Graph (SceneKit)

场景图(SceneKit)

In SceneKit, nodes own their properties. A node IS a renderable, collidable, animated thing.
在SceneKit中,节点拥有自身属性。一个节点就是可渲染、可碰撞、可动画的对象。

Entity-Component-System (RealityKit)

实体-组件-系统(ECS,RealityKit)

In RealityKit, entities are empty containers. Components add data. Systems process that data.
Entity (identity + hierarchy)
  ├── TransformComponent (position, rotation, scale)
  ├── ModelComponent (mesh + materials)
  ├── CollisionComponent (collision shapes)
  ├── PhysicsBodyComponent (mass, mode)
  └── [YourCustomComponent] (game-specific data)

System (processes entities with specific components each frame)
Why ECS matters:
  • Composition over inheritance: Combine any components on any entity
  • Data-oriented: Systems process arrays of components efficiently
  • Decoupled logic: Systems don't know about each other
  • Testable: Components are pure data, Systems are pure logic
在RealityKit中,实体是空容器。组件添加数据,系统处理数据。
Entity (标识 + 层级)
  ├── TransformComponent (位置、旋转、缩放)
  ├── ModelComponent (网格 + 材质)
  ├── CollisionComponent (碰撞形状)
  ├── PhysicsBodyComponent (质量、模式)
  └── [YourCustomComponent] (游戏专属数据)

System (每帧处理带有特定组件的实体)
ECS的重要性
  • 组合优于继承:可在任意实体上组合任意组件
  • 面向数据:系统高效处理组件数组
  • 逻辑解耦:系统之间互不依赖
  • 可测试性:组件是纯数据,系统是纯逻辑

The ECS Mental Shift

ECS的思维转变

Scene Graph ThinkingECS Thinking
"The player node moves""The movement system processes entities with MovementComponent"
"Add a method to the node subclass""Add a component, create a system"
"Override
update(_:)
in the node"
"Register a System that queries for components"
"The node knows its health""HealthComponent holds data, DamageSystem processes it"

场景图思维ECS思维
"玩家节点移动""移动系统处理带有MovementComponent的实体"
"为节点子类添加方法""添加组件,创建系统"
"在节点中重写
update(_:)
"
"注册一个查询组件的系统"
"节点知晓自身生命值""HealthComponent存储数据,DamageSystem处理逻辑"

2. Entity Hierarchy

2. 实体层级

Creating Entities

创建实体

swift
// Empty entity
let entity = Entity()
entity.name = "player"

// Entity with components
let entity = Entity()
entity.components[ModelComponent.self] = ModelComponent(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .blue, isMetallic: false)]
)

// ModelEntity convenience (has ModelComponent built in)
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: true)]
)
swift
// 空实体
let entity = Entity()
entity.name = "player"

// 带组件的实体
let entity = Entity()
entity.components[ModelComponent.self] = ModelComponent(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .blue, isMetallic: false)]
)

// ModelEntity便捷方式(内置ModelComponent)
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: true)]
)

Hierarchy Management

层级管理

swift
// Parent-child
parent.addChild(child)
child.removeFromParent()

// Find entities
let found = root.findEntity(named: "player")

// Enumerate
for child in entity.children {
    // Process children
}

// Clone
let clone = entity.clone(recursive: true)
swift
// 父子关系
parent.addChild(child)
child.removeFromParent()

// 查找实体
let found = root.findEntity(named: "player")

// 遍历子实体
for child in entity.children {
    // 处理子实体
}

// 克隆实体
let clone = entity.clone(recursive: true)

Transform

变换

swift
// Local transform (relative to parent)
entity.position = SIMD3<Float>(0, 1, 0)
entity.orientation = simd_quatf(angle: .pi / 4, axis: SIMD3(0, 1, 0))
entity.scale = SIMD3<Float>(repeating: 2.0)

// World-space queries
let worldPos = entity.position(relativeTo: nil)
let worldTransform = entity.transform(relativeTo: nil)

// Set world-space transform
entity.setPosition(SIMD3(1, 0, 0), relativeTo: nil)

// Look at a point
entity.look(at: targetPosition, from: entity.position, relativeTo: nil)

swift
// 局部变换(相对于父实体)
entity.position = SIMD3<Float>(0, 1, 0)
entity.orientation = simd_quatf(angle: .pi / 4, axis: SIMD3(0, 1, 0))
entity.scale = SIMD3<Float>(repeating: 2.0)

// 世界空间查询
let worldPos = entity.position(relativeTo: nil)
let worldTransform = entity.transform(relativeTo: nil)

// 设置世界空间变换
entity.setPosition(SIMD3(1, 0, 0), relativeTo: nil)

// 看向目标点
entity.look(at: targetPosition, from: entity.position, relativeTo: nil)

3. Components

3. 组件

Built-in Components

内置组件

ComponentPurpose
Transform
Position, rotation, scale
ModelComponent
Mesh geometry + materials
CollisionComponent
Collision shapes for physics and interaction
PhysicsBodyComponent
Mass, physics mode (dynamic/static/kinematic)
PhysicsMotionComponent
Linear and angular velocity
AnchoringComponent
AR anchor attachment
SynchronizationComponent
Multiplayer sync
PerspectiveCameraComponent
Camera settings
DirectionalLightComponent
Directional light
PointLightComponent
Point light
SpotLightComponent
Spot light
CharacterControllerComponent
Character physics controller
AudioMixGroupsComponent
Audio mixing
SpatialAudioComponent
3D positional audio
AmbientAudioComponent
Non-positional audio
ChannelAudioComponent
Multi-channel audio
OpacityComponent
Entity transparency
GroundingShadowComponent
Contact shadow
InputTargetComponent
Gesture input (visionOS)
HoverEffectComponent
Hover highlight (visionOS)
AccessibilityComponent
VoiceOver support
组件用途
Transform
位置、旋转、缩放
ModelComponent
网格几何 + 材质
CollisionComponent
用于物理系统和交互的碰撞形状
PhysicsBodyComponent
质量、物理模式(动态/静态/运动学)
PhysicsMotionComponent
线速度和角速度
AnchoringComponent
AR锚点附着
SynchronizationComponent
多人同步
PerspectiveCameraComponent
相机设置
DirectionalLightComponent
平行光
PointLightComponent
点光源
SpotLightComponent
聚光灯
CharacterControllerComponent
角色物理控制器
AudioMixGroupsComponent
音频混合
SpatialAudioComponent
3D位置音频
AmbientAudioComponent
非位置音频
ChannelAudioComponent
多通道音频
OpacityComponent
实体透明度
GroundingShadowComponent
接触阴影
InputTargetComponent
手势输入(visionOS)
HoverEffectComponent
悬停高亮(visionOS)
AccessibilityComponent
VoiceOver支持

Custom Components

自定义组件

swift
struct HealthComponent: Component {
    var current: Int
    var maximum: Int

    var percentage: Float {
        Float(current) / Float(maximum)
    }
}

// Register before use (typically in app init)
HealthComponent.registerComponent()

// Attach to entity
entity.components[HealthComponent.self] = HealthComponent(current: 100, maximum: 100)

// Read
if let health = entity.components[HealthComponent.self] {
    print(health.current)
}

// Modify
entity.components[HealthComponent.self]?.current -= 10
swift
struct HealthComponent: Component {
    var current: Int
    var maximum: Int

    var percentage: Float {
        Float(current) / Float(maximum)
    }
}

// 使用前注册(通常在应用初始化时)
HealthComponent.registerComponent()

// 附加到实体
entity.components[HealthComponent.self] = HealthComponent(current: 100, maximum: 100)

// 读取组件
if let health = entity.components[HealthComponent.self] {
    print(health.current)
}

// 修改组件
entity.components[HealthComponent.self]?.current -= 10

Component Lifecycle

组件生命周期

Components are value types (structs). When you read a component, modify it, and write it back, you're replacing the entire component:
swift
// Read-modify-write pattern
var health = entity.components[HealthComponent.self]!
health.current -= damage
entity.components[HealthComponent.self] = health
Anti-pattern: Holding a reference to a component and expecting mutations to propagate. Components are copied on read.

组件是值类型(结构体)。当你读取组件、修改并写回时,你是在替换整个组件:
swift
// 读取-修改-写入模式
var health = entity.components[HealthComponent.self]!
health.current -= damage
entity.components[HealthComponent.self] = health
反模式:持有组件引用并期望修改自动同步。组件在读取时会被复制。

4. Systems

4. 系统

System Protocol

系统协议

swift
struct DamageSystem: System {
    // Define which components this system needs
    static let query = EntityQuery(where: .has(HealthComponent.self))

    init(scene: RealityKit.Scene) {
        // One-time setup
    }

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            var health = entity.components[HealthComponent.self]!
            if health.current <= 0 {
                entity.removeFromParent()
            }
        }
    }
}

// Register system
DamageSystem.registerSystem()
swift
struct DamageSystem: System {
    // 定义该系统需要的组件
    static let query = EntityQuery(where: .has(HealthComponent.self))

    init(scene: RealityKit.Scene) {
        // 一次性初始化
    }

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            var health = entity.components[HealthComponent.self]!
            if health.current <= 0 {
                entity.removeFromParent()
            }
        }
    }
}

// 注册系统
DamageSystem.registerSystem()

System Best Practices

系统最佳实践

  • One responsibility per system: MovementSystem, DamageSystem, RenderingSystem — not GameLogicSystem
  • Query filtering: Use precise queries to avoid processing irrelevant entities
  • Order matters: Systems run in registration order. Register dependencies first.
  • Avoid storing entity references: Query each frame instead. Entity references can become stale.
  • 单一职责:每个系统只负责一项功能,比如MovementSystem、DamageSystem、RenderingSystem,而不是GameLogicSystem
  • 查询过滤:使用精确查询避免处理无关实体
  • 执行顺序:系统按注册顺序运行,先注册依赖系统
  • 避免存储实体引用:每帧查询实体,实体引用可能会失效

Event Handling

事件处理

swift
// Subscribe to collision events
scene.subscribe(to: CollisionEvents.Began.self) { event in
    let entityA = event.entityA
    let entityB = event.entityB
    // Handle collision
}

// Subscribe to scene update
scene.subscribe(to: SceneEvents.Update.self) { event in
    let deltaTime = event.deltaTime
    // Per-frame logic
}

swift
// 订阅碰撞事件
scene.subscribe(to: CollisionEvents.Began.self) { event in
    let entityA = event.entityA
    let entityB = event.entityB
    // 处理碰撞
}

// 订阅场景更新
scene.subscribe(to: SceneEvents.Update.self) { event in
    let deltaTime = event.deltaTime
    // 每帧逻辑
}

5. SwiftUI Integration

5. SwiftUI集成

RealityView (iOS 18+, visionOS 1.0+)

RealityView(iOS 18+, visionOS 1.0+)

swift
struct ContentView: View {
    var body: some View {
        RealityView { content in
            // make closure — called once
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: false)]
            )
            content.add(box)

        } update: { content in
            // update closure — called when SwiftUI state changes
        }
    }
}
swift
struct ContentView: View {
    var body: some View {
        RealityView { content in
            // make闭包 — 只调用一次
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: false)]
            )
            content.add(box)

        } update: { content in
            // update闭包 — SwiftUI状态变化时调用
        }
    }
}

RealityView with Camera (iOS)

iOS端带相机的RealityView

On iOS,
RealityView
provides a camera content parameter for configuring the AR or virtual camera:
swift
RealityView { content, attachments in
    // Load 3D content
    if let model = try? await ModelEntity(named: "scene") {
        content.add(model)
    }
}
在iOS上,
RealityView
提供相机内容参数用于配置AR或虚拟相机:
swift
RealityView { content, attachments in
    // 加载3D内容
    if let model = try? await ModelEntity(named: "scene") {
        content.add(model)
    }
}

Loading Content Asynchronously

异步加载内容

swift
RealityView { content in
    // Load from bundle
    if let entity = try? await Entity(named: "MyScene", in: .main) {
        content.add(entity)
    }

    // Load from URL
    if let entity = try? await Entity(contentsOf: modelURL) {
        content.add(entity)
    }
}
swift
RealityView { content in
    // 从Bundle加载
    if let entity = try? await Entity(named: "MyScene", in: .main) {
        content.add(entity)
    }

    // 从URL加载
    if let entity = try? await Entity(contentsOf: modelURL) {
        content.add(entity)
    }
}

Model3D (Simple Display)

Model3D(简单展示)

swift
// Simple 3D model display (no interaction)
Model3D(named: "toy_robot") { model in
    model
        .resizable()
        .scaledToFit()
} placeholder: {
    ProgressView()
}
swift
// 简单3D模型展示(无交互)
Model3D(named: "toy_robot") { model in
    model
        .resizable()
        .scaledToFit()
} placeholder: {
    ProgressView()
}

SwiftUI Attachments (visionOS)

SwiftUI附件(visionOS)

swift
RealityView { content, attachments in
    let entity = ModelEntity(mesh: .generateSphere(radius: 0.1))
    content.add(entity)

    if let label = attachments.entity(for: "priceTag") {
        label.position = SIMD3(0, 0.15, 0)
        entity.addChild(label)
    }
} attachments: {
    Attachment(id: "priceTag") {
        Text("$9.99")
            .padding()
            .glassBackgroundEffect()
    }
}
swift
RealityView { content, attachments in
    let entity = ModelEntity(mesh: .generateSphere(radius: 0.1))
    content.add(entity)

    if let label = attachments.entity(for: "priceTag") {
        label.position = SIMD3(0, 0.15, 0)
        entity.addChild(label)
    }
} attachments: {
    Attachment(id: "priceTag") {
        Text("$9.99")
            .padding()
            .glassBackgroundEffect()
    }
}

State Binding Pattern

状态绑定模式

swift
struct GameView: View {
    @State private var score = 0

    var body: some View {
        VStack {
            Text("Score: \(score)")

            RealityView { content in
                let scene = try! await Entity(named: "GameScene")
                content.add(scene)
            } update: { content in
                // React to state changes
                // Note: update is called when SwiftUI state changes,
                // not every frame. Use Systems for per-frame logic.
            }
        }
    }
}

swift
struct GameView: View {
    @State private var score = 0

    var body: some View {
        VStack {
            Text("Score: \(score)")

            RealityView { content in
                let scene = try! await Entity(named: "GameScene")
                content.add(scene)
            } update: { content in
                // 响应状态变化
                // 注意:update仅在SwiftUI状态变化时调用,
                // 并非每帧调用。每帧逻辑请使用Systems。
            }
        }
    }
}

6. AR on iOS

6. iOS端AR功能

AnchorEntity

AnchorEntity

swift
// Horizontal plane
let anchor = AnchorEntity(.plane(.horizontal, classification: .table,
                                  minimumBounds: SIMD2(0.2, 0.2)))

// Vertical plane
let anchor = AnchorEntity(.plane(.vertical, classification: .wall,
                                  minimumBounds: SIMD2(0.5, 0.5)))

// World position
let anchor = AnchorEntity(world: SIMD3<Float>(0, 0, -1))

// Image anchor
let anchor = AnchorEntity(.image(group: "AR Resources", name: "poster"))

// Face anchor (front camera)
let anchor = AnchorEntity(.face)

// Body anchor
let anchor = AnchorEntity(.body)
swift
// 水平面
let anchor = AnchorEntity(.plane(.horizontal, classification: .table,
                                  minimumBounds: SIMD2(0.2, 0.2)))

// 垂直面
let anchor = AnchorEntity(.plane(.vertical, classification: .wall,
                                  minimumBounds: SIMD2(0.5, 0.5)))

// 世界位置
let anchor = AnchorEntity(world: SIMD3<Float>(0, 0, -1))

// 图像锚点
let anchor = AnchorEntity(.image(group: "AR Resources", name: "poster"))

// 面部锚点(前置摄像头)
let anchor = AnchorEntity(.face)

// 身体锚点
let anchor = AnchorEntity(.body)

SpatialTrackingSession (iOS 18+)

SpatialTrackingSession(iOS 18+)

swift
let session = SpatialTrackingSession()
let configuration = SpatialTrackingSession.Configuration(tracking: [.plane, .object])
let result = await session.run(configuration)

if let notSupported = result {
    // Handle unsupported tracking on this device
    for denied in notSupported.deniedTrackingModes {
        print("Not supported: \(denied)")
    }
}
swift
let session = SpatialTrackingSession()
let configuration = SpatialTrackingSession.Configuration(tracking: [.plane, .object])
let result = await session.run(configuration)

if let notSupported = result {
    // 处理设备不支持的追踪模式
    for denied in notSupported.deniedTrackingModes {
        print("Not supported: \(denied)")
    }
}

AR Best Practices

AR最佳实践

  • Anchor entities to detected surfaces rather than world positions for stability
  • Use plane classification (
    .table
    ,
    .floor
    ,
    .wall
    ) to place content appropriately
  • Start with horizontal plane detection — it's the most reliable
  • Test on real devices; simulator AR is limited
  • Provide visual feedback during surface detection (coaching overlay)

  • 将实体锚定到检测到的表面而非世界位置,以提升稳定性
  • 使用平面分类(
    .table
    ,
    .floor
    ,
    .wall
    )合理放置内容
  • 从水平面检测开始,这是最可靠的模式
  • 在真实设备上测试,模拟器AR功能有限
  • 在表面检测过程中提供视觉反馈(引导层)

7. Interaction

7. 交互

ManipulationComponent (iOS, visionOS)

ManipulationComponent(iOS, visionOS)

swift
// Enable drag, rotate, scale gestures
entity.components[ManipulationComponent.self] = ManipulationComponent(
    allowedModes: .all  // .translate, .rotate, .scale
)

// Also requires CollisionComponent for hit testing
entity.generateCollisionShapes(recursive: true)
swift
// 启用拖拽、旋转、缩放手势
entity.components[ManipulationComponent.self] = ManipulationComponent(
    allowedModes: .all  // .translate, .rotate, .scale
)

// 还需要CollisionComponent用于命中测试
entity.generateCollisionShapes(recursive: true)

InputTargetComponent (visionOS)

InputTargetComponent(visionOS)

swift
// Required for visionOS gesture input
entity.components[InputTargetComponent.self] = InputTargetComponent()
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [.generateBox(size: SIMD3(0.1, 0.1, 0.1))]
)
swift
// visionOS手势输入必需
entity.components[InputTargetComponent.self] = InputTargetComponent()
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [.generateBox(size: SIMD3(0.1, 0.1, 0.1))]
)

Gesture Integration with SwiftUI

SwiftUI手势集成

swift
RealityView { content in
    let entity = ModelEntity(mesh: .generateBox(size: 0.1))
    entity.generateCollisionShapes(recursive: true)
    entity.components.set(InputTargetComponent())
    content.add(entity)
}
.gesture(
    TapGesture()
        .targetedToAnyEntity()
        .onEnded { value in
            let tappedEntity = value.entity
            // Handle tap
        }
)
.gesture(
    DragGesture()
        .targetedToAnyEntity()
        .onChanged { value in
            value.entity.position = value.convert(value.location3D,
                from: .local, to: .scene)
        }
)
swift
RealityView { content in
    let entity = ModelEntity(mesh: .generateBox(size: 0.1))
    entity.generateCollisionShapes(recursive: true)
    entity.components.set(InputTargetComponent())
    content.add(entity)
}
.gesture(
    TapGesture()
        .targetedToAnyEntity()
        .onEnded { value in
            let tappedEntity = value.entity
            // 处理点击
        }
)
.gesture(
    DragGesture()
        .targetedToAnyEntity()
        .onChanged { value in
            value.entity.position = value.convert(value.location3D,
                from: .local, to: .scene)
        }
)

Hit Testing

命中测试

swift
// Ray-cast from screen point
if let result = arView.raycast(from: screenPoint,
                                allowing: .estimatedPlane,
                                alignment: .horizontal).first {
    let worldPosition = result.worldTransform.columns.3
    // Place entity at worldPosition
}

swift
// 从屏幕点发射射线
if let result = arView.raycast(from: screenPoint,
                                allowing: .estimatedPlane,
                                alignment: .horizontal).first {
    let worldPosition = result.worldTransform.columns.3
    // 在worldPosition处放置实体
}

8. Materials and Rendering

8. 材质与渲染

Material Types

材质类型

MaterialPurposeCustomization
SimpleMaterial
Solid color or textureColor, metallic, roughness
PhysicallyBasedMaterial
Full PBRAll PBR maps (base color, normal, metallic, roughness, AO, emissive)
UnlitMaterial
No lighting responseColor or texture, always fully lit
OcclusionMaterial
Invisible but occludesAR content hiding behind real objects
VideoMaterial
Video playback on surfaceAVPlayer-driven
ShaderGraphMaterial
Custom shader graphReality Composer Pro
CustomMaterial
Metal shader functionsFull Metal control
材质用途自定义程度
SimpleMaterial
纯色或纹理颜色、金属度、粗糙度
PhysicallyBasedMaterial
完整PBR材质所有PBR贴图(基础色、法线、金属度、粗糙度、环境光遮蔽、自发光)
UnlitMaterial
无光照响应颜色或纹理,始终全亮
OcclusionMaterial
不可见但遮挡内容隐藏真实物体后方的3D内容
VideoMaterial
表面播放视频由AVPlayer驱动
ShaderGraphMaterial
自定义着色器图使用Reality Composer Pro
CustomMaterial
Metal着色器函数完全控制Metal

PhysicallyBasedMaterial

PhysicallyBasedMaterial

swift
var material = PhysicallyBasedMaterial()
material.baseColor = .init(tint: .white,
    texture: .init(try! .load(named: "albedo")))
material.metallic = .init(floatLiteral: 0.0)
material.roughness = .init(floatLiteral: 0.5)
material.normal = .init(texture: .init(try! .load(named: "normal")))
material.ambientOcclusion = .init(texture: .init(try! .load(named: "ao")))
material.emissiveColor = .init(color: .blue)
material.emissiveIntensity = 2.0

let entity = ModelEntity(
    mesh: .generateSphere(radius: 0.1),
    materials: [material]
)
swift
var material = PhysicallyBasedMaterial()
material.baseColor = .init(tint: .white,
    texture: .init(try! .load(named: "albedo")))
material.metallic = .init(floatLiteral: 0.0)
material.roughness = .init(floatLiteral: 0.5)
material.normal = .init(texture: .init(try! .load(named: "normal")))
material.ambientOcclusion = .init(texture: .init(try! .load(named: "ao")))
material.emissiveColor = .init(color: .blue)
material.emissiveIntensity = 2.0

let entity = ModelEntity(
    mesh: .generateSphere(radius: 0.1),
    materials: [material]
)

OcclusionMaterial (AR)

OcclusionMaterial(AR)

swift
// Invisible plane that hides 3D content behind it
let occluder = ModelEntity(
    mesh: .generatePlane(width: 1, depth: 1),
    materials: [OcclusionMaterial()]
)
occluder.position = SIMD3(0, 0, 0)
anchor.addChild(occluder)
swift
// 隐藏后方3D内容的不可见平面
let occluder = ModelEntity(
    mesh: .generatePlane(width: 1, depth: 1),
    materials: [OcclusionMaterial()]
)
occluder.position = SIMD3(0, 0, 0)
anchor.addChild(occluder)

Environment Lighting

环境光照

swift
// Image-based lighting
if let resource = try? await EnvironmentResource(named: "studio_lighting") {
    // Apply via RealityView content
}

swift
// 基于图像的光照
if let resource = try? await EnvironmentResource(named: "studio_lighting") {
    // 通过RealityView content应用
}

9. Physics and Collision

9. 物理与碰撞

Collision Shapes

碰撞形状

swift
// Generate from mesh (accurate but expensive)
entity.generateCollisionShapes(recursive: true)

// Manual shapes (prefer for performance)
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [
        .generateBox(size: SIMD3(0.1, 0.2, 0.1)),     // Box
        .generateSphere(radius: 0.1),                   // Sphere
        .generateCapsule(height: 0.3, radius: 0.05)     // Capsule
    ]
)
swift
// 从网格生成(精确但性能开销大)
entity.generateCollisionShapes(recursive: true)

// 手动创建形状(优先选择以提升性能)
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [
        .generateBox(size: SIMD3(0.1, 0.2, 0.1)),     // 盒子
        .generateSphere(radius: 0.1),                   // 球体
        .generateCapsule(height: 0.3, radius: 0.05)     // 胶囊体
    ]
)

Physics Body

物理体

swift
// Dynamic — physics simulation controls movement
entity.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    massProperties: .init(mass: 1.0),
    material: .generate(staticFriction: 0.5,
                        dynamicFriction: 0.3,
                        restitution: 0.4),
    mode: .dynamic
)

// Static — immovable collision surface
ground.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    mode: .static
)

// Kinematic — code-controlled, participates in collisions
platform.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    mode: .kinematic
)
swift
// 动态 — 物理模拟控制移动
entity.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    massProperties: .init(mass: 1.0),
    material: .generate(staticFriction: 0.5,
                        dynamicFriction: 0.3,
                        restitution: 0.4),
    mode: .dynamic
)

// 静态 — 不可移动的碰撞表面
ground.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    mode: .static
)

// 运动学 — 代码控制,参与碰撞
platform.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    mode: .kinematic
)

Collision Groups and Filters

碰撞组与过滤器

swift
// Define groups
let playerGroup = CollisionGroup(rawValue: 1 << 0)
let enemyGroup = CollisionGroup(rawValue: 1 << 1)
let bulletGroup = CollisionGroup(rawValue: 1 << 2)

// Filter: player collides with enemies and bullets
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [.generateSphere(radius: 0.1)],
    filter: CollisionFilter(
        group: playerGroup,
        mask: enemyGroup | bulletGroup
    )
)
swift
// 定义组
let playerGroup = CollisionGroup(rawValue: 1 << 0)
let enemyGroup = CollisionGroup(rawValue: 1 << 1)
let bulletGroup = CollisionGroup(rawValue: 1 << 2)

// 过滤器:玩家与敌人、子弹碰撞
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [.generateSphere(radius: 0.1)],
    filter: CollisionFilter(
        group: playerGroup,
        mask: enemyGroup | bulletGroup
    )
)

Collision Events

碰撞事件

swift
// Subscribe in RealityView make closure or System
scene.subscribe(to: CollisionEvents.Began.self, on: playerEntity) { event in
    let otherEntity = event.entityA == playerEntity ? event.entityB : event.entityA
    handleCollision(with: otherEntity)
}
swift
// 在RealityView的make闭包或System中订阅
scene.subscribe(to: CollisionEvents.Began.self, on: playerEntity) { event in
    let otherEntity = event.entityA == playerEntity ? event.entityB : event.entityA
    handleCollision(with: otherEntity)
}

Applying Forces

施加力

swift
if var motion = entity.components[PhysicsMotionComponent.self] {
    motion.linearVelocity = SIMD3(0, 5, 0)  // Impulse up
    entity.components[PhysicsMotionComponent.self] = motion
}

swift
if var motion = entity.components[PhysicsMotionComponent.self] {
    motion.linearVelocity = SIMD3(0, 5, 0)  // 向上的冲量
    entity.components[PhysicsMotionComponent.self] = motion
}

10. Animation

10. 动画

Transform Animation

变换动画

swift
// Animate to position over duration
entity.move(
    to: Transform(
        scale: SIMD3(repeating: 1.5),
        rotation: simd_quatf(angle: .pi, axis: SIMD3(0, 1, 0)),
        translation: SIMD3(0, 2, 0)
    ),
    relativeTo: entity.parent,
    duration: 2.0,
    timingFunction: .easeInOut
)
swift
// 在指定时长内动画到目标位置
entity.move(
    to: Transform(
        scale: SIMD3(repeating: 1.5),
        rotation: simd_quatf(angle: .pi, axis: SIMD3(0, 1, 0)),
        translation: SIMD3(0, 2, 0)
    ),
    relativeTo: entity.parent,
    duration: 2.0,
    timingFunction: .easeInOut
)

Playing USD Animations

播放USD动画

swift
if let entity = try? await Entity(named: "character") {
    // Play all available animations
    for animation in entity.availableAnimations {
        entity.playAnimation(animation.repeat())
    }
}
swift
if let entity = try? await Entity(named: "character") {
    // 播放所有可用动画
    for animation in entity.availableAnimations {
        entity.playAnimation(animation.repeat())
    }
}

Animation Playback Control

动画播放控制

swift
let controller = entity.playAnimation(animation)
controller.pause()
controller.resume()
controller.speed = 2.0      // 2x playback speed
controller.blendFactor = 0.5 // Blend with current state

swift
let controller = entity.playAnimation(animation)
controller.pause()
controller.resume()
controller.speed = 2.0      // 2倍速播放
controller.blendFactor = 0.5 // 与当前状态混合

11. Audio

11. 音频

Spatial Audio

空间音频

swift
// Load audio resource
let resource = try! AudioFileResource.load(named: "engine.wav",
    configuration: .init(shouldLoop: true))

// Create entity with spatial audio
let audioEntity = Entity()
audioEntity.components[SpatialAudioComponent.self] = SpatialAudioComponent()
let controller = audioEntity.playAudio(resource)

// Position the audio source in 3D space
audioEntity.position = SIMD3(2, 0, -1)
swift
// 加载音频资源
let resource = try! AudioFileResource.load(named: "engine.wav",
    configuration: .init(shouldLoop: true))

// 创建带空间音频的实体
let audioEntity = Entity()
audioEntity.components[SpatialAudioComponent.self] = SpatialAudioComponent()
let controller = audioEntity.playAudio(resource)

// 在3D空间中定位音频源
audioEntity.position = SIMD3(2, 0, -1)

Ambient Audio

环境音频

swift
entity.components[AmbientAudioComponent.self] = AmbientAudioComponent()
entity.playAudio(backgroundMusic)

swift
entity.components[AmbientAudioComponent.self] = AmbientAudioComponent()
entity.playAudio(backgroundMusic)

12. Performance

12. 性能优化

Entity Count

实体数量

  • Under 100 entities: No concerns
  • 100-1000 entities: Monitor with RealityKit debugger
  • 1000+ entities: Use instancing and LOD strategies
  • 少于100个实体:无需担心
  • 100-1000个实体:使用RealityKit调试器监控
  • 1000+个实体:使用实例化和LOD策略

Instancing

实例化

swift
// Share mesh and material across many entities
let sharedMesh = MeshResource.generateSphere(radius: 0.01)
let sharedMaterial = SimpleMaterial(color: .white, isMetallic: false)

for i in 0..<1000 {
    let entity = ModelEntity(mesh: sharedMesh, materials: [sharedMaterial])
    entity.position = randomPosition()
    parent.addChild(entity)
}
RealityKit automatically batches entities with identical mesh and material resources.
swift
// 为多个实体共享网格和材质
let sharedMesh = MeshResource.generateSphere(radius: 0.01)
let sharedMaterial = SimpleMaterial(color: .white, isMetallic: false)

for i in 0..<100 {
    let entity = ModelEntity(mesh: sharedMesh, materials: [sharedMaterial])
    entity.position = randomPosition()
    parent.addChild(entity)
}
RealityKit会自动批处理具有相同网格和材质资源的实体。

Component Churn

组件频繁更新

Anti-pattern: Creating and replacing components every frame.
swift
// BAD — component allocation every frame
func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
        entity.components[ModelComponent.self] = ModelComponent(
            mesh: .generateBox(size: 0.1),
            materials: [newMaterial]  // New allocation every frame
        )
    }
}

// GOOD — modify existing component
func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
        // Only update when actually needed
        if needsUpdate {
            var model = entity.components[ModelComponent.self]!
            model.materials = [cachedMaterial]
            entity.components[ModelComponent.self] = model
        }
    }
}
反模式:每帧创建和替换组件。
swift
// 错误示例 — 每帧分配组件
func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
        entity.components[ModelComponent.self] = ModelComponent(
            mesh: .generateBox(size: 0.1),
            materials: [newMaterial]  // 每帧新分配
        )
    }
}

// 正确示例 — 修改现有组件
func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
        // 仅在需要时更新
        if needsUpdate {
            var model = entity.components[ModelComponent.self]!
            model.materials = [cachedMaterial]
            entity.components[ModelComponent.self] = model
        }
    }
}

Collision Shape Optimization

碰撞形状优化

  • Use simple shapes (box, sphere, capsule) instead of mesh-based collision
  • generateCollisionShapes(recursive: true)
    is convenient but expensive
  • For static geometry, generate shapes once during setup
  • 使用简单形状(盒子、球体、胶囊体)替代基于网格的碰撞
  • generateCollisionShapes(recursive: true)
    便捷但性能开销大
  • 对于静态几何体,在初始化时一次性生成形状

Profiling

性能分析

Use Xcode's RealityKit debugger:
  • Entity Inspector: View entity hierarchy and components
  • Statistics Overlay: Entity count, draw calls, triangle count
  • Physics Visualization: Show collision shapes

使用Xcode的RealityKit调试器:
  • 实体检查器:查看实体层级和组件
  • 统计面板:实体数量、绘制调用、三角形数量
  • 物理可视化:显示碰撞形状

13. Multiplayer

13. 多人协作

Synchronization Basics

同步基础

swift
// Components sync automatically if they conform to Codable
struct ScoreComponent: Component, Codable {
    var points: Int
}

// SynchronizationComponent controls what syncs
entity.components[SynchronizationComponent.self] = SynchronizationComponent()
swift
// 符合Codable的组件会自动同步
struct ScoreComponent: Component, Codable {
    var points: Int
}

// SynchronizationComponent控制同步内容
entity.components[SynchronizationComponent.self] = SynchronizationComponent()

MultipeerConnectivityService

MultipeerConnectivityService

swift
let service = try MultipeerConnectivityService(session: mcSession)
// Entities with SynchronizationComponent auto-sync across peers
swift
let service = try MultipeerConnectivityService(session: mcSession)
// 带有SynchronizationComponent的实体会在设备间自动同步

Ownership

所有权

  • Only the owner of an entity can modify it
  • Request ownership before modifying shared entities
  • Non-Codable component data does not sync

  • 只有实体的所有者可以修改它
  • 修改共享实体前请求所有权
  • 非Codable组件数据不会同步

14. Anti-Patterns

14. 反模式

Anti-Pattern 1: UIKit-Style Thinking in ECS

反模式1:ECS中使用UIKit风格思维

Time cost: Hours of frustration from fighting the architecture
swift
// BAD — subclassing Entity for behavior
class PlayerEntity: Entity {
    func takeDamage(_ amount: Int) { /* logic in entity */ }
}

// GOOD — component holds data, system has logic
struct HealthComponent: Component { var hp: Int }
struct DamageSystem: System {
    static let query = EntityQuery(where: .has(HealthComponent.self))
    func update(context: SceneUpdateContext) {
        // Process damage here
    }
}
时间成本:与架构对抗导致数小时的挫败
swift
// 错误示例 — 为实体子类添加行为
class PlayerEntity: Entity {
    func takeDamage(_ amount: Int) { /* 逻辑在实体中 */ }
}

// 正确示例 — 组件存储数据,系统处理逻辑
struct HealthComponent: Component { var hp: Int }
struct DamageSystem: System {
    static let query = EntityQuery(where: .has(HealthComponent.self))
    func update(context: SceneUpdateContext) {
        // 在这里处理伤害逻辑
    }
}

Anti-Pattern 2: Monolithic Entities

反模式2:单体实体

Time cost: Untestable, inflexible architecture
Don't put all game logic in one entity type. Split into components that can be mixed and matched.
时间成本:不可测试、架构僵化
不要将所有游戏逻辑放在一个实体类型中。拆分为可组合的组件。

Anti-Pattern 3: Frame-Based Updates Without Systems

反模式3:不使用系统的帧更新

Time cost: Missed frame updates, inconsistent behavior
swift
// BAD — timer-based updates
Timer.scheduledTimer(withTimeInterval: 1/60, repeats: true) { _ in
    entity.position.x += 0.01
}

// GOOD — System update
struct MovementSystem: System {
    static let query = EntityQuery(where: .has(VelocityComponent.self))
    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            let velocity = entity.components[VelocityComponent.self]!
            entity.position += velocity.value * Float(context.deltaTime)
        }
    }
}
时间成本:错过帧更新、行为不一致
swift
// 错误示例 — 基于定时器的更新
Timer.scheduledTimer(withTimeInterval: 1/60, repeats: true) { _ in
    entity.position.x += 0.01
}

// 正确示例 — 系统更新
struct MovementSystem: System {
    static let query = EntityQuery(where: .has(VelocityComponent.self))
    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            let velocity = entity.components[VelocityComponent.self]!
            entity.position += velocity.value * Float(context.deltaTime)
        }
    }
}

Anti-Pattern 4: Not Generating Collision Shapes for Interactive Entities

反模式4:交互实体不生成碰撞形状

Time cost: 15-30 min debugging "why taps don't work"
Gestures require
CollisionComponent
. If an entity has
InputTargetComponent
(visionOS) or
ManipulationComponent
but no
CollisionComponent
, gestures will never fire.
时间成本:15-30分钟调试“为什么手势不生效”
手势需要
CollisionComponent
。如果实体有
InputTargetComponent
(visionOS)或
ManipulationComponent
但没有
CollisionComponent
,手势永远不会触发。

Anti-Pattern 5: Storing Entity References in Systems

反模式5:在系统中存储实体引用

Time cost: Crashes from stale references
swift
// BAD — entity might be removed between frames
struct BadSystem: System {
    var playerEntity: Entity?  // Stale reference risk

    func update(context: SceneUpdateContext) {
        playerEntity?.position.x += 0.1  // May crash
    }
}

// GOOD — query each frame
struct GoodSystem: System {
    static let query = EntityQuery(where: .has(PlayerComponent.self))

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            entity.position.x += Float(context.deltaTime)
        }
    }
}

时间成本:引用失效导致崩溃
swift
// 错误示例 — 实体可能在帧间被移除
struct BadSystem: System {
    var playerEntity: Entity?  // 引用失效风险

    func update(context: SceneUpdateContext) {
        playerEntity?.position.x += 0.1  // 可能崩溃
    }
}

// 正确示例 — 每帧查询
struct GoodSystem: System {
    static let query = EntityQuery(where: .has(PlayerComponent.self))

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            entity.position.x += Float(context.deltaTime)
        }
    }
}

15. Code Review Checklist

15. 代码审查清单

  • Custom components registered via
    registerComponent()
    before use
  • Systems registered via
    registerSystem()
    before scene loads
  • Components are value types (structs), not classes
  • Read-modify-write pattern used for component updates
  • Interactive entities have
    CollisionComponent
  • visionOS interactive entities have
    InputTargetComponent
  • Collision shapes are simple (box/sphere/capsule) where possible
  • No entity references stored across frames in Systems
  • Mesh and material resources shared across identical entities
  • Component updates only occur when values actually change
  • USD/USDZ format used for 3D assets (not .scn)
  • Async loading used for all model/scene loading
  • [weak self]
    in closure-based subscriptions if retaining view/controller

  • 自定义组件在使用前通过
    registerComponent()
    注册
  • 系统在场景加载前通过
    registerSystem()
    注册
  • 组件是值类型(结构体),而非类
  • 组件更新使用读取-修改-写入模式
  • 交互实体带有
    CollisionComponent
  • visionOS交互实体带有
    InputTargetComponent
  • 尽可能使用简单碰撞形状(盒子/球体/胶囊体)
  • 系统中不跨帧存储实体引用
  • 相同实体共享网格和材质资源
  • 仅在值实际变化时更新组件
  • 3D资源使用USD/USDZ格式(而非.scn)
  • 所有模型/场景加载使用异步方式
  • 闭包订阅中如果持有视图/控制器,使用
    [weak self]

16. Pressure Scenarios

16. 压力场景

Scenario 1: "ECS Is Overkill for Our Simple App"

场景1:“ECS对我们的简单应用来说太复杂”

Pressure: Team wants to avoid learning ECS, just needs one 3D model displayed
Wrong approach: Skip ECS, jam all logic into RealityView closures.
Correct approach: Even simple apps benefit from ECS. A single
ModelEntity
in a
RealityView
is already using ECS — you're just not adding custom components yet. Start simple, add components as complexity grows.
Push-back template: "We're already using ECS — Entity and ModelComponent. The pattern scales. Adding a custom component when we need behavior is one struct definition, not an architecture change."
压力:团队不想学习ECS,只需要展示一个3D模型
错误做法:跳过ECS,将所有逻辑塞进RealityView闭包。
正确做法:即使简单应用也能从ECS获益。RealityView中的单个
ModelEntity
已经在使用ECS——只是还没添加自定义组件。从简单开始,随着复杂度增加再添加组件。
反驳模板:“我们已经在使用ECS了——Entity和ModelComponent就是ECS的一部分。这种模式具备扩展性。当我们需要行为时,只需要定义一个结构体组件,而非改变架构。”

Scenario 2: "Just Use SceneKit, We Know It"

场景2:“就用SceneKit吧,我们熟悉它”

Pressure: Team has SceneKit experience, RealityKit is unfamiliar
Wrong approach: Build new features in SceneKit.
Correct approach: SceneKit is soft-deprecated. New features won't be added. Invest in RealityKit now — the ECS concepts transfer to other game engines (Unity, Unreal, Bevy) if needed.
Push-back template: "SceneKit is in maintenance mode — no new features, only security patches. Every line of SceneKit we write is migration debt. RealityKit's concepts (Entity, Component, System) are industry-standard ECS."
压力:团队有SceneKit经验,RealityKit不熟悉
错误做法:用SceneKit开发新功能。
正确做法:SceneKit已进入维护模式,不会再添加新功能。现在投入RealityKit——ECS概念可迁移到其他游戏引擎(Unity、Unreal、Bevy)。
反驳模板:“SceneKit处于维护模式——不再添加新功能,仅提供安全补丁。我们写的每一行SceneKit代码都是技术债务。RealityKit的概念(实体、组件、系统)是行业标准的ECS。”

Scenario 3: "Make It Work Without Collision Shapes"

场景3:“不用碰撞形状让它工作起来”

Pressure: Deadline, collision shape setup seems complex
Wrong approach: Skip collision shapes, use position-based proximity detection.
Correct approach:
entity.generateCollisionShapes(recursive: true)
takes one line. Without it, gestures won't work and physics won't collide. The "shortcut" creates more debugging time than it saves.
Push-back template: "Collision shapes are required for gestures and physics. It's one line:
entity.generateCollisionShapes(recursive: true)
. Skipping it means gestures silently fail — a harder bug to diagnose."

压力:截止日期临近,碰撞形状设置看起来复杂
错误做法:跳过碰撞形状,使用基于位置的 proximity 检测。
正确做法
entity.generateCollisionShapes(recursive: true)
只需要一行代码。没有它,手势无法工作,物理系统无法碰撞。“捷径”会导致比节省的时间更多的调试时间。
反驳模板:“碰撞形状是手势和物理系统的必需条件。只需要一行代码:
entity.generateCollisionShapes(recursive: true)
。跳过它会导致手势无声失效——这是更难诊断的bug。”

Resources

资源

WWDC: 2019-603, 2019-605, 2021-10074, 2022-10074, 2023-10080, 2023-10081, 2024-10103, 2024-10153
Docs: /realitykit, /realitykit/entity, /realitykit/realityview, /realitykit/modelentity, /realitykit/anchorentity, /realitykit/component
Skills: axiom-realitykit-ref, axiom-realitykit-diag, axiom-scenekit, axiom-scenekit-ref
WWDC:2019-603, 2019-605, 2021-10074, 2022-10074, 2023-10080, 2023-10081, 2024-10103, 2024-10153
文档:/realitykit, /realitykit/entity, /realitykit/realityview, /realitykit/modelentity, /realitykit/anchorentity, /realitykit/component
技能:axiom-realitykit-ref, axiom-realitykit-diag, axiom-scenekit, axiom-scenekit-ref