realitykit-visionos-developer

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

RealityKit visionOS Developer

RealityKit visionOS 开发者指南

Description and Goals

说明与目标

This skill provides comprehensive guidance for implementing RealityKit-based spatial experiences on visionOS. RealityKit uses an Entity Component System (ECS) architecture where entities are lightweight containers, behavior comes from components, and systems drive per-frame updates.

本技能为在visionOS上实现基于RealityKit的空间体验提供全面指导。RealityKit采用实体组件系统（ECS）架构，其中实体是轻量级容器，行为由组件提供，系统驱动每帧更新。

Goals

目标

Enable developers to build immersive 3D experiences on visionOS using RealityKit
Provide clear guidance on when to use each component and system
Help developers understand ECS patterns and best practices
Support debugging and optimization of RealityKit scenes
Ensure proper integration with SwiftUI via RealityView

帮助开发者使用RealityKit在visionOS上构建沉浸式3D体验
清晰指导何时使用各类组件与系统
帮助开发者理解ECS模式与最佳实践
支持RealityKit场景的调试与优化
确保通过RealityView与SwiftUI的正确集成

What This Skill Should Do

本技能的适用场景

When implementing RealityKit features on visionOS, this skill should:

Guide component selection - Help you choose the right components for rendering, interaction, physics, audio, and animation needs
Provide system implementation patterns - Show how to create custom systems for continuous behavior
Offer code examples - Demonstrate common patterns like async asset loading, interactive entities, and custom systems
Highlight best practices - Emphasize proper async loading, component registration, and performance considerations
Warn about pitfalls - Identify common mistakes like using ARView on visionOS or blocking the main actor

Load the appropriate component or system reference file from the tables below for detailed usage, code examples, and best practices.

在visionOS上实现RealityKit功能时，本技能可：

指导组件选择 - 帮助你根据渲染、交互、物理、音频和动画需求选择合适的组件
提供系统实现模式 - 展示如何创建自定义系统以实现持续行为
提供代码示例 - 演示异步资源加载、交互式实体和自定义系统等常见模式
强调最佳实践 - 重点说明异步加载、组件注册和性能考量的正确方式
警示常见陷阱 - 指出在visionOS上使用ARView或阻塞主actor等常见错误

从下方表格中加载对应的组件或系统参考文件，获取详细用法、代码示例和最佳实践。

Information About the Skill

技能相关信息

Core Concepts

核心概念

Entities and Components

实体与组件

Entities are lightweight containers; behavior comes from components.
Prefer composition over inheritance and use custom
```
Component
```
+
```
Codable
```
when you need per-entity state.
Register custom components once with
```
Component.registerComponent()
```
before use.
Keep entity transforms and component updates on the main actor.

实体是轻量级容器，行为由组件提供。
优先使用组合而非继承，当需要每个实体的独立状态时，使用自定义
```
Component
```
+
```
Codable
```
。
使用自定义组件前，需通过
```
Component.registerComponent()
```
注册一次。
实体变换和组件更新需在主actor上执行。

RealityView and Attachments

RealityView与附件

Use
```
RealityView
```
to bridge SwiftUI and RealityKit.
Load assets with
```
Entity(named:)
```
or
```
Entity(contentsOf:)
```
asynchronously and handle errors.
Always use
```
ViewAttachmentComponent
```
for SwiftUI overlays in 3D and avoid the
```
RealityView
```
attachments closure.

使用
```
RealityView
```
桥接SwiftUI与RealityKit。
异步使用
```
Entity(named:)
```
或
```
Entity(contentsOf:)
```
加载资源并处理错误。
在3D空间中嵌入SwiftUI叠加层时，始终使用
```
ViewAttachmentComponent
```
，避免使用
```
RealityView
```
的附件闭包。

Systems and Queries

系统与查询

Use a custom
```
System
```
for continuous, per-frame behavior.

Query entities with

EntityQuery

QueryPredicate

and process them in

update(context:)

Use
```
SystemDependency
```
to control update order when multiple systems interact.

使用自定义
```
System
```
实现持续的每帧行为。

通过

EntityQuery

QueryPredicate

查询实体，并在

update(context:)

中处理。

当多个系统交互时，使用
```
SystemDependency
```
控制更新顺序。

Components Reference

组件参考

Use this table to decide which component reference file to load when implementing RealityKit features:

在实现RealityKit功能时，可通过下表选择对应的组件参考文件：

Rendering and Appearance

渲染与外观

Component	When to Use
`ModelComponent`	When rendering 3D geometry with meshes and materials on entities.
`ModelSortGroupComponent`	When experiencing depth fighting (z-fighting) issues with overlapping geometry or need to control draw order.
`OpacityComponent`	When creating fade effects, making entities semi-transparent, or implementing visibility transitions.
`AdaptiveResolutionComponent`	When optimizing performance in large scenes by reducing render quality for distant objects.
`ModelDebugOptionsComponent`	When debugging rendering issues, visualizing model geometry, or inspecting bounding boxes during development.
`MeshInstancesComponent`	When rendering many copies of the same mesh efficiently (trees, crowds, particle-like objects).
`BlendShapeWeightsComponent`	When implementing facial animation, character expressions, or morphing mesh deformations.

组件	适用场景
`ModelComponent`	在实体上使用网格和材质渲染3D几何图形时。
`ModelSortGroupComponent`	出现重叠几何图形的深度冲突（z-fighting）问题，或需要控制绘制顺序时。
`OpacityComponent`	创建淡入淡出效果、使实体半透明，或实现可见性过渡时。
`AdaptiveResolutionComponent`	通过降低远处对象的渲染质量来优化大型场景性能时。
`ModelDebugOptionsComponent`	调试渲染问题、可视化模型几何图形，或在开发过程中检查边界框时。
`MeshInstancesComponent`	高效渲染大量相同网格副本（如树木、人群、类粒子对象）时。
`BlendShapeWeightsComponent`	实现面部动画、角色表情或网格变形时。

User Interaction

用户交互

Component	When to Use
`InputTargetComponent`	When making entities interactive (tappable, draggable) or handling user input events.
`ManipulationComponent`	When implementing built-in drag, rotate, and scale interactions with hand gestures or trackpad.
`GestureComponent`	When implementing custom gesture recognition beyond what ManipulationComponent provides.
`HoverEffectComponent`	When providing visual feedback when users look at or hover over interactive entities.
`AccessibilityComponent`	When making entities accessible to screen readers, VoiceOver, or other assistive technologies.
`BillboardComponent`	When creating 2D sprites, text labels, or UI elements that should always face the viewer.

组件	适用场景
`InputTargetComponent`	使实体可交互（可点击、可拖拽）或处理用户输入事件时。
`ManipulationComponent`	通过手势或触控板实现内置的拖拽、旋转和缩放交互时。
`GestureComponent`	实现 `ManipulationComponent` 之外的自定义手势识别时。
`HoverEffectComponent`	当用户注视或悬停在交互式实体上时提供视觉反馈时。
`AccessibilityComponent`	使实体适配屏幕阅读器、VoiceOver或其他辅助技术时。
`BillboardComponent`	创建始终面向观察者的2D精灵、文本标签或UI元素时。

Anchoring and Spatial

锚定与空间

Component	When to Use
`AnchoringComponent`	When anchoring virtual content to detected planes, tracked images, hand locations, or world targets.
`ARKitAnchorComponent`	When accessing the underlying ARKit anchor data for an anchored entity.
`SceneUnderstandingComponent`	When accessing scene understanding data like detected objects or room reconstruction.
`DockingRegionComponent`	When defining regions where content can automatically dock or snap into place.
`ReferenceComponent`	When implementing lazy loading of external entity assets or referencing entities in other files.
`AttachedTransformComponent`	When attaching an entity's transform to another entity for hierarchical positioning.

组件	适用场景
`AnchoringComponent`	将虚拟内容锚定到检测平面、跟踪图像、手部位置或世界目标时。
`ARKitAnchorComponent`	访问锚定实体的底层ARKit锚点数据时。
`SceneUnderstandingComponent`	访问场景理解数据（如检测到的对象或房间重建信息）时。
`DockingRegionComponent`	定义内容可自动停靠或对齐的区域时。
`ReferenceComponent`	实现外部实体资源的懒加载，或引用其他文件中的实体时。
`AttachedTransformComponent`	将一个实体的变换附加到另一个实体以实现层级定位时。

Cameras

相机

Component	When to Use
`PerspectiveCameraComponent`	When configuring a perspective camera with depth and field of view for 3D scenes.
`OrthographicCameraComponent`	When configuring an orthographic camera without perspective distortion for 2D-like views.
`ProjectiveTransformCameraComponent`	When implementing custom camera projection transforms for specialized rendering needs.

组件	适用场景
`PerspectiveCameraComponent`	为3D场景配置带深度和视野的透视相机时。
`OrthographicCameraComponent`	配置无透视畸变的正交相机以实现类2D视图时。
`ProjectiveTransformCameraComponent`	为特殊渲染需求实现自定义相机投影变换时。

Lighting and Shadows

光照与阴影

Component	When to Use
`PointLightComponent`	When adding an omnidirectional point light that radiates in all directions from a position.
`DirectionalLightComponent`	When adding a directional light with parallel rays (like sunlight) for consistent scene lighting.
`SpotLightComponent`	When adding a cone-shaped spotlight for focused, directional lighting effects.
`ImageBasedLightComponent`	When applying environment lighting from HDR textures for realistic reflections and ambient lighting.
`ImageBasedLightReceiverComponent`	When enabling entities to receive and respond to image-based lighting in the scene.
`GroundingShadowComponent`	When adding grounding shadows to visually anchor floating content to surfaces.
`DynamicLightShadowComponent`	When enabling real-time dynamic shadows cast by light sources onto entities.
`EnvironmentLightingConfigurationComponent`	When configuring environment lighting behavior, intensity, or blending modes.
`VirtualEnvironmentProbeComponent`	When implementing reflection probes for accurate reflections in virtual environments.

组件	适用场景
`PointLightComponent`	添加从某一位置向所有方向辐射的全向点光源时。
`DirectionalLightComponent`	添加具有平行光线的定向光源（如太阳光）以实现一致的场景照明时。
`SpotLightComponent`	添加锥形聚光灯以实现聚焦的定向照明效果时。
`ImageBasedLightComponent`	通过HDR纹理应用环境光以实现真实反射和环境照明时。
`ImageBasedLightReceiverComponent`	使实体能够接收并响应场景中的基于图像的照明时。
`GroundingShadowComponent`	添加地面阴影以将悬浮内容视觉锚定到表面时。
`DynamicLightShadowComponent`	启用光源投射到实体上的实时动态阴影时。
`EnvironmentLightingConfigurationComponent`	配置环境照明行为、强度或混合模式时。
`VirtualEnvironmentProbeComponent`	实现反射探针以在虚拟环境中获得准确反射时。

Audio

音频

Component	When to Use
`SpatialAudioComponent`	When playing 3D positioned audio that changes based on listener position and orientation.
`AmbientAudioComponent`	When playing non-directional ambient audio that doesn't change with listener position.
`ChannelAudioComponent`	When playing channel-based audio content (stereo, surround, etc.) without spatialization.
`AudioLibraryComponent`	When storing and managing multiple audio resources for reuse across entities.
`ReverbComponent`	When applying reverb effects to an entity's audio for spatial acoustic simulation.
`AudioMixGroupsComponent`	When grouping audio sources for centralized mixing control and volume management.

组件	适用场景
`SpatialAudioComponent`	播放随观察者位置和方向变化的3D定位音频时。
`AmbientAudioComponent`	播放不随观察者位置变化的无方向环境音频时。
`ChannelAudioComponent`	播放基于声道的音频内容（立体声、环绕声等）且无需空间化时。
`AudioLibraryComponent`	存储和管理多个音频资源以便在实体间复用时。
`ReverbComponent`	为实体音频应用混响效果以实现空间声学模拟时。
`AudioMixGroupsComponent`	对音频源进行分组以实现集中混音控制和音量管理时。

Animation and Character

动画与角色

Component	When to Use
`AnimationLibraryComponent`	When storing multiple animations (idle, walk, run) on a single entity for character animation.
`CharacterControllerComponent`	When implementing character movement with physics, collision, and ground detection.
`CharacterControllerStateComponent`	When storing runtime state (velocity, grounded status) for a character controller.
`SkeletalPosesComponent`	When providing skeletal pose data for skeletal animation and bone transformations.
`IKComponent`	When implementing inverse kinematics for procedural animation (e.g., reaching, pointing).
`BodyTrackingComponent`	When integrating ARKit body tracking data to animate entities based on real-world body poses.

组件	适用场景
`AnimationLibraryComponent`	在单个实体上存储多个动画（idle、walk、run）以实现角色动画时。
`CharacterControllerComponent`	实现带物理、碰撞和地面检测的角色移动时。
`CharacterControllerStateComponent`	存储角色控制器的运行时状态（速度、接地状态）时。
`SkeletalPosesComponent`	为骨骼动画和骨骼变换提供骨骼姿态数据时。
`IKComponent`	实现逆运动学以实现程序动画（如伸手、指向）时。
`BodyTrackingComponent`	集成ARKit身体跟踪数据以根据真实世界身体姿态驱动实体动画时。

Physics and Collision

物理与碰撞

Component	When to Use
`CollisionComponent`	When defining collision shapes for hit testing, raycasting, or physics interactions.
`PhysicsBodyComponent`	When adding physical behavior (mass, gravity, forces) to entities for physics simulation.
`PhysicsMotionComponent`	When controlling linear and angular velocity of physics bodies programmatically.
`PhysicsSimulationComponent`	When configuring global physics simulation parameters like gravity or timestep.
`ParticleEmitterComponent`	When emitting particle effects (smoke, sparks, debris) from an entity position.
`ForceEffectComponent`	When applying force fields (gravity wells, explosions) that affect multiple physics bodies.
`PhysicsJointsComponent`	When creating joints (hinges, springs) between physics bodies for articulated structures.
`GeometricPinsComponent`	When defining geometric attachment points for connecting entities at specific locations.

组件	适用场景
`CollisionComponent`	为命中测试、射线检测或物理交互定义碰撞形状时。
`PhysicsBodyComponent`	为实体添加物理行为（质量、重力、力）以进行物理模拟时。
`PhysicsMotionComponent`	以编程方式控制物理体的线速度和角速度时。
`PhysicsSimulationComponent`	配置全局物理模拟参数（如重力、时间步长）时。
`ParticleEmitterComponent`	从实体位置发射粒子效果（烟雾、火花、碎片）时。
`ForceEffectComponent`	应用力场（重力井、爆炸）以影响多个物理体时。
`PhysicsJointsComponent`	在物理体之间创建关节（铰链、弹簧）以实现铰接结构时。
`GeometricPinsComponent`	定义几何附着点以在特定位置连接实体时。

Portals and Environments

门户与环境

Component	When to Use
`PortalComponent`	When creating portals that render a separate world or scene through an opening.
`WorldComponent`	When designating an entity hierarchy as a separate renderable world for portal rendering.
`PortalCrossingComponent`	When controlling behavior (teleportation, scene switching) when entities cross portal boundaries.
`EnvironmentBlendingComponent`	When blending virtual content with the real environment for mixed reality experiences.

组件	适用场景
`PortalComponent`	创建可通过开口渲染独立世界或场景的门户时。
`WorldComponent`	将实体层级指定为门户渲染用的独立可渲染世界时。
`PortalCrossingComponent`	控制实体穿越门户边界时的行为（传送、场景切换）时。
`EnvironmentBlendingComponent`	将虚拟内容与真实环境混合以实现混合现实体验时。

Presentation and UI

展示与UI

Component	When to Use
`ViewAttachmentComponent`	When embedding SwiftUI views into 3D space for interactive UI elements or labels.
`PresentationComponent`	When presenting SwiftUI modals, sheets, or system UI from an entity interaction.
`TextComponent`	When rendering 3D text directly on entities without using SwiftUI views.
`ImagePresentationComponent`	When displaying images or textures on entities in 3D space.
`VideoPlayerComponent`	When playing video content on entity surfaces using AVPlayer.

组件	适用场景
`ViewAttachmentComponent`	将SwiftUI视图嵌入3D空间以实现交互式UI元素或标签时。
`PresentationComponent`	从实体交互中展示SwiftUI模态框、表单或系统UI时。
`TextComponent`	无需使用SwiftUI视图，直接在实体上渲染3D文本时。
`ImagePresentationComponent`	在3D空间中的实体上显示图像或纹理时。
`VideoPlayerComponent`	使用AVPlayer在实体表面播放视频内容时。

Networking and Sync

网络与同步

Component	When to Use
`SynchronizationComponent`	When synchronizing entity state, transforms, and components across networked multiplayer sessions.
`TransientComponent`	When marking entities as temporary, non-persistent, and excluded from network synchronization.

组件	适用场景
`SynchronizationComponent`	在联网多人会话中同步实体状态、变换和组件时。
`TransientComponent`	将实体标记为临时、非持久化且排除在网络同步外时。

Systems Reference

系统参考

Use this reference when implementing custom ECS behavior:

System/API	When to Use
`System and Component Creation`	When creating custom systems for continuous, per-frame behavior or custom components for per-entity state.

在实现自定义ECS行为时，可参考以下内容：

系统/API	适用场景
`System and Component Creation`	创建自定义系统以实现持续的每帧行为，或创建自定义组件以存储每个实体的状态时。

Implementation Patterns

实现模式

RealityView Async Load

RealityView 异步加载

swift

RealityView { content in
    do {
        let entity = try await Entity(named: "Scene")
        content.add(entity)
    } catch {
        print("Failed to load entity: \(error)")
    }
}

swift

RealityView { content in
    do {
        let entity = try await Entity(named: "Scene")
        content.add(entity)
    } catch {
        print("Failed to load entity: \(error)")
    }
}

Interactive Entity Setup

交互式实体设置

swift

let entity = ModelEntity(mesh: .generateBox(size: 0.1))
entity.components.set(CollisionComponent(shapes: [.generateBox(size: [0.1, 0.1, 0.1])]))
entity.components.set(InputTargetComponent())
entity.components.set(ManipulationComponent())

swift

let entity = ModelEntity(mesh: .generateBox(size: 0.1))
entity.components.set(CollisionComponent(shapes: [.generateBox(size: [0.1, 0.1, 0.1])]))
entity.components.set(InputTargetComponent())
entity.components.set(ManipulationComponent())

Custom System Skeleton

自定义系统框架

swift

import RealityKit

struct SpinComponent: Component, Codable {
    var speed: Float
}

struct SpinSystem: System {
    static let query = EntityQuery(where: .has(SpinComponent.self))

    init(scene: Scene) {}

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
            guard let spin = entity.components[SpinComponent.self] else { continue }
            entity.transform.rotation *= simd_quatf(angle: spin.speed * Float(context.deltaTime), axis: [0, 1, 0])
        }
    }
}

SpinSystem.registerSystem()

swift

import RealityKit

struct SpinComponent: Component, Codable {
    var speed: Float
}

struct SpinSystem: System {
    static let query = EntityQuery(where: .has(SpinComponent.self))

    init(scene: Scene) {}

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
            guard let spin = entity.components[SpinComponent.self] else { continue }
            entity.transform.rotation *= simd_quatf(angle: spin.speed * Float(context.deltaTime), axis: [0, 1, 0])
        }
    }
}

SpinSystem.registerSystem()

Pitfalls and Checks

常见陷阱与检查项

Always load assets asynchronously; avoid blocking the main actor.
Avoid
```
ARView
```
on visionOS; use
```
RealityView
```
.
Add
```
CollisionComponent
```
+
```
InputTargetComponent
```
for draggable or tappable entities.
Never use the
```
RealityView
```
update closure; use a custom
```
System
```
for continuous behavior instead.
Mesh generation is limited to
```
box
```
,
```
sphere
```
,
```
plane
```
,
```
cylinder
```
, and
```
cone
```
.

始终异步加载资源；避免阻塞主actor。
在visionOS上避免使用
```
ARView
```
；使用
```
RealityView
```
。
为可拖拽或可点击的实体添加
```
CollisionComponent
```
+
```
InputTargetComponent
```
。
切勿使用
```
RealityView
```
的更新闭包；改用自定义
```
System
```
实现持续行为。
网格生成仅支持
```
box
```
、
```
sphere
```
、
```
plane
```
、
```
cylinder
```
和
```
cone
```
。