Back to Details

google-cloud-solution-agentic-analytics-spark-knowledge-catalog

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Agentic analytics across cloud providers and data types

跨云服务商与数据类型的代理式分析（Agentic analytics）

This skill provides a workflow to design and implement a governed, secure pipeline for agentic analytics solution across structured and unstructured data that's distributed across Google Cloud, on-premises systems, and other cloud providers.
本技能提供一套工作流，用于设计和实现一个受管控、安全的代理式分析解决方案管道，处理分布在Google Cloud、本地系统及其他云服务商中的结构化与非结构化数据。

Overview of the workflow

工作流概述

The workflow consists of the following phases:
  • Phase 1: Requirements discovery. Gather detailed requirements related to the cloud workload or use case that the user needs assistance for.
  • Phase 2: Solution architecture. Use the requirements that were gathered in Phase 1 to generate a detailed solution architecture for the cloud workload or use case.
  • Phase 3: Solution validation. Create a plan to validate the generated solution, generate validation instructions and scripts, and run the validation.
  • Phase 4: Solution packing and presentation. Consolidate the generated content and present the solution.
Important notes about the workflow:
  • Strict phase separation: During Phase 1 (Requirements discovery), when you ask the user clarifying questions, DON'T recommend, propose, or outline any architectural designs, technical decompositions, cloud services, or component mappings.
  • When you can skip certain phases: If the user's prompt indicates that a specific phase or task in this workflow is already completed or approved (e.g., "requirements discovery stage is completed", "product selection is approved", or "architecture is confirmed"), DON'T repeat that phase or task. Instead, skip directly to the requested task (such as generating the technical decomposition, recommending products, or compiling the solution guide).
工作流包含以下阶段：
  • 阶段1：需求发现。收集与用户需要协助的云工作负载或用例相关的详细需求。
  • 阶段2：解决方案架构。利用阶段1收集的需求，为云工作负载或用例生成详细的解决方案架构。
  • 阶段3：解决方案验证。创建计划以验证生成的解决方案，生成验证说明和脚本，并执行验证。
  • 阶段4：解决方案打包与展示。整合生成的内容并展示解决方案。
工作流重要说明：
  • 严格的阶段分离：在阶段1（需求发现）中，当您向用户询问澄清问题时，不得推荐、提议或概述任何架构设计、技术分解、云服务或组件映射。
  • 可跳过特定阶段的情况：如果用户的提示表明此工作流中的某个特定阶段或任务已完成或获批（例如“需求发现阶段已完成”、“产品选择已获批”或“架构已确认”），请勿重复该阶段或任务。而是直接跳转到请求的任务（如生成技术分解、推荐产品或编制解决方案指南）。

Phase 1: Requirements discovery and analysis

阶段1：需求发现与分析

  1. Request the user to describe the functional requirements (business processes, activities, and use cases) of their workload. Ask the user the following questions, one question at a time:
    • What are your primary inventory data sources? Are they unstructured (e.g., PDF flavor recipes, invoices) or structured (e.g., historical sales in Iceberg)?
    • Where are these sources hosted? Are they split across AWS S3, Azure Blob, Google Cloud Storage, or databases like AlloyDB?
    • How do you manage and federate metadata across your data sources within Google Cloud and in external locations (such as other cloud providers)?
    • What are your analytical and computational requirements to join, clean, and run forecast models over large-scale distributed data?
    • What types of natural language prompts do your data scientists or operational agents expect to execute in their agentic IDE (VS Code or Antigravity IDE)?
  2. Request the user to describe the non-functional requirements of their workload.
    The following are examples of questions you can ask to gather non-functional requirements:
    • Security, privacy, and compliance: What data privacy rules, regulatory compliance (e.g., GDPR, HIPAA), or data governance requirements must the system adhere to?
    • Reliability: What are your uptime, high-availability, fault-tolerance, and disaster recovery objectives (RTO/RPO)?
    • Performance: What target query latencies and SLA expectations does your workload require?
    • Operations: What operational monitoring metrics do your data scientists and engineers need?
    • Cost & Sustainability: Do you have specific budget constraints and data egress/transfer cost requirements?
  3. Ask the user whether the workload currently runs on other cloud providers or on-premises.
    • If the user answers "yes", then ask the user to describe the architecture of the current deployment.
    • If the user answer "no", then proceed to the next step.
  4. Request the user to describe dependencies, if any, on other workloads, products, or tools. The following are examples of questions that you can ask to get information about the dependencies:
    • Do you have any upstream or downstream dependencies on external systems (e.g., identity providers, data curation platforms, CI/CD pipelines, or active data catalogs)?
    • Are there any requirements for your general data-engineering software delivery lifecycle (e.g., version control, testing, data quality assurance)? Provide the path to a directory or examples of these artifacts.
  5. Review the input that the user has provided so far, and check whether there are any ambiguities or contradictions.
    If you identify any ambiguities or contradictions in the requirements that the user has provided (e.g., zero-copy vs copying data to a repository), then do the following for each ambiguity or contradiction that you identify:
    • Describe the ambiguity or contradiction (e.g., explain why copying data contradicts the zero-copy requirement and also incurs data-transfer costs).
    • Ask the user how they wish to resolve the ambiguity or contradiction.
      • If the user delegates the choice to you (e.g., the user replies with "do what you think is best" or "you decide"), then provide a clear suggestion to resolve the ambiguity or contradiction (e.g., suggest prioritizing zero-copy remote queries), explain your reasoning (e.g., to eliminate multi-cloud fees and data duplication), and ask the user to approve your suggestion.
    Critical: Until all the ambiguities and contradictions that you identify are resolved according to the preceding guidance, you must NOT recommend or generate any architecture design, technical decomposition, or Google Cloud product recommendations.
  6. Important: DON'T start this step if there are unresolved contradictions or ambiguities from Step 5.
    Generate a technical decomposition of the components of the workload.
    • The technical decomposition must break down the solution into logical components.
    • The decomposition MUST address role-based security and credentials within the relevant layers.
    • The decomposition MUST be organized under the following four layers, which represent a standard architectural pattern for agentic analytics solutions, flowing from user interaction through data context and governance to core data processing:
      • User-interaction layer (IDE): e.g., agentic development environment.
      • Grounding and trusted data: e.g., foundation model, MCP servers, and data warehouse in the cloud.
      • Metadata curation: e.g., metadata scanning.
      • Data processing and analytics: e.g., analytics workflows, Spark data processing, and external data stores.
  7. Request the user to approve the generated technical decomposition.
  8. If the user requests changes, then generate an updated technical decomposition.
  9. Repeat steps 5 through 8 until the user approves the generated technical decomposition.
  10. After the user approves the technical decomposition, proceed to Phase 2. Important: Don't proceed to the next phase until the user approves the generated technical decomposition of the workload.
  1. 请求用户描述其工作负载的功能需求（业务流程、活动和用例）。逐个向用户询问以下问题：
    • 您的主要库存数据源是什么？是非结构化数据（如PDF格式的配方、发票）还是结构化数据（如Iceberg中的历史销售数据）？
    • 这些数据源托管在何处？是否分布在AWS S3、Azure Blob、Google Cloud Storage或AlloyDB等数据库中？
    • 您如何在Google Cloud内部和外部位置（如其他云服务商）管理和联合跨数据源的元数据？
    • 针对大规模分布式数据进行关联、清洗和运行预测模型，您有哪些分析和计算需求？
    • 您的数据科学家或运营代理期望在其代理式IDE（VS Code或Antigravity IDE）中执行哪些类型的自然语言提示？
  2. 请求用户描述其工作负载的非功能需求。
    以下是可用于收集非功能需求的示例问题：
    • 安全、隐私与合规：系统必须遵守哪些数据隐私规则、监管合规要求（如GDPR、HIPAA）或数据治理要求？
    • 可靠性：您的正常运行时间、高可用性、容错和灾难恢复目标（RTO/RPO）是什么？
    • 性能：您的工作负载需要哪些目标查询延迟和SLA预期？
    • 运维：您的数据科学家和工程师需要哪些运维监控指标？
    • 成本与可持续性：您是否有特定的预算限制以及数据流出/传输成本要求？
  3. 询问用户当前工作负载是否在其他云服务商或本地环境运行。
    • 如果用户回答“是”，则请求用户描述当前部署的架构。
    • 如果用户回答“否”，则进入下一步。
  4. 请求用户描述是否存在对其他工作负载、产品或工具的依赖关系。以下是可用于获取依赖关系信息的示例问题：
    • 您是否对外部系统（如身份提供商、数据治理平台、CI/CD管道或活跃数据目录）存在上游或下游依赖？
    • 您的通用数据工程软件交付生命周期是否有任何要求（如版本控制、测试、数据质量保证）？提供相关工件的目录路径或示例。
  5. 回顾用户到目前为止提供的输入，检查是否存在任何歧义或矛盾。
    如果您在用户提供的需求中发现任何歧义或矛盾（例如零拷贝与将数据复制到存储库），则针对每个发现的歧义或矛盾执行以下操作：
    • 描述歧义或矛盾（例如解释为什么复制数据与零拷贝要求相矛盾，并且会产生数据传输成本）。
    • 询问用户希望如何解决该歧义或矛盾。
      • 如果用户将选择权委托给您（例如用户回复“做你认为最好的”或“你来决定”），则提供明确的解决方案建议（例如建议优先选择零拷贝远程查询），解释您的理由（例如消除多云费用和数据重复），并请求用户批准您的建议。
    关键：在所有已识别的歧义或矛盾按照上述指导解决之前，不得推荐或生成任何架构设计、技术分解或Google Cloud产品建议。
  6. 重要：如果步骤5中存在未解决的矛盾或歧义，请不要开始此步骤。
    生成工作负载组件的技术分解。
    • 技术分解必须将解决方案拆分为逻辑组件。
    • 分解必须在相关层内解决基于角色的安全和凭证问题。
    • 分解必须组织在以下四个层之下，这些层代表代理式分析解决方案的标准架构模式，从用户交互到数据上下文和治理，再到核心数据处理：
      • 用户交互层（IDE）：例如代理式开发环境。
      • 基础与可信数据层：例如基础模型（foundation model）、MCP服务器和云端数据仓库。
      • 元数据管理层：例如元数据扫描。
      • 数据处理与分析层：例如分析工作流、Spark数据处理和外部数据存储。
  7. 请求用户批准生成的技术分解。
  8. 如果用户要求更改，则生成更新后的技术分解。
  9. 重复步骤5至8，直到用户批准生成的技术分解。
  10. 用户批准技术分解后，进入阶段2。 重要：在用户批准工作负载的技术分解之前，不得进入下一阶段。

Phase 2: Solution architecture

阶段2：解决方案架构

Ground all generated content

锚定所有生成内容

For each task in this phase, to ensure that the generated content aligns with the latest and official Google Cloud guidance, ground the generated content by using the following resources:
对于此阶段的每个任务，为确保生成的内容符合最新的官方Google Cloud指导，使用以下资源锚定生成的内容：

Task 2.1: Identify Google Cloud products and features required for the workload.

任务2.1：确定工作负载所需的Google Cloud产品和功能。

  1. For each component in the confirmed technical decomposition, identify the appropriate Google Cloud products and features, based on the guidance in the following resources and adjusted suitably based on the approved technical decomposition: 
    • references/product-selection-guidance.md
    • https://github.com/google/skills/blob/main/skills/cloud/google-cloud-solution-architecture/references/decision-making-guides.md
  2. Present the generated product recommendations and ask the user to approve the recommendations.
  3. If the user requests changes, then make the required changes.
  4. Repeat steps 2 and 3 until the user approves the product recommendations.
  5. After the user approves the product recommendations, proceed to Task 2.2.
  1. 对于已确认的技术分解中的每个组件，根据以下资源中的指导并结合已批准的技术分解进行适当调整，确定合适的Google Cloud产品和功能： 
    • references/product-selection-guidance.md
    • https://github.com/google/skills/blob/main/skills/cloud/google-cloud-solution-architecture/references/decision-making-guides.md
  2. 展示生成的产品建议，并请求用户批准这些建议。
  3. 如果用户要求更改，则进行必要的修改。
  4. 重复步骤2和3，直到用户批准产品建议。
  5. 用户批准产品建议后，进入任务2.2。

Task 2.2: Generate an architecture diagram.

任务2.2：生成架构图。

  1. Generate an architecture diagram in Mermaid format: https://github.com/mermaid-js/mermaid.
  2. Present the generated diagram to the user and ask the user to approve the architecture diagram.
  3. If the user requests changes, then make the required changes.
  4. Repeat steps 2 and 3 until the user approves the architecture diagram.
  5. After the user approves the architecture diagram, proceed to Task 2.3.
  1. 以Mermaid格式生成架构图： https://github.com/mermaid-js/mermaid.
  2. 向用户展示生成的图，并请求用户批准该架构图。
  3. 如果用户要求更改，则进行必要的修改。
  4. 重复步骤2和3，直到用户批准架构图。
  5. 用户批准架构图后，进入任务2.3。

Task 2.3: Generate an architecture description.

任务2.3：生成架构描述。

  1. Generate a description that explains the purpose of each component, the relationships between the components, and the task flow or data flow.
  2. Present the generated architecture description to the user and ask the user to approve the description.
  3. If the user requests any changes, then make the required changes.
  4. Repeat steps 2 and 3 until the user approves the architecture description.
  5. After the user approves the architecture description, proceed to Task 2.4.
  1. 生成描述，说明每个组件的用途、组件之间的关系以及任务流或数据流。
  2. 向用户展示生成的架构描述，并请求用户批准该描述。
  3. 如果用户要求更改，则进行必要的修改。
  4. 重复步骤2和3，直到用户批准架构描述。
  5. 用户批准架构描述后，进入任务2.4。

Task 2.4: Generate design recommendations.

任务2.4：生成设计建议。

  1. Generate design recommendations and best practices to optimally configure each component in the architecture based on the workload's requirements.
    Important:
    • When you generate design recommendations, consider the following:
      • Functional requirements that were gathered in Phase 1.
      • Non-functional requirements that were gathered in Phase 1.
    • Align the generated design recommendations with the recommendations in 
      references/design-recommendations.md
      .
    • To generate design recommendations for Knowledge Catalog, use the resources that are listed in 
      references/knowledge-catalog-documentation.md
    • To generate guidance for the non-functional requirements, use the following skills: 
      • google-cloud-waf-security
      • google-cloud-waf-reliability
      • google-cloud-waf-cost-optimization
      • google-cloud-waf-operational-excellence
      • google-cloud-waf-performance-optimization
      • google-cloud-waf-sustainability
  2. Present the generated recommendations to the user and ask whether the user needs any changes.
  3. If the user needs changes, then make the required changes.
  4. Repeat steps 2 and 3 until the user confirms that the generated design recommendations meet their requirements.
  5. Proceed to Task 2.5.
  1. 根据工作负载的需求，生成架构中每个组件的最佳配置设计建议和最佳实践。
    重要：
    • 生成设计建议时，请考虑以下内容：
      • 阶段1中收集的功能需求。
      • 阶段1中收集的非功能需求。
    • 使生成的设计建议与
      references/design-recommendations.md
      中的建议保持一致。
    • 要为Knowledge Catalog生成设计建议，请使用
      references/knowledge-catalog-documentation.md
      中列出的资源。
    • 要为非功能需求生成指导，请使用以下技能： 
      • google-cloud-waf-security
      • google-cloud-waf-reliability
      • google-cloud-waf-cost-optimization
      • google-cloud-waf-operational-excellence
      • google-cloud-waf-performance-optimization
      • google-cloud-waf-sustainability
  2. 向用户展示生成的建议，并询问用户是否需要任何更改。
  3. 如果用户需要更改，则进行必要的修改。
  4. 重复步骤2和3，直到用户确认生成的设计建议满足其需求。
  5. 进入任务2.5。

Task 2.5: Generate deployment guidance.

任务2.5：生成部署指导。

  1. Generate deployment guidance, including code and instructions to enable the user to deploy the solution.
    Important:
  2. Present the generated deployment guidance to the user and ask whether the user needs any changes.
  3. If the user requests changes, then make the required changes.
  4. Repeat steps 2 and 3 until the user confirms that the generated deployment guidance meets their requirements.
  5. Proceed to Phase 3.
  1. 生成部署指导，包括代码和说明，使用户能够部署解决方案。
    重要：
  2. 向用户展示生成的部署指导，并询问用户是否需要任何更改。
  3. 如果用户要求更改，则进行必要的修改。
  4. 重复步骤2和3，直到用户确认生成的部署指导满足其需求。
  5. 进入阶段3。

Phase 3: Solution validation

阶段3：解决方案验证

  1. Create a plan to validate the generated solution. The plan must outline the steps to verify that the generated solution meets the workload's requirements.
  2. Present the validation plan to the user and request feedback or approval.
  3. If the user requests changes, update the plan as required.
  4. Repeat steps 2 and 3 until the user approves the validation plan.
  5. Generate scripts or commands using tools like 
    curl
    or 
    gcloud
    to perform the steps in the approved validation plan.
  6. Request permission from the user to perform the validation checks.
  7. If the user gives permission, run the validation checks and troubleshoot any deployment issues.
  8. When all the validation checks pass, proceed to Phase 4.
  1. 创建验证生成的解决方案的计划。该计划必须概述验证生成的解决方案是否满足工作负载需求的步骤。
  2. 向用户展示验证计划，并请求反馈或批准。
  3. 如果用户要求更改，根据需要更新计划。
  4. 重复步骤2和3，直到用户批准验证计划。
  5. 使用
    curl
    或
    gcloud
    等工具生成脚本或命令，以执行已批准的验证计划中的步骤。
  6. 请求用户许可执行验证检查。
  7. 如果用户给予许可，执行验证检查并排查任何部署问题。
  8. 所有验证检查通过后，进入阶段4。

Phase 4: Solution packaging and presentation

阶段4：解决方案打包与展示

  1. Consolidate the text artifacts that were generated in Phase 2 and Phase 3 into a single Markdown file named 
    solution-architecture-guide.md
    , based on the template in 
    assets/output-template.md
    .
  2. Present the consolidated solution-architecture-guide.md to the user.
  3. Request the user's permission to write the code files in the user's workspace.
  4. After the user gives permission, write the code files in the user's workspace.
  1. 将阶段2和阶段3中生成的文本工件整合到一个名为
    solution-architecture-guide.md
    的Markdown文件中，基于
    assets/output-template.md
    中的模板。
  2. 向用户展示整合后的solution-architecture-guide.md。
  3. 请求用户许可将代码文件写入用户的工作区。
  4. 用户给予许可后，将代码文件写入用户的工作区。

Supporting resources

支持资源