volt-kubernetes

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Helper for deploying Volt's resources on Kubernetes

用于在Kubernetes上部署Volt资源的辅助指南

This skill focuses on creating cluster configuration as code using terraform, helmfile and helm.
本指南专注于使用Terraform、Helmfile和Helm以基础设施即代码的方式创建集群配置。

Tool Overview

工具概述

  • Helm: Configures kubernetes resources for a VoltDB cluster (called a release)
  • Helmfile: Manages multiple helm releases with dependencies
  • Terraform: Creates and manages cloud resources and high-level kubernetes resources
All tools use declarative scripts that can be version-controlled in git. After changes are approved, re-run the specific tool to update resources.
  • Helm:为VoltDB集群(称为release)配置Kubernetes资源
  • Helmfile:管理带有依赖关系的多个Helm release
  • Terraform:创建并管理云资源及高层级Kubernetes资源
所有工具均使用可在Git中进行版本控制的声明式脚本。变更获批后,重新运行对应工具即可更新资源。

Quick Reference for LLM

大语言模型快速参考

Key patterns to follow:
  1. Directory structure: Use modular, descriptive filenames (e.g.,
    voltdb-master.tf
    , not
    cluster.tf
    )
  2. StateValues separation:
    base.yaml
    (versions),
    env.yaml.gotmpl
    (env vars),
    override.yaml.local
    (local overrides)
  3. Create default environment for HelmFile: Always start with a default environment.
  4. Service DNS:
    <release-name>-<service-name>.<namespace>.svc.cluster.local:<port>
  5. Release dependencies: Use
    needs: [namespace/release-name]
    in helmfile
  6. Local charts: Use
    chart: {{ .StateValues.voltsp.chart.path }}
    for development
  7. VoltDB Published charts: Use
    chart: voltdb/voltdb
    with
    version: "{{ .StateValues.voltdb.chart.version }}"
  8. VoltSP Published charts: Use
    chart: voltdb/volt-streaming
    with
    version: "{{ .StateValues.voltsp.chart.version }}"
  9. Dangerous operations: Comment out
    initForce
    and
    deletePVC
    by default with warnings
  10. Gitignore patterns:
    *local
    in stateValues,
    *.tfvars
    in terraform
  11. Environment variables: Pass from terraform to helmfile via
    helmfile.tf
    using
    null_resource
    with
    local-exec
需遵循的核心模式:
  1. 目录结构:使用模块化、具描述性的文件名(例如
    voltdb-master.tf
    ,而非
    cluster.tf
  2. 状态值分离
    base.yaml
    (版本信息)、
    env.yaml.gotmpl
    (环境变量)、
    override.yaml.local
    (本地覆盖配置)
  3. 为HelmFile创建默认环境:始终从默认环境开始配置
  4. 服务DNS
    <release-name>-<service-name>.<namespace>.svc.cluster.local:<port>
  5. Release依赖:在helmfile中使用
    needs: [namespace/release-name]
  6. 本地图表:开发阶段使用
    chart: {{ .StateValues.voltsp.chart.path }}
  7. VoltDB官方发布图表:使用
    chart: voltdb/voltdb
    并指定
    version: "{{ .StateValues.voltdb.chart.version }}"
  8. VoltSP官方发布图表:使用
    chart: voltdb/volt-streaming
    并指定
    version: "{{ .StateValues.voltsp.chart.version }}"
  9. 高危操作:默认注释掉
    initForce
    deletePVC
    并添加警告
  10. Gitignore规则:StateValues中的
    *local
    文件,Terraform中的
    *.tfvars
    文件
  11. 环境变量传递:通过
    helmfile.tf
    中的
    null_resource
    结合
    local-exec
    将Terraform中的环境变量传递给Helmfile

Important note about resource management

资源管理重要说明

Terraform can execute helm with the helm-release resource, but we have found it to be unreliable. Both terraform and helm depend on some internal state and those two are not always compatible. That's why deployment scripts must be layered: Terraform as cloud orchestrator, Helm as kubernetes orchestrator. Helmfile manages multiple helm releases and must be run from terraform.
Resource ownership:
  • Terraform manages: Cloud resources (GKE/EKS/AKS), namespaces, secrets, network, infrastructure
  • Helm manages: VoltDB/VoltSP cluster resources (pods, services, configmaps created by charts)
If some kubernetes resources are not strictly related to a cluster deployment, those can be created and managed by terraform. For example, namespaces, config maps, pvc-s, secrets, etc.
Helm creates some kubernetes resources implicitly by setting values for the release. Those resources must be created and managed by helm. If in doubt, always check the helm chart and clarify with a user.
Terraform可通过helm-release资源执行Helm,但我们发现这种方式并不可靠。Terraform和Helm都依赖各自的内部状态,二者并不总是兼容。因此部署脚本必须分层设计:Terraform作为云编排工具,Helm作为Kubernetes编排工具。Helmfile用于管理多个Helm release,且必须通过Terraform运行。
资源归属:
  • Terraform管理:云资源(GKE/EKS/AKS)、命名空间、密钥、网络、基础设施
  • Helm管理:VoltDB/VoltSP集群资源(由图表创建的Pod、服务、配置映射等)
若某些Kubernetes资源与集群部署无直接关联,可由Terraform创建和管理,例如命名空间、配置映射、PVC、密钥等。
Helm会通过设置release值隐式创建部分Kubernetes资源,这些资源必须由Helm创建和管理。若存在疑问,请查看Helm图表并与用户确认。

Prerequisites

前置条件

Before generating deployment scripts, check that:
  • Helm v4.1.1+ is installed
  • kubectl v1.34.2+ is installed
  • Helmfile v1.2.3+ is installed
  • Terraform v1.5.7+ is installed
  • Java 17+ is installed
  • Maven 3.6+ is installed
  • Valid VoltDB Enterprise license file is available
kubectl must be configured to connect to a cloud provider.
生成部署脚本前,请确认:
  • 已安装Helm v4.1.1+
  • 已安装kubectl v1.34.2+
  • 已安装Helmfile v1.2.3+
  • 已安装Terraform v1.5.7+
  • 已安装Java 17+
  • 已安装Maven 3.6+
  • 拥有有效的VoltDB企业版许可证文件
kubectl必须已配置为可连接至对应云服务商。

Instructions

操作步骤

Step 1: Understand the application details

步骤1:了解应用详情

Before any code is generated, verify with a user that there is a running integration test using testcontainers. Such a test validates all required VoltDB components — helm values file, schema file, license, optional java code that can be packaged into a jar file. Such a test validates all required VoltSP components — helm values file, license, optional java code that can be packaged into a jar file. Such a test can be easily translated into a helm configuration. If a user cannot point to such a test, please recommend creating one and quit. If a user selects a testcontainer test, please run it locally and verify that it passes.
在生成任何代码前,请与用户确认是否存在使用Testcontainers运行的集成测试。此类测试可验证所有必需的VoltDB组件——Helm值文件、模式文件、许可证、可打包为Jar的可选Java代码。也可验证所有必需的VoltSP组件——Helm值文件、许可证、可打包为Jar的可选Java代码。
此类测试可轻松转换为Helm配置。若用户无法提供此类测试,请建议创建该测试并终止当前操作。若用户选择Testcontainers测试,请在本地运行并验证其可通过。

Step 2: Script directory layout

步骤2:脚本目录结构

Create the following directory structure at the root of the application:
text
k8s/
  helmfile/
    stateValues/
      base.yaml                      # Default values (versions, paths)
      env.yaml.gotmpl                # Environment-specific values (from env vars)
      override.yaml.local.template   # Template for local overrides
    values/
      <release-name>.yaml            # Helm values for each release
    helmfile.yaml.gotmpl             # Helmfile release configuration
    README.md                        # Instructions for running helmfile
    .gitignore                       # Ignore *local files
  terraform/
    provider.tf                      # Cloud provider configuration
    variables.tf                     # Variable definitions
    terraform.tfvars.template        # Template for sensitive values
    outputs.tf                       # Output values
    gke.tf (or eks.tf, aks.tf)       # Cluster configuration
    network.tf                       # Network configuration
    namespace.tf                     # Namespace and secrets
    <resource>-<name>.tf             # Resource-specific files (e.g., voltdb-master.tf)
    build-jar.tf                     # Application build (if needed)
    helmfile.tf                      # Helmfile execution
    .gitignore                       # Ignore *.tfvars, .terraform, etc.
    README.md                        # Instructions for running terraform
Key principles:
  • Use descriptive, resource-specific filenames in terraform (e.g.,
    voltdb-master.tf
    ,
    voltdb-replica.tf
    ) instead of generic names
  • Separate concerns: one file per major resource or logical grouping
  • Keep sensitive data in
    *.tfvars
    files or
    *.local
    for helmfile - those files will be gitignored
在应用根目录创建如下目录结构:
text
k8s/
  helmfile/
    stateValues/
      base.yaml                      # 默认值(版本、路径)
      env.yaml.gotmpl                # 环境特定值(来自环境变量)
      override.yaml.local.template   # 本地覆盖配置模板
    values/
      <release-name>.yaml            # 每个release的Helm值
    helmfile.yaml.gotmpl             # Helmfile release配置
    README.md                        # 运行Helmfile的说明文档
    .gitignore                       # 忽略*local文件
  terraform/
    provider.tf                      # 云服务商配置
    variables.tf                     # 变量定义
    terraform.tfvars.template        # 敏感值模板
    outputs.tf                       # 输出值
    gke.tf (或 eks.tf, aks.tf)       # 集群配置
    network.tf                       # 网络配置
    namespace.tf                     # 命名空间和密钥
    <resource>-<name>.tf             # 特定资源文件(例如voltdb-master.tf)
    build-jar.tf                     # 应用构建配置(若需要)
    helmfile.tf                      # Helmfile执行配置
    .gitignore                       # 忽略*.tfvars、.terraform等文件
    README.md                        # 运行Terraform的说明文档
核心原则:
  • 在Terraform中使用具描述性的特定资源文件名(例如
    voltdb-master.tf
    voltdb-replica.tf
    ),而非通用名称
  • 关注点分离:每个主要资源或逻辑分组对应一个文件
  • 将敏感数据存储在
    *.tfvars
    文件或Helmfile的
    *.local
    文件中——这些文件需被Git忽略

Step 3: Helm values

步骤3:Helm值配置

Look at the testcontainers test and identify all Volt services that are deployed and their resources like schema, deployment file, jar files, configuration, etc. For each identified cluster deployment, create a yaml file with the same name as the cluster. Those yaml files must be created in root project directory under 'k8s/helmfile/values' directory. If the directory is missing, create it.
Key points:
  • Add inline comments to warn about dangerous operations and explain critical settings
  • Comment out dangerous options like
    initForce
    and
    deletePVC
    by default
  • Add warnings about data loss risks
  • Include inline explanations for non-obvious settings
  • Schema can be inlined into yaml file or set with
    --set-file
    option with a local path
  • Configure topics for CDC (Change Data Capture) if needed
See
references/volt-helm-release.md
for additional details. Always check for the newest version of the helm chart and service version that the helm chart supports.
查看Testcontainers测试,确定所有已部署的Volt服务及其相关资源,如模式、部署文件、Jar文件、配置等。
为每个已确定的集群部署创建一个与集群同名的YAML文件,这些文件需放置在项目根目录的
k8s/helmfile/values
目录下。若该目录不存在,请创建它。
关键点:
  • 添加行内注释以警告高危操作并说明关键配置
  • 默认注释掉
    initForce
    deletePVC
    等高危选项
  • 添加数据丢失风险警告
  • 对非显而易见的配置添加行内说明
  • 模式可内联到YAML文件中,或通过
    --set-file
    选项指定本地路径
  • 若需要,为CDC(变更数据捕获)配置主题
更多细节请参考
references/volt-helm-release.md
。请始终使用Helm图表及其支持的服务的最新版本。

Step 4: Helmfile releases

步骤4:Helmfile Releases配置

Helmfile requires a strict directory structure and configuration pattern.
helmfile/
  stateValues/
  - base.yaml
  - env.yaml.gotmpl
  - override.yaml.local.template
  values/
  helmfile.yaml.gotmpl
  README.md
  .gitignore
The README.md file with instructions on how to run helmfile. Helmfile can be run from terraform which should pass environment variables to helmfile. If helmfile is run from shell
terraform output name
commnad is used to get values for required environment variables. The
helmfile.yaml.gotmpl
file contains helmfile release configuration. The
stateValues
directory contains *.yaml or *.yaml.gotmpl files with helmfile values.
Helmfile要求严格的目录结构和配置模式:
helmfile/
  stateValues/
  - base.yaml
  - env.yaml.gotmpl
  - override.yaml.local.template
  values/
  helmfile.yaml.gotmpl
  README.md
  .gitignore
README.md文件包含运行Helmfile的说明。Helmfile可通过Terraform运行,Terraform会将环境变量传递给Helmfile。若从Shell运行Helmfile,可使用
terraform output name
命令获取所需环境变量的值。
helmfile.yaml.gotmpl
文件包含Helmfile release配置。
stateValues
目录包含Helmfile值对应的*.yaml或*.yaml.gotmpl文件。

StateValues Directory

StateValues目录

Create the following files in
helmfile/stateValues/
:
1. base.yaml - Default values for all releases (versions, paths):
yaml
licensePath: ~/licence.xml
voltdb:
  chart:
    version: "3.14.0"  # Helm chart version
  version: "15.1.0"    # VoltDB application version
voltsp:
  chart:
    version: "1.6.0"   # Helm chart version
  version: "1.6.0"     # VoltSP application version
  jarPath: "/path/to/voltsp.jar"  # Path to VoltSP jar file
Always use the latest version of the helm chart and application version. Check the helm chart and application versions in the helm repository. Values related to paths in this file can be overridden with local specific values in
override.yaml.local.template
. But keep the defaults in base.yaml.
2. env.yaml.gotmpl - Environment-specific values (extracted from env vars):
yaml
namespace: {{ requiredEnv "NAMESPACE" }}
docker:
  pullImageSecretName: {{ requiredEnv "DOCKER_SECRET_NAME" }}
Use
{{ requiredEnv "ENV_NAME" }}
to extract required environment variables. Helmfile will fail if required env vars are not set.
3. override.yaml.local.template - Template for local overrides: The
override.yaml.local.template
file contains a template for the
override.yaml.local
file, with blank values. Do not create
override.yaml.local
file. The
override.yaml.local
file is created by users from the template and contains local-specific values. This
override.yaml.local
file should not be checked in as it can contain sensitive data. Make sure adding it to .gitignore.
yaml
undefined
helmfile/stateValues/
中创建以下文件:
1. base.yaml - 所有release的默认值(版本、路径):
yaml
licensePath: ~/licence.xml
voltdb:
  chart:
    version: "3.14.0"  # Helm图表版本
  version: "15.1.0"    # VoltDB应用版本
voltsp:
  chart:
    version: "1.6.0"   # Helm图表版本
  version: "1.6.0"     # VoltSP应用版本
  jarPath: "/path/to/voltsp.jar"  # VoltSP Jar文件路径
请始终使用Helm图表和应用的最新版本,可在Helm仓库中查看对应版本。该文件中与路径相关的值可通过
override.yaml.local.template
中的本地特定值覆盖,但默认值需保留在base.yaml中。
2. env.yaml.gotmpl - 环境特定值(从环境变量提取):
yaml
namespace: {{ requiredEnv "NAMESPACE" }}
docker:
  pullImageSecretName: {{ requiredEnv "DOCKER_SECRET_NAME" }}
使用
{{ requiredEnv "ENV_NAME" }}
提取必需的环境变量。若未设置必需的环境变量,Helmfile将运行失败。
3. override.yaml.local.template - 本地覆盖配置模板:
override.yaml.local.template
文件是
override.yaml.local
文件的模板,包含空值。请勿创建
override.yaml.local
文件,用户需根据该模板创建
override.yaml.local
并填充本地特定值。
override.yaml.local
文件不应提交至Git,因为它可能包含敏感数据,请确保将其添加到.gitignore中。
yaml
undefined

Copy this file to override.yaml.local and fill in your local values

复制此文件为override.yaml.local并填写本地值

override.yaml.local is gitignored and should contain sensitive data

override.yaml.local已被Git忽略,可包含敏感数据

licensePath: "" voltsp: jarPath: ""

**4. .gitignore** - Ignore local override files:
*local
This prevents sensitive local configuration from being committed.
licensePath: "" voltsp: jarPath: ""

**4. .gitignore** - 忽略本地覆盖文件:
*local
这可防止敏感的本地配置被提交。

Environment

环境配置

The helmfile.yaml.gotmpl should configure all releases using a default environment:
yaml
environments:
  default:
    values:
    - stateValues/base.yaml
    - stateValues/env.yaml.gotmpl
    - path: stateValues/override.yaml.local
      required: false
---
repositories:
  ...

releases:
  ...
User can later add more environments, but this is a good starting point.
helmfile.yaml.gotmpl应使用默认环境配置所有release:
yaml
environments:
  default:
    values:
    - stateValues/base.yaml
    - stateValues/env.yaml.gotmpl
    - path: stateValues/override.yaml.local
      required: false
---
repositories:
  ...

releases:
  ...
用户后续可添加更多环境,但这是一个良好的起点。

Docker Pull Secret

Docker拉取密钥

To pull images from private repositories, kubernetes requires a secret with credentials. The helmfile could release such a secret, but it's better to create it using terraform. The secret requires user, password, email - see terraform section. The name should be the same as the secret name in the helmfile.yaml.gotmpl file. The secret should be created in the namespace where the cluster is deployed.
要从私有仓库拉取镜像,Kubernetes需要包含凭证的密钥。Helmfile可发布此类密钥,但更建议通过Terraform创建。该密钥需要用户名、密码、邮箱——请查看Terraform部分。密钥名称需与helmfile.yaml.gotmpl文件中的密钥名称一致,且应在集群部署的命名空间中创建。

Helm Repository

Helm仓库

See
references/volt-helm-release.md
for information how to configure helm repository. Verify whether additional repositories are required and include them in the helmfile.yaml.gotmpl file. Example:
yaml
repositories:
  - name: voltdb
    url: https://voltdb-kubernetes-charts.storage.googleapis.com
  # any other valid helm repository for kafka, prometheus, etc.
请参考
references/volt-helm-release.md
了解如何配置Helm仓库。确认是否需要额外的仓库,并将其包含在helmfile.yaml.gotmpl文件中。示例:
yaml
repositories:
  - name: voltdb
    url: https://voltdb-kubernetes-charts.storage.googleapis.com
  # 其他有效的Kafka、Prometheus等Helm仓库

Releases

Releases配置

Helmfile can manage any number of releases with dependencies.
Key patterns:
  • Use
    needs
    to define release dependencies (ensures correct deployment order)
  • Use
    {{ .StateValues.voltsp.chart.path }}
    or
    {{ .StateValues.voltdb.chart.path }}
    for local charts during development
  • Use
    chart: voltdb/voltdb
    or
    chart: voltdb/volt-streams
    for published charts
  • Service DNS pattern:
    <release-name>-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:<port>
  • Kafka DNS pattern:
    <release-name>-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
Example releases: Example, for the
products
application:
yaml
releases:
  # VoltDB Products
  - name: voltdb-products
    chart: voltdb/voltdb
    version: "{{ .StateValues.voltdb.chart.version }}"
    namespace: {{ .StateValues.namespace | quote }}
    values:
      - values/voltdb-products-values.yaml
    set:
      - name: global.voltdbVersion
        value: {{ .StateValues.voltdb.version | quote }}
      - name: cluster.config.licenseXMLFile
        file: {{ .StateValues.licensePath | quote }}
      - name: cluster.config.schemas.products
        file: {{ .StateValues.voltdb.resources.products.schemaPath | quote }}
      - name: cluster.config.classes.products
        file: {{ .StateValues.voltdb.resources.products.jarPath | quote }}
      - name: cluster.clusterSpec.imagePullSecrets[0].name
        value: {{ .StateValues.docker.pullImageSecretName | quote }}

  # VoltSP Products
  - name: voltsp-products
    chart: voltdb/volt-streams
    version: "{{ .StateValues.voltsp.chart.version }}"
    namespace: {{ .StateValues.namespace | quote }}
    needs:
      - {{ .StateValues.namespace }}/voltdb-products
    values:
      - values/voltsp-products-values.yaml
    set:
      - name: streaming.licenseXMLFile
        file: {{ .StateValues.licensePath | quote }}
      - name: streaming.voltapps
        file: {{ .StateValues.voltsp.resources.products.jarPath | quote }}
      - name: imagePullSecrets[0].name
        value: {{ .StateValues.docker.pullImageSecretName | quote }}
      - name: podEnv.kafka-bootstrap-servers
        value: voltdb-products-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
      - name: podEnv.voltdb-products-server
        value: voltdb-products-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:21212
Important notes:
  • The
    needs
    field ensures releases are deployed in the correct order
  • Use
    namespace/release-name
    format in
    needs
    when releases are in the same namespace
  • The
    podEnv
    section sets environment variables used by the application
  • Service DNS names follow kubernetes conventions:
    <service-name>.<namespace>.svc.cluster.local:<port>
  • For local development, use
    chart: {{ .StateValues.voltsp.chart.path }}
    to reference local chart directories. no version is needed.
  • For production, switch to published charts:
    chart: voltdb/volt-streams
    with
    version
    specified
See
references/volt-helm-release.md
for detailed helm values configuration.
Helmfile可管理任意数量的带有依赖关系的release。
核心模式:
  • 使用
    needs
    定义release依赖(确保正确的部署顺序)
  • 开发阶段使用
    {{ .StateValues.voltsp.chart.path }}
    {{ .StateValues.voltdb.chart.path }}
    引用本地图表
  • 使用
    chart: voltdb/voltdb
    chart: voltdb/volt-streams
    引用官方发布的图表
  • 服务DNS模式:
    <release-name>-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:<port>
  • Kafka DNS模式:
    <release-name>-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
Releases示例:
products
应用为例:
yaml
releases:
  # VoltDB Products
  - name: voltdb-products
    chart: voltdb/voltdb
    version: "{{ .StateValues.voltdb.chart.version }}"
    namespace: {{ .StateValues.namespace | quote }}
    values:
      - values/voltdb-products-values.yaml
    set:
      - name: global.voltdbVersion
        value: {{ .StateValues.voltdb.version | quote }}
      - name: cluster.config.licenseXMLFile
        file: {{ .StateValues.licensePath | quote }}
      - name: cluster.config.schemas.products
        file: {{ .StateValues.voltdb.resources.products.schemaPath | quote }}
      - name: cluster.config.classes.products
        file: {{ .StateValues.voltdb.resources.products.jarPath | quote }}
      - name: cluster.clusterSpec.imagePullSecrets[0].name
        value: {{ .StateValues.docker.pullImageSecretName | quote }}

  # VoltSP Products
  - name: voltsp-products
    chart: voltdb/volt-streams
    version: "{{ .StateValues.voltsp.chart.version }}"
    namespace: {{ .StateValues.namespace | quote }}
    needs:
      - {{ .StateValues.namespace }}/voltdb-products
    values:
      - values/voltsp-products-values.yaml
    set:
      - name: streaming.licenseXMLFile
        file: {{ .StateValues.licensePath | quote }}
      - name: streaming.voltapps
        file: {{ .StateValues.voltsp.resources.products.jarPath | quote }}
      - name: imagePullSecrets[0].name
        value: {{ .StateValues.docker.pullImageSecretName | quote }}
      - name: podEnv.kafka-bootstrap-servers
        value: voltdb-products-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
      - name: podEnv.voltdb-products-server
        value: voltdb-products-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:21212
重要说明:
  • needs
    字段确保release按正确顺序部署
  • 当release位于同一命名空间时,
    needs
    使用
    namespace/release-name
    格式
  • podEnv
    部分设置应用使用的环境变量
  • 服务DNS名称遵循Kubernetes约定:
    <service-name>.<namespace>.svc.cluster.local:<port>
  • 本地开发时,使用
    chart: {{ .StateValues.voltsp.chart.path }}
    引用本地图表目录,无需指定版本
  • 生产环境中,切换为官方发布的图表:
    chart: voltdb/volt-streams
    并指定
    version
Helm值的详细配置请参考
references/volt-helm-release.md

Optional Observability release

可选可观测性Release

Propose to a user whether to include an observability release.
As the cluster is created from scratch, the assumption is that monitoring services are not installed. Volt uses Prometheus and Grafana for monitoring. See
references/volt-monitoring.md
for information how to enable monitoring. Prometheus creates special kubernetes resources to correctly target and scrape metrics from pods. By default, Prometheus is looking for pods in the same namespace. Example of the standalone prometheus and grafana releases:
yaml
releases:
  - name: prometheus
    chart: prometheus-community/prometheus-operator-crds
    namespace: {{ .StateValues.namespace | quote }}

  - name: monitoring
    chart: voltdb/management-console
    namespace: {{ .StateValues.namespace | quote }}
Mind that prometheus and grafana require additional cpu and memory resources. Monitoring has to be released first.
请询问用户是否需要包含可观测性Release。
由于集群是从头创建的,默认假设未安装监控服务。Volt使用Prometheus和Grafana进行监控,启用监控的相关信息请参考
references/volt-monitoring.md
。Prometheus会创建特定的Kubernetes资源以正确定位并抓取Pod的指标,默认情况下Prometheus会查找同一命名空间中的Pod。
独立Prometheus和Grafana Release示例:
yaml
releases:
  - name: prometheus
    chart: prometheus-community/prometheus-operator-crds
    namespace: {{ .StateValues.namespace | quote }}

  - name: monitoring
    chart: voltdb/management-console
    namespace: {{ .StateValues.namespace | quote }}
请注意,Prometheus和Grafana需要额外的CPU和内存资源,监控服务必须优先部署。

Step 5: Terraform resources

步骤5:Terraform资源配置

Terraform should define all cloud resources required for deployment.
Terraform应定义部署所需的所有云资源。

Questions to ask

需询问的问题

Ask a user the following questions:
  • which cloud provider to use
  • what is the project name
  • what is the project namespace
  • what type of nodes the k8s will use
  • whether to use a single node pool or to use multiple node pools for better transparency and isolation
  • what is region and zone for deployment
  • how nodes are visible to each other, should there be a specific security rules
  • how nodes are accessible from the outside
  • what is the name of the docker pull secret (default
    dockerio-registry
    ) – user should provide additional email, user, password – those should be saved in terraform.tfvars file Answer to those questions will drive values in variables.tf file.
请向用户询问以下问题:
  • 使用哪个云服务商
  • 项目名称是什么
  • 项目命名空间是什么
  • Kubernetes将使用哪种类型的节点
  • 使用单节点池还是多节点池以获得更好的透明度和隔离性
  • 部署的区域和可用区是什么
  • 节点之间的可见性如何,是否需要特定的安全规则
  • 节点如何从外部访问
  • Docker拉取密钥的名称(默认
    dockerio-registry
    )——用户需提供额外的邮箱、用户名、密码——这些信息应保存在terraform.tfvars文件中
这些问题的答案将决定variables.tf文件中的值。

Files and directories

文件和目录

Terraform resources are created in the root project directory under the 'k8s/terraform' directory.
Required files:
1. .gitignore - Ignore sensitive and generated files:
*.tfvars
.terraform
terraform.plan
terraform.tfstate
terraform.tfstate.backup
.terraform.lock.hcl
2. README.md - Instructions for running terraform with prerequisites and steps
3. variables.tf - Variable definitions with defaults:
terraform
variable "project_id" {
  description = "GCP project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "us-central1"
}

variable "namespace" {
  description = "Kubernetes namespace"
  type        = string
  default     = "voltdb"
}

variable "docker_username" {
  description = "Docker registry username"
  type        = string
  sensitive   = true
}

variable "docker_password" {
  description = "Docker registry password"
  type        = string
  sensitive   = true
}

variable "docker_email" {
  description = "Docker registry email"
  type        = string
  sensitive   = true
}
4. terraform.tfvars.template - Template for sensitive values:
terraform
undefined
Terraform资源需创建在项目根目录的
k8s/terraform
目录下。
必需文件:
1. .gitignore - 忽略敏感文件和生成文件:
*.tfvars
.terraform
terraform.plan
terraform.tfstate
terraform.tfstate.backup
.terraform.lock.hcl
2. README.md - 包含前置条件和步骤的Terraform运行说明
3. variables.tf - 带默认值的变量定义:
terraform
variable "project_id" {
  description = "GCP project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "us-central1"
}

variable "namespace" {
  description = "Kubernetes namespace"
  type        = string
  default     = "voltdb"
}

variable "docker_username" {
  description = "Docker registry username"
  type        = string
  sensitive   = true
}

variable "docker_password" {
  description = "Docker registry password"
  type        = string
  sensitive   = true
}

variable "docker_email" {
  description = "Docker registry email"
  type        = string
  sensitive   = true
}
4. terraform.tfvars.template - 敏感值模板:
terraform
undefined

Copy this file to terraform.tfvars and fill in your values

复制此文件为terraform.tfvars并填写值

project_id = "" docker_username = "" docker_password = "" docker_email = ""

**5. outputs.tf** - Export values for helmfile and verification:
```terraform
output "namespace" {
  value = kubernetes_namespace.voltdb.metadata[0].name
}

output "docker_secret_name" {
  value = kubernetes_secret.dockerio_registry.metadata[0].name
}

output "cluster_endpoint" {
  value = google_container_cluster.primary.endpoint
  sensitive = true
}
6. provider.tf - Cloud provider and kubernetes configuration
7. gke.tf (or eks.tf, aks.tf) - Cluster configuration with node pools
8. network.tf - Network, VPC, firewall rules configuration
9. namespace.tf - Namespace and global secrets (docker pull secret)
10. Resource-specific files - Use descriptive names:
  • voltdb-products.tf
    - VoltDB master cluster resources not controlled by helmfile or helm
  • build-jar.tf
    - Application build resources (if needed)
11. helmfile.tf - Helmfile execution with environment variables:
terraform
resource "null_resource" "helmfile_apply" {
  provisioner "local-exec" {
    command = "helmfile apply"
    working_dir = "${path.module}/../helmfile"
    environment = {
      NAMESPACE          = kubernetes_namespace.voltdb.metadata[0].name
      DOCKER_SECRET_NAME = kubernetes_secret.dockerio_registry.metadata[0].name
    }
  }

  depends_on = [
    kubernetes_namespace.voltdb,
    kubernetes_secret.dockerio_registry
  ]
}
Key principles:
  • Use modular, resource-specific filenames instead of generic names
  • One file per major resource type or logical grouping
  • Keep sensitive values in terraform.tfvars (gitignored)
  • Export values needed by helmfile via outputs.tf
  • Pass environment variables to helmfile via helmfile.tf
project_id = "" docker_username = "" docker_password = "" docker_email = ""

**5. outputs.tf** - 导出Helmfile和验证所需的值:
```terraform
output "namespace" {
  value = kubernetes_namespace.voltdb.metadata[0].name
}

output "docker_secret_name" {
  value = kubernetes_secret.dockerio_registry.metadata[0].name
}

output "cluster_endpoint" {
  value = google_container_cluster.primary.endpoint
  sensitive = true
}
6. provider.tf - 云服务商和Kubernetes配置
7. gke.tf (或 eks.tf, aks.tf) - 带节点池的集群配置
8. network.tf - 网络、VPC、防火墙规则配置
9. namespace.tf - 命名空间和全局密钥(Docker拉取密钥)
10. 特定资源文件 - 使用具描述性的名称:
  • voltdb-products.tf
    - 不由Helmfile或Helm控制的VoltDB主集群资源
  • build-jar.tf
    - 应用构建资源(若需要)
11. helmfile.tf - 带环境变量的Helmfile执行配置:
terraform
resource "null_resource" "helmfile_apply" {
  provisioner "local-exec" {
    command = "helmfile apply"
    working_dir = "${path.module}/../helmfile"
    environment = {
      NAMESPACE          = kubernetes_namespace.voltdb.metadata[0].name
      DOCKER_SECRET_NAME = kubernetes_secret.dockerio_registry.metadata[0].name
    }
  }

  depends_on = [
    kubernetes_namespace.voltdb,
    kubernetes_secret.dockerio_registry
  ]
}
核心原则:
  • 使用模块化的特定资源文件名,而非通用名称
  • 每个主要资源类型或逻辑分组对应一个文件
  • 将敏感值保存在terraform.tfvars中(已被Git忽略)
  • 通过outputs.tf导出Helmfile所需的值
  • 通过helmfile.tf将环境变量传递给Helmfile

Docker Pull Secret

Docker拉取密钥

In namespace.tf create a secret, with a default name
dockerio-registry
. Make sure variables contains placeholders for email, user and password that can be overridden in terraform.tfvars file. User can choose different name for the secret. The secret name should be added to a output.tf file. The downstream helmfile config should use the same secret name.
Example in namespace.tf:
terraform
resource "kubernetes_secret" "dockerio_registry" {
  metadata {
    name      = "dockerio-registry"
    namespace = kubernetes_namespace.voltdb.metadata[0].name
  }

  type = "kubernetes.io/dockerconfigjson"

  data = {
    ".dockerconfigjson" = jsonencode({
      auths = {
        "https://index.docker.io/v1/" = {
          username = var.docker_username
          password = var.docker_password
          email    = var.docker_email
          auth     = base64encode("${var.docker_username}:${var.docker_password}")
        }
      }
    })
  }

  depends_on = [kubernetes_namespace.voltdb]
}
Important: The secret must be created after the namespace and before helmfile execution.
在namespace.tf中创建一个密钥,默认名称为
dockerio-registry
。确保变量中包含邮箱、用户名和密码的占位符,这些值可在terraform.tfvars文件中覆盖。用户可选择其他密钥名称,该密钥名称需添加到output.tf文件中,下游Helmfile配置需使用相同的密钥名称。
namespace.tf示例:
terraform
resource "kubernetes_secret" "dockerio_registry" {
  metadata {
    name      = "dockerio-registry"
    namespace = kubernetes_namespace.voltdb.metadata[0].name
  }

  type = "kubernetes.io/dockerconfigjson"

  data = {
    ".dockerconfigjson" = jsonencode({
      auths = {
        "https://index.docker.io/v1/" = {
          username = var.docker_username
          password = var.docker_password
          email    = var.docker_email
          auth     = base64encode("${var.docker_username}:${var.docker_password}")
        }
      }
    })
  }

  depends_on = [kubernetes_namespace.voltdb]
}
重要说明: 密钥必须在命名空间创建后、Helmfile执行前创建。

Step 6: Verify the deployment

步骤6:验证部署

For the first time run
terraform init
command. Then run
terraform apply
command. Verify that the deployment is working as expected. Wait for specific pod to be ready -
kubectl wait --for=condition=ready pod -l release=<release_name> -n ${namespace} --timeout=300s;
or call
kubectl get pods -n <namespace>
and check that all pods are running without any errors.
首次运行时执行
terraform init
命令,然后运行
terraform apply
命令。
验证部署是否正常工作。等待特定Pod就绪:
kubectl wait --for=condition=ready pod -l release=<release_name> -n ${namespace} --timeout=300s;
或执行
kubectl get pods -n <namespace>
检查所有Pod是否正常运行且无错误。

Troubleshooting

故障排除

If expected pods are missing, crashing or have exited with error, check pods logs or deployment events with
kubectl describe
command. Once event or logs are checked, propose a correction to the helmfile release configuration or to the terraform resources. Re-run
terraform apply
command and check that all pods are running without any errors.
若预期Pod缺失、崩溃或报错,请查看Pod日志或使用
kubectl describe
命令查看部署事件。检查事件或日志后,建议对Helmfile release配置或Terraform资源进行修正。重新运行
terraform apply
命令并检查所有Pod是否正常运行且无错误。