volt-kubernetes
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseHelper for deploying Volt's resources on Kubernetes
用于在Kubernetes上部署Volt资源的辅助指南
This skill focuses on creating cluster configuration as code using terraform, helmfile and helm.
本指南专注于使用Terraform、Helmfile和Helm以基础设施即代码的方式创建集群配置。
Tool Overview
工具概述
- Helm: Configures kubernetes resources for a VoltDB cluster (called a release)
- Helmfile: Manages multiple helm releases with dependencies
- Terraform: Creates and manages cloud resources and high-level kubernetes resources
All tools use declarative scripts that can be version-controlled in git.
After changes are approved, re-run the specific tool to update resources.
- Helm:为VoltDB集群(称为release)配置Kubernetes资源
- Helmfile:管理带有依赖关系的多个Helm release
- Terraform:创建并管理云资源及高层级Kubernetes资源
所有工具均使用可在Git中进行版本控制的声明式脚本。变更获批后,重新运行对应工具即可更新资源。
Quick Reference for LLM
大语言模型快速参考
Key patterns to follow:
- Directory structure: Use modular, descriptive filenames (e.g., , not
voltdb-master.tf)cluster.tf - StateValues separation: (versions),
base.yaml(env vars),env.yaml.gotmpl(local overrides)override.yaml.local - Create default environment for HelmFile: Always start with a default environment.
- Service DNS:
<release-name>-<service-name>.<namespace>.svc.cluster.local:<port> - Release dependencies: Use in helmfile
needs: [namespace/release-name] - Local charts: Use for development
chart: {{ .StateValues.voltsp.chart.path }} - VoltDB Published charts: Use with
chart: voltdb/voltdbversion: "{{ .StateValues.voltdb.chart.version }}" - VoltSP Published charts: Use with
chart: voltdb/volt-streamingversion: "{{ .StateValues.voltsp.chart.version }}" - Dangerous operations: Comment out and
initForceby default with warningsdeletePVC - Gitignore patterns: in stateValues,
*localin terraform*.tfvars - Environment variables: Pass from terraform to helmfile via using
helmfile.tfwithnull_resourcelocal-exec
需遵循的核心模式:
- 目录结构:使用模块化、具描述性的文件名(例如,而非
voltdb-master.tf)cluster.tf - 状态值分离:(版本信息)、
base.yaml(环境变量)、env.yaml.gotmpl(本地覆盖配置)override.yaml.local - 为HelmFile创建默认环境:始终从默认环境开始配置
- 服务DNS:
<release-name>-<service-name>.<namespace>.svc.cluster.local:<port> - Release依赖:在helmfile中使用
needs: [namespace/release-name] - 本地图表:开发阶段使用
chart: {{ .StateValues.voltsp.chart.path }} - VoltDB官方发布图表:使用并指定
chart: voltdb/voltdbversion: "{{ .StateValues.voltdb.chart.version }}" - VoltSP官方发布图表:使用并指定
chart: voltdb/volt-streamingversion: "{{ .StateValues.voltsp.chart.version }}" - 高危操作:默认注释掉和
initForce并添加警告deletePVC - Gitignore规则:StateValues中的文件,Terraform中的
*local文件*.tfvars - 环境变量传递:通过中的
helmfile.tf结合null_resource将Terraform中的环境变量传递给Helmfilelocal-exec
Important note about resource management
资源管理重要说明
Terraform can execute helm with the helm-release resource, but we have found it to be unreliable.
Both terraform and helm depend on some internal state and those two are not always compatible.
That's why deployment scripts must be layered: Terraform as cloud orchestrator, Helm as kubernetes orchestrator.
Helmfile manages multiple helm releases and must be run from terraform.
Resource ownership:
- Terraform manages: Cloud resources (GKE/EKS/AKS), namespaces, secrets, network, infrastructure
- Helm manages: VoltDB/VoltSP cluster resources (pods, services, configmaps created by charts)
If some kubernetes resources are not strictly related to a cluster deployment, those can be created and managed by terraform.
For example, namespaces, config maps, pvc-s, secrets, etc.
Helm creates some kubernetes resources implicitly by setting values for the release. Those resources must be created and managed by helm.
If in doubt, always check the helm chart and clarify with a user.
Terraform可通过helm-release资源执行Helm,但我们发现这种方式并不可靠。Terraform和Helm都依赖各自的内部状态,二者并不总是兼容。因此部署脚本必须分层设计:Terraform作为云编排工具,Helm作为Kubernetes编排工具。Helmfile用于管理多个Helm release,且必须通过Terraform运行。
资源归属:
- Terraform管理:云资源(GKE/EKS/AKS)、命名空间、密钥、网络、基础设施
- Helm管理:VoltDB/VoltSP集群资源(由图表创建的Pod、服务、配置映射等)
若某些Kubernetes资源与集群部署无直接关联,可由Terraform创建和管理,例如命名空间、配置映射、PVC、密钥等。
Helm会通过设置release值隐式创建部分Kubernetes资源,这些资源必须由Helm创建和管理。若存在疑问,请查看Helm图表并与用户确认。
Prerequisites
前置条件
Before generating deployment scripts, check that:
- Helm v4.1.1+ is installed
- kubectl v1.34.2+ is installed
- Helmfile v1.2.3+ is installed
- Terraform v1.5.7+ is installed
- Java 17+ is installed
- Maven 3.6+ is installed
- Valid VoltDB Enterprise license file is available
kubectl must be configured to connect to a cloud provider.
生成部署脚本前,请确认:
- 已安装Helm v4.1.1+
- 已安装kubectl v1.34.2+
- 已安装Helmfile v1.2.3+
- 已安装Terraform v1.5.7+
- 已安装Java 17+
- 已安装Maven 3.6+
- 拥有有效的VoltDB企业版许可证文件
kubectl必须已配置为可连接至对应云服务商。
Instructions
操作步骤
Step 1: Understand the application details
步骤1:了解应用详情
Before any code is generated, verify with a user that there is a running integration test using testcontainers.
Such a test validates all required VoltDB components — helm values file, schema file, license, optional java code that can be packaged into a jar file.
Such a test validates all required VoltSP components — helm values file, license, optional java code that can be packaged into a jar file.
Such a test can be easily translated into a helm configuration.
If a user cannot point to such a test, please recommend creating one and quit.
If a user selects a testcontainer test, please run it locally and verify that it passes.
在生成任何代码前,请与用户确认是否存在使用Testcontainers运行的集成测试。此类测试可验证所有必需的VoltDB组件——Helm值文件、模式文件、许可证、可打包为Jar的可选Java代码。也可验证所有必需的VoltSP组件——Helm值文件、许可证、可打包为Jar的可选Java代码。
此类测试可轻松转换为Helm配置。若用户无法提供此类测试,请建议创建该测试并终止当前操作。若用户选择Testcontainers测试,请在本地运行并验证其可通过。
Step 2: Script directory layout
步骤2:脚本目录结构
Create the following directory structure at the root of the application:
text
k8s/
helmfile/
stateValues/
base.yaml # Default values (versions, paths)
env.yaml.gotmpl # Environment-specific values (from env vars)
override.yaml.local.template # Template for local overrides
values/
<release-name>.yaml # Helm values for each release
helmfile.yaml.gotmpl # Helmfile release configuration
README.md # Instructions for running helmfile
.gitignore # Ignore *local files
terraform/
provider.tf # Cloud provider configuration
variables.tf # Variable definitions
terraform.tfvars.template # Template for sensitive values
outputs.tf # Output values
gke.tf (or eks.tf, aks.tf) # Cluster configuration
network.tf # Network configuration
namespace.tf # Namespace and secrets
<resource>-<name>.tf # Resource-specific files (e.g., voltdb-master.tf)
build-jar.tf # Application build (if needed)
helmfile.tf # Helmfile execution
.gitignore # Ignore *.tfvars, .terraform, etc.
README.md # Instructions for running terraformKey principles:
- Use descriptive, resource-specific filenames in terraform (e.g., ,
voltdb-master.tf) instead of generic namesvoltdb-replica.tf - Separate concerns: one file per major resource or logical grouping
- Keep sensitive data in files or
*.tfvarsfor helmfile - those files will be gitignored*.local
在应用根目录创建如下目录结构:
text
k8s/
helmfile/
stateValues/
base.yaml # 默认值(版本、路径)
env.yaml.gotmpl # 环境特定值(来自环境变量)
override.yaml.local.template # 本地覆盖配置模板
values/
<release-name>.yaml # 每个release的Helm值
helmfile.yaml.gotmpl # Helmfile release配置
README.md # 运行Helmfile的说明文档
.gitignore # 忽略*local文件
terraform/
provider.tf # 云服务商配置
variables.tf # 变量定义
terraform.tfvars.template # 敏感值模板
outputs.tf # 输出值
gke.tf (或 eks.tf, aks.tf) # 集群配置
network.tf # 网络配置
namespace.tf # 命名空间和密钥
<resource>-<name>.tf # 特定资源文件(例如voltdb-master.tf)
build-jar.tf # 应用构建配置(若需要)
helmfile.tf # Helmfile执行配置
.gitignore # 忽略*.tfvars、.terraform等文件
README.md # 运行Terraform的说明文档核心原则:
- 在Terraform中使用具描述性的特定资源文件名(例如、
voltdb-master.tf),而非通用名称voltdb-replica.tf - 关注点分离:每个主要资源或逻辑分组对应一个文件
- 将敏感数据存储在文件或Helmfile的
*.tfvars文件中——这些文件需被Git忽略*.local
Step 3: Helm values
步骤3:Helm值配置
Look at the testcontainers test and identify all Volt services that are deployed and their resources like schema, deployment file, jar files, configuration, etc.
For each identified cluster deployment, create a yaml file with the same name as the cluster.
Those yaml files must be created in root project directory under 'k8s/helmfile/values' directory.
If the directory is missing, create it.
Key points:
- Add inline comments to warn about dangerous operations and explain critical settings
- Comment out dangerous options like and
initForceby defaultdeletePVC - Add warnings about data loss risks
- Include inline explanations for non-obvious settings
- Schema can be inlined into yaml file or set with option with a local path
--set-file - Configure topics for CDC (Change Data Capture) if needed
See for additional details.
Always check for the newest version of the helm chart and service version that the helm chart supports.
references/volt-helm-release.md查看Testcontainers测试,确定所有已部署的Volt服务及其相关资源,如模式、部署文件、Jar文件、配置等。
为每个已确定的集群部署创建一个与集群同名的YAML文件,这些文件需放置在项目根目录的目录下。若该目录不存在,请创建它。
k8s/helmfile/values关键点:
- 添加行内注释以警告高危操作并说明关键配置
- 默认注释掉和
initForce等高危选项deletePVC - 添加数据丢失风险警告
- 对非显而易见的配置添加行内说明
- 模式可内联到YAML文件中,或通过选项指定本地路径
--set-file - 若需要,为CDC(变更数据捕获)配置主题
更多细节请参考。请始终使用Helm图表及其支持的服务的最新版本。
references/volt-helm-release.mdStep 4: Helmfile releases
步骤4:Helmfile Releases配置
Helmfile requires a strict directory structure and configuration pattern.
helmfile/
stateValues/
- base.yaml
- env.yaml.gotmpl
- override.yaml.local.template
values/
helmfile.yaml.gotmpl
README.md
.gitignoreThe README.md file with instructions on how to run helmfile. Helmfile can be run from terraform which should pass environment variables to helmfile. If helmfile is run from shell commnad is used to get values for required environment variables.
The file contains helmfile release configuration.
The directory contains *.yaml or *.yaml.gotmpl files with helmfile values.
terraform output namehelmfile.yaml.gotmplstateValuesHelmfile要求严格的目录结构和配置模式:
helmfile/
stateValues/
- base.yaml
- env.yaml.gotmpl
- override.yaml.local.template
values/
helmfile.yaml.gotmpl
README.md
.gitignoreREADME.md文件包含运行Helmfile的说明。Helmfile可通过Terraform运行,Terraform会将环境变量传递给Helmfile。若从Shell运行Helmfile,可使用命令获取所需环境变量的值。
terraform output namehelmfile.yaml.gotmplstateValuesStateValues Directory
StateValues目录
Create the following files in :
helmfile/stateValues/1. base.yaml - Default values for all releases (versions, paths):
yaml
licensePath: ~/licence.xml
voltdb:
chart:
version: "3.14.0" # Helm chart version
version: "15.1.0" # VoltDB application version
voltsp:
chart:
version: "1.6.0" # Helm chart version
version: "1.6.0" # VoltSP application version
jarPath: "/path/to/voltsp.jar" # Path to VoltSP jar fileAlways use the latest version of the helm chart and application version.
Check the helm chart and application versions in the helm repository.
Values related to paths in this file can be overridden with local specific values in .
But keep the defaults in base.yaml.
override.yaml.local.template2. env.yaml.gotmpl - Environment-specific values (extracted from env vars):
yaml
namespace: {{ requiredEnv "NAMESPACE" }}
docker:
pullImageSecretName: {{ requiredEnv "DOCKER_SECRET_NAME" }}Use to extract required environment variables.
Helmfile will fail if required env vars are not set.
{{ requiredEnv "ENV_NAME" }}3. override.yaml.local.template - Template for local overrides:
The file contains a template for the file, with blank values.
Do not create file.
The file is created by users from the template and contains local-specific values.
This file should not be checked in as it can contain sensitive data. Make sure adding it to .gitignore.
override.yaml.local.templateoverride.yaml.localoverride.yaml.localoverride.yaml.localoverride.yaml.localyaml
undefined在中创建以下文件:
helmfile/stateValues/1. base.yaml - 所有release的默认值(版本、路径):
yaml
licensePath: ~/licence.xml
voltdb:
chart:
version: "3.14.0" # Helm图表版本
version: "15.1.0" # VoltDB应用版本
voltsp:
chart:
version: "1.6.0" # Helm图表版本
version: "1.6.0" # VoltSP应用版本
jarPath: "/path/to/voltsp.jar" # VoltSP Jar文件路径请始终使用Helm图表和应用的最新版本,可在Helm仓库中查看对应版本。该文件中与路径相关的值可通过中的本地特定值覆盖,但默认值需保留在base.yaml中。
override.yaml.local.template2. env.yaml.gotmpl - 环境特定值(从环境变量提取):
yaml
namespace: {{ requiredEnv "NAMESPACE" }}
docker:
pullImageSecretName: {{ requiredEnv "DOCKER_SECRET_NAME" }}使用提取必需的环境变量。若未设置必需的环境变量,Helmfile将运行失败。
{{ requiredEnv "ENV_NAME" }}3. override.yaml.local.template - 本地覆盖配置模板:
文件是文件的模板,包含空值。请勿创建文件,用户需根据该模板创建并填充本地特定值。文件不应提交至Git,因为它可能包含敏感数据,请确保将其添加到.gitignore中。
override.yaml.local.templateoverride.yaml.localoverride.yaml.localoverride.yaml.localoverride.yaml.localyaml
undefinedCopy this file to override.yaml.local and fill in your local values
复制此文件为override.yaml.local并填写本地值
override.yaml.local is gitignored and should contain sensitive data
override.yaml.local已被Git忽略,可包含敏感数据
licensePath: ""
voltsp:
jarPath: ""
**4. .gitignore** - Ignore local override files:*local
This prevents sensitive local configuration from being committed.licensePath: ""
voltsp:
jarPath: ""
**4. .gitignore** - 忽略本地覆盖文件:*local
这可防止敏感的本地配置被提交。Environment
环境配置
The helmfile.yaml.gotmpl should configure all releases using a default environment:
yaml
environments:
default:
values:
- stateValues/base.yaml
- stateValues/env.yaml.gotmpl
- path: stateValues/override.yaml.local
required: false
---
repositories:
...
releases:
...User can later add more environments, but this is a good starting point.
helmfile.yaml.gotmpl应使用默认环境配置所有release:
yaml
environments:
default:
values:
- stateValues/base.yaml
- stateValues/env.yaml.gotmpl
- path: stateValues/override.yaml.local
required: false
---
repositories:
...
releases:
...用户后续可添加更多环境,但这是一个良好的起点。
Docker Pull Secret
Docker拉取密钥
To pull images from private repositories, kubernetes requires a secret with credentials.
The helmfile could release such a secret, but it's better to create it using terraform. The secret requires user, password, email - see terraform section.
The name should be the same as the secret name in the helmfile.yaml.gotmpl file.
The secret should be created in the namespace where the cluster is deployed.
要从私有仓库拉取镜像,Kubernetes需要包含凭证的密钥。Helmfile可发布此类密钥,但更建议通过Terraform创建。该密钥需要用户名、密码、邮箱——请查看Terraform部分。密钥名称需与helmfile.yaml.gotmpl文件中的密钥名称一致,且应在集群部署的命名空间中创建。
Helm Repository
Helm仓库
See for information how to configure helm repository.
Verify whether additional repositories are required and include them in the helmfile.yaml.gotmpl file.
Example:
references/volt-helm-release.mdyaml
repositories:
- name: voltdb
url: https://voltdb-kubernetes-charts.storage.googleapis.com
# any other valid helm repository for kafka, prometheus, etc.请参考了解如何配置Helm仓库。确认是否需要额外的仓库,并将其包含在helmfile.yaml.gotmpl文件中。示例:
references/volt-helm-release.mdyaml
repositories:
- name: voltdb
url: https://voltdb-kubernetes-charts.storage.googleapis.com
# 其他有效的Kafka、Prometheus等Helm仓库Releases
Releases配置
Helmfile can manage any number of releases with dependencies.
Key patterns:
- Use to define release dependencies (ensures correct deployment order)
needs - Use or
{{ .StateValues.voltsp.chart.path }}for local charts during development{{ .StateValues.voltdb.chart.path }} - Use or
chart: voltdb/voltdbfor published chartschart: voltdb/volt-streams - Service DNS pattern:
<release-name>-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:<port> - Kafka DNS pattern:
<release-name>-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
Example releases:
Example, for the application:
productsyaml
releases:
# VoltDB Products
- name: voltdb-products
chart: voltdb/voltdb
version: "{{ .StateValues.voltdb.chart.version }}"
namespace: {{ .StateValues.namespace | quote }}
values:
- values/voltdb-products-values.yaml
set:
- name: global.voltdbVersion
value: {{ .StateValues.voltdb.version | quote }}
- name: cluster.config.licenseXMLFile
file: {{ .StateValues.licensePath | quote }}
- name: cluster.config.schemas.products
file: {{ .StateValues.voltdb.resources.products.schemaPath | quote }}
- name: cluster.config.classes.products
file: {{ .StateValues.voltdb.resources.products.jarPath | quote }}
- name: cluster.clusterSpec.imagePullSecrets[0].name
value: {{ .StateValues.docker.pullImageSecretName | quote }}
# VoltSP Products
- name: voltsp-products
chart: voltdb/volt-streams
version: "{{ .StateValues.voltsp.chart.version }}"
namespace: {{ .StateValues.namespace | quote }}
needs:
- {{ .StateValues.namespace }}/voltdb-products
values:
- values/voltsp-products-values.yaml
set:
- name: streaming.licenseXMLFile
file: {{ .StateValues.licensePath | quote }}
- name: streaming.voltapps
file: {{ .StateValues.voltsp.resources.products.jarPath | quote }}
- name: imagePullSecrets[0].name
value: {{ .StateValues.docker.pullImageSecretName | quote }}
- name: podEnv.kafka-bootstrap-servers
value: voltdb-products-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
- name: podEnv.voltdb-products-server
value: voltdb-products-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:21212Important notes:
- The field ensures releases are deployed in the correct order
needs - Use format in
namespace/release-namewhen releases are in the same namespaceneeds - The section sets environment variables used by the application
podEnv - Service DNS names follow kubernetes conventions:
<service-name>.<namespace>.svc.cluster.local:<port> - For local development, use to reference local chart directories. no version is needed.
chart: {{ .StateValues.voltsp.chart.path }} - For production, switch to published charts: with
chart: voltdb/volt-streamsspecifiedversion
See for detailed helm values configuration.
references/volt-helm-release.mdHelmfile可管理任意数量的带有依赖关系的release。
核心模式:
- 使用定义release依赖(确保正确的部署顺序)
needs - 开发阶段使用或
{{ .StateValues.voltsp.chart.path }}引用本地图表{{ .StateValues.voltdb.chart.path }} - 使用或
chart: voltdb/voltdb引用官方发布的图表chart: voltdb/volt-streams - 服务DNS模式:
<release-name>-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:<port> - Kafka DNS模式:
<release-name>-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
Releases示例:
以应用为例:
productsyaml
releases:
# VoltDB Products
- name: voltdb-products
chart: voltdb/voltdb
version: "{{ .StateValues.voltdb.chart.version }}"
namespace: {{ .StateValues.namespace | quote }}
values:
- values/voltdb-products-values.yaml
set:
- name: global.voltdbVersion
value: {{ .StateValues.voltdb.version | quote }}
- name: cluster.config.licenseXMLFile
file: {{ .StateValues.licensePath | quote }}
- name: cluster.config.schemas.products
file: {{ .StateValues.voltdb.resources.products.schemaPath | quote }}
- name: cluster.config.classes.products
file: {{ .StateValues.voltdb.resources.products.jarPath | quote }}
- name: cluster.clusterSpec.imagePullSecrets[0].name
value: {{ .StateValues.docker.pullImageSecretName | quote }}
# VoltSP Products
- name: voltsp-products
chart: voltdb/volt-streams
version: "{{ .StateValues.voltsp.chart.version }}"
namespace: {{ .StateValues.namespace | quote }}
needs:
- {{ .StateValues.namespace }}/voltdb-products
values:
- values/voltsp-products-values.yaml
set:
- name: streaming.licenseXMLFile
file: {{ .StateValues.licensePath | quote }}
- name: streaming.voltapps
file: {{ .StateValues.voltsp.resources.products.jarPath | quote }}
- name: imagePullSecrets[0].name
value: {{ .StateValues.docker.pullImageSecretName | quote }}
- name: podEnv.kafka-bootstrap-servers
value: voltdb-products-cluster-0-kafka.{{ .StateValues.namespace }}.svc.cluster.local:9092
- name: podEnv.voltdb-products-server
value: voltdb-products-cluster-client.{{ .StateValues.namespace }}.svc.cluster.local:21212重要说明:
- 字段确保release按正确顺序部署
needs - 当release位于同一命名空间时,使用
needs格式namespace/release-name - 部分设置应用使用的环境变量
podEnv - 服务DNS名称遵循Kubernetes约定:
<service-name>.<namespace>.svc.cluster.local:<port> - 本地开发时,使用引用本地图表目录,无需指定版本
chart: {{ .StateValues.voltsp.chart.path }} - 生产环境中,切换为官方发布的图表:并指定
chart: voltdb/volt-streamsversion
Helm值的详细配置请参考。
references/volt-helm-release.mdOptional Observability release
可选可观测性Release
Propose to a user whether to include an observability release.
As the cluster is created from scratch, the assumption is that monitoring services are not installed.
Volt uses Prometheus and Grafana for monitoring. See for information how to enable monitoring.
Prometheus creates special kubernetes resources to correctly target and scrape metrics from pods.
By default, Prometheus is looking for pods in the same namespace.
Example of the standalone prometheus and grafana releases:
references/volt-monitoring.mdyaml
releases:
- name: prometheus
chart: prometheus-community/prometheus-operator-crds
namespace: {{ .StateValues.namespace | quote }}
- name: monitoring
chart: voltdb/management-console
namespace: {{ .StateValues.namespace | quote }}Mind that prometheus and grafana require additional cpu and memory resources.
Monitoring has to be released first.
请询问用户是否需要包含可观测性Release。
由于集群是从头创建的,默认假设未安装监控服务。Volt使用Prometheus和Grafana进行监控,启用监控的相关信息请参考。Prometheus会创建特定的Kubernetes资源以正确定位并抓取Pod的指标,默认情况下Prometheus会查找同一命名空间中的Pod。
references/volt-monitoring.md独立Prometheus和Grafana Release示例:
yaml
releases:
- name: prometheus
chart: prometheus-community/prometheus-operator-crds
namespace: {{ .StateValues.namespace | quote }}
- name: monitoring
chart: voltdb/management-console
namespace: {{ .StateValues.namespace | quote }}请注意,Prometheus和Grafana需要额外的CPU和内存资源,监控服务必须优先部署。
Step 5: Terraform resources
步骤5:Terraform资源配置
Terraform should define all cloud resources required for deployment.
Terraform应定义部署所需的所有云资源。
Questions to ask
需询问的问题
Ask a user the following questions:
- which cloud provider to use
- what is the project name
- what is the project namespace
- what type of nodes the k8s will use
- whether to use a single node pool or to use multiple node pools for better transparency and isolation
- what is region and zone for deployment
- how nodes are visible to each other, should there be a specific security rules
- how nodes are accessible from the outside
- what is the name of the docker pull secret (default ) – user should provide additional email, user, password – those should be saved in terraform.tfvars file Answer to those questions will drive values in variables.tf file.
dockerio-registry
请向用户询问以下问题:
- 使用哪个云服务商
- 项目名称是什么
- 项目命名空间是什么
- Kubernetes将使用哪种类型的节点
- 使用单节点池还是多节点池以获得更好的透明度和隔离性
- 部署的区域和可用区是什么
- 节点之间的可见性如何,是否需要特定的安全规则
- 节点如何从外部访问
- Docker拉取密钥的名称(默认)——用户需提供额外的邮箱、用户名、密码——这些信息应保存在terraform.tfvars文件中
dockerio-registry
这些问题的答案将决定variables.tf文件中的值。
Files and directories
文件和目录
Terraform resources are created in the root project directory under the 'k8s/terraform' directory.
Required files:
1. .gitignore - Ignore sensitive and generated files:
*.tfvars
.terraform
terraform.plan
terraform.tfstate
terraform.tfstate.backup
.terraform.lock.hcl2. README.md - Instructions for running terraform with prerequisites and steps
3. variables.tf - Variable definitions with defaults:
terraform
variable "project_id" {
description = "GCP project ID"
type = string
}
variable "region" {
description = "GCP region"
type = string
default = "us-central1"
}
variable "namespace" {
description = "Kubernetes namespace"
type = string
default = "voltdb"
}
variable "docker_username" {
description = "Docker registry username"
type = string
sensitive = true
}
variable "docker_password" {
description = "Docker registry password"
type = string
sensitive = true
}
variable "docker_email" {
description = "Docker registry email"
type = string
sensitive = true
}4. terraform.tfvars.template - Template for sensitive values:
terraform
undefinedTerraform资源需创建在项目根目录的目录下。
k8s/terraform必需文件:
1. .gitignore - 忽略敏感文件和生成文件:
*.tfvars
.terraform
terraform.plan
terraform.tfstate
terraform.tfstate.backup
.terraform.lock.hcl2. README.md - 包含前置条件和步骤的Terraform运行说明
3. variables.tf - 带默认值的变量定义:
terraform
variable "project_id" {
description = "GCP project ID"
type = string
}
variable "region" {
description = "GCP region"
type = string
default = "us-central1"
}
variable "namespace" {
description = "Kubernetes namespace"
type = string
default = "voltdb"
}
variable "docker_username" {
description = "Docker registry username"
type = string
sensitive = true
}
variable "docker_password" {
description = "Docker registry password"
type = string
sensitive = true
}
variable "docker_email" {
description = "Docker registry email"
type = string
sensitive = true
}4. terraform.tfvars.template - 敏感值模板:
terraform
undefinedCopy this file to terraform.tfvars and fill in your values
复制此文件为terraform.tfvars并填写值
project_id = ""
docker_username = ""
docker_password = ""
docker_email = ""
**5. outputs.tf** - Export values for helmfile and verification:
```terraform
output "namespace" {
value = kubernetes_namespace.voltdb.metadata[0].name
}
output "docker_secret_name" {
value = kubernetes_secret.dockerio_registry.metadata[0].name
}
output "cluster_endpoint" {
value = google_container_cluster.primary.endpoint
sensitive = true
}6. provider.tf - Cloud provider and kubernetes configuration
7. gke.tf (or eks.tf, aks.tf) - Cluster configuration with node pools
8. network.tf - Network, VPC, firewall rules configuration
9. namespace.tf - Namespace and global secrets (docker pull secret)
10. Resource-specific files - Use descriptive names:
- - VoltDB master cluster resources not controlled by helmfile or helm
voltdb-products.tf - - Application build resources (if needed)
build-jar.tf
11. helmfile.tf - Helmfile execution with environment variables:
terraform
resource "null_resource" "helmfile_apply" {
provisioner "local-exec" {
command = "helmfile apply"
working_dir = "${path.module}/../helmfile"
environment = {
NAMESPACE = kubernetes_namespace.voltdb.metadata[0].name
DOCKER_SECRET_NAME = kubernetes_secret.dockerio_registry.metadata[0].name
}
}
depends_on = [
kubernetes_namespace.voltdb,
kubernetes_secret.dockerio_registry
]
}Key principles:
- Use modular, resource-specific filenames instead of generic names
- One file per major resource type or logical grouping
- Keep sensitive values in terraform.tfvars (gitignored)
- Export values needed by helmfile via outputs.tf
- Pass environment variables to helmfile via helmfile.tf
project_id = ""
docker_username = ""
docker_password = ""
docker_email = ""
**5. outputs.tf** - 导出Helmfile和验证所需的值:
```terraform
output "namespace" {
value = kubernetes_namespace.voltdb.metadata[0].name
}
output "docker_secret_name" {
value = kubernetes_secret.dockerio_registry.metadata[0].name
}
output "cluster_endpoint" {
value = google_container_cluster.primary.endpoint
sensitive = true
}6. provider.tf - 云服务商和Kubernetes配置
7. gke.tf (或 eks.tf, aks.tf) - 带节点池的集群配置
8. network.tf - 网络、VPC、防火墙规则配置
9. namespace.tf - 命名空间和全局密钥(Docker拉取密钥)
10. 特定资源文件 - 使用具描述性的名称:
- - 不由Helmfile或Helm控制的VoltDB主集群资源
voltdb-products.tf - - 应用构建资源(若需要)
build-jar.tf
11. helmfile.tf - 带环境变量的Helmfile执行配置:
terraform
resource "null_resource" "helmfile_apply" {
provisioner "local-exec" {
command = "helmfile apply"
working_dir = "${path.module}/../helmfile"
environment = {
NAMESPACE = kubernetes_namespace.voltdb.metadata[0].name
DOCKER_SECRET_NAME = kubernetes_secret.dockerio_registry.metadata[0].name
}
}
depends_on = [
kubernetes_namespace.voltdb,
kubernetes_secret.dockerio_registry
]
}核心原则:
- 使用模块化的特定资源文件名,而非通用名称
- 每个主要资源类型或逻辑分组对应一个文件
- 将敏感值保存在terraform.tfvars中(已被Git忽略)
- 通过outputs.tf导出Helmfile所需的值
- 通过helmfile.tf将环境变量传递给Helmfile
Docker Pull Secret
Docker拉取密钥
In namespace.tf create a secret, with a default name . Make sure variables contains placeholders for email, user and password that can be overridden in terraform.tfvars file.
User can choose different name for the secret. The secret name should be added to a output.tf file.
The downstream helmfile config should use the same secret name.
dockerio-registryExample in namespace.tf:
terraform
resource "kubernetes_secret" "dockerio_registry" {
metadata {
name = "dockerio-registry"
namespace = kubernetes_namespace.voltdb.metadata[0].name
}
type = "kubernetes.io/dockerconfigjson"
data = {
".dockerconfigjson" = jsonencode({
auths = {
"https://index.docker.io/v1/" = {
username = var.docker_username
password = var.docker_password
email = var.docker_email
auth = base64encode("${var.docker_username}:${var.docker_password}")
}
}
})
}
depends_on = [kubernetes_namespace.voltdb]
}Important: The secret must be created after the namespace and before helmfile execution.
在namespace.tf中创建一个密钥,默认名称为。确保变量中包含邮箱、用户名和密码的占位符,这些值可在terraform.tfvars文件中覆盖。用户可选择其他密钥名称,该密钥名称需添加到output.tf文件中,下游Helmfile配置需使用相同的密钥名称。
dockerio-registrynamespace.tf示例:
terraform
resource "kubernetes_secret" "dockerio_registry" {
metadata {
name = "dockerio-registry"
namespace = kubernetes_namespace.voltdb.metadata[0].name
}
type = "kubernetes.io/dockerconfigjson"
data = {
".dockerconfigjson" = jsonencode({
auths = {
"https://index.docker.io/v1/" = {
username = var.docker_username
password = var.docker_password
email = var.docker_email
auth = base64encode("${var.docker_username}:${var.docker_password}")
}
}
})
}
depends_on = [kubernetes_namespace.voltdb]
}重要说明: 密钥必须在命名空间创建后、Helmfile执行前创建。
Step 6: Verify the deployment
步骤6:验证部署
For the first time run command. Then run command.
Verify that the deployment is working as expected.
Wait for specific pod to be ready - or call and check that all pods are running without any errors.
terraform initterraform applykubectl wait --for=condition=ready pod -l release=<release_name> -n ${namespace} --timeout=300s;kubectl get pods -n <namespace>首次运行时执行命令,然后运行命令。
terraform initterraform apply验证部署是否正常工作。等待特定Pod就绪: 或执行检查所有Pod是否正常运行且无错误。
kubectl wait --for=condition=ready pod -l release=<release_name> -n ${namespace} --timeout=300s;kubectl get pods -n <namespace>Troubleshooting
故障排除
If expected pods are missing, crashing or have exited with error, check pods logs or deployment events with command.
Once event or logs are checked, propose a correction to the helmfile release configuration or to the terraform resources.
Re-run command and check that all pods are running without any errors.
kubectl describeterraform apply若预期Pod缺失、崩溃或报错,请查看Pod日志或使用命令查看部署事件。检查事件或日志后,建议对Helmfile release配置或Terraform资源进行修正。重新运行命令并检查所有Pod是否正常运行且无错误。
kubectl describeterraform apply