WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"
mkdir -p "$WORKDIR" && chmod 700 "$WORKDIR"
ENV_FILE="$WORKDIR/collector.env"

Two things about these values matter and are easy to get wrong:

- **Key names are exact.** The collector reads `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`,
  `CLICKHOUSE_ENDPOINT`, and `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE`. Store the SQL password
  under `CLICKHOUSE_PASSWORD` (not a custom name); if it is missing, the collector starts with an
  **empty** password and dies with `code: 516, Authentication failed`.
- **The password charset is constrained from three directions at once.** ClickHouse Cloud rejects
  passwords without at least one uppercase character and one special character, so a plain hex
  string fails at `CREATE USER`. At the same time, the collector's migration tool embeds the
  password in a connection URL, so `@`, `:`, `/`, `?`, `#`, and `%` corrupt it (symptom:
  `code: 516` at startup even though the password is "correct"). The recipe above is random hex
  (lowercase + digits) plus the suffix `Aa1-`, which adds the required uppercase, a digit, and a
  **URL-unreserved** special character (`-`). The OTLP token has no such rules (it is just a
  bearer token), so plain hex is fine for it.

The env file uses **bare `KEY=VALUE` lines with no quotes**: Docker's `--env-file` does not do
shell parsing, so any quotes you add become part of the value.

On the **existing-collector path** the `OTLP_AUTH_TOKEN` is not used by your collector (auth on
your receiver is your own setup); it is generated only so the same file works if you later switch
to the local collector. The `CLICKHOUSE_*` values are still used: they go into the exporter
config you add to your collector in Step 6.

**Every later step runs in a fresh shell, so `WORKDIR`, `ENV_FILE`, and any exported credentials do
not persist, and `WORKDIR`/`ENV_FILE` are not stored inside the env file, so sourcing it can't
recover them.** Begin each subsequent step's shell with this **standard preamble**, which
re-derives the paths from the deterministic default, loads the saved credentials (Step 3), and
loads the config:

```bash
WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env"
[ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a

which clickhousectl

curl -fsSL https://clickhouse.com/cli | sh

clickhousectl cloud auth status

clickhousectl cloud auth status
clickhousectl cloud service list --json | jq -r '.[].name'

undefined

**From now on, open every shell that calls `clickhousectl` with both loads**, because env vars do
not persist across shells:

```bash
. "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a

WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env"
[ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a

undefined

Extract the values you need, coercing the port to an integer. The port serializes as a float
(`8443.0`); if `:8443.0` leaks into the endpoint the collector's ClickHouse exporter cannot dial
it:

```bash
SERVICE_ID=$(jq -r '.id' "$WORKDIR/svc.json")
SERVICE_NAME=$(jq -r '.name' "$WORKDIR/svc.json")
STATE=$(jq -r '.state' "$WORKDIR/svc.json")
CLICKHOUSE_ENDPOINT=$(jq -r '.endpoints[] | select(.protocol=="https")
  | "https://\(.host):\(.port | tonumber | floor)"' "$WORKDIR/svc.json")

`STATE` must be `running`. If it is `stopped` or `starting`, ask the user to start the service (or
wait), and do not proceed. ClickHouse Cloud services **idle-suspend**, so even a "running" service
can be asleep; the next query both checks reachability and wakes it:

```bash
clickhousectl cloud service query --id "$SERVICE_ID" --query "SELECT version()"

WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env"
[ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a

Grant the least privilege the collector needs to create and write the `otel.*` schema. On the
current image the schema migrations and their version table also live in `otel`, so `otel.*` is
sufficient (this statement carries no secret):

```bash
clickhousectl cloud service query --id "$SERVICE_ID" --query \
  "GRANT SELECT, INSERT, CREATE DATABASE, CREATE TABLE, CREATE VIEW ON otel.* TO hyperdx_ingest"

clickhousectl cloud service query --id "$SERVICE_ID" --query "SHOW GRANTS FOR hyperdx_ingest"

WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env"
[ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a
docker info > /dev/null

cat > "$WORKDIR/docker-compose.yaml" <<'EOF'
name: clickstack
services:
  otel-collector:
    image: clickhouse/clickstack-otel-collector:latest
    container_name: clickstack-otel-collector
    env_file: ./collector.env
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "13133:13133" # health
    restart: unless-stopped
    networks: [clickstack-net]
networks:
  clickstack-net:
    name: clickstack-net
EOF

Compose creates the `clickstack-net` network for you (the guard above clears an orphaned one from a
prior run first). Skip to **Step 6d** to confirm health.

docker network create clickstack-net 2>/dev/null || true

docker rm -f clickstack-otel-collector 2>/dev/null || true
docker run -d \
  --name clickstack-otel-collector \
  --network clickstack-net \
  --env-file "$ENV_FILE" \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 13133:13133 \
  clickhouse/clickstack-otel-collector:latest

undefined

Add this to your collector config and reload it:

```yaml
receivers:
  otlp/hyperdx:
    protocols:
      grpc:
        include_metadata: true
        endpoint: "0.0.0.0:4317"
      http:
        cors:
          allowed_origins: ["*"]
          allowed_headers: ["*"]
        include_metadata: true
        endpoint: "0.0.0.0:4318"

processors:
  batch:
  memory_limiter:
    limit_mib: 1500
    spike_limit_mib: 512
    check_interval: 5s

connectors:
  routing/logs:
    default_pipelines: [logs/out-default]
    error_mode: ignore
    table:
      - context: log
        statement: route() where IsMatch(attributes["rr-web.event"], ".*")
        pipelines: [logs/out-rrweb]

exporters:
  clickhouse:
    database: otel
    endpoint: ${env:CLICKHOUSE_ENDPOINT}
    username: hyperdx_ingest
    password: ${env:CLICKHOUSE_PASSWORD}
    ttl: 720h
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
  clickhouse/rrweb:
    database: otel
    endpoint: ${env:CLICKHOUSE_ENDPOINT}
    username: hyperdx_ingest
    password: ${env:CLICKHOUSE_PASSWORD}
    ttl: 720h
    logs_table_name: hyperdx_sessions
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    traces:
      receivers: [otlp/hyperdx]
      processors: [memory_limiter, batch]
      exporters: [clickhouse]
    metrics:
      receivers: [otlp/hyperdx]
      processors: [memory_limiter, batch]
      exporters: [clickhouse]
    logs/in:
      receivers: [otlp/hyperdx]
      exporters: [routing/logs]
    logs/out-default:
      receivers: [routing/logs]
      processors: [memory_limiter, batch]
      exporters: [clickhouse]
    logs/out-rrweb:
      receivers: [routing/logs]
      processors: [memory_limiter, batch]
      exporters: [clickhouse/rrweb]

docker ps --filter name=clickstack-otel-collector --format '{{.Status}}'
curl -fsS http://localhost:13133/ && echo
docker logs --tail 40 clickstack-otel-collector 2>&1 | tail -40

WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env"
[ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a

TG_IMAGE=ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest
NET=clickstack-net
ENDPOINT=clickstack-otel-collector:4317
TG_LOG="$WORKDIR/telemetrygen.log"; : > "$TG_LOG"

tg() {
  # usage: tg <logs|traces|metrics> [extra telemetrygen flags...]
  # Output (which contains the token in the echoed config) goes to $TG_LOG, never the terminal.
  local signal="$1"; shift
  docker run --rm --network "$NET" "$TG_IMAGE" "$signal" \
    --otlp-endpoint "$ENDPOINT" \
    --otlp-insecure \
    --otlp-header "authorization=\"$OTLP_AUTH_TOKEN\"" \
    --rate 10 --duration 15s "$@" >>"$TG_LOG" 2>&1
  echo "$signal exit=$?"
}

tg logs --service checkout --severity-text Info  --severity-number 9 \
  --body "checkout completed" \
  --otlp-attributes 'deployment.environment="production"' \
  --telemetry-attributes 'http.method="POST"'
tg logs --service payment  --severity-text Error --severity-number 17 \
  --body "payment gateway timeout" \
  --otlp-attributes 'deployment.environment="production"' \
  --telemetry-attributes 'http.status_code="500"'

tg traces --service checkout --child-spans 4 --span-duration 120ms --status-code Ok \
  --otlp-attributes 'deployment.environment="production"' \
  --telemetry-attributes 'http.route="/cart"'
tg traces --service payment  --child-spans 3 --span-duration 400ms --status-code Error \
  --otlp-attributes 'deployment.environment="production"' \
  --telemetry-attributes 'http.route="/charge"'

tg metrics --service checkout --metric-type Sum
tg metrics --service checkout --metric-type Gauge
tg metrics --service payment  --metric-type Histogram

clickhousectl cloud service query --id "$SERVICE_ID" --query \
  "SELECT name FROM system.tables WHERE database='otel' ORDER BY name"

clickhousectl cloud service query --id "$SERVICE_ID" --query \
  "SELECT table, sum(rows) AS rows
   FROM system.parts
   WHERE database='otel' AND active
     AND table IN ('otel_logs','otel_traces',
                   'otel_metrics_sum','otel_metrics_gauge',
                   'otel_metrics_histogram','otel_metrics_exponential_histogram',
                   'otel_metrics_summary')
   GROUP BY table ORDER BY table"

clickhousectl cloud service query --id "$SERVICE_ID" --query "SELECT 1"

✅ ClickStack is set up and ingesting telemetry for service <SERVICE_NAME> (<SERVICE_ID>).
   Complete onboarding in the console (Step 8) to auto-detect the sources and see your data.

Collector
  ▸ New local collector via <Docker Compose | docker run>  (or: configured your existing collector)
  ▸ Send OTLP gRPC to: localhost:4317
  ▸ Send OTLP HTTP to: localhost:4318
  ▸ Health check:      http://localhost:13133/   (local collector only)
  ▸ Required header:   authorization: <OTLP token>
       (retrieve with:  grep OTLP_AUTH_TOKEN <WORKDIR>/collector.env)

ClickHouse target
  ▸ Endpoint: <CLICKHOUSE_ENDPOINT>
  ▸ SQL user: hyperdx_ingest   (password in <WORKDIR>/collector.env, mode 0600)
  ▸ Database: otel

Finish in the ClickHouse Cloud console:
  ▸ Open the service, select ClickStack in the left menu, then Getting Started,
    and complete onboarding to auto-detect sources.
  ▸ https://console.clickhouse.cloud/services/<SERVICE_ID>/clickstack

undefined

Do **not** drop the `otel` database: it contains telemetry the user may want to retain.