export GROUNDCOVER_API_KEY=gcsa_...   # must be a service-account key

gcsa_

Invalid token prefix. Expected 'gcsa_'

Invalid token length (N, expected 40)

• gcsa_…

  — service-account key. This is the one query/CRUD commands need.

• gcik_…

  — ingestion key (data push). Will NOT authenticate the API; wrong type.

gcsa_

gcsa_

groundcover service-accounts create

export GROUNDCOVER_API_KEY=gcsa_…

• --base-url https://api.groundcover.com

• --backend-id groundcover

--tenant-uuid

GROUNDCOVER_TENANT_UUID

X-Tenant-UUID

raw …

raw grafana …

raw grafana …

gcsa_

glsa_…

GROUNDCOVER_GRAFANA_SERVICE_ACCOUNT_TOKEN

--grafana-token

GROUNDCOVER_BACKEND_ID

GROUNDCOVER_TENANT_UUID

GROUNDCOVER_GRAFANA_SERVICE_ACCOUNT_TOKEN

GROUNDCOVER_BASE_URL

GC_*

--api-key

--backend-id

--tenant-uuid

--grafana-token

--base-url

• --set dotted.path=value

  — deep-merge overrides on top of the body

• --query key=value

  (repeatable) — arbitrary querystring overrides
• Built-in default body captured from the webapp HAR (SDK requires explicit

  --body-file

  /

  --body-json

  )
• Sends

  X-Tenant-UUID

  on non-grafana raw calls when you set

  --tenant-uuid

  /

  GROUNDCOVER_TENANT_UUID

  (for cross-tenant access; the SDK transport otherwise derives tenant from the API key). Grafana

  raw grafana …

  commands don't use it — they authenticate with the

  glsa_

  token instead.

• groundcover raw list

  to discover every captured endpoint

--body-file

--body-json

--raw

groundcover raw …

• logs:

  filters

  ,

  velocity

  (SDK only has

  search

  )
• traces:

  attributes

  ,

  details

  ,

  errors

  ,

  filters

  ,

  insights

  ,

  latencies

  ,

  requests

  ,

  values-distribution

  (SDK only has

  search

  )
• metrics:

  cardinality

  ,

  cardinality-graph

  ,

  discovery

  ,

  labels-cardinality

  ,

  query-range

  ,

  resources errors|latencies|list|requests

  (SDK has

  query

  ,

  names

  ,

  keys

  ,

  values

  )
• prometheus:

  prometheus api query

  (raw Prom passthrough — handy for ad-hoc PromQL via

  --query query='up'

  )
• Grafana dashboards:

  grafana search

  ,

  grafana dashboards get|save|delete

  ,

  grafana dashboards permissions get|update

  ,

  grafana dashboards versions|get|restore

  ,

  grafana folders list|get|create|update|delete

  ,

  grafana folders permissions get|update

  ,

  grafana annotations list|create|update|delete

  ,

  grafana prometheus rules

  ,

  grafana datasources label-values

  ,

  grafana ds query

• monitors drilldowns:

  instances filters|query|timeline

  ,

  labels keys

  ,

  silences

  ,

  summary filters|query

  ,

  timeline

  (SDK only has CRUD)
• k8s drilldowns:

  configmaps|cronjob|daemonsets|deployments|jobs|pods|pvcs|replicasets|statefulsets list

  ,

  container info

  ,

  context events

  ,

  namespaces info|list

  ,

  nodes info-with-resources|list|resources|usage top10

  ,

  pod container usage

  ,

  pods status-over-time

  ,

  workloads availability|events|usage top10

  ,

  network connections|cross-az|cross-az-regions|partners|throughput|top-connections

  ,

  events search-time-series

  (SDK only has

  clusters

  ,

  workloads

  ,

  events-search

  ,

  events-over-time

  )
• infra:

  infra hosts info-with-resources

• resources / RUM:

  resources apis errors|filters|latencies|list|requests

  ,

  rum sessions filters|query

  ,

  sources list

• pipelines stats:

  pipelines logs current-stats

  ,

  pipelines traces current-stats

  (SDK only does config CRUD)
• tenant / billing / RBAC reads:

  rbac seatsUsage

  ,

  rbac tenant ai-settings

  ,

  rbac tenant settings

  ,

  backend settings

  ,

  billing method

  ,

  agent token-budgets|token-usage|token-usage history|token-usage tenant

• storage management:

  storage-management get|update --data-type <logs|traces|events|measurements|monitor_instance>

  for retention, gcQL exception rules, and index-tier settings
• misc:

  graph

  ,

  graph filters

  ,

  views member

  ,

  views member defaults

  ,

  migrations

  ,

  connectors list org|personal

  ,

  synthetics rules

  ,

  aggregations metrics config|config default

  ,

  integrations data config

groundcover raw list

groundcover raw list <group>

• dashboards — also

  archive <id>

  ,

  restore <id>

• monitors —

  --query 'monitor_name = "cpu"'

• silences —

  list --active

• recurring-silences
• connected-apps —

  --query 'type:slack-webhook'

• notification-routes —

  --query 'prod'

• synthetics
• secrets — also

  hash <id>

• workflows

END=$(date -u +%FT%TZ); START=$(date -u -v-1H +%FT%TZ)   # GNU: date -u -d '1 hour ago' +%FT%TZ
jq -n --arg s "$START" --arg e "$END" '{
  start:$s, end:$e,
  query: "service_name = \"api\" AND level = \"error\"",
  filters: "", sources: []
}' | groundcover logs search --body-file /dev/stdin | jq '.[] | {ts:.timestamp, pod:.pod_name, body:(.body|fromjson?)}'

logs search

returns a bare JSON array at the root —

jq '.[]'

, not

.logs[]

. The human-readable message lives in the

body

field as a JSON string (parse with

fromjson

). Useful top-level fields:

pod_name

,

container_name

,

namespace

,

workload

,

level

(normalized lowercase —

error

/

warning

/

debug

, not the

WARN

/

DEBG

you see inside

body

),

cluster

,

timestamp

,

trace_id

.

• Equality:

  service_name = "api"

  (double quotes inside JSON →

  \"

  )
• Comparison:

  duration_ms > 1000

  ,

  status_code >= 500

• Boolean:

  AND

  ,

  OR

  ,

  NOT

• Substring:

  message ~ "timeout"

  , regex:

  message ~* "^panic:"

• Membership:

  level IN ("error", "fatal")

  — works on a named field only
• Freetext phrase: a bare quoted string, e.g.

  container = "my-service" AND "database unavailable"

  .

  IN(...)

  is not valid as freetext (errors

  freetext 'in(...)' is not supported

  ) — expand to

  level = "error" OR level = "fatal"

  .
• Never send an empty

  query

  —

  query: ""

  → 500

  pipeline cannot be nil

  . Always supply at least one term.
• Null check:

  trace_id IS NOT NULL

search discovery

search keys

Limit

Type

keys

body

dashboards get <uuid>

.preset

fromjson

json.loads

{
  "spec": { "layoutType": "ordered", "crosshairSyncEnabled": true },
  "layout": [ /* one entry per TOP-LEVEL item, placed on a 24-col grid */ ],
  "widgets": [ /* the actual panels, referenced by id from layout */ ],
  "duration": "Last 6 hours",   // UI duration string
  "variables": [ /* template vars, same shape as the deep-link recipe */ ],
  "schemaVersion": 7
}

• layout

  — top-level grid placement. Each entry

  { "id":"A", "h":12, "w":24, "x":0, "y":0, "minH":8 }

  . Grid is 24 columns wide (

  w:8

  = a third,

  w:12

  = half);

  id

  must match a widget

  id

  . You hand-compute

  y

  to stack rows — there is no auto-flow.

• widgets

  —

  { id, name, type, queries:[{id, expr, dataType:"metrics", editorMode:"editor"}], visualizationConfig:{ type, selectedUnit, ... } }

  .

  expr

  is PromQL with

  $var

  substitutions.
• Widget types:

  time-series

  ,

  stat

  ,

  text

  (markdown in

  content

  , no queries — use for headers/dividers),

  section

  .

400 {"message":"Dashboard validation failed"}

1. Stat widgets (

   visualizationConfig.type:"stat"

   ) validate ONLY as flat top-level

   layout

   entries — NEVER inside a section's

   children[]

   . Time-series widgets work in both. To build an overview stat row, place the stat panels as flat top-level items (with a

   text

   header widget above them) and keep time-series panels in sections.
2. Section

   color

   accepts only a fixed palette. Verified valid:

   gray

   ,

   purple

   ,

   teal

   . Verified rejected:

   green

   ,

   blue

   ,

   red

   ,

   yellow

   ,

   orange

   .

• Every query returns data: pull the deployed preset, substitute concrete values into each

  expr

  (a

  $var

  → a real label value; a wildcard var → a

  =~".+"

  regex), and run it through

  metrics query

  as a range query. Use a window ≥ the metric's emit cadence (sparse integration metrics may only emit every few minutes).
• Geometry: assert no two layout rects overlap and every

  x+w ≤ 24

  (top-level entries and each section's

  children

  ).
• Caveat: the CLI can't reliably run instant queries (see Common issues), so you can't reproduce the stat-panel instant path from the CLI — reason from staleness + range data.

• They lag real-time (see the stat-panel note above) — minutes, not seconds.
• They're often multiplied by a

  stat

  dimension (

  stat=average/maximum/minimum

  ) — every series appears 3×. Always filter to the one you want (e.g.

  stat="average"

  ) or aggregations double/triple-count.
• The env/scope label is integration-named, not a clean

  env

  . If the same resource is also covered by an in-cluster exporter, the two families carry different label keys for the same concept (e.g.

  instance_id

  vs

  instanceid

  ). Use a separate template variable per label scheme rather than forcing one variable to span both.

1. Scope the time window. Default to last 1h. Compute

   START

   /

   END

   with

   date -u

   (see recipes).
2. Logs first.

   groundcover logs search

   with a GCQL filter on

   service_name

   and

   level = "error"

   . Cheap and usually answers it.
3. Traces if latency / slow.

   groundcover traces search

   filtering on

   duration_ms

   ,

   status_code

   , or

   service_name

   . Grab a few

   trace_id

   s and pivot back to logs via

   trace_id = "..."

   .
4. Metrics for shape over time.

   groundcover metrics query

   with PromQL — error rate, p99, saturation. Use

   range

   for graphs,

   instant

   for a point check.
5. K8s if "is it even running".

   groundcover k8s workloads

   for status/restarts;

   groundcover k8s events-search

   for

   Warning

   events (OOMKilled, FailedScheduling, BackOff).
6. Don't know the field name?

   groundcover search discovery

   ,

   metrics names

   ,

   metrics keys

   ,

   metrics values

   to enumerate.
7. Need to silence noisy alerts during the incident?

   groundcover monitors list --query …

   →

   groundcover silences create

   .

1. Check the resource list above.

2. groundcover --help

   and

   groundcover <resource> --help

   .
3. For raw:

   groundcover raw list

   , then

   groundcover raw list <group>

   .

• Invalid token prefix

  /

  Invalid token length

  → the key isn't a

  gcsa_

  service-account key (see Auth). A

  gcik_

  ingestion key is the common mistake.

• 401

  with a valid-looking

  gcsa_

  key → the key is for a different tenant. For other tenants also set

  GROUNDCOVER_TENANT_UUID

  and (if applicable)

  GROUNDCOVER_BASE_URL

  .
• Otherwise verify the env var is actually exported in the current shell.

• Time window is wrong / too narrow.

  start

  must be before

  end

  , both RFC 3339 UTC.
• GCQL field name doesn't exist in this tenant. Use

  search discovery

  /

  search keys

  to enumerate.
• Query string isn't escaped — inside JSON,

  service_name = "api"

  becomes

  "service_name = \"api\""

  .

• Field names are capitalized (

  Promql

  ,

  Start

  ,

  End

  ,

  Step

  ,

  QueryType

  ) — lowercase is silently accepted by JSON but ignored.

• Step

  is a string (

  "60"

  ), not a number.
• For instant queries set

  "QueryType": "instant"

  and omit

  Step

  . Observed caveat: in testing the CLI

  metrics query

  has returned

  400 metricsQueryBadRequest

  for instant bodies (every shape tried —

  Start

  ==

  End

  ,

  Time

  , with/without

  QueryType

  ), even on fresh metrics. When you need a single current value, fall back to a range query (

  QueryType:"range"

  +

  Step

  ) over a short window and take the last point. (Genuine instant PromQL is also reachable via

  raw prometheus api query

  , but that passthrough may target a different store than

  metrics query

  .)
• Parsing the response: results live at

  .data.result // .result

  ; each

  values

  entry is

  [unix_ts, "stringvalue"]

  . Sum a range with

  [.values[][1]] | map(tonumber) | add

  ; format the timestamp with

  strftime

  .

• Always include a discriminating term (a phrase +

  container

  /

  pod

  ), not just a pod by itself.
• Narrow the window and raise

  --timeout

  (e.g.

  --timeout 120s

  ).
• Avoid large

  OR

  chains in one query — run them separately.
• For "how often / over time" questions prefer

  metrics query

  over scraping logs.

• monitors get

  /

  update

  speak YAML (even with

  --raw

  ). Workflow:

  monitors get <uuid> > mon.yaml

  , edit,

  monitors update <uuid> --body-file mon.yaml

  . A successful update returns

  {"status":202}

  (applied asynchronously — re-

  get

  to confirm).
• Monitor model lives under

  model.queries[].sqlPipeline.filters.conditions[]

  — each condition is

  {key, origin: root, type, filters:[{op, value}]}

  ; freetext uses

  type: freetext

  +

  op: phrase_search

  . Other knobs:

  instantRollup

  (e.g.

  1 minute

  ),

  thresholds[]

  ,

  evaluationInterval.{interval, pendingFor}

  .

• monitors list --query 'monitor_name ~ "..."'

  → 400

  regexp requires a string column type

  . Use

  monitor_name = "exact"

  , or list all and filter with

  jq

  .