• Create, set up, or configure a new alert rule.
• Get paged or notified when a metric, log volume, latency, or error rate crosses a threshold.
• Detect anomalous behavior on a service, host, or signal.
• Catch silent data loss ("alert if data stops arriving from X").

• Understand what an existing alert monitors →

  signoz-explaining-alerts

  .
• Diagnose why an existing alert fired →

  signoz-investigating-alerts

  .
• Modify thresholds, queries, or routing on an existing alert → call

  signoz:signoz_update_alert

  directly.

1. What to monitor — signal type (metrics / logs / traces / exceptions) and the specific condition (CPU, error rate, p99 latency, log count, ...).
2. Resource scope — which service, host, namespace, or environment.
3. Threshold — numeric value and comparison ("above 80%", "below 100/s").
4. Severity — implicit from urgency words ("page" → critical, default warning otherwise).
5. Channel — explicit channel name if the user provided one.

needs_input

1. Call

   signoz:signoz_get_field_keys

   with

   fieldContext=resource

   to enumerate resource attributes for the chosen signal.
2. Call

   signoz:signoz_get_field_values

   for the most likely attribute (typically

   service.name

   , then

   host.name

   , then

   k8s.namespace.name

   ) to get concrete values.
3. If the user mentioned a metric by name, call

   signoz:signoz_list_metrics

   with a search term to verify the exact OTel metric name. Wrong names create alerts that never fire.

needs_input

• Metrics:

  signoz:signoz_query_metrics

  (PromQL) or

  signoz:signoz_execute_builder_query

  with

  count()

  (or

  count_distinct(service.name)

  if scope-discovering).
• Logs:

  signoz:signoz_aggregate_logs

  with

  count()

  over the filter.
• Traces:

  signoz:signoz_aggregate_traces

  with

  count()

  over the filter.

• Probe returns rows → proceed to Step 3.
• Probe returns empty → STOP. Do not build an alert config the user will then be asked to throw away. Emit a

  needs_input

  block describing what was missing and offer concrete recovery:
  • Service doesn't emit the metric → call

    signoz:signoz_get_field_values signal=metrics name=service.name metricName=<metric>

    to list the services that do emit it; let the user pick a different service or a different metric.
  • Wrong attribute name (

    service

    instead of

    service.name

    ) → suggest the semantic-convention name and re-probe.
  • Service emits the metric but not in the expected time range → widen the probe window once (e.g. last 24h) before declaring no-data.

signoz://alert/instructions

signoz://alert/examples

METRIC_BASED_ALERT

signoz://alert/examples

signoz-generating-queries

op

matchType

"above"

"on_average"

"1"

"3"

op: "above"

"above_or_below"

op

matchType

"above"

"at_least_once"

"below"

"all_the_times"

"equals"

"on_average"

"not_equals"

"in_total"

"last"

• evalWindow: 5m0s

  ,

  frequency: 1m0s

  — change only if the intent implies a slower or faster cadence.

• matchType: "3"

  (on_average) for CPU / memory / latency — smooths transient spikes.

• matchType: "1"

  (at_least_once) for error counts / error rates — catches any breach.

warning

warning

service.name

host.name

k8s.namespace.name

deployment.environment.name

service

host

env

• summary

  — single-line headline. Include the resource scope and the numeric value:

  "checkoutservice error rate {{$value}}% above 3%"

  .

• description

  — longer message. Include

  {{$value}}

  ,

  {{$threshold}}

  , the groupBy values (e.g.

  {{$labels.service_name}}

  ), and a sentence on what to do or where to look. For count-based alerts include the count explicitly:

  "{{$value}} crash log lines in the last 5 minutes from service {{$labels.service_name}}"

  .

{{$value}}

{{$threshold}}

{{$labels.<key>}}

service.name

service_name

1. Call

   signoz:signoz_list_notification_channels

   to enumerate existing channels.
2. If the user named a channel ("send to slack-infra"), use it if it exists; if not, fall through.
3. Otherwise present the user with two options:
   • Pick from existing — list channels with their type (Slack, PagerDuty, email, webhook) so the user can choose.
   • Create new inline — call

     signoz:signoz_create_notification_channel

     with channel parameters the user provides (name, type, type-specific config like Slack webhook URL or PagerDuty integration key).
4. If neither path resolves a channel, emit

   needs_input: notification_channel

   and stop.

thresholds.spec[N].channels

1. Run the alert's full primary query (or formula) over the last hour using:

   • signoz:signoz_execute_builder_query

     for builder/formula queries.

   • signoz:signoz_query_metrics

     for PromQL queries.

   • signoz:signoz_aggregate_logs

     /

     signoz:signoz_aggregate_traces

     if those fit better.
2. Compute how many evaluation points in the last hour breached the proposed threshold. Surface this in the preview as "would have fired N times in the last 1h":
   • N = 0 → the threshold may be too loose or the gating too strict. Mention this so the user can adjust if intent was tighter.
   • N is large (e.g. > 30) → likely alert storm. Surface and recommend tightening or adding hysteresis (

     recoveryTarget

     ).
   • N is small and non-zero → calibrated; proceed.
3. Exceptions:
   • Anomaly alerts — skip the breach count entirely (Z-scores aren't directly comparable to raw values). Step 2.5 already verified the underlying metric × service has data; nothing more to validate here.
   • Log-based crash / panic / OOMKilled / FATAL alerts — these intentionally have zero matches in a healthy system. Step 2.5 has already surfaced the zero-match result and obtained user confirmation; skip the breach count.

needs_input

signoz:signoz_create_alert

{
  "alert": "<name>",
  "alertType": "...",
  "ruleType": "...",
  "condition": { ... },
  "labels": { "severity": "..." },
  "annotations": { "description": "...", "summary": "..." },
  "evaluation": { ... },
  "preferredChannels": ["..."]
}

Summary: This alert fires when [condition] for [resource scope], evaluated every [frequency] over the last [window]. Thresholds: warning at X, critical at Y. Notifications go to [channels]. Dry-run on the last hour: would have fired N times.

1. Call

   signoz:signoz_create_alert

   with the JSON payload from Step 7.
2. Name collision — if

   signoz:signoz_create_alert

   returns a duplicate-name error, do not suffix-append or call

   signoz:signoz_update_alert

   . Stop and tell the user the existing alert blocked creation; offer to use a different name or modify the existing alert (which is out of scope for this skill).
3. On success, report:
   • The alert ID and name.
   • What it watches and at what threshold.
   • Which channels are wired up.
   • The dry-run summary ("would have fired N times in last 1h").
   • Two follow-up offers: "Want to test the query live with

     signoz-generating-queries

     ?" and "Want me to add a runbook URL to the annotations?"

• Strict inputs over guessing. Resource attribute and channel are required. If missing, emit

  needs_input

  and stop. Creating an alert on a guessed service is harder to undo than asking.
• Always paginate

  signoz:signoz_list_alerts

  . Stopping at page 1 misses duplicates and produces noise.
• Dry-run is mandatory. Saving an alert whose query returns no data is a silent failure mode and must be prevented.
• No duplicate updates. Name collision → error and stop. Do not silently update an existing alert from a "create" skill.
• OTel attribute names only.

  service.name

  not

  service

  .
• Threshold codes are strings, not words.

  op: "1"

  not

  op: "above"

  .
• Signal must match alertType.

  signal: "logs"

  requires

  LOGS_BASED_ALERT

  . Mismatches fail validation.
• Anomaly rules are metrics-only.

  anomaly_rule

  + non-metric alertType is rejected.
• Channels must exist. Use names from

  signoz:signoz_list_notification_channels

  exactly, or create the channel inline first.
• Scope boundary. This skill only creates new rules. Modifications use

  signoz:signoz_update_alert

  directly.

1. Intent parsed: metric, CPU, scope

   service.name = checkout

   , thresholds 80% (warning) / 90% (critical), severity bumped to critical on the higher level because "page me" was used.

2. signoz:signoz_list_metrics searchText=cpu

   → confirms

   system.cpu.utilization

   .

3. signoz:signoz_list_alerts

   (paginated) → no existing CPU alert for checkout.

4. signoz:signoz_list_notification_channels

   → presents existing channels; user picks

   slack-infra

   for warning and

   pagerduty-oncall

   for critical.
5. Builds JSON:

   METRIC_BASED_ALERT

   ,

   threshold_rule

   ,

   signal=metrics

   , two thresholds (

   op="1"

   ,

   matchType="3"

   on_average,

   targetUnit="percent"

   ), filter

   service.name = 'checkout'

   .
6. Dry-run via

   signoz:signoz_execute_builder_query

   over last 1h: returns data, would have fired 0 times (clean baseline).
7. Emits JSON preview + summary.
8. Calls

   signoz:signoz_create_alert

   . Reports created alert with ID, threshold summary, channel routing, and dry-run result.

1. Intent parsed: traces, error-rate formula, scope

   service.name = payments

   , threshold 5%, severity default warning.

2. signoz:signoz_get_field_keys fieldContext=resource signal=traces

   confirms

   service.name

   .

   signoz:signoz_get_field_values

   confirms

   payments

   exists.
3. No existing payments error-rate alert.
4. Builds formula alert: query A counts spans with

   hasError = true

   for

   service.name = 'payments'

   , query B counts all spans for the same service, formula F1 =

   A * 100 / B

   ,

   selectedQueryName: "F1"

   , threshold target 5,

   targetUnit: "percent"

   ,

   op: "1"

   ,

   matchType: "1"

   (catch any breach).
5. Channel: user picks

   slack-payments

   .
6. Dry-run on last 1h: payments error rate hovered around 0.3%, would have fired 0 times. Clean — not too tight.
7. Preview, save, report.

1. Intent parsed: logs, volume threshold, scope = all services (groupBy

   service.name

   ), threshold 1000/min, severity warning.

2. signoz:signoz_get_field_keys signal=logs fieldContext=resource

   confirms

   service.name

   exists; no service filter required since the user said "any service" → groupBy provides the scoping.
3. Builds:

   LOGS_BASED_ALERT

   ,

   signal=logs

   ,

   aggregations: [{expression: "count()"}]

   ,

   filter: {expression: "severity_text IN ('ERROR', 'FATAL')"}

   ,

   groupBy: [{name: "service.name", fieldContext: "resource", fieldDataType: "string"}]

   , threshold 1000,

   targetUnit: ""

   ,

   evalWindow: 1m0s

   ,

   matchType: "1"

   (catch any minute that breaches).
4. Channels: user picks slack channel.
5. Dry-run: returned per-service counts, max in last 1h was 87 — would have fired 0 times. Within reasonable headroom.
6. Preview, save, report.

1. Intent parsed: metric, anomaly detection, scope

   service.name = api-gateway

   . Anomaly detection requires

   METRIC_BASED_ALERT

   +

   anomaly_rule

   .

2. signoz:signoz_list_metrics searchText=duration

   → finds

   http.server.request.duration

   .
3. Builds:

   anomaly_rule

   ,

   algorithm=zscore

   ,

   seasonality=daily

   , threshold target 3 (3 standard deviations),

   op: "1"

   ,

   matchType: "1"

   .
4. Channel: user picks slack-api.
5. Dry-run validates query returns data. Skip breach-count for anomaly alerts.
6. Preview emphasizes that the threshold is in standard deviations, not raw latency. Save, report.

• signoz://alert/instructions

  and

  signoz://alert/examples

  MCP resources — full alert config JSON schema, threshold codes, filter expression syntax, and version-current pattern examples. Always preferred over any transcribed copy.

• signoz-generating-queries

  skill — for authoring PromQL or testing queries before wrapping them in an alert.