• Read path: ES|QL against

  .ml-anomalies-*

  ,

  .ml-config

  ,

  .ml-notifications-*

  ,

  .ml-annotations-*

• Always-available:

  platform.core.execute_esql

  (plus additional platform tools for search, index mapping, and documentation — see

  scripts/agent_builder_constants.json

  )
• ML API spec (if available):

  .kibana_ai_openapi_spec_elasticsearch

  — see references/anomaly-detection-openapi-spec-discover.md for discovery pattern.
• Run

  ad_validate_ml_tool_permissions

  first when tools return empty/misleading results — missing privileges are the most common cause of false negatives. Full permissions matrix: references/permissions-matrix.md.

• record_score

  bands: >75 critical · 50–75 warning · 25–50 minor · <25 informational

• multi_bucket_impact ≥ 3

  → sustained shift (not a transient spike)

• initial_record_score >> record_score

  → renormalization (model saw worse anomalies later)

• actual << typical

  with

  count

  /

  low_count

  /

  low_mean

  → absence/outage, not just low value
• Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity

Full score definitions, renormalization mechanics, and

anomaly_score_explanation

components: references/score-reference.md.

.ml-anomalies-*

result_type

• bucket

  — bucket-level unusualness per

  bucket_span

  .

  anomaly_score

  is the aggregate across all detectors.

• record

  — finest-grained rows with

  actual

  vs

  typical

  ,

  probability

  ,

  record_score

  ,

  anomaly_score_explanation

  .

• influencer

  — entity contributions ranked within a bucket (

  influencer_score

  ).

• anomaly_score

  /

  record_score

  = current normalized values (move as the model sees new extremes).

• initial_anomaly_score

  /

  initial_record_score

  = immutable snapshots from detection time.
• Compare

  actual

  to

  typical

  ; use

  probability

  for raw likelihood.
• Map entities via

  partition_field_value

  /

  by_field_value

  /

  over_field_value

  .
• Read

  multi_bucket_impact

  (-5 to +5) to separate single-bucket spikes from sustained trends.

1. Multi-job entities are prime suspects; single-job entities are usually victims. Use

   min_job_count=2

   .
2. Earliest anomaly timestamp wins — sort

   ad_rca_correlation

   by timestamp; first-appearing entity = origin.

3. multi_bucket_impact ≥ 3

   = sustained behavioral shift, weight higher than transient spikes.
4. Never close an RCA without

   ad_rca_source_evidence

   — raw source documents are ground truth.
5. Use low

   min_score

   (25 or lower) for influencer queries — high thresholds miss correlated entities.

• Unexpectedly low:

  high_variance_penalty

  , renormalization, <3 weeks training for weekly seasonality,

  bucket_span

  too large, wrong detector function (

  mean

  vs

  high_mean

  ),

  incomplete_bucket_penalty

  , suppression by

  custom_rules

  .
• Unexpectedly high: insufficient history (early training over-flags), high-cardinality split (too few points per entity),

  use_null: true

  on a sparse field.

1. ad_get_jobs

   — ≥3 weeks data for weekly seasonality?

2. ad_ts_model_memory_health

   —

   memory_status

   healthy?

3. ad_ts_delayed_data_annotations

   — no incomplete buckets?

4. ad_query_anomaly_records

   — compare

   record_score

   vs

   initial_record_score

   .

5. ad_get_job_datafeed_config

   —

   bucket_span

   , detector function,

   custom_rules

   ,

   use_null

   .

6. ad_get_model_plot

   — wide bounds →

   high_variance_penalty

   .

7. ad_rca_score_reassessment

   — renormalization drift across history.
8. Explain

   anomaly_score_explanation

   factors.

1. Always show both

   initial_record_score

   and

   record_score

   — the gap is the renormalization story.
2. Explain renormalization before diagnosing config — score drift is the most common "score dropped" cause and needs no config change.

3. actual << typical

   with

   count

   /

   low_count

   is an absence anomaly — distinguish outages from value spikes.

4. high_variance_penalty

   and

   incomplete_bucket_penalty

   explain most "low score" surprises without remediation.
5. Weekly seasonality needs ≥3 weeks of training data — flag young jobs as the cause.

query_delay

ad_ts_delayed_data_annotations

ad_ts_bucket_event_gaps

ad_ts_ingest_latency_estimate

ad_update_datafeed_query_delay

ad_wf_troubleshoot_query_delay

soft_limit

hard_limit

ad_ts_model_memory_health

ad_wf_ts_field_cardinality

ad_estimate_memory_requirement

ad_update_model_memory_limit

ad_wf_troubleshoot_memory_limit

ad_get_jobs

ad_get_job_messages

ad_manage_datafeed

remote_cluster:

ad_ts_ccs_diagnostics

ad_wf_troubleshoot_anomaly_score

hard_limit

corrupts model state and causes downstream missing-doc false alarms (categorizer silently skips events for unknown categories). Fix memory before fixing

query_delay

.

• query_delay

  — how far behind real time the datafeed queries. Too small → missing docs; too large → slower alerts. Set to P95 ingest latency + buffer (default

  60s

  –

  120s

  ).

• delayed_data_check_config

  — how aggressively the datafeed checks for late data.

• bucket_span

  — analysis interval. Align with data granularity and detection window.

• frequency

  — defaults to

  min(query_delay, bucket_span / 2)

  .

1. Stop datafeed:

   ad_manage_datafeed

   (

   action=_stop

   )
2. Close job
3. Update config:

   ad_update_model_memory_limit

   ,

   ad_update_datafeed_query_delay

   ,

   ad_update_delayed_data_check_config

4. Open job:

   ad_open_job

5. Start datafeed:

   ad_manage_datafeed

   (

   action=_start

   )

ad_revert_model_snapshot

1. ad_validate_ml_tool_permissions

   first — missing privileges produce misleading empty results.
2. Fix memory before

   query_delay

   —

   hard_limit

   corrupts state;

   query_delay

   fixes on a memory-limited job are wasted.
3. Stop the datafeed before updating it. Updating a running datafeed is rejected.
4. Close the job before updating memory limit. Sequence above.
5. Prefer workflow tools (

   ad_wf_*

   ) over manually chaining diagnostics for complex decisions.

6. ad_preview_datafeed_with_latency

   before starting — confirm the datafeed returns data after config changes.

1. Build configs. Parse the user request into job + datafeed JSON with no null fields.
2. Apply smart defaults:

   Field	Default	Override when
   bucket_span	"15m"	User specifies a different span
   time_field	"@timestamp"	User names a different timestamp field
   index	"logs-*"	User specifies an index or pattern
   datafeed_query	{"match_all": {}}	User mentions filters, processes, or time windows
   influencers	by/over/partition fields from detectors	User adds extra influencer fields
   job_id	Generated from user description	User provides an explicit ID
   query_delay	"60s"	P95 ingest latency is higher

3. Choose detector function from user intent — full table in references/anomaly-detection-functions.md:
   • "high CPU" / "unusually large" →

     high_mean

     or

     high_sum

   • "rare logins" / "unusual values" →

     rare

     (variants below)
   • "too many requests" / "spike in count" →

     high_count

   rare

   variants:
   • Infrequent globally →

     rare by_field_name: X

   • Infrequent vs peers →

     rare by_field_name: X over_field_name: Y

   • Infrequent per segment →

     rare by_field_name: X partition_field_name: Y

   • Infrequent per segment vs peers →

     rare by_field_name: X over_field_name: Y partition_field_name: Z

4. Validate.

   platform.core.get_index_mapping

   on the target index to verify field existence/types →

   ad_validate_job_spec

   . If errors, fix and re-validate (max 3 attempts).
5. Present and confirm. Show the complete job + datafeed bodies formatted as the exact API calls. Ask for approval once. If feedback, incorporate and re-present (up to 3 rounds).
6. Deploy. After confirmation:

   ad_create_job

   →

   ad_create_datafeed

   →

   ad_open_job

   →

   ad_manage_datafeed

   (

   action=_start

   ). Report final

   job_id

   and

   datafeed_id

   .

start

end

Worked examples (rare-username, DNS exfil, large-downloads) with full JSON bodies and datafeed filters: references/job-creation-recipes.md.

1. Create job before datafeed. Datafeed references job by ID.
2. Open job before starting datafeed. Start on a closed job is rejected.

3. query_delay

   = P95 ingest latency + buffer (60s–120s safe default).
4. Forecasts require non-population jobs —

   over_field_name

   jobs cannot be forecasted; warn before attempting.

5. by_field_name

   vs

   over_field_name

   :

   by

   compares entity to its own history;

   over

   compares to peer group in the same bucket.

   partition_field_name

   = fully independent sub-model with its own normalization.

6. bucket_span

   matches detection granularity — 15m for high-frequency, 1h for operational metrics, 1d for daily patterns. Larger smooths short spikes; smaller increases noise.

1. Pick a mode first. Don't blend RCA logic with score-explanation logic in one response.

2. ad_validate_ml_tool_permissions

   first on empty results — privileges are the most common false-negative cause.
3. Score bands are absolute thresholds:

   >75

   critical,

   50–75

   warning,

   25–50

   minor,

   <25

   informational.
4. Multi-job entities are prime suspects. Use

   min_job_count=2

   in

   ad_rca_multi_job_entities

   .
5. Show

   initial_record_score

   alongside

   record_score

   — the gap tells the renormalization story.
6. Fix memory before

   query_delay

   .

   hard_limit

   invalidates downstream diagnostics.
7. Stop datafeed → close job → update config → open job → start datafeed for any config change to memory or query delay.
8. Confirm RCAs with

   ad_rca_source_evidence

   . Raw source documents are ground truth.