arize-trace

Original🇺🇸 English
Translated

INVOKE THIS SKILL when downloading or exporting Arize traces and spans. Covers exporting traces by ID, sessions by ID, and debugging LLM application issues using the ax CLI.

27installs
Added on

NPX Install

npx skill4agent add arize-ai/arize-skills arize-trace

Tags

Translated version includes tags in frontmatter

Arize Trace Skill

Concepts

  • Trace = a tree of spans sharing a
    context.trace_id
    , rooted at a span with
    parent_id = null
  • Span = a single operation (LLM call, tool call, retriever, chain, agent)
  • Session = a group of traces sharing
    attributes.session.id
    (e.g., a multi-turn conversation)
Use
ax spans export
to download trace data. This is the only supported command for retrieving spans.
Exploratory export rule: When exporting spans or traces without a specific
--trace-id
,
--span-id
, or
--session-id
(i.e., browsing/exploring a project), always start with
-l 50
to pull a small sample first. Summarize what you find, then pull more data only if the user asks or the task requires it. This avoids slow queries and overwhelming output on large projects.
Default output directory: Always use
--output-dir .arize-tmp-traces
on every
ax spans export
call. The CLI automatically creates the directory and adds it to
.gitignore
.

Prerequisites

Three things are needed:
ax
CLI, an API key (env var or profile), and a space ID. A project name is also needed but usually comes from the user's message.

Install ax

Verify
ax
is installed and working before proceeding:
  1. Check if
    ax
    is on PATH:
    command -v ax
    (Unix) or
    where ax
    (Windows)
  2. If not found, check common install locations:
    • macOS/Linux:
      test -x ~/.local/bin/ax && export PATH="$HOME/.local/bin:$PATH"
    • Windows: check
      %APPDATA%\Python\Scripts\ax.exe
      or
      %LOCALAPPDATA%\Programs\Python\Scripts\ax.exe
  3. If still not found, install it (requires shell access to install packages):
    • Preferred:
      uv tool install arize-ax-cli
    • Alternative:
      pipx install arize-ax-cli
    • Fallback:
      pip install arize-ax-cli
  4. After install, if
    ax
    is not on PATH:
    • macOS/Linux:
      export PATH="$HOME/.local/bin:$PATH"
    • Windows (PowerShell):
      $env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"
  5. If
    ax --version
    fails with an SSL/certificate error:
    • macOS:
      export SSL_CERT_FILE=/etc/ssl/cert.pem
    • Linux:
      export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
    • Windows (PowerShell):
      $env:SSL_CERT_FILE = "C:\Program Files\Common Files\SSL\cert.pem"
      (or use
      python -c "import certifi; print(certifi.where())"
      to find the cert bundle)
  6. ax --version
    must succeed before proceeding. If it doesn't, stop and ask the user for help.

Verify environment

Run a quick check for credentials:
macOS/Linux (bash):
bash
ax --version && echo "--- env ---" && echo "ARIZE_API_KEY: ${ARIZE_API_KEY:-(not set)}" && echo "ARIZE_SPACE_ID: ${ARIZE_SPACE_ID:-(not set)}" && echo "--- profiles ---" && ax profiles show 2>&1
Windows (PowerShell):
powershell
ax --version; Write-Host "--- env ---"; Write-Host "ARIZE_API_KEY: $env:ARIZE_API_KEY"; Write-Host "ARIZE_SPACE_ID: $env:ARIZE_SPACE_ID"; Write-Host "--- profiles ---"; ax profiles show 2>&1
Read the output and proceed immediately if either the env var or the profile has an API key. Only ask the user if both are missing. Resolve failures:
  • No API key in env and no profile → AskQuestion: "Arize API key (https://app.arize.com/admin > API Keys)"
  • Space ID unknown → AskQuestion, or run
    ax projects list -o json --limit 100 --space-id $ARIZE_SPACE_ID
    and present as selectable options
  • Project unclear → ask, or run
    ax projects list -o json --limit 100
    and search for a match
IMPORTANT:
--space-id
is required when using a human-readable project name as the
PROJECT
positional argument. It is not needed when using a base64-encoded project ID.

Export Spans:
ax spans export

The primary command for downloading trace data to a file.

By trace ID

bash
# Using project name (requires --space-id)
ax spans export PROJECT_NAME --trace-id TRACE_ID --space-id SPACE_ID --output-dir .arize-tmp-traces

# Using base64 project ID (no --space-id needed)
ax spans export PROJECT_ID --trace-id TRACE_ID --output-dir .arize-tmp-traces

By span ID

bash
ax spans export PROJECT_NAME --span-id SPAN_ID --space-id SPACE_ID --output-dir .arize-tmp-traces

By session ID

bash
ax spans export PROJECT_NAME --session-id SESSION_ID --space-id SPACE_ID --output-dir .arize-tmp-traces

Flags

FlagTypeRequiredDescription
--trace-id
stringmutexFilter:
context.trace_id = 'X'
--span-id
stringmutexFilter:
context.span_id = 'X'
--session-id
stringmutexFilter:
attributes.session.id = 'X'
PROJECT
string (positional)yes (or
$ARIZE_DEFAULT_PROJECT
)
Project name or base64 ID (positional arg, not a flag)
--space-id
stringyes (when
PROJECT
is a name)
Space ID; required to resolve project names
--days
intnoLookback window (default: 30)
--start-time
stringnoOverride start (ISO 8601)
--end-time
stringnoOverride end (ISO 8601)
--output-dir
stringnoOutput directory (default:
.arize-tmp-traces
; ensure it is gitignored — see above)
--stdout
boolnoPrint JSON to stdout instead of file
--all
boolnoUse Arrow Flight for bulk export (see below)
Exactly one of
--trace-id
,
--span-id
,
--session-id
is required.
Output is a JSON array of span objects. File naming:
{type}_{id}_{timestamp}/spans.json
.

Bulk export with
--all
(Arrow Flight)

By default,
ax spans export
uses the REST API which is limited to 500 spans per page and capped by
--limit
. Pass
--all
to switch to Arrow Flight for streaming bulk export with no span limit.
bash
ax spans export PROJECT_NAME --space-id SPACE_ID --filter "status_code = 'ERROR'" --all --output-dir .arize-tmp-traces
REST vs Flight trade-offs:
  • REST (default): Lower friction -- no Arrow/Flight dependency needed, uses standard HTTPS ports, works through any corporate proxy or firewall. Limited to 500 spans per page.
  • Flight (
    --all
    ): Required for bulk export beyond 500 spans. Uses gRPC+TLS on a separate host/port which some corporate networks may block.
When to use
--all
:
  • Exporting more than 500 spans
  • Downloading full traces with many child spans
  • Large time-range exports
Agent auto-escalation rule: If a REST export returns exactly the number of spans requested by
-l
(or 500 if no limit was set), the result is likely truncated. Increase
-l
or re-run with
--all
to get the full dataset — but only when the user asks or the task requires more data.
Requirements for
--all
:
  • --space-id
    is required (Flight uses
    space_id
    +
    project_name
    , not
    project_id
    )
  • --limit
    is ignored when
    --all
    is set
Networking notes for
--all
:
Arrow Flight connects to
flight.arize.com:443
via gRPC+TLS -- this is a different host from the REST API (
api.arize.com
). On internal or private networks, the Flight endpoint may use a different host/port. Configure via:
  • ax profile:
    flight_host
    ,
    flight_port
    ,
    flight_scheme
  • Environment variables:
    ARIZE_FLIGHT_HOST
    ,
    ARIZE_FLIGHT_PORT
    ,
    ARIZE_FLIGHT_SCHEME
The
--all
flag is also available on
ax traces export
,
ax datasets export
, and
ax experiments export
with the same behavior (REST by default, Flight with
--all
).

Export Traces:
ax traces export

Export full traces -- all spans belonging to traces that match a filter. Uses a two-phase approach:
  1. Phase 1: Find spans matching
    --filter
    (up to
    --limit
    via REST, or all via Flight with
    --all
    )
  2. Phase 2: Extract unique trace IDs, then fetch every span for those traces
bash
# Explore recent traces (start small with -l 50, pull more if needed)
ax traces export PROJECT_NAME --space-id SPACE_ID -l 50 --output-dir .arize-tmp-traces

# Export traces with error spans (REST, up to 500 spans in phase 1)
ax traces export PROJECT_NAME --space-id SPACE_ID --filter "status_code = 'ERROR'" --stdout

# Export all traces matching a filter via Flight (no limit)
ax traces export PROJECT_NAME --space-id SPACE_ID --filter "status_code = 'ERROR'" --all

Flags

FlagTypeDefaultDescription
PROJECT
stringrequiredPositional argument (name or base64 ID)
--filter
stringnoneFilter expression for phase-1 span lookup
--space-id
stringnoneSpace ID; required when PROJECT is a name or when using
--all
--limit, -l
int50Max number of traces to export
--days
int30Lookback window in days
--start-time
stringnoneOverride start (ISO 8601)
--end-time
stringnoneOverride end (ISO 8601)
--output-dir
string
.
Output directory
--stdout
boolfalsePrint JSON to stdout instead of file
--all
boolfalseUse Arrow Flight for both phases (see spans
--all
docs above)
-p, --profile
stringdefaultConfiguration profile

How it differs from
ax spans export

  • ax spans export
    exports individual spans matching a filter
  • ax traces export
    exports complete traces -- it finds spans matching the filter, then pulls ALL spans for those traces (including siblings and children that may not match the filter)

Filter Syntax Reference

SQL-like expressions passed to
--filter
.

Common filterable columns

ColumnTypeDescriptionExample Values
name
stringSpan name
'ChatCompletion'
,
'retrieve_docs'
status_code
stringStatus
'OK'
,
'ERROR'
,
'UNSET'
latency_ms
numberDuration in ms
100
,
5000
parent_id
stringParent span IDnull for root spans
context.trace_id
stringTrace ID
context.span_id
stringSpan ID
attributes.session.id
stringSession ID
attributes.openinference.span.kind
stringSpan kind
'LLM'
,
'CHAIN'
,
'TOOL'
,
'AGENT'
,
'RETRIEVER'
,
'RERANKER'
,
'EMBEDDING'
,
'GUARDRAIL'
,
'EVALUATOR'
attributes.llm.model_name
stringLLM model
'gpt-4o'
,
'claude-3'
attributes.input.value
stringSpan input
attributes.output.value
stringSpan output
attributes.error.type
stringError type
'ValueError'
,
'TimeoutError'
attributes.error.message
stringError message
event.attributes
stringError tracebacksUse CONTAINS (not exact match)

Operators

=
,
!=
,
<
,
<=
,
>
,
>=
,
AND
,
OR
,
IN
,
CONTAINS
,
LIKE
,
IS NULL
,
IS NOT NULL

Examples

status_code = 'ERROR'
latency_ms > 5000
name = 'ChatCompletion' AND status_code = 'ERROR'
attributes.llm.model_name = 'gpt-4o'
attributes.openinference.span.kind IN ('LLM', 'AGENT')
attributes.error.type LIKE '%Transport%'
event.attributes CONTAINS 'TimeoutError'

Tips

  • Prefer
    IN
    over multiple
    OR
    conditions:
    name IN ('a', 'b', 'c')
    not
    name = 'a' OR name = 'b' OR name = 'c'
  • Start broad with
    LIKE
    , then switch to
    =
    or
    IN
    once you know exact values
  • Use
    CONTAINS
    for
    event.attributes
    (error tracebacks) -- exact match is unreliable on complex text
  • Always wrap string values in single quotes

Workflows

Debug a failing trace

  1. ax traces export PROJECT --space-id SPACE_ID --filter "status_code = 'ERROR'" -l 50 --output-dir .arize-tmp-traces
  2. Read the output file, look for spans with
    status_code: ERROR
  3. Check
    attributes.error.type
    and
    attributes.error.message
    on error spans

Download a conversation session

  1. ax spans export PROJECT --session-id SESSION_ID --space-id SPACE_ID --output-dir .arize-tmp-traces
  2. Spans are ordered by
    start_time
    , grouped by
    context.trace_id
  3. If you only have a trace_id, export that trace first, then look for
    attributes.session.id
    in the output to get the session ID

Export for offline analysis

bash
ax spans export PROJECT --trace-id TRACE_ID --space-id SPACE_ID --output-dir .arize-tmp-traces --stdout | jq '.[]'

Span Column Reference (OpenInference Semantic Conventions)

Core Identity and Timing

ColumnDescription
name
Span operation name (e.g.,
ChatCompletion
,
retrieve_docs
)
context.trace_id
Trace ID -- all spans in a trace share this
context.span_id
Unique span ID
parent_id
Parent span ID.
null
for root spans (= traces)
start_time
When the span started (ISO 8601)
end_time
When the span ended
latency_ms
Duration in milliseconds
status_code
OK
,
ERROR
,
UNSET
status_message
Optional message (usually set on errors)
attributes.openinference.span.kind
LLM
,
CHAIN
,
TOOL
,
AGENT
,
RETRIEVER
,
RERANKER
,
EMBEDDING
,
GUARDRAIL
,
EVALUATOR

Where to Find Prompts and LLM I/O

Generic input/output (all span kinds):
ColumnWhat it contains
attributes.input.value
The input to the operation. For LLM spans, often the full prompt or serialized messages JSON. For chain/agent spans, the user's question.
attributes.input.mime_type
Format hint:
text/plain
or
application/json
attributes.output.value
The output. For LLM spans, the model's response. For chain/agent spans, the final answer.
attributes.output.mime_type
Format hint for output
LLM-specific message arrays (structured chat format):
ColumnWhat it contains
attributes.llm.input_messages
Structured input messages array (system, user, assistant, tool). Where chat prompts live in role-based format.
attributes.llm.input_messages.roles
Array of roles:
system
,
user
,
assistant
,
tool
attributes.llm.input_messages.contents
Array of message content strings
attributes.llm.output_messages
Structured output messages from the model
attributes.llm.output_messages.contents
Model response content
attributes.llm.output_messages.tool_calls.function.names
Tool calls the model wants to make
attributes.llm.output_messages.tool_calls.function.arguments
Arguments for those tool calls
Prompt templates:
ColumnWhat it contains
attributes.llm.prompt_template.template
The prompt template with variable placeholders (e.g.,
"Answer {question} using {context}"
)
attributes.llm.prompt_template.variables
Template variable values (JSON object)
Finding prompts by span kind:
  • LLM span: Check
    attributes.llm.input_messages
    for structured chat messages, OR
    attributes.input.value
    for serialized prompt. Check
    attributes.llm.prompt_template.template
    for the template.
  • Chain/Agent span: Check
    attributes.input.value
    for the user's question. Actual LLM prompts are on child LLM spans.
  • Tool span: Check
    attributes.input.value
    for tool input,
    attributes.output.value
    for tool result.

LLM Model and Cost

ColumnDescription
attributes.llm.model_name
Model identifier (e.g.,
gpt-4o
,
claude-3-opus-20240229
)
attributes.llm.invocation_parameters
Model parameters JSON (temperature, max_tokens, top_p, etc.)
attributes.llm.token_count.prompt
Input token count
attributes.llm.token_count.completion
Output token count
attributes.llm.token_count.total
Total tokens
attributes.llm.cost.prompt
Input cost in USD
attributes.llm.cost.completion
Output cost in USD
attributes.llm.cost.total
Total cost in USD

Tool Spans

ColumnDescription
attributes.tool.name
Tool/function name
attributes.tool.description
Tool description
attributes.tool.parameters
Tool parameter schema (JSON)

Retriever Spans

ColumnDescription
attributes.retrieval.documents
Retrieved documents array
attributes.retrieval.documents.ids
Document IDs
attributes.retrieval.documents.scores
Relevance scores
attributes.retrieval.documents.contents
Document text content
attributes.retrieval.documents.metadatas
Document metadata

Reranker Spans

ColumnDescription
attributes.reranker.query
The query being reranked
attributes.reranker.model_name
Reranker model
attributes.reranker.top_k
Number of results
attributes.reranker.input_documents.*
Input documents (ids, scores, contents, metadatas)
attributes.reranker.output_documents.*
Reranked output documents

Session, User, and Custom Metadata

ColumnDescription
attributes.session.id
Session/conversation ID -- groups traces into multi-turn sessions
attributes.user.id
End-user identifier
attributes.metadata.*
Custom key-value metadata. Any key under this prefix is user-defined (e.g.,
attributes.metadata.user_email
). Filterable.

Errors and Exceptions

ColumnDescription
attributes.exception.type
Exception class name (e.g.,
ValueError
,
TimeoutError
)
attributes.exception.message
Exception message text
event.attributes
Error tracebacks and detailed event data. Use
CONTAINS
for filtering.

Evaluations and Annotations

ColumnDescription
annotation.<name>.label
Human or auto-eval label (e.g.,
correct
,
incorrect
)
annotation.<name>.score
Numeric score (e.g.,
0.95
)
annotation.<name>.text
Freeform annotation text

Embeddings

ColumnDescription
attributes.embedding.model_name
Embedding model name
attributes.embedding.texts
Text chunks that were embedded

Troubleshooting

ProblemSolution
ax: command not found
Check
~/.local/bin/ax
; if missing:
uv tool install arize-ax-cli
(requires shell access to install packages). Then
export PATH="$HOME/.local/bin:$PATH"
SSL: CERTIFICATE_VERIFY_FAILED
macOS:
export SSL_CERT_FILE=/etc/ssl/cert.pem
. Linux:
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
. Windows:
$env:SSL_CERT_FILE = (python -c "import certifi; print(certifi.where())")
No such command
on a subcommand that should exist
The installed
ax
is outdated. Reinstall from the local workspace:
uv tool install --force --reinstall /path/to/arize/sdk/python/arize-ax-cli
(requires shell access to install packages)
No profile found
Follow "Resolve credentials" in Prerequisites to auto-discover or prompt for the API key
401 Unauthorized
with valid API key
You are likely using a project name (e.g.,
my-project
) without
--space-id
. Add
--space-id SPACE_ID
or use the base64 project ID instead
No spans found
Expand
--days
(default 30), verify project ID
Filter error
Check column name spelling, wrap string values in single quotes
Timeout on large export
Use
--days 7
to narrow the time range

Save Credentials

At session end, if the user manually provided an API key, space ID, or project name (not loaded from an existing profile), offer to save them to
~/.arize/config.toml
. Use AskQuestion with "Yes, save them" / "No thanks". Skip if all values were already in the profile.
Read the existing file (or create it), add/update only the new fields, and write it back:
toml
[auth]
api_key = "THE_API_KEY"

[defaults]
space_id = "THE_SPACE_ID"
project = "THE_PROJECT_NAME"