Apigee as an Agent Gateway Service Extension
Wires an Apigee proxy in as an ext-proc callout on a GCP Agent Gateway. The
proxy is a processor, not an API proxy: it has no backend, inspects the LLM
request and response the gateway hands it, and returns a verdict.
You build the proxy; the user does not.
Two modes, one schema. Decide which before anything else, and say which.
Input file present →
prod mode: it is the contract, every value comes from
it, and a missing or malformed field is a refusal, not a default. No input file
and the user is describing an outcome →
dev mode: apply the documented
defaults, show a plan with durations and get approval, start with identity and
metering only, write
apigee-agent-gateway-governance.yaml
at the end as the
handoff.
assets/apigee-agent-gateway-governance.example.yaml
documents the
schema both modes use.
Modes, defaults, durations, cost disclosure and the interlocks that bind both:
.
Read it before you start.
Colour the plan's ACTION column — green
, cyan
/
,
yellow
/
; full palette in
. Yellow is the
one that earns its place: it is the only signal that a resource the user already
owns is being changed. Rendered plain, the one row that modifies something is
indistinguishable from the dozen that do not — which is exactly the row the
approval exists for.
Prerequisite: where the agent runs
Check this before building anything. It decides whether traffic can reach
the gateway at all, and it is silent when wrong — every step below still
succeeds while nothing arrives.
Use Agent Runtime, created with
/
. It is the supported path and the orchestrator builds
the client-side PSC, routing, DNS and cert trust for you. There is no
surface for this and no console path — it is SDK-only, at agent creation time.
See
Route traffic through Agent Gateway
for the binding, and
references/networking-psc.md
for a worked example.
Set GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES: "False"
in the
agent's , or every invocation 401s on the agent's own
, before user code runs. Deploy-time invisible. Full symptom
and isolation:
, Agent Gateway.
Where the agent runs is otherwise settled — the premise is a working gateway and
a working agent. Do not explain how to build one or list unsupported compute
unprompted;
references/networking-psc.md
has the constraints if it turns out
to matter.
Tool versions
Do not assume these are installed or current.
| Tool | Minimum | Why |
|---|
| 588.0.0 | apigee apis import --from-template
is on the GA track at this version; older SDKs need |
google-cloud-aiplatform[agent_engines,adk]
| | Agent Runtime client used to bind the agent to the gateway |
| | only if you build the test agent with ADK |
On gcloud >= 588.0.0: every feature must keep its , and the template must
never declare or . All six features set
today and
the template declares neither, so the shipped bundle is correct — but both are
load-bearing now, and breaking either one still compiles, still deploys, and
reports nothing. Mechanism and symptoms:
, under
Authoring (AFT).
bash
gcloud version | head -1 # confirm >= 588.0.0
gcloud components update # if older
pip install "google-cloud-aiplatform[agent_engines,adk]>=2.1,<3" "google-adk>=2.9,<3"
needs PyYAML and names the interpreter if it is
missing.
, or run it with a Python 3 that has it; gcloud's
bundled interpreter does not, and the SDK's vendored
is Python 2.
bash
python3 scripts/compile_check.py assets/apigee-proxy-template.yaml
Execution model
Three layers, and the choice between them is not stylistic:
| Layer | Used for | Why |
|---|
| AFT YAML template | the proxy itself | version-controlled, reviewable, diffable |
| PSC, DNS, AuthzExtension, AuthzPolicy | native commands exist |
| REST () | env properties, envgroups, deployments, debug sessions | no equivalent today |
The customer owns
one input file
(
assets/apigee-agent-gateway-governance.example.yaml
). It is the
version-controlled artifact. Never hand-edit the generated proxy: regenerate it.
Run the Apigee-side sequence with the script, not by hand. Steps 1, 3 and 4
below are order-dependent and each fails in a way that reads as a different
problem.
does all three, preserves the properties
map, polls the async operation, and exits non-zero if the deployment does not
come up
:
bash
scripts/deploy_proxy.sh --org "$ORG" --env "$ENV" --proxy "$PROXY" \
--template assets/apigee-proxy-template.yaml [--service-account "$SA"] --yes \
[--index-endpoint-id ID --index-id ID --sc-project P --sc-region R --embeddings-model M]
Run it without
first: it prints the plan and changes nothing. Show that
plan, then re-run with
once the user agrees.
The vector-search flags are the one exception to "edit the default in every
feature that declares it". Those three ids are substituted at deploy time
and checked for leftovers; hand-editing them into
feature-semantic-cache.yaml
deploys fine, skips the check, and makes the file single-use.
scripts/ensure_gateway_connectivity.sh
is the same idea for the gateway leg
(step 0). It is the only part of this skill that modifies a resource the user
already owns, so it applies nothing without
.
The steps are written out below so you can diagnose a failure, not so you can
retype them.
Inputs you must resolve first
Environment prerequisites: — gcloud SDK (588+),
project, billing,
, region, credentials, IAM. Nothing
below creates any of it.
Prod: from the input file, then environment, then ask — never invent. Dev: apply
the
defaults, and ask only for what has none.
- ,
- , , , ,
- , ,
network.networkAttachment
- , (a list, one entry per wire shape),
gateway.manageNetworkConfig
drives TWO artifacts; write both from the same list. The
AuthzPolicy gets one suffix per entry, verbatim with
leading punctuation kept — it decides what the gateway routes to Apigee. The
template's
gets one
(request.uri Like "*<entry>*")
clause
per entry, OR-ed, leading
or
stripped — it decides what the proxy
acts on once it arrives. The shipped two-entry default gives:
((request.uri Like "*generateContent*") or (request.uri Like "*chat/completions*"))
The AuthzPolicy is the outer gate and wins: a suffix list narrower than
means the missing shape never reaches Apigee at all. Derive both
from the input file and never from the default when the file disagrees — the
file is the reviewable contract and a mismatch is invisible once deployed.
A one-entry list is legitimate and governs one shape. Dropping an entry
governs one fewer
silently — deploys clean, zero transactions for the shape
dropped — and narrows
, Model Armor and the cache together.
, Runtime.
is a KNOWN ISSUE whenever it is on — warn, then build it.
The key is the prompt embedding alone: no caller, no model, no shape. A hit
therefore crosses callers — similar prompts share an answer — and one model
behind one shape does not make it safe. With more shapes the first model to
answer answers for the other too: empty HTTP 200 across shapes, wrong-model
body within one. Warn whenever the cache is on, not only on multi-shape, then
build. Wording:
; mechanism:
.
Each entry is a SUFFIX match. You write these, not the user — derive them
from which endpoint their agent actually calls, and do not ask them to reason
about suffixes. Vertex's OpenAI-compatible endpoint and
need
different entries, and getting it wrong governs nothing silently; the forms and
the evidence are in
, Runtime.
There is no field. Metering does not depend on the wire shape —
the count is normalised in JS across both — so the field selected nothing and
was removed. Adding a shape is an
branch in
plus an
entry in
.
Three places carry the callout hostname and all three must match — the
Apigee envgroup, the AuthzExtension
/
, and the private DNS A
record. You set all three from one value: a build invariant to hold, not a
question for the user. It becomes a topic only when a callout never lands, and
then it is the first check. Build order matters alongside it — the
AuthzExtension is inert until the host resolves and the proxy is deployed, so
"the extension exists" is not evidence that leg works.
,
Runtime.
Workflow
0. Gateway: give it a route into the VPC
A gateway with no
has no route into the customer's VPC and
cannot resolve the callout hostname.
This skill builds it — network
attachment, egress and DNS peering — so its absence is a build step, not a
blocker, and should be reported that way. Do not open with an alarm about a
resource you are about to create.
One thing here does need the user, and it is the reason this is step 0 rather
than a silent prerequisite: attaching egress is one-way and affects traffic
they already run. Settle it in the plan, before anything executes — see
"Confirm everything up front" below. Capturing a known-good agent turn first is
worth doing when they are nervous about a production gateway, because it
separates "never worked" from "we broke it"; it is a precaution to offer, not a
step to impose.
bash
gcloud network-services agent-gateways describe "$GATEWAY" \
--location="$REGION" --project="$PROJECT" --format='value(networkConfig)'
gcloud network-security authz-policies list \
--location="$REGION" --project="$PROJECT"
A console-created gateway already has a CUSTOM AuthzPolicy — an IAP
one, which coexists with ours and is not needed for the egress
JWT. You never create the gateway, so assess it rather than assuming.
, Agent Gateway.
Empty
output means missing, and it is easy to miss: the gateway
still serves ordinary agent traffic perfectly, because that path goes to public
Google endpoints and never enters the VPC. Only the Apigee callout fails.
If it is missing, behaviour depends on
gateway.manageNetworkConfig
:
- — run the script below. It creates the network attachment, then
read-modify-writes the gateway to add the egress attachment and DNS peering.
Agreement for this is taken in the plan, not here; carries it.
- — the user has said they want to make this change themselves.
Print the script invocation and let them run it. Do not modify the gateway.
bash
scripts/ensure_gateway_connectivity.sh \
--gateway "$GATEWAY" --project "$PROJECT" --region "$REGION" \
--vpc "$VPC" --subnet "$SUBNET" --attachment "$ATTACHMENT" \
--domain "$DNS_DOMAIN" # add --yes only after the user agrees
Either way
has to exist before the Apigee work starts — not
because its absence is an error, but because every later step then succeeds and
governs nothing.
Use the script, not the raw commands:
is a whole-resource upsert and a
hand-built YAML silently drops the gateway's
and
. Why,
and the
and
handling:
.
1. Apigee: enable the environment
The env must carry
apigee-service-extension-enabled=true
. No
flag
exists; PATCH the properties map.
bash
curl -s -H "Authorization: Bearer $TOKEN" \
"https://apigee.googleapis.com/v1/organizations/$ORG/environments/$ENV"
The PATCH replaces the whole map. Read it first and re-send every property
you intend to keep, or you will silently drop them.
bash
curl -X PATCH -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
"https://apigee.googleapis.com/v1/organizations/$ORG/environments/$ENV?updateMask=properties" \
-d '{"properties":{"property":[{"name":"apigee-service-extension-enabled","value":"true"}]}}'
This does
not disable Apigee Analytics — the two coexist; if analytics looks
missing it is the separate per-environment add-on.
.
Use a dedicated environment, attached to
exactly one envgroup. An
environment in two makes the gateway 404 every request with the extension never
invoked. It also leaves the
base path free, which is what the proxy ships
with.
scripts/ensure_apigee_environment.sh
creates the environment, attaches it to
the regional runtime instance, and creates and attaches the envgroup — so step 2
is already done when it succeeds. It creates billable resources and applies
nothing without
; that agreement is taken in the plan, not here.
bash
scripts/ensure_apigee_environment.sh \
--org "$ORG" --env "$ENV" --region "$REGION" \
--envgroup "$ENVGROUP" --hostname "$HOSTNAME" # add --yes to apply
It is idempotent and resumable, and refuses with exit 2 when the environment is
already in another envgroup or the envgroup exists on a different hostname. It
also handles three things a runbook tends to get wrong — the omitted
on
create, the four-minute instance attachment that
locks the environment, and
polling the operation rather than a
that succeeds too early.
2. Apigee: envgroup on the callout hostname
Done already if step 1's script ran. Verify, or do it by hand for an envgroup
that already exists. The hostname must be exactly the callout host, and the
group must be attached to the env.
bash
curl -s -H "Authorization: Bearer $TOKEN" \
"https://apigee.googleapis.com/v1/organizations/$ORG/envgroups/$ENVGROUP/attachments"
How many envgroups is the environment in? More than one and the callout
never lands.
enforces this before touching anything and exits
2 naming them;
reports it. Read the failure literally if
it fires — the environment is wrong, not the script.
3. Build the proxy from the template
bash
gcloud apigee apis import organizations/$ORG/apis/$PROXY \
--from-template=assets/apigee-proxy-template.yaml
Each
entry resolves
relative to the template file's own
directory, never the working directory, so a bare filename must sit beside the
template (a subpath like
features/feature-identity.yaml
works too).
is flat by convention, not compiler demand — but moving a file without updating
fails the import with a bare
.
If you get
Permission denied on resource ... (or it may not exist)
on an org
you demonstrably have access to, it is the credential, not the access. With
context-aware access enabled the default gcloud credential is rejected. Pass the
ADC token instead:
bash
CLOUDSDK_AUTH_ACCESS_TOKEN=$(gcloud auth application-default print-access-token) \
gcloud apigee apis import ...
To change a parameter,
edit the in every FEATURE that declares it
— not in the template.
gcloud apigee apis import
passes an empty
caller-parameter map, so each feature resolves its own default and the
template's declaration of the same name is never read. Editing the template
changes nothing, silently, and the compile still passes. Measured: setting
to DRY_RUN in the template shipped ENFORCE on a
fail-closed policy.
is declared in four features and all four
must agree.
fails on any disagreement. There is no
flag.
Enable an optional recipe by adding its feature to the template's
list. See
.
in the input file is NOT generative. It documents
which buckets exist; it does not create them. A bucket is real only when it has
an
step in
feature-token-metering.yaml
and a matching
parameter. Three exist:
,
and
— the catch-all for any model whose name contains neither
nor
.
Adding a fourth entry to the list meters nothing, and
omitting
silently unmeters every non-Gemini model: the count is extracted and then
discarded, with no
reference anywhere in the transaction. Adding a
model is a feature-file edit, not a config change.
4. Deploy
Deploy to the service-extension env, then confirm the deployment reports
proxyDeploymentType: EXTENSIBLE
.
bash
curl -s -H "Authorization: Bearer $TOKEN" \
".../environments/$ENV/deployments"
means it will
not govern the agent's off-Apigee tool egress.
If any policy calls a Google API — Model Armor, embeddings, Vector Search —
deploy
with a service account or the deploy is rejected with
:
bash
curl -X POST -H "Authorization: Bearer $TOKEN" \
".../apis/$PROXY/revisions/$REV/deployments?override=true&serviceAccount=$DEPLOY_SA"
5. Networking
This is the
gateway → Apigee leg only. PSC endpoint to the Apigee instance
service attachment, plus a private DNS A record. See
references/networking-psc.md
. The Apigee instance
must
include your project or the connection never reaches
.
The agent → gateway leg is the prerequisite above, and is not built here.
6. Bind the gateway
AuthzExtension then AuthzPolicy — in that order, and only once the host resolves
and the proxy is deployed. The extension is inert until then.
Both document bodies ship as assets. Copy and substitute; do not invent them
—
assets/authz-extension.example.yaml
and
assets/authz-policy.example.yaml
.
They carry the fields that are load-bearing and have no obvious default:
wireFormat: EXT_PROC_GRPC
,
, the
/
pair, the
body flags, and the policy's
/
nesting.
bash
gcloud service-extensions authz-extensions import $AUTHZ_EXT \
--source=authz-extension.yaml --location=$REGION --project=$PROJECT
gcloud beta network-security authz-policies import $AUTHZ_POLICY \
--source=authz-policy.yaml --location=$REGION --project=$PROJECT
The policy's resource paths take the project NUMBER, not the ID — both
and
customProvider.authzExtension.resources
. Resolve it
yourself; narrating two numbers at the user does not help. The trap, for when a
callout never lands: the number in the gateway's
serviceExtensionsServiceAccount
is the Google-managed tenant project, not
theirs, and it is the only one on screen, so it is the natural thing to copy. A
wrong number imports cleanly and targets nothing. Copy the slots from
assets/authz-policy.example.yaml
.
7. Verify
You
cannot test with curl — the host is internal and the proxy expects the
gateway's IAP-egress JWT. Run
scripts/read_debug_session.sh --arm
, drive a
turn, read it back with
. Expect
one transaction per turn per
governed shape under a suffix list this narrow; zero alone does not mean the
callout is dead — the script prints the probe that settles it.
Hard rules
Gate every step — how urgently depends on your AuthzPolicy. An ungated
policy faults on any call that carries no prompt, and
that
breaks the agent's turn. Historically the most common failure here.
A narrow suffix list routes only the model call to Apigee, so the trap mostly
does not fire. Broaden the policy and it fires at once: the ext-proc then sees
every agent egress — session create/read,
, OpenTelemetry exports —
measured at 12 calls for one single-question turn, of which one was the model
call. Gate the steps regardless; a condition is cheap and it is what makes the
policy safe to widen later. Full measurement:
.
- model call:
(request.uri Like "*generateContent*")
Never fail-close on your own metering. Keep
on the
JavaScript and quota policies so a metering bug cannot break LLM traffic.
This skill meters by default and can enforce on request. Enforcement is
assets/feature-token-enforce.yaml
: request-flow
policies sharing
each
with the response-flow
half. Add it to the
template's feature list. Verified live on ext-proc traffic — the over-budget
caller gets a
429, model never called.
Never offer
as a substitute: in the response flow it cannot
gate (the answer is already on the wire) and it freezes the counter, losing the
metering too.
State two things whenever you offer it. It is
one request late — the turn
that crosses the limit completes, because a token cost is not known until the
model answers. And it buckets on the
requested model, which under an OpenAI
alias can differ from the resolved model the counting side bills. The two traps
that both present as "it never refuses":
.
Normalize the model to a bucket. With an OpenAI-compatible endpoint the
model is an alias in the request body and the
resolved name is in the response
body (
). Exact-matching a raw name fails on the response
path.
needs its third argument so it returns a scalar. Without
it the quota policy receives a node it cannot parse.
References
-
—
read first. How to answer "what can you do", dev
vs prod, the defaults, durations, cost disclosure, the interlocks, and
reading an assess report
-
scripts/assess_integration.sh
—
read-only. Setup Status Summary then a
per-check table;
carries both. Lead with the summary and read out
the
Enforcement row. Run it before planning: it decides CREATE vs REUSE.
is a build step, not a failure; only
exits non-zero
-
references/networking-psc.md
— both network legs, PSC, DNS, acceptance
-
— Model Armor and semantic cache
-
— debug sessions and the checklist script
-
— SDK, project, billing, APIs, region, auth, IAM
-
— 63 failure modes and their symptoms
-
assets/apigee-agent-gateway-governance.example.yaml
— the input contract
-
assets/apigee-proxy-template.yaml
— the AFT template. Features merge by
and all sit beside it:
(10),
(15),
(20),
feature-semantic-cache.yaml
(30),
feature-token-metering.yaml
(50).
-
assets/authz-extension.example.yaml
,
assets/authz-policy.example.yaml
—
the step-6 bodies. Copy and substitute; paths take the project
number
Anything read out of the request body is normalised once, at priority 15,
not inside whichever feature needs it first. is shared, so
widening it arms every feature gated on it simultaneously — including Model
Armor, which fails closed. Model Armor and the cache both read
; the quotas read
, set on the response
path by the metering feature's own JS for the same reason.
-
scripts/ensure_apigee_environment.sh
— create the environment, attach it
to the instance, create and attach the envgroup. Idempotent,
gated
-
scripts/ensure_gateway_connectivity.sh
— route the gateway into the VPC. Read-modify-write;
gated
-
scripts/ensure_vector_search.sh
— only when
is enabled:
build index and endpoint, deploy one on the other, read back the domain.
gated, ~34 min, bills continuously
-
scripts/teardown_vector_search.sh
— the only thing that STOPS that charge. Refuses until the revision drops
.
gated
-
— the Apigee-side sequence, idempotent,
-
scripts/read_debug_session.sh
—
a session (printing when it
actually closes) and read it back as the
checklist, instead of
parsing the transaction JSON by hand
-
— compile the template offline, no org needed