GKE Workload Identity Federation Troubleshooting Skill

Use this skill to systematically diagnose why a Pod using Workload Identity Federation for GKE cannot authenticate to Google Cloud APIs. Typical symptoms:

HTTP/403 ... Permission 'iam.serviceAccounts.getAccessToken' denied on resource

google.auth.exceptions ... could not find default credentials

ComputeEngineCredentials cannot find the metadata server

API calls that unexpectedly use the node's default Compute Engine service account instead of the workload's identity.

Read-only boundary

This skill is diagnostic and non-interactive. It only reads cluster, IAM, and logging state and proposes fixes as commands or GitOps manifest changes for a human to apply. It must never create or modify IAM bindings, KSA annotations, node pools, or clusters automatically. When evidence is missing or the fix requires a privileged change, summarize findings and hand off to a human (see Step 7).

Output discipline (apply to every conclusion)

When you report a diagnosis, always:

Name the single most-likely root cause (not an open-ended list of possibilities).
Explicitly rule out the other plausible causes, citing the evidence that excludes them. In particular, when the cause is node-pool configuration, state plainly that it is not a KSA annotation or IAM binding problem; when the cause is a missing role on a target resource, state that Workload Identity itself is not misconfigured.
Give the exact remediation as a proposed change for a human — a concrete
```
gcloud
```
command (or GitOps manifest edit) with the real identifiers filled in — and never apply it automatically.

Diagnostic Workflow

Step 0: Context discovery & time window

Collect (from the user or the failing resource):

PROJECT_ID

PROJECT_NUMBER

CLUSTER

, cluster

LOCATION

NAMESPACE

, the KSA the Pod runs as, the node and node pool the Pod is scheduled on (used in Step 2), the target resource / API being called, and the exact error string. Define a time window around the first observed failure for log queries.

bash

# Resolve the project number (used in the direct-binding principal identifier).
gcloud projects describe "{PROJECT_ID}" --format="value(projectNumber)"

# Confirm which KSA the workload runs as.
kubectl get pod "{pod_name}" -n "{namespace}" \
  -o jsonpath='{.spec.serviceAccountName}'

# Identify the node the Pod runs on, then the node pool that node belongs to
# (Step 2 checks the node pool's Workload Identity mode).
NODE=$(kubectl get pod "{pod_name}" -n "{namespace}" -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" \
  -o jsonpath='{.metadata.labels.cloud\.google\.com/gke-nodepool}'

Step 1: Capture the exact error signature

Read the workload's own logs and the

gke-metadata-server

logs to classify the failure.

bash

kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix
kubectl describe pod "{pod_name}" -n "{namespace}"

Equivalent via Cloud Logging (preferred for historical events). Open it as a Logs Explorer deep link — URL-encode the query and append the project and time window:

https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}

(encode

%2F

, or use

;duration=PT1H

for a rolling hour):

resource.type="k8s_container"
resource.labels.namespace_name="{namespace}"
resource.labels.pod_name="{pod_name}"
severity>=WARNING

Classify the signature:

```
iam.serviceAccounts.getAccessToken
```
denied /
```
HTTP/403
```
→ the workload is using the GSA impersonation path. This is expected for the legacy setup, but if you intend to use direct binding, it means the KSA still carries a leftover
```
iam.gke.io/gcp-service-account
```
annotation (or the client SDK is configured to impersonate) and is unintentionally impersonating a GSA; go to Step 3.
```
403 PERMISSION_DENIED
```
on the target API/resource (no
```
getAccessToken
```
in the error) → the resolved identity lacks the required IAM role on that resource; go to Step 4.
```
could not find default credentials
```
/
```
cannot find the metadata server
```
→ metadata-server connectivity or a startup race; go to Step 5.
Calls succeed but as the node default service account → Workload Identity is not in effect for this node pool; go to Step 2.

Step 2: Verify Workload Identity is enabled (cluster + node pool)

Both the cluster and the node pool the Pod runs on must have Workload Identity enabled. A node pool with

GCE_METADATA

(instead of

GKE_METADATA

) causes Pods to fall back to the node's default Compute Engine service account.

bash

# Cluster must have a workload identity pool (PROJECT_ID.svc.id.goog).
gcloud container clusters describe "{cluster}" --location "{location}" \
  --format="value(workloadIdentityConfig.workloadPool)"

# Node pool must have workloadMetadataConfig.mode = GKE_METADATA.
gcloud container node-pools describe "{node_pool}" --cluster "{cluster}" \
  --location "{location}" \
  --format="value(config.workloadMetadataConfig.mode)"

Empty workload pool → Workload Identity is not enabled on the cluster.
Node-pool mode is
```
GCE_METADATA
```
(or empty) → the node pool is not using the GKE metadata server; this is the usual cause of "runs as the node default service account". Remediation: enable
```
--workload-metadata=GKE_METADATA
```
on the node pool (propose to a human; recreates nodes). This is a node-pool configuration problem — not a KSA annotation or IAM binding problem — so do not change KSA annotations or IAM bindings to fix it. When the symptom is "runs as the node default service account", say so explicitly: the root cause is the node pool's
```
workloadMetadataConfig.mode
```
, and the KSA annotation and IAM bindings are ruled out as the cause. Propose the exact fix, e.g.:

bash

gcloud container node-pools update "{node_pool}" --cluster "{cluster}" \
  --location "{location}" --workload-metadata=GKE_METADATA

Step 3: Verify the KSA → identity binding

There are two supported models. Prefer direct binding (current default); treat GSA impersonation as the legacy path.

How the two models fail differently: with direct binding the KSA principal accesses resources directly, so failures show up as a plain
403 PERMISSION_DENIED
on the target API (fix in Step 4). A
403 iam.serviceAccounts.getAccessToken
instead means an impersonation attempt — intended under the legacy path, or unintended if a leftover
iam.gke.io/gcp-service-account
annotation remains on a KSA that was meant to use direct binding.

(a) Direct KSA binding (no GSA impersonation). The KSA principal is granted roles directly. Construct the principal identifier and search for its bindings:

principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}

(b) Legacy: KSA + GSA impersonation. The KSA must be annotated to point at a GSA, and the KSA must hold

roles/iam.workloadIdentityUser

on that GSA.

bash

# The KSA annotation must reference the intended GSA.
kubectl get serviceaccount "{ksa_name}" -n "{namespace}" \
  -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}'

# The GSA's IAM policy must bind the KSA member to workloadIdentityUser.
gcloud iam service-accounts get-iam-policy \
  "{gsa_name}@{project_id}.iam.gserviceaccount.com" \
  --format=json
# Expect a binding: role roles/iam.workloadIdentityUser,
# member serviceAccount:{PROJECT_ID}.svc.id.goog[{NAMESPACE}/{KSA_NAME}]

If the annotation is present but the binding is missing, the binding was likely removed — check the
setIamPolicy
audit logs around the failure time to find the responsible principal.

Step 4: Verify IAM permissions on the target resource

Even with a correct binding, the identity (the KSA principal for direct binding, or the GSA for legacy) must hold the role required by the API call (for example

roles/storage.objectViewer

). The standard IAM Policy Troubleshooter has limited support for Workload Identity principals; use Cloud Asset Inventory to search all IAM policies for the principal instead.

bash

# Direct binding: search for the KSA principal's bindings across the project.
gcloud asset search-all-iam-policies \
  --scope="projects/{PROJECT_ID}" \
  --query='policy:"{PROJECT_ID}.svc.id.goog"'

# Legacy: search for the GSA's bindings on the target resource's project.
gcloud asset search-all-iam-policies \
  --scope="projects/{TARGET_PROJECT_ID}" \
  --query='policy:"{gsa_name}@{project_id}.iam.gserviceaccount.com"'

If no binding grants the required role on the target resource, that missing role is the root cause (and Workload Identity itself is not misconfigured). Present the fix as a proposed change for a human to apply — the exact

add-iam-policy-binding

with the real principal and role — never applying it automatically. For example, for direct binding on a project-level resource:

bash

gcloud projects add-iam-policy-binding "{TARGET_PROJECT_ID}" \
  --member="principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}" \
  --role="{REQUIRED_ROLE}"   # e.g. roles/storage.objectViewer

Step 5: GKE metadata server & connectivity

Applies when the signature is

could not find the metadata server

cannot find the metadata server

, or a connection/timeout error. The

gke-metadata-server

DaemonSet (in

kube-system

) brokers the token exchange on each node; requests to

169.254.169.254

are redirected to it, so a Pod fails closed if it cannot reach a healthy metadata-server Pod on its node.

(a) Check gke-metadata-server Pod health on the workload's node.

bash

# The DaemonSet Pods must be healthy on the workload's node.
kubectl get pods -n kube-system -l k8s-app=gke-metadata-server -o wide

# A gke-metadata-server Pod can be OOM-evicted when the cluster has many
# (>3,000) Kubernetes service accounts. Look for CrashLoopBackOff, then confirm
# the eviction was OOMKilled.
kubectl get pods -n kube-system | grep CrashLoopBackOff
kubectl describe pod {gke_metadata_server_pod} --namespace=kube-system | grep OOMKilled

(b) Inspect the
gke-metadata-server
logs (historical, via Cloud Logging; open as a Logs Explorer deep link, see Step 1):

resource.type="k8s_container"
resource.labels.namespace_name="kube-system"
labels."k8s-pod/k8s-app"="gke-metadata-server"
severity>=WARNING

(c) Connectivity test from the affected Pod — it must reach the metadata server and (for some client libraries) resolve its DNS name:

bash

# Token endpoint via the hardcoded metadata IP (a healthy path returns a token).
kubectl exec {pod_name} -n {namespace} -- \
  curl -sS -H 'Metadata-Flavor: Google' \
  'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token'

Possible causes:

Startup race: the metadata server needs a few seconds after a Pod starts. Applications that authenticate immediately may fail; the fix is application-side retries or an
```
initContainer
```
that waits for the metadata server, not a cluster change.
DNS: some client libraries resolve
```
metadata.google.internal
```
; broken in-cluster DNS surfaces as
```
cannot find the metadata server
```
. As a workaround, set
```
GCE_METADATA_HOST=169.254.169.254
```
to skip DNS resolution.
Network policy / egress: a cluster network policy must allow egress to
```
169.254.169.254/32
```
on port
```
80
```
(GKE Dataplane V2), or
```
169.254.169.252/32
```
on port
```
988
```
. A default-deny egress rule that blocks HTTPS to the public Security Token Service (
```
sts.googleapis.com:443
```
) produces
```
504 Gateway Timeout
```
or
```
context deadline exceeded
```
.
gke-metadata-server
crashing / high restarts: see (a) — reduce the number of Kubernetes service accounts (<3,000) to restore functionality.

Step 6: "Invalid form of account ID" edge case (legacy GSA impersonation)

Applies only to the legacy GSA impersonation setup (a KSA annotated with
iam.gke.io/gcp-service-account
), not to direct principal binding.

Signature: an operation that requires an IAM service account email (for example, manually creating a Cloud Storage signed URL) fails with:

ERROR: Invalid form of account ID SERVICEACCOUNT_NAME.svc.id.goog.
Should be [Gaia ID | Email | Unique ID | ] of the account

Cause: for a KSA linked to a GSA via annotation, the GKE metadata server by default returns

SERVICEACCOUNT_NAME.svc.id.goog

as the identifier, which is not a valid IAM service account email. This is not a missing IAM binding and does not require reverting the setup.

Fix (propose to a human): add the

iam.gke.io/return-principal-id-as-email="true"

annotation to the Pod's KSA so the metadata server returns the identity in the expected form:

bash

kubectl annotate serviceaccount "{ksa_name}" --namespace "{namespace}" \
  iam.gke.io/return-principal-id-as-email="true"

Step 7: Remediation boundary & escalation

Present the root cause + evidence (the exact error, the missing binding / annotation / node-pool mode, or the metadata-server state). Include a Cloud Logging deep link (see Step 1) to the supporting entries.
Propose the fix as a command or GitOps manifest change for a human to apply — never modify IAM, KSAs, or node pools automatically.

Escalate (instead of proposing more self-service diagnostics) when either:

the relevant logs are unavailable (excluded by a filter, or past the log bucket's retention); or
the identity/binding is correct, the target-resource IAM is correct, and the metadata server is healthy, yet the failure persists (undetermined root cause).

In those cases: state the limitation plainly, summarize the findings gathered (cluster/node-pool config, bindings, annotations, metadata-server state, and any

setIamPolicy

audit logs), and route to GKE support / engineering escalation. Do not fabricate a diagnosis when evidence is missing.

References

This skill is derived from public Google Cloud documentation:

Troubleshoot GKE authentication issues — the Workload Identity Federation section: WI-enabled check, KSA annotation,
```
roles/iam.workloadIdentityUser
```
,
```
iam.serviceAccounts.getAccessToken
```
403,
```
iam.gke.io/return-principal-id-as-email
```
, metadata-server DNS and the
```
gke-metadata-server
```
Pod is crashing
guidance.
About Workload Identity Federation for GKE — identities, the GKE metadata server, and why Pods don't inherit the node identity.
Configure applications to use Workload Identity Federation for GKE — node-pool
```
--workload-metadata=GKE_METADATA
```
, the direct-binding
```
principal://...
```
identifier, and the legacy
```
serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]
```
member format.
Search IAM policies (Cloud Asset Inventory) —
```
gcloud asset search-all-iam-policies
```
for auditing a principal's bindings.
View GKE logs — Cloud Logging queries for workload and system logs.

gke-workload-identity

NPX Install

Tags

SKILL.md Content

GKE Workload Identity Federation Troubleshooting Skill

Read-only boundary

Output discipline (apply to every conclusion)

Diagnostic Workflow

Step 0: Context discovery & time window

Step 1: Capture the exact error signature

Step 2: Verify Workload Identity is enabled (cluster + node pool)

Step 3: Verify the KSA → identity binding

Step 4: Verify IAM permissions on the target resource

Step 5: GKE metadata server & connectivity

Step 6: "Invalid form of account ID" edge case (legacy GSA impersonation)

Step 7: Remediation boundary & escalation

References