undefined

--------------------------------------------------------------------------------

kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix
kubectl describe pod "{pod_name}" -n "{namespace}"

resource.type="k8s_container"
resource.labels.namespace_name="{namespace}"
resource.labels.pod_name="{pod_name}"
severity>=WARNING

undefined

-   Empty workload pool → Workload Identity is **not enabled on the cluster**.
-   Node-pool mode is `GCE_METADATA` (or empty) → the node pool is **not** using
    the GKE metadata server; this is the usual cause of "runs as the node
    default service account". Remediation: enable
    `--workload-metadata=GKE_METADATA` on the node pool (propose to a human;
    recreates nodes). This is a **node-pool configuration** problem — not a KSA
    annotation or IAM binding problem — so do not change KSA annotations or IAM
    bindings to fix it. When the symptom is "runs as the node default service
    account", say so explicitly: the root cause is the node pool's
    `workloadMetadataConfig.mode`, and the KSA annotation and IAM bindings are
    **ruled out** as the cause. Propose the exact fix, e.g.:

```bash
gcloud container node-pools update "{node_pool}" --cluster "{cluster}" \
  --location "{location}" --workload-metadata=GKE_METADATA

principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}

undefined

> If the annotation is present but the binding is missing, the binding was
> likely removed — check the `setIamPolicy` audit logs around the failure time
> to find the responsible principal.

--------------------------------------------------------------------------------

undefined

If no binding grants the required role on the target resource, **that missing
role is the root cause** (and Workload Identity itself is **not**
misconfigured). Present the fix as a **proposed change for a human to apply** —
the exact `add-iam-policy-binding` with the real principal and role — never
applying it automatically. For example, for direct binding on a project-level
resource:

```bash
gcloud projects add-iam-policy-binding "{TARGET_PROJECT_ID}" \
  --member="principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}" \
  --role="{REQUIRED_ROLE}"   # e.g. roles/storage.objectViewer

undefined

**(b) Inspect the `gke-metadata-server` logs** (historical, via Cloud Logging;
open as a Logs Explorer deep link, see Step 1):

**(c) Connectivity test** from the affected Pod — it must reach the metadata
server and (for some client libraries) resolve its DNS name:

```bash

Possible causes:

-   **Startup race**: the metadata server needs a few seconds after a Pod
    starts. Applications that authenticate immediately may fail; the fix is
    application-side retries or an `initContainer` that waits for the metadata
    server, not a cluster change.
-   **DNS**: some client libraries resolve `metadata.google.internal`; broken
    in-cluster DNS surfaces as `cannot find the metadata server`. As a
    workaround, set `GCE_METADATA_HOST=169.254.169.254` to skip DNS resolution.
-   **Network policy / egress**: a cluster network policy must allow egress to
    `169.254.169.254/32` on port `80` (GKE Dataplane V2), or
    `169.254.169.252/32` on port `988`. A default-deny egress rule that blocks
    HTTPS to the public Security Token Service (`sts.googleapis.com:443`)
    produces `504 Gateway Timeout` or `context deadline exceeded`.
-   **`gke-metadata-server` crashing / high restarts**: see (a) — reduce the
    number of Kubernetes service accounts (<3,000) to restore functionality.

--------------------------------------------------------------------------------

ERROR: Invalid form of account ID SERVICEACCOUNT_NAME.svc.id.goog.
Should be [Gaia ID | Email | Unique ID | ] of the account

kubectl annotate serviceaccount "{ksa_name}" --namespace "{namespace}" \
  iam.gke.io/return-principal-id-as-email="true"