Deco Site Scaling Tuning

When to Use This Skill

• A site is overscaled (too many pods for its traffic)
• A site oscillates between scaling up and down (panic mode loop)
• Need to switch scaling metric (concurrency vs CPU vs RPS)
• Need to find the right target value for a site
• After deploying scaling changes, to verify they're working

Prerequisites

• kubectl

  access to the target cluster
• Prometheus accessible via port-forward (from

  kube-prometheus-stack

  in monitoring namespace)
• Python 3 for analysis scripts
• At least 6 hours of metric history for meaningful analysis
• For direct latency data: queue-proxy PodMonitor must be applied (see Step 0)

Quick Start

0. ENABLE METRICS   → Apply queue-proxy PodMonitor if not already done
1. PORT-FORWARD     → kubectl port-forward prometheus-pod 19090:9090
2. COLLECT DATA     → Run analysis scripts against Prometheus
3. ANALYZE          → Find CPU threshold where latency degrades
4. RECOMMEND        → Choose scaling metric and target
5. APPLY            → Use deco-site-deployment skill to apply changes
6. VERIFY           → Monitor for 1-2 hours after change

Files in This Skill

SKILL.md

analysis-scripts.md

Step 0: Enable Queue-Proxy Metrics (one-time)

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: knative-queue-proxy
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    any: true
  selector:
    matchExpressions:
      - key: serving.knative.dev/revision
        operator: Exists
  podMetricsEndpoints:
    - port: http-usermetric
      path: /metrics
      interval: 15s

kubectl apply -f queue-proxy-podmonitor.yaml
# Wait 2-3 hours for data to accumulate before running latency analysis

• revision_app_request_latencies_bucket

  — request latency histogram (p50/p95/p99)

• revision_app_request_latencies_sum

  /

  _count

  — for avg latency

• revision_app_request_count

  — request rate by response code

Step 1: Establish Prometheus Connection

PROM_POD=$(kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n monitoring $PROM_POD 19090:9090 &
# Verify
curl -s "http://127.0.0.1:19090/api/v1/query?query=up" | jq '.status'

Step 2: Collect Current State

2a. Read current autoscaler config

SITENAME="<sitename>"
NS="sites-${SITENAME}"

# Current revision annotations
kubectl get rev -n $NS -o json | \
  jq '.items[] | select(.status.conditions[]?.status == "True" and .status.conditions[]?.type == "Active") |
  {name: .metadata.name, annotations: .metadata.annotations | with_entries(select(.key | startswith("autoscaling")))}'

# Global autoscaler defaults
kubectl get cm config-autoscaler -n knative-serving -o json | jq '.data | del(._example)'

2b. Current pod count and resources

kubectl get pods -n $NS --no-headers | wc -l
kubectl top pods -n $NS --no-headers | head -20

Step 3: Run Analysis

analysis-scripts.md

Methodology: Finding the Optimal CPU Target

1. Collect CPU per pod, concurrency per pod, pod count, and (if available) request latency over 6-12 hours
2. Bucket data by CPU range (0-200m, 200-300m, ..., 700m+)
3. For each bucket, compute avg/p95 concurrency per pod
4. Compute the "latency inflation factor" — how much concurrency increases beyond what the pod count reduction explains:

   excess = (avg_conc_above_threshold / avg_conc_below_threshold) / (avg_pods_below / avg_pods_above)

   • excess = 1.0 → concurrency increase fully explained by fewer pods (no latency degradation)
   • excess > 1.0 → latency is inflating concurrency (pods are slowing down)
   • The CPU level where excess crosses ~1.5x is your inflection point
5. If queue-proxy latency is available, directly plot avg latency vs CPU — the hockey stick inflection is your target

What to Look For

CPU vs Concurrency/pod:

  Low CPU   (0-200m)   →  Low conc/pod   →  Pods are idle (overprovisioned)
  Medium CPU (200-400m) →  Moderate conc  →  Healthy range
  ★ INFLECTION ★       →  Conc jumps      →  Latency starting to degrade
  High CPU  (500m+)    →  High conc/pod   →  Pods overloaded, latency bad

Decision Matrix

target: 400

Common Patterns

• Baseline CPU: 200-300m (Deno runtime + V8 JIT)
• Inflection: 400-500m
• Recommendation: CPU scaling with target = inflection (e.g., 400 millicores)

• CPU stays low even under high concurrency
• Inflection not visible in CPU
• Recommendation: Keep concurrency scaling, tune the target

• Symptoms: pods cycle between min and max
• Cause: concurrency scaling + low target +

  scale-down-delay

  ratchet
• Fix: Switch to CPU scaling (breaks the latency→concurrency feedback loop)

Step 4: Apply Changes

deco-site-deployment

1. Update the

   state

   secret with new scaling config
2. Redeploy on both clouds

NEW_STATE=$(echo "$STATE" | jq '
  .scaling.metric = {
    "type": "cpu",
    "target": 400
  }
')

Step 5: Verify After Change

# Watch pod count stabilize
watch -n 10 "kubectl get pods -n sites-<sitename> --no-headers | wc -l"

# Check if panic mode triggers (should be N/A for HPA/CPU)
# HPA doesn't have panic mode — this is one of the advantages

# Verify HPA is active
kubectl get hpa -n sites-<sitename>

# Check HPA status
kubectl describe hpa -n sites-<sitename>

Success Criteria

• Pod count stabilizes (no more oscillation)
• Avg CPU per pod stays below your target during normal traffic
• CPU crosses target only during genuine traffic spikes (and scales up proportionally)
• No panic mode events (HPA doesn't have panic mode)
• Latency stays acceptable (check with queue-proxy metrics if available)

Rollback

NEW_STATE=$(echo "$STATE" | jq '
  .scaling.metric = {
    "type": "concurrency",
    "target": 15,
    "targetUtilizationPercentage": 70
  }
')

Related Skills

• deco-site-deployment

  — Apply scaling changes and redeploy

• deco-site-memory-debugging

  — Debug memory issues on running pods

• deco-incident-debugging

  — Incident response and triage