python3 scripts/huawei-cloud.py

Execution method: Cloud service queries are executed through the local

hcloud

CLI. AOM Prometheus

query_range

calls are the only exception and use signed HTTPS requests because the required Prometheus range-query path is not compatible with hcloud. Do not call Huawei Cloud SDKs, curl IAM flows, openstack, or hand-written cloud APIs outside the bundled dispatcher.

• Pod CPU/memory TopN ranking and single Pod time-series metrics
• Node CPU/memory/disk TopN ranking and single Node time-series metrics
• Node GPU and xGPU metrics, including GPU utilization, memory, temperature, power, schedule policy, xGPU allocation, usage, and health
• CoreDNS QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replica count, and per-Pod CPU/memory metrics
• nginx-ingress QPS, 4xx/5xx rate, success rate, P95 latency, active connections, per-Pod CPU/memory, and Ingress TLS certificate expiration status
• Autoscaler unschedulable Pods, node state count, scale-up/down events, errors, node groups, HPA current/desired replicas, and per-Pod CPU/memory metrics
• Kubernetes control-plane metrics for apiserver, etcd, controller-manager, and scheduler
• ECS instance CPU/memory/disk/network metrics
• ELB connection, bandwidth, QPS metrics
• EIP bandwidth, traffic, packet loss metrics
• NAT Gateway SNAT connection metrics
• Full-cluster monitoring aggregation with anomaly detection (80% threshold)
• Threshold-based status classification (critical/warning/normal/unknown)

• Python 3.8+ for the dispatcher and result processing
• hcloud (KooCLI) 7.2.2+ for CCE/ECS/ELB/VPC/EIP/NAT/CES/IAM cloud service queries

• kubectl

  only for Kubernetes resource reads that cannot be derived from AOM/hcloud, such as Pod

  label_selector

  filtering, Ingress TLS certificate checks, and LoadBalancer Service discovery for ELB/EIP association; clusters without external EIP require the

  kubectl cce

  plugin from

  kubectl-cce-plugin/README.md

• Prometheus-related monitoring data is queried from AOM Prometheus with signed HTTPS requests; the cluster must have the Prometheus add-on integrated with AOM, otherwise these tools may return empty metric series
• Controller-manager, scheduler, and etcd metrics require the

  kube-controller-manager

  ,

  kube-scheduler

  , and

  etcd-server

  ServiceMonitors to be enabled separately in AOM; otherwise these tools may return empty metric series
• Autoscaler, ingress-controller, and NVIDIA GPU metrics require the corresponding

  autoscaler

  ,

  ingress-controller

  , and

  nvidia-gpu-device-plugin

  PodMonitors to be enabled separately in AOM; ingress request metrics also require

  nginx_ingress_controller_requests

  to be explicitly allowed in the ingress-controller PodMonitor
• Run environment check before first use (see Verification section)
• Install and use

  kubectl-cce

  according to references/kubectl-cce.md

• Valid Huawei Cloud credentials via hcloud profile or AK/SK mode
• Security Rules:
  • 🚫 Never expose AK/SK values in code, conversation, or commands
  • 🚫 Never use

    echo $HUAWEI_AK

    or

    echo $HUAWEI_SK

    to check credentials
  • ✅ Credential priority for hcloud calls is: explicit tool parameters > local hcloud profile > environment variables
  • ✅ AOM Prometheus signed HTTP and Kubernetes certificate setup cannot use encrypted hcloud profile material, so they use explicit tool parameters first and environment variables as the signing fallback
  • ✅ Prefer IAM users over root account for cloud operations
  • ✅ Enable MFA for sensitive operations

hcloud configure list

export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4

cce:cluster:get

aom:instance:list

aom:metricsData:get

ces:metricsData:get

ecs:cloudServers:list

elb:loadbalancers:list

vpc:eips:list

nat:natGateways:list

1. When any command fails due to IAM permission errors, display the required permission list
2. Guide the user to create a custom policy in the IAM console and grant authorization
3. Pause execution and wait for user confirmation that permissions have been granted

• success

  — boolean, true if query completed

• region

  — Huawei Cloud region

• cluster_id

  /

  cluster_name

  — CCE cluster identity

• aom_instance_id

  — AOM Prometheus instance used for metric queries

• metrics

  — Dict with cpu/memory/disk data per resource, including status classification

• certificate_check

  — nginx-ingress Ingress TLS certificate expiration summary when certificate checking is enabled

• time_series

  — Historical data points with

  timestamp

  ,

  time

  ,

  average

  ,

  min

  ,

  max

• status

  — Threshold classification:

  critical

  (>80% CPU, >85% memory/disk),

  warning

  (>50% CPU/memory, >70% disk),

  normal

  (below warning),

  unknown

  (no data)

1. Resolve region, cluster ID, and credentials using the documented priority.
2. Discover the AOM Prometheus instance from the CCE cluster add-on binding.
3. Start with Pod/Node TopN or aggregation, then drill into a Pod, Node, component, or cloud resource.
4. Keep PromQL scoped by

   cluster="<cluster_id>"

   ; add namespace, pod, or resource filters only to reduce noise.
5. Use status classification as an investigation lead, then correlate anomalies with events or alarm history.

1. Run

   python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN region=cn-north-4 cluster_id=<cluster-id> namespace=default top_n=5

   to verify Pod metric queries
2. Run

   python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN region=cn-north-4 cluster_id=<cluster-id> top_n=5

   to verify Node metric queries
3. Run

   python3 scripts/huawei-cloud.py huawei_get_ecs_metrics region=cn-north-4 instance_id=<instance-id>

   to verify CES metric connectivity

1. Start with Pod/Node TopN before drilling into individual resources.
2. Keep

   hours

   small (1-4) for recent analysis; cap historical reviews at 24 hours.
3. Provide

   namespace

   to reduce Pod noise while preserving the cluster filter.
4. Focus on

   critical

   and

   warning

   resources first.
5. Use

   huawei_cce_cluster_monitoring_aggregation

   for full-cluster health checks.
6. Correlate metric anomalies with

   huawei-cloud-cce-kubernetes-event-analyzer

   .
7. Do not expose production Pod names, node IPs, or cluster IDs in public summaries.

• This skill is strictly read-only and never modifies resources or configurations.
• Thresholds are predefined baselines; tune them against workload SLOs before making operational decisions.
• AK/SK must never be hardcoded; use hcloud profile for normal hcloud calls or environment fallback for signed AOM/Kubernetes calls.

• scripts/huawei-cloud.py

  is the only user-facing execution method
• AOM Prometheus instance is auto-discovered; no need to manually specify

  aom_instance_id

• Cloud resource metrics (ECS/ELB/EIP/NAT) use CES (Cloud Eye Service), not AOM
• Do not make automatic scaling or remediation decisions based solely on metric analysis.

• AOM Prometheus data requires the cluster Prometheus add-on to be integrated with AOM.
• Control-plane ServiceMonitors and component PodMonitors must be enabled before related metrics appear.
• Query results reflect collected monitoring data only; missing series are not proof that the workload is healthy.
• This skill does not remediate, scale, restart, create, update, or delete cloud or Kubernetes resources.