AI Agent Skill: Operational knowledge for managing Proxmox Virtual Environment infrastructure

• VM and LXC lifecycle management - From creation to decommissioning
• Storage operations - Configuration, content management, backup strategies
• High Availability - HA groups, resource management, failover
• Cluster operations - Multi-node management, migration, replication
• Certificate management - Installation, renewal, ACME integration
• ACME configuration - Provider setup, certificate ordering, automation
• Notifications - Target configuration, delivery verification, alerting
• Troubleshooting - Common issues, API quirks, resolution patterns
• Security - Permission models, API token best practices
• Performance - Monitoring, resource optimization

• Hosts VMs and LXC containers
• Provides local storage
• Participates in cluster quorum

• Types: Directory, LVM, ZFS, Ceph, NFS, iSCSI
• Content types: Images, ISOs, backups, templates
• Can be node-local or cluster-shared

• Linux bridges for VM/LXC connectivity
• VLANs for network segmentation
• SDN (Software Defined Networking) for advanced scenarios

• Shared configuration via pmxcfs
• HA for automatic failover
• Live migration between nodes

1. Get next available VMID
2. Create VM with basic config (CPU, memory, OS type)
3. Add disk(s) from storage
4. Configure network interface(s)
5. Set boot order
6. Start VM

• Choose appropriate storage for disk (performance vs capacity)
• Use virtio drivers for best performance (requires guest support)
• Configure QEMU guest agent for better management

1. Review current config
2. Resize resources (CPU, memory, disk) as needed
3. Add/remove network interfaces
4. Configure firewall rules
5. Set up snapshots for rollback capability

• Snapshot before major changes
• Use cloud-init for automated provisioning
• Enable QEMU guest agent for graceful operations

1. Check VM status (running, stopped, paused)
2. Monitor resource usage (CPU, memory, disk I/O)
3. Review task history for recent operations
4. Check logs for errors or warnings

• CPU usage and steal time
• Memory pressure and swap usage
• Disk I/O wait times
• Network throughput

1. Create snapshot for quick rollback
2. Schedule backup job (vzdump)
3. Verify backup completed successfully
4. Test restore periodically
5. Prune old backups to manage space

• Snapshot mode: Fast, requires storage support
• Suspend mode: Pauses VM during backup
• Stop mode: Stops VM for consistent backup

• Lighter weight (shared kernel)
• Faster startup times
• Lower overhead
• Less isolation than VMs

1. Create from template
2. Configure resources (CPU, memory, swap)
3. Add mount points for storage
4. Configure network
5. Start container
6. Access via console or SSH

• Use

  mp0

  ,

  mp1

  for mount points (not disk0, disk1)
• No BIOS/UEFI configuration
• Direct kernel access (privileged) or restricted (unprivileged)
• Faster snapshot/restore operations

1. Create backup jobs (manual or scheduled)
2. Configure retention policy
3. Prune old backups automatically
4. Restore from backup
5. Verify backup integrity

• Use compression to save space
• Store backups on separate storage
• Test restore procedures regularly
• Document backup schedules
• Monitor backup job success/failure

1. Create HA group (define node priorities)
2. Add VM/LXC to HA management
3. Configure HA settings (max relocate, max restart)
4. Monitor HA status
5. Test failover scenarios

• started: Resource running on assigned node
• stopped: Resource intentionally stopped
• fence: Node fenced, resource will be restarted elsewhere
• error: HA manager encountered an error

• Critical services requiring high uptime
• Automatic failover needed
• Cluster has 3+ nodes (for quorum)

1. Verify target node has resources
2. Check shared storage access
3. Initiate migration
4. Monitor migration progress
5. Verify VM running on new node

• Shared storage for VM disks
• Network connectivity between nodes
• Compatible CPU types (or CPU flags masked)

1. Stop VM/LXC
2. Migrate to target node
3. Start on new node

• No shared storage available
• Maintenance on source node
• CPU incompatibility

• Insufficient resources on node
• Storage unavailable
• Lock file present
• Configuration error

1. Check node resources (memory, CPU)
2. Verify storage is mounted and accessible
3. Remove lock file if stale
4. Review VM config for errors
5. Check logs: /var/log/pve/tasks/

• Network connectivity issues
• Storage not shared
• CPU incompatibility
• Insufficient resources on target

1. Verify network between nodes
2. Check storage is accessible from both nodes
3. Review CPU flags compatibility
4. Ensure target node has capacity
5. Try offline migration if live fails

• Insufficient storage space
• VM locked by another operation
• Snapshot creation failed
• Network timeout (for remote storage)

1. Check storage space availability
2. Verify no other operations running on VM
3. Try manual backup to isolate issue
4. Review backup job logs
5. Prune old backups to free space

• Cluster quorum lost
• HA service not running
• Fencing not configured
• All nodes in HA group unavailable

1. Check cluster quorum status
2. Verify HA service running on all nodes
3. Review HA group configuration
4. Check fencing configuration
5. Manually start VM if needed

• Storage backend overloaded
• Network bottleneck (for remote storage)
• Disk cache settings suboptimal
• Too many VMs on same storage

1. Monitor storage backend performance
2. Check network throughput to storage
3. Adjust VM disk cache settings
4. Distribute VMs across multiple storage
5. Consider faster storage tier

1. Create dedicated user for automation
2. Assign minimal required permissions
3. Generate API token (not password)
4. Store token securely (environment variables, secrets manager)
5. Rotate tokens periodically

• Use roles to group permissions
• Assign roles to users/tokens
• Follow principle of least privilege
• Audit permission usage regularly

• Use realms (PAM, LDAP, AD) for authentication
• Create groups for role-based access
• Assign users to groups
• Review access periodically

• Restrict API access by IP (firewall rules)
• Use SSL/TLS for API connections
• Enable two-factor authentication for users
• Monitor authentication logs

• Don't overcommit CPU cores excessively
• Use CPU limits for non-critical VMs
• Pin CPUs for latency-sensitive workloads
• Monitor CPU steal time

• Enable ballooning for dynamic allocation
• Set appropriate memory limits
• Monitor swap usage (should be minimal)
• Use hugepages for large memory VMs

• Use virtio-scsi for best performance
• Enable discard/TRIM for SSDs
• Configure appropriate I/O scheduler
• Monitor disk latency and throughput

• Node CPU, memory, disk usage
• VM resource consumption
• Storage performance (IOPS, latency)
• Network throughput
• Task completion times

• Built-in Proxmox metrics (RRD data)
• External monitoring (Prometheus, Grafana)
• Log aggregation (syslog, ELK stack)
• Alerting for critical thresholds

• proxmox-workflows.md - Common operational patterns

• proxmox-troubleshooting.md - API quirks and solutions

• running: VM is powered on
• stopped: VM is powered off
• paused: VM execution suspended
• suspended: VM state saved to disk

• dir: Directory-based storage
• lvm: LVM volume groups
• zfs: ZFS pools
• ceph: Ceph RBD
• nfs: NFS shares
• iscsi: iSCSI targets

• snapshot: Fast, requires storage support
• suspend: Pauses VM during backup
• stop: Stops VM for backup

• started: Running on assigned node
• stopped: Intentionally stopped
• fence: Node fenced, restarting elsewhere
• error: HA manager error