Design and implement a complete ML pipeline for: $ARGUMENTS
为$ARGUMENTS设计并实现完整的机器学习（ML）流水线
Use this skill when
适用场景
Working on machine learning pipeline - multi-agent mlops orchestration tasks or workflows
Needing guidance, best practices, or checklists for machine learning pipeline - multi-agent mlops orchestration
处理机器学习流水线 - 多Agent MLOps编排相关任务或工作流时
需要机器学习流水线 - 多Agent MLOps编排的指导方案、最佳实践或检查清单时
Do not use this skill when
不适用场景
The task is unrelated to machine learning pipeline - multi-agent mlops orchestration
You need a different domain or tool outside this scope
任务与机器学习流水线 - 多Agent MLOps编排无关时
需要该范围之外的其他领域或工具时
Instructions
操作说明
Clarify goals, constraints, and required inputs.
Apply relevant best practices and validate outcomes.
Provide actionable steps and verification.
If detailed examples are required, open
resources/implementation-playbook.md
.
明确目标、约束条件和所需输入。
应用相关最佳实践并验证结果。
提供可执行步骤和验证方法。
若需要详细示例，请打开
resources/implementation-playbook.md
。
Thinking
思路
This workflow orchestrates multiple specialized agents to build a production-ready ML pipeline following modern MLOps best practices. The approach emphasizes:
Phase-based coordination: Each phase builds upon previous outputs, with clear handoffs between agents
Modern tooling integration: MLflow/W&B for experiments, Feast/Tecton for features, KServe/Seldon for serving
Production-first mindset: Every component designed for scale, monitoring, and reliability
Reproducibility: Version control for data, models, and infrastructure
Continuous improvement: Automated retraining, A/B testing, and drift detection
The multi-agent approach ensures each aspect is handled by domain experts:
<Task>
subagent_type: data-engineer
prompt: |
Analyze and design data pipeline for ML system with requirements: $ARGUMENTS
Deliverables:
Data source audit and ingestion strategy:
Source systems and connection patterns
Schema validation using Pydantic/Great Expectations
Data versioning with DVC or lakeFS
Incremental loading and CDC strategies
Data quality framework:
Profiling and statistics generation
Anomaly detection rules
Data lineage tracking
Quality gates and SLAs
Storage architecture:
Raw/processed/feature layers
Partitioning strategy
Retention policies
Cost optimization
Provide implementation code for critical components and integration patterns.
</Task>
<Task>
subagent_type: data-scientist
prompt: |
Design feature engineering and model requirements for: $ARGUMENTS
Using data architecture from: {phase1.data-engineer.output}
Deliverables:
Feature engineering pipeline:
Transformation specifications
Feature store schema (Feast/Tecton)
Statistical validation rules
Handling strategies for missing data/outliers
Model requirements:
Algorithm selection rationale
Performance metrics and baselines
Training data requirements
Evaluation criteria and thresholds
Experiment design:
Hypothesis and success metrics
A/B testing methodology
Sample size calculations
Bias detection approach
Include feature transformation code and statistical validation logic.
</Task>
<Task>
subagent_type: ml-engineer
prompt: |
Implement training pipeline based on requirements: {phase1.data-scientist.output}
Using data pipeline: {phase1.data-engineer.output}
Build comprehensive training system:
Training pipeline implementation:
Modular training code with clear interfaces
Hyperparameter optimization (Optuna/Ray Tune)
Distributed training support (Horovod/PyTorch DDP)
Cross-validation and ensemble strategies
Experiment tracking setup:
MLflow/Weights & Biases integration
Metric logging and visualization
Artifact management (models, plots, data samples)
Experiment comparison and analysis tools
Model registry integration:
Version control and tagging strategy
Model metadata and lineage
Promotion workflows (dev -> staging -> prod)
Rollback procedures
Provide complete training code with configuration management.
</Task>
<Task>
subagent_type: python-pro
prompt: |
Optimize and productionize ML code from: {phase2.ml-engineer.output}
Focus areas:
Code quality and structure:
Refactor for production standards
Add comprehensive error handling
Implement proper logging with structured formats
Create reusable components and utilities
Performance optimization:
Profile and optimize bottlenecks
Implement caching strategies
Optimize data loading and preprocessing
Memory management for large-scale training
Testing framework:
Unit tests for data transformations
Integration tests for pipeline components
Model quality tests (invariance, directional)
Performance regression tests
Deliver production-ready, maintainable code with full test coverage.
</Task>
<Task>
subagent_type: observability-engineer
prompt: |
Implement comprehensive monitoring for ML system deployed in: {phase3.mlops-engineer.output}
Using Kubernetes infrastructure: {phase3.kubernetes-architect.output}
Monitoring framework:
Model performance monitoring:
Prediction accuracy tracking
Latency and throughput metrics
Feature importance shifts
Business KPI correlation
Data and model drift detection:
Statistical drift detection (KS test, PSI)
Concept drift monitoring
Feature distribution tracking
Automated drift alerts and reports
System observability:
Prometheus metrics for all components
Grafana dashboards for visualization
Distributed tracing with Jaeger/Zipkin
Log aggregation with ELK/Loki
Alerting and automation:
PagerDuty/Opsgenie integration
Automated retraining triggers
Performance degradation workflows
Incident response runbooks
Cost tracking:
Resource utilization metrics
Cost allocation by model/experiment
Optimization recommendations
Budget alerts and controls
Deliver monitoring configuration, dashboards, and alert rules.
</Task>