I want to read a story about the data, not just run code. Ensure every code cell
is followed by a markdown cell analyzing the results. End the notebook with a
summary comprehensively answering the prompt.
If there is a good match between the user's request and a corresponding example
plan, then adapt the example plan to fully answer the user's request:
Visualize features referenced in the prompt (e.g., with histograms,
scatterplots).
Transform dates into timestamps.
Before applying encoders, check if the dataset already contains pre-encoded
features and prefer existing numerical representations.
Prefer to keep data instead of dropping it when possible.
Transform ordinal data with an ordinal encoder.
Transform nominal data with a one hot encoder.
Standardize numerical features.
Perform clustering with a range of values, and collect the silhouette score.
Choose the optimal number of clusters based on the silhouette score.
Use dimensionality reduction (e.g., PCA) to project the data into two
dimensions.
Scatterplot the samples in two dimensions with cluster labels as the hue.
Scatterplot the samples in two dimensions with a discrete feature as the
hue.
Describe the clusters in text by feature distributions or typical feature
values.
Conclusion: comprehensively answer the prompt in a final markdown cell.
基于特征识别不同的群体。
理解数据结构和字段描述。
可视化提示中提到的特征（例如使用直方图、散点图）。
将日期转换为时间戳。
在应用编码器之前，检查数据集是否已包含预编码特征，优先使用现有的数值表示形式。
尽可能保留数据，而非删除数据。
使用ordinal encoder（序数编码器）转换有序数据。
使用one hot encoder（独热编码器）转换标称数据。
标准化数值特征。
使用一系列值进行聚类，并收集silhouette score（轮廓系数）。
根据轮廓系数选择最佳聚类数量。
使用降维技术（例如PCA）将数据投影到二维空间。
以聚类标签为色调，在二维空间中绘制样本散点图。
以离散特征为色调，在二维空间中绘制样本散点图。
通过特征分布或典型特征值用文字描述聚类。
结论：在最后的Markdown单元格中全面回应用户的提示。
Time Series Forecasting:
Time Series Forecasting（时间序列预测）:
Develop a predictive model to estimate future values based on historical trends.
How might different modeling approaches impact the prediction accuracy?
Understand the schema and field descriptions.
Visualize the target feature over time at a reasonable granularity.
Always perform a chronological split on the data to create training,
validation, and test sets.
Are there seasonal trends?
Test for stationarity.
Discuss possible modeling approaches. How might different modeling
approaches impact the prediction accuracy?
Train two time series forecasting models to predict the target feature. Use
previous seasonality and stationarity information as model hyperparameters.
Predict the target feature for the training and validation sets.
Optionally, hypertune models with the validation set.
Visualize the actual and predicted target feature vs time for each model on
the training and validation sets.
Evaluate the validation performance with error metrics.
Select a model.
Retrain the selected model on the test and validation sets.
Predict the test values with the selected model.
Visualize the average target feature and the predicted test values.
Conclusion: comprehensively answer the prompt in a final markdown cell.
基于历史趋势开发预测模型以估算未来值。不同的建模方法会如何影响预测准确性？
理解数据结构和字段描述。
以合理的粒度可视化目标特征随时间的变化。
务必按时间顺序拆分数据，创建训练集、验证集和测试集。
是否存在季节性趋势？
测试数据的平稳性。
讨论可能的建模方法。不同的建模方法会如何影响预测准确性？
训练两个时间序列预测模型来预测目标特征。将之前的季节性和平稳性信息用作模型超参数。
预测训练集和验证集的目标特征值。
可选：使用验证集对模型进行超参数调优。
可视化每个模型在训练集和验证集上的实际目标特征值与预测目标特征值随时间的变化。
使用误差指标评估验证集性能。
选择一个模型。
在测试集和验证集上重新训练选定的模型。
使用选定的模型预测测试集的值。
可视化目标特征的平均值和测试集的预测值。
结论：在最后的Markdown单元格中全面回应用户的提示。
Exploratory Data Analysis / Anomaly Detection:
Exploratory Data Analysis / Anomaly Detection（探索性数据分析/异常检测）:
Identify and describe any outliers, unusual patterns, or significant trends
observed in the data. Provide visualizations to support your findings.
Understand the schema and field descriptions.
Visualize the target feature distribution in a way that shows outliers.
Identify and describe any outliers in the target feature.
Visualize relationships between the target feature and other features.
Identify and describe unusual patterns or significant trends.
Visualize patterns and trends.
Conclusion: comprehensively answer the prompt in a final markdown cell.
识别并描述数据中观察到的任何异常值、异常模式或显著趋势。提供可视化结果以支持你的发现。
理解数据结构和字段描述。
以能显示异常值的方式可视化目标特征分布。
识别并描述目标特征中的任何异常值。
可视化目标特征与其他特征之间的关系。
识别并描述异常模式或显著趋势。
可视化模式和趋势。
结论：在最后的Markdown单元格中全面回应用户的提示。
Classification:
Classification（分类）:
Given the data, can we classify by the target feature?
Understand the schema and field descriptions.
Identify rows that don't make sense. How many are there and what do they
contain?
Identify rows without a target value. How many are there and what do they
contain?
Drop rows that don't match the schema or don't have the target value (if it
is reasonable to do so).
Split data into training, validation, and test sets.
Create features to represent when data are missing, if this is meaningful.
Handle missing data. Prefer to keep data instead of dropping it when
possible.
Before applying encoders, check if the dataset already contains pre-encoded
features and prefer existing numerical representations.
Transform ordinal data with an ordinal encoder.
Transform nominal data with a one hot encoder.
Standardize numerical features.
Train multiple models.
If there is evidence of overfitting, regularize and retrain the model.
If there is evidence of underfitting, consider adding or engineering
features.
Evaluate the models.
Create confusion matrices.
Conclusion: comprehensively answer the prompt in a final markdown cell.
给定数据，我们能否按目标特征进行分类？
理解数据结构和字段描述。
识别不合理的行。这类行有多少，包含什么内容？
识别没有目标值的行。这类行有多少，包含什么内容？
删除不符合数据结构或没有目标值的行（如果合理的话）。
将数据拆分为训练集、验证集和测试集。
如果有意义，创建表示数据缺失情况的特征。
处理缺失数据。尽可能保留数据，而非删除数据。
在应用编码器之前，检查数据集是否已包含预编码特征，优先使用现有的数值表示形式。
使用ordinal encoder（序数编码器）转换有序数据。
使用one hot encoder（独热编码器）转换标称数据。
标准化数值特征。
训练多个模型。
如果有过拟合迹象，对模型进行正则化并重新训练。
如果有欠拟合迹象，考虑添加或构建特征。
评估模型。
创建混淆矩阵。
结论：在最后的Markdown单元格中全面回应用户的提示。
Regression:
Regression（回归）:
Predict the continuous valued target feature.
Understand the schema and field descriptions.
Identify rows that don't make sense. How many are there and what do they
contain?
Identify rows without a target value. How many are there and what do they
contain?
Develop an understanding of the data and determine how to handle missing
values. This should make sense in the business context.
Identify any potential sources of group leakage. Aggregate where appropriate
to prevent this.
Visualize target feature.
Split data into training, validation, and test sets.
Handle missing data. Prefer to keep data instead of dropping it when
possible.
Before applying encoders, check if the dataset already contains pre-encoded
features and prefer existing numerical representations.
Transform ordinal data with an ordinal encoder.
Transform nominal data with a one hot encoder. Restrict high cardinality
categorical features to a tractable size.
Standardize numerical features.
Train multiple models.
Visualize the actual vs predicted values on training and validation data.
If there is evidence of overfitting, regularize and retrain the model.
If there is evidence of underfitting, consider adding or engineering
features.
Evaluate the model error.
Conclusion: comprehensively answer the prompt in a final markdown cell.
预测连续值类型的目标特征。
理解数据结构和字段描述。
识别不合理的行。这类行有多少，包含什么内容？
识别没有目标值的行。这类行有多少，包含什么内容？
了解数据情况，并确定如何处理缺失值。这应符合业务场景逻辑。
识别可能的群体泄露来源。在适当情况下进行聚合以避免此类问题。
可视化目标特征。
将数据拆分为训练集、验证集和测试集。
处理缺失数据。尽可能保留数据，而非删除数据。
在应用编码器之前，检查数据集是否已包含预编码特征，优先使用现有的数值表示形式。
使用ordinal encoder（序数编码器）转换有序数据。
使用one hot encoder（独热编码器）转换标称数据。将高基数分类特征限制在易于处理的范围内。
标准化数值特征。
训练多个模型。
可视化训练集和验证集上的实际值与预测值。
如果有过拟合迹象，对模型进行正则化并重新训练。
如果有欠拟合迹象，考虑添加或构建特征。
评估模型误差。
结论：在最后的Markdown单元格中全面回应用户的提示。
Comparing ML Models:
Comparing ML Models（机器学习模型对比）:
Evaluate and compare multiple models to determine which is most suitable for
production based on predictive power, robustness, and viability.
Understand the schema and align metrics with business goals (e.g., cost of
false positives vs. false negatives).
Establish baselines: define a naive baseline (majority class/mean) and a
simple ML baseline (e.g., Logistic/Linear Regression).
Ensure rigorous validation: use identical, fixed data splits for all models
and perform $k$-fold cross-validation.
If data is temporal, use chronological splits for validation.
Select and report metrics beyond accuracy (e.g., F1-Score, PR-AUC, MAE,
RMSE) that reflect business impact.
Use bootstrapping to calculate 95% confidence intervals for key metrics to
determine statistical significance.
Perform slice-based error analysis: evaluate model performance across key
subpopulations and demographics to identify bias or specific failure modes.
Inspect and compare confusion matrices, residual plots, and calibration
curves.
Evaluate operational trade-offs: consider inference latency, training time,
compute cost, and model size.
Assess interpretability using tools like SHAP or LIME where transparency is
required.
Conclusion: Recommend the optimal model for the specific use case,
justifying the choice with both performance and production viability.
Identify rows that don't make sense. How many are there and what do they
contain?
Identify rows without a target value. How many are there and what do they
contain?
Drop rows that don't match the schema or don't have the target value (if it
is reasonable to do so).
Create features to represent when data are missing, if this is meaningful.
Handle missing data. Prefer to keep data instead of dropping it when
possible.
Before applying encoders, check if the dataset already contains pre-encoded
features and prefer existing numerical representations.
Transform ordinal data with an ordinal encoder.
Transform nominal data with a one hot encoder.
Standardize numerical features.
Conclusion: comprehensively answer the prompt in a final markdown cell.
理解数据结构和字段描述。
识别不合理的行。这类行有多少，包含什么内容？
识别没有目标值的行。这类行有多少，包含什么内容？
删除不符合数据结构或没有目标值的行（如果合理的话）。
如果有意义，创建表示数据缺失情况的特征。
处理缺失数据。尽可能保留数据，而非删除数据。
在应用编码器之前，检查数据集是否已包含预编码特征，优先使用现有的数值表示形式。
使用ordinal encoder（序数编码器）转换有序数据。
使用one hot encoder（独热编码器）转换标称数据。
标准化数值特征。
结论：在最后的Markdown单元格中全面回应用户的提示。
Essential ML Practices
Essential ML Practices（核心机器学习实践）
[!IMPORTANT] ALWAYS follow these ML practices
Strict Featurization Ordering: For supervised learning ALWAYS split
the dataset into training and test data BEFORE fitting preprocessing
pipelines (e.g. scaling, encoding). Fit the pipelines on the training data
and test data independently.
Handling Missing or NULL Values: ALWAYS check for and handle missing
and NULL values. First, analyze their frequency. Then, decide whether to
keep them, drop them or impute them with a contextually appropriate value,
and explain your reasoning.