causal-insights

Causal Insight Analysis for

doubleml-pipeline

Outputs

The pipeline has already estimated debiased causal effects with Double Machine Learning (DML). Read the completed

output_dir

files directly using the exact metric definitions in references/metric_formulas.md — never refit raw regressions on the input panel.

Adapt to the run's schema (
metadata.json
first): Treatments, channels, interventions, outcome KPIs, entity dimensions (

geo_col

item_col

), and covariates vary across users and domains (including non-monetary treatments with no spend column). Derive every lever name, unit type, segment label, and driver strictly from

metadata.json

and the columns present in

consolidated_1a_full_df.csv

— never assume specific channels, treatments, or domain constructs that are not in the data. When a question cannot be answered from the run's outputs, say so explicitly and name the exact pipeline setting (

treatment_cols

x_cols_list

covariates_for_cate_list

geo_col

item_col

optimize=True

, or

run_sensitivity=True

) needed to answer it.

Output Directory Reading Order


outcome_col
estimated_incremental_KPI_<d>
model == "median"
total_estimated_incremental_KPI
robustness_value
decision
optimization_metrics.csv

Step	File in `<output_dir>`	What to extract
1	`metadata.json`	`outcome_col` , `treatment_cols` , `treatment_types` , `treatment_spend_cols` , `x_cols_list` , `covariates_for_cate_list` , `geo_col` , `item_col` , `date_col` , `opt_threshold_roi` ( `threshold_roi` ), `saturation_family` , `saturation_phase0` , `phase4_optimization` , `cluster_dml` .
2	`step4_consolidated_results/consolidated_1a_full_df.csv`	Row-level panel + `estimated_incremental_KPI_<d>` , `_2.5%` , `_97.5%` , `roi_amount_based_<d>` , and (saturation optimizer only) `optimized_<d>_spend` / `optimized_incremental_KPI_<d>` . Aggregate all overall, segment, calendar, and driver slices from this table.
3	`step4_consolidated_results/graph/004_cate/cate_coef_dict_pooling.csv`	Pooled `model == "median"` row `cate_pvalue_dict` for Bonferroni-adjusted heterogeneity grades ( `strong` , `weak` , `none` ).
4	`step4_consolidated_results/consolidated_3a_detail.csv`	Phase 3 repetition stability ( `total_estimated_incremental_KPI` sign consistency and CV across `n_reps` ).
5	`step4_consolidated_results/graph/006_sensitivity/sensitivity_metrics.csv`	OVB `robustness_value` vs. `benchmark_covariate` ( `benchmark_cf_d` , `benchmark_cf_y` ).
6	`step0_saturation/summary_alpha_selection.json` & `diag6_t_ref_consistency.csv`	Phase 0 `decision` ( `saturation` vs. `linear` ), `decision_reason` ( `1se_pass` , `1se_fail` , `grid_edge_upper` ), `selected_family` , `alpha_hat` , `half_saturation_T` , `T_ref` , `d_bar` , `se` , `diag1_fired` , `diag2_fired` , `r_T` , `messages` . (Per-treatment subdirs when `len(treatment_cols) > 1` .)
7	`step4_consolidated_results/graph/007_optimization/` or `roi_output/<d>/007_optimization/`	`optimization_metrics.csv` and `optimization_yearly_summary.csv` ( `spend` treatments use `graph/007_optimization/` ; non- `spend` treatments with spend mapping use `roi_output/<d>/007_optimization/` ).

Four Non-Negotiable Calculation Rules

Always apply the formulas in references/metric_formulas.md to avoid silent calculation errors:

Order row-level 95% bounds before summing:
```
estimated_incremental_KPI_<d>_2.5%
```
and
```
_97.5%
```
are
```
(T - zero_point) * bound
```
, so they swap order whenever
```
T < zero_point
```
. Always compute
```
np.minimum(c25, c975).sum()
```
and
```
np.maximum(c25, c975).sum()
```
. Compute
```
effect_per_unit
```
(and its ordered 95% interval) by dividing by
```
T.sum()
```
, leaving it unset if a slice mixes positive and negative treatment values.
Divide by monetary spend for ROI (
treatment_spend_cols
): In
```
consolidated_3b_summary*.csv
```
,
```
total_estimated_roi
```
is
```
incremental_kpi / total_input
```
(effect per raw unit of treatment
```
<d>
```
, NOT monetary ROI when
```
treatment_types[d]
```
is
```
percentage
```
,
```
price
```
,
```
impressions
```
, or another non-spend unit). Always compute monetary ROI from
```
consolidated_1a_full_df.csv
```
by dividing
```
incremental_kpi
```
(and its ordered 95% bounds) by
```
df[treatment_spend_cols[d]].sum()
```
when spend is mapped (otherwise report
```
effect_per_unit
```
and its 95% interval). Compare
```
[roi_low, roi_high]
```
against
```
opt_threshold_roi
```
:
```
roi_high < threshold_roi
```
->
```
not_worth_investing
```
;
```
roi_low > threshold_roi
```
->
```
worth_investing
```
; straddling ->
```
undetermined
```
(omit verdict if
```
opt_threshold_roi
```
is
```
null
```
or no spend column exists). A
```
price
```
treatment is estimated as a gap
```
benchmark price - price
```
(its column is named
```
<price>_gap
```
;
```
metadata["price_benchmarks"][d]
```
gives the benchmark, the benchmark column and the investment column), so
```
effect_per_unit
```
is "incremental KPI per currency unit of price cut", rows with a zero gap were sold at the benchmark price, and a negative gap (price above the benchmark) counts as zero investment.
Grade CATE heterogeneity from the
median
row with Bonferroni scaling: In
```
cate_coef_dict_pooling.csv
```
(
```
model == "median"
```
), match all basis p-values for covariate
```
c
```
(
```
<c>_pvalue
```
,
```
bs(<c>, ...)
```
,
```
C(<c>)...
```
), compute
```
p_adj = min(1.0, min(ps) * len(ps))
```
, and classify
```
< 0.05
```
as
```
strong (p=...)
```
,
```
< 0.10
```
as
```
weak (p=...)
```
, and
```
>= 0.10
```
as
```
none (p=...)
```
(quoting
```
p<0.001
```
below
```
0.001
```
). Pooled BLP p-values ignore first-stage estimation error, so treat them as an approximate grade rather than a formal hypothesis test.
Exclude synthetic ground truth and derived cluster columns: Never read, profile, or report
```
TrueContrib_*
```
,
```
total_ground_truth_KPI
```
,
```
true_incremental_KPI_*
```
,
```
ground_truth_*
```
,
```
true_total_roi
```
,
```
SMAPE
```
,
```
R_squared
```
,
```
MAE
```
,
```
RMSE
```
, or
```
metadata["ground_truth_metrics"]
```
unless the user explicitly asks to audit synthetic recovery. Exclude
```
cluster_dml.derived_columns
```
(
```
cluster_date_block
```
) from driver analyses.

Allowed manual arithmetic & multi-run rule: Break-even gross margin

1 / roi_on_spend

when

outcome_col

represents gross revenue, and rescaling

effect_per_unit

to natural units (e.g., dividing by

when

treatment_types[d] == "percentage"

stored on

[0, 1]

to report per 1 percentage point). Never invent margins or benchmarks (ask the user instead), and never extrapolate outside observed support. Across multiple

output_dir

runs estimated separately, compare side by side only when

outcome_col

matches, and never sum

incremental_kpi

across runs. Speak in plain domain terms matching the user's column semantics and language without leaking internal schema keys.

7-Step Analysis Playbook

Work through the steps needed to answer the user's question. For a comprehensive review, execute all applicable steps and follow references/report_template.md.

Frame the run: Read

metadata.json

and inspect date frequency (

daily

weekly

monthly

), row count, active entity axes (

geo_col

item_col

), treatments (

treatment_cols

treatment_types

treatment_spend_cols

), confounders (

x_cols_list

covariates_for_cate_list

opt_threshold_roi

cluster_dml

, and Phase 0 / Phase 4 flags.

Headline causal impact: Summarize total
```
spend
```
(or treatment volume) and treated row count (Fact:), then
```
incremental_kpi
```
with 95% interval,
```
roi_on_spend
```
(or rescaled
```
effect_per_unit
```
),
```
share_of_outcome
```
, break-even margin (if revenue KPI), and payback verdict (Inference:).
Where it works (
geo_col
&
item_col
): Group
```
consolidated_1a_full_df.csv
```
by each configured entity axis (
```
geo_col
```
,
```
item_col
```
). Sort by
```
roi_on_spend
```
for efficiency questions or
```
incremental_kpi
```
for volume questions. Check whether
```
geo_col
```
/
```
item_col
```
is in
```
covariates_for_cate_list
```
.
When it works (calendar cadence): When
```
date_col
```
is present, group by month-of-year (and day-of-week when median date step <= 2 days, plus year-month or quarter for trajectory).
What amplifies it (drivers & 2-D interactions): Bin continuous drivers with within-entity quantiles (
```
within_quantile_bins
```
in references/metric_formulas.md, grouping by
```
geo_col
```
or
```
item_col
```
when present) so each unit is judged against its own baseline, and cross-tabulate 2-D interactions (e.g., a CATE covariate x treatment intensity, or two covariates from the run).
Can we trust it? (Diagnostics & raw segment profiles): Check repetition stability, OVB sensitivity (
```
robustness_value
```
vs.
```
max(benchmark_cf_d, benchmark_cf_y)
```
overall and least-robust segments), Phase 0 saturation diagnostics,
```
low_sample
```
(
```
< 30
```
treated rows), and
```
plausibility_flags
```
. Whenever two segments have non-overlapping 95% intervals (
```
ci_separation
```
on
```
roi_on_spend
```
or
```
effect_per_unit
```
) or
```
plausibility_flags
```
fire, profile that segment's raw data (
```
gap_in_row_sd
```
,
```
ratio_to_overall
```
,
```
treated_share
```
, and treated
```
p25/p50/p75
```
) before interpreting.
How should allocation move? (Optimization): When
```
007_optimization/
```
exists, report historical vs. optimized spend, incremental KPI, and ROI (total, by
```
geo_col
```
/
```
item_col
```
, and yearly from
```
optimization_yearly_summary.csv
```
), row counts shifted (
```
increased
```
,
```
reduced
```
,
```
cut_to_zero
```
), reconciliation with
```
optimization_metrics.csv
```
, and optimizer assumptions/warnings. Never design custom allocations, scale to unmodeled budgets, or shift budget across treatments or runs. If absent, note that optimization requires a single-treatment run with
```
optimize=True
```
,
```
date_col
```
, and spend mapping.

Causal & Domain Interpretation Rules

Reading 95% intervals: Always lead with the point estimate and
```
effect_direction
```
(
```
positive
```
,
```
negative
```
,
```
zero
```
). Aggregate 95% bands sum row-level bounds conservatively and routinely span zero. Never call an estimate "statistically significant", "not significant", "inconclusive", or "indistinguishable from zero", and never use the words significant or p-value in prose (except quoting CATE grades verbatim like
```
strong (p=0.0022)
```
as an approximate heterogeneity grade).
Average vs. Marginal ROI:
```
roi_on_spend
```
is the average return on past spend.
```
worth_investing
```
justifies keeping historical spend;
```
not_worth_investing
```
justifies cutting/stopping;
```
undetermined
```
calls for a controlled test. Scaling budget up requires marginal returns from an active saturation curve or KKT
```
optimization
```
. Under saturation, per-unit effect naturally falls at higher spend along diminishing returns.
Column Role (
cate_covariate
|
confounder
|
unused
) & Constant-Effect Runs: Per-unit treatment effects vary only through
```
covariates_for_cate_list
```
. If
```
covariates_for_cate_list
```
is empty, per-unit effect is constant by construction and
```
incremental_kpi
```
differs across slices only with treatment volume. If a slice column is
```
confounder
```
(in
```
x_cols_list
```
only, including entity dummies) or
```
unused
```
, splitting by it is descriptive — showing where effects landed via covariate mix and intensity, not modeled slice-specific response. Say so explicitly and recommend adding the column to
```
covariates_for_cate_list
```
to test it. Warn against conditioning on post-treatment variables (columns moved by the treatment itself) as targeting pre-conditions.
Confounding, Carryover & Spillover: Inspect
```
x_cols_list
```
against the other columns in
```
consolidated_1a_full_df.csv
```
to flag unadjusted concurrent treatments or levers, intertemporal pull-forward or lagged carryover effects (
```
_l<k>
```
lagged columns), and cross-unit spillover or substitution across
```
geo_col
```
/
```
item_col
```
.

Trust Checklist

Check	Required Action
Payback & multiplicity	Keep `worth_investing` , cut `not_worth_investing` , test `undetermined` . Across many `geo_col` or `item_col` slices, ~2.5% per tail can cross `threshold_roi` by chance unless consistent across periods and repetitions.
Driver evidence	Claim a covariate shifts per-unit response only when CATE grade is `strong` (quote verbatim, e.g., `strong (p=0.0022)` ). Treat `weak` as a lead to validate and `none` as no modeled heterogeneity.
Plausibility & support	Flag `< 30` treated rows ( `low_sample` ) as directional only. If `incremental_kpi_exceeds_outcome` , `kpi_without_spend` , or `negative_spend` fires, treat that slice's estimate as unreliable and inspect raw segment profiles.
Model & Phase 0	Caveat high repetition CV ( `> 0.3` ), mixed repetition signs, `robustness_value <= max(benchmark_cf_d, benchmark_cf_y)` (passing rules out confounders only up to the strongest observed `x_col` ), few clusters ( `< 30` in `cluster_dml.messages` ), or Phase 0 warnings ( `1se_fail` , `diag1_fired` , `diag2_fired` , low `r_T` , `fallback_half_saturation_T_used` ).
Optimizer discipline	Frame `optimization` strictly as a retrospective model-implied proposal (no carryover/adstock, point-estimate CATE, no 95% interval). State whether Phase 4 ran `saturation` , `simple` , or `linear_fallback` , warn if `kpi_after > 2.0 * kpi_before` (upper bound from cutting low-scored rows), if `kpi_before <= 0 < kpi_after` , or if `> 30%` of active units are cut to zero, and ask about operational floors/contracts.

Mandatory Statement Tagging (

Fact:

Inference:

Hypothesis:

)

Start every bullet point (and every standalone result sentence) with exactly one bold tag — never rely on section headers, and split mixed bullets:

Fact: Raw input panel quantities (
```
spend
```
, observed KPI, row/treated counts, treatment intensity quantiles, bin cut points, covariate means).
Inference: DML model outputs (
```
incremental_kpi
```
, 95% intervals,
```
roi_on_spend
```
, payback verdicts, CATE evidence grades, sensitivity
```
RV
```
, model-implied optimization numbers).
Hypothesis: Proposed mechanism explaining why an inference occurred, always paired with a concrete verification step (config change or holdout experiment).

Bundled References

Exact aggregation, ROI, CATE grading, diagnostics, and optimization formulas: references/metric_formulas.md
Full causal review structure and statement-tagging examples: references/report_template.md

causal-insights

NPX Install

Tags

SKILL.md Content

Causal Insight Analysis for
`doubleml-pipeline`
Outputs

Output Directory Reading Order

Four Non-Negotiable Calculation Rules

7-Step Analysis Playbook

Causal & Domain Interpretation Rules

Trust Checklist

Mandatory Statement Tagging (
`Fact:`
/
`Inference:`
/
`Hypothesis:`
)

Bundled References

causal-insights

NPX Install

Tags

SKILL.md Content

Causal Insight Analysis for doubleml-pipeline Outputs

Output Directory Reading Order

Four Non-Negotiable Calculation Rules

7-Step Analysis Playbook

Causal & Domain Interpretation Rules

Trust Checklist

Mandatory Statement Tagging (Fact: / Inference: / Hypothesis:)

Bundled References

Causal Insight Analysis for
`doubleml-pipeline`
Outputs

Mandatory Statement Tagging (
`Fact:`
/
`Inference:`
/
`Hypothesis:`
)