A/B Test Interaction Effects

This report summarizes interaction effects detected in the A/B test.
We analyzed both binary outcomes (conversion, sign-up) and continuous outcomes (revenue, engagement).

Binary Outcomes

We modeled conversion probability using logistic regression with interaction terms. Significant effects include:

  • Habitual Buyer and Tail in Query bin: increased likelihood of conversion (p < 0.0463).
  • Low Results Segment and Treatment: smaller effect compared to baseline.
Plot of Binary Outcome Test
Continuous Outcomes

We modeled engagement and revenue using elastic net regression with bootstrapped confidence intervals.

  • Internal Channel and Visit Frequency 1-10: strongly positive effect on average revenue.
  • '1' in Segment Visitor x Treatment: moderate effect, not statistically significant.
Plot of Continuous Outcome Test
Key Takeaways
  • Interaction effects reveal dependencies between features not visible in main effects alone.
  • Habitual Buyer should be paired with Tail in Query bin to maximize conversions.
  • Further testing is needed for continuous outcomes in Segment Visitor due to high uncertainty.