Calibrating an inverse geo test for a PyMC MMM model (original) (raw)

December 15, 2024, 3:00pm 1

Hi everyone,

I’m building an MMM model using PyMC and need some help with calibrating an inverse geo test. Not sure if I set this up right, and I’d appreciate any guidance!

Test details:

• Pre-test spend: $4500/day (whole market)
• During test: 1700/day(controlgeos,whichareamostunaffected),1700/day (control geos, which are amost unaffected), 1700/day(controlgeos,whichareamostunaffected),0 (test geos)
• Result: Δy = -400 installs in the test vs the control
• Std deviation: 5000

Here’s the way I summarized it:

df_lift_test = pd.DataFrame({
    "channel": "channel_name",
    "x": 1700,
    "delta_x": -1700,
    "delta_y": -400,
    "sigma": 5000
})

Does this setup look right for calibrating the impact?

Sorry if this is a basic question - any feedback would be super helpful!

Thanks a lot!

Hey @A_Bell you may know already but here its a good description.

A_Bell December 22, 2024, 8:16am 3

Hey @cetagostini, thanks! I’ve seen it, but I’m still unsure. My situation is a bit different since it’s inverse- I’ve lowered the budget. So I’m not sure if x and Δx should be for the entire market or just the test market.

The counterfactual thinking behind geo tests is that we created a test group where the budget was reduced to 0,andacomparablecontrolgroupwherethebudgetremainedat0, and a comparable control group where the budget remained at 0,andacomparablecontrolgroupwherethebudgetremainedat1700 (their pre-test allocation). So the difference in treatment between test and control is $1700, and it resulted in a -400 change in y.

I feel like I need to incorporate that somehow into the model. What do you think? Can you help me understand what to input into the model?

Thanks! BTW- your webinar on MMM was incredibly interesting and insightful, thanks for that!