Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion website/docs/About.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ id: About
---

## What is GeoLift?
GeoLift is an open-source and end-to-end solution to calculate Lift through geo-experiments. Based on some of the latest developments on **Synthetic Control Models**, the tool helps users plan, implement, and analyze complex geographic causal tests in a simple and easy-to-use way. Through GeoLift tests, users are able to measure the true value of their marketing campaigns, revamp their strategies, estimate their omni-channel perfomance, and even perform cross-channel optiimzations all while using the common currency of incrmentality. Moreover, GeoLift offers all of these features in a privacy-secure way through the exclusive use of aggregated data.
GeoLift is an open-source and end-to-end solution to calculate Lift through geo-experiments. Based on some of the latest developments on **Synthetic Control Models**, the tool helps users plan, implement, and analyze complex geographic causal tests in a simple and easy-to-use way. Through GeoLift tests, users are able to measure the true value of their marketing campaigns, revamp their strategies, estimate their omni-channel perfomance, and even perform cross-channel optimizations all while using the common currency of incrementality. Moreover, GeoLift offers all of these features in a privacy-secure way through the exclusive use of aggregated data.

---

Expand Down
4 changes: 2 additions & 2 deletions website/docs/Best Practices/BestPractices.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sidebar_position: 1
## Data
* Daily granularity in the data is strongly recommended over weekly.
* It is recommended to use the highest level of geographical granularity at which we can target Facebook campaigns in the study's region (Zip Codes, Cities, etc.).
* We recommend having at least 4 - 5x the test duration of pre-campaign historical data of stable data (must not contain structural changes or any other impactful deviation from their data-generating process.).
* We recommend having at least 4 - 5x the test duration of pre-campaign historical data of stable data (must not contain structural changes or any other impactful deviation from their data-generating process).
* At minimum we recommend having 25 pre-treatment periods of 20 or more geo-units.
* Under normal circumstances we advise having historical information to be over the last 52 weeks, this may take into consideration any seasonal variations across brand product sales as well as account for other factors that may not be taken into consideration over a shorter duration.
* The test duration should cover at least one purchase cycle for the product.
Expand All @@ -32,5 +32,5 @@ sidebar_position: 1
---

## National Marketing Efforts
* Take into consideration any significant variations across national media such as TV, Print etc. during pre-test or test period will make it hard to really isolate the impact of Facebook on the outcome of interest.
* Taking into consideration any significant variations across national media such as TV, Print, etc. during pre-test or test period will make it hard to really isolate the impact of Facebook on the outcome of interest.
* For sales to be truly attributed to Facebook variations, all other media should be held constant across the markets and if there are significant variations, make sure to address these before the test.
12 changes: 6 additions & 6 deletions website/docs/Methodology/Methodology.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,30 +12,30 @@ sidebar_position: 1
---

## Quasi-Experiments and Synthetic Control Methods
Quasi-Experiments offer a great alternative to measure Lift whenever an RCT is not viable and GeoLift leverages some of the latest developments in this area to empower advertisers to embrace incrementality. GeoLift is based on [Synthetic Control Methods (SCMs)](https://economics.mit.edu/files/11859), which work by creating an artificial unit that is as similar as possible to the test unit. Using historical information prior to the treatment SCMs find the combination of untreated units that most closely replicate the treated. This effectively creates a constructed or synthetic control which will provide a robust counterfactual to which we will measure the campaign’s effectiveness. Prior to the intervention the test and synthetic control are virtually identical, therefore, any differences after the treatment started between these two units is the campaign’s incrementality.
Quasi-Experiments offer a great alternative to measure Lift whenever an RCT is not viable and GeoLift leverages some of the latest developments in this area to empower advertisers to embrace incrementality. GeoLift is based on [Synthetic Control Methods (SCMs)](https://economics.mit.edu/sites/default/files/publications/Synthetic%20Control%20Methods.pdf), which work by creating an artificial unit that is as similar as possible to the test unit. Using historical information prior to the treatment SCMs find the combination of untreated units that most closely replicate the treated. This effectively creates a constructed or synthetic control which will provide a robust counterfactual to which we will measure the campaign’s effectiveness. Prior to the intervention the test and synthetic control are virtually identical, therefore, any differences after the treatment started between these two units is the campaign’s incrementality.


![GeoLift Example](/img/Methodology_GeoLift_Lift.png)


The idea behind SCMs is that a combination of units often provides a better comparison for the unit exposed to the intervention than any single unit alone (like in matching analyses). Moreover, by constructing the counterfactual as a weighted average of the units of observation, these approaches provide additional robustness against omitted variable biases as long as the control units have similar characteristics to the test. In practice, unit comparability and similarity are a given since GeoLift studies we use locations of the same region or country as test and control units. Moreover, SCMs rely on panel data and can reliably account for confounders changing over time unlike traditional Difference-In-Difference quasi-experimental methods.
The idea behind SCMs is that a combination of units often provides a better comparison for the unit exposed to the intervention than any single unit alone (like in matching analyses). Moreover, by constructing the counterfactual as a weighted average of the units of observation, these approaches provide additional robustness against omitted variable biases as long as the control units have similar characteristics to the test. In practice, unit comparability and similarity are a given since in GeoLift studies we use locations of the same region or country as test and control units. Moreover, SCMs rely on panel data and can reliably account for confounders changing over time unlike traditional Difference-In-Difference quasi-experimental methods.


---
## Making Synthetic Control Methods Even Better
Nevertheless, SCMs are not perfect and are subject to biases due to inexact matching which happen when we can’t reliably re-create the test unit with the controls. Fortunately, GeoLift is based on the combination of two cutting-edge SCM methods that address these limitations and provide a powerful foundation for our tool. Specifically, GeoLift combines the sturdiness of synthetic control estimations of [Augmented Synthetic Control Methods (ASCM)](https://eml.berkeley.edu/~jrothst/workingpapers/BMFR_Synth_Nov_2018.pdf) with the powerful inference capabilities of [Generalized Synthetic Controls (GSC)](https://www.cambridge.org/core/journals/political-analysis/article/generalized-synthetic-control-method-causal-inference-with-interactive-fixed-effects-models/B63A8BD7C239DD4141C67DA10CD0E4F3).
Nevertheless, SCMs are not perfect and are subject to biases due to inexact matching which happens when we can’t reliably re-create the test unit with the controls. Fortunately, GeoLift is based on the combination of two cutting-edge SCM methods that address these limitations and provide a powerful foundation for our tool. Specifically, GeoLift combines the sturdiness of synthetic control estimations of [Augmented Synthetic Control Methods (ASCM)](https://eml.berkeley.edu/~jrothst/workingpapers/BMFR_Synth_Nov_2018.pdf) with the powerful inference capabilities of [Generalized Synthetic Controls (GSC)](https://www.cambridge.org/core/journals/political-analysis/article/generalized-synthetic-control-method-causal-inference-with-interactive-fixed-effects-models/B63A8BD7C239DD4141C67DA10CD0E4F3).

Specifically, GeoLift uses ASCM to estimate and de-bias the synthetic control estimate and then uses GSC’s robustness on small samples and on heterogeneous effects across units to perform inference. Moreover, the two-step implementation of these approaches addresses imbalances caused by the curse of dimensionality which typically causes bias as the amount of historical data and units increases given that the likelihood of finding exact matching decreases rapidly as more dimensions are added to the solutions-space. Finally, GSC provides powerful parametric bootstrapping approaches to provide valid and reliable inference and uncertainty estimates. All-in-all, the combination of these two approaches provide robustness against biases in GeoLift at the cost of additional processing power.

SCMs have been regarded as [“arguably the most important innovation in the policy evaluation literature in the last 15 years”](https://faculty.smu.edu/millimet/classes/eco7377/papers/athey%20imbens%202017.pdf) (Athey and Imbens 2015). However, their adoption in other areas such as marketing has been slow. This is mainly due to the lack of power calculations for, which makes them difficult to plan and design future studies. These calculations are particularly important for geo-testing experiments, where the effect sizes are usually small and where there is often a significant chance to fail to find the effect of the experiment. Therefore, running a geo-test without a robust prior power analysis leads to a high chance of failing to find lift, even if it actually happened.
SCMs have been regarded as [“arguably the most important innovation in the policy evaluation literature in the last 15 years”](https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.31.2.3) (Athey and Imbens 2017). However, their adoption in other areas such as marketing has been slow. This is mainly due to the lack of power calculations for campaigns, which makes it difficult to plan and design future studies. These calculations are particularly important for geo-testing experiments, where the effect sizes are usually small and where there is often a significant chance to fail to find the effect of the experiment. Therefore, running a geo-test without a robust prior power analysis leads to a high chance of failing to find lift, even if it actually happened.

Moreover, through a power analysis we can not only align expectations before the test, but we can even set it up for success by finding which test set-up has the best chance to detect the lift. Through this analysis we can find which are the best test locations, how many we should include, investment, and even how long we should run the test for in order to be able to detect the lift.
Moreover, through a power analysis we can not only align expectations before the test, but we can even set it up for success by finding which test set-up has the best chance to detect the lift. Through this analysis we can find which are the best test locations, how many we should include, investment level, and even how long we should run the test for in order to be able to detect the lift.

GeoLift addresses these issues by providing three power calculators for three common use-cases:

#### Test length and investment/lift for known test locations

This calculator is useful when an advertiser knows which test locations he wants to use for an experiment, but needs help finding out the investment and test length. The calculator takes as input the dataset, a list of thest locations, and a Cost Per Incremental Conversion (if available) to help determine the necessary investment to execute a well-powered test.
This calculator is useful when an advertiser knows which test locations he wants to use for an experiment, but needs help finding out the investment and test length. The calculator takes as input the dataset, a list of test locations, and a Cost Per Incremental Conversion (if available) to help determine the necessary investment to execute a well-powered test.



Expand Down