Skip to main content
When Dema generates a suggested experiment, it provides a detailed table to help you understand how the test is structured and what to expect. This table breaks down the test’s estimated performance metrics and key characteristics of the treatment and control groups.

​
Understanding the suggested experiment table

When Dema generates a suggested experiment, it provides a detailed table to help you understand how the test is structured and what to expect. This table breaks down the test’s estimated performance metrics and key characteristics of the treatment and control groups. Here’s what each column represents:

​
Experiment statistics

  1. Estimated spend
    • The amount of advertising spend that will be removed (in a decrease spend test) or added (in an increase spend test) in the treatment group.
    • This helps you understand the magnitude of the change being tested.
  2. Spend previous period
    • The amount of advertising spend during the same time period as the planned test, but before the test begins.
    • This serves as a baseline to compare current spending and measure changes.
  3. Estimated total gross sales
    • The total gross sales expected across all regions in the test, including both treatment and control groups.
    • This provides an overview of total expected revenue during the test period.
  4. Estimated incremental gross sales
    • The amount of sales expected to be lost (in a decrease spend test) or gained (in an increase spend test) in the treatment group due to the test.
    • This is calculated based on the ROAS or epROAS applied to the estimated spend changes.
  5. Treatment group proportion
    • The percentage of regions assigned to the treatment group versus the control group.
    • A lower proportion indicates that fewer regions will experience the test intervention, helping to reduce potential disruptions.
  6. Estimated lift
    • The percentage change in performance (e.g., sales, profit) expected as a result of the test.
    • This metric indicates the potential impact of the experiment.
  7. Minimum detectable effect (MDE)
    • The smallest measurable change that can be reliably detected by the test.
    • This reflects the sensitivity of the test design; smaller MDE values indicate higher sensitivity.
  8. Treatment group correlation
    • The correlation between treatment and control groups in terms of historical performance.
    • Higher correlation values (e.g., above 90%) indicate that the groups are well-matched, which improves the reliability of the test.
  9. Required scaling (for scaling tests)
    • A helper metric that shows how large the required scaling should be for the test.
    • This metric helps you understand the magnitude of spend changes needed in the treatment group to achieve meaningful results.

​
Geographical targeting

Treatment

Lists the regions (e.g., zip codes or commute zones) included in the treatment group, where the test intervention will occur (e.g., spend increased or decreased).

Control

Lists the regions included in the control group, which serves as the baseline for comparison.
Download CSV: Provides a downloadable file with the full list of treatment and control regions for detailed review and implementation.

​
How to use this information

​
Interpreting estimated metrics

  • Understand the impact on spend and sales: Compare the estimated spend and incremental gross sales to gauge the scale and potential outcomes of the test.
  • Plan for changes in overall performance: Use the estimated total gross sales to anticipate any shifts in overall revenue during the test period.
  • Assess test sensitivity*: Review the MDE and treatment group correlation to ensure the test design is robust and likely to yield actionable insights.*

​
Preparing for the test

To ensure successful test implementation:
  • Align with your team: Share the treatment and control group details with your team to ensure everyone is aligned on the regions affected.
  • Download the targeting details: Use the CSV file to streamline campaign adjustments in ad platforms.
By understanding the suggested experiment table, you can effectively plan and execute tests that drive meaningful insights into the incremental impact of your marketing efforts.