Unpacking Statistical Significance in Brand Lift

Statistical significance is a crucial concept in the realm of data analysis, providing researchers and analysts with a powerful tool to draw meaningful conclusions from their findings. This blog post aims to demystify the meaning of statistical significance, exploring its meaning and value relative to our brand lift studies.

At its core, statistical significance measures the confidence in the meaningfulness of results, indicating the likelihood that the observed outcomes are not random. The confidence level reflects the strength of belief in the results being influenced by factors other than luck. In short, a statistically significant result would imply there is a confounding variable that would explain the difference in scoring, and the test would be the evidence to support this claim.  

As it relates to our brand lift studies, our goal is to find instances where these differences exist, then identify what these differences look like and articulate why these variables might lead to better performing assets to the clients.

Independence for Statistical Testing

In order to test two samples for significance, the samples should be independent.  In the statistical sense, independence means that the outcome of one event does not affect the outcome of the other.  At Veritonic, we consider each respondent an independent event to avoid bias being introduced, which could invalidate any statistical test run on the results.  

Significance Testing for Brand Lift 

The current statistical significance test we use for Brand Lift is called the two sample Z-test for proportions.  Brand Lift meets the independence condition for this test since the control and exposed groups have no overlapping respondents.  It is directly comparing the proportion of respondents meeting our success metric in the control group to the same proportion in the exposed group, and therefore, the two sample Z-test for proportions is a viable statistical significance test for Brand Lift.

Statistical significance is a cornerstone in the world of data analysis, providing a reliable framework for drawing meaningful conclusions. Its power lies in the ability to distinguish between real effects and chance occurrences, guiding informed decision-making. Although there are many elements that go into sound data and analysis, we believe in helping our clients understand statistical significance as one of them, and the role it plays in the methodology that is behind the data that they use to make informed, confident decisions.