Throw down some knowledge: Statistical Significance

Time and again, one of the things I struggle with is statistical significance. I did OK when I took stats in college but have since forgotten everything, and the information online is scant. The calculator on A4D (thanks Mike Powell and Jason) is helpful, but I’m still constantly wondering: how many impressions / clicks / conversions is enough to determine a winner? Question to the veterans on this board: what do you typically think is "enough"?

@Lorenzo – "statistical significance" would be an awesome subject for a future blog post

User Comment:
Hey, It really depends on the type of offer, with most offers a 90 or 95% confidence rate isn’t needed, but you can do it on a gut feeling. Personally, I do a really rough 3x cpa = enough data to start optimizing off of, and that’s not mathematical at all but rather based on my experiences.

If you’re doing statistical sig in the stats way, it depends, if you’re doing a rebill for example, you’re going to spend a fortune before hitting any kind of meaningful sig level (hence why we do the rough non mathematical gut ways), while if you’re doing a $1 email submit you may have a chance to hit those levels and optimize based off of your formula (usually a z or t score, whatever one you want to use)…. hope that makes sense…

Personally I use my gut/quick and dirty math to optimize because it limits spends and I’ve found it results in more profit rather than dumping more money down the drain to say, hey this DEFINITELY isn’t working…

User Comment:

Thanks for the response. I agree the rules of thumb / gut check method is useful, and is basically what I do as well. I guess I’m wondering if there’s a middle ground between 95% confidence intervals that cost thousands of dollars to achieve, and the totally non-scientific "I’ll give it 5k impressions to see if it’s a good ad" approach.

User Comment:
Ive been using AND

From my Marketing Research background, I can tell you brands make million dollar descisions with 80% confidence.

User Comment:
Interesting, and good to know. Thanks Russian.

User Comment:
great post as always.

How do you account for offers that are only around for a few weeks, as far as, getting enough relevant info fast enough to make 80% conf. decisions?

Its when you discover its around for awhile, but that is something few know. Even offers that have track record could stop in a month and you not know it.

Keep in mind most times we do not know when the offer will be pulled.


The Article Published IN 06-09-2011 07:36 PM

Share To More ()