One of the things I’ve been struggling with as I test is wondering if data is significant or not. I decided to calculate the statistical significance of my tests by "hand".
I went through some college class’s page on the topic and ended up putting together a spreadsheet that will calculate statistical significance with chi squares. It’s pretty similar to splittester.com, but has a little more detail. Using this, you can tell if your confidence is low b/c there simply isn’t enough data or if it’s because there really isn’t that much of a difference in the two ads/LPs/offers. Then you can tell if you’re 95%, 99% or 99.9% confident.
You only have to enter 4 numbers . Here’s how you use it:
– Figure out what you want to test. In this case I did 2 landing pages. You will be either proving or disproving a hypothesis with this method to determine if a relationship exists and then figure out how strong that relationship is. Mine could have been "Is there a relationship between landing page and CTR?"
I know that in my testing, LP1 had a higher CTR than LP2, but the exact numbers were different and I wanted to be certain that if I made a decision it would be based on fact and not pure speculation.
1. Split up the data in the first section. In the example below I have my 2 LPs. It’s split up into Clicked, did not click. You could do converted, did not, etc. The link to the info above may help if you’re unsure of how to enter whatever it is you’re testing (They were doing some kind of yes/no in their example). It doesn’t work with %, just whole numbers.
2. All the rest of the numbers will calculate on their own. All calculations in the ‘expected’ section must be over 5. If they aren’t, they will turn red and you need to collect more data. This helps me a lot. Tells me to keep on getting data and also not to make snap judgements just b/c some offer started out with a better CR or CTR.
3. The Chi Square total is in the gray box. The value determines the confidence. It must be higher than the numbers at the bottom that reference a stat table to have a particular confidence level. (again, green is good, red is bad).
In my case, I could be 99.9% sure that there was a significant relationship between the LP and the CTR and that LP1 was superior.
Not the fanciest thing ever, but I geeked out on it a little when I made it so I thought I’d share. Hopefully this will come in handy for someone else also.
Fantastic… now if you can post this up on a website you can start collecting affiliate IPs
The Article Published IN 09-29-2011 05:40 AM