What does a statistically significant difference mean in an A/B test?

Q: I just ran an A/B test and on the Publish Winner page it tells me that my results are statistically significant. What does that mean?

A: An A/B test is exactly that, a test. What you are testing is a hypothesis of some sort. An example of an A/B hypothesis could be, "if I use a subject line with the recipient's name in the subject line, then I will get more people to open the email"

Statistical significance is a way to measure the validity of the result. Basically, when something is statistically significant, then the results of the test are most likely due to your efforts. If something is not statistically significant, then it means that the results may be because of your test, but they could also just be a random result.