http://www.evanmiller.org/how-not-to-run-an-ab-test.html

Why you should run a test for a set number of observations instead of until statistically significant.