* The Business of Software

A former community discussing the business of software, from the smallest shareware operation to Microsoft. A part of Joel on Software.

We're closed, folks!


» Business of Software FAQ
» The Business of Software Conference (held every fall, usually in Boston)
» Forum guidelines (Please read before posting!)


Andy Brice
Successful Software

Doug Nebeker ("Doug")

Jonathan Matthews
Creator of DeepTrawl, CloudTrawl, and LeapDoc

Nicholas Hebb
BreezeTree Software

Bob Walsh
host, Startup Success Podcast author of The Web Startup Success Guide and Micro-ISV: From Vision To Reality

Patrick McKenzie
Bingo Card Creator

Google Experiments and the Multi-Armed Bandit Method

I started writing a long post about Google Experiments and the method it uses for allocating traffic to experiment alternatives and how it seems to be skewing my landing page evaluation results.  The post got more than long enough to be worth putting into blog post instead.  If you're using Google Experiments it might be worth reading.

Mark Nemtsas Send private email
Wednesday, October 16, 2013
Assuming this was also the case with the old website optimiser it makes a lot of sense. I remember running a few experiments where in the first day the statistics for both were rapidly gaining views. Then everything would slow down and it would take a long time to reach statistical significance.

This always annoyed me. I generally only try variation which I think are intuitively likely to produce a good result, so am not expecting a massive drop from the B version. What I want was fast results. I want to run a test asap, then follow it up with another. You could argue I should have used multi-variant. Unfortunately that has the side-effect of making my brain melt.

At least now I know why and can switch it off in the future. Thanks Mark.
Jonathan Matthews Send private email
Thursday, October 17, 2013
Favorited :o)

That's a good, clear examination of a thorny issue, thanks.

Reluctantlyregistered Send private email
Thursday, October 17, 2013

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz