The Business of Software

Let's say your product is $99,

Half of the users would see a special for $10 off
Half of the users would see a special for $20 off

Split test the results and see which one makes you more money.

Is this dirty? Is it even possible? Are customers likely to feel burned if they ever find out?

What if a customer comes to the page twice and sees a different price? I think people would be annoyed.

Why should it be dirty?  (by which I asumme you mean immoral)

If a customer gets more $99 of value out of a product for which they would pay $99, and therefore decides to buy, that remains true regardless of whether other people are paying more or less.


That said, it is a sensitive area so needs to be handled carefully. You may get found out (although unless you're Amazon or something, your customers are probably not paying enough attention to find you out). I'd work on the basis, always (not just in this case), that you may get found out - I think that's a good rule in general. 

In general, don't do stuff that you wouldn't want to be found out on. 

But, this isn't such a case.  If you're found out, just say "I'm trying different prices".  Supermarkets do it all the time (hint: look on the bottom shelf in the store when shopping for groceries).

Liam - that problem is easily avoided. Cookies is a crude solution. You can also use geolocation - if the person comes from states A,B,C price X, states D,E,F price Y - being careful to balance the states.

I did customer support for a small ISV and did price testing all the time. We did get "caught" a few times. We were straight with our customers and said what we were doing. We then sent them to a link for the cheaper offer.

On top of that, we continuously ran a coupon program. One coupon to buy quickly after download and another for return customers. From my experience of answering 10,000 customer emails I never had one customer get really made about pricing. As long as you talk straight, most reasonable people get it.

testing like this is incredibly common, online and in normal retail channels

there's nothing wrong with it

I'm thinking of the same thing right now.  Can anyone suggest HOW to determine the state?

Google geolocation.  There are quite a few products that convert IP address to approximate location.

Yeah, I've been looking.  Most seem to either be wrong or expensive.  Anyone know of an accurate Free service?

Rather than using geolocation or cookies, you can also use a hashing function on IP address, then mod 2 it.  This gets you a random but mostly static and predictable coinflip, although it doesn't help if they access your site from multiple computers.  (Here's the rub though: nothing will, short of a login.)

Amazon has run A/B pricing experiments, and learned the hard way that customers *do* find out, and *are* upset about it.
So their new policy is: if they run a price experiment, you'll always get the lowest price in the experiment, when you get to the checkout.  (So you might click on something with 5% off - when you get to the checkout, it'll give it to you for 10% off, because that was the other offer that you didn't see).

rather than running with 2 prices how about having one price one week then the other price the next week.  If you can keep your promotion efforts reasonably even throughout you should get some ok data.  Personally Id have lower quality data than have people wandering about thinking were not trustworthy which is what they will think if they find out.

Just remember that a paying customer will regard any funny business regarding their money as dishonesty on your part regardless of whats really going on.

I can tell you that 10$ is not much of a discount for a 100$ product..  if someones gona pay 90$ they are just as willing to pay 100$..

so split test game aside.. you might be wasting time .. and headache trying to figure out of 10 is better than 20$ discount.

Depends if your product is price sensitive. If it's a B2B product there's probably little point in running these type of promotions.

I do also remember when Amazon got caught out over this though.

If you use Patricks method you might want to ignore the least significant byte as this may change regularly for some ISP's, otherwise I'd say this is probably the best method.

Hashing the IP alone unfortunately won't produce a consistent result. A substantial proportion, maybe about half of people don't static IPs, and several larger ISPs change IP even during a session.

Probably the least changeable bytes in the IP are the 1st and 2nd. The 1st in particular is highly correlated with country, so it will screw up your test to use this, unless you only count orders in say a particular country.  The 2nd might be better.  Even so, I wouldn't want to solely rely on it - it will change sometimes.

Using geolocation avoids that problem to some extent, as people don't usually move between computers in different states.

Or the third option is to simply pick at random 1 or 2.

In any of the 3 cases, what you can do to improve reliability somewhat is stuff the 1 or 2 in a cookie, and use the cookie value for subsequent accesses.  The IP hash, geolocation or random value is only used when the cookie is not there (first time visit usually).


Regarding Amazon, there method unfortunately means their test is not complete. Usually at commerce sites a substantial proportion of people drop out at the checkout, so this remains untested in their system.

Also Amazon is very different from you, because of their high media profile, millions of people watching on them, and low prices being one of the main motivators to shop at Amazon in the first place.

Use time() % 2 - that is, current second is even for $10, odd for $20 (could also use minutes). Store it in a cookie.

Count the number of visitors for each reduction... eg, if there's more $89 sold then it could be because there's more $10 off offers shown.

Nothing wrong with having a different price; Many people sell at different prices for different countries/currencies.

NaN, I think what these guys are discussing are ways to ensure the same price is displayed to any single person that checks it more than once.

Personally I like the sound of the IP-hashing idea combined with cookies, though the geo-location idea sounds better if you have the database of data.

Databases: ipligence.com maxmind.com ip2location.com netgeo.com probably others

This reminds me about a situation a few years ago.  I wanted to outsource work for a logo.  I came across a site that offered a x% off coupon. They may even have had a "only good until _____".  The coupon appeared on a pop-up window from the site via javascript.  I noticed that the link that the javascript opening have a parameter.  Some thing like http://domain.com/coupon.php?code=b.  Naturally, I tried browsing to http://domain.com/coupon.php?code=a. Guess what popped up? Another x% off coupon. If code b = 25% off, a = 10% off.  So naturally I tried code c, c equaled something like 33% off. I tried code d... d = 50% off. I tried code e... not valid.  I eventually used the company and submitted  a coupon code generated from using code d (50% off).  The company said they didn't/weren't offering any 50% off coupons and thought I must have made a typo so they still gave me the 33% off coupon. I wasn't upset since I still saved more then the original 25% off.  However the coupon was a major factor in determining whether to use them or not. I may have been a little upset if I found out I could have saved more after the fact but I was still pleased with the work they did. Side moral of the  story... don't make it obvious on how to get a better coupon code.