While shop­ping on Ama­zon and Newegg, I’ve come across a most frus­trat­ing prob­lem. Both web­sites of­fer a sort­ing fea­ture where you can view items in a par­tic­u­lar cat­e­go­ry usu­al­ly by price or name or re­lease date or things like that. Both al­so al­low buy­ers to rate items after they’ve used them, and al­low po­ten­tial buy­ers to sort the items based on other cus­tomers’ re­views. There is a fair­ly frus­trat­ing prob­lem with this sys­tem, how­ev­er.

Both Ama­zon and Newegg (and pre­sum­ably thou­sands of other on­line stores) sort by av­er­age cus­tomer rat­ing when told to sort by rat­ing. If I were shop­ping for, say, com­put­er mon­i­tors on Newegg, and I was look­ing for one that was re­li­able and rat­ed high­ly by a lot of peo­ple, the av­er­age rat­ing sys­tem might be mis­lead­ing. Say there is an HP mon­i­tor for sale, which 500 peo­ple bought and liked. One per­son, how­ev­er, bought it and found a few dead pix­els (man­u­fac­tur­ing de­fect - it hap­pens), and so rat­ed it low. The mon­i­tor’s av­er­age rat­ing would be 4.98 out of 5.00 or some­thing like that.

Now con­sid­er a no-name mon­i­tor that one per­son bought and liked. He or she is the on­ly per­son to have ev­er rat­ed this mon­i­tor, and rat­ed it a 5, mak­ing the mon­i­tor’s av­er­age rat­ing 5.00. 5.00 for this no-name mon­i­tor is high­er than 4.98 for the HP mon­i­tor - this would put the no-name mon­i­tor high­er up on the list if I asked Newegg to sort by “Best rat­ing”.

For a se­lec­tion of two prod­ucts this seems like a mi­nor prob­lem, but if I were look­ing for, per­haps, a new DVD burn­er, the hun­dreds of prod­ucts which re­ceived two or three 5.00 re­views would fill up many pages of my search with ir­rel­e­vant prod­ucts - I would not con­sid­er one or two peo­ple rat­ing a pro­duct a 5.00 to be an ac­cu­rate indi­ca­tion that the pro­duct is, on the whole, re­li­ably con­struct­ed. I would have to wade through all those pages be­fore I found the first pro­duct rat­ed by more than about five peo­ple.

There must be a bet­ter way to sort prod­ucts by cus­tomer rat­ing than sim­ply by av­er­age rat­ing, which pro­duces mis­lead­ing re­sults. Newegg’s sort by num­ber of rat­ings is bet­ter but still not ex­act­ly it. What if five hun­dred peo­ple bought a pro­duct and rat­ed it bad­ly? I think a bet­ter so­lu­tion might be some sort of weight­ed av­er­age - 4.98s can be pulled high­er than 5.00s if they have more rat­ings. Some­thing along the li­nes of (score * num­ber-of-re­views), so that the 500 5.00s that the bet­ter pro­duct re­ceived counts more than the two 5.00s that the in­fe­ri­or pro­duct re­ceived.