A bit of a rant about Which? magazine

As we’re renovating our house and need to buy quite a few new household appliances, etc I decided to take up Which? magazine’s offer of trial membership so I could get access to their online reviews and test results. As far as I remember, Which? always had a pretty good reputation and it seems like the smart thing to do, to research stuff before you buy. But I’ve been unimpressed, and a little bit surprised, by the service.

Here are the things I found within the first few hours of using the service:

• The website itself is littered with usability problems. For example, after I successfully log in and go to look at a review, I still see a large box in the middle of the page asking me to sign up or log in! When I go to add a comment to a review, it asks me to create a new identity, then drops me somewhere else completely and I have to hunt for the comment thread again in order to write the review I wanted to write in the first place. Basic stuff.
• The website has security issues. I was dismayed to see that the password I entered was promptly emailed to me. Standard best-practice is to never email out passwords, but only to allow the user to reset them if required.
• The content on the website can be a bit patchy in places. When looking at the review for one item I noticed that the photo didn’t match what it said in the ’specs’ section. Got the impression it had been automatically pulled in from some database and no-one had checked it.
• But far more serious than all that is that I don’t really believe they are giving a balanced judgement about what to buy. They highlight a series of scientific-type tests that give measurable results, and these are easy to communicate and look good. But they often seem to completely ignore other important considerations like quality of construction, how long things might last, and aesthetic appeal. So for instance a £700 range cooker might get a score of 66% based on tests like how long the oven takes to heat up, how easy it is to clean, and so on. This implies it is better overall than another, premium cooker costing £3000 that gets a score of 65%. Yes, oven performance and ease of use are important, but they don’t mention that the cheaper cooker might have cheap-looking plastic dials, for example, and be much less rugged in construction.

It probably is useful to look at resources like Which? when researching expensive items, but I wouldn’t take their ratings as definitive, and definitely go and see stuff yourself before you buy. And don’t use a valuable password if you register for their website!