A customer recently wrote:
I think you really need to add comments from users under each product (like sparkfun does). Makes it easy to review good/bad about a project and ask a question not in docs.
I rejected the idea when some of our developers pushed for comments on our web site a few years ago, and upon reconsideration, I am still firm in my opposition. However, I am interested in what others think, so rather than just giving the customer a quick reply, I spent a bit more time to present my view here. Several of my thoughts on this particular topic are based on broader philosophies of how people and organizations should operate, but I’ll try to focus on comment systems without getting too sidetracked.
First off, I use and like the customer review or feedback systems on sites like Amazon and Newegg, and I think it’s clear that the feature benefits customers and the sites. However, there are also many sites like CNN, where even minor reports can be appended with thousands of reader posts expounding on everything from the story not being news to some kid in the accompanying picture looking cute to some other reader being an idiot; the only thing missing in the comments is anything useful to anyone but psychologists and future anthropologists. Aside from some possible reductions in page load times, at least there is no real harm in the useless comments, and although part of me thinks that many of these sites are putting in user comment systems just because everyone else has one, I hope they know their business well enough to conclude that the cost of including a bunch of worthless content is outweighed by the benefit of enticing readers to participate in the site and thereby capturing a bit more of their attention. It might not be making the world better, but there are worse places than news sites for people to waste their time.
The spectrum of comment system value does not extend merely from obviously positive to approximately harmless; for a web site like ours, bad content is certainly harmful, and the likelihood of the user-generated content being bad, as a whole, is almost guaranteed. The second claim, that comment-based content on a site like ours will necessarily be bad content, is the bulk of my argument, and I’ll get to that after elaborating on the first point, that bad content is harmful to a site like ours. I see four reasons for bad user content to be more damaging to Pololu product pages, and by extension to our customers, than to product pages on Amazon:
First, unlike sites like Amazon or Newegg, we do not carry almost everything. When we make something like a motor controller, we might make a few versions to cover different current and voltage options, but we are not in the business of carrying fourteen different models of AVR programmer: if we think we have a good offering for a problem or application, we would rather spend our effort developing or looking for solutions to other problems rather than slightly different solutions for the same problem. If you are looking for a graphics card on Newegg and see a bad review for one, you are still probably going to get some other graphics card at that site. With our selection, a bad comment by a disgruntled customer could not just sway a potential customer toward another product we carry, but rather drive them from our site altogether. This isn’t meant to be some kind of sob story, but just my assessment of the situation, and it should not be surprising that I am against putting work into something that is against our own interest. (For those thinking that a site welcoming negative comments is a badge of openness or integrity, I’ll get to that later.)
Second, our products generally require much more expertise to use and evaluate than the typical Amazon product, making a review by an incompetent user harder to spot. If we see a review saying something like, “I would’ve given this hammer a higher rating but it keeps missing the nails, especially when I use it with my left hand”, we know right away to discount it. A more specialized site like Newegg might have the problem to some extent, too, but installing a graphics card is generally much easier than controlling a motor driver from a microcontroller. An electronics or robotics beginner might be competent enough to work through the documentation we provide but not necessarily be able to tell apart good advice or comments from bad. The bad comments could cause unnecessary frustration that might not only hurt our relationship with the customer but also the customer’s budding enthusiasm for engineering.
Third, for products like ours, it is difficult to separate the personal, subjective experience from objective facts, by which our products should be judged. If someone comments that a book is poorly written, that a pen is awkward to hold, that some shoes did not fit well, that a TV’s stand is ugly, or that some speakers sound bad, we are very aware that these are opinions, and it’s easier to take them with a grain of salt or go with two good impressions over one bad one, and those subjective opinions are still generally useful. However, it’s much more difficult to resolve conflicting statements in objective-sounding comments: you might see mixed reviews on the look of a sweater and still get it for your wife if you personally like how it looks; if you see some people saying a servo controller doesn’t work and others saying it works fine, it’s not as easy to dismiss the negative comments as just opinions or results in some particular applications that might not represent a typical application. This is not to say there is no room for subjective evaluation of our products or that comments cannot be helpful for products like ours; rather, the point is that there is a lot more room for comments to be disproportionately damaging.
These first three reasons are not just pie-in-the-sky, theoretical musings about why comment systems might be bad, and they do not apply only to small sites like ours. Just this past weekend, after I started writing this post, I went to the Lowe’s site to look into a blow torch for burning weeds in my front yard. They had one model, and it had a single customer review that claimed that contrary to what a Lowe’s staffer told him, the torch did not work with standard propane tanks. Of course, I wouldn’t be bringing this example up if the comment was correct: I eventually bought the thing at a physical store, and it worked just fine with the tanks they had there. However, if I were just shopping online, I would not have ordered it. In the store, I wasted a bunch of time trying to figure out why the thing would not fit the tanks since it sure looked like it would. There was no reason to expect the review to just be flat-out wrong, and no real way to tell that it was wrong. Since I was writing this post, I felt a greater need to address this particular problem, so I wrote a review of my own. I think it’s the first time I’ve written an online review for a product like that.
So now, there are two reviews sitting there, claiming opposite things, and I have a feeling that if I had originally come across those two reviews there, I still would not have ordered the torch online. As soon as that first review was posted, anyone who looked at it either avoided ordering the torch or wasted time trying to understand it, and even with my comment there, the result is not as good as if the first comment were never there.
My fourth and final reason for thinking bad comments would be damaging to Pololu’s web site is more personal: I am trying to create good content and a useful resource for other engineers and future engineers, and hosting a bunch of crap on our site would be contrary to that mission. I have had arguments with customers who cited user comments from other sites when I was trying to explain why some of what they were doing was poor practice. There’s no shortage of places on the web for people who don’t know what they’re talking about to spout their ill-informed nonsense, and unlike CNN, I don’t think it’s okay to have a bunch of worthless content on my site just because it’s easy to tell that it’s worthless.
So, why am I so sure the comments would be bad? One simple way to see it is that if the comments contained anything good, we should put them in the main content on our site, making the comments redundant at best. From this argument, we can see that the value of a comment system is dependent on how much effort we put into maintaining a web site. If we put up minimal information and never modified it, comments could be better than nothing. But, that is not our situation, so let’s consider some individual claims of comment system proponents, starting with our customer’s message: “Makes it easy to review good/bad about a project and ask a question not in docs.” I think this sentiment is fairly representative in that it shows three basic desires: easy evaluation of a product, access to more information than is available in the (non-comment) documentation, and interaction with others, possibly including the site operators.
There’s almost no limit to the alternatives consumers have ready access to these days, so it’s understandable that they look to the advice and judgment of others to help evaluate their many options. In many cases, going with what works for others can be a good strategy, and that’s the main reason user reviews work at all. However, the “go with the crowd” approach depends on relatively consistent application scenarios and reviewers who have a good understanding of the material or are representative of the next user. In other words, for a review to be useful to me, my intended use for a product has to be similar to that of the reviewer, and my general background (as it relates to the product) needs to be similar to the reviewer’s. That turns out to be the case for most consumer applications, where the intended use of the product and the target customer is well defined.
For instance, for most customers, the expectation of a set of speakers is roughly the same (assuming they’re not some fancy audiophile-specific version), and a bus driver or professional golfer could be fairly representative of a typical user: after actually using the speakers, they could know about as much as there is to know about them for basic use. Reviews of speakers are mostly going to center on whether they subjectively sound good and maybe some other aspects like ease of use. The utility of the reviews would quickly go down if every reviewer had a different application, with one talking about how easy the speaker magnets were to detach and another commenting about how well the cases work to scare off pigeons. And this is still assuming the posts offer accurate information; once the comments get flooded with questions and inconclusive answers, there’s no hope of finding good information. Yet this is the reality for customers using our components, which can be used in countless combinations for any number of applications, and the customer’s experience with the product depends on the details of the combination. Our motor controllers might be used to turn a turret, drive a robot, propel an underwater ROV, spin a fan, or power LEDs, all from a multitude of different power supplies and interfaced to all kinds of different microcontrollers or other control signal sources.
Another fundamental problem with reviews from different customers is that everyone has a different scale, and I’m not just talking about opinions like “good” and “bad”. Even if a person is being completely honest and consistent, his sense of “small” can be different from someone else’s. Many adjectives like “small” are relative to the problem domain, and it’s difficult even for us to maintain some consistent descriptions as our product selection grows.
Of course, getting to the consistent and relevant user-generated content is important for mainstream retail sites, too, and various sites have different approaches for trying to rate the ratings and evaluating the credibility of reviewers. However, many of these approaches boil down to hoping that a high volume of reviews can be refined into the “right answer”, which is usually some kind of overall numerical rating. For our market size, we probably do not have a sufficient number of customers to even get a statistically valid number of ratings, and the concept of a “right answer” is in itself misleading. Viewer reviews might give you an idea of whether a movie is worth seeing, but strangers’ views should not be very relevant if your choice is whether you should read a good book or watch a good movie.
An important final point regarding user comments simplifying product evaluation is that even if user content is relevant and accurate, it becomes bad if it is poorly organized. There are successful user-generated content sites, like Wikipedia and Stack Overflow, that do not just reduce everything to a single number. However, we see that those sites have far more elaborate content management and organization tools than just a chronological list of comments.
The second benefit people want in comments is more fundamental information (as opposed to opinions or judgments) about a product. Ultimately, the hope people have for comments is that they will quickly illuminate some crucial aspect of a product; unfortunately, for complicated products like most of the ones we carry, there are many important aspects, and the likelihood that some detail you care about will be in a handful of comments or that you will find it in hundreds of comments gets vanishingly small. If there is fundamentally a lot of material to consider, an organized book with a table of contents and index is going to beat a pile of post-it notes.
As I said before, if the web site operator does not put effort into maintaining the content on a product page, the pile of post-it notes is better than nothing. For a large reseller like Amazon, it’s not just about effort: there is only so much we can expect its employees to know about a product, and if a product has thousands or millions of users, it is reasonable to expect those users to develop more expertise than the people maintaining the page. In that situation, the web site editors do not have the expertise to edit the metaphorical book, and we are left to make sense of the notes ourselves.
That is not our case, at least for the majority of our products. Our product pages and documentation are made by the people who design the products, and we do have the expertise to modify and maintain the pages as necessary. Rejecting a comment system therefore represents our commitment to providing good content, rather than hoping someone else will happen to do it for us. Please note that this is not some claim of infallibility or omniscience on our part; I’m just saying that if there is a mistake or missing information, it’s better to fix it directly than to make people rely on the comments.
The third common appeal for comments centers on terms like communication, feedback, and community. Once again, my problem is not with those goals; rather, my point is that comments on a product page are a poor means of achieving the goal. I think it’s quite easy to reach us: our phone number is on every page, along with a link for giving us feedback. If you call about a product, chances are that you can talk to one of the people that designed it. If you email, you should get a reply within a day. Our engineers also constantly monitor our forum, where customers can help each other and where we do not have any restrictions on linking to products from other vendors. Participating in discussions is also a good way to learn (both for us and for customers), but looking at a bunch of conversations is not an efficient way to learn if you don’t know ahead of time if the conversation you are looking at is a good one. Keeping around a log of outdated or otherwise irrelevant comments makes the likelihood of finding a good conversation lower.
My objections to comments might seem more valid if we consider some actual examples. Besides citing SparkFun, the customer was specifically interested in one of our stepper motor driver carriers, so I took a look at the page for a comparable SparkFun product. (The product actually got replaced since I started this post, but I’ll stick to the older version since the newer product’s comments, though also generally useless, might not be representative of what accumulates.)
SparkFun’s EasyDriver has over 100 comments, which probably represent more than 80% of the content on the page. If I am a user of the product, is it my responsibility to read all of the comments? The first several discussion threads look like they are discussing mistakes in the design that are probably no longer relevant or are separately called out in the product description. Then there are several comments praising the designer (“yeeeeeeeeeeesssss!! I love you!!” is the entirety of one post). Some comments are from customers that do not have the board yet. There are some comments about the parts being out of stock back in October. There are a bunch of unanswered questions. Need I go on? (By the way, the point here is not to pick on SparkFun, but rather to illustrate what I think is the inevitable outcome of user comments on sites like ours.)
A final aspect of comment systems that I want to discuss has to do with integrity or transparency. I have heard claims that comment systems increase trust in a site and that allowing public criticism of products shows humility or willingness by the web site operator to admit mistakes. While I am a fan of those two qualities, I am skeptical about the integrity of the execution of such systems. Unless a web site operator allows for absolutely anything in comments (which would lead to even more junk on it), some lines need to be drawn. Outright spam might be out. What about links to competitors’ products? What about comments that are of the form, “This product sucks because X” when X is objectively false? Ultimately, the web site administrator has to end up micromanaging the comments, and thus potentially changing the picture they portray, or accept that the comments will become a disorganized trash heap of inaccurate, misleading, and otherwise unsavory content. My impression is that the latter outcome is the more common one, with the mechanism that was supposed to lead to increased communication and accountability instead leading to indifference and even abandonment of the basic goal of providing the customer with good content.
My views on product comments might change as we continue offering more items we did not design and as I hear back from other customers. And of course, part of the point of this blog is to welcome comments: what do you think?
14 Comments New Comment
25 Jan 2011
Couple ironic things (for what it is worth)
1) I am "williams" in Sparkfun comment you posted :-)
2) The comments actually closed the deal for me to buy the driver.
I am coder, but newbe to hardware side of things. I received a Fez couple months ago which made me dive into the hw stuff. Have a box full of boards and tools now. Anyway.
I think it reduces down to a simple truth - Knowledge is good, and lack of it is bad. Everyone will buy something here for the first time and can be unclear about xyz. If they are unsure about how it may work for their needs, they may go away or not attempt it. Knowledge also Scales and Plays itself forward. Growing knowledge and Community is always good.
Most product pages (these days) have some form of comments. Sure, they can be abused - that will just be part of it. The good will outweight the bad. As logins are required, most people will play nice. Obvious issues can be moderated. The SparkFun comments do need some usability work (i.e. collapsed), but still find them very useful. Unless you bought the product before, clicking Buy always has a bit of unknown risk. You want to limit that if possible. Sales are good.
How much time would save someone to see a comment:
"If your motor amp limit is .33, your VREF will be .132 using this formula VREF = .33 * (8 * .05). Use MM from VREF to ground while adjusting pot..."
I don't see it as any kind of trick to keep people on a product page, it is about pre-sales knowledge. Couple quick lines from experts is worth a thousand words.
BTW - I got the EasyDriver first, but had some issues with it. Could be me or maybe I burnt something, not sure. But your stepper works like a charm and has the extra 16th microstep.
A question I want to see answered in the comments is "can you remove the motor leads while the power to board is on?". You will fry the easydriver that way. Same with the A4983 stepper motor driver?
Just some thoughts from a customer. Cheers!
26 Jan 2011
after reading all of this, i still gotta say, i agree with William.
but i almost didnt write this.. after all... you really dont like comments...
26 Jan 2011
Ha, that's an amusing coincidence.
Disconnecting powered stepper motors is usually a bad idea, and we've added a notice to that effect on our driver pages. By the way, changing connections while power is applied is generally bad even when you're not talking about stepper motors.
I think your reduction of comments to "Knowledge" is the logical step I disagree with; the ideal is not borne in reality. There is no shortage of information out there, and combing through comments on product pages has to be one of the worse ways of acquiring knowledge. Datasheets, web sites, and books by people who know what they are doing will have a much higher density of the "quick lines from experts" you're looking for.
2 Feb 2011
Hi Jan. I was not refering to general knowledge. But specific knowledge of the product. Things that are not addressed in the datasheets are many. Are we supposed to somehow just know everything? Like, what screws you use with that motor? Why I fried that thing backwards. Pot direction mislabels. Need any other product to hook that up to my IMU, etc. I mean come on. Are you pulling my chain here? Where else would misc info live to be searched in the first place? You want me to search Bing for comments that don't exist on your product page or forum? Look at it not from an experts perspective, but as someone looking to buy a product for the first time. Even if you did manage to add everything in docs someone had a question on, that does not address the questions people will not spend time to ask. They can't find the answer, so move on. You also don't know what you don't know. Community is what drives products.
2 Feb 2011
I am not pulling your chain; let's consider your examples:
"What screws you use with that motor" or "pot direction mislabels": This kind of basic info should definitely be in docs and not relegated to comments. I believe you can find this level of answer in comments, but the answer existing there should not be considered a virtue: it's an indication of poor documentation in the first place.
"Why I fried that thing backwards" and "need any other product to hook that up to my IMU": This *is* general knowledge, and it's also not likely to be covered well in comments.
I am not convinced of the existence of this intersection of "misc info" that is not general knowledge yet specifically relevant to a product but also should not be in the docs. I think the forum does capture the misc info aspect; if your quest is for knowledge, why do you care if you're searching in product comments or in forums and official docs? Do you at least agree that having all the info in organized documentation is a better goal than having all the info in comments?
I don't understand your statements toward the end. How would comments have answers to "questions people will not spend time to ask" any more than any other format?
15 Jun 2011
LOL I deleted my post about suggesting a forum because when I went back up to the top of the page I saw the forum menu item LOL
so questions should be asked and answered in the forum - no comments - I think that works for me :)
Long time pololu customer
27 Aug 2011
I have been a long time customer, but unfortunately, after reading this I have lost quite a bit of respect for pololu because all the arguments against having comments are based on an underlying assumption which is not explicitly stated. This assumption is that the reader of the comments is an idiot who cannot spot a fake review or quickly sort through the junk to find the useful pieces of information in the comments. It is also rather selfish and backward looking (if it is not good for 'us', we reject it). Folks, it's about engagement and interaction, not hiding and protecting.
I use sparkfun quite frequently and no matter how much junk exists in the comments, there is always a few posts that provide unique information. And my opinion of sparkfun (and pololu for that matter) will not change just because some idiot posts a "this controller is junk, I tried it and it doesn't work" comment. Again, most people who would be shopping in these sites can tell better, and they also deserve a bit more respect from pololu.
I for one would love to see comments, and even having pololu staff provide some product specific feedback using that context.
Defensiveness and protectionism does not make a progressive company!!
29 Aug 2011
I am disappointed in your take, but I appreciate the time you took to provide the feedback, so I want to address each of your points.
I know that I don't think of potential comment readers as idiots, and it's been a while since I wrote this post, so I just reread it. I don't see the evidence to back up your claim in my post. You say it's not explicitly stated, but it's just not there. I gave the example of my own experience, where the intelligence of the reader is irrelevant when the information in a comment is just flat wrong. I talked about beginners, but that has nothing to do with being idiots or not, and I stand by the claim that comments can do those kinds of comment readers a great disservice (I have seen several cases of it).
Regarding your selfishness claim, I don't think any business would be well-served by embracing things that are bad for it. I think comments on a site like ours are bad compared to the available alternatives, and I think you're deluding yourself if you somehow think that Sparkfun thinks that comments are bad for them but that they tolerate them out of some kind of altruism. Sparkfun has made the calculation that on the balance, comments are good for them; I have presented my reasons for thinking otherwise. I don't understand the "backward looking" part.
I think I specifically addressed your "it's about engagement and interaction, not hiding and protecting" sentiment: those goals are fine, but comments on a product page are a bad way of achieving them. I think it's pretty easy for you to interact with us if you want to, but you should not have to just because there isn't decent documentation in the first place.
Regarding the occasional useful information in comments, I think I addressed that, too: the comments are not a good way of organizing and presenting information. I suspect that most competent readers digging through a hundred comments that are 95% junk do it because they are at a desperate point where that is a last resort. I think it's disrespectful for a web site to expect me to dig through a bunch of assorted comments rather than to just present the information well in the first place.
I do appreciate that you want the comments, and I suspect that on some level, it's independent of the arguments of if they are good or bad. I am genuinely interested in what others think, and I think I am open-minded enough about it that I could change my mind. In this case, though, you're speculating about an unstated (and untrue) assumption rather than arguing any of the points directly.
Your use of "context" caught my attention. In some sense, my argument is that there is a place for official, organized information and a place for informal discussion (e.g. the forums, this blog, etc.). What is special about having the discussion on the product page as opposed to in the forum? What if there were a dedicated forum thread for each product?
I don't know what to make of your last statement since I do not know what you mean by "progressive company" and therefore do not know if being a progressive company is something to strive for.
14 Jan 2012
Comments are the gasoline of the Internet. In one way, they make it run. But on the other hand, they produce noxious fumes and pollutants and all one has to do is drop a match close by to set it all alight and ruin everything.
Which is a long, over-flowery way of saying "I agree with you".
1 Feb 2012
You need comments. Just install a moderation system, and erase all the crap - heck - you've got zillions of fans out here in internetland guys; I'm certain you'll find some more than willing to keep an eye on all new postings, and remove all the dross, all for free.
Comments often tell folks what they need to know pre-sales, as well as post-sales, and provide excellent food for SEO.
Do it. You won't have any regrets.
22 Aug 2012
All the respect, and a little more, that I lose because I similarly assume that your action is more "backwards" is balanced by the solid arguments you offer for it. And my positive experience with pololu products. If you're going to make this decision (as you did), then I think the rationale in your post is the best way it could be justified.
That said, there are some simple ways to get around a number of your problems. You can have a link to the "product page on the forum" or otherwise a more hidden comment section, for the major concern of not wanting irrelevant content to be a majority of the product page.
I would reiterate what other people say, that when I glance through comments I realize they're mostly full of crap, but I often find quick answers to questions that aren't major enough to ask a question or post about. These days I more often comb through datasheets & application notes because I want a more thorough understanding, or because what I'm working with is more complicated, but when I was starting out it was very helpful to get the quickstart comments from other users, and not from complicated datasheets. I was also led towards several tutorials and libraries that I could adapt, because of the comment system (on other sites.)
If you ever revisit the idea of comments, remember that a simple prompt reminding people of certain norms can strongly influence the character of the comments. E.g, having a reminder that "the comment system is for constructive remarks, make sure to be clear as to what you're saying and why you suggest it. We may respond to or moderate comments that address flaws in the product that have since been revised" [not the best worded reminder, but the general idea is contained.]
(thanks for everything!)
P.S, the product search engine could really be spruced up. I always find myself going through the category sort, because the product search is not very effective unless I have a part/chip number.
25 Aug 2012
Thank you very much for your feedback. At this point, I think that if I could just flip a switch and have comments in a way that would not be that objectionable (for instance, with some of the suggestions you had), I would turn that feature on. The way we're organized right now, though, our few web programmers are also involved in product design, and I would rather spend our time on new products or on more important features for the web site (such as the search system).
9 Oct 2012
Meh, add a link from the product pages, to a support category in the forum for that product. Issue solved.
(I'm actually here, re-reading your servo stuff, because I'm thinking of using them as introductory information for someone.)
26 Nov 2012
You might also consider that distributor sites that professionals rely on every day, such as digikey.com or mouser.com, do not have product comments. What they do have is good parametric search, which is something that your site now also has.
Another commonality between you and Digi-key is that what you sell, requires skills and research to use, is well documented, and works as documented. On Amazon or Newegg: not so much!
Btw: I think you allow open discussion in your forums. Good enough for me! A link to the appropriate forum section for each product would be icing.