----------------------------------------------------------------- Note: you can also find this text at http://se.ethz.ch/~meyer/publications/wikipedia/wikipedia.pdf with better formating. ----------------------------------------------------------------- _____________________________________ Defense and Illustration of Wikipedia _____________________________________ Bertrand Meyer 6-7 January 2006 This note, more precisely Part One, was prepared in late December of 2005 as a reaction to a critique of Wikipedia by four well-known computer scientists. Their article highlights, Cassandra-style, all that can go wrong with the Wikipedia concept of an encyclopedia produced by an iterative "community process" allowing anyone to edit any entry. A scary prospect indeed. As my critique of their critique acknowledges, it's hard to find fault with their cogently argued analysis -- except for one detail: in its eagerness to paint a theoretical picture of Wikipedia as a disaster waiting to happen, it skipped checking the real Wikipedia, which would have revealed that the disaster has not happened, A more pragmatic look at Wikipedia as it exists today indicates that the project, while perhaps not living up to the hype of its most fervent promoters, has become a superbly useful tool for Web-based fact-finding. The original critique suffered, in my opinion, from a half-empty-glass perspective, and from a misunderstanding of Wikipedia's role in the world. Without in the end disagreeing fundamentally with the authors' analysis, I take a half-full-glass view, based on a different understanding of what is Wikipedia's competition: not the traditional professionally produced encyclopedias, but the legions of sites that, springing up all over the Web, purport to contain answers, unverified and often unverifiable, to every topic on earth. Against that standard, Wikipedia is a resounding success. That's the analysis I produced in December, based on an assessment of what Wikipedia is, not what it could degrade into. You will find it in Part One. Because of other commitments I had to put aside finishing it up; but a rather personal Wikipedia incident in early January, recounted in the second part, gives enough reason to get back to it, and an interesting twist to the first. ________________ Part One: Before ________________ 1. SURELY THIS CAN'T WORK! -------------------------- By now few people need to be told what Wikipedia [17] is: an online encyclopedia, with close to one million entries for the English version in early 2006, plus versions in a couple dozen languages, all growing quickly and sharing a remarkable feature: anyone can, from any browser, edit any entry at any time in any way he pleases, including of course amending or outright canceling a previous edit. If you're just given this description, the natural reaction is to shake your head in disbelief. Surely this can't work. The result will be awful. Bad scholarship, bad language, endless edit-counteredit wars. The whole thing will be a mess. That's the view I have heard from several distinguished computer scientists, and the one argued by Denning, Horning, Parnas and Weinstein in a reasoned but strongly critical comment on Wikipedia published in the venerable Communications of the ACM [1]. DHPW (I'll call them that way from their initials) cite a number of dangers, from contributors' possibly dark motives to lack of guarantee of accuracy, which can "pollute enough information to undermine trust in the work as a whole". If we were five years ago, just before Wikipedia started [17], and you presented me with the concept, I would have agreed. The 15-year experience of Usenet, this free-wheeling global discussion forum where the postings of cranks have exactly the same value as those of Nobel prizes, and cranks always win in the end because they have the time to continue posting, would have been enough to convince me that "community-based processes" can't succeed on a large scale. Surely, as "DHPW" argue, the Wikipedia process can only mean impending doom. In this they are right, but after five years if doom is still impending it's at least been suspended. Wikipedia, defying all the smart forecasts, has become a key tool for hordes of Web users, and there is no sign that it leads to a decline in the quality of Web information -- rather, as I will argue, the reverse. The most striking feature of the DHPW criticism -- striking in particular because the article warns of the risk that Wikipedia contributors will "supply speculations" in lieu of facts -- is that it is hypothetical. It doesn't directly present any example, such as a wrong Wikipedia entry, of the risks it cites. (The only concrete reference to a specific Wikipedia article is to an entry about one of the authors, which he complains he can edit himself. Shouldn't he instead be glad that he can correct misstatements?) 2. SUMMONING THE EVIDENCE ------------------------- General principles and in abstracto warnings are good, but it doesn't hurt to look at the evidence. To anyone who uses the Web routinely for writing, fact-finding, research and other intellectual pursuits, as I have done for a decade now, the evidence is overwhelming: Wikipedia is, ever more often, the appropriate source of information. I could cite hundreds of examples, but let me take just one. A few weeks ago I wanted to get up to date on a concept I had never studied formally: Levenshtein distance, also called edit distance. It's a beautiful algorithm to find the "distance" between two strings, defined as the minimum numbers of operations -- add an item, remove an item, or substitute an item -- that will turn one into the other. I first fetched from my shelves the Song of Songs of algorithms, Knuth's treatise [8]. Tough luck, Levenshtein is not there, and the author confirmed to me later that it will be in volume 4, but alas volume 4, now appearing in fascicles, hasn't reached Levenshtein yet. So I turned I also noted that the loop invariant wasn't quite strong enough, and started a short correspondence with the entry's author, Derrick Coetzee. He mentioned that he was not sure his discussion proved minimality; indeed it doesn't, so I looked at a possible minimality proof. We haven't updated the entry yet -- so little time... -- but will. In the face of such an experience, repeated almost daily, the theoretical admonitions of DHPW sound somewhat academic. 3. A TALE OF THREE PRESIDENTS (VICE AND FULL) --------------------------------------------- Does the irrefutable evidence of Wikipedia's usefulness mean DHPW are wrong? No. In fact the six dangers they point out -- Accuracy, Motives, Uncertain Expertise, Volatility, Coverage, Sources -- are undeniable, and one could most likely find specific Wikipedia entries to illustrate every one of them. In fact, while the article itself, as noted, includes no examples (which can be justified by its brevity), it cites two references that do. One of them [9] is particularly interesting. Written by Robert McHenry, former editor-in-chief of Encyclopedia Britannica, and titled "The Faith-Based Encyclopedia", it delivers a much more scathing critique of Wikipedia than DHPW. Dissecting the entry on Alexander Hamilton, McHenry points out among other deficiencies the lack of a mention of the uncertainty about Hamilton's birth date -- 1755 or 1757 -- and further notes that values given for his age at various times are sometimes computed from one date and sometimes from the other. He dismisses the entry overall as a paper that "might be expected of a high school student, and at that [...] a C paper at best" (a reference to the grade, not the programming language). He also notes, by looking at the version history, that the original Hamilton entry was tighter and more cogent, but that through successive revisions "th This is useful and well-substantiated criticism. But because it is based on concrete evidence and not abstract generalities it lends itself to objective examination. Checking the Hamilton entry a year and two months after Mr. McHenry, I notice that -- at least to someone who knows little about the topic -- it looks far better than how he described it then. No doubt this was due in part to his article; in fact the version history shows that his most salient points, like the 1755/1757 mess, were corrected on 15 November 2004 -- the same day his article appeared on the Web! You could argue that it's a special case because the article was widely publicized, but I prefer to look at the positive aspects (the half-full view): * First, imagine the time it would have taken, to correct such a deficiency, even a widely publicized one, in a traditional, refereed encyclopedia -- and the impossibility of removing erroneous copies in circulation. No one, by the way, should assert such errors can't occur there. Shortly after the incident reported in Part Two, Hartmut Scheible from the University of Frankfurt wrote to me and to the Spiegel that editions of Microsoft's Encarta, German version, have for years carried an entry about the German writer Ernst Jünger, stating that Jünger was visited on his 100th birthday in 1995 by the German Chancellor Helmut Kohl and "the Israeli President Chaim Herzog". Well, Israel did have a president of that name, but - as you can check on the Web! -- his term ended in 1993, and the suggestion that he would have traveled to Germany just to meet J ünger raises eyebrows (Jünger was stationed in Paris during the war, where he was a staple of the city's intellectual life and behaved more decently than some * Coming back to Hamilton in Wikipedia, the page's history shows constant and extensive improvements since the time of McHenry's criticism. This is consistent with the experience of many other pages, there for anyone to see. I have, on the other hand, also seen more than a few Wikipedia entries of mediocre or bad quality. That's, I think, the main difference with traditional encyclopedias. The best Wikipedia entries and even the merely good ones are at the professional level, or quickly reaching it. The bad ones can be far below the professional standard. C student papers, or D minus. 4. THE TRUE POINT OF COMPARISON ------------------------------- The last observation highlights what I believe is a misunderstanding by the DHPW paper and other criticism such as McHenry's. They are not entirely to blame, since the misunderstanding is fueled by the Wikipedia community's own over-representation of their goals and achievements, including lofty pronouncements cited by McHenry: Wikipedia is intended to become "the largest encyclopedia in history, both in terms of breadth and depth and also to become a reliable resource" (McHenry takes delight in this "and also"); the process implies that "incomplete or poorly written first drafts of articles can evolve into polished, presentable masterpieces through the process of collaborative editing" (the kind of statement that McHenry dismisses, not entirely without ground, as "faith-based"). But a comparison to professional encyclopedias or other refereed publications is, in my view, the wrong one to make. I don't think it's offensive to Mr. Coetzee to conjecture that Knuth's treatment of the Levenshtein distance algorithm, when it appears, will be far superior to the Wikipedia entry. If -- perhaps because a good Wikipedia entry caught my fancy -- I get excited about Alexander Hamilton, I will consult the Encyclopedia Britannica or, more likely, I'll buy a book by an established author. (See note 1 at end of text.) On the other hand, if I want to check who was president of Germany in 1995, I can save a trip to the library or the bookstore, and chances are Wikipedia will help me (it did). When I choose to go to Wikipedia, the competition is not certified, refereed sources, which I will use in other circumstances. The competition is non-Wikipedia Web pages. Millions of people, like me, use the Web for fact-finding and fact-checking. Prior to Wikipedia, and ever since 1993 or so, the Web has become a primary source for finding out answers to simple but specific questions. What 8-bit extended ASCII character has the same code that Unicode uses for the "British pound" sign? If I didn't have Wikipedia, I wouldn't buy a book. I would go to the Web, and I would find some answer. But how good an answer? With Wikipedia the quality is overall much better, precisely thanks to the community-based process that the Wikipedia critics deride. (See note 2 at end of text.) 5. DUBITO, ERGO SUM ------------------- In using Wikipedia, like other Web pages and indeed any source, the key rule -- which, to my surprise, the critics don't seem to emphasize -- is never to forget the usual principles of objective enquiry: since when are we supposed to trust everything that we read, printed, electronic or otherwise? A critical approach to the source documents is one of the first things that we teach, or should teach, to our students. As anyone who has spent more than 15 minutes browsing the Web knows, the general credibility level of what you find there is far lower than in most of the other sources of information one can access, with only three attested exceptions (horoscopes, US talk radio, and White House communiqués). If you are looking for information and really need it to be accurate -- you usually do! -- you must, on the Web, have your defenses up all the time. For example: * Try several Web pages * Check whether they corroborate each other or not. * If they do, check that they don't just parrot some common original source, as is often the case, but were derived truly independently. I always pay attention to the overall quality of a page: numerous spelling mistakes, psychedelic colors, popups that spring up from all over are as many reasons to decrease my confidence level. To check the dates of German and Israeli presidential terms after Prof. Scheible's message, I didn't just read the corresponding Wikipedia entries but went to a few other places as well. I have learned to associate a likelihood rating with Web results. For example I have little doubt that I got the right dates for the presidential terms of the two Herzogs; on the other hand, while I am perhaps 99.8% certain that the visit to Jünger was by Roman and not by Chaim, at the time of writing I would check a really authoritative source (after all, politicians will sometimes do strange things) before betting a large sum on Encarta being wrong. If you take that skeptical view -the very basis of any rational endeavor -- what becomes most striking about Wikipedia is not DHPW's list of undeniable risks; it's not that Wikipedia is sometimes wrong or inept; it's that Wikipedia is so often right, competent and useful. Critics like to point out the entries which cause repeated edit wars, usually affecting pages on hot political or religious topics. They are a real problem, and the Wikipedia community will have to find solutions for these -- it's trying to -- but they really don't affect the fundamental picture, at least for professional use. Who would seriously go to a Web site for definitive answers about such issues? Anyone who succumbs to outright manipulation of Wikipedia entries by activists on these topics would probably have fallen prey, sooner or later, to some other manipulation. I have not yet seen, on the other hand, a protracted edit war on Levenshtein distance or 8-bit extended ASCII. I did see extensive, reasoned arguments in the discussion pages of entries for advanced technical topics such as "abstract interpretation" (a program verification technique), where some people criticized edits listing a particular vendor of related software tools, and others justified these edits as relevant. It's interesting to see how contributors who are tactless, a little less than fully honest or simply a bit over-enthusiastic get civilized by others over time. As to the real crooks, they can hide behind an IP address, but only so long. You are not as anonymous as you think, at least not forever. 6. THEORETICAL ADVANTAGES ------------------------- Even if we rise above the evidence and limit ourselves to the abstract, hypothetical level of the DHPW article, we'll find some advantages of Wikipedia over traditional encyclopedias. Three come readily to mind. The first is breadth: what encyclopedia in print or CD has entries on (to take one example among countless ones, from the English Wikipedia) "20 Minuten", the free tabloid-style available every morning in Zurich trams and trains? The second is internationalism. I remember being shocked, one of the first times I used a US dictionary (I have forgotten which), to see Picasso listed as a "Spanish painter" and Einstein as an "American physicist". Surely the criterion, whatever it is - country of birth, of major achievements, of citizenship -- should be the same in both cases. Such instances of petty chauvinism would quickly be corrected in a medium that by nature has a broad international cadre of contributors. The universal editability that puts off critics has many positive aspects too. Being the kind of person who notices typos (alas, in other people's publications more effectively than in mine), I like to correct them. When I see a small error in an otherwise well-written entry -- and am absolutely sure that it's an error -- I often take the time to fix it. I like the feeling of doing one's own bit, ever so tiny, in the endless global fight to minimize entropy. I can mark an error in a book, but only in one copy (I confess I have done it on library books, which even with a pencil may be bad citizenship); I cannot fix a typo on an ordinary Web page. Even the editability of a page that talks about you, which so scares DHPW, is often preferable to alternatives. If a newspaper publishes something wrong on you -- and, to put it politely, not all journalists always get all details right -- don't you wish you could change it? You can't. On the other hand, I noticed a few months ago that, like one of the DHPW authors, I have an entry [15] in the English Wikipedia. I was amused and flattered, especially since I have no idea of the authors. I noticed an obsolete piece of information (the mention of a position which I no longer hold, but my own Web page used to list). I was tempted to correct the text, but thought it tacky to edit one's entry unless for something really important, so I let it go and never mentioned it to anyone. A few months later, I checked the page again out of curiosity, and sure enough the information had been removed. That was pretty amazing. 7. WIKIPEDIA AND ITS ROLE ------------------------- Nothing above proves DHPW and the other critics wrong. They have a point. The very success of Wikipedia raises the public's expectations. In spite of all warnings, people will attach value to what they find there. The risks are there, and the people driving Wikipedia would be well inspired to address them. I believe in particular that the development of a rating system for both authors and entries is inevitable. Authors should have the ability to prove their skills through successive achievements, as in some open-source projects. Perhaps even more importantly, there should be a way to certify an entry as it existed on a certain date, the rated snapshot being preserved on the entry's page and non editable. Some kind of refereeing process should also appear. All this, I think, will happen. But even before then it would be a blatant mistake to deny the great role that Wikipedia entries play, as a better kind of informational Web pages, to help us day to day in the search for truthful information. If you look at Wikipedia today, that's what it is, and in that role it is tremendously successful. Will it go further? The Wikipedia ambition seems to be to replace traditional reference books. The most systematic assessment so far is a study in Nature [4], which applied a standard peer-review process to entries in both Encyclopaedia Britannica and Wikipedia for accuracy and more generally for quality. If you are a Wikipedia advocate, you'll find much encouragement in it, since it found errors in both works, and concludes that Britannica is still ahead but not by much. On the other hand, Peter Denning (the D in DHPW) points out that "Nature concluded Wikipedia is 'pretty good' for science. Others think that a 30% difference in error rates is significant." He is right; 30% is scary. Another lesson from the Nature results is that if "community-based processes" have their limits peer review is not a guarantee of quality either. The tale of the two Herzogs confirms this, as does the publication experience of many researchers (I must not be the only one to note that there is often little correlation between how easy it was to publish a scientific paper and how well received, cited, etc. it was after publication). The recent Hwang cloning-research debacle should suffice to remove any remaining illusion that peer review is infallible. Still, there hardly seems to be a better strategy, when the aim is to reach the best possible result overall, than to have new work assessed by the recognized experts in the field. For that reason I don't think that efforts such as Wikipedia will ever displace professionally edited works of reference. The comparison with open-source software development, hinted at above, is telling. In an article of a few years ago [10], which didn't make me any new friends in the open-source community, I argued that open source is (whatever its more wild-eyed proponents may say) not an ethical choice but a business model. It's different from the commercial model, neither intrinsically better nor intrinsically worse, but adapted better in some cases and less well in others. I would suggest the same attitude towards "free" vs traditional reference publications: sometimes you need one, sometimes the other. They will both find their place. Wikipedia has already carved an initial one for itself, and it's a pretty impressive place. _______________ Part Two: After _______________ AND NOW FOR SOMETHING A BIT DIFFERENT ------------------------------------- Part One -- now updated a bit as a result of recent emails -- is what I had until a week ago. Writing on 6 January 2006 gives it a new perspective. It turns out that along with the English Wikipedia as mentioned above the German version also had an entry [16] on me, although I didn't know about it. Three days ago, Christian Kirsch from Heise.de, one of the major German online publishers, looked up the entry (as he explained to me in an email of this morning, "We are gathering basic facts of 'important' IT persons, and I was looking for your birthdate") and discovered that the entry had been edited to report that I died on the preceding 24th of December. The edit had been there for four days until he noticed it. (Four days! That's the really humiliating part. You'd think people would look me up on Wikipedia, even just German Wikipedia, far more often than that. Especially during the holidays! Or maybe they did, but weren't surprised as they thought I'd long been dead already?) After calling me and reaching my secretary Claudia Günthart who confirmed that although away in California I had given recent signs of apparent life, technically known as "emails", he immediately wrote a story about the affair [7]. He contacted the "Wikipedia press people" who promptly corrected the entry -- but not before he had taken a snapshot of the page, a wise step since in the following days some Wikipede removed the spurious version from the version history, so that there is no trace of it on the site. (The discussion section contains an extensive discussion of the matter, and some criticism of that decision by other Wikipedes who argue that the appropriate action was just to cancel the edit on the page, leaving it in the history.) Mr. Kirsch was later kind enough to send me a copy of the snapshot (see [1]). The Heise story was quickly picked up by the online site of the Spiegel, the best known German news magazine [3], from there by one of the main Zurich dailies (Tages Anzeiger) and by a Swiss news agency, out of which it spread throughout the German-speaking world. In particular all the Zurich dailies had it. Yesterday, I was told, it was on the morning news on the radio. As of this morning the Google search ?Bertrand Meyer? Wikipedia tot [6] (tot is German for dead) yields 170 hits, and growing. The main French-language Swiss papers have started reporting it too, although I haven't seen any mention yet beyond Germany, Austria and Switzerland. (As an aside it's fascinating to follow the spread of such a quickly exploding story, with only a handful of original sources, chief among them here Kirsch's story, and the others parroting previous ones, sometimes adding details out of nowhere -- for example that I was said to be in a car accident, whereas the faked entry didn't give a cause for my regrettable pass Some press comments are interesting. Many journalists went into speeches about the fundamental unreliability of Wikipedia and - again following Kirsch's lead -- related the incident to the recent story of John Seigenthaler, whose English Wikipedia entry was edited to relate him to the assassination of John Kennedy, an edit that its author, when caught, described as a prank, but which Mr. Seigenthaler apparently didn't consider funny. In reading the German articles I have the impression -- I can't point to more than an impression -- of a kind of almost proud "me too" attitude: it's not just the Americans! We are hip and networked too! We have our own Wikipedia scandal! Journalists called up my office a lot; they called the ETH head of public relations, Rolf Probala, who told them (correctly) that I was in the US, and (correctly too, but I think it was a wild guess on his part, because with the time difference this must have been before he contacted me) that I was taking the matter with "sarcastic humor" and The whole thing, as the press reported, looks rather obviously like a student's joke. The edit, using European-style dates, stated the details of my death as "According to the latest reports, Bertrand Meyer died on 24.12.2005 in Zurich. On 23.12.2005, exam results were published; links between that publication and his death couldn't be confirmed". OK, it's not too tasteful; I have seen funnier student jokes. But I have seen less funny ones too. I've also on the Internet seen stuff that is not funny at all. A few years ago a number of well-known people in the object-oriented community were the target of vicious and gratuitous slander, on newsgroups and Web pages, by a well-known crank (he was posting anonymously but everyone knew who was behind the messages), accusing us of various vices and misdeeds; some of them, for example, were faked confessions of alcoholism, purportedly signed by one of his targets; they initially seemed so obviously ludicrous as to cause little more than a shrug, but in fact were crafted well enough to appear genuine to some naïve readers. Those were not jokes. And there was no way to edit them! They stayed on for a long time, faithfully preserved in newsgroup archives, with no possibility of correction. Asserting that their sole role is to preserve snapshots of what once was, archive maintainers flatly refuse to remove even the most absurd libel, Wikipedia looks much more civilized: mistakes, mischief and misdeeds will occur -- what else can we expect? -, but the correction mechanisms are there. In the case of the student joke, good or not so good, it took a few minutes from detection to correction. Even if you consider the full period the page was up, four days isn't that much. So to Lauren Weinstein (the W in DHPW) who asked me, after seeing a first draft of this, "Would you feel less charitable about the editing of your biography if, for example, it had reported that you had previously been arrested for some truly reprehensible offense, and this false fact started circulating around?" the answer is obvious. Yes. Fiction for fiction, I'd rather be dead. But if someone is going to slander me horribly on the Web, I'd rather that he chose an editable medium. On a less ominous note it may be interesting to speculate what would have happened if, as planned, I had sent a condensed version of Part One of this note to the Risks Forum [12], where I first read about the DHPW paper; it would have been natural, then, to interpret the entry edit as a consequence. The sequence, it turned out, happened in the reverse order. In the end does this little episode change my view of Wikipedia as expressed above? No. I can't really see ground for any modification of substance. The system succumbed to one of its potential flaws, and quickly healed itself. This doesn't affect the big picture. Just like those about me, rumors about Wikipedia's downfall have been grossly exaggerated. ACKNOWLEDGMENTS --------------- I thank Lauren Weinstein, Peter Denning and David Parnas for perceptive comments on the first draft of this article, which I have tried to incorporate in this revision (without of course implying that they agree with everything in it). Christian Kirsch from Heise was not only the one who first detected the macabre prank; he had the good sense to preserve the record, told the world, and later helped me reconstruct the sequence of events. Andreas Leitner from ETH contributed useful comments and alerted me to the Nature study. I am grateful to Claudia Gü nthart and Luc de Louw for first alerting me to the incident, and (along with Ralf Probala) for keeping their cool when the press started to call, and guessing how I would react. Several other people also sent me useful notices. The German Wikipedia community was fast, effective and considerate in dealing with the issue. And I was particularly touched by kind words from a number of ETH students. BIBLIOGRAPHY ------------ 1. Anonymous: Spurious edit to "Bertrand Meyer" entry in German Wikipedia (see [16]), early January 2006. Available for the record (6 January 2006) at http://se.ethz.ch/~meyer/publications/wikipedia/page_hack_record.html. 2. Peter Denning, Jim Horning, David Parnas, and Lauren Weinstein: Wikipedia Risks, in Inside Risks 186, Communications of the ACM, vol. 48, no. 12, December 2005; available (5 January 2006) at http://www.csl.sri.com/users/neumann/insiderisks05.html. 3. Holger Dambeck: BÖSER SCHERZ AUF WIKIPEDIA -Professor wird totgeschrieben ("Bad joke on Wikipedia -- Professor is written off as dead"), in Spiegel Online, 3 January 2006. Available (6 January 2006) at http://www.spiegel.de/netzwelt/netzkultur/0,1518,393270,00.html. 4. Jim Giles: Internet Encyclopaedias Go Head To Head, in Nature 438, 15 December 2005, pages 900-901, available (6 January 2006) at http://www.nature.com/nature/journal/v438/n7070/full/438900a.html. 5. Google search for: "Defense and Illustration", available as http://tinyurl.com/7t4vc (last run 8 January 2006); see 6th entry returned. 6. Google search for: "Bertrand Meyer" Wikipedia tot, available as http://tinyurl.com/9ur8p (last run 6 January 2006). 7. Christian Kirsch (signing as "ck"), Wikipedia schrieb Bertrand Meyer tot ("Wikipedia wrote off Bertrand Meyer as dead"), 3 January 2006 at 9:49, available (6 January 2006) at http://www.heise.de/newsticker/meldung/67895. 8. Donald Knuth, The Art of Computer Programming, volumes 1 to 3 (out of 7 announced), Addison-Wesley. 9. Robert McHenry, The Faith-Based Encyclopedia, 15 November 2004, available (5 January 2006) at http://www.techcentralstation.com/111504A.html. 10. Bertrand Meyer, The Ethics of Free Software, in Software Development, March 2000; revised version at http://se.ethz.ch/~meyer/publications/sd/ethics.pdf. 11. Bertrand Meyer, Object-Oriented Software Construction, 2nd edition, Prentice Hall, 1997. 12. Peter Neumann (editor), The Risks Digest: Forum on Risks to the Public in Computers and Related Systems, available as the Usenet newsgroup comp.risks and, among numerous other venues, at http://catless.ncl.ac.uk/Risks/latest.html (accessed 5 January 2006). 13. Hartmut Scheible, University of Frankfurt, Leserbrief zum Artikel "Böser Scherz auf Wikipedia: Professor wird totgeschrieben" email to the author (a cc of a message sent to the reader's mail section of Der Spiegel as a response to [3]), 3 January 2006. 14. Wikipedia: at http://www.wikipedia.org. 15. Wikipedia (English) entry: Bertrand Meyer, available (5 January 2006) at http://en.wikipedia.org/wiki/Bertrand_Meyer. 16. Wikipedia (German) entry: Bertrand Meyer, available (5 January 2006) at http://de.wikipedia.org/wiki/Bertrand_Meyer. 17. Wikipedia entry: History of Wikipedia, available (5 January 2006) at http://en.wikipedia.org/wiki/History_of_Wikipedia. NOTES ----- 1 If you realize that the title of the present paper is a nod to a famous book but can't quite place it, a Google search will, at the time of writing, find it for you -- in the Britannica (see [5]). Wikipedia just doesn't measure up for that particular example. 2 Another problem with Web entries is stability. Before publishing a book [11] that extensively relied on references from the Web, I wrote to the author of every referenced page to ask for some commitment that it would stay up for a reasonable period. This is tedious, and not a guarantee anyway. One particular page raised a tricky issue: when they received my query the authors decided they wanted to keep the page private, and removed it from the Web. I could neither unlearn what I had learned from it, nor cite it (I ended up using the information as little as I could). With Wikipedia, there's at least some hope that the entries will persist.