Surveys in Debian (original) (raw)

This is my page trying to list surveys done by members of the debian project, which alsogives some pointers to adviceand why I care. Most of the survey discussions are posted todebian-projectand I try to contribute constructive criticism when possible. I'd welcome suggestions for the contents of this page.

Surveys by debian developers

1. bits from the DMs/NMs/AMs?, Paul Wise (2008-03)
2. The construction and deconstruction of the legitimacy of people and ideas in the Free Software Community - A study about Debian, Daniel Ruoso (2006-07, results, method, pt and partial en)
3. Poll of debian-user readers about the GNU FDL, Brian Nelson (2005-04, results, some method)
4. Debian usage survey, Enrico Zini (2004-04,results 2005-03)
5. The debian-women supporters survey, Amaya (2004-07, some results, method)
6. Strawpoll on proper usage of @debian.org email address, Michael Banck (2004-01, results, method)

Surveys on debian, rather than by developers

1. Academy of Management Journal, Oct 2007, Vol. 50 Issue 5, p1079-1106, 28p, Siobhán O'Mahony and Fabrizio Ferraro (summary by André Felipe Machado)
2. Managing Open Source Software as an Integrated Part of Business, Niklas Vainio (2006-01, results to appear?)
3. Political Motives of Developers for Collaboration on GNU/Linux, Tobias Escher (2004-07, results, method, non-free)
4. FLOSSPOLS Second Survey, FLOSSPOLS (2005-03, results to appear?)
5. YAQ - Yet Another Questionnaire, Frauke Lehmann (2004-07, results, method (in German-language dissertation), non-free)
6. Short Survey, Biella Coleman (2003-05, in dissertation)
7. Who is Doing it, Gregorio Robles (2001-07, results 2001-08, method, non-free)

Surveys on free software in general

1. Open Source Case Studies, IDABC - Interchange of Data Between Administrations (viewed 2008-03)
2. DebianInThePublicSector, WMartinBorgert and TorstenWerner (viewed 2008-03)
3. Free software in public administration, beta, Ciarán O'Riardon (viewed 2008-03)

I find it disappointing that non-developer surveys take a lot longer to publish results, only about half seem to publish at all, and non-free software terms like CC-nd, -nc and -by are common for ones which do. I have not listed projects which don't look like they'll ever publish.

Advice to surveyors

Because I criticise bad surveys, people expect me to give advice about how to write good surveys. I'm not sure why: there are more than enough elementary statistics texts which cover surveying far better than I can. It is not a subject which is hugely different for debian.

I like "Statistics Explained" by John Parry Lewis and Alasdair Traill, but I'm not sure how easy it is to get and it's rather big. Much easier to obtain is "Statistics Without Tears: A Primer for Non-Mathematicians" by Derek Rowntree. If you want something online, start withSome Aspects of Study Designfrom Gerard E. Dallal's "Little Handbook of Statistical Practice" and follow the link to the full thing when you're ready, but I strongly suggest getting "Tears" from your local library too. If you know of a free software work on the subject, please let me know and I'll promote it here.

A survey is evidence, not proof. You will probably not convince those who vehemently disagree with you and how many "floating voters" you convince is correlated with how good your survey is. Basically, as with any other surveys, the golden rules are:

1. Privately, start the write-up before you start collecting data and define:
   • what population you are trying to sample,
   • what question(s) you are trying to investigate,
   • your method of sampling (sometimes you can do better than a self-selecting listener opinion poll, while other times you need to compile a census of all members rather than sampling them),
   • the desired strength of outcome (if you can),
   • and any prior beliefs you hold about the topic;
2. Be as simple, clear and neutral as you can: in question phrasing and procedure details for a questionnaire, or in measurement methods for automated data collection, or so on;
3. Be considerate:
   • try to be polite when asking people to participate,
   • tell people how and when their responses will be published (if you're doing interviews, do this at the end, to avoid biasing your data: some people will tell you what they think you want to hear; but include it in the mailshot if you're asking people to send back forms),
   • accommodate community opinions (so for a survey about debian, please release the results and analysis as free software),
   • test the survey yourself before involving any other people
   • and try to follow any applicable guidelines, as far as possible.
     Remember that respondants are giving time to your research;
4. Carry out a pilot survey. Use it to debug the survey, as well as estimating the sample size you need to get the desired strength of outcome;
5. Carry out the main survey;
6. The first summary results should be a simple descriptive compilation of responses. Leave any commentary until the second part, then try to use any applicable methods to support or debunk your commentary;
7. Publish as much as you can, so others can check, reproduce or extend your findings.

To recap: define, clear, considerate, pilot, survey, summary, publish.

For more specific details, consider this advice from:

• Wouter's HOWTO perform a study on Open Source or Free Software (and get away with it without annoying people)
• Independent Media Centres (aka indymedia)
• European Telework Online
• Method diffusion in large open source projects, Ph.D. research by Martin F. Krafftseems like a good introduction that could stop researchers asking silly questions about debian, as well as interesting research in its own right.
• Web Content Accessibility Guidelinesaren't specifically about surveys, but there's no good reason for most surveys about debian to require permission to execute programs (javascript) and save data (cookies) on the user's computer or to require the user to have perfect eyesight. How would you feel reporting your results with "we surveyed all debian people with perfect eyesight that don't care about computer security or privacy"? Doesn't sound so good, does it?
• PHPSurveyorseems capable of doing accessible web surveys, if configured appropriately. I've not yet tested it myself.

Who am I and why I care

I am a debian developer who studied degree-level statistics for about ten years. I have been trained to do surveys by public and private sector groups. I trained others on-and-off for about 5 years. I'll link an example of my work here when it's online. These days, I earn a living offering web site visitor analysis, accessibility checking and market research, along with some programming.

Primarily, I want to help debian contributors carry out better surveys and so help manage their debian work better. The project is large and anecdotes don't always help. Some think that if you state an opinion, you must be ready to defend it forever, else it has less value: I reject that and believe good solid research will make collections of "ordinary" opinions defensible against this view.

Also, I find it really frustrating to see research projects surveying debian, but wasting our time. It is especially annoying when these projects are funded with large amounts of government money which could have gone to groups related toFSF Europe, but for politics.

Acknowledgements

Thanks to Andreas Schuldei for motivating me to work on this. Many thanks to Enrico Zini for commenting on an early draft and suggesting some "be considerate" advice. Thanks to all who commented on the announcementto debian-project. All errors are still probably mine, though.

You can copy this page under the same terms asmy personal site.

17:00, 23 Apr 2008MJR