A consensus-based transparency checklist (original) (raw)

We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository.

Good science requires transparency

Ideally, science is characterized by a ‘show me’ norm, meaning that claims should be based on observations that are reported transparently, honestly and completely1. When parts of the scientific process remain hidden, the trustworthiness of the associated conclusions is eroded. This erosion of trust affects the credibility not only of specific articles, but—when a lack of transparency is the norm—perhaps even entire disciplines. Transparency is required not only for evaluating and reproducing results (from the same data), but also for research synthesis and meta-analysis from the raw data and for effective replication and extension of that work. Particularly when the research is funded by public resources, transparency and openness constitute a societal obligation.

In recent years many social and behavioural scientists have expressed a lack of confidence in some past findings2, partly due to unsuccessful replications. Among the causes for this low replication rate are underspecified methods, analyses and reporting practices. These research practices can be difficult to detect and can easily produce unjustifiably optimistic research reports. Such lack of transparency need not be intentional or deliberately deceptive. Human reasoning is vulnerable to a host of pernicious and often subtle biases, such as hindsight bias, confirmation bias and motivated reasoning, all of which can drive researchers to unwittingly present a distorted picture of their results.

The practical side of transparency

How can scientists increase the transparency of their work? To begin with, they could adopt open research practices such as study preregistration and data sharing3,4,5. Many journals, institutions and funders now encourage or require researchers to adopt these practices. Some scientific subfields have seen broad initiatives to promote transparency standards for reporting and summarizing research findings, such as START, SPIRIT, PRISMA, STROBE and CONSORT (see https://www.equator-network.org). A few journals ask authors to answer checklist questions about statistical and methodological practices (e.g., the Nature Life Sciences Reporting Summary)6 and transparency (for example, Psychological Science). Journals can signal that they value open practices by offering ‘badges’ that acknowledge open data, code and materials7. The Transparency and Openness Promotion (TOP) guidelines8, endorsed by many journals, promote the availability of all research items, including data, materials and code. Authors can declare their adherence to these TOP standards by adding a transparency statement in their articles (TOP Statement)[9](/articles/s41562-019-0772-6#ref-CR9 "Aalbersberg, I. J. et al. Making Science Transparent By Default; Introducing the TOP Statement. Preprint at OSF https://osf.io/preprints/sm78t/

             (2018)."). Collectively, these somewhat piecemeal innovations illustrate a science-wide shift toward greater transparency in research reports.

Transparency Checklist

We provide a consensus-based, comprehensive transparency checklist that behavioural and social science researchers can use to improve and document the transparency of their research, especially for confirmatory work. The checklist reinforces the norm of transparency by identifying concrete actions that researchers can take to enhance transparency at all the major stages of the research process. Responses to the checklist items can be submitted along with a manuscript, providing reviewers, editors and, eventually, readers with critical information about the research process necessary to evaluate the robustness of a finding. Journals could adopt this checklist as a standard part of the submission process, thereby improving documentation of the transparency of the research that they publish.

We developed the checklist contents using a preregistered ‘reactive-Delphi’ expert consensus process10, with the goal of ensuring that the contents cover most of the elements relevant to transparency and accountability in behavioural research. The initial set of items was evaluated by 45 behavioural and social science journal editors-in-chief and associate editors, as well as 18 open-science advocates. The Transparency Checklist was iteratively modified by deleting, adding and rewording the items until a sufficiently high level of acceptability and consensus were reached and no strong counter arguments for single items were made (for the selection of the participants and the details of the consensus procedure see Supplementary Information). As a result, the checklist represents a consensus among these experts.

The final version of the Transparency Checklist 1.0 contains 36 items that cover four components of a study: preregistration; methods; results and discussion; and data, code and materials availability. For each item, authors select the appropriate answer from prespecified options. It is important to emphasize that none of the responses on the checklist is a priori good or bad and that the transparency report provides researchers the opportunity to explain their choices at the end of each section.

In addition to the full checklist, we provide a shortened 12-item version (Fig. 1). By reducing the demands on researchers’ time to a minimum, the shortened list may facilitate broader adoption, especially among journals that intend to promote transparency but are reluctant to ask authors to complete a 36-item list. We created online applications for the two checklists that allow users to complete the form and generate a report that they can submit with their manuscript and/or post to a public repository (Box 1). The checklist is subject to continual improvement, and users can always access the most current version on the checklist website; access to previous versions will be provided on a subpage.

Fig. 1

The Shortened Transparency Checklist 1.0. After each section, the researchers can add free text if they find that further explanation of their response is needed. The full version of the checklist can be found at http://www.shinyapps.org/apps/TransparencyChecklist/.

This checklist presents a consensus-based solution to a difficult task: identifying the most important steps needed for achieving transparent research in the social and behavioural sciences. Although this checklist was developed for social and behavioural researchers who conduct and report confirmatory research on primary data, other research approaches and disciplines might find value in it and adapt it to their field’s needs. We believe that consensus-based solutions and user-friendly tools are necessary to achieve meaningful change in scientific practice. While there may certainly remain important topics the current version fails to cover, nonetheless we trust that this version provides a useful to facilitate starting point for transparency reporting. The checklist is subject to continual improvement, and we encourage researchers, funding agencies and journals to provide feedback and recommendations. We also encourage meta-researchers to assess the use of the checklist and its impact in the transparency of research.

Data availability

All anonymized raw and processed data as well as the survey materials are publicly shared on the Open Science Framework page of the project: https://osf.io/v5p2r/. Our methodology and data-analysis plan were preregistered before the project. The preregistration document can be accessed at: https://osf.io/v5p2r/registrations.

Change history

A Correction to this paper has been published: https://doi.org/10.1038/s41562-019-0812-2

References

  1. Merton, R. The Sociology of Science: Theoretical and Empirical Investigations (University of Chicago Press, 1973).
  2. Baker, M. Nature 533, 452–454 (2016).
    Article CAS Google Scholar
  3. Chambers, C. D. Cortex 49, 609–610 (2013).
    Article Google Scholar
  4. Gernsbacher, M. A. Adv. Methods Pract. Psychol. Sci. 1, 403–414 (2018).
    Article Google Scholar
  5. Munafò, M. R. et al. Nat. Hum. Behav. 1, 0021 (2017).
    Article Google Scholar
  6. Campbell, P. Nature 496, 398 (2013).
    Article Google Scholar
  7. Kidwell, M. C. et al. PLoS Biol. 14, e1002456 (2016).
    Article Google Scholar
  8. Nosek, B. A. et al. Science 348, 1422–1425 (2015).
    Article CAS Google Scholar
  9. Aalbersberg, I. J. et al. Making Science Transparent By Default; Introducing the TOP Statement. Preprint at OSF https://osf.io/preprints/sm78t/ (2018).
  10. McKenna, H. P. J. Adv. Nurs. 19, 1221–1225 (1994).
    Article CAS Google Scholar

Download references

Acknowledgements

We thank F.Schönbrodt and A.T. Foldes for their technical help with the application.

Author information

Authors and Affiliations

  1. ELTE, Eotvos Lorand University, Budapest, Hungary
    Balazs Aczel, Barnabas Szaszi & Zoltan Kekecs
  2. University of Amsterdam, Amsterdam, Netherlands
    Alexandra Sarafoglou, Šimon Kucharský, Agneta Fisher & Eric-Jan Wagenmakers
  3. University of Southern California, Los Angeles, CA, USA
    Daniel Benjamin
  4. Cardiff University, Cardiff, UK
    Christopher D. Chambers & Candice C. Morey
  5. Columbia University, New York, NY, USA
    Andrew Gelman, Eric Johnson & Don P. Green
  6. University of Wisconsin-Madison, Madison, WI, USA
    Morton A. Gernsbacher & John Curtin
  7. Stanford University, Stanford, CA, USA
    John P. Ioannidis
  8. Maastricht University, Maastricht, Netherlands
    Kai Jonas
  9. Nature Human Behaviour, Springer Nature, London, UK
    Stavroula Kousta
  10. Emory University, Atlanta, GA, USA
    Scott O. Lilienfeld
  11. University of Melbourne, Melbourne, Victoria, Australia
    Scott O. Lilienfeld
  12. University of Victoria, Saanich, British Columbia, Canada
    D. Stephen Lindsay
  13. University of Bristol, Bristol, UK
    Marcus Munafò
  14. University of New South Wales, Sydney, New South Wales, Australia
    Benjamin R. Newell
  15. University of California San Diego, San Diego, CA, USA
    Harold Pashler
  16. University College London, London, UK
    David R. Shanks & Pasco Fearon
  17. University of Illinois, Chicago, IL, USA
    Daniel J. Simons & Dolores Albarracin
  18. Tilburg University, Tilburg, Netherlands
    Jelte M. Wicherts
  19. Rotman Research Institute, Baycrest, Toronto, Ontario, Canada
    Nicole D. Anderson
  20. University of Lausanne, Lausanne, Switzerland
    John Antonakis
  21. Ohio State University, Columbus, OH, USA
    Hal R. Arkes
  22. University of Münster, Münster, Germany
    Mitja D. Back
  23. University of North Carolina at Charlotte, Charlotte, NC, USA
    George C. Banks
  24. University of Texas at Austin, Austin, TX, USA
    Christopher Beevers
  25. Old Dominion University, Norfolk, VA, USA
    Andrew A. Bennett
  26. University of California Davis, Davis, CA, USA
    Wiebke Bleidorn & Simine Vazire
  27. Georgia Southern University, Statesboro, GA, USA
    Ty W. Boyer
  28. University of Modena-Reggio Emilia, Modena, Italy
    Cristina Cacciari
  29. University of Massachusetts, Boston, Boston, MA, USA
    Alice S. Carter
  30. Michigan State University, East Lansing, MI, USA
    Joseph Cesario
  31. University of Massachusetts, Amherst, Amherst, MA, USA
    Charles Clifton
  32. Royal College of Surgeons in Ireland, Dublin, Ireland
    Ronán M. Conroy
  33. University of Nebraska Omaha, Omaha, NE, USA
    Mike Cortese
  34. University of Florence, Florence, Italy
    Fiammetta Cosci
  35. University of Missouri, Columbia, MO, USA
    Nelson Cowan
  36. The College of New Jersey, Ewing Township, NJ, USA
    Jarret Crawford
  37. Leiden University, Leiden, Netherlands
    Eveline A. Crone
  38. Georgia Institute of Technology, Atlanta, GA, USA
    Randall Engle
  39. University of Western Australia, Perth, Western Australia, Australia
    Simon Farrell
  40. Carnegie Mellon University, New York, NY, USA
    Mark Fichman
  41. Radboud University, Nijmegen, Netherlands
    Willem Frankenhuis
  42. University of Zurich, Zurich, Switzerland
    Alexandra M. Freund & Lea Moersdorf
  43. University of York, York, UK
    M. Gareth Gaskell
  44. University of Kent, Kent, UK
    Roger Giner-Sorolla
  45. Case Western Reserve University, Cleveland, OH, USA
    Robert L. Greene
  46. University of Rhode Island, Providence, RI, USA
    Lisa L. Harlow
  47. University of California, Berkeley, Berkeley, CA, USA
    Fernando Hoces de la Guardia
  48. Northeastern University, Boston, MA, USA
    Derek Isaacowitz
  49. Boston College, Boston, MA, USA
    Janet Kolodner
  50. University of Miami, Coral Gables, FL, USA
    Debra Lieberman
  51. Vanderbilt University, Nashville, TN, USA
    Gordon D. Logan
  52. University of California, San Francisco, San Francisco, CA, USA
    Wendy B. Mendes
  53. University of Michigan, Ann Arbor, MI, USA
    Brendan Nyhan
  54. North Carolina State University, Raleigh, NC, USA
    Jeffrey Pollack
  55. University of Cincinnati, Cincinnati, OH, USA
    Christopher Sullivan

Authors

  1. Balazs Aczel
  2. Barnabas Szaszi
  3. Alexandra Sarafoglou
  4. Zoltan Kekecs
  5. Šimon Kucharský
  6. Daniel Benjamin
  7. Christopher D. Chambers
  8. Agneta Fisher
  9. Andrew Gelman
  10. Morton A. Gernsbacher
  11. John P. Ioannidis
  12. Eric Johnson
  13. Kai Jonas
  14. Stavroula Kousta
  15. Scott O. Lilienfeld
  16. D. Stephen Lindsay
  17. Candice C. Morey
  18. Marcus Munafò
  19. Benjamin R. Newell
  20. Harold Pashler
  21. David R. Shanks
  22. Daniel J. Simons
  23. Jelte M. Wicherts
  24. Dolores Albarracin
  25. Nicole D. Anderson
  26. John Antonakis
  27. Hal R. Arkes
  28. Mitja D. Back
  29. George C. Banks
  30. Christopher Beevers
  31. Andrew A. Bennett
  32. Wiebke Bleidorn
  33. Ty W. Boyer
  34. Cristina Cacciari
  35. Alice S. Carter
  36. Joseph Cesario
  37. Charles Clifton
  38. Ronán M. Conroy
  39. Mike Cortese
  40. Fiammetta Cosci
  41. Nelson Cowan
  42. Jarret Crawford
  43. Eveline A. Crone
  44. John Curtin
  45. Randall Engle
  46. Simon Farrell
  47. Pasco Fearon
  48. Mark Fichman
  49. Willem Frankenhuis
  50. Alexandra M. Freund
  51. M. Gareth Gaskell
  52. Roger Giner-Sorolla
  53. Don P. Green
  54. Robert L. Greene
  55. Lisa L. Harlow
  56. Fernando Hoces de la Guardia
  57. Derek Isaacowitz
  58. Janet Kolodner
  59. Debra Lieberman
  60. Gordon D. Logan
  61. Wendy B. Mendes
  62. Lea Moersdorf
  63. Brendan Nyhan
  64. Jeffrey Pollack
  65. Christopher Sullivan
  66. Simine Vazire
  67. Eric-Jan Wagenmakers

Contributions

B.A., B.S., A.S., Z.K., and E-J.W. conceptualized the project, conducted the survey study, analysed the data and drafted the initial version of the manuscript. Š.K. developed and designed the online application. D.B., C.D.C., A.F., A.G., M.A.G., J.P.I., E.J., K.J., S.K., S.O.L., D.S.L., C.C.M., M.M., B.R.N., H.P., D.R.S., D.J.S., and J.M.W. took part in the preparation and conclusion of the checklist items. D.A., N.D.A., J.A., H.A., M.D.B., G.C.B., C.B., A.A.B., W.B., T.W.B., C.C., A.S.C., J.C., C. Clifton, R.M.C., M.C., F.C., N.C., J. Crawford, E.A.C., J. Curtin, R.E., S.F., P.F., M.F., W.F., A.M.F., M.G.G., R.G-S., D.P.G., R.L.G., L.L.H., F.H.G., D.I., J.K., D.L., G.D.L., W.B.M., L.M., B.N., J.P., C.S., and S.V. evaluated the checklist items. All authors were involved in reviewing and editing the final version of the manuscript.

Corresponding author

Correspondence toBalazs Aczel.

Ethics declarations

Competing interests

S.K. is Chief Editor of the journal Nature Human Behaviour. S.K. has recused herself from any aspect of decision-making on this manuscript and played no part in the assignment of this manuscript to in-house editors or peer reviewers. She was also separated and blinded from the editorial process from submission inception to decision. The other authors declared no competing interests.

Supplementary information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Cite this article

Aczel, B., Szaszi, B., Sarafoglou, A. et al. A consensus-based transparency checklist.Nat Hum Behav 4, 4–6 (2020). https://doi.org/10.1038/s41562-019-0772-6

Download citation