A large-scale study on research code quality and execution - PubMed (original) (raw)

A large-scale study on research code quality and execution

Ana Trisovic et al. Sci Data. 2022.

Abstract

This article presents a study on the quality and execution of research code from publicly-available replication datasets at the Harvard Dataverse repository. Research code is typically created by a group of scientists and published together with academic papers to facilitate research transparency and reproducibility. For this study, we define ten questions to address aspects impacting research reproducibility and reuse. First, we retrieve and analyze more than 2000 replication datasets with over 9000 unique R files published from 2010 to 2020. Second, we execute the code in a clean runtime environment to assess its ease of reuse. Common coding errors were identified, and some of them were solved with automatic code cleaning to aid code execution. We find that 74% of R files failed to complete without error in the initial execution, while 56% failed when code cleaning was applied, showing that many errors can be prevented with good coding practices. We also analyze the replication datasets from journals' collections and discuss the impact of the journal policy strictness on the code re-execution rate. Finally, based on our results, we propose a set of recommendations for code dissemination aimed at researchers, journals, and repositories.

© 2022. The Author(s).

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Fig. 1

Fig. 1

Most popular code file types on Harvard Dataverse (Oct, 2020). Of the top two, R is open source and free.

Fig. 2

Fig. 2

Implementation on the AWS Batch.

Fig. 3

Fig. 3

Dataset sizes and file counts.

Fig. 4

Fig. 4

File name lengths and presence of documentation.

Fig. 5

Fig. 5

Number of code line and relative number of comments.

Fig. 6

Fig. 6

Distribution of unique dependencies per dataset.

Fig. 7

Fig. 7

Most frequently used R libraries.

Fig. 8

Fig. 8

Success rate and errors before and after code cleaning. To objectively determine the effects of code cleaning, we subset the results that have explicit “successes” and errors while excluding the ones with TLE values as the outcome. As a result, the count of files in this figure is lower than the total count.

Fig. 9

Fig. 9

Types of workflows in research analyses.

Fig. 10

Fig. 10

Re-execution success rate per journal Dataverse collection. In the brackets are the number of datasets and the number of R files, respectively.

Fig. 11

Fig. 11

Re-execution success rates per year per R software version.

Fig. 12

Fig. 12

Success rate per research field.

References

    1. Digital Science et al. The State of Open Data Report 2019. _Digital Science_10.6084/m9.figshare.9980783.v2 (2019).
    1. National Academies of Sciences, Engineering, and Medicine and others. Reproducibility and replicability in science (National Academies Press, 2019). -PubMed
    1. Baker M. 1,500 scientists lift the lid on reproducibility. Nature News. 2016;533:452. doi: 10.1038/533452a. -DOI -PubMed
    1. Knuth DE. Literate programming. The Computer Journal. 1984;27:97–111. doi: 10.1093/comjnl/27.2.97. -DOI
    1. Pasquier T, et al. Sharing and Preserving Computational Analyses for Posterity with encapsulator. Computing in Science Engineering. 2018;20:111–124. doi: 10.1109/MCSE.2018.042781334. -DOI

Publication types

LinkOut - more resources