pyOpenSci Software Review Editor Guide (original) (raw)

Thank you for your time in serving as an editor for a PyOpenSci package! Below you will find some information about the role that editors have in the pyOpenSci Python open peer review process.

Experience needed to become an editor#

Editors generally should:

Two types of editors#

There are two types of editors involved in our open peer review process:

Both types of editors are considered a part of the editorial board for pyOpenSci. The major difference between guest and full editors is:

Guest editors#

A guest editor is is invited to lead a review in the case where we need specific expertise for a single review. We also consider editors who are performing their first review as guest editors as they may require more guidance or mentorship to complete the review (if they are new to our organization).

New editors who wish to continue on as full editors for pyOpenSci may do so as long as both parties (pyOpenSci and the guest editors) feel like it is a healthy fit for them and the organization.

“Full” editors#

A full editor is most often someone who has experience with the pyOpenSci open package review process. A full editor ideally:

We also appreciate when editors have experience working with or in the Python open source software community be it maintaining packages, contributing to packages, or supporting the community. This is certainly not a requirement however if you are interested in getting involved with pyOpenSci!

Note

There could be certain situations when an editor is on boarded with less experience! The above are simply guidelines that we like to follow.

What does an editor do? (Responsibilities)#

An editor is normally recruited by the Editor in Chief, other editors on the board, or the software review lead. More on recruiting editors can be found here.

An editor is responsible for:

Note

Decisions surrounding policy, updates to peer review guides and decisions on package review are generally made in the private editorial-board channel in the pyOpenSci Slack organization. Please make sure that you are comfortable with checking Slack regularly.

Editor support of other reviews#

Editors are not charged with tracking other submissions that they are not leading. However, if you are serving as an editor and notice an issue with another review, please raise that issue either directly with the editor for that review. Or you can raise the issue in the editors channel in slack.

Editor-in-Chief rotation#

The editorial board normally participates in the Editor in Chief rotation. You are eligible to enter this rotation after 3 months of serving on the editorial board and/or after your first review as it makes sense. Read more about the roles and responsibilities of the Editor in Chief, here.

If the Editor in Chief role feels like too much responsibility, an editor can also decline being a part of this rotation.

How long does an editor serve on the editorial board?#

Ideally an editor can commit to serving for at least one year as an editor for pyOpenSci. During that year we expect that you will lead review of 3-4 packages. However, we understand that in certain situations, an editor may need to step down before the 1 year time period has ended.

We also understand that life gets busy and you are always welcome to “say no” to a review during a particularly busy time.

We welcome editors staying on for longer as long as they are happy serving with us and they get along well with other members of the editorial board, the software review lead and the current Editor in Chief.

Editor checklist: Get Started With Leading a Package Review#

Follow the checklist below when serving as an editor for a package submitted to pyOpenSci for review.

All reviews happen in GitHub issues. The template for theyaml header of a review submission below will be referenced multiple times in the steps below:


name: Submit Software for Review about: Use to submit your Python package for pyOpenSci peer review title: '' labels: 1/editor-checks, New Submission! assignees: ''

✔️ 1. First, tag the submission issue on GitHub#

Once you begin the review process as an editor:

Submitting Author: SUBMITTING AUTHOR NAME HERE (Name @github_handle) All current maintainers: ALL CURRENT MAINTAINERS LISTED HERE (Name @github_handle1, Name @github_handle2) Package Name: PACKAGE-NAME-HERE One-Line Description of Package: DESCRIPTION OF THE PACKAGE HERE Repository Link: REPO-LINK Version submitted: VERSION-SUBMITTED

Editor in Chief checks for structure & scope should be completed first

The editor in chief who initially engaged with this review should have already evaluated the package level Editor Checks section for Fit, Automated Tests, Documentation, License, and Repository.

They also should have checked whether the package is in scope for pyOpenSci. And whether there is functionality overlap with functionality of any other existing Python packages.

However, in some instances the editor-in-chief may request that an editor perform these tasks. Be sure to check the issue to ensure the above checks have been implemented prior to initiating the review.

If the package does not fit the pyOpenSci scope and policies and needs to be rejected, seethis section in the editor in chief guideabout how to respond.

✔️ 2. Respond to the submitter in the GitHub issue#

Once the above is complete, you are ready to add an editor response to the issue. This step ensures that the package is ready to be reviewed. It also ensures that we are using our volunteer reviewer time effectively.

Editor response to review:


Editor comments

:wave: Hi @reviewer-one and @reviewer-two! Thank you for volunteering to review for pyOpenSci!

Please fill out our pre-review survey

Before beginning your review, please fill out our pre-review survey. This helps us improve all aspects of our review and better understand our community. No personal data will be shared from this survey - it will only be used in an aggregated format by our Executive Director to improve our processes and programs.

The following resources will help you complete your review:

  1. Here is the reviewers guide. This guide contains all of the steps and information needed to complete your review.
  2. Here is the review template that you will need to fill out and submit here as a comment, once your review is complete.

Please get in touch with any questions or concerns! Your review is due:

Reviewers: Due date:

Important

Important: If in the initial checks you find any major gaps in the package’s structure, request changes before assigning reviewers.

✔️ 3. Identify scientific Python package reviewers#

Each review should have at least 2 reviewers.

A review consisting of a domain expert and a Pythonista is ideal as it provides two distinct perspectives for review. Further it can often be difficult to find two people with both the specific domain expertise AND packaging expertise.

Finding package reviewers#

Often times, finding reviewers for a package can be the trickiest part of the review process. Expect this to take a bit of time. Check out this page for tips related to finding reviewers.

If you can, try to find two people to serve as reviewers within two weeks of responding to the issue as the editor. If it takes longer, as often does, make a point to keep the author posted on the issue as you continue your search. You may add language such as:

Hey, @authorGithubHandle I just wanted to drop in to let you know that I’m searching for reviewers for your package. It may take a bit more time.

This type of communication just lets the author know that the process is moving forward. Even if it takes longer to find reviews, authors generally appreciate the communication and understand it’s a volunteer-lead process.

If you wish, you can use the email template below to invite reviewers.

Dear [REVIEWER-Name],

Hi, this is [EDITOR-Name]. [FRIENDLY BANTER]. I'm writing to ask if you have time to review a package for pyOpenSci. pyOpenSci has an open peer review of open source Python packages that support science. Accepted packages become a part of our pyOpenSci ecosystem of vetted community-adopted tools. Our review process is similar to that of open journals however it focuses on ensuring high quality packaging approaches that are inline with community adopted Python standards.

The package, [PACKAGE] by [AUTHOR(S)], does [FUNCTION]. You can find it on GitHub here: [REPO LINK]. We conduct our open review process via GitHub as well, here: [ONBOARDING ISSUE]

If you accept, note that we ask reviewers to complete reviews in three weeks (We’ve found it takes a similar amount of time to review a package as an academic paper.).

Our reviewers guide details what we look for in a package review, and includes links to example reviews. We also include a reviewer template on that page that you can use to guide your review. Our Python packaging standards are detailed in our packaging guide.

If you have time, please have a look at the package first, to make sure that you do not have a conflict of interest with the package authors.

If you have questions or feedback, feel free to ask me.

{IF APPLICABLE: The authors have also chosen to jointly submit their package to the Journal of Open Source Software, so this package includes a short paper.md manuscript describing the software that we ask you include in your review.}

Are you able to review? If you can not, I welcome any suggestions that you have for other reviewers. If I don't hear from you within a week, I will assume that you are unable to review this package at this time.

Thank you for your time.

Sincerely,

[EDITOR]

When inviting reviewers, include something like “if I don’t hear from you in a week, I’ll assume you are unable to review,” so as to give a clear deadline when you’ll move on to looking for someone else to keep the processing moving.

  1. reviewer GitHub handles and
  2. the review deadline date.

Warning

Make sure to ask the reviewers for their preferred means of contact, or a reliable way to get in touch with them.

✔️ 4. Onboard reviewers#

Once reviewers have been identified:

Editor: Full Name (@github_username) Reviewer 1: Full Name (@github_username) Reviewer 2: Full Name (@github_username) Archive: Filled out when the review is complete. Version accepted: Filled out when the review is complete.

Editor responsibilities during the review#

During the review process, it is important to check in with the reviewers to ensure that things are moving smoothly:

✔️ 5. What to do when reviews are in#

If the author stops responding, refer to the policies and/or ping the other editors in the Slack channel <_Not available publicly yet_> for discussion.

Once the author has responded to the reviews and made appropriate changes made changes:

Important

If for some reason during the process you need to halt the review and close the issue, be sure to let all parties know (maintainer and reviewers) prior to closing the issue.

Be sure to:

Putting a review on hold & handling non-responsive authors#

In some cases, an author may need more time to respond to review comments. In this case, the author can choose to have their submission put on hold. As an editor you should apply the holding label to the GitHub issue.

The holding status will be revisited every 3 months.

After one year the issue will be closed if there is no movement towards responding to reviews by the author.

If the author simply not responding, the editor should:

If needed you can also chose to email the author. using the email provided in the package metadata file of the package. The email could be in any of the three files in the package: setup.py, pyproject.toml (preferred) or setup.cfg.

Important

If a submission is closed and the author wishes to re-submit, they will have to start a new submission.

If the package is still in scope, the author will have to respond to the first round of reviews first. After that is complete, you can begin looking for new reviewers to evaluated the package. This ensures that none of the review energy spent in the first review, goes to waste.

✔️ 6. How to accept a package into the pyOpenSci ecosystem#

Once the package has been accepted through the review process:


🎉 has been approved by pyOpenSci! Thank you for submitting and many thanks to for reviewing this package! 😸

Author Wrap Up Tasks

There are a few things left to do to wrap up this submission:

It looks like you would like to submit this package to JOSS. Here are the next steps:

🎉 Congratulations! You are now published with both JOSS and pyOpenSci! 🎉

Editor Final Checks

Please complete the final steps to wrap up this review. Editor, please do the following:


If you have any feedback for us about the review process please feel free to share it here. We are always looking to improve our process and documentation in the peer-review-guide.

Archive: UPDATE-THIS-WITH-A-ZENODO-ARCHIVE-BADGE-TBD Version accepted: UPDATE-THIS-TBD Date accepted (month/day/year): UPDATE-THIS-TBD

Note

Closing notes about the editorial process#

✔️ OPTIONAL: Instructions for Submitting to JOSS#

If the package fits within the JOSS Scope, once the package has been approved by pyOpenSci:

These instructions loosely include:

  1. Login to the JOSS website and fill out the JOSS submission form. When you fill out the form, be sure to mention and link to the approved pyOpenSci review.
  2. Wait for a JOSS editor to approve the presubmission (which includes a scope check).

Important

The scope of packages accepted by pyOpenSci is sometimes different from those accepted by JOSS. Not all pyOpenSci accepted packages will be accepted by JOSS. Further, packages that have been previously published elsewhere may not be eligible to be published with JOSS unless significant changes and improvements to package functionality have been made.

JOSS will accept the pyOpenSci review and direct the author to check their paper.md file. Once the package is accepted by JOSS, the author will be instructed to add the JOSS DOI badge to their package README file.

Once the package is accepted by JOSS and the DOI badge resolves properly:

Last Steps Before Closing the Review Issue#

Once the review is complete, you can close the issue. Before doing that:

Congratulations, you have completed a review for pyOpenSci!