[llvm-dev] [RFC] Require PRs for XFAILing tests (original) (raw)

Krzysztof Parzyszek via llvm-dev llvm-dev at lists.llvm.org
Mon Oct 3 12:05:22 PDT 2016

On 10/3/2016 12:55 PM, Robinson, Paul wrote:

If the XFAIL-linked PRs are content-free, that is worse than useless. The point is to have those PRs say something useful about the specific XFAIL case, which is a vast improvement over what we have today (i.e., almost nothing).

They will be content-free.

Following of processes tends to degrade to meet the required minimum, especially when large groups of people are involved. In this proposal there is no way to enforce meaningful content, only that the PR exists.

To make sure that the process works, it needs a motivation that goes beyond being solely a requirement. If the PR indicated a problem that will need to be fixed, adding content would become natural. If the PR is there only to keep a test from failing, adding information to it is just extra work.

All that aside, what would be "meaningful information" in case of an xfailed test? Doing enough analysis to find out why it fails is often good enough to just fix it. A common thing for PRs is that they only serve as a record that a problem exists, ideally with information on how to reproduce it (they usually come from users, not developers). In case of xfailed testcases we already know that they fail and how to reproduce them.

-Krzysztof

-- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

More information about the llvm-dev mailing list