[Python-Dev] please help triage VSTS failures (original) (raw)

Gregory P. Smith greg at krypto.org
Fri May 18 20:49:14 EDT 2018

Hah, yes, that was un-intuitive for me. I was looking for something labelled "logs" to click on. thanks. (I hope MS is taking notes on UX issues here)

So these failures were in fact the known flakes and not infrastructure. good!

On the high level view of VSTS output on a failure:

"Issues: phase1 Cmd.exe exited with code '2'."

is not useful. Can we make that display the tail of the failing phase's log? that'd avoid needing to find the inner most red X and click on it which is what I'll always need to do otherwise as there doesn't appear to be a direct hyperlink to that.

It looks like VSTS also had an API for surfacing individual test results into their Tests / Test Results summary pane? Doing something to integrate with that would likely be a nicer UI. We have something similar at work, here's how we make unittest.TestCase emit the JUnit XML files of test results for CI systems to display:

https://github.com/abseil/abseil-py/blob/master/absl/testing/xml_reporter.py

If there are industry standard format(s) that make Python test suites play nicer for reporting on CI systems such as that, they'd be worthy of stdlib inclusion.

-gps

On Fri, May 18, 2018 at 5:35 PM Nathaniel Smith <njs at pobox.com> wrote:

On Fri, May 18, 2018 at 1:13 PM, Steve Dower <steve.dower at python.org> wrote: > According to the VSTS dev team, an easy “rerun this build” button and > filtering by changed paths are coming soon, which should clean things up.

If you're talking to them, please ask them to make sure that the "rerun this build" button doesn't erase the old log. (That's what it does on Travis. Appveyor is better.) The problem is that when you have a flaky/intermittent failure, your todo list is always (a) rerun the build so at least it's not blocking whatever this unrelated change is, (b) file some sort of bug, or comment on some existing bug, and link to the log to help track down the intermittent failure. If you click the "rebuild" button on Travis, then it solves (a), while deleting the information you need for (b) – and for rare intermittent bugs you might not have much information to go on besides that build log. -n -- Nathaniel J. Smith -- https://vorpus.org

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180518/9ff7419b/attachment.html>

More information about the Python-Dev mailing list