Please add more pre-merge testing for LLDB PRs (original) (raw)

March 27, 2025, 12:03am 1

Currently only Linux x86-64 tests run on LLDB PRs before merge. Recently my reverse-execution PR was merged and stuck thanks to the heroic efforts of many people, especially @labath. It would have required much less heroism if more platforms were tested in CI before merge, especially MacOS. Currently MacOS test bots only run after changes have been merged, which makes debugging Mac test failures almost impossible if you don’t own a Mac, or if you do own a Mac but can’t reproduce the bot test failures on it. The problem is especially acute because Mac debugserver is quite different from Linux lldbserver.

I think everyone agrees that it would be great to have MacOS in premerge testing. Why it is not comes down to resources, both in terms of compute and engineering. The current premerge infra is funded by Google, specifically through GCP. GCP does not support MacOS virtual machines. There are alternatives, but so far no one has stepped up to fund the amount of compute/donate enough time on existing MacOS machines that would be needed to run the current premerge pipeline on all PRs.

MacOS also presents some engineering challenges for building a scalable CI system. We use Github ARC (actions runner controller) and k8s with containers for Linux/Windows premerge testing, which makes things scalable and simplifies maintenance on the node side (we basically don’t need to think about it). MacOS does not have support for containers or k8s which makes things a bit more involved. Although it is certainly possible to spin up some Github runners on MacOS machines.

We could probably configure opt-in testing where if you add a label testing runs on one of the free github runners, but that adds some complexity that I would prefer to be justified by significant interest from the community.

GCP does not support MacOS virtual machines.

But it does support Linux ARM64, so is that a no-brainer? Linux ARM64 would certainly help.

The costs and benefits of setting up Mac pre-merge testing are no doubt complex. But engineer time is typically worth a lot more than machine time and I burned through plenty of the former (my own and others) with my PR!

One thing to consider is, how accessible do you want LLDB to be to contributors? Pre-merge testing is especially valuable to new contributors and contributors who don’t find it easy to ask for favors.

Endill March 27, 2025, 2:45am 4

For quite a while, pre-merge CI has been a question of compute resources and labor needed to maintain it. Important point here is that lack of coverage isn’t caused by someone not understanding how valuable pre-merge CI is. I think it’s also worth mentioning that we don’t have even a single dedicated person to maintain our infrastructure.

Some of the coverage has been disabled, because we don’t have enough compute resources. For instance, despite being a heavy user of Clang C++ API, LLDB is not tested on Clang changes, and that causes breaks.

It would certainly be nice to have, but given that we don’t have dedicated maintainers for this infrastructure, who is going to do the work, and then pay the bills to keep it running?