shell_completions test issues · Issue #14545 · rust-lang/cargo (original) (raw)

Problem

The new shell_completions tests have a few issues that need to be resolved before they can be enabled.

The tests randomly fail. I have seen fish and zsh randomly fail. Gate shell_completions tests on availability #14541 (comment) indicates that it might be an interactive timeout.
They require external tools to be installed to run. Tests generally should not require that.
- Looking for the binaries is not sufficient, since for example the bash test also requires the bash-completions package to be installed.
The elvish test has a weird failure for me:

++++ actual:   In-memory
        1 + Deprecation: the legacy temporary assignment syntax is deprecated; use "tmp" instead
        2 +   /tmp/cargo/target/tmp/cit/t0/home/elvish/rc.elv:3:7: eval (E:CARGO_COMPLETE=elvish cargo | slurp)

Tests should be using the ignore attribute so that it reports in the output when it isn't running. Currently they exit with return; on macos, but that should use the ignore attribute instead.
Should figure out why the tests are ignored on macos, since I would expect that to work with homebrew. I'm unaware why these are currently ignored.

Steps

cargo test --test testsuite -- shell_completions

Possible Solution(s)

One idea I had for limiting where these tests run is to add a CI_EXTENDED environment variable. The tests would only be required to run if that environment variable is set, and then we can set that in the appropriate jobs (like test). Then the cargo_test macro could have something like requires_extended_fish or something like that. That would also be useful for the currently hard-coded hg and lldb.

However, that doesn't help with some of the more complex issues like the bash-completions problem.

We could extend the cargo_test attribute to have ignore_macos to fix the macos ignore problem.

Notes

No response

Version

Currently on 643a025 of master.