[RFC] Enabling the Lit Internal Shell by Default (original) (raw)

We propose to enable llvm-lit internal shell by default for all LLVM projects. We believe that by using the internal shell, we will eliminate platform dependencies, improve test run times by reducing the number of subprocess invocations, and improving cross-platform testing consistency.

Background

LLVM’s main test running tool, lit, is used as a test driver for end-to-end tests. It is designed to parse and execute shell-like commands, which are specified in the comments of source files. Currently, lit’s default behavior is to use the system’s shell when executing RUN commands. However, lit also has an internal shell implementation, which is currently the default for Windows systems. This internal shell can be enabled regardless of platform by turning on the LIT_USE_INTERNAL_SHELL environment variable.

Motivations

llvm-lit‘s dependency on the system shell introduces inconsistencies across the various platforms LLVM is expected to support. On POSIX-compliant systems, like Linux, the commands typically execute as expected. However, on non-POSIX platforms, such as Windows and Fuchsia, these commands sometimes fail due to differences in shell environment.

The LLVM community has been discussing ways to improve the usability and functionality of lit for a while now. There has been previous discussion[1] about making improvements to lit’s internal shell, and even to deprecate the use of external shells and convert lit’s internal shell to be the default. There are several reasons that suggest a transition to using lit’s internal shell:

1. Fewer CI Breakages: Inconsistencies in shell behavior across different platforms are a frequent cause of CI breakages. These issues result in increased maintenance efforts as someone has to manually identify and fix the platform-specific failures.

2. Improved run times: This was brought up in a previous discussion[2], where it was observed that using lit’s internal shell resulted in faster times when running lit tests.

3. Enhanced Debug Output: The current debug output often repeats shell commands, making it unnecessarily difficult to navigate the code segments. For example, shell commands appear twice and the execution trace is not always clear. However, in RFC: Improving lit’s debug output[1], the usability of debug output for lit’s internal shell was improved upon to address difficulties with the debug output for external shells.

Implementation Plan

The internal shell currently has missing capabilities necessary for it to run tests across all LLVM subprojects. We have identified 5 missing or incomplete features that would allow us to enable lit’s internal shell by default.

1. Built-in Commands (cat, diff, env, ulimit, unset)

• Adding additional options to the cat, diff, and env commands.
• Implementing the ulimit and unset commands.
• Subprojects affected: clang-tools-extra, compiler-rt, lldb, llvm

2. Environment Variables (XRAY_OPTIONS, DEFAULT_TRIPLE, JITDUMPDIR, etc)

• Implementing the ability to set environment variables without the use of the env command. For example:
  RUN: XRAY_OPTIONS="patch_premain=false" %run %t | FileCheck %s
• Subprojects affected: clang, compiler-rt, lld, llvm

3. Brackets (“{}“, “()“, “[ ]“ )

• Implementing bracket/parentheses/brace recognition and execution.
• Subprojects affected: bolt, clang, compiler-rt, lld

4. Command Substitution

• Implementing the use of $() syntax for command substitution
• Subprojects affected: clang, compiler-rt

5. Command Output Redirection

• Implementing recognition of |& syntax to redirect stdout and stderr to the following command
• Subprojects affected: bolt

6. (Other) Python Implementation Fixes

• Fixing bugs in current lit internal shell implementation (e.g. TypeError, Unsupported Redirect, No Such File or Directory)
• Subprojects affected: bolt, compiler-rt, lldb, llvm

Implementation Strategy

We plan to address internal shell failures subproject by subproject, starting with LLVM. We hope that by doing this, we can switch on the internal shell for individual test suites as we fill in the missing functionalities, handling any regressions as we tackle each subproject. Our hope is to eventually be able to switch on the internal shell for all of the test suites and make it the default shell across all lit testing.

References

[1] RFC: Improving lit’s debug output RFC: Improving lit's debug output

[2] Why does llvm-lit use bash? Why does llvm-lit use bash?

CC: @connieyzhu @ilovepi