Monthly LLVM libc meeting (original) (raw)

October 19, 2023, 11:42pm 1

Hello,

I’m setting up a regular (every 4 weeks) meeting for developers of, and anyone interested in, LLVM’s libc (partly inspired by the one being set up for libc++).

The goal is to provide a space for future and current libc developers and users where we can discuss what we’re working on, what is open for more work, and what is blocking us.

There should be an event on the LLVM community events calendar by the time this is posted.

The meeting will be held on Google Meet (meet.google.com/otv-dmps-mcc) at 09:00 PST/PDT which is also 12:00 EST/EDT and 18:00 CET/CEST. The first meeting will be on Thursday, October 26th, 2023, and then every four weeks after that (excluding holidays, such as US Thanksgiving).

The meeting notes will be posted on this thread after each meeting, and the agenda will be posted before each meeting. If you want to add to the agenda, the easiest way is to send a message in the #libc channel in discord, or send me a message on discourse/discord.

Here’s the agenda for tomorrow’s meeting:

• Introductions
  • Explaining the structure of this meeting
    * Start with the topics in the agenda
    * If we get through those, move to Q&A/open discussion
    * If we finish with that, end the meeting, even if it’s early.
• (michaelrj): How many people are interested in working on the project, and how many are interested in using the project?
  • How can we enable what you’re doing?
  • What needs to be done before the libc is “complete” for your purposes?
• Here’s what we’re working on:
  • (michaelrj): working on finalizing printf for google use
  • (lntue): working on double precision math functions
• (jhuber): GPU math status?
• (mcgrathr): How should special compiler flags be handled?
  • Should we allow changes that make certain compiler flags necessary for optimal performance?
    * This can cause problems for customers that don’t use our build system

Here are the notes from today’s meeting (lightly edited):

• What needs to be done before the libc is “complete” for your use case?
  • Jhuber: Using LLVM libc for GPUs
    * Waiting on compiler fixes to land before varargs are available, as well as math tests for the GPU.
    * Tue is working on improving the exhaustive math tests so that they can test against the host system, since if we test that against MPFR then we can know they all match.
  • Phosek: Using LLVM libc for Fuchsia and Pigweed
    * Fuchsia is a microkernel OS
    * Currently using an old fork of musl, they want to move to something with better non-linux support, and getting updates.
    * Fuchsia needs a complete C library, and ideally C++ as well.
    * The Fuchsia libc has been slowly replacing parts of musl with functions from LLVM libc.
    * Pigweed is an embedded framework
    * LLVM libc would be a replacement for newlib
    * Most embedded systems don’t need all of libc
    * Memory functions, math, maybe sprintf
    * The libc might be complete for embedded, testing is ongoing.
    * The pixel buds pro are the first product they’re trying to move to LLVM libc, and it looks like all the functions they need are available.
    * There are other ones in progress, but they may need more. Every embedded system is different.
• What are people interested in working on?
  • Lucian: Curious about streams, security and performance critical functions.
    * For security critical functions: The networking functions are very relevant, as well as having more fuzzers for existing string functions.
    * Having better support for operating with sanitizers, which traditionally intercept the libc functions since other libcs like glibc don’t cooperate with the sanitizer
    * Fuchsia builds sanitizers into their libc, since they build from source, and they’d be willing to upstream their internal API.
    * They’d also be interested in (if possible) working with several libc vendors to make this API more generally available.
    * Fortify source is something we’d like to support, but haven’t yet.
    * We’d like to coordinate our fortify source development with clang, since that would make the implementation cleaner within libc.
    * Another option would be matching the hardening modes that libc++ has.
• Here’s what we’re working on:
  • (michaelrj): working on finalizing printf for google use
  • (lntue): working on single precision math functions and some initial support for C23 math functions
  • (jhuber): Poked at scanf, but has had issues getting it to compile. Also would like to build libc for GPUs and CPUs at the same time.
    * One of the important things for the GPU build is that it has to be built with an up-to-date clang since it requires some magic builtins only added to clang.
    * The current problem is that the libc cmake assumes that there’s only one target at a time.
    * It might be easier to just run cmake multiple times as separate builds using the runtimes build.
    * We might need to add ordering to the runtimes build, since it currently builds in any order.
    * The current way that libc++ can do multiple builds for multiple targets is by putting them in lib/target-triple/…
    * Another way would be to build the libcs with different suffixes (e.g. libc-target-triple.a)
    * One of the things being considered for the runtimes build is having a list of targets to build for, as well as a list of which of those can run tests.
• Status of GPU math functions
  • The next step is getting exhaustive testing running so that we can test against the openCL standard.
  • We’d like to be able to provide multiple implementations so that we can provide a switch between speed and accuracy.
• How do we balance having broad hardware support vs having the best performance
  • Using ifuncs to select a specific implementation is the standard method, but not necessarily the best one.
  • One way to handle this is to treat libc sources as part of your program, which allows for more targeted optimization but harms portability
  • How do we inform external users about special flags that certain files need?
    * Some users don’t use our provided build system, so having flags that are necessary for optimal performance can be a stumbling block if they aren’t made aware
    * We should probably add those flags to our config system since it’s JSON and so fairly easy to consume for downstream users
• Lucian brought up the idea of benchmarking functions using shortcuts through undefined behavior, such as what strlen does: https://github.com/llvm/llvm-project/blob/main/libc/src/string/string_utils.h#L61
  • We currently have benchmarking for memory functions, math functions, and strtof.
  • Having more benchmarks would be nice.
    * Are we benchmarking to check our own performance or to compare against other libcs?
• It would be nice to make our test suite available to use with other libc implementations
  • (michaelrj) I’d like to follow the design that libc++ uses, which is having the main test suite and marking some parts as testing internal details.
    * This allows us to have tests that check our implementation thoroughly, but also work for other libcs that only match us in the public API.

Just as a heads up: There will be no meeting on November 23rd since it falls on US Thanksgiving, currently the plan is to have the next meeting on December 21st.

Here is the current list of topics for tomorrow’s meeting:

• (michaelrj): Most of the Google contributors are going to be on vacation from Dec 22, 2023 to Jan 1, 2023
• (jhuber): remove the special handling of -ffreestanding
• (jhuber): need a way to build the GPU and CPU libraries in the same LLVM config
• (SchrodingerZhu): wants to know priorities for symbols/syscall wrappers
• (SchrodingerZhu): interested in new buildbot configurations
  • “hermetic build on archlinux fails again because __stack_chk_fail is not included in integration test and hermetic builds”

Here are the notes from today’s meeting:

• -ffreestanding
  • Need to talk to Guillaume (gchatelet)
  • The original reason for the flag was to avoid the compiler calling builtin memcpy inside memcpy
  • -fno-builtin prevents the memcpy inlining, but it also prevents inlining in later, and also causes issues with LTO
    * LTO and inlining are very important in GPU builds, but also anyone else who is building from source
  • Original patch: ⚙ D74162 [Inliner] Inlining should honor nobuiltin attributes
  • It would be nice to have a reproducer so it can be checked if the problem is fixed
• Need a way to build CPU/GPU libraries at the same time
  • Basically we’d need something similar to runtimes which lets you do two similar builds at the same time
  • lntue is working on fixing our cmake, but there’s a lot of “if target_arch”
  • Runtimes already supports this
    * The target triple for your host is the “default” runtimes build, and ends up in the normal spot
    * There’s another cmake variable that has a list of target triples, and this will build those as well.
    * You can also have target specific build flags passed through
    * There’s basic support for multilibs
  • Right now the GPU target isn’t entirely based on the triple, there are also specific arguments it needs.
  • See for example llvm-project/clang/cmake/caches/Fuchsia-stage2.cmake at 11140cc238b8c4124e6f9efacb1601f81da096a0 · llvm/llvm-project · GitHub
  • It would be nice to look into building the libc with FatLTO
  • The runtimes build is being moved away from being as dependent on target triples
  • There is a brute-force version of math tests working on GPUs
• What are the priorities for new development?
  • Syscall wrappers are always useful
  • The symbols necessary to build for libc++ would be nice
  • Dynamic linking/loading isn’t a priority, but it would be nice to have eventually
    * Fuchsia has their own loader which has a compatible license, but hasn’t upstreamed it. That would be a good starting point.

Here’s the current list of topics for tomorrow’s meeting:

• (michaelrj): List of things Google users have asked for:
  • Finishing pthreads
    * (michaelrj) I am currently working on this
  • Finish up stdio design
    * (michaelrj) going to talk to internal stakeholders about this today
    * Previous discussion: [RFC] Modernizing and formalizing the File abstraction interface
  • Finishing floating point functions
    * Double precision, float128, float16?
    * In progress by lntue
  • Fixed point functions
  • Wide character support
  • Embedded support
    * Code size optimization
    * arm32/risc-v 32 support
    * Various non-standard embedded utilities (e.g. sys/queue.h)
  • First-class sanitizer support
    * Maybe phosek can talk about this?
• (michaelrj): What do external users need?

Notes from today’s meeting:

• First-class Sanitizer support
  • Still in the exploration phase
  • Motivated by experience from Fuchsia
  • Existing sanitizers work by intercepting libc calls, which has similar challenges to llvm-libc’s overlay mode
  • The Fuchsia sanitizers use lower level apis which go underneath the libc to use the sanitizers
  • They also instrument most libc functions
  • This experience could be reused for LLVM-libc
  • We could also try to use this for embedded since it would save binary size and performance overhead
• What do external users need
  • Jhuber: needs to finish the freestanding compile flag fix with gchatelet
    * [RFC] libc -ffreestanding / -fno-builtin
    * Doing -fno-builtin isn’t good enough since the GPU builds need to include stdint.h and that may pull in extra pieces without -ffreestanding
    * This may end up needing clang changes
    * We should likely look into making an LTO buildbot
    * This may be a good use case for fat-LTO
    * fat-LTO embeds the bitcode alongside the machine code into the object so that it can be used for optimized linking
    * We should look into disabling loop idiom recognition for certain sections of code
    * We don’t want the memcpy implementation to be turned into a builtin call
    * There is some special handling on some special functions that can’t get eliminated during LTO
    * This isn’t actually useful for GPUs since it’s a holdover from dynamic linking
    * llvm-project/llvm/lib/Object/IRSymtab.cpp at main · llvm/llvm-project · GitHub
    * https://github.com/llvm/llvm-project/pull/73031
  • Mikhail
    * Working on porting to risc-v
    * The buildbot is currently broken, working on fixing it
    * A few patches to hopefully enable fullbuild on risc-v 32
• Build system changes
  • Added -werror
  • Had some troubles since rooting the build in the llvm directory was setting some default flags
    * It might be good to remove LLVM libc from the enable projects list and move it exclusively to runtimes
    * LLVM 18 branch is the 23rd, so we could add a warning that we’re removing it in the next major version (6 months).
    * We’d need to figure out a way to build headergen, or replace it with something that doesn’t need tablegen
  • We’d like to enable crossbuilding fullbuild, but we need to figure out how to set up getting kernel headers
• C23 additions
  • Bitwise operations in the library
    * It would be good to have these since it might be a good differentiator
• Math status
  • GPUs are very interested in float16 support
  • Math smoke tests are off on GPU right now
    * Broken by a recent patch, hopefully fixed soon
    * [libc][math] Add min/max/min_denorm/max_denorm constants to FPBits an… · llvm/llvm-project@d2361b2 · GitHub

Here’s the current list of topics for tomorrow’s meeting:

• (michaelrj): Commit Policy
  • Context: [RFC] Proposed Policy for committing to LLVM-libc
  • It seems like the current path forward is to make waiting for review and passing presubmits strongly recommended
• (michaelrj): “good first issue”
  • First one (filed by SchrodingerZhu):
    * [libc] Support C23 binary notation in `sprintf` · Issue #80727 · llvm/llvm-project · GitHub
    * Completed by a new contributor!
  • Second one (filed by michaelrj):
    * Add bit width length modifier to printf · Issue #81685 · llvm/llvm-project · GitHub
    * Claimed by a new contributor, in progress
  • This is good, we should do more
    * Make sure to have clear instructions on how to fix the issue when you mark it as such
• (michaelrj): Fixed point support
  • An internal user requested this, so it’s the current project
• (michaelrj): new flags system for tests
  • Testing adding _get_common_test_compile_options
• (michaelrj): Possibly migrate to new fuzz test backend?
  • GitHub - google/fuzztest
    * Used for google internal testing
  • Makes fuzz test semantics more unit-test like
  • Would require a dependency on fuzztest and possibly gtest, but only for the fuzz tests.
    * Do people have any problems with this?
• (jhuber) --ffreestanding
  • Context: [RFC] libc -ffreestanding / -fno-builtin - #14 by efriedma-quic
  • This is a priority issue for the GPU build

Notes from today’s meeting:

• Commit Policy
  • People seem in agreement that review being strongly recommended is fine
  • Jhuber: being able to push in an emergency is important
    * Michaelrj: fix forward can be good, but also can break things more, be careful with it
• “Good first issue”
  • We should possibly go looking for more of these since they are good for the project
  • Make issues on github, there’s a page for “good first issue” bugs: https://github.com/llvm/llvm-project/contribute
• Fixed Point:
  • Support for fixed point arithmetic in libc · Issue #74560 · llvm/llvm-project · GitHub
  • Currently in the process of setting up support
  • Compiler support is spotty
  • Leonard Chan has been working on compiler-side support
    * Currently we are only supporting clang 18+ for fixed point, since that’s the only compiler we are sure will work
  • Using fixed points instead of soft floats gives significant performance improvements on Cortex M0s and other embedded targets
  • Fixed point support isn’t in the standard, it’s technically still just a proposal
    * The proposal hasn’t changed since 2006, so it’s likely to be stable
    * Clang and gcc have support currently
    * Gcc requires a recompile to use it
  • As far as we know, there’s currently no open-source libc that supports it right now, so we will be the first
    * Erikson has a toolchain that supports fixed point support, but their libc is proprietary
• Update on dependencies, compile + linking options, and flags propagation
  • Cmake has been fragile
  • lntue has been working through the technical debt
  • Planning on removing all the generator expressions
  • There may be a few small patches, but there will have to be a big patch doing the main change
  • Planning to have a new file for compile options
    * Put the test options in there
• Fuzztest
  • People seem fine with it
  • Current documentation: Fuzzing for LLVM-libc functions — The LLVM C Library
• Freestanding
  • Currently can’t do LTO with a libc since the compiler doesn’t want to inline libc
  • Look into having specific builtin disabling flags on specific functions
  • Jhuber wants ffreestanding since that simplifies the GPU build
    * The whole GPU environment needs to be freestanding
  • Look into making the inliner recognize specific “-no-builtin” functions to avoid inlining them
  • Being able to LTO the entire application is an important feature
• Jhuber: working on updating the GPU build
  • Currently the GPU build has a lot of exceptions in the cmake
  • After some new clang changes, the GPU targets can be a single target
  • Waiting for this patch to land: [LLVM] Add `LLVM__RUNTIME_TARGETS` to set targets per-project by jhuber6 · Pull Request #81557 · llvm/llvm-project · GitHub
    * This will allow the GPU target to be a runtimes cross compile target
    * Might conflict with Petr’s runtimes cmake rewrite, so Petr and jhuber need to coordinate
    * Would cmake cache files simplify this?
    * It would be a different workflow, currently the workflow is to set these options and then add what you (as a user) need
  • Looking into updating the install setup since it’s desirable for GPUs to always overwrite the install target
  • GPU loader: llvm-project/libc/utils/gpu/loader at main · llvm/llvm-project · GitHub
• Future cmake design
  • Currently the cmake assumes we’re building for the host, in future we’d like to make cross compiling and emulator testing easier
  • Ideally we’d match how libc++ and compiler-rt do cross compiling so that we can all work well together
• GSoC
  • Proposals are still open (possibly)
  • Need to set up your idea as a mentor, then get a participant to write up a proposal
  • Deadline for the proposals is end of February
    * LLVM had a soft deadline of February 5th
    * If we want to do it we need to do it as soon as possible.
  • Idea for proposal:
    * Float16 support
    * Work on GPU support?
• C23 support
  • Stdbit is in progress
    * We have most of it done
  • There are internal customers asking for C23 support (since it’s not as available elsewhere)
• Nick is working on some integration for Bionic (Android)
  • Currently working on continuous release
• For emulation
  • ARM FVP? Fixed Virtual Platforms
  • There’s a cmake flag to just run under emulation: CMAKE_CROSSCOMPILING_EMULATOR
    * News to me
  • llvm-project/libc/cmake/modules/LLVMLibCTestRules.cmake at 1343ea126775d77ad97223b2b86aae0dab6c8ac8 · llvm/llvm-project · GitHub
    * This is from the draft: libc support for hexagon linux by androm3da · Pull Request #81815 · llvm/llvm-project · GitHub PR
    * Discussion here: Tutorial: testing libc in riscv32 qemu - #7 by mikhailrgadelha

Here’s the current list of topics for tomorrow’s meeting:

• (michaelrj): Project Hand In Hand
  • A project that will hopefully improve LLVM-libc and libc++ interoperation
  • I have a doc that I will present, then I will post it on Discourse as an RFC with any comments I got from this meeting
  • I’m also hoping to present on this briefly at the libc++ meeting next tuesday
    * Or maybe it’s on the 26th? See Monthly libc++ meeting - #6 by ldionne
    * If it’s on the 26th I’ll probably just post the RFC to the libcxx channel on discord
• (michaelrj): First Issues
  • Currently in progress (claimed and have an open PR)
    * Add bit width length modifier to printf Add bit width length modifier to printf · Issue #81685 · llvm/llvm-project · GitHub
    * Add strfrom functions Add strfrom functions to LLVM-libc · Issue #84244 · llvm/llvm-project · GitHub
    * add support for stdio.h’s rename [libc][c11] add support for stdio.h's rename() · Issue #84980 · llvm/llvm-project · GitHub
  • If the people working on these have questions, this would be a good place to ask

Notes from today’s meeting:

• Hand In Hand
  • Link to RFC: [RFC] Project Hand In Hand (LLVM-libc/libc++ code sharing)
  • What are the goals?
    * We want to close the gap on libc building for libc++
    * We also want to stop duplicating code between the projects
  • Libc++ currently depends on a conforming libc
    * This may be like adding a new libc that it supports
    * Some functions will probably have options to use the system libc, but others may not require this
    * Calling ctype functions on the default c locale is the same in all libcs
    * Similarly, string to float should always give the same result
  • Currently, libc++ can’t be built fully on top of LLVM-libc
    * Embedded targets and some Fuchsia uses
  • The new design would change the libc surface that libc++ uses
    * Some platform independent functions would always come from LLVM-libc
    * Some other functions would still come from the public libc interface
  • This would allow us to share code that can’t go through the public interface
    * The from_chars interface is similar to the strtod interface, but from_chars can’t easily call strtod
  • Libc++ already depends on other directories
    * Specifically libcxxabi and libunwind
    * Adding a new interface will hopefully not be too large
  • Some of the duplicated interfaces are in installed headers
    * We need to (at least initially) avoid touching those since we don’t want to leak the libc internal interface
  • Build system integration
    * We need to be careful of the case where users take just individual folders for individual libraries
    * Examples:
    * Fuchsia uses libc as an individual folder
    * Pigweed uses libc and libcxx as individual slices
    * Chromium builds libcxx as part of chromium
    * We’ll need to decide on include path policies, and warn downstream users so that they can update their build systems
    * Officially libc++ doesn’t support users who take just an individual folder from the monorepo
    * Their policy is basically just copy any extra folders too
  • The existing libcxxabi interface is problematic
    * There is automated code copying, and some parallel headers
    * We don’t want our project to end up like that
  • The libc++ project has previously tried to share code that shows up in the installed headers
    * It was a lot of effort, there’s a lot of macros and other specific code style choices that need to be kept in sync
  • Shared underlying API design
    * Mutexes in libc currently have a flag to mark it as recursive or not
    * This means that lock/unlock always has a branch to check
    * Libc++ has two different types for mutex and recursive mutex
    * Since they sit on top of libc, they have to use the mutex type with flag under the hood, causing performance bottlenecks
    * A shared underlying interface could avoid these bottlenecks
    * Louis Dionne would be very interested in this
  • If the existing C interface is designed in a way that doesn’t work well with C++, it is possible, though difficult, to change the C standard
    * Android went through the process to make this change with pthread clock waits, and got changes in bionic+glibc and libc++ and POSIX
    * the specific “this solves an impedance mismatch problem between libc and libc++” use case was sufficiently obvious+convincing that it “only” took about a year start to end — everybody “got it”.
• Petr Hosek is working on porting code for a baremetal codebase
  • They moved to using clang, LLVM-libc, and are submitting patches for libc++ so they can use it
    * Top level bug: Support use of libc++ with LLVM libc in embedded development · Issue #84879 · llvm/llvm-project · GitHub
  • LLVM-libc has the ability to pick your list of entrypoints for your target
  • Libc++ has the carve out system to remove pieces you don’t need, but there are still things that fall through the cracks
    * Currently working to clean that up
  • Embedded support is a place where both libc and libc++ can make a lot of impact
• Nick Desaulniers: libc and libc++ interface layer
  • Libc++ currently needs ~80 symbols that LLVM-libc doesn’t provide
  • Carve outs can remove some of these, but not all
• SchrodingerZhu Macro policy inside of LLVM-libc
  • Existing:
    * Headers in libc/include have dependencies on macros and types
    * Some macros are defined in UAPI headers, but some we provide ourselves
    * Some macros are defined in llvm-libc-macros, some in API.td
    * For internal usage
    * Current use is split between using the public header vs depending on our internal header directly
  • Future
    * We should try to provide a complete set of macros for C23
    * We may need to use linux UAPI headers for specific macros/types
    * But also some are unusable since they don’t match the userspace interface
    * Linux headers are mostly stable between versions
    * Posix headers often have a lot of implementation defined pieces, we’ll need to keep that in mind
    * For internal usage
    * Types and macros should always be defined in llvm-libc-types/llvm-libc-macros
    * Ideally headergen would only be used for functions
• Nick Desaulniers: Lint rules
  • We have a rule that checks if a header comes from the system
    * We need to clean up code that currently is doing this

Here’s the current list of topics for tomorrow’s meeting:

• Nick Desaulniers: Document policy on assembly
• Nick Desaulniers: Raise minimum supported compiler versions (a la libc++) so that we can Move to C++20
• Michael Jones: New header inclusion design
  • [RFC][libc] Codify header inclusion policy by nickdesaulniers · Pull Request #87017 · llvm/llvm-project · GitHub
• Joseph Huber: -ffreestanding design
  • [RFC] libc -ffreestanding / -fno-builtin - #34 by jhuber6
• Joseph Huber: What is the status of baremetal libc++?

Feel free to send me a message if you have another topic, or bring it up during the meeting.

Notes from today’s meeting:

• Freestanding Design:
  • Inlining isn’t the only problem for doing optimizations with libc
    * Some libc functions are “magic” in that the compiler will identify them and handle them differently
    * E.g. sin is a “nowrite” function
  • We might have to make the compiler not do its magic if the symbol is available
  • The meaning of freestanding may need to be changed from what it currently is
    * The idea would be freestanding turns off libcall optimizations, since any given function (e.g. strlen) should be defined locally and may be whatever the user defines.
    * We may want to have a different flag for “the libc is in the module, but the semantics are the same as normal” so that it can do libcall optimizations, but using the provided libc.
  • Freestanding implies nobuiltins, which can prevent inlining in LTO
  • We want inlining, libcall optimizations and LTO, but the compiler generally doesn’t handle having the libc built with the user code well
    * We don’t want a libc function to get inlined, then loop idiom recognized back to a call of the libc function
    * E.g. call to strlen gets inlined, then the optimizer sees the loop for strlen and turns it back into a call to strlen
    * One design for this would be to have the optimizer mark which functions it’s touched (“poisoning” it) so it doesn’t try to do more to it
  • In different stages, we need to handle these functions differently
    * During constant folding (for example) we still want these functions to be treated as magic (e.g. sin(pi) should still be the constant 0)
    * During LTO (for example) we want these to just be regular functions
    * Maybe inlined or otherwise transformed for better performance
  • It may be useful to have a way to mark a function as the “canonical” version of a libc function
    * That way the compiler can treat it as the abstract libc function
    * But also the compiler might have multiple copies of the same function so it can be optimized/inlined as a regular function as well
  • The linux kernel has already handled some of these issues so that it can have LTO and libcall optimizations, but there are specific kludges to get that working.
• Baremetal libc++
  • Joseph would like to use something similar on the GPU
  • Libc++ has been linked against a baremetal project using LLVM-libc
    * It involved many carve outs and some changes
    * Overarching bug: Support use of libc++ with LLVM libc in embedded development · Issue #84879 · llvm/llvm-project · GitHub
  • There are some places where the carve outs being more fine grained would help for baremetal
• Raising Compiler Version
  • Currently the compiler version we require is based on what our buildbots have
    * The builtbots got updated recently, so the minimum version would need to be updated
  • If we want to move to C++20 we need to increase the minimum compiler version
  • The proposal is to move to something like what libc++ does, which is the most recent two versions of clang and the most recent version of GCC
    * Matching libc++ would let us move towards LLVM runtimes having a unified support window
  • So far we don’t have any users clamoring for old compiler support
  • The GPU server code needs to be built in the libc projects phase
    * This kinda puts it into the overall LLVM compiler support window, which goes back to much older versions.
    * It also reaches into libc sources right now, including touching the printf converters which have a lot of dependencies in __support
  • While libc is still available in projects, we should probably survey the linux distributions for which compilers we can use
  • LLVM 19 will be branching in July, so we could add a warning to not use the libc as a project now and then remove project support after the branch point.
    * The GPU build might be able to make this deadline, but it depends on the offloading runtime. There’s complexity there.
• Policy on assembly
  • The plan is to avoid assembly for performance (since it tends to be brittle)
    * It’s generally better to improve the compiler instead
  • For functions that specifically need assembly (E.g. setjmp/longjmp, syscall) we need to decide how we’ll write that
    * Things like startup code will also need assembly

Here’s the current list of topics for tomorrow’s meeting:

• Improving extern build integration support (Roland McGrath)
  • Header generator vs external builds
  • File / header lists
  • Per-function test vs specific test file support
• LibcTest.h vs gtest/gmock APIs (Roland McGrath)
  • Ability to use EXPECT_ /ASSERT_ in helper functions (outside fixture methods).
  • Custom matcher API: explainError vs MatchAndExplain, etc.
  • Death tests.
• Builds (Roland McGrath)
  • Would like to only support -DLLVM_ENABLE_RUNTIMES=”libc” and …/runtimes
    * No more -DLLVM_ENABLE_PROJECTS=“libc”
    * Libc-hdrgen
    * No more …/llvm
    * Documentation is all over the place.
    * Too many build configurations to test
    * Can we make the bootstrap build explicit?
• Fenv tests (Roland McGrath)
• Landing the LTO/inlining fix soon (Joseph Huber)
• Performance Testing (Joseph Huber)
• Libc++ status (Joseph Huber)

Since this has more topics than usual, I’m planning to try to get through them relatively quickly.

Notes from today’s meeting:

• Extern build integration support:
  • Users of the LLVM-libc code but providing their own build system
  • Header Generator:
    * Hasn’t been a problem yet since users haven’t been using fullbuild, but they’d like to move to fullbuild in the future
    * External users want a more standalone header generator
    * I (Michael) am hosting interns this summer, they start in ~2 weeks and will be rewriting the header generator to be more standalone
  • File organization
    * Currently external builds need to hand maintain their build system
    * Every time a new file is created they need to add it manually
    * New interdependencies cause a lot of issues
    * Most of the code has a uniform structure
    * Headers, sources, and tests all have a consistent pattern
    * But there are exceptions, e.g. new fenv functions
    * There are also some tests that depend on multiple functions.
    * It would be good for us to try to keep that uniform structure.
    * It would be nice to have a way to export the dependencies to simplify the downstream build maintenance
    * Something machine ingestible
    * It would also be nice to have annotations for what dependencies a function needs (e.g. OS, architecture)
    * This could also potentially be used by bazel
    * Enforcing fine grained dependencies inside cmake is in progress
    * Tue is working on it
    * Once this is done there should be a cmake tool to print out the dependencies
    * Ideally the bazel build will be moved to match the structure of the existing libc directory.
• Libctest
  • Not a subset of gtest anymore, it’s just a similar framework that’s standalone at this point
  • There are some incompatibilities,
    * specifically in gtest all the assert macros are allowed to be within sub functions, whereas our current framework doesn’t.
    * The custom matchers also don’t work as gtest matchers
  • We should probably move to being gtest compatible
    * If we do, we should have an upstream gtest buildbot
  • Roland is working on some patches to fix the assert macro issue, but would like some help on the matchers.
• LTO/Inlining change
  • Landed but reverted due to issues in the AMDGPU backend
    * Seems like a bug in the backend
  • The suggested solution is if we’re using LLVM as a linker when we have a libcall function in the module it can’t be treated as a black box as normal, since it might have changed semantics from optimizations. Everything gets marked as nobuiltin
    * (notes may be a bit inaccurate, look at the source change)
  • After this change the inliner will allow nobuiltin inlining
  • The Change: [Linker] Propagate `nobuiltin` attributes when linking known libcalls by jhuber6 · Pull Request #89431 · llvm/llvm-project · GitHub
  • Discourse RFC: [RFC] libc -ffreestanding / -fno-builtin
• Performance testing
  • There’s some code in tree for performance testing already, but it might not be usable for GPUs
    * The existing method on GPUs requires a significant amount of inline asm to measure clock cycle latency
    * It might need to be put in a GPU subdirectory
  • The end goal is to measure the latency of things like math calls on the GPU.

Here’s the current list of topics for tomorrow’s meeting:

• Welcome interns! (michaelrj)
  • They’ll be working on redoing Headergen
• New headergen (michaelrj)
  • Probably written in python with spec files in json for ease of maintenance.
• Math testing on GPUs (jhuber)
• libc/libc++ baremetal config (jhuber)
• Anything else people want to bring up

Notes from today’s meeting:

• New headergen
  • Design next week
  • Writing json by hand is painful
    * Fuchsia uses fidl that gets converted to json for the actual code
  • Look into yaml
  • Elliot is more interested in taking just the implementation and handwriting his own headers (because he’s got 15 years of history he can’t change ), so headergen isn’t a huge deal for him.
  • Clang has a feature called extract API
    * It parses headers and spits out json representing that API
    * If we use the same representation then we could share tooling
    * Definitely look into it
    * Useful for extracting metadata on different libc’s function annotations
  • libc-coord mailing list for folks working on libcs to talk to one another: libc-coord mailing list
• Math testing on GPUs
  • Goal is to use all of the LLVM-libc math functions on GPUs once they’re available
  • Before they’re rolled out it would be good to run the brute force math tests
  • We could compare the CPU version of LLVM-libc to the GPU version
    * We’ve already tested the CPU version against MPFR, so if the GPU version matches we know it’s good.
  • Right now the blocker is separating out the cmake configuration for CPU and GPU
  • There’s also issues around dependency chains when using the offloading runtime
  • GPUs are also interested in float16, so that work will be helpful
• Baremetal libc++/libc config
  • Petr has a local config that works (but isn’t finished) for building libc++ for baremetal
    * It’s a configuration that turns off pretty much everything possible in libc++
    * Can put it up on github so that people can look at it
  • There’s some cleanup needed to use libc++ and LLVM-libc together even for baremetal
  • Libc++abi expects malloc which is a problem for baremetal, so we might need to create a switch to turn that off
    * There’s also a baremetal malloc being added to LLVM-libc by Leonard
    * It’s a basic freelist allocator
    * It would be good to have multiple options for baremetal allocators since there are a variety of uses.
  • There’s some talk of merging libc++abi back into libc++ so that might help with this
• Math_errhandling
  • Historically math functions set errno, but c99 added the math_errhandling macro to show to the user how math functions report errors
    * They can set errno, or do floating point exceptions
  • The compiler also has a switch for if math functions are expected to set errno, which allows it to inline math functions if they won’t set errno.
  • Our existing implementation is similar to glibc and if we’re in no math errno then we just use floating point exceptions, else use errno.
    * We also have cutouts for platforms that don’t have fpexceptions
    * We need to make sure that if the macro says errno will be set then we do that
    * The default for linux is math sets errno, but pretty much every other platform doesn’t set errno by default.
    * (Android sets WANT_ERRNO=0 when building arm-optimized-routines’ libm routines, because bionic doesn’t set errno from libm)
  • We should make sure that as part of the target configuration we choose how math error handling works
    * Needs two switches: one for fp exceptions and one for errno
    * We also need to make sure that the compiler switch doesn’t override our macro for how math errno works since setting the compiler switch when building user code doesn’t actually change the libc behavior
  • Possible future area of expansion: provide fast math versions of libm functions, and use those instead of the normal accurate ones when fast-math is set.

NOTE: THIS MEETING IS HAPPENING ONE DAY EARLY (9am on July 3rd) TO AVOID JULY 4TH

Here’s the current list of topics for tomorrow’s meeting:
Topics

• Progress update on new headergen (rosezhang and aaryanshukla)

Feel free to bring up topics day-of, it looks like we’ll have extra time.

Additional Topics:

• Math optimization options (libc’s “fast math”) (lntue)
  • Current located in src/__support/macros/optimization.h
  • Current list:
    * LIBC_MATH_SKIP_ACCURATE_PASS
    * LIBC_MATH_SMALL_TABLES
    * LIBC_MATH_NO_ERRNO
    * LIBC_MATH_NO_EXCEPT
    * LIBC_MATH_FAST
  • Control macro:
    * LIBC_MATH
  • Future options:
    * LIBC_MATH_FLOAT_INTERMEDIATE
    * LIBC_MATH_HALF_INTERMEDIATE
  • Q: Any other options?
  • Q: Move them to a public header?
• Embedded libc (Volodymyr Turanskyy)
• How to handle visibility for API functions (jhuber)

Notes

• New Headergen
  • Inside python it’s represented by classes
  • Currently the specs are in yaml files
  • On track to get things done in time for the internship
    * In the process of landing code into LLVM
• Math optimization options (libc’s “fast math”) (lntue)
  • Some math functions currently have options that make them faster at the cost of accuracy
    * Specifically the new sin/cos functions have several options and they could be expanded to more functions
  • The amount of error introduced by skipping the accurate pass has been measured
    * For GPUs, the maximum error they can accept is defined in the OpenCL standard: The OpenCL™ C Specification
    * E.g. sin has a max of 4 ULP (Unit of Least Precision) error
  • Generally the fast path is accurate in almost all cases, but it may be incorrectly rounded
  • Need to make sure that the flags also change the tests to match
    * E.g. disable rounding tests if the rounding isn’t accurate
  • We might want to export these options into the public headers so that the user can tell how the libc was configured
    * Design TBD
    * Could be put in features.h, which already is used to inform the user about the libc’s capabilities.
    * Possibly use new headergen to set up these macros
    * One of the macros should probably be “default rounding mode” for platforms where rounding mode has to be set at build time
    * E.g. nvidia GPUs use different instructions for operations in different rounding modes
  • It may also be good to create math functions where you pass the rounding mode as an argument
    * Similar to how isdigit_l takes a locale
    * Hopefully get this standardized
  • Next math functions to get implemented will be cube root then inverse trig functions
• Embedded libc
  • When setting up ARM’s embedded toolchain they considered LLVM-libc
    * It wasn’t complete enough at the time to use it
    * But in the meantime we’ve added a bunch of functions so they’re going to evaluate it again
    * The biggest concern is around code size
  • Fuchsia/Pigweed has been using LLVM-libc in embedded for a bit now
    * They haven’t had a good public platform in the past, but now they’ve been working on getting the Raspberry Pi Pico working
    * It’s got a lot of useful features, but also it’s got a Cortex M0 core which is very similar to the actual CPU in real products
    * It’s close to building with vanilla clang+LLVM-libc, just missing a couple pieces
    * Next up is hopefully a buildbot that tests this configuration on a set of Picos
    * Pigweed is also using LLVM-libc in some products
    * Binary size has been an improvement over the previous toolchain
    * They haven’t been using much floating point math, so that may be a concern
    * But we added fixed point math LLVM-libc which works better for embedded systems, usually both reducing size and improving performance.
    * Two big parts of LLVM-libc that are missing for embedded right now:
    * Semihosting for stdio
    * Very useful during bringup
    * Threading support
    * Not sure what the design will be, since threading is very platform specific
    * Probably have an ABI that allows vendors to plug in their own threading support similar to libc++
    * Recently added a simple freelist allocator for embedded
    * Not perfectly optimal, looking into replacing it with a TLSF (Two-Level Segregate Fit) allocator
• Status of AOR
  • Added many years ago, may be out of date now
  • A lot of these functions are implemented in assembly, which doesn’t match the style of LLVM-libc
  • It may be useful to remove the parts of AOR that we aren’t using
  • From the ARM side there may be interest in contributing
• Visibility for API functions
  • We should probably mark all the internal symbols as hidden
  • The GPU currently compiles with -fvisibility=hidden
  • For other platforms we don’t currently do this but we probably should
  • If we do that we’d also need to mark the public functions with [[gnu::visiblity(“default”)]]
    * We also need to mark the public globals (e.g. stdout)
  • Probably create a macro like #define LLVM_LIBC_VISIBILITY [[gnu::hidden]]
  • The decision will probably come down to vendors
    * Hidden visibility is good for static linking cases, but dynamic linking may want default visibility.
  • There are some pieces that should be always hidden, e.g. the internal cpp namespace
    * It may be best to put the visibility annotations on namespaces.
  • Whatever we decide it needs to be configurable, probably through a macro and config
  • Visibility is used in LTO to determine if something can be fully internalized
• -stdlibinc vs -ffreestanding
  • -ffreestanding is similar to -nostdinc -fnobuiltin which causes issues in lto
• GPU and baremetal sharing code
  • For rand_utils they are probably going to be using the same code
  • The GPU RPC system may be used as a basis for something like semihosting for baremetal