Build options for Valgrind (original) (raw)

I’m aware of building OpenMP with -DLIBOMP_TSAN_SUPPORT=1 for TSAN. And I’ve started looking at libarcher - is there a manual fof that anywhere?

Is there any equivalent to allow LLVM OpenMP to work with Valgrind DRD and Helgrind? GCC OpenMP has --disable-linux-futex which causes it to use pthread functions rather than directly using futex.

(Asking as a Valgrind developer).

JCownie July 15, 2024, 11:35am 2

I do not see an explicit option in the LLVM runtime code, however, you should be able to achieve it from the CMAKE file if you add an option to define the KMP_USE_FUTEX macro as 0.

The important code is in kmp_lock.h, (here)

#ifndef KMP_USE_FUTEX
#define KMP_USE_FUTEX                                                          \
  (KMP_OS_LINUX &&                                                             \
   (KMP_ARCH_X86 || KMP_ARCH_X86_64 || KMP_ARCH_ARM || KMP_ARCH_AARCH64))
#endif

Here you can see it checking whether the macro is already defined (which it normally won’t be, but you’re about to change that :-)), and then setting it depending on the OS and target architecture.

So, by setting it to 0 yourself you should be able to eliminate the use of futexes.

However, that may not solve your real problem, since it won’t make the runtime use pthread locks, but rather one of its own lock implementations.

If you really need it to use pthread locks, then you would need to implement the necessary shims, or, maybe you could instrument the existing locks.

HTH

– Jim

I’ll take a look. Using pthreads would be easiest. Otherwise I need to be able to intercept the equivalent lock functions.

JCownie July 16, 2024, 9:51am 4

I think there is a more general issue here, though, which is that locks are not the only OpenMP synchronisation construct. In particular, barriers, single and master constructs also exist, which are likely not implemented using pthread locks. So even if you ensure that the OpenMP lock calls ultimately use pthread locks that isn’t enough to track thread safety.

Sure. I was just keeping things simple.

JCownie July 17, 2024, 8:49am 6

Keeping things simple is fine, but it doesn’t help us to work out what the best overall approach might be if we’re only thinking about a small part of the problem…

For instance, adding support to instrument the OpenMP level calls to the lock routines (which are in the standard, so can’t change their names) would be a general solution for both the LLVM and GCC runtimes, but doesn’t address the other issues where the names of the relevant functions for barriers, fork, join, … are not specified by the OpenMP standard, so each runtime uses its own naming convention.

How are those handled for the GCC OpenMP runtime support?

Valgrind doesn’t work with GCC OpenMP out of the box.

It needs a build of GCC OpenMP with --disable-linux-futex at configure time.

JCownie July 17, 2024, 9:07am 8

Valgrind doesn’t work with GCC OpenMP out of the box.

It needs a build of GCC OpenMP with --disable-linux-futex at configure time.

Yes, that was already clear, but it doesn’t address the issue of how you are handling all of the other, non-lock related, synchronisations which you need to be tracking, and which are presumably unaffected by whether futexes are used or not.

I am trying to understand what the whole job is to get LLVM’s OpenMP implementation working with Valgrind/Helgrind, and how that can best be achieved.
Clearly {Val,Hel}grind do not need modifications to libpthread, or libc, so you have mechanisms for handling relevant functions and understanding their semantics that do not require changes to the underlying libraries.

I’m therefore wondering whether adding more information to {Val,Hel}grind might be a better approach. Even just intercepting the OpenMP lock functions ought to remove the need for a non-standard GCC runtime build, which seems a plus.

The way that Valgrind works for redirection and wrapping is very much brute force. It sees all mmap system calls, used to load shared libraries. When it sees an ELF library being loaded it reads the ELF headers and looks at all the functions. It records all those that need to be wrapped or redirected. Then when the guest is executing any function calls that correspond to the wrapped/redirected functions.

In GCC --disable-linux-futex sets the LIBGOMP_USE_PTHREADS macro, which as far as I know causes gomp to only use pthread interfaces.

JCownie July 17, 2024, 12:03pm 10

The issue, though, is where to apply the brute force

Part of the issue is what “only using pthread interfaces” means. There may be operations which are not sanely available via pthread interfaces, but are required by the OpenMP semantics.

It’s not as if the issue is “Don’t make system calls directly, but go through the libc wrappers”, since there there are no other options, as you ultimately have to ask the kernel to do something, whereas here you don’t.

So the issue is wider than just the lock functions, and what is needed in the other areas matters too.

If it were just the lock functions, then applying your brute force approach to those would give you a solution for any OpenMP runtime and require no changes to the runtime itself, which seems quite attractive. Unfortunately I doubt that it’s enough.

(FWIW, I’m off on holiday tomorrow, and won’t be back until Wednesday, so I’m unlikely to reply rapidly :-))

As I understand it GCC OpenMP built with --disable-linux-futex really means ‘use the pthread API’. Historically there was a gomp barrier wrapper but it was too unstable and got removed.

If there’s no way to build LLVM OpenMP so that it only uses pthreads I’ll have to look at alternatives, probably starting by looking at how libarcher sits between OpenMP and TSAN.

There’s no urgency for this. Valgrind development advances at a leisurely pace.

jprotze October 27, 2025, 1:32pm 12

I would be interested in extending Archer, so that it can also work with valgrind.

If there’s no way to build LLVM OpenMP so that it only uses pthreads I’ll have to look at alternatives, probably starting by looking at how libarcher sits between OpenMP and TSAN.

libarcher understands OpenMP synchronization semantics and injects them to TSAN by calling the annotation functions AnnotateHappensBefore, AnnotateHappensAfter, AnnotateIgnoreWritesBegin, AnnotateIgnoreWritesEnd.

Because libarcher should be dynamically loadable and not fail for some missing functions, all function calls are done using function pointers after lookup with dlsym.

In the current version, if a function is not available (like in the valgrind setup), the pointer points to a template function: llvm-project/openmp/tools/archer/ompt-tsan.cpp at main · llvm/llvm-project · GitHub

This means, that in the current version, you would not be able to distinguish happensBefore from happensAfter calls.

The second issue with the current version is that it always deregisters itself, if TSan is not present. We can easily workaround this issue by adding an env variable.