Use probe-stack=inline-asm in LLVM 11+ by erikdesjardins · Pull Request #77885 · rust-lang/rust (original) (raw)

Looks like a comparison is inverted. This is a code snippet from by-value-trait-object-safety-rpass. Immediately before this, the size of the alloca is loaded into ebx, which in our case is 0 because the unsized trait object has size 0.

add	ebx, 15
and	ebx, -16
mov	eax, esp
sub	eax, ebx
mov	dword ptr [ebp - 368], edx
mov	dword ptr [ebp - 372], esi
mov	dword ptr [ebp - 376], edi
mov	dword ptr [ebp - 380], eax

.LBB87_43: mov eax, dword ptr [ebp - 380] cmp eax, esp jl .LBB87_45 mov dword ptr [esp], 0 sub esp, 4096 jmp .LBB87_43 .LBB87_45: mov eax, dword ptr [ebp - 380] mov esp, eax

Due to:

cmp eax, esp jl .LBB87_45

...if ebx is nonzero, the new stack pointer in eax will be below esp, and it will exit the loop without probing.
...if ebx is zero, the new stack pointer in eax will be equal to, and then greater than esp, and it will probe infinitely.
(and that's what happens running it locally under gdb)

The same thing shows up in Clang 11 with -fstack-clash-protection (https://godbolt.org/z/vxbaxd) with just:

int size;

void foo(void*);

int main() { foo(alloca(size)); }

I suppose nobody noticed this in C because it's rare to call alloca(0), and alloca(<nonzero>) will still run, it just won't actually probe anything.

Assuming I haven't misinterpreted something, can you report this to LLVM? I don't have an account yet.