ill-typed unused FFI declarations can cause UB · Issue #46188 · rust-lang/rust (original) (raw)

This compiles and prints "p is not null and 0x0":

pub mod bad { #[allow(improper_ctypes)] extern { pub fn malloc(x: usize) -> &'static mut (); }

#[no_mangle]
pub fn bar() {
    let _m = malloc as unsafe extern "C" fn(usize) -> &'static mut ();
}

}

pub mod good { extern { fn malloc(x: usize) -> *const u8; }

pub fn foo() {
    unsafe {
        let p = malloc(0x13371337deadbeef); // your computer doesn't have enough memory
        if p.is_null() {
            panic!("p is null");
        } else {
            panic!("p is not null and {:?}", p);
        }
    }
}

}

fn main() { bad::bar(); good::foo(); }

The problem is that we have two declarations of the "malloc" symbol, but LLVM uses a global namespace for these. So during codegen, the 2nd declaration we generate overwrites the first. In this case, the "ill-typed" malloc declaration (bad::malloc) comes last, up putting a nonnull attribute on malloc, which causes mod good to be miscompiled.

Here's another example that does not involve malloc. It does not get miscompiled currently, but it demonstrates the issue.

See here for a document that summarizes why this happens, and what our options are.