Self build toolchains for RISC-V lacking correct libc (original) (raw)
I’m trying to bootstrap and compile the toolchains for Vortex, which is an open-source GPGPU based on RISC-V ISA. I followed the README.vortex in the pocl source file pulled from GitHub - vortexgpgpu/pocl at vortex.
#### Compiling pocl on RiscV ####
## Dependencies:
- sudo apt-get -y install \
binutils build-essential libtool texinfo \
gzip zip unzip patchutils curl git \
make cmake ninja-build automake bison flex gperf \
grep sed gawk python bc \
zlib1g-dev libexpat1-dev libmpc-dev \
libglib2.0-dev libfdt-dev libpixman-1-dev
## Setting tools directory
export TOOLDIR=$HOME/tools
## Building RiscV GNU Toolchain (gcc, binutils, etc..)
- git clone --depth=1 --recursive https://github.com/riscv-collab/riscv-gnu-toolchain.git
- mkdir build && cd build
- export CPATH=$TOOLDIR/GNU/include
- export LIBRARY_PATH=$TOOLDIR/GNU/lib
- ../configure --prefix=$TOOLDIR/riscv32-gnu-toolchain --with-cmodel=medany --with-arch=rv32imf --with-abi=ilp32f
#../configure --prefix=$TOOLDIR/riscv64-gnu-toolchain --with-cmodel=medany --with-arch=rv64imafd --with-abi=lp64d
- make -j`nproc`
- make -j`nproc` build-qemu
## Building LLVM for Vortex
- git clone --recursive --branch vortex https://github.com/vortexgpgpu/llvm.git
- cd llvm
- mkdir build && cd build
- export LLVM_PREFIX=$TOOLDIR/llvm-vortex
- export RISCV_TOOLCHAIN_PATH=$TOOLDIR/riscv64-gnu-toolchain
- cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$LLVM_PREFIX -DLLVM_ENABLE_PROJECTS="clang;lld" -DBUILD_SHARED_LIBS=True -DLLVM_TARGETS_TO_BUILD="RISCV" -DLLVM_ABI_BREAKING_CHECKS=FORCE_OFF -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF -DDEFAULT_SYSROOT=$RISCV_TOOLCHAIN_PATH/riscv32-unknown-elf -DLLVM_DEFAULT_TARGET_TRIPLE="riscv32-unknown-elf" ../llvm
- make -j`nproc`
- make install
## Sanity test your new RISC-V LLVM
- echo -e '#include <stdio.h>\n int main(void) { printf("Hello world!\\n"); return 0; }' > hello.c
- clang hello.c
- qemu-riscv32 hello
## Building llvm-spirv
- git clone --depth=1 -b release/10.x https://github.com/KhronosGroup/SPIRV-LLVM-Translator.git
- cd llvm-spirv
- mkdir build && cd build
- export LLVM_VORTEX=$TOOLDIR/llvm-vortex
- cmake .. -DLLVM_DIR=$LLVM_VORTEX -DCMAKE_INSTALL_PREFIX=$LLVM_VORTEX
- make llvm-spirv -j`nproc`
- make install
# manually copy over llvm-spirv binary
- cp ./tools/llvm-spirv/llvm-spirv $LLVM_VORTEX/bin
## build POCL compiler
- git clone --branch vortex --recursive https://github.com/vortexgpgpu/pocl
- cd pocl
- mkdir build_cc && cd build_cc
- export POCL_CC_PATH=$TOOLDIR/pocl/compiler
- export LLVM_VORTEX=$TOOLDIR/llvm-vortex
- cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$POCL_CC_PATH -DOCS_AVAILABLE=ON -DWITH_LLVM_CONFIG=$LLVM_VORTEX/bin/llvm-config -DENABLE_VORTEX=ON -DBUILD_TESTS=OFF -DPOCL_DEBUG_MESSAGES=ON -DENABLE_ICD=OFF -DCLANG_MARCH_FLAG:STRING=-mcpu= -DLLC_HOST_CPU=generic-rv32 ..
- make -j`nproc`
- make install
## build POCL runtime
- git clone --branch vortex --recursive https://github.com/vortexgpgpu/pocl
- cd pocl
- mkdir build_rt && cd build_rt
- export POCL_RT_PATH=$TOOLDIR/pocl/runtime
- export VORTEX_DRIVER_INC=$HOME/dev/vortex/runtime/include
- export VORTEX_DRIVER_LIB=$HOME/dev/vortex/build/runtime/stub/libvortex.so
- cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DHOST_DEVICE_BUILD_HASH=riscv32-unknown-unknown-elf -DCMAKE_INSTALL_PREFIX=$POCL_RT_PATH -DOCS_AVAILABLE=OFF -DENABLE_LLVM=OFF -DVORTEX_DRIVER_INC=$VORTEX_DRIVER_INC -DVORTEX_DRIVER_LIB=$VORTEX_DRIVER_LIB -DENABLE_VORTEX=ON -DBUILD_TESTS=OFF -DPOCL_DEBUG_MESSAGES=ON -DENABLE_ICD=OFF ..
- make -j`nproc`
- make install
- cp -r ../include $POCL_RT_PATH
And I built the source as follows:
- RiscV GNU Toolchain
- LLVM for Vortex
- llvm-spirv
- POCL compiler
Then I encountered a problem when testing my new RISC-V LLVM as the README.vortex says. The error log showed as follows:
ld.lld: error: unable to find library -lclang_rt.builtins-riscv32
clang-16: error: ld.lld command failed with exit code 1 (use -v to see invocation)
As the error reported, I built compiler-rt in llvm standalone to get the libclang_rt.builtins-riscv32.a and rebuilt hello.c :
$ ~/share/vortex/toolchain/out/llvm-vortex/bin/clang -march=rv32imf -mabi=ilp32f hello.c
It returned with new errors:
ld.lld: error: undefined symbol: _close
>>> referenced by closer.c
>>> libc_a-closer.o:(_close_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
>>> did you mean: fclose
>>> defined in: /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a(libc_a-fclose.o)
ld.lld: error: undefined symbol: _exit
>>> referenced by abort.c
>>> libc_a-abort.o:(abort) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _fstat
>>> referenced by fstatr.c
>>> libc_a-fstatr.o:(_fstat_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _isatty
>>> referenced by isattyr.c
>>> libc_a-isattyr.o:(_isatty_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _lseek
>>> referenced by lseekr.c
>>> libc_a-lseekr.o:(_lseek_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _read
>>> referenced by readr.c
>>> libc_a-readr.o:(_read_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _kill
>>> referenced by signalr.c
>>> libc_a-signalr.o:(_kill_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _getpid
>>> referenced by signalr.c
>>> libc_a-signalr.o:(_getpid_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _sbrk
>>> referenced by sbrkr.c
>>> libc_a-sbrkr.o:(_sbrk_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
ld.lld: error: undefined symbol: _write
>>> referenced by writer.c
>>> libc_a-writer.o:(_write_r) in archive /home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/riscv32-unknown-elf/lib/libc.a
So I checked the prebuilt toolchains provided by the author of Vortex, and found a file folder named libc32 :
libc32/lib$ tree
.
├── crt1.o
├── crti.o
├── crtn.o
├── libc.a
├── libcrypt.a
├── libdl.a
├── libm.a
├── libpthread.a
├── libresolv.a
├── librt.a
├── libutil.a
├── libxnet.a
├── rcrt1.o
└── Scrt1.o
It seems to be a musl libc built standalone. But there is no document showing how to build it. So my first problem is how to bootstrap build this libc32 myself.
Since I do not know how to build this libc32, I use the prebuilt one instead and build hello.c as follows:
$ ~/share/vortex/toolchain/out/llvm-vortex/bin/clang -march=rv32imf -mabi=ilp32f -L/home/soc_szq/share/vortex/toolchain/out/libc32/lib hello.c
However, it still returns with a warning:
ld.lld: warning: cannot find entry symbol _start; not setting start address
So although it generated an a.out, but test with qemu returned with segmentation fault.
/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/bin/qemu-riscv32 a.out -v
Segmentation fault (core dumped)
Besides I skipped this test and went through the README.vortex to build pocl runtime. Since it needs a library named libvortex.so, I build vortex at first to get it. But build vortex with my self-build riscv32-gnu-toolchain returned with error:
make[3]: Entering directory '/home/soc_szq/share/vortex/toolchain/src/vortex/build/tests/kernel/conform'
/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/bin/riscv32-unknown-elf-gcc -march=rv32imaf -mabi=ilp32f -O3 -mcmodel=medany -fno-exceptions -nostartfiles -nostdlib -fdata-sections -ffunction-sections -I/home/soc_szq/share/vortex/toolchain/src/vortex/kernel/include -I/home/soc_szq/share/vortex/toolchain/src/vortex/build/hw -DXLEN_32 -DNDEBUG /home/soc_szq/share/vortex/toolchain/src/vortex/tests/kernel/conform/main.cpp /home/soc_szq/share/vortex/toolchain/src/vortex/tests/kernel/conform/tests.cpp -Wl,-Bstatic,--gc-sections,-T,/home/soc_szq/share/vortex/toolchain/src/vortex/kernel/scripts/link32.ld,--defsym=STARTUP_ADDR=0x80000000 /home/soc_szq/share/vortex/toolchain/src/vortex/build/kernel/libvortex.a -L/home/soc_szq/share/vortex/toolchain/out/libc32/lib -lm -lc /home/soc_szq/share/vortex/toolchain/out/libcrt32/lib/baremetal/libclang_rt.builtins-riscv32.a -o conform.elf
/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/lib/gcc/riscv32-unknown-elf/15.0.0/../../../../riscv32-unknown-elf/bin/ld: /home/soc_szq/share/vortex/toolchain/src/vortex/build/kernel/libvortex.a(vx_start.S.o): in function `_start':
(.init+0x64): undefined reference to `__libc_fini_array'
/home/soc_szq/share/vortex/toolchain/out/riscv32-gnu-toolchain/lib/gcc/riscv32-unknown-elf/15.0.0/../../../../riscv32-unknown-elf/bin/ld: (.init+0x70): undefined reference to `__libc_init_array'
collect2: error: ld returned 1 exit status
make[3]: *** [../common.mk:43: conform.elf] Error 1
I checked the symbol table of my self-build libc.a in riscv32-gnu-toolchain and it has the entry __libc_fini_array , I wonder why the linker reports there is no entry named `__libc_fini_array’ when I use my self-build riscv32-gnu-toolchain. Here is the symbol table of self-build libc.a in riscv32-gnu-toolchains:
File: libc.a(lib_a-fini.o)
Symbol table '.symtab' contains 16 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 00000000 0 NOTYPE LOCAL DEFAULT UND
1: 00000000 0 FILE LOCAL DEFAULT ABS fini.c
2: 00000000 0 SECTION LOCAL DEFAULT 1 .text
3: 00000000 0 SECTION LOCAL DEFAULT 2 .data
4: 00000000 0 SECTION LOCAL DEFAULT 3 .bss
5: 00000000 0 SECTION LOCAL DEFAULT 4 .text.__libc_fin[...]
6: 00000000 0 NOTYPE LOCAL DEFAULT 4 $xrv32i2p1_m2p0_[...]
7: 00000008 0 NOTYPE LOCAL DEFAULT 4 .L0
8: 00000010 0 NOTYPE LOCAL DEFAULT 4 .L0
9: 00000048 0 NOTYPE LOCAL DEFAULT 4 .L1
10: 00000034 0 NOTYPE LOCAL DEFAULT 4 .L3
11: 00000000 0 SECTION LOCAL DEFAULT 6 .comment
12: 00000000 0 SECTION LOCAL DEFAULT 7 .riscv.attributes
13: 00000000 92 FUNC GLOBAL DEFAULT 4 __libc_fini_array
14: 00000000 0 NOTYPE WEAK DEFAULT UND __fini_array_start
15: 00000000 0 NOTYPE WEAK DEFAULT UND __fini_array_end
If I use the prebuilt riscv32-gnu-toolchains provided by the author, the vortex source can be built correctly. I wonder whether this error comes with the mismatch of the prebuilt libc32 and my self-build gnu toolchains, or the GNU toolchain built following the README.vortex just could not work. And why the linker of my self-build toolchain could not find the entry named __libc_fini_array.