What is the best way to JIT compile C++ code contained in a string? (original) (raw)
February 28, 2024, 10:08pm 1
Background
I have a code and framework that I’m working on which generates C++ source code for GPU kernels and I need a CPU fallback for systems which don’t support CUDA or HIP, or for when I need to use double or complex types on Metal GPUs. Currently what I’m doing is writing the code to disk foreach thread, fork a process and call the system compiler to generate a dynamic library which I load using dlopen/dlclose. Here an example code with some things omitted for brevity.
void compile(const std::string kernel_source) {
// Create a name for the source code.
std::ostringstream temp_stream;
temp_stream << reinterpret_cast<size_t> (this);
const std::string thread_id = temp_stream.str();
temp_stream.str(std::string());
temp_stream.clear();
temp_stream << "temp_" << thread_id << ".cpp";
const std::string filename = temp_stream.str();
// Write source code to disk.
std::ofstream out(filename);
out << kernel_source;
out.close();
temp_stream.str(std::string());
temp_stream.clear();
// Create a name for the shared library.
temp_stream << "./" << filename << ".so";
library_name = temp_stream.str();
temp_stream.str(std::string());
temp_stream.clear();
// Generate command line
temp_stream << CXX << " " << CXX_FLAGS;
temp_stream << filename << " -o " << library_name;
// Compile the library.
auto pid = fork();
int error = 0;
if (pid == 0) {
auto args = split_string(temp_stream.str());
auto c_args = to_c_str(args);
error = execvp(c_args[0], c_args.data());
}
waited(pid, &error, 0);
// Load the library
lib_handle = dlopen(library_name.c_str(), RTLD_LAZY);
}
From here I load the kernel and return lambda to call it. Note that here the resulting kernels can have different numbers of arguments. I’m using a std::map so I can have consistent function signature which I then unpack inside the kernel.
std::function<void(void)> create_kernel_call(const std::string kernel_name,
graph:arg_nodes<T> args) {
// Load kernel function pointer.
void *kernel = dlsym(lib_handle, kernel_name.c_str());
// Create argument map.
std::map<size_t, T *> buffers;
for (auto &arg : args) {
buffers[reinterpret_cast<size_t> (arg)] = kernel_arguments[arg].data();
}
// Create a callable function.
return [kernel, buffers] () mutable {
((void (*)(std::map<size_t, T*> &))kernel)(buffers);
}
}
Where a kernel_source string looks something like
#include <map>
using namespace std;
extern "c" void example_kernel(std::map<size_t, float *> &args) {
const float v_105553159292360 = args[105553159292360];
const float o_105553150985176 = args[105553150985176];
const float o_105553150985816 = args[105553150985816];
for (size_t i = 0; i < 100000; i++)
const float r_105553159292360 = v_105553159292360[i];
const float r_105553150985816 = 10;
const float r_105553150985416 = -0.300000012;
const float r_105553150985336 = r_105553159292360 + r_105553150985416;
const float r_105553150985176 = r_105553150985816*r_105553150985336;
o_105553150985176[i] = r_105553150985176;
o_105553150985816[i] = r_105553150985816;
}
}
This works well enough most of the type but on some systems, dlclose fails to unload. Then a different library can be generated with the same name and dlopen fails to load it.
Using LLVM for JIT
I want to eliminate the need to explicitly write the source to disk and avoid loading the library using the dlopen\dlclose functions. I would also like a better way to handle the fact that the number of kernel arguments can vary in a better way without using std::map.
However, I’m a bit overwhelmed by the JIT user guide and unsure if should focus on MCJIT or ORC. I would be calling this from multiple threads but in a manner where each thread is compiling independent kernels. Some of the kernels will use functions from <cmath> and <complex>. I would need these to apply optimizations as if I was compiling with -O3.
It looks like the LLJIT would replace most of what I’m doing.
auto JIT = LLJITBuilder.create();
JIT->addIRModule(Some thing that converts a string of c++ code to IR module);
auto EntrySym = JIT->lookup("kernel_name");
auto *kernel = EntrySym.getAddress().toPtr<void(*)()> ();
How do I obtain IR for C++ code. I assume this would involve clang libraries? Does this automatically apply the same -O3 optimizations or would I need to apply those when generating the IR? Does this manually handle the run time libraries? Or would I need to use something lower level?
Are there any up-to-date examples of JIT compiling C++ code?
After tripping over llvm::ErrorOr and llvm::Expected a few times, I was able to get a working hello world level example working. I’m still forking to system compiler to generate but I was able to load a function and run it.
#include <string>
#include <fstream>
#include <array>
#include <iostream>
#include <unistd.h>
#include <sys/wait.h>
#include "llvm/Support/TargetSelect.h"
#include "llvm/ExecutionEngine/Orc/LLJIT.h"
int main(int argc, const char * argv[]) {
llvm::InitializeNativeTarget();
const std::string source(
"#include <iostream>\n"
"extern \"C\" void hello() {\n"
" std::cout << \"Hello World\" << std::endl;\n"
"}"
);
std::ofstream out("hello.cpp");
out << source;
out.close();
auto pid = fork();
int error = 0;
if (pid == 0) {
auto compiler = std::string("clang");
auto flag = std::string("-c");
auto file = std::string("hello.cpp");
std::array<char *, 4> args = {
const_cast<char *> (compiler.c_str()),
const_cast<char *> (flag.c_str()),
const_cast<char *> (file.c_str()),
static_cast<char *> (NULL)
};
error = execvp(args[0], args.data());
}
waitpid(pid, &error, 0);
auto object = llvm::MemoryBuffer::getFileAsStream("hello.o");
auto jit_try = llvm::orc::LLJITBuilder().create();
if (auto jiterror = jit_try.takeError()) {
std::cerr << "Failed to build JIT : " << toString(std::move(jiterror)) << std::endl;
return 1;
}
auto jit = std::move(jit_try.get());
jit->addObjectFile(std::move(object.get()));
llvm::orc::ExecutorAddr entry = std::move(jit->lookup("hello")).get();
auto kernel = entry.toPtr<void(*)()> ();
kernel();
return 0;
}
Anyee March 7, 2024, 1:04am 3
I think you should lookup how to invoke Clang compiler from in-process way and trap errors; that way the Clang API can give you a LLVM Module and you can generate bitcode and perhaps run this on your CPU or accelerator etc.
Using the example I found here, I am able to get a to a point where I can get clang to emit some error messages.
#include <string>
#include <iostream>
#include <memory>
#include "clang/Frontend/TextDiagnosticPrinter.h"
#include "clang/Frontend/CompilerInvocation.h"
#include "clang/Frontend/CompilerInstance.h"
#include "clang/Basic/TargetInfo.h"
#include "clang/CodeGen/CodeGenAction.h"
#include "llvm/Support/TargetSelect.h"
#include "llvm/ExecutionEngine/Orc/LLJIT.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/ADT/IntrusiveRefCntPtr.h"
#include "llvm/TargetParser/Host.h"
void jitFunction(const std::string message) {
llvm::InitializeNativeTarget();
const std::string source(
"#include <iostream>\n"
"extern \"C\" void hello() {\n"
" std::cout << \"" + message + "\" << std::endl;\n"
"}"
);
clang::DiagnosticOptions diagOpts;
auto textDiagPrinter = std::make_unique<clang::TextDiagnosticPrinter> (llvm::errs(),
&diagOpts);
llvm::IntrusiveRefCntPtr<clang::DiagnosticIDs> pDiagIDs;
clang::DiagnosticsEngine DE(pDiagIDs, &diagOpts,
textDiagPrinter.get());
std::vector<const char *> args = {
"-O3",
"hello.cpp"
};
auto CI = std::make_shared<clang::CompilerInvocation> ();
clang::CompilerInvocation::CreateFromArgs(*(CI.get()), args, DE);
llvm::StringRef source_code_data(source);
auto buffer = llvm::MemoryBuffer::getMemBufferCopy(source_code_data);
CI->getPreprocessorOpts().addRemappedFile(args[1], buffer.get());
clang::CompilerInstance clang;
clang.setInvocation(CI);
clang.createDiagnostics();
const auto targetOptions = std::make_shared<clang::TargetOptions> ();
targetOptions->Triple = llvm::sys::getProcessTriple();
auto *pTI = clang::TargetInfo::CreateTargetInfo(DE, targetOptions);
clang.setTarget(pTI);
clang::EmitObjAction compilerAction;
clang.ExecuteAction(compilerAction);
auto object = llvm::MemoryBuffer::getFileAsStream("hello.o");
auto jit_try = llvm::orc::LLJITBuilder().create();
if (auto jiterror = jit_try.takeError()) {
std::cerr << "Failed to build JIT : " << toString(std::move(jiterror)) << std::endl;
exit(1);
}
auto jit = std::move(jit_try.get());
jit->addObjectFile(std::move(object.get()));
auto entry = std::move(jit->lookup("hello")).get();
auto kernel = entry.toPtr<void(*)()> ();
kernel();
}
But the problem I’m running into now is trying to find the c++ standard library headers. Adding explicit include paths to args
std::vector<const char *> args = {
"-O3",
"hello.cpp",
"-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/",
"-I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/include",
"-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include"
};
results in the error.
In file included from hello.cpp:1:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iostream:43:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/ios:221:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__ios/fpos.h:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iosfwd:106:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__std_mbstate_t.h:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__mbstate_t.h:45:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/_types/_mbstate_t.h:31:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/machine/types.h:37:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/types.h:50:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:81:2: warning: "Unsupported compiler detected" [-W#warnings]
81 | #warning "Unsupported compiler detected"
| ^
In file included from hello.cpp:1:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iostream:43:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/ios:222:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__locale:21:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/mutex:192:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__condition_variable/condition_variable.h:17:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__mutex/unique_lock.h:17:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__system_error/system_error.h:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__system_error/error_category.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/string:569:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/remove.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/find.h:21:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/cwchar:114:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/wchar.h:126:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:90:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/stdio.h:108:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h:75:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/_types/_va_list.h:32:26: error: typedef redefinition with different types ('__darwin_va_list' (aka 'void *') vs '__builtin_va_list')
32 | typedef __darwin_va_list va_list;
| ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/include/stdarg.h:14:27: note: previous definition is here
14 | typedef __builtin_va_list va_list;
| ^
1 warning and 1 error generated.
However, I would like to avoid hard coding these paths since I intend this to be cross platform. What is the best way to find the system headers?
I was able to get the example working.
#include <string>
#include <iostream>
#include <memory>
#include "clang/Frontend/TextDiagnosticPrinter.h"
#include "clang/Frontend/CompilerInvocation.h"
#include "clang/Frontend/CompilerInstance.h"
#include "clang/Basic/TargetInfo.h"
#include "clang/CodeGen/CodeGenAction.h"
#include "llvm/Support/TargetSelect.h"
#include "llvm/ExecutionEngine/Orc/LLJIT.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/ADT/IntrusiveRefCntPtr.h"
#include "llvm/TargetParser/Host.h"
#include "llvm/ExecutionEngine/Orc/ThreadSafeModule.h"
void jitFunction(const std::string message) {
llvm::InitializeNativeTarget();
LLVMInitializeAArch64AsmPrinter();
LLVMInitializeAArch64AsmParser();
const std::string source(
"#include <iostream>\n"
"extern \"C\" void hello() {\n"
" std::cout << \"" + message + "\" << std::endl;\n"
"}"
);
llvm::IntrusiveRefCntPtr<clang::DiagnosticOptions> diagOpts;
auto textDiagPrinter = std::make_unique<clang::TextDiagnosticPrinter> (llvm::errs(),
diagOpts.get());
llvm::IntrusiveRefCntPtr<clang::DiagnosticIDs> pDiagIDs;
clang::DiagnosticsEngine DE(pDiagIDs, diagOpts,
textDiagPrinter.release());
std::vector<const char *> args = {
"-O3",
"hello.cpp",
"-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1",
"-I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/include",
"-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include",
"-fgnuc-version=4.2.1"
};
auto CI = std::make_shared<clang::CompilerInvocation> ();
clang::CompilerInvocation::CreateFromArgs(*(CI.get()), args, DE);
llvm::StringRef source_code_data(source);
auto buffer = llvm::MemoryBuffer::getMemBuffer(source_code_data);
CI->getPreprocessorOpts().addRemappedFile(args[1], buffer.release());
clang::CompilerInstance clang;
clang.setInvocation(CI);
clang.createDiagnostics();
const auto targetOptions = std::make_shared<clang::TargetOptions> ();
targetOptions->Triple = llvm::sys::getProcessTriple();
auto *pTI = clang::TargetInfo::CreateTargetInfo(DE, targetOptions);
clang.setTarget(pTI);
clang::EmitCodeGenOnlyAction compilerAction;
clang.ExecuteAction(compilerAction);
auto ir_module = compilerAction.takeModule();
auto ctx = std::unique_ptr<llvm::LLVMContext> (compilerAction.takeLLVMContext());
auto jit_try = llvm::orc::LLJITBuilder().create();
if (auto jiterror = jit_try.takeError()) {
std::cerr << "Failed to build JIT : " << toString(std::move(jiterror)) << std::endl;
exit(1);
}
auto jit = std::move(jit_try.get());
jit->addIRModule(llvm::orc::ThreadSafeModule(std::move(ir_module),
llvm::orc::ThreadSafeContext(std::move(ctx))));
auto entry = std::move(jit->lookup("hello")).get();
auto kernel = entry.toPtr<void(*)()> ();
kernel();
}
int main(int argc, const char * argv[]) {
jitFunction("Hello World1");
jitFunction("Hello World2");
jitFunction("Hello World3");
return 0;
}
However, I would like to avoid hard coding these paths since I intend this to be cross platform. What is the best way to find the system headers?
I tested with system clang with clang -### demo.cc, and I got
➜ ~ clang -### demo.cc
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
"/Library/Developer/CommandLineTools/usr/bin/clang" "-cc1" "-triple" "arm64-apple-macosx13.0.0" "-Wundef-prefix=TARGET_OS_" "-Wdeprecated-objc-isa-usage" "-Werror=deprecated-objc-isa-usage" "-Werror=implicit-function-declaration" "-emit-obj" "-mrelax-all" "--mrelax-relocations" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "demo.cc" "-mrelocation-model" "pic" "-pic-level" "2" "-mframe-pointer=non-leaf" "-fno-strict-return" "-ffp-contract=on" "-fno-rounding-math" "-funwind-tables=1" "-fobjc-msgsend-selector-stubs" "-target-sdk-version=13.3" "-fvisibility-inlines-hidden-static-local-var" "-target-cpu" "apple-m1" "-target-feature" "+v8.5a" "-target-feature" "+crc" "-target-feature" "+lse" "-target-feature" "+rdm" "-target-feature" "+crypto" "-target-feature" "+dotprod" "-target-feature" "+fp-armv8" "-target-feature" "+neon" "-target-feature" "+fp16fml" "-target-feature" "+ras" "-target-feature" "+rcpc" "-target-feature" "+zcm" "-target-feature" "+zcz" "-target-feature" "+fullfp16" "-target-feature" "+sm4" "-target-feature" "+sha3" "-target-feature" "+sha2" "-target-feature" "+aes" "-target-abi" "darwinpcs" "-fallow-half-arguments-and-returns" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=lldb" "-target-linker-version" "857.1" "-fcoverage-compilation-dir=/Users/shuoshu.yh" "-resource-dir" "/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3" "-isysroot" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" "-I/usr/local/include" "-stdlib=libc++" "-internal-isystem" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1" "-internal-isystem" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include" "-internal-isystem" "/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3/include" "-internal-externc-isystem" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" "-internal-externc-isystem" "/Library/Developer/CommandLineTools/usr/include" "-Wno-reorder-init-list" "-Wno-implicit-int-float-conversion" "-Wno-c99-designator" "-Wno-final-dtor-non-final-class" "-Wno-extra-semi-stmt" "-Wno-misleading-indentation" "-Wno-quoted-include-in-framework-header" "-Wno-implicit-fallthrough" "-Wno-enum-enum-conversion" "-Wno-enum-float-conversion" "-Wno-elaborated-enum-base" "-Wno-reserved-identifier" "-Wno-gnu-folding-constant" "-fdeprecated-macro" "-fdebug-compilation-dir=/Users/shuoshu.yh" "-ferror-limit" "19" "-stack-protector" "1" "-fstack-check" "-mdarwin-stkchk-strong-link" "-fblocks" "-fencode-extended-block-signature" "-fregister-global-dtors-with-atexit" "-fgnuc-version=4.2.1" "-fno-cxx-modules" "-no-opaque-pointers" "-fcxx-exceptions" "-fexceptions" "-fmax-type-align=16" "-fcommon" "-fcolor-diagnostics" "-clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation" "-fno-odr-hash-protocols" "-clang-vendor-feature=+enableAggressiveVLAFolding" "-clang-vendor-feature=+revert09abecef7bbf" "-clang-vendor-feature=+thisNoAlignAttr" "-clang-vendor-feature=+thisNoNullAttr" "-mllvm" "-disable-aligned-alloc-awareness=1" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/var/folders/8j/094tp2hd3h986hp51n_3pgz80000gp/T/demo-627ba4.o" "-x" "c++" "demo.cc"
"/Library/Developer/CommandLineTools/usr/bin/ld" "-demangle" "-lto_library" "/Library/Developer/CommandLineTools/usr/lib/libLTO.dylib" "-no_deduplicate" "-dynamic" "-arch" "arm64" "-platform_version" "macos" "13.0.0" "13.3" "-syslibroot" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" "-o" "a.out" "-L/usr/local/lib" "/var/folders/8j/094tp2hd3h986hp51n_3pgz80000gp/T/demo-627ba4.o" "-lSystem" "/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3/lib/darwin/libclang_rt.osx.a"
the system related options is
-resource-dir" "/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3" "-isysroot" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" "-I/usr/local/include" "-stdlib=libc++" "-internal-isystem" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1" "-internal-isystem" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include" "-internal-isystem" "/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3/include" "-internal-externc-isystem" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" "-internal-externc-isystem" "/Library/Developer/CommandLineTools/usr/include"
I found use xcrun --sdk macosx --show-sdk-path, then we got
➜ ~ xcrun --sdk macosx --show-sdk-path
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
I also found
➜ ~ clang -print-resource-dir
/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3
for "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1", I have no idea.