Build System - ESP32 - — ESP-IDF Programming Guide v5.4.1 documentation (original) (raw)

[中文]

This document explains the implementation of the ESP-IDF build system and the concept of "components". Read this document if you want to know how to organize and build a new ESP-IDF project or component.

Overview

An ESP-IDF project can be seen as an amalgamation of a number of components. For example, for a web server that shows the current humidity, there could be:

ESP-IDF makes these components explicit and configurable. To do that, when a project is compiled, the build system will look up all the components in the ESP-IDF directories, the project directories and (optionally) in additional custom component directories. It then allows the user to configure the ESP-IDF project using a text-based menu system to customize each component. After the components in the project are configured, the build system will compile the project.

Concepts

Some things are not part of the project:

Using the Build System

idf.py

The idf.py command-line tool provides a front-end for easily managing your project builds. It manages the following tools:

You can read more about configuring the build system using idf.py here.

Using CMake Directly

idf.py is a wrapper around CMake for convenience. However, you can also invoke CMake directly if you prefer.

When idf.py does something, it prints each command that it runs for easy reference. For example, the idf.py build command is the same as running these commands in a bash shell (or similar commands for Windows Command Prompt):

mkdir -p build cd build cmake .. -G Ninja # or 'Unix Makefiles' ninja

In the above list, the cmake command configures the project and generates build files for use with the final build tool. In this case, the final build tool is Ninja: running ninja actually builds the project.

It's not necessary to run cmake more than once. After the first build, you only need to run ninja each time. ninja will automatically re-invoke cmake if the project needs reconfiguration.

If using CMake with ninja or make, there are also targets for more of the idf.py sub-commands. For example, running make menuconfig or ninja menuconfig in the build directory will work the same as idf.py menuconfig.

Note

If you're already familiar with CMake, you may find the ESP-IDF CMake-based build system unusual because it wraps a lot of CMake's functionality to reduce boilerplate. See writing pure CMake components for some information about writing more "CMake style" components.

Flashing with Ninja or Make

It's possible to build and flash directly from ninja or make by running a target like:

Or:

Available targets are: flash, app-flash (app only), bootloader-flash (bootloader only).

When flashing this way, optionally set the ESPPORT and ESPBAUD environment variables to specify the serial port and baud rate. You can set environment variables in your operating system or IDE project. Alternatively, set them directly on the command line:

ESPPORT=/dev/ttyUSB0 ninja flash

Note

Providing environment variables at the start of the command like this is Bash shell Syntax. It will work on Linux and macOS. It won't work when using Windows Command Prompt, but it will work when using Bash-like shells on Windows.

Or:

make -j3 app-flash ESPPORT=COM4 ESPBAUD=2000000

Note

Providing variables at the end of the command line is make syntax, and works for make on all platforms.

Using CMake in an IDE

You can also use an IDE with CMake integration. The IDE will want to know the path to the project's CMakeLists.txt file. IDEs with CMake integration often provide their own build tools (CMake calls these "generators") to build the source files as part of the IDE.

When adding custom non-build steps like "flash" to the IDE, it is recommended to execute idf.py for these "special" commands.

For more detailed information about integrating ESP-IDF with CMake into an IDE, see Build System Metadata.

Setting up the Python Interpreter

ESP-IDF works well with Python version 3.8+.

idf.py and other Python scripts will run with the default Python interpreter, i.e., python. You can switch to a different one like python3 $IDF_PATH/tools/idf.py ..., or you can set up a shell alias or another script to simplify the command.

If using CMake directly, running cmake -D PYTHON=python3 ... will cause CMake to override the default Python interpreter.

If using an IDE with CMake, setting the PYTHON value as a CMake cache override in the IDE UI will override the default Python interpreter.

To manage the Python version more generally via the command line, check out the tools pyenv or virtualenv. These let you change the default Python version.

Example Project

An example project directory tree might look like this:

This example "myProject" contains the following elements:

Component directories each contain a component CMakeLists.txt file. This file contains variable definitions to control the build process of the component, and its integration into the overall project. See Component CMakeLists Files for more details.

Each component may also include a Kconfig file defining the component configuration options that can be set via menuconfig. Some components may also include Kconfig.projbuild and project_include.cmake files, which are special files for overriding parts of the project.

Project CMakeLists File

Each project has a single top-level CMakeLists.txt file that contains build settings for the entire project. By default, the project CMakeLists can be quite minimal.

Minimal Example CMakeLists

Minimal project:

cmake_minimum_required(VERSION 3.16) include($ENV{IDF_PATH}/tools/cmake/project.cmake) project(myProject)

Mandatory Parts

The inclusion of these three lines, in the order shown above, is necessary for every project:

Optional Project Variables

These variables all have default values that can be overridden for custom behavior. Look in /tools/cmake/project.cmake for all of the implementation details.

Any paths in these variables can be absolute paths, or set relative to the project directory.

To set these variables, use the cmake set command ie set(VARIABLE "VALUE"). The set() commands should be placed after the cmake_minimum(...) line but before the include(...) line.

Renaming main Component

The build system provides special treatment to the main component. It is a component that gets automatically added to the build provided that it is in the expected location, PROJECT_DIR/main. All other components in the build are also added as its dependencies, saving the user from hunting down dependencies and providing a build that works right out of the box. Renaming the main component causes the loss of these behind-the-scenes heavy lifting, requiring the user to specify the location of the newly renamed component and manually specify its dependencies. Specifically, the steps to renaming main are as follows:

  1. Rename main directory.
  2. Set EXTRA_COMPONENT_DIRS in the project CMakeLists.txt to include the renamed main directory.
  3. Specify the dependencies in the renamed component's CMakeLists.txt file via REQUIRES or PRIV_REQUIRES arguments on component registration.

Overriding Default Build Specifications

The build sets some global build specifications (compile flags, definitions, etc.) that gets used in compiling all sources from all components.

For example, one of the default build specifications set is the compile option -Wextra. Suppose a user wants to use override this with -Wno-extra, it should be done after project():

cmake_minimum_required(VERSION 3.16) include($ENV{IDF_PATH}/tools/cmake/project.cmake) project(myProject)

idf_build_set_property(COMPILE_OPTIONS "-Wno-error" APPEND)

This ensures that the compile options set by the user won't be overridden by the default build specifications, since the latter are set inside project().

Component CMakeLists Files

Each project contains one or more components. Components can be part of ESP-IDF, part of the project's own components directory, or added from custom component directories (see above).

A component is any directory in the COMPONENT_DIRS list which contains a CMakeLists.txt file.

Searching for Components

The list of directories in COMPONENT_DIRS is searched for the project's components. Directories in this list can either be components themselves (ie they contain a CMakeLists.txt file), or they can be top-level directories whose sub-directories are components.

When CMake runs to configure the project, it logs the components included in the build. This list can be useful for debugging the inclusion/exclusion of certain components.

Multiple Components with the Same Name

When ESP-IDF is collecting all the components to compile, the search precedence is as follows (from highest to lowest):

If two or more of these directories contain component sub-directories with the same name, the component with higher precedence is used. This allows, for example, overriding ESP-IDF components with a modified version by copying that component from the ESP-IDF components directory to the project components directory and then modifying it there. If used in this way, the ESP-IDF directory itself can remain untouched.

Note

If a component is overridden in an existing project by moving it to a new location, the project will not automatically see the new component path. Run idf.py reconfigure (or delete the project build folder) and then build again.

Minimal Component CMakeLists

The minimal component CMakeLists.txt file simply registers the component to the build system using idf_component_register:

idf_component_register(SRCS "foo.c" "bar.c" INCLUDE_DIRS "include" REQUIRES mbedtls)

A library with the name of the component will be built and linked to the final app.

Directories are usually specified relative to the CMakeLists.txt file itself, although they can be absolute.

There are other arguments that can be passed to idf_component_register. These arguments are discussed here.

See example component requirements and example component CMakeLists for more complete component CMakeLists.txt examples.

Preset Component Variables

The following component-specific variables are available for use inside component CMakeLists, but should not be modified:

The following variables are set at the project level, but available for use in component CMakeLists:

Build/Project Variables

The following are some project/build variables that are available as build properties and whose values can be queried using idf_build_get_property from the component CMakeLists.txt:

Other build properties are listed here.

Controlling Component Compilation

To pass compiler options when compiling source files belonging to a particular component, use the target_compile_options function:

target_compile_options(${COMPONENT_LIB} PRIVATE -Wno-unused-variable)

To apply the compilation flags to a single source file, use the CMake set_source_files_properties command:

set_source_files_properties(mysrc.c PROPERTIES COMPILE_FLAGS -Wno-unused-variable )

This can be useful if there is upstream code that emits warnings.

Note

CMake set_source_files_properties command is not applicable when the source files have been populated with help of the SRC_DIRS variable in idf_component_register. See File Globbing & Incremental Builds for more details.

When using these commands, place them after the call to idf_component_register in the component CMakeLists file.

Component Configuration

Each component can also have a Kconfig file, alongside CMakeLists.txt. This contains configuration settings to add to the configuration menu for this component.

These settings are found under the "Component Settings" menu when menuconfig is run.

To create a component Kconfig file, it is easiest to start with one of the Kconfig files distributed with ESP-IDF.

For an example, see Adding conditional configuration.

Preprocessor Definitions

The ESP-IDF build system adds the following C preprocessor definitions on the command line:

Component Requirements

When compiling each component, the ESP-IDF build system recursively evaluates its dependencies. This means each component needs to declare the components that it depends on ("requires").

When Writing a Component

idf_component_register(... REQUIRES mbedtls PRIV_REQUIRES console spiffs)

If a component only supports some target chips (values of IDF_TARGET) then it can specify REQUIRED_IDF_TARGETS in the idf_component_register call to express these requirements. In this case, the build system will generate an error if the component is included in the build, but does not support the selected target.

Note

In CMake terms, REQUIRES & PRIV_REQUIRES are approximate wrappers around the CMake functions target_link_libraries(... PUBLIC ...) and target_link_libraries(... PRIVATE ...).

Example of Component Requirements

Imagine there is a car component, which uses the engine component, which uses the spark_plug component:

Car Component

The car.h header file is the public interface for the car component. This header includes engine.h directly because it uses some declarations from this header:

/* car.h */ #include "engine.h"

#ifdef ENGINE_IS_HYBRID #define CAR_MODEL "Hybrid" #endif

And car.c includes car.h as well:

/* car.c */ #include "car.h"

This means the car/CMakeLists.txt file needs to declare that car requires engine:

idf_component_register(SRCS "car.c" INCLUDE_DIRS "." REQUIRES engine)

Engine Component

The engine component also has a public header file include/engine.h, but this header is simpler:

/* engine.h */ #define ENGINE_IS_HYBRID

void engine_start(void);

The implementation is in engine.c:

/* engine.c */ #include "engine.h" #include "spark_plug.h"

...

In this component, engine depends on spark_plug but this is a private dependency. spark_plug.h is needed to compile engine.c, but not needed to include engine.h.

This means that the engine/CMakeLists.txt file can use PRIV_REQUIRES:

idf_component_register(SRCS "engine.c" INCLUDE_DIRS "include" PRIV_REQUIRES spark_plug)

As a result, source files in the car component don't need the spark_plug include directories added to their compiler search path. This can speed up compilation, and stops compiler command lines from becoming longer than necessary.

Spark Plug Component

The spark_plug component doesn't depend on anything else. It has a public header file spark_plug.h, but this doesn't include headers from any other components.

This means that the spark_plug/CMakeLists.txt file doesn't need any REQUIRES or PRIV_REQUIRES clauses:

idf_component_register(SRCS "spark_plug.c" INCLUDE_DIRS ".")

Source File Include Directories

Each component's source file is compiled with these include path directories, as specified in the passed arguments to idf_component_register:

idf_component_register(.. INCLUDE_DIRS "include" PRIV_INCLUDE_DIRS "other")

Main Component Requirements

The component named main is special because it automatically requires all other components in the build. So it's not necessary to pass REQUIRES or PRIV_REQUIRES to this component. See renaming main for a description of what needs to be changed if no longer using the main component.

Common Component Requirements

To avoid duplication, every component automatically requires some "common" IDF components even if they are not mentioned explicitly. Headers from these components can always be included.

The list of common components is: cxx, newlib, freertos, esp_hw_support, heap, log, soc, hal, esp_rom, esp_common, esp_system, xtensa/riscv.

Including Components in the Build

Circular Dependencies

It's possible for a project to contain Component A that requires (REQUIRES or PRIV_REQUIRES) Component B, and Component B that requires Component A. This is known as a dependency cycle or a circular dependency.

CMake will usually handle circular dependencies automatically by repeating the component library names twice on the linker command line. However this strategy doesn't always work, and the build may fail with a linker error about "Undefined reference to ...", referencing a symbol defined by one of the components inside the circular dependency. This is particularly likely if there is a large circular dependency, i.e., A > B > C > D > A.

The best solution is to restructure the components to remove the circular dependency. In most cases, a software architecture without circular dependencies has desirable properties of modularity and clean layering and will be more maintainable in the long term. However, removing circular dependencies is not always possible.

To bypass a linker error caused by a circular dependency, the simplest workaround is to increase the CMake LINK_INTERFACE_MULTIPLICITY property of one of the component libraries. This causes CMake to repeat this library and its dependencies more than two times on the linker command line.

For example:

set_property(TARGET ${COMPONENT_LIB} APPEND PROPERTY LINK_INTERFACE_MULTIPLICITY 3)

Advanced Workaround: Undefined Symbols

If only one or two symbols are causing a circular dependency, and all other dependencies are linear, then there is an alternative method to avoid linker errors: Specify the specific symbols required for the "reverse" dependency as undefined symbols at link time.

For example, if component A depends on component B but component B also needs to reference reverse_ops from component A (but nothing else), then you can add a line like the following to the component B CMakeLists.txt to resolve the cycle at link time:

This symbol is provided by 'Component A' at link time

target_link_libraries(${COMPONENT_LIB} INTERFACE "-u reverse_ops")

See the target_link_libraries documentation for more information about this CMake function.

Requirements in the Build System Implementation

Component Dependency Order

The order of components in the BUILD_COMPONENTS variable determines other orderings during the build:

The ESP-IDF CMake helper function idf_component_add_link_dependency adds a link-only dependency between one component and another. In almost all cases, it is better to use the PRIV_REQUIRES feature in idf_component_register to create a dependency. However, in some cases, it's necessary to add the link-time dependency of another component to this component, i.e., the reverse order to PRIV_REQUIRES (for example: Overriding Default Chip Drivers).

To make another component depend on this component at link time:

idf_component_add_link_dependency(FROM other_component)

Place this line after the line with idf_component_register.

It's also possible to specify both components by name:

idf_component_add_link_dependency(FROM other_component TO that_component)

Overriding Parts of the Project

Project_include.cmake

For components that have build requirements that must be evaluated before any component CMakeLists files are evaluated, you can create a file called project_include.cmake in the component directory. This CMake file is included when project.cmake is evaluating the entire project.

project_include.cmake files are used inside ESP-IDF, for defining project-wide build features such as esptool.py command line arguments and the bootloader "special app".

Unlike component CMakeLists.txt files, when including a project_include.cmake file the current source directory (CMAKE_CURRENT_SOURCE_DIR and working directory) is the project directory. Use the variable COMPONENT_DIR for the absolute directory of the component.

Note that project_include.cmake isn't necessary for the most common component uses, such as adding include directories to the project, or LDFLAGS to the final linking step. These values can be customized via the CMakeLists.txt file itself. See Optional Project Variables for details.

project_include.cmake files are included in the order given in BUILD_COMPONENTS variable (as logged by CMake). This means that a component's project_include.cmake file will be included after it's all dependencies' project_include.cmake files, unless both components are part of a dependency cycle. This is important if a project_include.cmake file relies on variables set by another component. See also above.

Take great care when setting variables or targets in a project_include.cmake file. As the values are included in the top-level project CMake pass, they can influence or break functionality across all components!

KConfig.projbuild

This is an equivalent to project_include.cmake for Component Configuration KConfig files. If you want to include configuration options at the top level of menuconfig, rather than inside the "Component Configuration" sub-menu, then these can be defined in the KConfig.projbuild file alongside the CMakeLists.txt file.

Take care when adding configuration values in this file, as they will be included across the entire project configuration. Where possible, it's generally better to create a KConfig file for Component Configuration.

Wrappers to Redefine or Extend Existing Functions

Thanks to the linker's wrap feature, it is possible to redefine or extend the behavior of an existing ESP-IDF function. To do so, you will need to provide the following CMake declaration in your project's CMakeLists.txt file:

target_link_libraries(${COMPONENT_LIB} INTERFACE "-Wl,--wrap=function_to_redefine")

Where function_to_redefine is the name of the function to redefine or extend. This option will let the linker replace all the calls to function_to_redefine functions in the binary libraries with calls to __wrap_function_to_redefine function. Thus, you must define this new symbol in your application.

The linker will provide a new symbol named __real_function_to_redefine which points to the former implementation of the function to redefine. It can be called from the new implementation, making it an extension of the former one.

This mechanism is shown in the example build_system/wrappers. Check examples/build_system/wrappers/README.md for more details.

Override the Default Bootloader

Thanks to the optional bootloader_components directory present in your ESP-IDf project, it is possible to override the default ESP-IDF bootloader. To do so, a new bootloader_components/main component should be defined, which will make the project directory tree look like the following:

Here the my_bootloader.c file becomes source code for the new bootloader, which means that it will need to perform all the required operations to set up and load the main application from flash.

It is also possible to conditionally replace the bootloader depending on a certain condition, such as the target for example. This can be achieved thanks to the BOOTLOADER_IGNORE_EXTRA_COMPONENT CMake variable. This list can be used to tell the ESP-IDF bootloader project to ignore and not compile the given components present in bootloader_components. For example, if one wants to use the default bootloader for ESP32 target, then myProject/CMakeLists.txt should look like the following:

include($ENV{IDF_PATH}/tools/cmake/project.cmake)

if(${IDF_TARGET} STREQUAL "esp32") set(BOOTLOADER_IGNORE_EXTRA_COMPONENT "main") endif()

project(main)

It is important to note that this can also be used for any other bootloader components than main. In all cases, the prefix bootloader_component must not be specified.

See custom_bootloader/bootloader_override for an example of overriding the default bootloader.

Configuration-Only Components

Special components which contain no source files, only Kconfig.projbuild and KConfig, can have a one-line CMakeLists.txt file which calls the function idf_component_register() with no arguments specified. This function will include the component in the project build, but no library will be built and no header files will be added to any included paths.

Debugging CMake

For full details about CMake and CMake commands, see the CMake v3.16 documentation.

Some tips for debugging the ESP-IDF CMake-based build system:

When included from a project CMakeLists file, the project.cmake file defines some utility modules and global variables and then sets IDF_PATH if it was not set in the system environment.

It also defines an overridden custom version of the built-in CMake project function. This function is overridden to add all of the ESP-IDF specific project functionality.

Warning On Undefined Variables

By default, the function of warnings on undefined variables is disabled.

To enable this function, we can pass the --warn-uninitialized flag to CMake or pass the --cmake-warn-uninitialized flag to idf.py so it will print a warning if an undefined variable is referenced in the build. This can be very useful to find buggy CMake files.

Browse the /tools/cmake/project.cmake file and supporting functions in /tools/cmake/ for more details.

Example Component CMakeLists

Because the build environment tries to set reasonable defaults that will work most of the time, component CMakeLists.txt can be very small or even empty (see Minimal Component CMakeLists). However, overriding preset_component_variables is usually required for some functionality.

Here are some more advanced examples of component CMakeLists files.

Adding Conditional Configuration

The configuration system can be used to conditionally compile some files depending on the options selected in the project configuration.

Kconfig:

config FOO_ENABLE_BAR bool "Enable the BAR feature." help This enables the BAR feature of the FOO component.

CMakeLists.txt:

set(srcs "foo.c" "more_foo.c")

if(CONFIG_FOO_ENABLE_BAR) list(APPEND srcs "bar.c") endif()

idf_component_register(SRCS "${srcs}" ...)

This example makes use of the CMake if function and list APPEND function.

This can also be used to select or stub out an implementation, as such:

Kconfig:

config ENABLE_LCD_OUTPUT bool "Enable LCD output." help Select this if your board has an LCD.

config ENABLE_LCD_CONSOLE bool "Output console text to LCD" depends on ENABLE_LCD_OUTPUT help Select this to output debugging output to the LCD

config ENABLE_LCD_PLOT bool "Output temperature plots to LCD" depends on ENABLE_LCD_OUTPUT help Select this to output temperature plots

CMakeLists.txt:

if(CONFIG_ENABLE_LCD_OUTPUT) set(srcs lcd-real.c lcd-spi.c) else() set(srcs lcd-dummy.c) endif()

We need font if either console or plot is enabled

if(CONFIG_ENABLE_LCD_CONSOLE OR CONFIG_ENABLE_LCD_PLOT) list(APPEND srcs "font.c") endif()

idf_component_register(SRCS "${srcs}" ...)

Conditions Which Depend on the Target

The current target is available to CMake files via IDF_TARGET variable.

In addition to that, if target xyz is used (IDF_TARGET=xyz), then Kconfig variable CONFIG_IDF_TARGET_XYZ will be set.

Note that component dependencies may depend on IDF_TARGET variable, but not on Kconfig variables. Also one can not use Kconfig variables in include statements in CMake files, but IDF_TARGET can be used in such context.

Source Code Generation

Some components will have a situation where a source file isn't supplied with the component itself but has to be generated from another file. Say our component has a header file that consists of the converted binary data of a BMP file, converted using a hypothetical tool called bmp2h. The header file is then included in as C source file called graphics_lib.c:

add_custom_command(OUTPUT logo.h COMMAND bmp2h -i ${COMPONENT_DIR}/logo.bmp -o log.h DEPENDS ${COMPONENT_DIR}/logo.bmp VERBATIM)

add_custom_target(logo DEPENDS logo.h) add_dependencies(${COMPONENT_LIB} logo)

set_property(DIRECTORY "${COMPONENT_DIR}" APPEND PROPERTY ADDITIONAL_CLEAN_FILES logo.h)

This answer is adapted from the CMake FAQ entry, which contains some other examples that will also work with ESP-IDF builds.

In this example, logo.h will be generated in the current directory (the build directory) while logo.bmp comes with the component and resides under the component path. Because logo.h is a generated file, it should be cleaned when the project is cleaned. For this reason, it is added to the ADDITIONAL_CLEAN_FILES property.

Note

If generating files as part of the project CMakeLists.txt file, not a component CMakeLists.txt, then use build property PROJECT_DIR instead of ${COMPONENT_DIR} and ${PROJECT_NAME}.elf instead of ${COMPONENT_LIB}.)

If a a source file from another component included logo.h, then add_dependencies would need to be called to add a dependency between the two components, to ensure that the component source files were always compiled in the correct order.

Embedding Binary Data

Sometimes you have a file with some binary or text data that you'd like to make available to your component, but you don't want to reformat the file as a C source.

You can specify argument EMBED_FILES in the component registration, giving space-delimited names of the files to embed:

idf_component_register(... EMBED_FILES server_root_cert.der)

Or if the file is a string, you can use the variable EMBED_TXTFILES. This will embed the contents of the text file as a null-terminated string:

idf_component_register(... EMBED_TXTFILES server_root_cert.pem)

The file's contents will be added to the .rodata section in flash, and are available via symbol names as follows:

extern const uint8_t server_root_cert_pem_start[] asm("_binary_server_root_cert_pem_start"); extern const uint8_t server_root_cert_pem_end[] asm("_binary_server_root_cert_pem_end");

The names are generated from the full name of the file, as given in EMBED_FILES. Characters /, ., etc. are replaced with underscores. The _binary prefix in the symbol name is added by objcopy and is the same for both text and binary files.

To embed a file into a project, rather than a component, you can call the function target_add_binary_data like this:

target_add_binary_data(myproject.elf "main/data.bin" TEXT)

Place this line after the project() line in your project CMakeLists.txt file. Replace myproject.elf with your project name. The final argument can be TEXT to embed a null-terminated string, or BINARY to embed the content as-is.

For an example of using this technique, see the "main" component of the file_serving example protocols/http_server/file_serving/main/CMakeLists.txt - two files are loaded at build time and linked into the firmware.

It is also possible to embed a generated file:

add_custom_command(OUTPUT my_processed_file.bin COMMAND my_process_file_cmd my_unprocessed_file.bin) target_add_binary_data(my_target "my_processed_file.bin" BINARY)

In the example above, my_processed_file.bin is generated from my_unprocessed_file.bin through some command my_process_file_cmd, then embedded into the target.

To specify a dependence on a target, use the DEPENDS argument:

add_custom_target(my_process COMMAND ...) target_add_binary_data(my_target "my_embed_file.bin" BINARY DEPENDS my_process)

The DEPENDS argument to target_add_binary_data ensures that the target executes first.

Code and Data Placements

ESP-IDF has a feature called linker script generation that enables components to define where its code and data will be placed in memory through linker fragment files. These files are processed by the build system, and is used to augment the linker script used for linking app binary. See Linker Script Generation for a quick start guide as well as a detailed discussion of the mechanism.

Fully Overriding the Component Build Process

Obviously, there are cases where all these recipes are insufficient for a certain component, for example when the component is basically a wrapper around another third-party component not originally intended to be compiled under this build system. In that case, it's possible to forego the ESP-IDF build system entirely by using a CMake feature called ExternalProject. Example component CMakeLists:

External build process for quirc, runs in source dir and

produces libquirc.a

externalproject_add(quirc_build PREFIX ${COMPONENT_DIR} SOURCE_DIR ${COMPONENT_DIR}/quirc CONFIGURE_COMMAND "" BUILD_IN_SOURCE 1 BUILD_COMMAND make CC=${CMAKE_C_COMPILER} libquirc.a INSTALL_COMMAND "" )

Add libquirc.a to the build process

add_library(quirc STATIC IMPORTED GLOBAL) add_dependencies(quirc quirc_build)

set_target_properties(quirc PROPERTIES IMPORTED_LOCATION ${COMPONENT_DIR}/quirc/libquirc.a) set_target_properties(quirc PROPERTIES INTERFACE_INCLUDE_DIRECTORIES ${COMPONENT_DIR}/quirc/lib)

set_directory_properties( PROPERTIES ADDITIONAL_CLEAN_FILES "${COMPONENT_DIR}/quirc/libquirc.a")

(The above CMakeLists.txt can be used to create a component named quirc that builds the quirc project using its own Makefile.)

Note

When using an external build process with PSRAM, remember to add -mfix-esp32-psram-cache-issue to the C compiler arguments. See CONFIG_SPIRAM_CACHE_WORKAROUND for details of this flag.

ExternalProject Dependencies and Clean Builds

CMake has some unusual behavior around external project builds:

The best of these approaches for building an external project will depend on the project itself, its build system, and whether you anticipate needing to frequently recompile the project.

Custom Sdkconfig Defaults

For example projects or other projects where you don't want to specify a full sdkconfig configuration, but you do want to override some key values from the ESP-IDF defaults, it is possible to create a file sdkconfig.defaults in the project directory. This file will be used when creating a new config from scratch, or when any new config value hasn't yet been set in the sdkconfig file.

To override the name of this file or to specify multiple files, set the SDKCONFIG_DEFAULTS environment variable or set SDKCONFIG_DEFAULTS in top-level CMakeLists.txt. File names that are not specified as full paths are resolved relative to current project's directory.

When specifying multiple files, use a semicolon as the list separator. Files listed first will be applied first. If a particular key is defined in multiple files, the definition in the latter file will override definitions from former files.

Some of the IDF examples include a sdkconfig.ci file. This is part of the continuous integration (CI) test framework and is ignored by the normal build process.

Target-dependent Sdkconfig Defaults

If and only if an sdkconfig.defaults file exists, the build system will also attempt to load defaults from an sdkconfig.defaults.TARGET_NAME file, where TARGET_NAME is the value of IDF_TARGET. For example, for esp32 target, default settings will be taken from sdkconfig.defaults first, and then from sdkconfig.defaults.esp32. If there are no generic default settings, an empty sdkconfig.defaults still needs to be created if the build system should recognize any additional target-dependent sdkconfig.defaults.TARGET_NAME files.

If SDKCONFIG_DEFAULTS is used to override the name of defaults file/files, the name of target-specific defaults file will be derived from SDKCONFIG_DEFAULTS value/values using the rule above. When there are multiple files in SDKCONFIG_DEFAULTS, target-specific file will be applied right after the file bringing it in, before all latter files in SDKCONFIG_DEFAULTS

For example, if SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig_devkit1", and there is a file sdkconfig.defaults.esp32 in the same folder, then the files will be applied in the following order: (1) sdkconfig.defaults (2) sdkconfig.defaults.esp32 (3) sdkconfig_devkit1.

Flash Arguments

There are some scenarios that we want to flash the target board without IDF. For this case we want to save the built binaries, esptool.py and esptool write_flash arguments. It's simple to write a script to save binaries and esptool.py.

After running a project build, the build directory contains binary output files (.bin files) for the project and also the following flashing data files:

You can pass any of these flasher argument files to esptool.py as follows:

python esptool.py --chip esp32 write_flash @build/flash_project_args

Alternatively, it is possible to manually copy the parameters from the argument file and pass them on the command line.

The build directory also contains a generated file flasher_args.json which contains project flash information, in JSON format. This file is used by idf.py and can also be used by other tools which need information about the project build.

Building the Bootloader

The bootloader is a special "subproject" inside /components/bootloader/subproject. It has its own project CMakeLists.txt file and builds separate .ELF and .BIN files to the main project. However, it shares its configuration and build directory with the main project.

The subproject is inserted as an external project from the top-level project, by the file /components/bootloader/project_include.cmake. The main build process runs CMake for the subproject, which includes discovering components (a subset of the main components) and generating a bootloader-specific config (derived from the main sdkconfig).

Writing Pure CMake Components

The ESP-IDF build system "wraps" CMake with the concept of "components", and helper functions to automatically integrate these components into a project build.

However, underneath the concept of "components" is a full CMake build system. It is also possible to make a component which is pure CMake.

Here is an example minimal "pure CMake" component CMakeLists file for a component named json:

add_library(json STATIC cJSON/cJSON.c cJSON/cJSON_Utils.c)

target_include_directories(json PUBLIC cJSON)

Using Third-Party CMake Projects with Components

CMake is used for a lot of open-source C and C++ projects — code that users can tap into for their applications. One of the benefits of having a CMake build system is the ability to import these third-party projects, sometimes even without modification! This allows for users to be able to get functionality that may not yet be provided by a component, or use another library for the same functionality.

Importing a library might look like this for a hypothetical library foo to be used in the main component:

Register the component

idf_component_register(...)

Set values of hypothetical variables that control the build of foo

set(FOO_BUILD_STATIC OFF) set(FOO_BUILD_TESTS OFF)

Create and import the library targets

add_subdirectory(foo)

Publicly link foo to main component

target_link_libraries(main PUBLIC foo)

For an actual example, take a look at build_system/cmake/import_lib. Take note that what needs to be done in order to import the library may vary. It is recommended to read up on the library's documentation for instructions on how to import it from other projects. Studying the library's CMakeLists.txt and build structure can also be helpful.

It is also possible to wrap a third-party library to be used as a component in this manner. For example, the mbedtls component is a wrapper for Espressif's fork of mbedtls. See its component CMakeLists.txt .

The CMake variable ESP_PLATFORM is set to 1 whenever the ESP-IDF build system is being used. Tests such as if (ESP_PLATFORM) can be used in generic CMake code if special IDF-specific logic is required.

Using ESP-IDF Components from External Libraries

The above example assumes that the external library foo (or tinyxml in the case of the import_lib example) doesn't need to use any ESP-IDF APIs apart from common APIs such as libc, libstdc++, etc. If the external library needs to use APIs provided by other ESP-IDF components, this needs to be specified in the external CMakeLists.txt file by adding a dependency on the library target idf::<componentname>.

For example, in the foo/CMakeLists.txt file:

add_library(foo bar.c fizz.cpp buzz.cpp)

if(ESP_PLATFORM)

On ESP-IDF, bar.c needs to include esp_flash.h from the spi_flash component

target_link_libraries(foo PRIVATE idf::spi_flash) endif()

Using Prebuilt Libraries with Components

Another possibility is that you have a prebuilt static library (.a file), built by some other build process.

The ESP-IDF build system provides a utility function add_prebuilt_library for users to be able to easily import and use prebuilt libraries:

add_prebuilt_library(target_name lib_path [REQUIRES req1 req2 ...] [PRIV_REQUIRES req1 req2 ...])

where:

Optional arguments REQUIRES and PRIV_REQUIRES specify dependency on other components. These have the same meaning as the arguments for idf_component_register.

Take note that the prebuilt library must have been compiled for the same target as the consuming project. Configuration relevant to the prebuilt library must also match. If not paid attention to, these two factors may contribute to subtle bugs in the app.

For an example, take a look at build_system/cmake/import_prebuilt.

Using ESP-IDF in Custom CMake Projects

ESP-IDF provides a template CMake project for easily creating an application. However, in some instances the user might already have an existing CMake project or may want to create a custom one. In these cases it is desirable to be able to consume IDF components as libraries to be linked to the user's targets (libraries/executables).

It is possible to do so by using the build system APIs provided by tools/cmake/idf.cmake. For example:

cmake_minimum_required(VERSION 3.16) project(my_custom_app C)

Include CMake file that provides ESP-IDF CMake build system APIs.

include($ENV{IDF_PATH}/tools/cmake/idf.cmake)

Include ESP-IDF components in the build, may be thought as an equivalent of

add_subdirectory() but with some additional processing and magic for ESP-IDF build

specific build processes.

idf_build_process(esp32)

Create the project executable and plainly link the newlib component to it using

its alias, idf::newlib.

add_executable(${CMAKE_PROJECT_NAME}.elf main.c) target_link_libraries(${CMAKE_PROJECT_NAME}.elf idf::newlib)

Let the build system know what the project executable is to attach more targets, dependencies, etc.

idf_build_executable(${CMAKE_PROJECT_NAME}.elf)

The example in build_system/cmake/idf_as_lib demonstrates the creation of an application equivalent to hello world application using a custom CMake project.

Note

The IDF build system can only set compiler flags for source files that it builds. When an external CMakeLists.txt file is used and PSRAM is enabled, remember to add -mfix-esp32-psram-cache-issue to the C compiler arguments. See CONFIG_SPIRAM_CACHE_WORKAROUND for details of this flag.

ESP-IDF CMake Build System API

Idf-build-commands

idf_build_get_property(var property [GENERATOR_EXPRESSION])

Retrieve a build property property and store it in var accessible from the current scope. Specifying GENERATOR_EXPRESSION will retrieve the generator expression string for that property, instead of the actual value, which can be used with CMake commands that support generator expressions.

idf_build_set_property(property val [APPEND])

Set a build property property with value val. Specifying APPEND will append the specified value to the current value of the property. If the property does not previously exist or it is currently empty, the specified value becomes the first element/member instead.

idf_build_component(component_dir [component_source])

Present a directory component_dir that contains a component to the build system. Relative paths are converted to absolute paths with respect to current directory.

An optional component_source argument can be specified to indicate the source of the component. (default: "project_components")

This argument determines the overriding priority for components with the same name. For detailed information, see Multiple Components with the Same Name.

This argument supports the following values (from highest to lowest priority):

For instance, if a component named "json" is present as both "idf_components", and "project_components", the component as "project_components" takes precedence over the one as "idf_components".

Warning

All calls to this command must be performed before idf_build_process. This command does not guarantee that the component will be processed during build (see the COMPONENTS argument description for idf_build_process).

idf_build_process(target [PROJECT_DIR project_dir] [PROJECT_VER project_ver] [PROJECT_NAME project_name] [SDKCONFIG sdkconfig] [SDKCONFIG_DEFAULTS sdkconfig_defaults] [BUILD_DIR build_dir] [COMPONENTS component1 component2 ...])

Performs the bulk of the behind-the-scenes magic for including ESP-IDF components such as component configuration, libraries creation, dependency expansion and resolution. Among these functions, perhaps the most important from a user's perspective is the libraries creation by calling each component's idf_component_register. This command creates the libraries for each component, which are accessible using aliases in the form idf::component_name. These aliases can be used to link the components to the user's own targets, either libraries or executables.

The call requires the target chip to be specified with target argument. Optional arguments for the call include:

idf_build_executable(executable)

Specify the executable executable for ESP-IDF build. This attaches additional targets such as dependencies related to flashing, generating additional binary files, etc. Should be called after idf_build_process.

idf_build_get_config(var config [GENERATOR_EXPRESSION])

Get the value of the specified config. Much like build properties, specifying GENERATOR_EXPRESSION will retrieve the generator expression string for that config, instead of the actual value, which can be used with CMake commands that support generator expressions. Actual config values are only known after call to idf_build_process, however.

Idf-build-properties

These are properties that describe the build. Values of build properties can be retrieved by using the build command idf_build_get_property. For example, to get the Python interpreter used for the build:

idf_build_get_property(python PYTHON) message(STATUS "The Python interpreter is: ${python}")

Idf-component-commands

idf_component_get_property(var component property [GENERATOR_EXPRESSION])

Retrieve a specified component's component property, property and store it in var accessible from the current scope. Specifying GENERATOR_EXPRESSION will retrieve the generator expression string for that property, instead of the actual value, which can be used with CMake commands that support generator expressions.

idf_component_set_property(component property val [APPEND])

Set a specified component's component property, property with value val. Specifying APPEND will append the specified value to the current value of the property. If the property does not previously exist or it is currently empty, the specified value becomes the first element/member instead.

idf_component_register([[SRCS src1 src2 ...] | [[SRC_DIRS dir1 dir2 ...] [EXCLUDE_SRCS src1 src2 ...]] [INCLUDE_DIRS dir1 dir2 ...] [PRIV_INCLUDE_DIRS dir1 dir2 ...] [REQUIRES component1 component2 ...] [PRIV_REQUIRES component1 component2 ...] [LDFRAGMENTS ldfragment1 ldfragment2 ...] [REQUIRED_IDF_TARGETS target1 target2 ...] [EMBED_FILES file1 file2 ...] [EMBED_TXTFILES file1 file2 ...] [KCONFIG kconfig] [KCONFIG_PROJBUILD kconfig_projbuild] [WHOLE_ARCHIVE])

Register a component to the build system. Much like the project() CMake command, this should be called from the component's CMakeLists.txt directly (not through a function or macro) and is recommended to be called before any other command. Here are some guidelines on what commands can not be called before idf_component_register:

Commands that set and operate on variables are generally okay to call before idf_component_register.

The arguments for idf_component_register include:

The following are used for embedding data into the component, and is considered as source files when determining if a component is config-only. This means that even if the component does not specify source files, a static library is still created internally for the component if it specifies either:

Idf-component-properties

These are properties that describe a component. Values of component properties can be retrieved by using the build command idf_component_get_property. For example, to get the directory of the freertos component:

idf_component_get_property(dir freertos COMPONENT_DIR) message(STATUS "The 'freertos' component directory is: ${dir}")

File Globbing & Incremental Builds

The preferred way to include source files in an ESP-IDF component is to list them manually via SRCS argument to idf_component_register:

idf_component_register(SRCS library/a.c library/b.c platform/platform.c ...)

This preference reflects the CMake best practice of manually listing source files. This could, however, be inconvenient when there are lots of source files to add to the build. The ESP-IDF build system provides an alternative way for specifying source files using SRC_DIRS:

idf_component_register(SRC_DIRS library platform ...)

This uses globbing behind the scenes to find source files in the specified directories. Be aware, however, that if a new source file is added and this method is used, then CMake won't know to automatically re-run and this file won't be added to the build.

The trade-off is acceptable when you're adding the file yourself, because you can trigger a clean build or run idf.py reconfigure to manually re-run CMake. However, the problem gets harder when you share your project with others who may check out a new version using a source control tool like Git...

For components which are part of ESP-IDF, we use a third party Git CMake integration module (/tools/cmake/third_party/GetGitRevisionDescription.cmake) which automatically re-runs CMake any time the repository commit changes. This means if you check out a new ESP-IDF version, CMake will automatically rerun.

For project components (not part of ESP-IDF), there are a few different options:

The best option will depend on your particular project and its users.

Build System Metadata

For integration into IDEs and other build systems, when CMake runs the build process generates a number of metadata files in the build/ directory. To regenerate these files, run cmake or idf.py reconfigure (or any other idf.py build command).

JSON Configuration Server

A tool called kconfserver is provided to allow IDEs to easily integrate with the configuration system logic. kconfserver is designed to run in the background and interact with a calling process by reading and writing JSON over process stdin & stdout.

You can run kconfserver from a project via idf.py confserver or ninja kconfserver, or a similar target triggered from a different build generator.

For more information about kconfserver, see the esp-idf-kconfig documentation.

Build System Internals

Build Scripts

The listfiles for the ESP-IDF build system reside in /tools/cmake. The modules which implement core build system functionality are as follows:

  • build.cmake - Build related commands i.e., build initialization, retrieving/setting build properties, build processing.
  • component.cmake - Component related commands i.e., adding components, retrieving/setting component properties, registering components.
  • kconfig.cmake - Generation of configuration files (sdkconfig, sdkconfig.h, sdkconfig.cmake, etc.) from Kconfig files.
  • ldgen.cmake - Generation of final linker script from linker fragment files.
  • target.cmake - Setting build target and toolchain file.
  • utilities.cmake - Miscellaneous helper commands.

Aside from these files, there are two other important CMake scripts in /tools/cmake:

  • idf.cmake - Sets up the build and includes the core modules listed above. Included in CMake projects in order to access ESP-IDF build system functionality.
  • project.cmake - Includes idf.cmake and provides a custom project() command that takes care of all the heavy lifting of building an executable. Included in the top-level CMakeLists.txt of standard ESP-IDF projects.

The rest of the files in /tools/cmake are support or third-party scripts used in the build process.

Build Process

This section describes the standard ESP-IDF application build process. The build process can be broken down roughly into four phases:

ESP-IDF Build System Process

Initialization

This phase sets up necessary parameters for the build.

  • Upon inclusion of idf.cmake in project.cmake, the following steps are performed:
    • Set IDF_PATH from environment variable or inferred from path to project.cmake included in the top-level CMakeLists.txt.
    • Add /tools/cmake to CMAKE_MODULE_PATH and include core modules plus the various helper/third-party scripts.
    • Set build tools/executables such as default Python interpreter.
    • Get ESP-IDF git revision and store as IDF_VER.
    • Set global build specifications i.e., compile options, compile definitions, include directories for all components in the build.
    • Add components in components to the build.
  • The initial part of the custom project() command performs the following steps:
    • Set IDF_TARGET from environment variable or CMake cache and the corresponding CMAKE_TOOLCHAIN_FILE to be used.
    • Add components in EXTRA_COMPONENT_DIRS to the build.
    • Prepare arguments for calling command idf_build_process() from variables such as COMPONENTS/EXCLUDE_COMPONENTS, SDKCONFIG, SDKCONFIG_DEFAULTS.

The call to idf_build_process() command marks the end of this phase.

Enumeration

This phase builds a final list of components to be processed in the build, and is performed in the first half of idf_build_process().

  • Retrieve each component's public and private requirements. A child process is created which executes each component's CMakeLists.txt in script mode. The values of idf_component_register REQUIRES and PRIV_REQUIRES argument is returned to the parent build process. This is called early expansion. The variable CMAKE_BUILD_EARLY_EXPANSION is defined during this step.
  • Recursively include components based on public and private requirements.
  • Unless IDF Component Manager is disabled, it is called to resolve the dependencies of the components: - Looks for manifests and dependencies contained in the project. - Starts the version solving process to resolve the dependencies of the components. - When the version solving process succeeds, the IDF Component Manager downloads dependencies, integrates them into the build, and creates a dependencies.lock file that contains a list of the exact versions of the dependencies installed by the IDF Component Manager.

Processing

This phase processes the components in the build, and is the second half of idf_build_process().

Finalization

This phase is everything after idf_build_process().

Browse /tools/cmake/project.cmake for more details.

Migrating from ESP-IDF GNU Make System

Some aspects of the CMake-based ESP-IDF build system are very similar to the older GNU Make-based system. The developer needs to provide values the include directories, source files etc. There is a syntactical difference, however, as the developer needs to pass these as arguments to the registration command, idf_component_register.

Automatic Conversion Tool

An automatic project conversion tool is available in tools/cmake/convert_to_cmake.py in ESP-IDF v4.x releases. The script was removed in v5.0 because of its make build system dependency.

No Longer Available in CMake

Some features are significantly different or removed in the CMake-based system. The following variables no longer exist in the CMake-based build system:

No Default Values

Unlike in the legacy Make-based build system, the following have no default values:

No Longer Necessary

Flashing from Make

make flash and similar targets still work to build and flash. However, project sdkconfig no longer specifies serial port and baud rate. Environment variables can be used to override these. See Flashing with Ninja or Make for more details.

Application Examples