Heap Memory Debugging - ESP32 (original) (raw)

Overview

ESP-IDF integrates tools for requesting heap information, heap corruption detection, and heap tracing. These can help track down memory-related bugs.

For general information about the heap memory allocator, see Heap Memory Allocation.

Heap Information

To obtain information about the state of the heap, call the following functions:

heap_caps_get_free_size() can be used to return the current free memory for different memory capabilities.
heap_caps_get_largest_free_block() can be used to return the largest free block in the heap, which is also the largest single allocation currently possible. Tracking this value and comparing it to the total free heap allows you to detect heap fragmentation.
heap_caps_get_minimum_free_size() can be used to track the heap "low watermark" since boot.
heap_caps_get_info() returns a multi_heap_info_t structure, which contains the information from the above functions, plus some additional heap-specific data (number of allocations, etc.).
heap_caps_print_heap_info() prints a summary of the information returned by heap_caps_get_info() to stdout.
heap_caps_dump() and heap_caps_dump_all() output detailed information about the structure of each block in the heap. Note that this can be a large amount of output.

Heap Allocation and Free Function Hooks

Users can use allocation and free detection hooks to be notified of every successful allocation and free operation:

Providing a definition of esp_heap_trace_alloc_hook() allows you to be notified of every successful memory allocation operation.
Providing a definition of esp_heap_trace_free_hook() allows you to be notified of every successful memory-free operations.

This feature can be enabled by setting the CONFIG_HEAP_USE_HOOKS option. esp_heap_trace_alloc_hook() and esp_heap_trace_free_hook() have weak declarations (e.g., __attribute__((weak))), thus it is not necessary to provide declarations for both hooks. Given that it is technically possible to allocate and free memory from an ISR (though strongly discouraged from doing so), the esp_heap_trace_alloc_hook() and esp_heap_trace_free_hook() can potentially be called from an ISR.

It is not recommended to perform (or call API functions to perform) blocking operations or memory allocation/free operations in the hook functions. In general, the best practice is to keep the implementation concise and leave the heavy computation outside of the hook functions.

The example below shows how to define the allocation and free function hooks:

#include "esp_heap_caps.h"

void esp_heap_trace_alloc_hook(void* ptr, size_t size, uint32_t caps) { ... } void esp_heap_trace_free_hook(void* ptr) { ... }

void app_main() { ... }

Memory Allocation Failed Hook

Users can use heap_caps_register_failed_alloc_callback() to register a callback that is invoked every time an allocation operation fails.

Additionally, users can enable the CONFIG_HEAP_ABORT_WHEN_ALLOCATION_FAILS, which will automatically trigger a system abort if any allocation operation fails.

The example below shows how to register an allocation failure callback:

#include "esp_heap_caps.h"

void heap_caps_alloc_failed_hook(size_t requested_size, uint32_t caps, const char *function_name) { printf("%s was called but failed to allocate %d bytes with 0x%X capabilities. \n",function_name, requested_size, caps); }

void app_main() { ... esp_err_t error = heap_caps_register_failed_alloc_callback(heap_caps_alloc_failed_hook); ... void *ptr = heap_caps_malloc(allocation_size, MALLOC_CAP_DEFAULT); ... }

Heap Corruption Detection

Heap corruption detection allows you to detect various types of heap memory errors:

Out-of-bound writes & buffer overflows
Writes to freed memory
Reads from freed or uninitialized memory

Three levels of corruption detection are available. Each one providing a finer level of detection than the previous:

Basic (No Poisoning)
Light Impact
Comprehensive

Assertions

The heap implementation (heap/multi_heap.c, etc.) includes numerous assertions that will fail if the heap memory is corrupted. To detect heap corruption most effectively, ensure that assertions are enabled in the project configuration via the CONFIG_COMPILER_OPTIMIZATION_ASSERTION_LEVEL option.

If a heap integrity assertion fails, a line will be printed like CORRUPT HEAP: multi_heap.c:225 detected at 0x3ffbb71c. The memory address printed is the address of the heap structure that has corrupt content.

It is also possible to manually check heap integrity by calling heap_caps_check_integrity_all() or related functions. This function checks all of the requested heap memory for integrity and can be used even if assertions are disabled. If the integrity checks detects an error, it will print the error along with the address(es) of corrupt heap structures.

Finding Heap Corruption

Memory corruption can be one of the hardest classes of bugs to find and fix, as the source of the corruption could be completely unrelated to the symptoms of the corruption. Here are some tips:

A crash with a CORRUPT HEAP: message usually includes a stack trace, but this stack trace is rarely useful. The crash is the symptom of memory corruption when the system realizes the heap is corrupt. But usually, the corruption happens elsewhere and earlier in time.
Increasing the heap memory debugging Configuration level to "Light impact" or "Comprehensive" gives you a more accurate message with the first corrupt memory address.
Adding regular calls to heap_caps_check_integrity_all() or heap_caps_check_integrity_addr() in your code helps you pin down the exact time that the corruption happened. You can move these checks around to "close in on" the section of code that corrupted the heap.
Based on the memory address that has been corrupted, you can use JTAG debugging to set a watchpoint on this address and have the CPU halt when it is written to.
If you do not have JTAG, but you do know roughly when the corruption happens, set a watchpoint in software just beforehand via esp_cpu_set_watchpoint(). A fatal exception will occur when the watchpoint triggers. The following is an example of how to use the function - esp_cpu_set_watchpoint(0, (void *)addr, 4, ESP_WATCHPOINT_STORE). Note that watchpoints are per-CPU and are set on the current running CPU only. So if you do not know which CPU is corrupting memory, call this function on both CPUs.
For buffer overflows, heap tracing in HEAP_TRACE_ALL mode tells which callers are allocating which addresses from the heap. See Heap Tracing To Find Heap Corruption for more details. You can try to find the function that allocates memory with an address immediately before the corrupted address, since it is probably the function that overflows the buffer.
Calling heap_caps_dump() or heap_caps_dump_all() can give an indication of what heap blocks are surrounding the corrupted region and may have overflowed or underflowed, etc.

Configuration

Temporarily increasing the heap corruption detection level can give more detailed information about heap corruption errors.

In the project configuration menu, under Component config, there is a menu Heap memory debugging. The option CONFIG_HEAP_CORRUPTION_DETECTION can be set to one of the following three levels:

Basic (No Poisoning)

This is the default level. By default, no special heap corruption features are enabled, but the provided assertions are enabled. A heap corruption error will be printed if any of the heap's internal data structures appear overwritten or corrupted. This usually indicates a buffer overrun or out-of-bounds write.

If assertions are enabled, an assertion will also trigger if a double-free occurs (the same memory is freed twice).

Calling heap_caps_check_integrity() in Basic mode checks the integrity of all heap structures, and print errors if any appear to be corrupted.

Light Impact

This level incorporates the "Basic" detection features. Additionally, each block of memory allocated is "poisoned" with head and tail "canary bytes". If an application writes over the "canary bytes", they will be seen as corrupted and integrity checks will fail.

The head canary word is 0xABBA1234 (3412BAAB in byte order), and the tail canary word is 0xBAAD5678 (7856ADBA in byte order).

With basic heap corruption checks, most out-of-bound writes can be detected and the number of overrun bytes before a failure is detected depends on the properties of the heap. However, the Light Impact mode is more precise as even a single-byte overrun can be detected.

Enabling light-impact checking increases the memory usage since each individual allocation uses additional bytes of metadata.

Each time heap_caps_free() is called in Light Impact mode, the head and tail canary bytes of the buffer being freed are checked against the expected values.

When heap_caps_check_integrity() or heap_caps_check_integrity_all() is called, all allocated blocks of heap memory have their canary bytes checked against the expected values.

In both cases, the functions involve checking that the first 4 bytes of an allocated block (before the buffer is returned to the user) should be the word 0xABBA1234, and the last 4 bytes of the allocated block (after the buffer is returned to the user) should be the word 0xBAAD5678.

Different values usually indicate buffer underrun or overrun. Overrun indicates that when writing to memory, the data written exceeds the size of the allocated memory, resulting in writing to an unallocated memory area; underrun indicates that when reading memory, the data read exceeds the allocated memory and reads data from an unallocated memory area.

Comprehensive

This level incorporates the "Light Impact" detection features. Additionally, it checks for uninitialized-access and use-after-free bugs. In this mode, all freshly allocated memory is filled with the pattern 0xCE, and all freed memory is filled with the pattern 0xFE.

Enabling Comprehensive mode has a substantial impact on runtime performance, as all memory needs to be set to the allocation patterns each time a heap_caps_malloc() or heap_caps_free() completes, and the memory also needs to be checked each time. However, this mode allows easier detection of memory corruptions which are much more subtle to find otherwise. It is recommended to only enable this mode when debugging, not in production.

The checks for allocated and free patterns (0xCE and 0xFE, respectively) are also done when calling heap_caps_check_integrity() or heap_caps_check_integrity_all().

Crashes in Comprehensive Mode

If an application crashes when reading or writing an address related to 0xCECECECE in Comprehensive mode, it indicates that it has read uninitialized memory. The application should be changed to either use heap_caps_calloc() (which zeroes memory), or initialize the memory before using it. The value 0xCECECECE may also be seen in stack-allocated automatic variables, because, in ESP-IDF, most task stacks are originally allocated from the heap, and in C, stack memory is uninitialized by default.

If an application crashes, and the exception register dump indicates that some addresses or values were 0xFEFEFEFE, this indicates that it is reading heap memory after it has been freed, i.e., a "use-after-free bug". The application should be changed to not access heap memory after it has been freed.

If a call to heap_caps_malloc() or heap_caps_realloc() causes a crash because it was expected to find the pattern 0xFEFEFEFE in free memory and a different pattern was found, it indicates that the app has a use-after-free bug where it is writing to memory that has already been freed.

Manual Heap Checks in Comprehensive Mode

Calls to heap_caps_check_integrity() or heap_caps_check_integrity_all() may print errors relating to 0xFEFEFEFE, 0xABBA1234, or 0xBAAD5678. In each case the checker is expected to find a given pattern, and will error out if not found:

For free heap blocks, the checker expects to find all bytes set to 0xFE. Any other values indicate a use-after-free bug where free memory has been incorrectly overwritten.
For allocated heap blocks, the behavior is the same as for the Light Impact mode. The canary bytes 0xABBA1234 and 0xBAAD5678 are checked at the head and tail of each allocated buffer, and any variation indicates a buffer overrun or underrun.

Heap Task Tracking

The Heap Task Tracking can be enabled via the menuconfig: Component config > Heap memory debugging > Enable heap task tracking (see CONFIG_HEAP_TASK_TRACKING).

The feature allows users to track the heap memory usage of each task created since startup and provides a series of statistics that can be accessed via getter functions or simply dumped into the stream of the user's choosing. This feature is useful for identifying memory usage patterns and potential memory leaks.

An additional configuration can be enabled by the user via the menuconfig: Component config > Heap memory debugging > Keep information about the memory usage of deleted tasks (see CONFIG_HEAP_TRACK_DELETED_TASKS) to keep the statistics collected for a given task even after it is deleted.

Note

Note that the Heap Task Tracking cannot detect the deletion of statically allocated tasks. Therefore, users will have to keep in mind while reading the following section that statically allocated tasks will always be considered alive in the scope of the Heap Task Tracking feature.

It is important to mention that its usage is strongly discouraged for other purposes than debugging for the following reasons:

Tracking the allocations and storing the resulting statistics for each task requires a non-negligible RAM usage overhead.
The overall performance of the heap allocator is severely impacted due to the additional processing required for each allocation and free operation.

Note

Note that the memory allocated by the heap task tracking feature will not be visible when dumping or accessing the statistics.

Structure of the statistics and information

For a given task, the heap task tracking feature categorizes statistics on three different levels:

The task level statistics
The heap level statistics
The allocation level statistics

The task level statistics provides the following information:

Name of the given task
Task handle of the given task
Status of the given task (if the task is running or deleted)
Peak memory usage of the given task (the maximum amount of memory used by the given task during the task lifetime)
Current memory usage of the given task
Number of heaps in which the task has allocated memory

The heap level statistics provides the following information for each heap used by the given task:

Name of the given heap
Caps the capabilities of the given heap (without priority)
Size the total size of the given heap
Current usage of the given task on the given heap
Peak usage of the given task on the given heap
Number of allocations done by the given task for on the given heap

The allocation level statistics provides the following information for each allocation done by the given task on the given heap:

Address of the given allocation
Size of the given allocation

Dumping the statistics and information

The heap_caps_print_single_task_stat_overview() API prints an overview of heap usage for a specific task to the provided output stream.

┌────────────────────┬─────────┬──────────────────────┬───────────────────┬─────────────────┐ │ TASK │ STATUS │ CURRENT MEMORY USAGE │ PEAK MEMORY USAGE │ TOTAL HEAP USED │ ├────────────────────┼─────────┼──────────────────────┼───────────────────┼─────────────────┤ │ task_name │ ALIVE │ 0 │ 7152 │ 1 │ └────────────────────┴─────────┴──────────────────────┴───────────────────┴─────────────────┘

heap_caps_print_all_task_stat_overview() prints an overview of heap usage for all tasks (including the deleted tasks if CONFIG_HEAP_TRACK_DELETED_TASKS is enabled).

┌────────────────────┬─────────┬──────────────────────┬───────────────────┬─────────────────┐ │ TASK │ STATUS │ CURRENT MEMORY USAGE │ PEAK MEMORY USAGE │ TOTAL HEAP USED │ ├────────────────────┼─────────┼──────────────────────┼───────────────────┼─────────────────┤ │ task_name │ DELETED │ 11392 │ 11616 │ 1 │ │ other_task_name │ ALIVE │ 0 │ 9408 │ 2 │ │ main │ ALIVE │ 3860 │ 7412 │ 2 │ │ ipc1 │ ALIVE │ 32 │ 44 │ 1 │ │ ipc0 │ ALIVE │ 10080 │ 10092 │ 1 │ │ Pre-scheduler │ ALIVE │ 2236 │ 2236 │ 1 │ └────────────────────┴─────────┴──────────────────────┴───────────────────┴─────────────────┘

Note

Note that the task named “Pre-scheduler” represents allocations that occurred before the scheduler was started. It is not an actual task, so the "status" field (which is shown as "ALIVE") is not meaningful and should be ignored.

Use heap_caps_print_single_task_stat() to dump the complete set of statistics for a specific task, or heap_caps_print_all_task_stat() to dump statistics for all tasks:

[...] ├ ALIVE: main, CURRENT MEMORY USAGE 308, PEAK MEMORY USAGE 7412, TOTAL HEAP USED 2: │ ├ HEAP: RAM, CAPS: 0x0010580e, SIZE: 344400, USAGE: CURRENT 220 (0%), PEAK 220 (0%), ALLOC COUNT: 2 │ │ ├ ALLOC 0x3fc99024, SIZE 88 │ │ ├ ALLOC 0x3fc99124, SIZE 132 │ └ HEAP: RAM, CAPS: 0x0010580e, SIZE: 22308, USAGE: CURRENT 88 (0%), PEAK 7192 (32%), ALLOC COUNT: 5 │ ├ ALLOC 0x3fce99f8, SIZE 20 │ ├ ALLOC 0x3fce9a10, SIZE 12 │ ├ ALLOC 0x3fce9a20, SIZE 16 │ ├ ALLOC 0x3fce9a34, SIZE 20 │ ├ ALLOC 0x3fce9a4c, SIZE 20 [...] └ ALIVE: Pre-scheduler, CURRENT MEMORY USAGE 2236, PEAK MEMORY USAGE 2236, TOTAL HEAP USED 1: └ HEAP: RAM, CAPS: 0x0010580e, SIZE: 344400, USAGE: CURRENT 2236 (0%), PEAK 2236 (0%), ALLOC COUNT: 11 ├ ALLOC 0x3fc95cb0, SIZE 164 ├ ALLOC 0x3fc95dd8, SIZE 12 ├ ALLOC 0x3fc95dfc, SIZE 12 ├ ALLOC 0x3fc95e20, SIZE 16 ├ ALLOC 0x3fc95e48, SIZE 24 ├ ALLOC 0x3fc95e78, SIZE 88 ├ ALLOC 0x3fc95ee8, SIZE 88 ├ ALLOC 0x3fc95f58, SIZE 88 ├ ALLOC 0x3fc95fc8, SIZE 88 ├ ALLOC 0x3fc96038, SIZE 1312 ├ ALLOC 0x3fc96570, SIZE 344

Note

The dump shown above has been truncated (see "[...]") for readability reasons and only displays the statistics and information of the main task and the Pre-scheduler. The goal here is only to demonstrate the information displayed when calling the heap_caps_print_all_task_stat() (resp. heap_caps_print_single_task_stat()) API functions.

Getting the statistics and information

heap_caps_get_single_task_stat() allows the user to access information of a specific task. The information retrieved by calling this API is identical to the one dumped using heap_caps_print_single_task_stat().

heap_caps_get_all_task_stat() allows the user to access an overview of the information of all tasks (including the deleted tasks if CONFIG_HEAP_TRACK_DELETED_TASKS is enabled). The information retrieved by calling this API is identical to the one dumped using heap_caps_print_all_task_stat().

Each getter function requires a pointer to the data structure that will be used by the heap task tracking to gather the statistics and information of a given task (or all tasks). This data structure contains pointers to arrays that the user can allocate statically or dynamically.

Due to the difficulty of estimating the size of the arrays used to store information — since it's hard to determine the number of allocations per task, the number of heaps used by each task, and the number of tasks created since startup — the heap task tracking also provides heap_caps_alloc_single_task_stat_arrays() (resp. heap_caps_alloc_all_task_stat_arrays()) to dynamically allocate the required amount of memory for those arrays.

Similarly, the heap task tracking also provides heap_caps_free_single_task_stat_arrays() (resp. heap_caps_free_all_task_stat_arrays()) to free the memory dynamically allocated when calling heap_caps_alloc_single_task_stat_arrays() (resp. heap_caps_alloc_all_task_stat_arrays()).

Heap Tracing

Heap Tracing allows the tracing of code which allocates or frees memory. Two tracing modes are supported:

Standalone. In this mode, traced data are kept on-board, so the size of the gathered information is limited by the buffer assigned for that purpose, and the analysis is done by the on-board code. There are a couple of APIs available for accessing and dumping collected info.
Host-based. This mode does not have the limitation of the standalone mode, because traced data are sent to the host over JTAG connection using app_trace library. Later on, they can be analyzed using special tools.

Heap tracing can perform two functions:

Leak checking: find memory that is allocated and never freed.
Heap use analysis: show all functions that are allocating or freeing memory while the trace is running.

How to Diagnose Memory Leaks

If you suspect a memory leak, the first step is to figure out which part of the program is leaking memory. Use the heap_caps_get_free_size() or related functions in heap information to track memory use over the life of the application. Try to narrow the leak down to a single function or sequence of functions where free memory always decreases and never recovers.

Standalone Mode

Once you have identified the code which you think is leaking:

Enable the CONFIG_HEAP_TRACING_DEST option.
Call the function heap_trace_init_standalone() early in the program, to register a buffer that can be used to record the memory trace.
Call the function heap_trace_start() to begin recording all mallocs or frees in the system. Call this immediately before the piece of code which you suspect is leaking memory.
Call the function heap_trace_stop() to stop the trace once the suspect piece of code has finished executing. This state will stop the tracing of both allocations and frees.
Call the function heap_trace_alloc_pause() to pause the tracing of new allocations while continuing to trace the frees. Call this immediately after the piece of code which you suspect is leaking memory to prevent any new allocations to be recorded.
Call the function heap_trace_dump() to dump the results of the heap trace.

The following code snippet demonstrates how application code would typically initialize, start, and stop heap tracing:

#include "esp_heap_trace.h"

#define NUM_RECORDS 100 static heap_trace_record_t trace_record[NUM_RECORDS]; // This buffer must be in internal RAM

...

void app_main() { ... ESP_ERROR_CHECK( heap_trace_init_standalone(trace_record, NUM_RECORDS) ); ... }

void some_function() { ESP_ERROR_CHECK( heap_trace_start(HEAP_TRACE_LEAKS) );

do_something_you_suspect_is_leaking();

ESP_ERROR_CHECK( heap_trace_stop() );
heap_trace_dump();
...

}

The output from the heap trace has a similar format to the following example:

====== Heap Trace: 8 records (8 capacity) ====== 6 bytes (@ 0x3fc9f620, Internal) allocated CPU 0 ccount 0x1a31ac84 caller 0x40376321:0x40376379 0x40376321: heap_caps_malloc at /path/to/idf/examples/components/heap/heap_caps.c:84 0x40376379: heap_caps_malloc_default at /path/to/idf/examples/components/heap/heap_caps.c:110

freed by 0x403839e4:0x42008096 0x403839e4: free at /path/to/idf/examples/components/newlib/heap.c:40 0x42008096: test_func_74 at /path/to/idf/examples/components/heap/test_apps/heap_tests/main/test_heap_trace.c:104 (discriminator 3)

9 bytes (@ 0x3fc9f630, Internal) allocated CPU 0 ccount 0x1a31b618 caller 0x40376321:0x40376379