Debugging NGINX (original) (raw)

Debugging helps to identify a bug in the program code if something goes wrong. It is generally used in developing or testing third-party or experimental modules.

NGINX debugging features include the debugging log and creation of a core dump file with its further backtrace.

Configuring NGINX Binary For Debugging

First, you will need to enable debugging in NGINX binary. NGINX Plus already provides you with nginx-debug binary while NGINX Open Source requires recompilation.

Configuring F5 NGINX Plus Binary

Starting from Release 8, NGINX Plus ships the nginx-debug binary together with the standard binary. To enable debugging in NGINX Plus, you will need to switch from nginx to nginx-debug binary. Open terminal and run the command:

service nginx stop && service nginx-debug start

service nginx stop && service nginx-debug start

When finished, enable the debugging log in the configuration file.

Compiling NGINX Open Source Binary

To enable debugging in NGINX Open Source, you will need to recompile it with the --with-debug flag specified in the configure script.

To compile NGINX Open Source with the debug support:

Download and unpack NGINX source files, go to the directory with the source files. See Downloading the Sources.
Get the list of NGINX configure arguments. Run the command:

nginx -V 2>&1 | grep arguments

nginx -V 2>&1 | grep arguments

Add the --with-debug option to the list of configure commands and run the configure script:

./configure --with-debug <other configure arguments>

./configure --with-debug <other configure arguments>

Compile and install NGINX:
shell

sudo make  
sudo make install

sudo make  
sudo make install

Restart NGINX.

NGINX and Debugging Symbols

Debug symbols helps obtain additional information for debugging, such as functions, variables, data structures, source file and line number information.

NGINX by default is compiled with the “_-g_” flag that includes debug symbols.

However, if you get the “No symbol table info available” error when you run a backtrace, then debugging symbols are missing and you will need to recompile NGINX with support of debugging symbols.

The exact set of compiler flags depends on the compiler. For example, for the GCC compiler system:

include debugging symbols with the “_-g_” flag
make the debugger output easier to understand by disabling compiler optimization with the “_-O0_” flag:

./configure --with-debug --with-cc-opt='-O0 -g' ...

./configure --with-debug --with-cc-opt='-O0 -g' ...

Enabling Debug Logging in NGINX Configuration

The debugging log records errors and any debugging-related information and is disabled by default. To enable it, make sure NGINX is compiled to support debugging (see Configuring NGINX Binary For Debugging) and then enable it in NGINX configuration file with the debug parameter of the error_log directive. The debugging log may be written to a file, an allocated buffer in memory, stderr output, or to syslog.

It is recommended enabling the debugging log on the ”_main_“ level of NGINX configuration to get the full picture of what’s going on.

Writing the Debugging Log to a File

Writing the debugging log to a file may slow down performance under high load. Also note that the file can grow very large and quickly eat up disk space. To reduce the negative impact, you can configure the debugging log to be written into a memory buffer, or set the debugging log for particular IP addresses. See Writing the Debugging Log to Memory and Debug Log for Selected IPs for details.

To enable writing the debugging log to a file:

Make sure your NGINX is configured with the --with-debug configuration option. Run the command and check if the output contains the --with-debug line:

nginx -V 2>&1 | grep -- '--with-debug'

nginx -V 2>&1 | grep -- '--with-debug'

Open NGINX configuration file:

sudo vi /etc/nginx/nginx.conf

sudo vi /etc/nginx/nginx.conf

Find the error_log directive which is by default located in the main context, and change the logging level to debug. If necessary, change the path to the log file:

error_log  /var/log/nginx/error.log debug;

error_log  /var/log/nginx/error.log debug;

Save the configuration and exit the configuration file.

Writing the Debugging Log to Memory

The debugging log can be written to a memory using a cyclic buffer. The advantage is that logging on the debug level will not have significant impact on performance under high load.

To enable writing the debug log to memory:

Make sure your NGINX is configured with the --with-debug configuration option. Run the command and check if the output contains the --with-debug line:

nginx -V 2>&1 | grep -- '--with-debug'

nginx -V 2>&1 | grep -- '--with-debug'

In NGINX configuration file, enable a memory buffer for debug logging with the error_log directive specified in the main context:
nginx

error_log memory:32m debug;  
...  
http {  
    ...  
}

error_log memory:32m debug;  
...  
http {  
    ...  
}

Extracting Debug Log From Memory

The log can be extracted from the memory buffer using a script executed in the GDB debugger.

To extract the debugging log from memory:

Obtain the PID of NGINX worker process:
Launch the GDB debugger:

sudo gdb -p <nginx PID obtained at the previous step>

sudo gdb -p <nginx PID obtained at the previous step>

Copy the script, paste it to GDB and press “Enter”. The script will save the log in the debug_log.txt file located in the current directory:
nginx

set $log = ngx_cycle->log  
while $log->writer != ngx_log_memory_writer  
    set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi><mi>o</mi><mi>g</mi><mo>=</mo></mrow><annotation encoding="application/x-tex">log = </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.01968em;">l</span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span></span></span></span>log->next  
end  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mi>u</mi><mi>f</mi><mo>=</mo><mo stretchy="false">(</mo><mi>n</mi><mi>g</mi><msub><mi>x</mi><mi>l</mi></msub><mi>o</mi><msub><mi>g</mi><mi>m</mi></msub><mi>e</mi><mi>m</mi><mi>o</mi><mi>r</mi><msub><mi>y</mi><mi>b</mi></msub><mi>u</mi><msub><mi>f</mi><mi>t</mi></msub><mo>∗</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">buf = (ngx_log_memory_buf_t *) </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">b</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">n</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em;">l</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">o</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">m</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">e</span><span class="mord mathnormal">m</span><span class="mord mathnormal" style="margin-right:0.02778em;">or</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">b</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">u</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.1076em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord">∗</span><span class="mclose">)</span></span></span></span>log->wdata  
dump binary memory debug_log.txt <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mi>u</mi><mi>f</mi><mo>−</mo><mo>&gt;</mo><mi>s</mi><mi>t</mi><mi>a</mi><mi>r</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">buf-&gt;start </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">b</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mord">−</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">&gt;</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6151em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.02778em;">r</span><span class="mord mathnormal">t</span></span></span></span>buf->end

set $log = ngx_cycle->log  
while $log->writer != ngx_log_memory_writer  
    set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi><mi>o</mi><mi>g</mi><mo>=</mo></mrow><annotation encoding="application/x-tex">log = </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.01968em;">l</span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span></span></span></span>log->next  
end  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mi>u</mi><mi>f</mi><mo>=</mo><mo stretchy="false">(</mo><mi>n</mi><mi>g</mi><msub><mi>x</mi><mi>l</mi></msub><mi>o</mi><msub><mi>g</mi><mi>m</mi></msub><mi>e</mi><mi>m</mi><mi>o</mi><mi>r</mi><msub><mi>y</mi><mi>b</mi></msub><mi>u</mi><msub><mi>f</mi><mi>t</mi></msub><mo>∗</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">buf = (ngx_log_memory_buf_t *) </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">b</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">n</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em;">l</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">o</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">m</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">e</span><span class="mord mathnormal">m</span><span class="mord mathnormal" style="margin-right:0.02778em;">or</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em;">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">b</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">u</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.1076em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord">∗</span><span class="mclose">)</span></span></span></span>log->wdata  
dump binary memory debug_log.txt <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mi>u</mi><mi>f</mi><mo>−</mo><mo>&gt;</mo><mi>s</mi><mi>t</mi><mi>a</mi><mi>r</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">buf-&gt;start </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">b</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mord">−</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">&gt;</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6151em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.02778em;">r</span><span class="mord mathnormal">t</span></span></span></span>buf->end

Quit GDB by pressing CTRL+D.
Open the file “_debug_log.txt_” located in the current directory:

Debug Log for Selected IPs

It is possible to enable the debugging log for a particular IP address or a range of IP addresses. Logging particular IPs may useful in a production environment as it will not negatively affect performance. The IP address is specified in the debug_connection directive within the events block; the directive can be defined more than once:

nginx

error_log /path/to/log;
...
events {
    debug_connection 192.168.1.1;
    debug_connection 192.168.10.0/24;
}

error_log /path/to/log;
...
events {
    debug_connection 192.168.1.1;
    debug_connection 192.168.10.0/24;
}

Debug Log for Each Virtual Host

Generally, the error_log directive is specified in the main context and thus is applied to all other contexts including server and location. But if there is another error_log directive specified inside a particular server or a location block, the global settings will be overridden and such error_log directive will set its own path to the log file and the level of logging.

To set up the debugging log for a particular virtual host, add the error_log directive inside a particular server block, in which set a new path to the log file and the debug logging level:

nginx

error_log /path1/to/log debug;
...
http {
    ...
    server {
    error_log /path2/to/log debug;
    ...
    }
}

error_log /path1/to/log debug;
...
http {
    ...
    server {
    error_log /path2/to/log debug;
    ...
    }
}

To disable the debugging log per a particular virtual host, specify the error_log directive inside a particular server block, and specify a path to the log file only:

nginx

error_log /path/to/log debug;
...
http {
    ...
    server {
    error_log /path/to/log;
    ...
    }
}

error_log /path/to/log debug;
...
http {
    ...
    server {
    error_log /path/to/log;
    ...
    }
}

A core dump file can help identify and fix problems that are causing NGINX to crash. A core dump file may contain sensitive information such as passwords and private keys, so ensure that they are treated securely.

In order to create a core dump file, they must be enabled in both the operating system and the NGINX configuration file.

Enabling Core Dumps in the Operating System

Perform the following steps in your operating system:

Specify a working directory in which a core dump file will be saved, for example, “_/tmp/cores_”:
Make sure the directory is writable by NGINX worker process:
shell

sudo chown root:root /tmp/cores  
sudo chmod 1777 /tmp/cores

sudo chown root:root /tmp/cores  
sudo chmod 1777 /tmp/cores

Disable the limit for the maximum size of a core dump file:

sudo prlimit --core=unlimited:unlimited --pid $(cat /run/nginx.pid)

sudo prlimit --core=unlimited:unlimited --pid $(cat /run/nginx.pid)

If the operation ends up with “Cannot modify limit: operation not permitted”, run the command:

sudo sh -c "ulimit -c unlimited && exec su $LOGNAME"

sudo sh -c "ulimit -c unlimited && exec su $LOGNAME"

Enable core dumps for the setuid and setgid processes.
For CentOS 7.0, Debian 8.2, Ubuntu 14.04, run the commands:
shell

echo "/tmp/cores/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern  
sudo sysctl -w fs.suid_dumpable=2  
sysctl -p

echo "/tmp/cores/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern  
sudo sysctl -w fs.suid_dumpable=2  
sysctl -p

For FreeBSD, run the commands:
shell

sudo sysctl kern.sugid_coredump=1  
sudo sysctl kern.corefile=/tmp/cores/%N.core.%P

sudo sysctl kern.sugid_coredump=1  
sudo sysctl kern.corefile=/tmp/cores/%N.core.%P

Enabling Core Dumps in NGINX Configuration

To enable core dumps in the NGINX configuration file:

Open the NGINX configuration file:

sudo vi /usr/local/etc/nginx/nginx.conf

sudo vi /usr/local/etc/nginx/nginx.conf

Define a directory that will keep core dump files with the working_directory directive. The directive is specified on the main configuration level:

working_directory /tmp/cores/;

working_directory /tmp/cores/;

Make sure the directory exists and is writable by NGINX worker process. Open terminal and run the commands:
shell

sudo chown root:root /tmp/cores  
sudo chmod 1777 /tmp/cores

sudo chown root:root /tmp/cores  
sudo chmod 1777 /tmp/cores

Specify the maximum possible size of the core dump file with the worker_rlimit_core directive. The directive is also specified on the main configuration level. If the core dump file size exceeds the value, the core dump file will not be created.

Example:

nginx

worker_processes   auto;
error_log          /var/log/nginx/error.log debug;
working_directory  /tmp/cores/;
worker_rlimit_core 500M;

events {
    ...
}

http {
    ...
}

worker_processes   auto;
error_log          /var/log/nginx/error.log debug;
working_directory  /tmp/cores/;
worker_rlimit_core 500M;

events {
    ...
}

http {
    ...
}

With these settings, a core dump file will be created in the “_/tmp/cores/_” directory, and only if its size does not exceed 500 megabytes.

Obtaining Backtrace From a Core Dump File

Backtraces provide information from a core dump file about what was wrong when a program crashed.

To get a backtrace from a core dump file:

Open a core dump file with the GDB debugger using the pattern:

sudo gdb <nginx_executable_path> <coredump_file_path>

sudo gdb <nginx_executable_path> <coredump_file_path>

Type-in the “backtrace command to get a stack trace from the time of the crash:

If the “_backtrace_” command resulted with the “No symbol table info available” message, you will need to recompile NGINX binary to include debugging symbols. See NGINX and Debugging Symbols.

Dumping NGINX Configuration From a Running Process

You can extract the current NGINX configuration from the master process in memory. This can be useful when you need to:

verify which configuration has been loaded
restore a previous configuration if the version on disk has been accidentally removed or overwritten

The configuration dump can be obtained with a GDB script provided that your NGINX has the debug support.

Make sure your NGINX is built with the debug support (the --with-debug configure option in the list of the configure arguments). Run the command and check if the output contains the --with-debug line:

nginx -V 2>&1 | grep -- '--with-debug'

nginx -V 2>&1 | grep -- '--with-debug'

Obtain the PID of NGINX worker process:
Launch the GDB debugger:

sudo gdb -p <nginx PID obtained at the previous step>

sudo gdb -p <nginx PID obtained at the previous step>

Copy and paste the script to GDB and press “Enter”. The script will save the configuration in the nginx_conf.txt file in the current directory:
nginx

set $cd = ngx_cycle->config_dump  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo>=</mo></mrow><annotation encoding="application/x-tex">nelts = </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span></span></span></span>cd.nelts  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo>=</mo><mo stretchy="false">(</mo><mi>n</mi><mi>g</mi><msub><mi>x</mi><mi>c</mi></msub><mi>o</mi><mi>n</mi><msub><mi>f</mi><mi>d</mi></msub><mi>u</mi><mi>m</mi><msub><mi>p</mi><mi>t</mi></msub><mo>∗</mo><mo stretchy="false">)</mo><mo stretchy="false">(</mo></mrow><annotation encoding="application/x-tex">elts = (ngx_conf_dump_t*)(</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">n</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1076em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">d</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord">∗</span><span class="mclose">)</span><span class="mopen">(</span></span></span></span>cd.elts)  
while ($nelts-- > 0)  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mi>a</mi><mi>m</mi><mi>e</mi><mo>=</mo></mrow><annotation encoding="application/x-tex">name = </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">nam</span><span class="mord mathnormal">e</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span></span></span></span>elts[$nelts]->name.data  
printf "Dumping %s to nginx_conf.txt\n", $name  
append memory nginx_conf.txt \  
      <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo stretchy="false">[</mo></mrow><annotation encoding="application/x-tex">elts[</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mopen">[</span></span></span></span>nelts]->buffer.start <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo stretchy="false">[</mo></mrow><annotation encoding="application/x-tex">elts[</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mopen">[</span></span></span></span>nelts]->buffer.end  
end

set $cd = ngx_cycle->config_dump  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo>=</mo></mrow><annotation encoding="application/x-tex">nelts = </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span></span></span></span>cd.nelts  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo>=</mo><mo stretchy="false">(</mo><mi>n</mi><mi>g</mi><msub><mi>x</mi><mi>c</mi></msub><mi>o</mi><mi>n</mi><msub><mi>f</mi><mi>d</mi></msub><mi>u</mi><mi>m</mi><msub><mi>p</mi><mi>t</mi></msub><mo>∗</mo><mo stretchy="false">)</mo><mo stretchy="false">(</mo></mrow><annotation encoding="application/x-tex">elts = (ngx_conf_dump_t*)(</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">n</span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1076em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">d</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord">∗</span><span class="mclose">)</span><span class="mopen">(</span></span></span></span>cd.elts)  
while ($nelts-- > 0)  
set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mi>a</mi><mi>m</mi><mi>e</mi><mo>=</mo></mrow><annotation encoding="application/x-tex">name = </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">nam</span><span class="mord mathnormal">e</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span></span></span></span>elts[$nelts]->name.data  
printf "Dumping %s to nginx_conf.txt\n", $name  
append memory nginx_conf.txt \  
      <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo stretchy="false">[</mo></mrow><annotation encoding="application/x-tex">elts[</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mopen">[</span></span></span></span>nelts]->buffer.start <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>l</mi><mi>t</mi><mi>s</mi><mo stretchy="false">[</mo></mrow><annotation encoding="application/x-tex">elts[</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">s</span><span class="mopen">[</span></span></span></span>nelts]->buffer.end  
end

Quit GDB by pressing CTRL+D.
Open the file nginx_conf.txt located in the current directory:

When asking for help with debugging, please provide the following information:

NGINX version, compiler version, and configure parameters. Run the command:
Current full NGINX configuration. See Dumping NGINX Configuration From a Running Process
The debugging log. See Enabling Debug Logging in NGINX Configuration
The obtained backtrace. See Enabling Core Dumps, Obtaining Backtrace