GitHub - aws/aws-imds-packet-analyzer (original) (raw)

Introduction

The AWS ImdsPacketAnalyzer is a tool that traces TCP interactions with the EC2 Instance Metadata Service (IMDS). This can assist in identifying the processes making IMDSv1 calls on a host. Traces contain the pid, the argv used to launch the process, and the parent pids up to four levels deep. This information allows you to identify a Process making IMDSv1 calls for further investigation.

The ImdsPacketAnalyzer leverages the BCC (BPF Compiler Collection). In order to successfully run the analyzer the BCC pre-requisites need to be installed.

AWS ImdsPacketAnalyzer

Packages - Installing BCC

For hosts with internet access, the install script can be used. It is advised that this script is run only on non-production instances. Installation will update dependancies and may affect other functionality. For instances without internet access you will need to share the files on an S3 folder.

sudo bash install-deps.sh

OR run the following commands per OS

Amazon Linux 2023

Install BCC

sudo dnf install bcc-tools

Amazon Linux 2

Install BCC (BPF Compiler Collection):

sudo amazon-linux-extras enable BCC
sudo yum install kernel-devel-$(uname -r)
sudo yum install bcc

Amazon Linux 1, 2018.03

Install BCC (BPF Compiler Collection):

sudo yum install kernel-headers-$(uname -r | cut -d'.' -f1-5)
sudo yum install kernel-devel-$(uname -r | cut -d'.' -f1-5)
sudo yum install bcc

Debian 11

echo deb http://cloudfront.debian.net/debian sid main | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install -y bpfcc-tools libbpfcc libbpfcc-dev linux-headers-$(uname -r)
sudo apt-get install linux-headers-$(uname -r) bcc

Debian 10

Note : During the Dependency installation, the "libcrypt1" related error occurs so the execution has step to fix and continue with the installation process further, also the OS libraries can cause the restart of the system releated services like sshd and crond.

echo deb http://cloudfront.debian.net/debian sid main | sudo tee -a /etc/apt/sources.list
sudo -i         # Need to switch to root user in the CLI before running below command
apt-get update
#Set the environment variable DEBIAN_FRONTEND to 'noninteractive' to avoid the prompts and accept the default answers
export DEBIAN_FRONTEND=noninteractive
apt-get install -y bpfcc-tools libbpfcc libbpfcc-dev linux-headers-$(uname -r) bcc --no-install-recommends
#Steps to fix the libcrypt1 error
cd /tmp/
apt -y download libcrypt1
dpkg-deb -x libcrypt1* .
cp -av lib/x86_64-linux-gnu/* /lib/x86_64-linux-gnu/
#Re-run the install command
apt-get install -y bpfcc-tools libbpfcc libbpfcc-dev linux-headers-$(uname -r) bcc --no-install-recommends
apt install -y --fix-broken
# Run the install command
apt-get install -y bpfcc-tools libbpfcc libbpfcc-dev linux-headers-$(uname -r) bcc --no-install-recommends

Ubuntu 20 / 22

For Ubuntu 20 -

sudo apt install -y bison build-essential cmake flex git libedit-dev libllvm12 llvm-12-dev libclang-12-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils zip

For Ubuntu 22 -

sudo apt install -y bison build-essential cmake flex git libedit-dev libllvm14 llvm-14-dev libclang-14-dev python3 zlib1g-dev libelf-dev libfl-dev python3-distutils zip

git clone https://github.com/iovisor/bcc.git
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd
sudo apt-get install linux-headers-$(uname -r)

RHEL 8 / 9 / Fedora

sudo yum -y install bcc-tools libbpf

SLES 15

sudo zypper ref
sudo zypper in bcc-tools bcc-examples
sudo zypper in --oldpackage kernel-default-devel-$(zypper se -s kernel-default-devel | awk '{split($0,a,"|"); print a[4]}' | grep $(uname -r | awk '{gsub("-default", "");print}') | sed -e 's/^[ \t]*//' | tail -n 1)

CentOS 9

WINDOWS

INSTALL PYTHON

INSTALL PIP

INSTALL GIT

INSTALL AWS CLI

INSTALL METABADGER (https://github.com/salesforce/metabadger)

Note: Troubleshooting + Installation on other distros please see: BCC (BPF Compiler Collection)

Usage

BCC requires that the analyzer is run with root permissions. Typically, you can execute the following script and IMDS calls will be logged to the console and to a log file by default (see logging.conf).

sudo python3 src/imds_snoop.py

Example v1 call:

The following IMDSv1 curl command

curl http://169.254.169.254/latest/meta-data/

will result the following IMDS packet analyzer output

IMDSv1(!) (pid:6028:curl argv:curl http://169.254.169.254/latest/meta-data/) called by -> (pid:6027:makeCalls.sh argv:/bin/bash ./makeCalls.sh) -> (pid:4081:zsh argv:-zsh) -> (pid:4081:sshd argv:sshd: kianred@pts/0)

Amazon Linux 2023

sudo python3 src/imds_snoop.py

Amazon Linux 2

sudo python3 src/imds_snoop.py

Amazon Linux 1

sudo python3 src/imds_snoop.py

Debian 11

sudo python3 src/imds_snoop.py

Debian 10

sudo python3 src/imds_snoop.py

Ubuntu 20 / 22

sudo LD_PRELOAD=/home/ubuntu/bcc/build/src/cc/libbcc.so.0 PYTHONPATH=/home/ubuntu/bcc/build/src/python src/imds_snoop.py

RHEL 8 / 9 / Fedora

sudo python3 src/imds_snoop.py

SLES 15

sudo python3 src/imds_snoop.py

CentOS 9

sudo python src/imds_snoop.py

WINDOWS

The output table will highlight if the instance has made IMDSv1 calls

To find the specific app making the IMDSv1 calls, use the inbuilt Windows Resource Monitor Network monitor to find the Image and PID of the application making calls.

To do this, open Resource Monitor (Start->Search ->Resource Monitor) and click on the Network tab. Then look for calls in the Network Activity section made to either the IP or DNS entries listed:

Network Analyzer will show the calls and you should proceed to update the software/application.

More details and thanks to https://github.com/salesforce/metabadger and https://www.greystone.co.uk/2022/03/24/how-greystone-upgraded-its-aws-ec2-instances-to-use-instance-meta-data-service-version-2-imdsv2/

Logging

The ImdsPacketAnalyzer will also capture IMDS calls to log files. Log entries follow the format: [Time] [Level] [message] where:

Example of a IMDSv1 log entry:

[2022-12-20T11:03:58] [WARNING] IMDSv1(!) (pid:1016:curl argv:curl http://169.254.169.254/latest/meta-data/) called by -> (pid:1015:makeCalls.sh argv:/bin/bash ./makeCalls.sh) -> (pid:32678:zsh argv:-zsh) -> (pid:32678:sshd argv:sshd: )

Example of a IMDSv2 log entry:

[2022-12-20T11:03:58] [INFO] IMDSv2 (pid:1018:curl argv:curl -H X-aws-ec2-metadata-token: AQAEAFEOMInKb-S7me-hLqzu83lYdeDV7r-sPh2D4SJF6v5IwD4S8g== -v http://169.254.169.254/latest/meta-data/) called by -> (pid:1015:makeCalls.sh argv:/bin/bash ./makeCalls.sh) -> (pid:32678:zsh argv:-zsh) -> (pid:32678:sshd argv:sshd: )

Log configuration

The logging configuration can be adjusted by editing the logging.conf file.

By default:

Analyzing log files

Assuming default logging setup:

Running the tool as a service

Activating the tool as a service

Configuring the analyzer to run as a service will ensure that the tool will run as soon as possible upon the boot up of an instance. This will increase the chances of identifying services making IMDSv1 calls as early as the instance is inited onto a network.

A shell script has been provided in the package that will automate the process of setting up the analyzer tool as a service. Note: the script/service will only work if the structure of the package is left unchanged.

Run the script from the command line as follows:

sudo ./activate-tracer-service.sh

or

sudo bash activate-tracer-service.sh

The permissions for the shell script may need to be changed using:

chmod +x activate-tracer-service.sh

You can check if the service is running after activating the service or a host reboot:

systemctl status -l imds_tracer_tool.service

Deactivating the tool as a service

When the tool is configured as a service using the previous script, a service file is added into the OS. In order to restore the system, run the script from the command line:

sudo ./deactivate-tracer-service.sh

or

sudo bash deactivate-tracer-service.sh

Permissions for the script may need to be changed:

chmod +x deactivate-tracer-service.sh

Limitations

We are aware of some limitations with the current version of the ImdsPacketAnalyzer. Contributions are welcomed.