DTrace/One-Liners - FreeBSD Wiki (original) (raw)

The following library of DTrace one-liners were last tested on FreeBSD 10.0. In FreeBSD 12.0, dwatch was added to the base.

Version

Show dtrace version

dtrace -V

Show dwatch version

dwatch -V

Listing Probes

List probes and search for string "foo":

dtrace -l | grep foo

Summarize probes by providers:

dtrace -l | awk '{ print $2 }' | sort | uniq -c | sort -n

List probes for a particular provider:

dtrace -l -P syscall

Using dwatch:

List probes and search for string "foo":

dwatch -lr foo

List unique providers:

dwatch -lP

List probes for a particular provider:

dwatch -lP syscall

List unique functions:

dwatch -lf

List probes for functions ending in read:

dwatch -lf '*read'

List probe-tuples ending in read using regex:

dwatch -lr 'read$'

Watch probe traversal for a given command:

dwatch -F 'pid$target:::' -c true

Syscalls

Trace file opens with process and filename:

dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'

Count system calls by program name:

dtrace -n 'syscall:::entry { @[execname] = count(); }'

Count system calls by syscall:

dtrace -n 'syscall:::entry { @[probefunc] = count(); }'

Count system calls by syscall, for PID 123 only:

dtrace -n 'syscall:::entry /pid == 123/ { @[probefunc] = count(); }'

Count system calls by syscall, for all processes with a specific program name ("nginx"):

dtrace -n 'syscall:::entry /execname == "nginx"/ { @[probefunc] = count(); }'

Count system calls by PID and program name:

dtrace -n 'syscall:::entry { @[pid, execname] = count(); }'

Find the parent of a process calling a syscall

dtrace -n 'syscall::read:entry { printf("execname=%s parent_execname=%s", execname, curthread->td_proc->p_pptr->p_comm); }'

Summarize requested read() sizes by program name, as power-of-2 distributions (bytes):

dtrace -n 'syscall::read:entry { @[execname] = quantize(arg2); }'

Summarize returned read() sizes by program name, as power-of-2 distributions (bytes or error):

dtrace -n 'syscall::read:return { @[execname] = quantize(arg1); }'

Summarize read() latency as a power-of-2 distribution by program name (ns):

dtrace -n 'syscall::read:entry { self->ts = timestamp; } syscall::read:return /self->ts/ { @[execname, "ns"] = quantize(timestamp - self->ts); self->ts = 0; }'

Summarize read() latency as a linear distribution (0 to 1000, step 5) by program name (ms):

dtrace -n 'syscall::read:entry { self->ts = timestamp; } syscall::read:return /self->ts/ { @[execname, "ms"] = lquantize((timestamp - self->ts) / 1000000, 0, 1000, 5); self->ts = 0; }'

Summarize read() on-CPU duration as a power-of-2 distribution by program name (ns):

dtrace -n 'syscall::read:entry { self->ts = vtimestamp; } syscall::read:return /self->ts/ { @[execname, "ns"] = quantize(vtimestamp - self->ts); self->ts = 0; }'

Count read() variants that "nginx" is using (if previous one-liners didn't work):

dtrace -n 'syscall::read:entry /execname == "nginx"/ { @[probefunc] = count(); }'

Summarize returned pread() sizes for "nginx" as distributions (bytes or error):

dtrace -n 'syscall::pread:return /execname == "nginx"/ { @ = quantize(arg1); }'

Count socket accept() variants by process name:

dtrace -n 'syscall::accept:return { @[execname] = count(); }'

Count socket connect() variants by process name:

dtrace -n 'syscall::connect:return { @[execname] = count(); }'

Summarize returned pread() sizes for "nginx"... and label the output:

dtrace -n 'syscall::pread:return /execname == "nginx"/ { @["rval (bytes)"] = quantize(arg1); }'

Using dwatch:

Trace file opens with process and filename:

dwatch -X open

Find the parent of a process calling a syscall

dwatch -R syscall::read:entry

Watch syscall probe traversal:

dwatch -F syscall

Watch syscall traversal by ruby processes:

dwatch -k 'ruby*' -F syscall

Watch syscall traversal by processes containing `daemon' in their name:

dwatch -k 'daemon' -F syscall

Watch open and openat syscalls until pid 1234 exits:

dwatch -X open -- -p 1234

Process Tracing

Trace new processes showing program name (and args if available):

dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

Count process-level events:

dtrace -n 'proc::: { @[probename] = count(); }'

Find the kernel function returning EINVAL for a given process

dtrace -n 'fbt:::return /(int)arg1 == EINVAL && pid == $target/ { stack(); }' -c ./a.out

Note that $pid is also available, but refers to the dtrace(1) process

itself rather than the command being run underneath it.

Using dwatch:

Trace new processes showing program name (and args if available):

dwatch exec-success

Trace new processes showing parent program and command executed:

dwatch execve

Trace new processes showing mini process-tree for command executed:

dwatch -R execve

Trace new processes belonging to root super-user:

dwatch -u root execve

Trace new processes belonging to wheel super-group:

dwatch -g wheel execve

Trace new processes belonging either to user daemon or nobody:

dwatch -u '1|65534' execve

Trace new processes belonging either to group daemon or nobody:

dwatch -g '1|65534' execve

Trace new processes running inside jail named myjail:

dwatch -j myjail execve

Trace new processes running unjailed (jailed processes ignored):

dwatch -j 0 execve

Trace new processes forked by pid 1234:

dwatch -p 1234 execve

Trace new processes forked by either pid 1234 or a child of pid 1234:

dwatch -R -p 1234 execve

Trace new processes forked by either pid 1234 or 5678:

dwatch -p '1234|5678' execve

Trace new processes matching either mkdir or rmdir using regex:

dwatch -z '(mk|rm)dir' execve

Signals

Using dwatch:

Watch signals being passed to kill syscall:

dwatch -X kill

Watch signals being passed between bash(1) and vi(1):

dwatch -k bash -k vi -X kill

Display the first process to call kill syscall and then exit:

dwatch -N 1 kill

Profiling

Count sampled thread names on-CPU at 997 Hertz:

dtrace -n 'profile-997 { @[stringof(curthread->td_name)] = count(); }'

Count sampled non-idle thread names on-CPU at 997 Hertz:

dtrace -n 'profile-997 /!(curthread->td_flags & 0x20)/ { @[stringof(curthread->td_name)] = count(); }'

Count sampled on-CPU kernel stacks at 99 Hertz:

dtrace -n 'profile-99 /arg0/ { @[stack()] = count(); }'

Count sampled process names and on-CP user stacks at 99 Hertz:

dtrace -n 'profile-99 /arg1/ { @[execname, ustack()] = count(); }'

Using dwatch:

Trace statfs for 5 minutes and then exit:

dwatch -T 5m statfs

Storage I/O

Count kernel stacks leading to block device I/O:

dtrace -n 'io:::start { @[stack()] = count(); }'

Using dwatch:

Trace kernel stacks leading to block device I/O:

dwatch -X io-start

Scheduler

Count kernel stacks leading to a context-switch off-CPU:

dtrace -n 'sched:::off-cpu { @[stack()] = count(); }'

Using dwatch:

Watch processes entering system CPU scheduler:

dwatch on-cpu

View the first 100 scheduler preemptions with less:

dwatch -y -N 100 preempt | less -R

IP

Count IP-level events:

dtrace -n 'ip::: { @[probename] = count(); }'

Using dwatch:

Trace IP-level events:

dwatch -X ip

UDP

Count UDP-level events:

dtrace -n 'udp::: { @[probename] = count(); }'

Using dwatch:

Trace UDP sent messages by remote IP address:

dwatch -X udp-send

Trace UDP received messages by remote IP address:

dwatch -X udp-receive

TCP

Count TCP-level events:

dtrace -n 'tcp::: { @[probename] = count(); }'

Trace TCP accepted connections by remote IP address:

dtrace -n 'tcp:::accept-established { trace(args[3]->tcps_raddr); }'

Count TCP passive opens by remote IP address:

dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr] = count(); }'

Count TCP active opens by remote IP address:

dtrace -n 'tcp:::connect-established { @[args[3]->tcps_raddr] = count(); }'

Count TCP sent messages by remote IP address:

dtrace -n 'tcp:::send { @[args[2]->ip_daddr] = count(); }'

Count TCP received messages by remote IP address:

dtrace -n 'tcp:::receive { @[args[2]->ip_saddr] = count(); }'

Summarize TCP sent messages by IP payload size, as a power-of-2 distribution:

dtrace -n 'tcp:::send { @[args[2]->ip_daddr] = quantize(args[2]->ip_plength); }'

Using dwatch:

Trace TCP accepted connections by remote IP address:

dwatch -X tcp-accept-established

Trace TCP active opens by remote IP address:

dwatch -X tcp-connect-established

Trace TCP sent messages by remote IP address:

dwatch -X tcp-send

Trace TCP received messages by remote IP address:

dwatch -X tcp-receive

Trace TCP activity while given nc command runs:

dwatch -X tcp -- -c "nc -zvw10 google.com 22"

Kernel Locks

Sum kernel adaptive lock block time by process name (ns):

dtrace -n 'lockstat:::adaptive-block { @[execname] = sum(arg1); }'

Summarize adaptive lock block time distribution by process name (ns):

dtrace -n 'lockstat:::adaptive-block { @[execname] = quantize(arg1); }'

Sum kernel adaptive lock block time by kernel stack trace (ns):

dtrace -n 'lockstat:::adaptive-block { @[stack()] = sum(arg1); }'

Sum kernel adaptive lock block time by lock name (ns):

dtrace -n 'lockstat:::adaptive-block { @[arg0] = sum(arg1); } END { printa("%40a %@16d ns\n", @); }'

Sum kernel adaptive lock block time by calling function (ns):

dtrace -n 'lockstat:::adaptive-block { @[caller] = sum(arg1); } END { printa("%40a %@16d ns\n", @); }'

Namecache

Count namecache lookups by program

dtrace -n 'vfs:namecache:lookup: { @missing[execname] = count(); }'

Count namecache misses by program

dtrace -n 'vfs:namecache:lookup:miss { @missing[execname] = count(); }'

VFS

Using dwatch:

Trace VFS lookup events by path:

dwatch -X vop_lookup

Trace VFS lookup (VOP_LOOKUP(9)) paths containing /lib/:

dwatch -X vop_lookup -r /lib/

Trace VFS create events by path:

dwatch -X vop_create

Trace VFS remove events by path:

dwatch -X vop_remove

Trace VFS symlink events by link and path:

dwatch -X vop_symlink

Trace VFS readdir events by path:

dwatch -X vop_readdir

Trace VFS mkdir events by path:

dwatch -X vop_mkdir

Trace VFS rmdir events by path:

dwatch -X vop_rmdir

Trace VFS rename events by path and destination:

dwatch -X vop_rename

Trace VFS mknod events by path:

dwatch -X vop_mknod

Raw Kernel Tracing

Count kernel slab memory allocation by function:

dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }'

Count kernel slab memory allocation by calling function:

dtrace -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'

Count kernel malloc() by calling function:

dtrace -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'

Count kernel malloc() by kernel stack trace:

dtrace -n 'fbt::malloc:entry { @[stack()] = count(); }'

Summarize vmem_alloc()s by arena name and size distribution:

dtrace -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }'

Summarize TCP life span in seconds:

dtrace -n 'fbt::tcp_close:entry { @["TCP life span (seconds):"] = quantize((uint32_t)(ticks - args[0]->t_starttime) / hz); }'

Using dwatch:

Trace processes performing writes with small buffers:

dwatch -t 'arg2<10' -E 'printf("%d",arg2)' write

Similar to above but ignore dtrace executable:

dwatch -X write -t 'execname != "dtrace" && this->nbytes < 10'

CategoryDtrace CategoryHowTo