Chapter 9. System tips (original) (raw)

Here, I describe basic tips to configure and manage systems, mostly from the console.

There are some utility programs to help your console activities.

Table 9.1. List of programs to support console activities

package popcon size description

mc

V:50, I:209

1542

See Section 1.3, “Midnight Commander (MC)”

bsdutils

V:519, I:999

356

script(1) command to make a record of terminal session

screen

V:71, I:230

1003

terminal multiplexer with VT100/ANSI terminal emulation

tmux

V:43, I:146

1180

terminal multiplexer alternative (Use "Control-B" instead)

fzf

V:4, I:16

3648

fuzzy text finder

fzy

V:0, I:0

54

fuzzy text finder

rlwrap

V:1, I:15

330

readline feature command line wrapper

ledit

V:0, I:11

331

readline feature command line wrapper

rlfe

V:0, I:0

45

readline feature command line wrapper

ripgrep

V:5, I:19

5152

fast recursive string search in the source code tree with automatic filtering

The simple use of script(1) (see Section 1.4.9, “Recording the shell activities”) to record shell activity produces a file with control characters. This can be avoided by using col(1) as the following.

$ script Script started, file is typescript

Do whatever … and press Ctrl-D to exit script.

$ col -bx < typescript > cleanedfile $ vim cleanedfile

There are alternative methods to record the shell activities:

In Section 1.4.2, “Customizing bash”, 2 tips to allow quick navigation around directories are described: $CDPATH and mc.

If you use fuzzy text filter program, you can do without typing the exact path. For fzf, include following in ~/.bashrc.

FZF_KEYBINDINGS_PATH=/usr/share/doc/fzf/examples/key-bindings.bash if [ -f $FZF_KEYBINDINGS_PATH ]; then . $FZF_KEYBINDINGS_PATH fi

For example:

After you learn basics of vim(1) through Section 1.4.8, “Using vim”, please read Bram Moolenaar's "Seven habits of effective text editing (2000)" to understand how vim should be used.

The behavior of vim can be changed significantly by enabling its internal features through the Ex-mode commands such as "set ..." to set vim options.

These Ex-mode commands can be included in user's vimrc file, traditional "~/.vimrc" or git-friendly "~/.vim/vimrc". Here is a very simple example[2]:

""" Generic baseline Vim and Neovim configuration (~/.vimrc) """ - For NeoVim, use "nvim -u ~/.vimrc [filename]" """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" let mapleader = ' ' " :h mapleader """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" set nocompatible " :h 'cp -- sensible (n)vim mode syntax on " :h :syn-on filetype plugin indent on " :h :filetype-overview set encoding=utf-8 " :h 'enc (default: latin1) -- sensible encoding """ current vim option value can be verified by :set encoding? set backspace=indent,eol,start " :h 'bs (default: nobs) -- sensible BS set statusline=%<%f%m%r%h%w%=%y[U+%04B]%2l/%2L=%P,%2c%V set listchars=eol:¶,tab:⇄\ ,extends:↦,precedes:↤,nbsp:␣ set viminfo=!,'100,<5000,s100,h " :h 'vi -- bigger copy buffer etc. """ Pick "colorscheme" from blue darkblue default delek desert elflord evening """ habamax industry koehler lunaperche morning murphy pablo peachpuff quiet ron """ shine slate torte zellner colorscheme industry """ don't pick "colorscheme" as "default" which may kill SpellUnderline settings set scrolloff=5 " :h 'scr -- show 5 lines around cursor set laststatus=2 " :h 'ls (default 1) k """ boolean options can be unset by prefixing "no" set ignorecase " :h 'ic set smartcase " :h 'scs set autoindent " :h 'ai set smartindent " :h 'si set nowrap " :h 'wrap "set list " :h 'list (default nolist) set noerrorbells " :h 'eb set novisualbell " :h 'vb set t_vb= " :h 't_vb -- termcap visual bell set spell " :h 'spell set spelllang=en_us,cjk " :h 'spl -- english spell, ignore CJK set clipboard=unnamedplus " :h 'cb -- cut/copy/paste with other app set hidden " :h 'hid set autowrite " :h 'aw set timeoutlen=300 " :h 'tm

The keymap of vim can be changed in user's vimrc file. E.g.:

[Caution] Caution
Don't try to change the default key bindings without very good reasons.

""" Popular mappings (imitating LazyVim etc.) """ Window moves without using CTRL-W which is dangerous in INSERT mode nnoremap h nnoremap j nnoremap k silent! nnoremap l """ Window resize nnoremap vertical resize -2 nnoremap resize -2 nnoremap resize +2 nnoremap vertical resize +2 """ Clear hlsearch with ( is mapped as above) nnoremap noh inoremap noh """ center after jump next nnoremap n nzz nnoremap N Nzz """ fast "jk" to get out of INSERT mode () inoremap jk noh """ fast "" to get out of TERM mode (CTRL-\ CTRL-N) tnoremap <C-> """ fast "jk" to get out of TERM mode (CTRL-\ CTRL-N) tnoremap jk <C-> """ previous/next trouble/quickfix item nnoremap [q cprevious nnoremap ]q cnext """ buffers nnoremap bprevious nnoremap bnext nnoremap [b bprevious nnoremap ]b bnext """ Add undo break-points inoremap , ,u inoremap . .u inoremap ; ;u """ save file inoremap w xnoremap w nnoremap w snoremap w """ better indenting vnoremap < >gv """ terminal (Somehow under Linux, becomes in Vim) nnoremap terminal "nnoremap terminal """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" if ! has('nvim') """ Toggle paste mode with p for Vim (no need for Nvim) set pastetoggle=p """ nvim default mappings for Vim. See :h default-mappings in nvim """ copy to EOL (no delete) like D for d noremap Y y$ """ sets a new undo point before deleting inoremap u inoremap u """ is re-purposed as above """ execute the previous macro recorded with Q nnoremap Q @@ """ repeat last substitute and KEEP flags nnoremap & :&& """ search visual selected string for visual mode xnoremap * y/\V" xnoremap # y?\V" endif

In order for the above keybindings to function properly, the terminal program needs to be configured to generate "ASCII DEL" for Backspace-key and "Escape sequence" for Delete-key.

Other miscellaneous configuration can be changed in user's vimrc file. E.g.:

""" Use faster 'rg' (ripgrep package) for :grep if executable("rg") set grepprg=rg\ --vimgrep\ --smart-case set grepformat=%f:%l:%c:%m endif """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" """ Retain last cursor position :h '" augroup RetainLastCursorPosition autocmd! autocmd BufReadPost * \ if line("'"") > 0 && line ("'"") <= line("$") | \ exe "normal! g'"" | \ endif augroup END """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" """ Force to use underline for spell check results augroup SpellUnderline autocmd! autocmd ColorScheme * highlight SpellBad term=Underline gui=Undercurl autocmd ColorScheme * highlight SpellCap term=Underline gui=Undercurl autocmd ColorScheme * highlight SpellLocal term=Underline gui=Undercurl autocmd ColorScheme * highlight SpellRare term=Underline gui=Undercurl augroup END """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" """ highlight tailing spaces except when typing as red (set after colorscheme) highlight TailingWhitespaces ctermbg=red guibg=red """ \s+ 1 or more whitespace character: and """ %#@<! Matches with zero width if the cursor position does NOT match. match TailingWhitespaces /\s+%#@<!$/

Interesting external plugin packages can be found:

Plugin packages in the vim-scripts package can be enabled using user's vimrc file. E.g.:

packadd! secure-modelines packadd! winmanager " IDE-like UI for files and buffers with w nnoremap w :WMToggle

The new native Vim package system works nicely with "git" and "git submodule". One such example configuration can be found at my git repository: dot-vim. This does essentially:

For more, please start vim with "vim --startuptime vimstart.log" to check actual execution sequence and time spent for each step.

It is quite confusing to see too many ways[3] to manage and load these external packages to vim. Checking the original information is the best cure.

Here are notable log analyzers ("~Gsecurity::log-analyzer" in aptitude(8)).

Table 9.4. List of system log analyzers

package popcon size description

logwatch

V:11, I:13

2328

log analyzer with nice output written in Perl

fail2ban

V:98, I:111

2126

ban IPs that cause multiple authentication errors

analog

V:3, I:96

3739

web server log analyzer

awstats

V:6, I:10

6928

powerful and featureful web server log analyzer

sarg

V:1, I:1

845

squid analysis report generator

pflogsumm

V:1, I:4

109

Postfix log entry summarizer

fwlogwatch

V:0, I:0

481

firewall log analyzer

squidview

V:0, I:0

189

monitor and analyze squid access.log files

swatch

V:0, I:0

99

log file viewer with regexp matching, highlighting, and hooks

crm114

V:0, I:0

1119

Controllable Regex Mutilator and Spam Filter (CRM114)

icmpinfo

V:0, I:0

44

interpret ICMP messages

[Note] Note

CRM114 provides language infrastructure to write fuzzy filters with the TRE regex library. Its popular use is spam mail filter but it can be used as log analyzer.

You can record the editor activities for complex repeats.

For Vim, as follows.

For Emacs, as follows.

Program activities can be monitored and controlled using specialized tools.

Table 9.8. List of tools for monitoring and controlling program activities

package popcon size description

coreutils

V:880, I:999

18307

nice(1): run a program with modified scheduling priority

bsdutils

V:519, I:999

356

renice(1): modify the scheduling priority of a running process

procps

V:766, I:999

2389

"/proc" filesystem utilities: ps(1), top(1), kill(1), watch(1), …

psmisc

V:420, I:775

908

"/proc" filesystem utilities: killall(1), fuser(1), peekfd(1), pstree(1)

time

V:7, I:132

129

time(1): run a program to report system resource usages with respect to time

sysstat

V:148, I:170

1904

sar(1), iostat(1), mpstat(1), …: system performance tools for Linux

isag

V:0, I:3

109

Interactive System Activity Grapher for sysstat

lsof

V:422, I:945

482

lsof(8): list files opened by a running process using "-p" option

strace

V:12, I:119

2897

strace(1): trace system calls and signals

ltrace

V:0, I:16

330

ltrace(1): trace library calls

xtrace

V:0, I:0

353

xtrace(1): trace communication between X11 client and server

powertop

V:18, I:217

677

powertop(1): information about system power use

cron

V:872, I:995

244

run processes according to a schedule in background from cron(8) daemon

anacron

V:396, I:479

93

cron-like command scheduler for systems that don't run 24 hours a day

at

V:101, I:154

158

at(1) or batch(1): run a job at a specified time or below certain load level

[Tip] Tip
The procps packages provide very basics of monitoring, controlling, and starting program activities. You should learn all of them.

For the command-line interface (CLI), the first program with the matching name found in the directories specified in the $PATH environment variable is executed. See [Section 1.5.3, “The "$PATH" variable”](ch01.en.html#%5Fthe%5Fliteral%5Fpath%5Fliteral%5Fvariable "1.5.3. The "$PATH" variable").

For the graphical user interface (GUI) compliant to the freedesktop.org standards, the *.desktop files in the /usr/share/applications/ directory provide necessary attributes for the GUI menu display of each program. Each package which is compliant to Freedesktop.org's xdg menu system installs its menu data provided by "*.desktop" under "/usr/share/applications/". Modern desktop environments which are compliant to Freedesktop.org standard use these data to generate their menu using the xdg-utils package. See "/usr/share/doc/xdg-utils/README".

For example, the chromium.desktop file defines attributes for the "Chromium Web Browser" such as "Name" for the program name, "Exec" for the program execution path and arguments, "Icon" for the icon used, etc. (see the Desktop Entry Specification) as follows:

[Desktop Entry] Version=1.0 Name=Chromium Web Browser GenericName=Web Browser Comment=Access the Internet Comment[fr]=Explorer le Web Exec=/usr/bin/chromium %U Terminal=false X-MultipleArgs=false Type=Application Icon=chromium Categories=Network;WebBrowser; MimeType=text/html;text/xml;application/xhtml_xml;x-scheme-handler/http;x-scheme-handler/https; StartupWMClass=Chromium StartupNotify=true

This is an oversimplified description. The *.desktop files are scanned as follows.

The desktop environment sets $XDG_DATA_HOME and $XDG_DATA_DIR environment variables. For example, under the GNOME 3:

So the base directories (see XDG Base Directory Specification) and the applications directories are as follows.

The *.desktop files are scanned in these applications directories in this order.

[Tip] Tip
A user custom GUI menu entry can be created by adding a *.desktop file in the $HOME/.local/share/applications/ directory.
[Tip] Tip
The "Exec=..." line isn't parsed by the shell. Use the env(1) command if environment variables need to be set.
[Tip] Tip
Similarly, if a *.desktop file is created in the autostart directory under these base directories, the specified program in the *.desktop file is executed automatically when the desktop environment is started. See Desktop Application Autostart Specification.
[Tip] Tip
Similarly, if a *.desktop file is created in the HOME/DesktopdirectoryandtheDesktopenvironmentisconfiguredtosupportthedesktopiconlauncherfeature,thespecifiedprograminitisexecuteduponclickingtheicon.PleasenotethattheactualnameoftheHOME/Desktop directory and the Desktop environment is configured to support the desktop icon launcher feature, the specified program in it is executed upon clicking the icon. Please note that the actual name of the HOME/DesktopdirectoryandtheDesktopenvironmentisconfiguredtosupportthedesktopiconlauncherfeature,thespecifiedprograminitisexecuteduponclickingtheicon.PleasenotethattheactualnameoftheHOME/Desktop directory is locale dependent. See xdg-user-dirs-update(1).

Some programs start another program automatically. Here are check points for customizing this process.

[Tip] Tip
update-mime(8) updates the "/etc/mailcap" file using "/etc/mailcap.order" file (see mailcap.order(5)).
[Tip] Tip
The debianutils package provides sensible-browser(1), sensible-editor(1), and sensible-pager(1) which make sensible decisions on which editor, pager, and web browser to call, respectively. I recommend you to read these shell scripts.
[Tip] Tip
In order to run a console application such as mutt under GUI as your preferred application, you should create an GUI application as following and set "/usr/local/bin/mutt-term" as your preferred application to be started as described. # cat /usr/local/bin/mutt-term <<EOF #!/bin/sh gnome-terminal -e "mutt \$@" EOF # chmod 755 /usr/local/bin/mutt-term

Use cron(8) to schedule tasks regularly. See crontab(1) and crontab(5).

You can schedule to run processes as a normal user, e.g. foo by creating a crontab(5) file as "/var/spool/cron/crontabs/foo" with "crontab -e" command.

Here is an example of a crontab(5) file.

use /usr/bin/sh to run commands, no matter what /etc/passwd says

SHELL=/bin/sh

mail any output to paul, no matter whose crontab this is

MAILTO=paul

Min Hour DayOfMonth Month DayOfWeek command (Day... are OR'ed)

run at 00:05, every day

5 0 * * * HOME/bin/daily.job>>HOME/bin/daily.job >> HOME/bin/daily.job>>HOME/tmp/out 2>&1

run at 14:15 on the first of every month -- output mailed to paul

15 14 1 * * $HOME/bin/monthly

run at 22:00 on weekdays(1-5), annoy Joe. % for newline, last % for cc:

0 22 * * 1-5 mail -s "It's 10pm" joe%Joe,%%Where are your kids?%.%% 23 */2 1 2 * echo "run 23 minutes after 0am, 2am, 4am ..., on Feb 1" 5 4 * * sun echo "run at 04:05 every Sunday"

run at 03:40 on the first Monday of each month

40 3 1-7 * * [ "$(date +%a)" == "Mon" ] && command -args

[Tip] Tip
For the system not running continuously, install the anacron package to schedule periodic commands at the specified intervals as closely as machine-uptime permits. See anacron(8) and anacrontab(5).
[Tip] Tip
For scheduled system maintenance scripts, you can run them periodically from root account by placing such scripts in "/etc/cron.hourly/", "/etc/cron.daily/", "/etc/cron.weekly/", or "/etc/cron.monthly/". Execution timings of these scripts can be customized by "/etc/crontab" and "/etc/anacrontab".

Systemd has low level capability to schedule programs to run without cron daemon. For example, /lib/systemd/system/apt-daily.timer and /lib/systemd/system/apt-daily.service set up daily apt download activities. See systemd.timer(5) .

Although most of the hardware configuration on modern GUI desktop systems such as GNOME and KDE can be managed through accompanying GUI configuration tools, it is a good idea to know some basics methods to configure them.

Table 9.14. List of hardware configuration tools

package popcon size description

console-setup

V:88, I:967

428

Linux console font and keytable utilities

x11-xserver-utils

V:302, I:528

568

X server utilities: xset(1), xmodmap(1)

acpid

V:84, I:148

158

daemon to manage events delivered by the Advanced Configuration and Power Interface (ACPI)

acpi

V:9, I:136

47

utility to display information on ACPI devices

sleepd

V:0, I:0

86

daemon to put a laptop to sleep during inactivity

hdparm

V:178, I:335

256

hard disk access optimization (see Section 9.6.9, “Optimization of hard disk”)

smartmontools

V:207, I:250

2358

control and monitor storage systems using S.M.A.R.T.

setserial

V:4, I:6

103

collection of tools for serial port management

memtest86+

V:1, I:21

12711

collection of tools for memory hardware management

scsitools

V:0, I:2

346

collection of tools for SCSI hardware management

setcd

V:0, I:0

37

compact disc drive access optimization

big-cursor

I:0

26

larger mouse cursors for X

Here, ACPI is a newer framework for the power management system than APM.

[Tip] Tip
CPU frequency scaling on modern system is governed by kernel modules such as acpi_cpufreq.

Device drivers for sound cards for current Linux are provided by Advanced Linux Sound Architecture (ALSA). ALSA provides emulation mode for previous Open Sound System (OSS) for compatibility.

Application softwares may be configured not only to access sound devices directly but also to access them via some standardized sound server system. Currently, PulseAudio, JACK, and PipeWire are used as sound server system. See Debian wiki page on Sound for the latest situation.

There is usually a common sound engine for each popular desktop environment. Each sound engine used by the application can choose to connect to different sound servers.

[Tip] Tip
Use "cat /dev/urandom > /dev/audio" or speaker-test(1) to test speaker (^C to stop).
[Tip] Tip
If you can not get sound, your speaker may be connected to a muted output. Modern sound system has many outputs. alsamixer(1) in the alsa-utils package is useful to configure volume and mute settings.

Table 9.15. List of sound packages

package popcon size description

alsa-utils

V:330, I:466

2605

utilities for configuring and using ALSA

oss-compat

V:1, I:17

18

OSS compatibility under ALSA preventing "/dev/dsp not found" errors

pipewire

V:265, I:319

120

audio and video processing engine multimedia server - metapackage

pipewire-bin

V:274, I:319

1631

audio and video processing engine multimedia server - audio server and CLI programs

pipewire-alsa

V:105, I:157

206

audio and video processing engine multimedia server - audio server to replace ALSA

pipewire-pulse

V:160, I:214

50

audio and video processing engine multimedia server - audio server to replace PulseAudio

pulseaudio

V:256, I:308

6472

PulseAudio server

libpulse0

V:413, I:580

975

PulseAudio client library

jackd

V:2, I:18

9

JACK Audio Connection Kit. (JACK) server (low latency)

libjack0

V:1, I:9

326

JACK Audio Connection Kit. (JACK) library (low latency)

libgstreamer1.0-0

V:429, I:597

4455

GStreamer: GNOME sound engine

libphonon4qt5-4

V:72, I:162

594

Phonon: KDE sound engine

Poor system maintenance may expose your system to external exploitation.

For system security and integrity check, you should start with the following.

Table 9.18. List of tools for system security and integrity check

package popcon size description

logcheck

V:6, I:7

110

daemon to mail anomalies in the system logfiles to the administrator

debsums

V:5, I:35

98

utility to verify installed package files against MD5 checksums

chkrootkit

V:8, I:17

925

rootkit detector

clamav

V:9, I:45

27455

anti-virus utility for Unix - command-line interface

tiger

V:1, I:2

7800

report system security vulnerabilities

tripwire

V:1, I:2

5016

file and directory integrity checker

john

V:1, I:9

471

active password cracking tool

aide

V:1, I:1

293

Advanced Intrusion Detection Environment - static binary

integrit

V:0, I:0

2659

file integrity verification program

crack

V:0, I:1

149

password guessing program

Here is a simple script to check for typical world writable incorrect file permissions.

find / -perm 777 -a ! -type s -a ! -type l -a ! ( -type d -a -perm 1777 )

[Caution] Caution
Since the debsums package uses

MD5 checksums stored locally, it can not be fully trusted as the system security audit tool against malicious attacks.

Booting your system with Linux live CDs or debian-installer CDs in rescue mode makes it easy for you to reconfigure data storage on your boot device.

You may need to umount(8) some devices manually from the command line before operating on them if they are automatically mounted by the GUI desktop system.

For disk partition configuration, although fdisk(8) has been considered standard, parted(8) deserves some attention. "Disk partitioning data", "partition table", "partition map", and "disk label" are all synonyms.

Older PCs use the classic Master Boot Record (MBR) scheme to hold disk partitioning data in the first sector, i.e., LBA sector 0 (512 bytes).

Recent PCs with Unified Extensible Firmware Interface (UEFI), including Intel-based Macs, use GUID Partition Table (GPT) scheme to hold disk partitioning data not in the first sector.

Although fdisk(8) has been standard for the disk partitioning tool, parted(8) is replacing it.

[Caution] Caution
Although parted(8) claims to create and to resize filesystem too, it is safer to do such things using best maintained specialized tools such as mkfs(8) (mkfs.msdos(8), mkfs.ext2(8), mkfs.ext3(8), mkfs.ext4(8), …) and resize2fs(8).
[Note] Note
In order to switch between

GPT and MBR, you need to erase first few blocks of disk contents directly (see Section 9.8.6, “Clearing file contents”) and use "parted /dev/sdx mklabel gpt" or "parted /dev/sdx mklabel msdos" to set it. Please note "msdos" is use here for MBR.

LVM2 is a logical volume manager for the Linux kernel. With LVM2, disk partitions can be created on logical volumes instead of the physical harddisks.

LVM requires the following.

Please start learning LVM2 from the following manpages.

For ext4 filesystem, the e2fsprogs package provides the following.

The mkfs(8) and fsck(8) commands are provided by the e2fsprogs package as front-ends to various filesystem dependent programs (mkfs.fstype and fsck.fstype). For ext4 filesystem, they are mkfs.ext4(8) and fsck.ext4(8) (they are symlinked to mke2fs(8) and e2fsck(8)).

Similar commands are available for each filesystem supported by Linux.

Table 9.20. List of filesystem management packages

package popcon size description

e2fsprogs

V:767, I:999

1499

utilities for the ext2/ext3/ext4 filesystems

btrfs-progs

V:44, I:72

5078

utilities for the Btrfs filesystem

reiserfsprogs

V:12, I:25

473

utilities for the Reiserfs filesystem

zfsutils-linux

V:29, I:30

1762

utilities for the OpenZFS filesystem

dosfstools

V:196, I:541

315

utilities for the FAT filesystem. (Microsoft: MS-DOS, Windows)

exfatprogs

V:29, I:371

301

utilities for the exFAT filesystem maintained by Samsung.

exfat-fuse

V:5, I:120

73

read/write exFAT filesystem (Microsoft) driver for FUSE.

exfat-utils

V:4, I:106

231

utilities for the exFAT filesystem maintained by the exfat-fuse author.

xfsprogs

V:21, I:95

3476

utilities for the XFS filesystem. (SGI: IRIX)

ntfs-3g

V:197, I:513

1474

read/write NTFS filesystem (Microsoft: Windows NT, …) driver for FUSE.

jfsutils

V:0, I:8

1577

utilities for the JFS filesystem. (IBM: AIX, OS/2)

reiser4progs

V:0, I:2

1367

utilities for the Reiser4 filesystem

hfsprogs

V:0, I:4

394

utilities for HFS and HFS Plus filesystem. (Apple: Mac OS)

zerofree

V:5, I:131

25

program to zero free blocks from ext2/3/4 filesystems

Solid state drive (SSD) is auto detected now.

Reduce unnecessary disk accesses to prevent disk wear out by mounting "tmpfs" on volatile data path in /etc/fstab.

You can monitor and log your hard disk which is compliant to SMART with the smartd(8) daemon.

  1. Enable SMART feature in BIOS.
  2. Install the smartmontools package.
  3. Identify your hard disk drives by listing them with df(1).
    • Let's assume a hard disk drive to be monitored as "/dev/hda".
  4. Check the output of "smartctl -a /dev/hda" to see if SMART feature is actually enabled.
    • If not, enable it by "smartctl -s on -a /dev/hda".
  5. Enable smartd(8) daemon to run by the following.
    • uncomment "start_smartd=yes" in the "/etc/default/smartmontools" file.
    • restart the smartd(8) daemon by "sudo systemctl restart smartmontools".
[Tip] Tip
The smartd(8) daemon can be customized with the /etc/smartd.conf file including how to be notified of warnings.

Here, we discuss manipulations of the disk image.

The disk image "partition.img" containing a single partition image can be mounted and unmounted by using the loop device as follows.

losetup --show -f partition.img

/dev/loop0

mkdir -p /mnt/loop0

mount -t auto /dev/loop0 /mnt/loop0

...hack...hack...hack

umount /dev/loop0

losetup -d /dev/loop0

This can be simplified as follows.

mkdir -p /mnt/loop0

mount -t auto -o loop partition.img /mnt/loop0

...hack...hack...hack

umount partition.img

Each partition of the disk image "disk.img" containing multiple partitions can be mounted by using the loop device.

losetup --show -f -P disk.img

/dev/loop0

ls -l /dev/loop0*

brw-rw---- 1 root disk 7, 0 Apr 2 22:51 /dev/loop0 brw-rw---- 1 root disk 259, 12 Apr 2 22:51 /dev/loop0p1 brw-rw---- 1 root disk 259, 13 Apr 2 22:51 /dev/loop0p14 brw-rw---- 1 root disk 259, 14 Apr 2 22:51 /dev/loop0p15

fdisk -l /dev/loop0

Disk /dev/loop0: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 6A1D9E28-C48C-2144-91F7-968B3CBC9BD1

Device Start End Sectors Size Type /dev/loop0p1 262144 4192255 3930112 1.9G Linux root (x86-64) /dev/loop0p14 2048 8191 6144 3M BIOS boot /dev/loop0p15 8192 262143 253952 124M EFI System

Partition table entries are not in disk order.

mkdir -p /mnt/loop0p1

mkdir -p /mnt/loop0p15

mount -t auto /dev/loop0p1 /mnt/loop0p1

mount -t auto /dev/loop0p15 /mnt/loop0p15

mount |grep loop

/dev/loop0p1 on /mnt/loop0p1 type ext4 (rw,relatime) /dev/loop0p15 on /mnt/loop0p15 type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) ...hack...hack...hack

umount /dev/loop0p1

umount /dev/loop0p15

losetup -d /dev/loop0

Alternatively, similar effects can be done by using the device mapper devices created by kpartx(8) from the kpartx package as follows.

kpartx -a -v disk.img

add map loop0p1 (253:0): 0 3930112 linear 7:0 262144 add map loop0p14 (253:1): 0 6144 linear 7:0 2048 add map loop0p15 (253:2): 0 253952 linear 7:0 8192

fdisk -l /dev/loop0

Disk /dev/loop0: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 6A1D9E28-C48C-2144-91F7-968B3CBC9BD1

Device Start End Sectors Size Type /dev/loop0p1 262144 4192255 3930112 1.9G Linux root (x86-64) /dev/loop0p14 2048 8191 6144 3M BIOS boot /dev/loop0p15 8192 262143 253952 124M EFI System

Partition table entries are not in disk order.

ls -l /dev/mapper/

total 0 crw------- 1 root root 10, 236 Apr 2 22:45 control lrwxrwxrwx 1 root root 7 Apr 2 23:19 loop0p1 -> ../dm-0 lrwxrwxrwx 1 root root 7 Apr 2 23:19 loop0p14 -> ../dm-1 lrwxrwxrwx 1 root root 7 Apr 2 23:19 loop0p15 -> ../dm-2

mkdir -p /mnt/loop0p1

mkdir -p /mnt/loop0p15

mount -t auto /dev/mapper/loop0p1 /mnt/loop0p1

mount -t auto /dev/mapper/loop0p15 /mnt/loop0p15

mount |grep loop

/dev/loop0p1 on /mnt/loop0p1 type ext4 (rw,relatime) /dev/loop0p15 on /mnt/loop0p15 type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) ...hack...hack...hack

umount /dev/mapper/loop0p1

umount /dev/mapper/loop0p15

kpartx -d disk.img

The ISO9660 image file, "cd.iso", from the source directory tree at "source_directory" can be made using genisoimage(1) provided by cdrkit by the following.

genisoimage -r -J -T -V volume_id -o cd.iso source_directory

Similarly, the bootable ISO9660 image file, "cdboot.iso", can be made from debian-installer like directory tree at "source_directory" by the following.

genisoimage -r -o cdboot.iso -V volume_id \

-b isolinux/isolinux.bin -c isolinux/boot.cat
-no-emul-boot -boot-load-size 4 -boot-info-table source_directory

Here Isolinux boot loader (see Section 3.1.2, “Stage 2: the boot loader”) is used for booting.

You can calculate the md5sum value and make the ISO9660 image directly from the CD-ROM device as follows.

$ isoinfo -d -i /dev/cdrom CD-ROM is in ISO 9660 format ... Logical block size is: 2048 Volume size is: 23150592 ...

dd if=/dev/cdrom bs=2048 count=23150592 conv=notrunc,noerror | md5sum

dd if=/dev/cdrom bs=2048 count=23150592 conv=notrunc,noerror > cd.iso

[Warning] Warning
You must carefully avoid ISO9660 filesystem read ahead bug of Linux as above to get the right result.

Here, we discuss direct manipulations of the binary data on storage media.

The most basic viewing method of binary data is to use "od -t x1" command.

Table 9.21. List of packages which view and edit binary data

package popcon size description

coreutils

V:880, I:999

18307

basic package which has od(1) to dump files (HEX, ASCII, OCTAL, …)

bsdmainutils

V:11, I:315

17

utility package which has hd(1) to dump files (HEX, ASCII, OCTAL, …)

hexedit

V:0, I:9

73

binary editor and viewer (HEX, ASCII)

bless

V:0, I:2

924

full featured hexadecimal editor (GNOME)

okteta

V:1, I:12

1585

full featured hexadecimal editor (KDE4)

ncurses-hexedit

V:0, I:1

130

binary editor and viewer (HEX, ASCII, EBCDIC)

beav

V:0, I:0

137

binary editor and viewer (HEX, ASCII, EBCDIC, OCTAL, …)

Software RAID systems offered by the Linux kernel provide data redundancy in the kernel filesystem level to achieve high levels of storage reliability.

There are tools to add data redundancy to files in application program level to achieve high levels of storage reliability, too.

There are tools for data file recovery and forensic analysis.

Table 9.24. List of packages for data file recovery and forensic analysis

package popcon size description

testdisk

V:2, I:28

1413

utilities for partition scan and disk recovery

magicrescue

V:0, I:2

255

utility to recover files by looking for magic bytes

scalpel

V:0, I:3

89

frugal, high performance file carver

myrescue

V:0, I:2

83

rescue data from damaged harddisks

extundelete

V:0, I:8

147

utility to undelete files on the ext3/4 filesystem

ext4magic

V:0, I:4

233

utility to undelete files on the ext3/4 filesystem

ext3grep

V:0, I:2

293

tool to help recover deleted files on the ext3 filesystem

scrounge-ntfs

V:0, I:2

50

data recovery program for NTFS filesystems

gzrt

V:0, I:0

33

gzip recovery toolkit

sleuthkit

V:3, I:24

1671

tools for forensics analysis. (Sleuthkit)

autopsy

V:0, I:1

1026

graphical interface to SleuthKit

foremost

V:0, I:5

102

forensics application to recover data

guymager

V:0, I:0

1021

forensic imaging tool based on Qt

dcfldd

V:0, I:3

114

enhanced version of dd for forensics and security

[Tip] Tip
You can undelete files on the ext2 filesystem using list_deleted_inodes and undel commands of debugfs(8) in the e2fsprogs package.

With physical access to your PC, anyone can easily gain root privilege and access all the files on your PC (see Section 4.6.4, “Securing the root password”). This means that login password system can not secure your private and sensitive data against possible theft of your PC. You must deploy data encryption technology to do it. Although GNU privacy guard (see Section 10.3, “Data security infrastructure”) can encrypt files, it takes some user efforts.

Dm-crypt facilitates automatic data encryption via native Linux kernel modules with minimal user efforts using device-mapper.

[Caution] Caution
Data encryption costs CPU time etc. Encrypted data becomes inaccessible if its password is lost. Please weigh its benefits and costs.

You can encrypt contents of removable mass devices, e.g. USB memory stick on "/dev/sdx", using dm-crypt/LUKS. You simply format it as the following.

fdisk /dev/sdx

... "n" "p" "1" "return" "return" "w"

cryptsetup luksFormat /dev/sdx1

...

cryptsetup open /dev/sdx1 secret

...

ls -l /dev/mapper/

total 0 crw-rw---- 1 root root 10, 60 2021-10-04 18:44 control lrwxrwxrwx 1 root root 7 2021-10-04 23:55 secret -> ../dm-0

mkfs.vfat /dev/mapper/secret

...

cryptsetup close secret

Then, it can be mounted just like normal one on to "/media/_username/disklabel_", except for asking password (see Section 10.1.7, “Removable storage device”) under modern desktop environment using the udisks2 package. The difference is that every data written to it is encrypted. The password entry may be automated using keyring (see Section 10.3.6, “Password keyring”).

You may alternatively format media in different filesystem, e.g., ext4 with "mkfs.ext4 /dev/mapper/sdx1". If btrfs is used instead, the udisks2-btrfs package needs to be installed. For these filesystems, the file ownership and permissions may need to be configured.

Debian distributes modularized Linux kernel as packages for supported architectures.

If you are reading this documentation, you probably don't need to compile Linux kernel by yourself.

Debian has its own method of compiling the kernel and related modules.

If you use initrd in Section 3.1.2, “Stage 2: the boot loader”, make sure to read the related information in initramfs-tools(8), update-initramfs(8), mkinitramfs(8) and initramfs.conf(5).

[Warning] Warning
Do not put symlinks to the directories in the source tree (e.g. "/usr/src/linux*") from "/usr/include/linux" and "/usr/include/asm" when compiling the Linux kernel source. (Some outdated documents suggest this.)
[Note] Note
When compiling the latest Linux kernel on the Debian stable system, the use of backported latest tools from the Debian unstable may be needed. module-assistant(8) (or its short form m-a) helps users to build and install module package(s) easily for one or more custom kernels. The dynamic kernel module support (DKMS) is a new distribution independent framework designed to allow individual kernel modules to be upgraded without changing the whole kernel. This is used for the maintenance of out-of-tree modules. This also makes it very easy to rebuild modules as you upgrade kernels.

The hardware driver is the code running on the main CPUs of the target system. Most hardware drivers are available as free software now and are included in the normal Debian kernel packages in the main area.

The firmware is the code or data loaded on the device attach to the target system (e.g., CPU microcode, rendering code running on GPU, or FPGA / CPLD data, …). Some firmware packages are available as free software but many firmware packages are not available as free software since they contain sourceless binary data. Installing these firmware data is essential for the device to function as expected.

Please note that access to non-free-firmware packages are provided by the official installation media to offer functional installation experience to the user since Debian 12 Bookworm. The non-free-firmware area is described in Section 2.1.5, “Debian archive basics”.

Please also note that the firmware data downloaded by fwupd from Linux Vendor Firmware Service and loaded to the running Linux kernel may be non-free.

Use of virtualized system enables us to run multiple instances of system simultaneously on a single hardware.

There are several virtualization and emulation tool platforms.

The container virtualization uses Section 4.7.5, “Linux security features” and is the backend technology of Section 7.7, “Sandbox”.

Here are some packages to help you to setup the virtualized system.

Table 9.27. List of virtualization tools

package popcon size description

coreutils

V:880, I:999

18307

GNU core utilities which contain chroot(8)

systemd-container

V:53, I:61

1330

systemd container/nspawn tools which contain systemd-nspawn(1)

schroot

V:5, I:7

2579

specialized tool for executing Debian binary packages in chroot

sbuild

V:1, I:3

243

tool for building Debian binary packages from Debian sources

debootstrap

V:5, I:54

314

bootstrap a basic Debian system (written in sh)

cdebootstrap

V:0, I:1

115

bootstrap a Debian system (written in C)

cloud-image-utils

V:1, I:17

66

cloud image management utilities

cloud-guest-utils

V:3, I:13

71

cloud guest utilities

virt-manager

V:11, I:44

2296

Virtual Machine Manager: desktop application for managing virtual machines

libvirt-clients

V:46, I:65

1241

programs for the libvirt library

incus

V:0, I:0

56209

Incus: system container and virtual machine manager (for Debian 13 "Trixie")

lxd

V:0, I:0

52119

LXD: system container and virtual machine manager (for Debian 12 "Bookworm")

podman

V:14, I:16

41948

podman: engine to run OCI-based containers in Pods

podman-docker

V:0, I:0

249

engine to run OCI-based containers in Pods - wrapper for docker

docker.io

V:41, I:43

150003

docker: Linux container runtime

games-emulator

I:0

21

games-emulator: Debian's emulators for games

bochs

V:0, I:0

6956

Bochs: IA-32 PC emulator

qemu

I:14

97

QEMU: fast generic processor emulator

qemu-system

I:22

66

QEMU: full system emulation binaries

qemu-user

V:1, I:6

93760

QEMU: user mode emulation binaries

qemu-utils

V:12, I:106

10635

QEMU: utilities

qemu-system-x86

V:33, I:91

58140

KVM: full virtualization on x86 hardware with the hardware-assisted virtualization

virtualbox

V:6, I:8

130868

VirtualBox: x86 virtualization solution on i386 and amd64

gnome-boxes

V:1, I:7

6691

Boxes: Simple GNOME app to access virtual systems

xen-tools

V:0, I:2

719

tools to manage debian XEN virtual server

wine

V:13, I:60

132

Wine: Windows API Implementation (standard suite)

dosbox

V:1, I:15

2696

DOSBox: x86 emulator with Tandy/Herc/CGA/EGA/VGA/SVGA graphics, sound and DOS

lxc

V:9, I:12

25890

Linux containers user space tools

python3-venv

I:88

6

venv for creating virtual python environments (system library)

python3-virtualenv

V:9, I:50

356

virtualenv for creating isolated virtual python environments

pipx

V:3, I:19

3324

pipx for installing python applications in isolated environments

See Wikipedia article Comparison of platform virtual machines for detail comparison of different platform virtualization solutions.

[Note] Note
Default Debian kernels support

KVM since lenny.

Typical work flow for virtualization involves several steps.

For the raw disk image file, see Section 9.7, “The disk image”.

For other virtual disk image files, you can use qemu-nbd(8) to export them using network block device protocol and mount them using the nbd kernel module.

qemu-nbd(8) supports disk formats supported by QEMU: raw, qcow2, qcow, vmdk, vdi, bochs, cow (user-mode Linux copy-on-write), parallels, dmg, cloop, vpc, vvfat (virtual VFAT), and host_device.

The network block device can support partitions in the same way as the loop device (see Section 9.7.3, “Mounting the disk image file”). You can mount the first partition of "disk.img" as follows.

modprobe nbd max_part=16

qemu-nbd -v -c /dev/nbd0 disk.img

...

mkdir /mnt/part1

mount /dev/nbd0p1 /mnt/part1

[Tip] Tip
You may export only the first partition of "disk.img" using "-P 1" option to qemu-nbd(8).

If you wish to try a new Debian environment from a terminal console, I recommend you to use chroot. This enables you to run console applications of Debian unstable and testing without usual risks associated and without rebooting. chroot(8) is the most basic way.

[Caution] Caution
Examples below assumes both parent system and chroot system share the same amd64 CPU architecture.

Although you can manually create a chroot(8) environment using debootstrap(1), this requires non-trivial efforts.

The sbuild package to build Debian packages from source uses the chroot environment managed by the schroot package. It comes with helper script sbuild-createchroot(1). Let's learn how it works by running it as follows.

$ sudo mkdir -p /srv/chroot $ sudo sbuild-createchroot -v --include=eatmydata,ccache unstable /srv/chroot/unstable-amd64-sbuild http://deb.debian.org/debian ...

You see how debootstrap(8) populates system data for unstable environment under "/srv/chroot/unstable-amd64-sbuild" for a minimal build system.

You can login to this environment using schroot(1).

$ sudo schroot -v -c chroot:unstable-amd64-sbuild

You see how a system shell running under unstable environment is created.

[Note] Note
The "/usr/sbin/policy-rc.d" file which always exits with 101 prevents daemon programs to be started automatically on the Debian system. See "/usr/share/doc/init-system-helpers/README.policy-rc.d.gz".
[Note] Note
Some programs under chroot may require access to more files from the parent system to function than sbuild-createchroot provides as above. For example, "/sys", "/etc/passwd", "/etc/group", "/var/run/utmp", "/var/log/wtmp", etc. may need to be bind-mounted or copied.
[Tip] Tip
The systemd-nspawn(1) command helps to run a command or OS in a light-weight container in similar ways to chroot. It is more powerful since it uses namespaces to fully virtualize the the process tree, IPC, hostname, domain name and, optionally, networking and user databases. See systemd-nspawn.