mikas blog (original) (raw)

The mysterious XF86AudioPlay issue

May 20th, 2026

I was getting “ is undefined” in the status bar of Emacs displayed every 2-3 seconds. Nowhere else I noticed any misbehavior or problems, and also couldn’t find any related log entries. It didn’t stop, though didn’t want to reboot my system to see whether that would fix the problem, but it was driving me nuts.

Now, as a starting point I adjusted my sway configuration, to react to the XF86AudioPlay key press event:

bindsym XF86AudioPlay exec playerctl play-pause

After reloading sway, my music player started to play for 2-3 seconds, stopped playing, started again, etc. It wasn’t a Emacs bug, but something indeed seemed to send the XF86AudioPlay key event every 2-3 seconds. It wasn’t my USB keyboard or any stuck key on it, as verified also by unplugging it. So which device was causing this?

libinput from libinput-tools to the rescue:

% sudo libinput debug-events [...] -event12 KEYBOARD_KEY +0.000s KEY_PLAYPAUSE (164) pressed event12 KEYBOARD_KEY +0.000s KEY_PLAYPAUSE (164) released event12 KEYBOARD_KEY +2.887s KEY_PLAYPAUSE (164) pressed event12 KEYBOARD_KEY +2.887s KEY_PLAYPAUSE (164) released event12 KEYBOARD_KEY +5.773s KEY_PLAYPAUSE (164) pressed event12 KEYBOARD_KEY +5.774s KEY_PLAYPAUSE (164) released [...]

The `_event12_` device was sending this event, what’s behind this?

% sudo udevadm info /dev/input/event12 P: /devices/pci0000:00/0000:00:1f.3/skl_hda_dsp_generic/sound/card0/input17/event12 M: event12 R: 12 J: c13:76 U: input D: c 13:76 N: input/event12 L: 0 S: input/by-path/pci-0000:00:1f.3-platform-skl_hda_dsp_generic-event E: DEVPATH=/devices/pci0000:00/0000:00:1f.3/skl_hda_dsp_generic/sound/card0/input17/event12 E: DEVNAME=/dev/input/event12 E: MAJOR=13 E: MINOR=76 E: SUBSYSTEM=input E: USEC_INITIALIZED=12468722 E: ID_INPUT=1 E: ID_INPUT_KEY=1 E: ID_INPUT_SWITCH=1 E: ID_PATH=pci-0000:00:1f.3-platform-skl_hda_dsp_generic E: ID_PATH_TAG=pci-0000_00_1f_3-platform-skl_hda_dsp_generic E: XKBMODEL=pc105 E: XKBLAYOUT=us E: XKBOPTIONS=lv3:ralt_switch,compose:rctrl E: BACKSPACE=guess E: LIBINPUT_DEVICE_GROUP=0/0/0:ALSA E: DEVLINKS=/dev/input/by-path/pci-0000:00:1f.3-platform-skl_hda_dsp_generic-event E: TAGS=:power-switch: E: CURRENT_TAGS=:power-switch:

% sudo udevadm info -a /dev/input/event12 | grep -iE 'kernels|drivers|name' KERNELS=="input17" DRIVERS=="" ATTRS{name}=="sof-hda-dsp Headphone" KERNELS=="card0" DRIVERS=="" KERNELS=="skl_hda_dsp_generic" DRIVERS=="skl_hda_dsp_generic" KERNELS=="0000:00:1f.3" DRIVERS=="sof-audio-pci-intel-tgl" KERNELS=="pci0000:00" DRIVERS==""

Behind this event12 is sof-hda-dsp Headphone, and evtest confirms that:

% sudo evtest No device specified, trying to scan all of /dev/input/event* Available devices: /dev/input/event0: AT Translated Set 2 keyboard /dev/input/event1: Sleep Button /dev/input/event10: ThinkPad Extra Buttons /dev/input/event11: sof-hda-dsp Mic /dev/input/event12: sof-hda-dsp Headphone /dev/input/event13: sof-hda-dsp HDMI/DP,pcm=3 /dev/input/event14: sof-hda-dsp HDMI/DP,pcm=4 /dev/input/event15: sof-hda-dsp HDMI/DP,pcm=5 /dev/input/event16: Yubico YubiKey OTP+FIDO+CCID /dev/input/event17: Apple Inc. Magic Keyboard with Numeric Keypad /dev/input/event18: Apple Inc. Magic Keyboard with Numeric Keypad [...] Select the device event number [0-24]: ^C

We can even get further information:

% sudo evtest /dev/input/event12 Input driver version is 1.0.1 Input device ID: bus 0x0 vendor 0x0 product 0x0 version 0x0 Input device name: "sof-hda-dsp Headphone" Supported events: Event type 0 (EV_SYN) Event type 1 (EV_KEY) Event code 114 (KEY_VOLUMEDOWN) Event code 115 (KEY_VOLUMEUP) Event code 164 (KEY_PLAYPAUSE) Event code 582 (KEY_VOICECOMMAND) Event type 5 (EV_SW) Event code 2 (SW_HEADPHONE_INSERT) state 0 Properties: Testing ... (interrupt to exit) Event: time 1779295060.175766, type 5 (EV_SW), code 2 (SW_HEADPHONE_INSERT), value 1 Event: time 1779295060.175766, -------------- SYN_REPORT ------------ Event: time 1779295061.951168, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295061.951168, -------------- SYN_REPORT ------------ Event: time 1779295061.951194, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295061.951194, -------------- SYN_REPORT ------------ Event: time 1779295064.548671, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295064.548671, -------------- SYN_REPORT ------------ Event: time 1779295064.548689, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295064.548689, -------------- SYN_REPORT ------------ Event: time 1779295067.437172, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295067.437172, -------------- SYN_REPORT ------------ Event: time 1779295067.437187, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295067.437187, -------------- SYN_REPORT ------------ Event: time 1779295070.323775, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295070.323775, -------------- SYN_REPORT ------------ Event: time 1779295070.323790, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295070.323790, -------------- SYN_REPORT ------------ Event: time 1779295073.200350, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295073.200350, -------------- SYN_REPORT ------------ Event: time 1779295073.200373, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295073.200373, -------------- SYN_REPORT ------------ Event: time 1779295076.076228, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295076.076228, -------------- SYN_REPORT ------------ Event: time 1779295076.076250, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295076.076250, -------------- SYN_REPORT ------------ Event: time 1779295078.961740, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295078.961740, -------------- SYN_REPORT ------------ Event: time 1779295078.961754, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295078.961754, -------------- SYN_REPORT ------------ Event: time 1779295081.850156, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1 Event: time 1779295081.850156, -------------- SYN_REPORT ------------ Event: time 1779295081.850175, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0 Event: time 1779295081.850175, -------------- SYN_REPORT ------------ Event: time 1779295083.306612, type 5 (EV_SW), code 2 (SW_HEADPHONE_INSERT), value 0 Event: time 1779295083.306612, -------------- SYN_REPORT ------------

So when I plug in my headphone (see the `SW_HEADPHONE_INSERT` event), the unexpected behavior starts, unplugging stops the problem.
Good! But what was totally unexpected for me: my headphone, being a Beyerdynamic DT-990 Pro, does not have any keys. 8-)

As it turned out, the headphone jack seemed to have been not entirely clean. The analog side of the jack triggers a behavior within the audio codec, where it seems to interpret the fluctuating impedance as a play button of the headset, being pressed, again and again.

I cleaned the jack of my headphone and my XF86AudioPlay problem is gone, case closed.

Posted in Computer, Debian, English, Hardware | 1 Comment »

apt, SHA-1 keys + 2026-02-01

January 31st, 2026

You might have seen Policy will reject signature within a year warnings in apt(-get) update runs like this:

root@424812bd4556:/# apt update Get:1 http://foo.example.org/debian demo InRelease [4229 B] Hit:2 http://deb.debian.org/debian trixie InRelease Hit:3 http://deb.debian.org/debian trixie-updates InRelease Hit:4 http://deb.debian.org/debian-security trixie-security InRelease Get:5 http://foo.example.org/debian demo/main amd64 Packages [1097 B] Fetched 5326 B in 0s (43.2 kB/s) All packages are up to date. Warning: http://foo.example.org/debian/dists/demo/InRelease: Policy will reject signature within a year, see --audit for details

root@424812bd4556:/# apt --audit update Hit:1 http://foo.example.org/debian demo InRelease Hit:2 http://deb.debian.org/debian trixie InRelease Hit:3 http://deb.debian.org/debian trixie-updates InRelease Hit:4 http://deb.debian.org/debian-security trixie-security InRelease All packages are up to date.
Warning: http://foo.example.org/debian/dists/demo/InRelease: Policy will reject signature within a year, see --audit for details Audit: http://foo.example.org/debian/dists/demo/InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Signing key on 54321ABCD6789ABCD0123ABCD124567ABCD89123 is not bound: No binding signature at time 2024-06-19T10:33:47Z because: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance because: SHA1 is not considered secure since 2026-02-01T00:00:00Z Audit: The sources.list(5) entry for 'http://foo.example.org/debian' should be upgraded to deb822 .sources Audit: Missing Signed-By in the sources.list(5) entry for 'http://foo.example.org/debian' Audit: Consider migrating all sources.list(5) entries to the deb822 .sources format Audit: The deb822 .sources format supports both embedded as well as external OpenPGP keys Audit: See apt-secure(8) for best practices in configuring repository signing. Audit: Some sources can be modernized. Run 'apt modernize-sources' to do so.

If you ignored this for the last year, I would like to tell you that 2026-02-01 is not that far away (hello from the past if you’re reading this because you’re already affected).

Let’s simulate the future:

root@424812bd4556:/# apt --update -y install faketime [...] root@424812bd4556:/# export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 FAKETIME="2026-08-29 23:42:11" root@424812bd4556:/# date Sat Aug 29 23:42:11 UTC 2026

root@424812bd4556:/# apt update Get:1 http://foo.example.org/debian demo InRelease [4229 B] Hit:2 http://deb.debian.org/debian trixie InRelease
Err:1 http://foo.example.org/debian demo InRelease Sub-process /usr/bin/sqv returned an error code (1), error message is: Signing key on 54321ABCD6789ABCD0123ABCD124567ABCD89123 is not bound: No binding signature at time 2024-06-19T10:33:47Z because: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance because: SHA1 is not considered secure since 2026-02-01T00:00:00Z [...] Warning: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://foo.example.org/debian demo InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Signing key on 54321ABCD6789ABCD0123ABCD124567ABCD89123 is not bound: No binding signature at time 2024-06-19T10:33:47Z because: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance because: SHA1 is not considered secure since 2026-02-01T00:00:00Z [...] root@424812bd4556:/# echo $? 100

Now, the proper solution would have been to fix the signing key underneath (via e.g. sq cert lint ‐‐fix ‐‐cert-file PRIVATKEYFILE>PRIVAT_KEY_FILE > PRIVATKEYFILE>PRIVAT_KEY_FILE-fixed).

If you don’t have access to the according private key (e.g. when using an upstream repository that has been ignoring this issue), you’re out of luck for a proper fix.

But there’s a workaround for the apt situation (related see apt commit 0989275c2f7afb7a5f7698a096664a1035118ebf):

root@424812bd4556:/# cat /usr/share/apt/default-sequoia.config

Default APT Sequoia configuration. To overwrite, consider copying this

to /etc/crypto-policies/back-ends/apt-sequoia.config and modify the

desired values.

[asymmetric_algorithms] dsa2048 = 2024-02-01 dsa3072 = 2024-02-01 dsa4096 = 2024-02-01 brainpoolp256 = 2028-02-01 brainpoolp384 = 2028-02-01 brainpoolp512 = 2028-02-01 rsa2048 = 2030-02-01

[hash_algorithms] sha1.second_preimage_resistance = 2026-02-01 # Extend the expiry for legacy repositories sha224 = 2026-02-01

[packets] signature.v3 = 2026-02-01 # Extend the expiry

Adjust this according to your needs:

root@424812bd4556:/# mkdir -p /etc/crypto-policies/back-ends/

root@424812bd4556:/# cp /usr/share/apt/default-sequoia.config /etc/crypto-policies/back-ends/apt-sequoia.config

root@424812bd4556:/# $EDITOR /etc/crypto-policies/back-ends/apt-sequoia.config

root@424812bd4556:/# cat /etc/crypto-policies/back-ends/apt-sequoia.config

APT Sequoia override configuration

[asymmetric_algorithms] dsa2048 = 2024-02-01 dsa3072 = 2024-02-01 dsa4096 = 2024-02-01 brainpoolp256 = 2028-02-01 brainpoolp384 = 2028-02-01 brainpoolp512 = 2028-02-01 rsa2048 = 2030-02-01

[hash_algorithms] sha1.second_preimage_resistance = 2026-09-01 # Extend the expiry for legacy repositories sha224 = 2026-09-01

[packets] signature.v3 = 2026-02-01 # Extend the expiry

Then we’re back into the original situation, being a warning instead of an error:

root@424812bd4556:/# apt update Hit:1 http://deb.debian.org/debian trixie InRelease Get:2 http://foo.example.org/debian demo InRelease [4229 B] Hit:3 http://deb.debian.org/debian trixie-updates InRelease Hit:4 http://deb.debian.org/debian-security trixie-security InRelease Warning: http://foo.example.org/debian/dists/demo/InRelease: Policy will reject signature within a year, see --audit for details [..]

Please note that this is a workaround, and not a proper solution.

Posted in Computer, Debian, English, Open Source | Comments Off on apt, SHA-1 keys + 2026-02-01

Bookdump 2025

January 10th, 2026

Mein Lesejahr 2025 war mit durchschnittlich bisschen mehr als einem Buch pro Woche vergleichbar mit 2024. Mein Best-Of der von mir 2025 fertig gelesenen B�cher (jene die ich besonders lesenswert fand bzw. empfehlen m�chte, die Reihenfolge entspricht dem Foto und stellt keinerlei Reihung dar):

• Russische Spezialit�ten, Dmitrij Kapitelman. Was f�r ein Feuerwerk von einem Buch, sprachgewaltig, traurig, am�sant.
• Die Jungfrau, Monika Helfer. Nach Helfers “Die Bagage”, “L�wenherz” und “Vati” war nat�rlich auch dieses Buch Pflichtlekt�re f�r mich.
• Das Buch zum Film, Clemens J. Setz. Wunderbare Alltagsbeobachtungen und Bonmots – ich hab eigentlich nur eine Kritik: mit 192 Seiten zu kurz.
• Wackelkontakt, Wolf Haas. Jaja, ein bekannter Bestseller etc. Aber er ist und bleibt einer meiner Lieblingsautoren. Ich war bei seiner Lesung in Graz und habe das Buch im Anschluss sogar noch ein zweites Mal gelesen, und es keine Sekunde bereut. Sprachk�nstler, Hilfsausdruck!
• Fleisch ist mein Gem�se, Heinz Strunk. Ich liebe Background-Geschichten, speziell wenn es um Musik bzw. das Musikerleben geht, und das ist hier mit dem Ausflug in die Branche der Tanzmusik der Fall. Bis auf einige wenige Ausnahmen flutscht es beim Lesen.
• Wut und Wertung: Warum wir �ber Geschmack streiten, Johannes Franzen. Warum eskalieren Konflikte �ber Geschmack, Kunst und Kanon? Warum ist Streiten �ber Geschmack eine wichtige Kulturtechnik? Franzen arbeitet das anhand von tats�chlich existierenden Kontroversen und Skandalen auf, lehrreich und anregend.
• Klapper, Kurt Pr�del. Fans von Clemens J. Setz kennen nat�rlich Pr�del, und da ich auch Coming-of-Age-Romanen mag, war das ein doppelter Volltreffer. Ich freue mich schon auf sein neues Buch “Salto”!
• Hier treibt mein Kartoffelherz, Anna Weidenholzer. Ich kann absolut nichts mehr zu diesem Buch sagen, aber ich hab’s echt gern gelesen.
• Die Infantin tr�gt den Scheitel links, Helena Adler. Das Buch hatte einen interessanten Sog auf mich, ich wollte es einfach weiterlesen. Die verspielte Sprache und Wortspiele haben es noch feiner gemacht.
• Das sch�ne Leben, Christiane R�singer. Ich hab R�singers B�cher von Kathrin Passig empfohlen bekommen (Volltreffer, danke!). Ich hab mir auch alle anderen B�cher von R�singer (“Berlin – Baku. Meine Reise zum Eurovision Song Contest”, “Zukunft machen wir sp�ter: Meine Deutschstunden mit Gefl�chteten”, “Liebe wird oft �berbewertet”) besorgt, und sehr gerne gelesen.

Posted in Allgemein, B�cher & CO | Comments Off on Bookdump 2025

HTU Bigband: Weihnachtskonzert am 12.12.2025

December 3rd, 2025

Weihnachten im Dezember – und das mit guter Bigband-Musik! 🎄

Am 12. Dezember 2025 laden wir euch zu einem Weihnachtskonzert der besonderen Art ein – mit der HTU Bigband im Mo.xx! Es gibt Jazz-Rock, Swing, Soul-Pop, Latin + Funk und nat�rlich jede Menge gute Stimmung. Einlass ist um 19 Uhr, Eintritt freiwillige Spende, und ab 19:30 Uhr gibt es dann feine Musik auf die Ohren. 🎶

🎵 Was: feine Bigband-Musik
⏰ Wann: Freitag, 12. Dezember 2025, ab 19:30 Uhr
📍 Wo: mo.xx, Moserhofgasse 34, Graz

Kommt vorbei, am Besten mit Familie und Freunden! 🎅

Posted in Allgemein, Events, Musik | Comments Off on HTU Bigband: Weihnachtskonzert am 12.12.2025

HTU Bigband @ 30 Jahre Radio Helsinki am 22.11.2025

November 14th, 2025

Radio Helsinki wird 30, und feiert das mit einem prall gef�llten Musikprogramm am Samstag, 22.11.2025 im Forum Stadtpark. 🥳 Wir sind mit der HTU-Bigband mit dabei, und spielen ab ~19:30 Uhr f�r ~1,5 Stunden. Schaut vorbei und feiert mit!

PS: wer es wirklich nicht hinschafft, m�ge zumindest das Radio einschalten oder den Livestream aufdrehen! 🤓

Foto-Quelle / Copyright: graz.social/@radiohelsinki/115428633169757433

Posted in Allgemein, Events, Musik | Comments Off on HTU Bigband @ 30 Jahre Radio Helsinki am 22.11.2025

What to expect from Debian/trixie #newintrixie

July 20th, 2025

Update on 2025-07-28: added note about Debian 13/trixie support for OpenVox (thanks, Ben Ford!)

Debian v13 with codename trixie is scheduled to be published as new stable release on 9th of August 2025.

I was the driving force at several of my customers to be well prepared for the upcoming stable release (my efforts for trixie started in August 2024). On the one hand, to make sure packages we care about are available and actually make it into the release. On the other hand, to ensure there are no severe issues that make it into the release and to get proper and working upgrades. So far everything is looking pretty well and working fine, the efforts seemed to have payed off. :)

As usual with major upgrades, there are some things to be aware of, and hereby I’m starting my public notes on trixie that might be worth for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.

Further readings

As usual start at the official Debian release notes, make sure to especially go through What’s new in Debian 13 + issues to be aware of for trixie (strongly recommended read!).

Package versions

As a starting point, let’s look at some selected packages and their versions in bookworm vs. trixie as of 2025-07-20 (mainly having amd64 in mind):

Misc unsorted

• The asterisk package once again didn’t make it into trixie (see #1031046)
• The new debian-repro-status package provides the identically named command-line tool debian-repro-status to query the reproducibility status of your installed Debian packages
• The Grml live system project provided further of their packages into Debian. Available as of trixie are now also grml-keyring (OpenPGP certificates used for signing the Grml repositories), grml-hwinfo (a tool which collects information of the hardware ) + grml-paste (command line interface for paste.debian.net)
• If you use pacemaker, be aware that its fence-agents package is now a transitional package. All the fence-agents got split into separate packages (fence-agents-$whatever). If you want to have all the fence-agents available, make sure to install the fence-agents-all package. If you have Recommends disabled, you definitely should be aware of this.
• usrmerge is finalized (also see dpkg warning issue in release notes)
• For an overview of the XMPP/Jabber situation in trixie see xmpp-team’s blog post
• The curl package now includes the wcurl command line tool, being a simple wrapper around curl to easily download files

apt

The new apt version 3.0 brings several new features, including:

• support for colors (f.e. green for installs/upgrades, yellow for downgrades, red for removals, can be disabled via ‐‐no-color, APT_NO_COLOR=1 or NO_COLOR=1 and customized via e.g. APT::Color::Action::Install “cyan”)
• organizes output in more readable sections and shows removals more prominently
• uses sequoia to verify signatures
• includes a new solver
• the new apt modernize-sources command converts /etc/apt/sources.list.d/*.list files into the new .sources format (AKA DEB822)
• the new apt distclean command removes all files under $statedir/lists except Release, Release.gpg, and InRelease (it can be used for example, when finalizing images distributed to users)
• new configuration option APT::NeverAutoRemove::KernelCount for keeping a configurable amount of kernels, f.e. setting APT::NeverAutoRemove::KernelCount 3 will keep 3 kernels (including the running, and most recent)
• new command line option ‐‐snapshot, and configuration option APT::Snapshot, controlling the snapshot chosen for archives with Snapshot: enable
• new command line option ‐‐update to run the update command before the specified command, like apt ‐‐update install zsh,
  apt ‐‐update remove foobar or apt ‐‐update safe-upgrade
• apt-key is gone, and there’s no replacement for it available (if you need an interface for listing present keys)

systemd

systemd got upgraded from v252.36-1~deb12u1 to 257.7-1 and there are lots of changes.

Be aware that systemd v257 has a new net.naming_scheme, v257 being PCI slot number is now read from firmware_node/sun sysfs file. The naming scheme based on devicetree aliases was extended to support aliases for individual interfaces of controllers with multiple ports. This might affect you, see e.g. #1092176 and #1107187, the Debian Wiki provides further useful information.

There are new systemd tools available:

• run0: temporarily and interactively acquire elevated or different privileges (serves a similar purpose as sudo)
• systemd-ac-power: Report whether we are connected to an external power source
• systemd-confext: Activates System Extension Images
• systemd-vpick: Resolve paths to ‘.v/’ versioned directories
• varlinkctl: Introspect with and invoke Varlink services

The tools provided by systemd gained several new options:

• busctl: new option ‐‐limit‐messages=NUMBER (Stop monitoring after receiving the specified number of message)
• hostnamectl: new option ‐j (same as ‐‐json=pretty on tty, ‐‐json=short otherwise)
• journalctl: new options ‐‐image‐policy=POLICY (Specify disk image dissection policy), ‐‐invocation=ID (Show logs from the matching invocation ID), ‐I (Show logs from the latest invocation of unit), ‐‐exclude-identifier=STRING (Hide entries with the specified syslog identifier),‐‐truncate-newline (Truncate entries by first newline character), ‐‐list-invocations (Show invocation IDs of specified unit), ‐‐list-namespaces (Show list of journal namespaces)
• kernel-install: new commands add‐all + list and plenty of new command line options
• localectl: new option ‐‐full (Do not ellipsize output)
• loginctl: new options ‐‐json=MODE (Generate JSON output for list-sessions/users/seats) + ‐j (Same as ‐‐json=pretty on tty, ‐‐json=short otherwise)
• networkctl: new commands edit FILES|DEVICES… (Edit network configuration files), cat [FILES|DEVICES…] (Show network configuration files), mask FILES… (Mask network configuration files) + unmask FILES… (Unmask network configuration files) + persistent-storage BOOL (Notify systemd-networkd if persistent storage is ready), and new options ‐‐no-ask-password (Do not prompt for password), ‐‐no-reload (Do not reload systemd-networkd or systemd-udevd after editing network config), ‐‐drop-in=NAME (Edit specified drop-in instead of main config file), ‐‐runtime (Edit runtime config files) + ‐‐stdin (Read new contents of edited file from stdin)
• systemctl” new commands list-paths [PATTERN] (List path units currently in memory, ordered by path), whoami [PID…] (Return unit caller or specified PIDs are part of), soft-reboot (Shut down and reboot userspace) + sleep (Put the system to sleep), and new options ‐‐capsule=NAME (Connect to service manager of specified capsule), ‐‐before (Show units ordered before with ‘list-dependencies’), ‐‐after (Show units ordered after with ‘list-dependencies’), ‐‐kill-value=INT (Signal value to enqueue), ‐‐no-warn (Suppress several warnings shown by default), ‐‐message=MESSAGE (Specify human readable reason for system shutdown), ‐‐image‐policy=POLICY (Specify disk image dissection policy), ‐‐reboot‐argument=ARG (Specify argument string to pass to reboot()), ‐‐drop-in=NAME (Edit unit files using the specified drop-in file name), ‐‐when=TIME (Schedule halt/power-off/reboot/kexec action after a certain timestamp) + ‐‐stdin (Read
  new contents of edited file from stdin)
• systemd-analyze” new commands architectures [NAME…] (List known architectures), smbios11 (List strings passed via SMBIOS Type #11), image-policy POLICY… (Analyze image policy string), fdstore SERVICE… (Show file descriptor store contents of service), malloc [D-BUS SERVICE…] (Dump malloc stats of a D-Bus service), has-tpm2 (Report whether TPM2 support is available), pcrs [PCR…] (Show TPM2 PCRs and their names) + srk [>FILE] (Write TPM2 SRK (to FILE)) and new options ‐‐no-legend (Disable column headers and hints in plot with either ‐‐table or ‐‐json=), ‐‐instance=NAME (Specify fallback instance name for template units), ‐‐unit=UNIT (Evaluate conditions and asserts of unit), ‐‐table (Output plot’s raw time data as a table), ‐‐scale-svg=FACTOR (Stretch x-axis of plot by FACTOR (default: 1.0)), ‐‐detailed (Add more details to SVG plot), ‐‐tldr (Skip comments and empty lines), ‐‐image
  -policy=POLICY (Specify disk image dissection policy) + ‐‐mask (Parse parameter as numeric capability mask)
• systemd-ask-password: new options ‐‐user (Ask only our own user’s agents) + ‐‐system (Ask agents of the system and of all users)
• systemd-cat: new option ‐‐namespace=NAMESPACE (Connect to specified journal namespace)
• systemd-creds: new options ‐‐user (Select user-scoped credential encryption), ‐‐uid=UID (Select user for scoped credentials) + ‐‐allow-null (Allow decrypting credentials with empty key)
• systemd-detect-virt: new options ‐‐cvm (Only detect whether we are run in a confidential VM) + ‐‐list-cvm (List all known and detectable types of confidential virtualization)
• systemd-firstboot: new options ‐‐image-policy=POLICY (Specify disk image dissection policy), ‐‐kernel-command-line=CMDLINE (Set kernel command line) + ‐‐reset (Remove existing files)
• systemd-id128: new commands var-partition-uuid (Print the UUID for the /var/ partition) + show [NAME|UUID] (Print one or more UUIDs), and new options ‐‐no-pager (Do not pipe output into a pager), ‐‐no-legend (Do not show the headers and footers), ‐‐json=FORMAT (Output inspection data in JSON), ‐j (Equivalent to ‐‐json=pretty (on TTY) or ‐‐json=short (otherwise)) + ‐P ‐‐value (Only print the value)
• systemd-inhibit: new option ‐‐no-ask-password (Do not attempt interactive authorization)
• systemd-machine-id-setup: new option ‐‐image-policy=POLICY (Specify disk image dissection policy)
• systemd-mount: new options ‐‐json=pretty|short|off (Generate JSON output) + ‐‐tmpfs (Create a new tmpfs on the mount point)
• systemd-notify: new options ‐‐reloading (Inform the service manager about configuration reloading), ‐‐stopping (Inform the service manager about service shutdown), ‐‐exec (Execute command line separated by ‘;’ once done), ‐‐fd=FD (Pass specified file descriptor with along with message) + ‐‐fdname=NAME (Name to assign to passed file descriptor(s))
• systemd-path: new option ‐‐no-pager (Do not pipe output into a pager)
• systemd-run: new options ‐‐expand-environment=BOOL (Control expansion of environment variables), ‐‐json=pretty|short|off (Print unit name and invocation id as JSON), ‐‐ignore-failure (Ignore the exit status of the invoked process) + ‐‐background=COLOR (Set ANSI color for background)
• systemd-sysext: new options ‐‐mutable=yes|no|auto|import|ephemeral|ephemeral-import (Specify a mutability mode of the merged hierarchy), ‐‐no-reload (Do not reload the service manager), ‐‐image-policy=POLICY (Specify disk image dissection policy) + ‐‐noexec=BOOL (Whether to mount extension overlay with noexec)
• systemd-sysusers: new options ‐‐tldr (Show non-comment parts of configuration) + ‐‐image-policy=POLICY (Specify disk image dissection policy)
• systemd-tmpfiles: new command ‐‐purge(Delete files and directories marked for creation in specified configuration files (careful!)), and new options ‐‐user (Execute user configuration), ‐‐tldr (Show non-comment parts of configuration files), ‐‐graceful (Quietly ignore unknown users or groups), ‐‐image-policy=POLICY (Specify disk image dissection policy) + ‐‐dry-run (Just print what would be done)
• systemd-umount: new options ‐‐json=pretty|short|off (Generate JSON output) + ‐‐tmpfs (Create a new tmpfs on the mount point)
• timedatectl: new commands ntp-servers INTERFACE SERVER (Set the interface specific NTP servers) + revert INTERFACE (Revert the interface specific NTP servers) and new option ‐P NAME (Equivalent to ‐‐value ‐‐property=NAME)

Debian’s systemd ships new binary packages:

• systemd-boot-efi-amd64-signed (Tools to manage UEFI firmware updates (signed))
• systemd-boot-tools (simple UEFI boot manager – tools)
• systemd-cryptsetup (Provides cryptsetup, integritysetup and veritysetup utilities)
• systemd-netlogd (journal message forwarder)
• systemd-repart (Provides the systemd-repart and systemd-sbsign utilities)
• systemd-standalone-shutdown (standalone shutdown binary for use in exitrds)
• systemd-ukify (tool to build Unified Kernel Images)

Linux Kernel

The trixie release ships a Linux kernel based on latest longterm version 6.12. As usual there are lots of changes in the kernel area, including better hardware support, and this might warrant a separate blog entry. To highlight some changes with Debian trixie:

• New Debian package linux-bpf-dev, providing the header file for BPF CO-RE builds
• New Debian package intel-sdsi, Intel On Demand (SDSi) provisioning tool
• New Debian package virtme-ng, providing helper scripts to easily test a Linux kernel on a QEMU VM
• The kernel modules are installed xz compressed
• Several new syscalls, like cachestat, fchmodat2, futex_wake, futex_wait, futex_requeue, listmount, statmount, lsm_get_self_attr/lsm_set_self_attr/lsm_list_modules/, mseal + setxattrat/getxattrat/listxattrat/removexattrat
• New Integrity Policy Enforcement (IPE) LSM
• Plenty of changes in io_uring, including support for bind/listen
• support devices with a block size larger than system page sizes in the XFS filesystem + VFS
• ntfs driver was replaced by ntfs3
• Device Memory TCP for faster network device transfers
• Support for ‘perf ftrace profile’
• support for atomic write operations in the block layer
• FUSE pass-through for file I/O
• ability to prevent writes to block devices containing mounted filesystems
• support for data-type profiling in the perf tool
• Guest-first memory for KVM
• TCP Authentication Option Linux implementation [RFC5925]
• mount beneath support for filesystems
• new noswap mount option for tmpfs filesystems
• Linux NFS server gained support for RPC-with-TLS [RFC 9289 (Towards Remote Procedure Call Encryption by Default)]
• PLB (Protective Load Balancing) for IPv6 (PLB is a host based mechanism for load balancing across switch links)

See Kernelnewbies.org for further changes between kernel versions.

Configuration management

For puppet users, Debian provides the puppet-agent (v8.10.0), puppetserver (v8.7.0) and puppetdb (v8.4.1) packages. Puppet’s upstream does not provide packages for trixie, yet. Given how long it took them for Debian bookworm, and with their recent Plans for Open Source Puppet in 2025, it’s unclear when (and whether at all) we might get something. As a result of upstream behavior, also the OpenVox project evolved, and they already provide Debian 13/trixie support (https://apt.voxpupuli.org/openvox8-release-debian13.deb). FYI: the AIO puppet-agent package for bookworm (v7.34.0-1bookworm) so far works fine for me on Debian/trixie. Be aware that due to the apt-key removal you need a recent version of the puppetlabs-apt for usage with trixie. The puppetlabs-ntp module isn’t yet ready for trixie (regarding ntp/ntpsec), if you should depend on that.

ansible is available and made it with version 2.19 into trixie.

Prometheus stack

Prometheus server was updated from v2.42.0 to v2.53, and all the exporters that got shipped with bookworm are still around (in more recent versions of course). Trixie gained some new exporters:

• prometheus-dnsmasq-exporter
• prometheus-mysqlrouter-exporter
• prometheus-pgbackrest-exporter
• prometheus-pgbouncer-exporter
• prometheus-phpfpm-exporter

Virtualization

docker (v26.1.5), ganeti (v3.1.0), libvirt (v11.3.0, be aware of significant changes to libvirt packaging), lxc (v6.0.4), podman (v5.4.2), openstack (see openstack-team on Salsa), qemu/kvm (v10.0.2), xen (v4.20.0) are all still around.

Proxmox already announced their PVE 9.0 BETA, being based on trixie and providing 6.14.8-1 kernel, QEMU 10.0.2, LXC 6.0.4, OpenZFS 2.3.3.

Vagrant is available in version 2.3.7, but Vagrant upstream does not provide packages for trixie yet. Given that HashiCorp adopted the BSL, the future of vagrant in Debian is unclear.

If you’re relying on VirtualBox, be aware that upstream doesn’t provide packages for trixie, yet. VirtualBox is available from Debian/unstable (version 7.1.12-dfsg-1 as of 2025-07-20), but not shipped with stable release since quite some time (due to lack of cooperation from upstream on security support for older releases, see #794466). Be aware that starting with Linux kernel 6.12, KVM initializes virtualization on module loading by default. This prevents VirtualBox VMs from starting. In order to avoid this, either add “kvm.enable_virt_at_load=0” parameter into kernel command line or unload the corresponding kvm_intel / kvm_amd module.

If you want to use Vagrant with VirtualBox on trixie, be aware that Debian’s vagrant package as present in trixie doesn’t support the VirtualBox package version 7.1 as present in Debian/unstable (manually patching vagrant’s meta.rb and rebuilding the package without Breaks: virtualbox (>= 7.1) is known to be working).

util-linux

The are plenty of new options available in the tools provided by util-linux:

• blkdiscard: new option ‐‐quiet (suppress warning messages)
• blockdev: new options ‐‐getdiskseq (get disk sequence number) + ‐‐getzonesz (get zone size)
• dmesg: new option ‐‐kmsg-file … (use the file in kmsg format), new ‐‐time-format … argument ‘raw’
• findmnt: new options ‐‐list-columns (list the available columns), ‐‐dfi (imitate the output of df(1) with -i option), ‐‐id … (filter by mount node ID), ‐‐filter … (apply display filter) + ‐‐uniq-id … (filter by
  mount node 64-bit ID)
• fstrim: new option -types …. (limit the set of filesystem types)
• hardlink: new options ‐‐respect-dir (directory names have to be identical), ‐‐exclude-subtree … (regular expression to exclude directories), ‐‐prioritize-trees (files found in the earliest specified top-level directory have higher priority), ‐‐list-duplicates (print every group of duplicate files), ‐‐mount (stay within the same filesystem) + ‐‐zero (delimit output with NULs instead of newlines)
• ipcmk: new options ‐‐posix-shmem … (create POSIX shared memory segment of size), ‐‐posix-semaphore … (create POSIX semaphore), ‐‐posix-mqueue … (create POSIX message queue) + ‐‐name … (name of the POSIX resource)
• ipcrm: new options ‐‐posix-shmem … (remove POSIX shared memory segment by name), ‐‐posix-mqueue … (remove POSIX message queue by name), ‐‐posix-semaphore (remove POSIX semaphore by name) + ‐‐all=… (remove all in specified category)
• lsblk: new options ‐‐ct-filter … (restrict the next counter), ‐‐ct … (define a custom counter), ‐‐highlight … (colorize lines matching the expression), ‐‐list-columns (list the available columns), ‐‐nvme (output info about NVMe devices), ‐‐properties-by … (methods used to gather data), ‐‐filter … (print only lines matching the expression), ‐‐virtio (output info about virtio devices)
• lscpu: new options ‐‐raw (use raw output format (for -e, -p and -C)) + ‐‐hierarchic=… (use subsections in summary (auto, never, always))
• lsipc: new options ‐‐posix-shmems (POSIX shared memory segments), ‐‐posix-mqueues (POSIX message queues), ‐‐posix-semaphores (POSIX semaphores), ‐‐name … (POSIX resource identified by name)
• lslocks: new option ‐‐list-columns (list the available columns)
• lslogins: new option ‐‐lastlog2 … (set an alternate path for lastlog2)
• lsns: new options ‐‐persistent (namespaces without processes), ‐‐filter … (apply display filter) + ‐‐list-columns (list the available columns)
• mkswap: new options ‐‐endianness=… (specify the endianness to use (native, little or big)), ‐‐offset … (specify the offset in the device), ‐‐size … (specify the size of a swap file in bytes) + ‐‐file (create a swap file)
• namei: ‐‐context (print any security context of each file)
• nsenter: new options ‐‐net-socket … (enter socket’s network namespace), ‐‐user-parent (enter parent user namespace), ‐‐keep-caps (retain capabilities granted in user namespaces), ‐‐env (inherit environment variables from tar get process) + ‐‐join-cgroup (join the cgroup of the target process)
• runuser: new option ‐‐no-pty (do not create a new pseudo-terminal)
• setarch: new option ‐‐show=… (show current or specific personality and exit)
• setpriv: new options ‐‐ptracer … (allow ptracing from the given process), ‐‐landlock-access … (add Landlock access), ‐‐landlock-rule … (add Landlock rule) + ‐‐seccomp-filter … (load seccomp filter from file)
• su: new option ‐‐no-pty (do not create a new pseudo-terminal)
• unshare: new option ‐‐load-interp … ( load binfmt definition in the namespace)
• whereis: new option -g (interpret name as glob (pathnames pattern))
• wipefs: new argument option feature for ‐‐backup=… option to specify directory (instead of default $HOME)
• zramctl: new option ‐‐algorithm-params … (algorithm parameters to use)

Now no longer present in util-linux as of trixie:

• addpart (tell the kernel about the existence of a specified partition): use partx instead
• delpart (tell the kernel to forget about a specified partition): use partx instead
• last (show a listing of last logged in users, binary got moved to wtmpdb), lastb (show a listing of last logged in users), mesg (control write access of other users to your terminal), utmpdump (dump UTMP and WTMP files in raw format): see Debian release notes for details

The following binaries got moved from util-linux to the util-linux-extra package:

• ctrlaltdel (set the function of the Ctrl-Alt-Del combination)
• mkfs.bfs (make an SCO bfs filesystem)
• fsck.cramfs + mkfs.cramfs (compressed ROM file system)
• fsck.minix + mkfs.minix (Minix filesystem)
• resizepart (tell the kernel about the new size of a partition)

And the util-linux-extra package also provides new tools:

• bits: convert bit masks from/to various formats
• blkpr: manage persistent reservations on a device
• coresched: manage core scheduling cookies for tasks
• enosys: utility to make syscalls fail with ENOSYS
• exch: atomically exchanges paths between two files
• fadvise: utility to use the posix_fadvise system call
• pipesz: set or examine pipe buffer sizes and optionally execute command.
• waitpid: utility to wait for arbitrary processes

OpenSSH

OpenSSH was updated from v9.2p1 to 10.0p1-5, so if you’re interested in all the changes, check out the release notes between those versions (9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9 + 10.0).

Let’s highlight some notable behavior changes in Debian:

• OpenSSH no longer supports DSA keys: see Debian’s release notes for further details
• openssh-server no longer reads ~/.pam_environment: see Debian’s release notes for further details

There are some notable new features:

• allow forwarding Unix Domain sockets via ssh -W
• OpenSSH penalty behavior: visit my separate blog post for more details
• add support for reading ED25519 private keys in PEM PKCS8 format. Previously only the OpenSSH private key format was supported.
• the new hybrid post-quantum algorithm mlkem768x25519-sha256 (based on the FIPS 203 Module-Lattice Key Encapsulation mechanism (ML-KEM) combined with X25519 ECDH) is now used by default for key agreement. This algorithm is considered to be safe against attack by quantum computers, is guaranteed to be no less strong than the popular curve25519-sha256 algorithm, has been standardised by NIST and is considerably faster than the previous default.
• the ssh-agent will now delete all loaded keys when signaled with SIGUSR1. This allows deletion of keys without having access to $SSH_AUTH_SOCK.
• support systemd-style socket activation in ssh-agent using the LISTEN_PID/LISTEN_FDS mechanism. Activated when these environment variables are set, the agent is started with the -d or -D option and no socket path is set.
• add a sshd -G option that parses and prints the effective configuration without attempting to load private keys and perform other checks. (This allows usage of the option before keys have been generated and for configuration evaluation and verification by unprivileged users.)
• add support for configuration tags to ssh(1). This adds a ssh_config(5) “Tag” directive and corresponding “Match tag” predicate that may be used to select blocks of configuration.
• add a “match localnetwork” predicate. This allows matching on the addresses of available network interfaces and may be used to vary the effective client configuration based on network location.
• add a %j token that expands to the configured ProxyJump hostname
• add support for “Match sessiontype” to ssh_config. Allows matching on the type of session initially requested, either “shell” for interactive sessions, “exec” for command execution sessions, “subsystem” for subsystem requests, such as sftp, or “none” for transport/forwarding-only sessions.
• allow glob(3) patterns to be used in sshd_config AuthorizedKeysFile and AuthorizedPrincipalsFile directives.

Thanks to everyone involved in the release, looking forward to trixie + and happy upgrading!
Let’s continue with working towards Debian/forky. :)

Posted in Computer, Debian, English, Open Source | 3 Comments »

Grml 2025.05 � codename Nudlaug

May 16th, 2025

Debian hard freeze on 2025-05-15? We bring you a new Grml release on top of that! 2025.05 🚀 – codename Nudlaug.

There’s plenty of new stuff, check out our official release announcement for all the details. But I’d like to highlight one feature that I particularly like: SSH service announcement with Avahi. The grml-full flavor ships Avahi, and when you enable SSH, it automatically announces the SSH service on your local network. So when f.e. booting Grml with boot option `_ssh=debian_`, you should be able to login on your Grml live system with `_ssh grml@grml.local_` and password ‘_debian_‘:

% insecssh grml@grml.local Warning: Permanently added 'grml.local' (ED25519) to the list of known hosts. grml@grml.local's password: Linux grml 6.12.27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.27-1 (2025-05-06) x86_64 Grml - Linux for geeks

grml@grml ~ %

Hint: grml-zshrc provides that useful shell alias `_insecssh_`, which is aliased to `_ssh -o “StrictHostKeyChecking=no” -o “UserKnownHostsFile=/dev/null”_`. Using those options, you aren’t storing the SSH host key of the (temporary) Grml live system (permanently) in your UserKnownHostsFile.

BTW, you can run `_avahi-browse -d local _ssh._tcp –resolve -t_` to discover the SSH services on your local network. 🤓

Happy Grml-ing!

Posted in Computer, Debian, English, Links, Open Source | Comments Off on Grml 2025.05 � codename Nudlaug

HTU Bigband Konzert am 05.06.2025

May 16th, 2025

Wie letztes Jahr schon, spielen wir auch heuer wieder mit der HTU-Bigband ein Konzert an der TU Graz.

Und zwar am Donnerstag, 5. Juni 2025! Das Konzert startet um 19:30 Uhr, bei Sch�nwetter im Innenhof der TU Graz (Alte Technik, Rechbauerstra�e 12, 8010 Graz), und bei Schlechtwetter geht es an der gleichen Adresse in den H�rsaal 2. Wir sind �ber 25 Musikerinnen und Musiker und haben ein anspruchsvolles Programm, von Swing, �ber Soul, Funk und Latin bis Pop ist alles dabei. Es gibt �ber 2 Stunden Musik vom Feinsten, die Set-List ist spitze, und das Ganze bei freiem Eintritt.

Ich freue mich schon tierisch darauf und w�rde mich wieder �ber bekannte und gut gelaunte Gesichter freuen. Ich hoffe man sieht und h�rt sich! :-)

Posted in Allgemein, Events | Comments Off on HTU Bigband Konzert am 05.06.2025

Lessons learned from running an open source project for 20 years @ GLT25

April 23rd, 2025

Time flies by so quickly, it’s >20 years since I started the Grml project.

I’m giving a (german) talk about the lessons learned from 20 years of running the Grml project this Saturday, 2025-04-26 at the Grazer Linuxtage (Graz/Austria). Would be great to see you there!

Posted in Computer, Debian, Debian-German, English, Events, Links, Open Source | Comments Off on Lessons learned from running an open source project for 20 years @ GLT25

OpenSSH penalty behavior in Debian/trixie #newintrixie

April 13th, 2025

This topic came up at a customer of mine in September 2024, when working on Debian/trixie support. Since then I wanted to blog about it to make people aware of this new OpenSSH feature and behavior. I finally found some spare minutes at Debian’s BSP in Vienna, so here we are. :)

Some of our Q/A jobs failed to run against Debian/trixie, in the debug logs we found:

debug1: kex_exchange_identification: banner line 0: Not allowed at this time

This Not allowed at this time pointed to a new OpenSSH feature. OpenSSH introduced options to penalize undesirable behavior with version 9.8p1, see OpenSSH Release Notes, and also sshd source code.

FTR, on the SSH server side, you’ll see messages like that:

Apr 13 08:57:11 grml sshd-session[2135]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55792 ssh2 [preauth] Apr 13 08:57:11 grml sshd-session[2135]: Disconnecting authenticating user root 10.100.15.42 port 55792: Too many authentication failures [preauth] Apr 13 08:57:12 grml sshd-session[2137]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55800 ssh2 [preauth] Apr 13 08:57:12 grml sshd-session[2137]: Disconnecting authenticating user root 10.100.15.42 port 55800: Too many authentication failures [preauth] Apr 13 08:57:13 grml sshd-session[2139]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55804 ssh2 [preauth] Apr 13 08:57:13 grml sshd-session[2139]: Disconnecting authenticating user root 10.100.15.42 port 55804: Too many authentication failures [preauth] Apr 13 08:57:13 grml sshd-session[2141]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55810 ssh2 [preauth] Apr 13 08:57:13 grml sshd-session[2141]: Disconnecting authenticating user root 10.100.15.42 port 55810: Too many authentication failures [preauth] Apr 13 08:57:13 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55818 on [10.100.15.230]:22 penalty: failed authentication Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55824 on [10.100.15.230]:22 penalty: failed authentication Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55838 on [10.100.15.230]:22 penalty: failed authentication Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55854 on [10.100.15.230]:22 penalty: failed authentication

This feature certainly is useful and has its use cases. But if you f.e. run automated checks to ensure that specific logins aren’t working, be careful: you might hit the penalty feature, lock yourself out but also consecutive checks then don’t behave as expected. Your login checks might fail, but only because the penalty behavior kicks in. The login you’re verifying still might be working underneath, but you don’t actually check for it exactly. Furthermore legitimate traffic from systems which accept connections from many users or behind shared IP addresses, like NAT and proxies could be denied.

To disable this new behavior, you can set PerSourcePenalties no in your sshd_config, but there are also further configuration options available, see PerSourcePenalties and PerSourcePenaltyExemptList settings in sshd_config(5) for further details.

Posted in Computer, Debian, English, Open Source | Comments Off on OpenSSH penalty behavior in Debian/trixie #newintrixie

Mein Lesejahr 2024

December 31st, 2024

Mein Lesejahr 2024 war mit durchschnittlich einem Buch pro Woche �hnlich wie 2023. Mein Best-Of der von mir 2024 fertig gelesenen B�cher (jene die ich besonders lesenswert fand bzw. empfehlen m�chte, die Reihenfolge entspricht dem Foto und stellt keinerlei Reihung dar):

• Die Geschichten in uns: Vom Schreiben und vom Leben, Benedict Wells. Sch�n gemacht und speziell f�r jene interessant, die sich f�r das “was steckt an Schreibarbeit hinter einem Buch” interessieren. Wells ist mit dem Buch die deutsche Version von Stephen Kings “On Writing” gelungen.
• Sand, Wolfgang Herrndorf. Ich bin bekannterweise Fan von Herrndorfs “Tschick” und “Arbeit und Struktur”, und damit war dieses Buch entsprechend eine Pflichtlekt�re f�r mich, und wow.
• F, Daniel Kehlmann. Von einem lieben Nachbarn vor die T�r gelegt bekommen und es war wirklich _f_antastisch zu lesen, es _f_lutschte nur so.
• Wo der spitzeste Zahn der Karawanken in den Himmel hinauf fletscht, Julia Jost. Das Buch wurde in der Literaturszene ziemlich gehypt, aber nach nur wenigen Seiten hab ich mich schon auf die Sprache eingegroovt. Viele Schaupl�tze sind mir bekannt und das Buch hatte dann eine interessante Sog-Wirkung. Die kursiv gesetzten umgangssprachlichen Ausdr�cke finde ich sehr gelungen.
• Herrndorf: Eine Biographie, Tobias R�ther. Ein Buch das f�r mich zum Verst�ndnis der Texte von Herrndorf beitr�gt. Eine sch�ne und stringente Geschichte, oder um es mit den Worten von Herrndorf zu sagen: “hohe Durchlesbarkeit”. Man findet auch viele bekannte Namen wieder (Kathrin Passig, Daniela Strigl, Sascha Lobo, Tex Rubinowitz, Klaus N�chtern,�).
• Iowa: Ein Ausflug nach Amerika, Stefanie Sargnagel. Sehr unterhaltsam und mit der gewohnten Beobachtungsgabe der Autorin, vielen Bonmots sowie super Fu�noten von Christiane R�singer.
• L�wenherz, Monika Helfer. Nachdem mir schon ihr “Die Bagage” (siehe Blogeintrag von 2020) und “Vati” (siehe Blogeintrag von 2023) so gut gefallen haben, war das Buch �ber ihren Bruder Richard entsprechend ein Must-Read, und es wird nicht mein letztes Buch von Helfer sein.
• Mitte des Lebens, Barbara Bleisch. Die Gedichte und Zitate h�tte es f�r mich nicht gebraucht, aber das Buch regt zum Nachdenken rund um die eigene Lebensmitte an, und es hat bei mir zeitlich gut in mein Leben gepasst.
• Ein sch�nes Ausl�nderkind, Toxische Pommes. Sch�n und traurig, aber wunderbar zu lesen – ich hoffe auf weitere B�cher von Toxische Pommes.
• Man kann auch in die H�he fallen, Joachim Meyerhoff. Wer mich kennt, wei�, dass ich Fan von Meyerhoffs “Alle Toten fliegen hoch”-Serie bin. Auch der neue Band ist wieder wunderbar geworden. (Ich bin nach wie vor auf der Suche nach �hnlich unterhaltsamen B�chern!)

Mein SuB bzw. Lesestapel f�r 2025 ist bereits gut gef�llt, ich freue mich aber trotzdem �ber etwaige Leseempfehlungen. Ebenso freue ich mich �ber Feedback, wenn jemand ein Buch aufgrund dieses Beitrags hier gelesen hat.

Posted in Allgemein, B�cher & CO | Comments Off on Mein Lesejahr 2024

Grml 2024.12 – codename Adventgrenze

December 20th, 2024

We did it again�! Just in time, we�re excited to announce the release of Grml stable version 2024.12, code-named �Adventgrenze�! (If you�re not familiar with Grml, it�s a Debian-based live system tailored for system administrators.)

This new release is built on Debian trixie, and for the first time, we�re introducing support for 64-bit ARM CPUs (arm64 architecture)!

I�m incredibly proud of the hard work that went into this release. A significant amount of behind-the-scenes effort went into reworking our infrastructure and redesigning the build process. Special thanks to Chris and Darsha – our Grml developer days in November and December were a blast!

For a detailed overview of the changes between releases 2024.02 and 2024.12, check out our official release announcement. And, as always, after a release comes the next one – exciting improvements are already in the works!

BTW: recently we also celebrated 20(!) years of Grml Releases. If you’re a Grml and or grml-zsh user, please join us in celebrating and send us a postcard!

Posted in Computer, Debian, English, Links, Open Source | Comments Off on Grml 2024.12 – codename Adventgrenze

HTU Bigband Konzert am 04.06.2024

May 28th, 2024

Am Dienstag den 4. Juni spielen wir ab 19:30 Uhr unser n�chstes HTU-Bigband-Konzert. Das Konzert findet im Innenhof der TU Graz (Alte Technik, Rechbauerstra�e 12, 8010 Graz) statt, bei Schlechtwetter geht es an der gleichen Adresse in den wunderbaren H�rsaal 2. Wir sind �ber 25 Musikerinnen und Musiker und haben ein anspruchsvolles Programm – von Swing, �ber Soul, Funk und Latin bis Pop ist alles dabei. Es gibt �ber 2 Stunden Musik vom Feinsten, und das Ganze bei freiem Eintritt.

Das Event gibt es auch auf Facebook und Instagram zum “Liken” und Weiterverteilen.

Ich freue mich schon tierisch darauf und w�rde mich �ber bekannte Gesichter freuen. Ich hoffe man sieht und h�rt sich! :-)

Posted in Allgemein, Events | Comments Off on HTU Bigband Konzert am 04.06.2024

Being a Debian Developer since 15 years

May 28th, 2024

15 years ago I became an official Debian Developer. Incredible how time flies.

Posted in Computer, English, Open Source | Comments Off on Being a Debian Developer since 15 years

Vortrag: We got hacked: Lektionen aus realen Security-Vorf�llen @ GLT24

April 7th, 2024

Auf den Grazer Linuxtagen 2024 (GLT24) war ich als Referent mit einem Vortrag zum Thema “We got hacked: Lektionen aus realen Security-Vorf�llen” vertreten. In meinem Vortrag gibt es einen Einblick in reale Security-Incidents und welche Lektionen sich aus solchen Vorf�llen mitnehmen lassen.

Es gibt den Vortrag dank des fantastischen c3voc-Teams bereits als Videomitschnitt online. Meine Vortragsfolien (2.1MB, PDF) stehen ebenfalls online zur Verf�gung. Viel Spa� beim Anschauen!

Posted in Computer, Events | Comments Off on Vortrag: We got hacked: Lektionen aus realen Security-Vorf�llen @ GLT24

Mein Lesejahr 2023

January 3rd, 2024

Ich habe auch 2023 keine Bookdumps geschrieben (zu viel Aufwand), darum gibt es auch diesmal wieder (siehe Lesejahr 2022 f�r die letzte Ausgabe) eine Art Best-Of der von mir 2023 fertig gelesenen B�cher, also jene die ich besonders lesenswert fand bzw. empfehlen m�chte (die Reihenfolge entspricht dem Foto und stellt keinerlei Reihung oder dergleichen dar):

• Das gl�cklichste Volk: Sieben Jahre bei den Pirah�-Indianern am Amazonas, Daniel L. Everett. Diese Buch ist eine Empfehlung von Khaled Hakami, der u.a. im Erkl�r-mir-die-Welt-Podcast zum Thema Erkl�r mir J�ger und Sammle zu Gast war. Das Buch ist eine gro�e Empfehlung speziell f�r all jene Leute, die sich f�r andere Kulturen interessieren. Es ist eines der horizonterweiterndsten B�cher, das ich in den letzten Jahren gelesen habe.
• Kleine Probleme, Nele Pollatschek. Der letzte Tag des Jahres, eine To-do-Liste, unerledigte Dinge und die Sehnsucht nach Ordnung. Die Autorin schreibt aus Sicht von Lars, Familienvater und Endvierziger ein unterhaltsames Buch.
• Gentleman �ber Bord, Herbert Clyde Lewis. Ein Roman aus dem Jahr 1937, in dem der Protagonist Henry bei einer Schiffsreise bei einem Missgeschick �ber Bord geht, und die restliche Besatzung des Schiffes jede Menge Ausreden f�r sich (er)findet, um das Verschwinden des Passagiers Henry zu entschuldigen. Bedeutungstief und zeitlos.
• The Undoing Project: A Friendship That Changed Our Minds, Michael Lewis. Ein Buch das die Freundschaft und Lebensweg von Daniel Kahneman (bekannt u.a. f�r das Buch “Thinking, Fast and Slow”) und Amos Tversk beleuchtet, jene Herren die u.a. f�r die Prospect Theory bekannt sind. Ich wurde auf das Buch �ber die wunderbare Rezension von Kathrin Passig aufmerksam, und mir hat das Buch voll zugesagt (auch wenn ein Lesebuddy zurecht anmerkte, dass man kein Problem mit amerikanischen Journalisten als Autor wie auch ein bisserl Drama haben sollte, mich hat beides nicht gest�rt). F�r mich war das Buch insgesamt sehr gut gemacht, es gab einige interessante Stellen und diente mir als Erinnerung, dass ich die Werke von Kahneman (wieder) mal (fertig)lesen sollte.
• Mir geht’s gut, wenn nicht heute, dann morgen, Dirk Stermann. Eine wunderbare Mischung aus ernsten Themen und Schm�h.
• Vati, Monika Helfer. Eine Fortsetzung ihrer eigenen Familiengeschichte, ich mag die schlichte und trotzdem ber�hrende Sprache.
• Eigentum, Wolf Haas. Ich habe Haas erst 2022 f�r mich entdeckt, er ist einer meiner Lieblingsautoren und ich bin seither auf dem Weg alles von ihm zu lesen. Auch Eigentum ist ein wundersch�nes Buch, in dem Haas von seiner sterbenden Mutter schreibt. Sprachk�nstler, Hilfsausdruck!
• Arbeit und Struktur, Wolfgang Herrndorf. Los ging es mit dem Schlie�en einer Bildungsl�cke: Tschick vom selbigen Autor hat mich derma�en reingezogen, dass ich auch endlich mal dessen Arbeit und Struktur angefangen habe, ein Buch das Clemens J. Setz im Podcast “Das Lesen der Anderen” empfohlen hat. Das Buch ist die Autobiografie der letzten Lebensjahre des Autors. Urspr�nglich als Blog aufgesetzt nachdem der Autor die Diagnose Hirntumor bekommen hat, wurde sein digitales Tagebuch dann in Papier- und Buchform gebracht. Man m�ge sich von dem vielleicht etwas sperrigen Titel nicht aufhalten lassen. Ein beeindruckendes, bewegendes und gro�artiges Buch.
• Die Herrenausstatterin, Mariana Leky. Ich bin ein Fan von Mariana Leky und habe �ber die letzten Jahre regelm��ig B�cher von ihr gelesen, so auch dieses wunderbare Buch �ber eine Dreiecksgeschichte.

Ich freue mich �brigens �ber Feedback, wenn jemand von euch ein Buch aufgrund dieses Beitrags hier gelesen oder selbst Lese-Empfehlungen f�r mich hat.

Posted in Allgemein, B�cher & CO | 2 Comments »

Postfix failing with “no shared cipher”

September 25th, 2023

I’m one of the few folks left who run and maintain mail servers. Recently I had major troubles receiving mails from the mail servers used by a bank, and when asking my favourite search engine, I’m clearly not the only one who ran into such an issue. Actually, I should have checked off the issue and not become a customer at that bank, but the tech nerd in me couldn’t resist getting to the bottom of the problem. Since I got it working and this might be useful for others, here we are. :)

I was trying to get an online banking account set up, but the corresponding account creation mail didn’t arrive me, at all. Looking at my mail server logs, my postfix mail server didn’t accept the mail due to:

postfix/smtpd[3319640]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:2283: postfix/smtpd[3319640]: lost connection after STARTTLS from mx01.arz.at[193.110.182.61]

Huh, what’s going on here?! Let’s increase the TLS loglevel (setting smtpd_tls_loglevel = 2) and retry. But how can I retry receiving yet another mail? Luckily, on the registration website of the bank there was a URL available, that let me request a one-time password. This triggered another mail, so I did that and managed to grab this in the logs:

postfix/smtpd[3320018]: initializing the server-side TLS engine postfix/tlsmgr[3320020]: open smtpd TLS cache btree:/var/lib/postfix/smtpd_scache postfix/tlsmgr[3320020]: tlsmgr_cache_run_event: start TLS smtpd session cache cleanup postfix/smtpd[3320018]: connect from mx01.arz.at[193.110.182.61] postfix/smtpd[3320018]: setting up TLS connection from mx01.arz.at[193.110.182.61] postfix/smtpd[3320018]: mx01.arz.at[193.110.182.61]: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH" postfix/smtpd[3320018]: SSL_accept:before SSL initialization postfix/smtpd[3320018]: SSL_accept:before SSL initialization postfix/smtpd[3320018]: SSL3 alert write:fatal:handshake failure postfix/smtpd[3320018]: SSL_accept:error in error postfix/smtpd[3320018]: SSL_accept error from mx01.arz.at[193.110.182.61]: -1 postfix/smtpd[3320018]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:2283: postfix/smtpd[3320018]: lost connection after STARTTLS from mx01.arz.at[193.110.182.61] postfix/smtpd[3320018]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 starttls=0/1 commands=1/2 postfix/smtpd[3320018]: connect from mx01.arz.at[193.110.182.61] postfix/smtpd[3320018]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2

Ok, so this TLS cipher list “aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH” looked like the tls_medium_cipherlist setting in postfix, but which ciphers might we expect? Let’s see what their SMTP server would speak to us:

% testssl --cipher-per-proto -t=smtp mx01.arz.at:25 [...] Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (IANA/RFC)

SSLv2 SSLv3 TLS 1 TLS 1.1 TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA TLS 1.3

Looks like a very small subset of ciphers, and they don’t seem to be talking TLS v1.3 at all? Not great. :(

A nice web service to verify the situation from another point of view is checktls, which also confirmed this:

[000.705] <-- 220 2.0.0 Ready to start TLS [000.705] STARTTLS command works on this server [001.260] Connection converted to SSL SSLVersion in use: TLSv1_2 Cipher in use: ECDHE-RSA-AES256-GCM-SHA384 Perfect Forward Secrecy: yes Session Algorithm in use: Curve P-256 DHE(256 bits) Certificate #1 of 3 (sent by MX): Cert VALIDATED: ok Cert Hostname VERIFIED (mx01.arz.at = .arz.at | DNS:.arz.at | DNS:arz.at) [...] [001.517] TLS successfully started on this server

I got distracted by some other work, and when coming back to this problem, the one-time password procedure no longer worked, as the password reset URL was no longer valid. :( I managed to find the underlying URL, and with some web developer tools tinkering I could still use the website to let me trigger sending further one-time password mails, phew.

Let’s continue, so my mail server was running Debian/bullseye with postfix v3.5.18-0+deb11u1 and openssl v1.1.1n-0+deb11u5, let’s see what it offers:

% testssl --cipher-per-proto -t=smtp mail.example.com:25 [...] Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (IANA/RFC)

SSLv2 SSLv3 TLS 1 xc00a ECDHE-ECDSA-AES256-SHA ECDH 253 AES 256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA xc019 AECDH-AES256-SHA ECDH 253 AES 256 TLS_ECDH_anon_WITH_AES_256_CBC_SHA x3a ADH-AES256-SHA DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA x89 ADH-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA xc009 ECDHE-ECDSA-AES128-SHA ECDH 253 AES 128 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA xc018 AECDH-AES128-SHA ECDH 253 AES 128 TLS_ECDH_anon_WITH_AES_128_CBC_SHA x34 ADH-AES128-SHA DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA x9b ADH-SEED-SHA DH 2048 SEED 128 TLS_DH_anon_WITH_SEED_CBC_SHA x46 ADH-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA TLS 1.1 xc00a ECDHE-ECDSA-AES256-SHA ECDH 253 AES 256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA xc019 AECDH-AES256-SHA ECDH 253 AES 256 TLS_ECDH_anon_WITH_AES_256_CBC_SHA x3a ADH-AES256-SHA DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA x89 ADH-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA xc009 ECDHE-ECDSA-AES128-SHA ECDH 253 AES 128 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA xc018 AECDH-AES128-SHA ECDH 253 AES 128 TLS_ECDH_anon_WITH_AES_128_CBC_SHA x34 ADH-AES128-SHA DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA x9b ADH-SEED-SHA DH 2048 SEED 128 TLS_DH_anon_WITH_SEED_CBC_SHA x46 ADH-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA TLS 1.2 xc02c ECDHE-ECDSA-AES256-GCM-SHA384 ECDH 253 AESGCM 256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 xc024 ECDHE-ECDSA-AES256-SHA384 ECDH 253 AES 256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 xc00a ECDHE-ECDSA-AES256-SHA ECDH 253 AES 256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA xcca9 ECDHE-ECDSA-CHACHA20-POLY1305 ECDH 253 ChaCha20 256 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 xc0af ECDHE-ECDSA-AES256-CCM8 ECDH 253 AESCCM8 256 TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8 xc0ad ECDHE-ECDSA-AES256-CCM ECDH 253 AESCCM 256 TLS_ECDHE_ECDSA_WITH_AES_256_CCM xc073 ECDHE-ECDSA-CAMELLIA256-SHA384 ECDH 253 Camellia 256 TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384 xc019 AECDH-AES256-SHA ECDH 253 AES 256 TLS_ECDH_anon_WITH_AES_256_CBC_SHA xa7 ADH-AES256-GCM-SHA384 DH 2048 AESGCM 256 TLS_DH_anon_WITH_AES_256_GCM_SHA384 x6d ADH-AES256-SHA256 DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA256 x3a ADH-AES256-SHA DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA xc5 ADH-CAMELLIA256-SHA256 DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA256 x89 ADH-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA xc05d ECDHE-ECDSA-ARIA256-GCM-SHA384 ECDH 253 ARIAGCM 256 TLS_ECDHE_ECDSA_WITH_ARIA_256_GCM_SHA384 xc02b ECDHE-ECDSA-AES128-GCM-SHA256 ECDH 253 AESGCM 128 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 xc023 ECDHE-ECDSA-AES128-SHA256 ECDH 253 AES 128 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 xc009 ECDHE-ECDSA-AES128-SHA ECDH 253 AES 128 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA xc0ae ECDHE-ECDSA-AES128-CCM8 ECDH 253 AESCCM8 128 TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 xc0ac ECDHE-ECDSA-AES128-CCM ECDH 253 AESCCM 128 TLS_ECDHE_ECDSA_WITH_AES_128_CCM xc072 ECDHE-ECDSA-CAMELLIA128-SHA256 ECDH 253 Camellia 128 TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256 xc018 AECDH-AES128-SHA ECDH 253 AES 128 TLS_ECDH_anon_WITH_AES_128_CBC_SHA xa6 ADH-AES128-GCM-SHA256 DH 2048 AESGCM 128 TLS_DH_anon_WITH_AES_128_GCM_SHA256 x6c ADH-AES128-SHA256 DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA256 x34 ADH-AES128-SHA DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA xbf ADH-CAMELLIA128-SHA256 DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA256 x9b ADH-SEED-SHA DH 2048 SEED 128 TLS_DH_anon_WITH_SEED_CBC_SHA x46 ADH-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA xc05c ECDHE-ECDSA-ARIA128-GCM-SHA256 ECDH 253 ARIAGCM 128 TLS_ECDHE_ECDSA_WITH_ARIA_128_GCM_SHA256 TLS 1.3 x1302 TLS_AES_256_GCM_SHA384 ECDH 253 AESGCM 256 TLS_AES_256_GCM_SHA384 x1303 TLS_CHACHA20_POLY1305_SHA256 ECDH 253 ChaCha20 256 TLS_CHACHA20_POLY1305_SHA256 x1301 TLS_AES_128_GCM_SHA256 ECDH 253 AESGCM 128 TLS_AES_128_GCM_SHA256

Not so bad, but sadly no overlap with any of the ciphers that mx01.arz.at offers.

What about disabling STARTTLS for the mx01.arz.at (+ mx02.arz.at being another one used by the relevant domain) mail servers when talking to mine? Let’s try that:

% sudo postconf -nf smtpd_discard_ehlo_keyword_address_maps smtpd_discard_ehlo_keyword_address_maps = hash:/etc/postfix/smtpd_discard_ehlo_keywords

% cat /etc/postfix/smtpd_discard_ehlo_keywords

disable starttls for mx01.arz.at / mx02.arz.at:

193.110.182.61 starttls 193.110.182.62 starttls

But the remote mail server doesn’t seem to send mails without TLS:

postfix/smtpd[4151799]: connect from mx01.arz.at[193.110.182.61] postfix/smtpd[4151799]: discarding EHLO keywords: STARTTLS postfix/smtpd[4151799]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2

Let’s verify this further, but without fiddling with the main mail server too much. We can add a dedicated service to postfix (see serverfault), and run it in verbose mode, to get more detailled logging:

% sudo postconf -Mf [...] 10025 inet n - - - - smtpd -o syslog_name=postfix/smtpd/badstarttls -o smtpd_tls_security_level=none -o smtpd_helo_required=yes -o smtpd_helo_restrictions=pcre:/etc/postfix/helo_badstarttls_allow,reject -v

[...]

% cat /etc/postfix/helo_badstarttls_allow /mx01.arz.at/ OK /mx02.arz.at/ OK /193.110.182.61/ OK /193.110.182.62/ OK

We redirect the traffic from mx01.arz.at + mx02.arz.at towards our new postfix service, listening on port 10025:

% sudo iptables -t nat -A PREROUTING -p tcp -s 193.110.182.61 --dport 25 -j REDIRECT --to-port 10025 % sudo iptables -t nat -A PREROUTING -p tcp -s 193.110.182.62 --dport 25 -j REDIRECT --to-port 10025

With this setup we get very detailed logging, and it seems to confirm our suspicion that the mail server doesn’t want to talk unencrypted with us:

[...] postfix/smtpd/badstarttls/smtpd[3491900]: connect from mx01.arz.at[193.110.182.61] [...] postfix/smtpd/badstarttls/smtpd[3491901]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2 postfix/smtpd/badstarttls/smtpd[3491901]: master_notify: status 1 postfix/smtpd/badstarttls/smtpd[3491901]: connection closed [...]

Let’s step back and revert those changes, back to our original postfix setup. Might the problem be related to our Let’s Encrypt certificate? Let’s see what we have:

% echo QUIT | openssl s_client -connect mail.example.com:25 -starttls [...] issuer=C = US, O = Let's Encrypt, CN = R3

No client certificate CA names sent Peer signing digest: SHA384 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits

SSL handshake has read 4455 bytes and written 427 bytes Verification: OK

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 384 bit [...]

We have an ECDSA based certificate, what about switching to RSA instead? Thanks to the wonderful dehydrated, this is as easy as:

% echo KEY_ALGO=rsa > certs/mail.example.com/config % ./dehydrated -c --domain mail.example.com --force % sudo systemctl reload postfix

With switching to RSA type key we get:

% echo QUIT | openssl s_client -connect mail.example.com:25 -starttls smtp CONNECTED(00000003) [...] issuer=C = US, O = Let's Encrypt, CN = R3

No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits

SSL handshake has read 5295 bytes and written 427 bytes Verification: OK

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 4096 bit

Which ciphers do we offer now? Let’s check:

% testssl --cipher-per-proto -t=smtp mail.example.com:25 [...] Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (IANA/RFC)

SSLv2 SSLv3 TLS 1 xc014 ECDHE-RSA-AES256-SHA ECDH 253 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x39 DHE-RSA-AES256-SHA DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA x88 DHE-RSA-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA xc019 AECDH-AES256-SHA ECDH 253 AES 256 TLS_ECDH_anon_WITH_AES_256_CBC_SHA x3a ADH-AES256-SHA DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA x89 ADH-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 253 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x33 DHE-RSA-AES128-SHA DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA x9a DHE-RSA-SEED-SHA DH 2048 SEED 128 TLS_DHE_RSA_WITH_SEED_CBC_SHA x45 DHE-RSA-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA xc018 AECDH-AES128-SHA ECDH 253 AES 128 TLS_ECDH_anon_WITH_AES_128_CBC_SHA x34 ADH-AES128-SHA DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA x9b ADH-SEED-SHA DH 2048 SEED 128 TLS_DH_anon_WITH_SEED_CBC_SHA x46 ADH-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA x96 SEED-SHA RSA SEED 128 TLS_RSA_WITH_SEED_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA TLS 1.1 xc014 ECDHE-RSA-AES256-SHA ECDH 253 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x39 DHE-RSA-AES256-SHA DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA x88 DHE-RSA-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA xc019 AECDH-AES256-SHA ECDH 253 AES 256 TLS_ECDH_anon_WITH_AES_256_CBC_SHA x3a ADH-AES256-SHA DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA x89 ADH-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc013 ECDHE-RSA-AES128-SHA ECDH 253 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x33 DHE-RSA-AES128-SHA DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA x9a DHE-RSA-SEED-SHA DH 2048 SEED 128 TLS_DHE_RSA_WITH_SEED_CBC_SHA x45 DHE-RSA-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA xc018 AECDH-AES128-SHA ECDH 253 AES 128 TLS_ECDH_anon_WITH_AES_128_CBC_SHA x34 ADH-AES128-SHA DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA x9b ADH-SEED-SHA DH 2048 SEED 128 TLS_DH_anon_WITH_SEED_CBC_SHA x46 ADH-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA x96 SEED-SHA RSA SEED 128 TLS_RSA_WITH_SEED_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 253 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 253 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xc014 ECDHE-RSA-AES256-SHA ECDH 253 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x9f DHE-RSA-AES256-GCM-SHA384 DH 2048 AESGCM 256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 253 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xccaa DHE-RSA-CHACHA20-POLY1305 DH 2048 ChaCha20 256 TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xc0a3 DHE-RSA-AES256-CCM8 DH 2048 AESCCM8 256 TLS_DHE_RSA_WITH_AES_256_CCM_8 xc09f DHE-RSA-AES256-CCM DH 2048 AESCCM 256 TLS_DHE_RSA_WITH_AES_256_CCM x6b DHE-RSA-AES256-SHA256 DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 x39 DHE-RSA-AES256-SHA DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA xc077 ECDHE-RSA-CAMELLIA256-SHA384 ECDH 253 Camellia 256 TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 xc4 DHE-RSA-CAMELLIA256-SHA256 DH 2048 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256 x88 DHE-RSA-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA xc019 AECDH-AES256-SHA ECDH 253 AES 256 TLS_ECDH_anon_WITH_AES_256_CBC_SHA xa7 ADH-AES256-GCM-SHA384 DH 2048 AESGCM 256 TLS_DH_anon_WITH_AES_256_GCM_SHA384 x6d ADH-AES256-SHA256 DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA256 x3a ADH-AES256-SHA DH 2048 AES 256 TLS_DH_anon_WITH_AES_256_CBC_SHA xc5 ADH-CAMELLIA256-SHA256 DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA256 x89 ADH-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 xc0a1 AES256-CCM8 RSA AESCCM8 256 TLS_RSA_WITH_AES_256_CCM_8 xc09d AES256-CCM RSA AESCCM 256 TLS_RSA_WITH_AES_256_CCM x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc0 CAMELLIA256-SHA256 RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA xc051 ARIA256-GCM-SHA384 RSA ARIAGCM 256 TLS_RSA_WITH_ARIA_256_GCM_SHA384 xc053 DHE-RSA-ARIA256-GCM-SHA384 DH 2048 ARIAGCM 256 TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384 xc061 ECDHE-ARIA256-GCM-SHA384 ECDH 253 ARIAGCM 256 TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 253 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 253 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 xc013 ECDHE-RSA-AES128-SHA ECDH 253 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x9e DHE-RSA-AES128-GCM-SHA256 DH 2048 AESGCM 128 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 xc0a2 DHE-RSA-AES128-CCM8 DH 2048 AESCCM8 128 TLS_DHE_RSA_WITH_AES_128_CCM_8 xc09e DHE-RSA-AES128-CCM DH 2048 AESCCM 128 TLS_DHE_RSA_WITH_AES_128_CCM xc0a0 AES128-CCM8 RSA AESCCM8 128 TLS_RSA_WITH_AES_128_CCM_8 xc09c AES128-CCM RSA AESCCM 128 TLS_RSA_WITH_AES_128_CCM x67 DHE-RSA-AES128-SHA256 DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 x33 DHE-RSA-AES128-SHA DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA xc076 ECDHE-RSA-CAMELLIA128-SHA256 ECDH 253 Camellia 128 TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 xbe DHE-RSA-CAMELLIA128-SHA256 DH 2048 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 x9a DHE-RSA-SEED-SHA DH 2048 SEED 128 TLS_DHE_RSA_WITH_SEED_CBC_SHA x45 DHE-RSA-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA xc018 AECDH-AES128-SHA ECDH 253 AES 128 TLS_ECDH_anon_WITH_AES_128_CBC_SHA xa6 ADH-AES128-GCM-SHA256 DH 2048 AESGCM 128 TLS_DH_anon_WITH_AES_128_GCM_SHA256 x6c ADH-AES128-SHA256 DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA256 x34 ADH-AES128-SHA DH 2048 AES 128 TLS_DH_anon_WITH_AES_128_CBC_SHA xbf ADH-CAMELLIA128-SHA256 DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA256 x9b ADH-SEED-SHA DH 2048 SEED 128 TLS_DH_anon_WITH_SEED_CBC_SHA x46 ADH-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA xba CAMELLIA128-SHA256 RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 x96 SEED-SHA RSA SEED 128 TLS_RSA_WITH_SEED_CBC_SHA x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA xc050 ARIA128-GCM-SHA256 RSA ARIAGCM 128 TLS_RSA_WITH_ARIA_128_GCM_SHA256 xc052 DHE-RSA-ARIA128-GCM-SHA256 DH 2048 ARIAGCM 128 TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256 xc060 ECDHE-ARIA128-GCM-SHA256 ECDH 253 ARIAGCM 128 TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256 TLS 1.3 x1302 TLS_AES_256_GCM_SHA384 ECDH 253 AESGCM 256 TLS_AES_256_GCM_SHA384 x1303 TLS_CHACHA20_POLY1305_SHA256 ECDH 253 ChaCha20 256 TLS_CHACHA20_POLY1305_SHA256 x1301 TLS_AES_128_GCM_SHA256 ECDH 253 AESGCM 128 TLS_AES_128_GCM_SHA256

With switching our SSL certificate to RSA, we gained around 51 new cipher options, amongst them being ones that also mx01.arz.at claimed to support.

FTR, the result from above is what you get with the default settings for postfix v3.5.18, being:

smtpd_tls_ciphers = medium smtpd_tls_mandatory_ciphers = medium smtpd_tls_mandatory_exclude_ciphers = smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3

But the delay between triggering the password reset mail and getting a mail server connect was getting bigger and bigger. Therefore while waiting for the next mail to arrive, I decided to capture the network traffic, to be able to look further into this if it should continue to be failing:

% sudo tshark -n -i eth0 -s 65535 -w arz.pcap -f "host 193.110.182.61 or host 193.110.182.62"

A few hours later the mail server connected again, and the mail went through!

postfix/smtpd[4162835]: connect from mx01.arz.at[193.110.182.61] postfix/smtpd[4162835]: Anonymous TLS connection established from mx01.arz.at[193.110.182.61]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) postfix/smtpd[4162835]: E50D6401E6: client=mx01.arz.at[193.110.182.61] postfix/smtpd[4162835]: disconnect from mx01.arz.at[193.110.182.61] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7

Now also having the captured network traffic, we can check the details there:

[...] % tshark -o smtp.decryption:true -r arz.pcap 1 0.000000000 193.110.182.61 → 203.0.113.42 TCP 74 24699 → 25 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=2261106119 TSecr=0 WS=128 2 0.000042827 203.0.113.42 → 193.110.182.61 TCP 74 25 → 24699 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=3233422181 TSecr=2261106119 WS=128 3 0.020719269 193.110.182.61 → 203.0.113.42 TCP 66 24699 → 25 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=2261106139 TSecr=3233422181 4 0.022883259 203.0.113.42 → 193.110.182.61 SMTP 96 S: 220 mail.example.com ESMTP 5 0.043682626 193.110.182.61 → 203.0.113.42 TCP 66 24699 → 25 [ACK] Seq=1 Ack=31 Win=29312 Len=0 TSval=2261106162 TSecr=3233422203 6 0.043799047 193.110.182.61 → 203.0.113.42 SMTP 84 C: EHLO mx01.arz.at 7 0.043811363 203.0.113.42 → 193.110.182.61 TCP 66 25 → 24699 [ACK] Seq=31 Ack=19 Win=65280 Len=0 TSval=3233422224 TSecr=2261106162 8 0.043898412 203.0.113.42 → 193.110.182.61 SMTP 253 S: 250-mail.example.com | PIPELINING | SIZE 20240000 | VRFY | ETRN | AUTH PLAIN | AUTH=PLAIN | ENHANCEDSTATUSCODES | 8BITMIME | DSN | SMTPUTF8 | CHUNKING 9 0.064625499 193.110.182.61 → 203.0.113.42 SMTP 72 C: QUIT 10 0.064750257 203.0.113.42 → 193.110.182.61 SMTP 81 S: 221 2.0.0 Bye 11 0.064760200 203.0.113.42 → 193.110.182.61 TCP 66 25 → 24699 [FIN, ACK] Seq=233 Ack=25 Win=65280 Len=0 TSval=3233422245 TSecr=2261106183 12 0.085573715 193.110.182.61 → 203.0.113.42 TCP 66 24699 → 25 [FIN, ACK] Seq=25 Ack=234 Win=30336 Len=0 TSval=2261106204 TSecr=3233422245 13 0.085610229 203.0.113.42 → 193.110.182.61 TCP 66 25 → 24699 [ACK] Seq=234 Ack=26 Win=65280 Len=0 TSval=3233422266 TSecr=2261106204 14 1799.888108373 193.110.182.61 → 203.0.113.42 TCP 74 10330 → 25 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=2262906007 TSecr=0 WS=128 15 1799.888161311 203.0.113.42 → 193.110.182.61 TCP 74 25 → 10330 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=3235222069 TSecr=2262906007 WS=128 16 1799.909030335 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=2262906028 TSecr=3235222069 17 1799.956621011 203.0.113.42 → 193.110.182.61 SMTP 96 S: 220 mail.example.com ESMTP 18 1799.977229656 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=1 Ack=31 Win=29312 Len=0 TSval=2262906096 TSecr=3235222137 19 1799.977229698 193.110.182.61 → 203.0.113.42 SMTP 84 C: EHLO mx01.arz.at 20 1799.977266759 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=31 Ack=19 Win=65280 Len=0 TSval=3235222158 TSecr=2262906096 21 1799.977351663 203.0.113.42 → 193.110.182.61 SMTP 267 S: 250-mail.example.com | PIPELINING | SIZE 20240000 | VRFY | ETRN | STARTTLS | AUTH PLAIN | AUTH=PLAIN | ENHANCEDSTATUSCODES | 8BITMIME | DSN | SMTPUTF8 | CHUNKING 22 1800.011494861 193.110.182.61 → 203.0.113.42 SMTP 76 C: STARTTLS 23 1800.011589267 203.0.113.42 → 193.110.182.61 SMTP 96 S: 220 2.0.0 Ready to start TLS 24 1800.032812294 193.110.182.61 → 203.0.113.42 TLSv1 223 Client Hello 25 1800.032987264 203.0.113.42 → 193.110.182.61 TLSv1.2 2962 Server Hello 26 1800.032995513 203.0.113.42 → 193.110.182.61 TCP 1266 25 → 10330 [PSH, ACK] Seq=3158 Ack=186 Win=65152 Len=1200 TSval=3235222214 TSecr=2262906151 [TCP segment of a reassembled PDU] 27 1800.053546755 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=186 Ack=3158 Win=36096 Len=0 TSval=2262906172 TSecr=3235222214 28 1800.092852469 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=186 Ack=4358 Win=39040 Len=0 TSval=2262906212 TSecr=3235222214 29 1800.092892905 203.0.113.42 → 193.110.182.61 TLSv1.2 900 Certificate, Server Key Exchange, Server Hello Done 30 1800.113546769 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=186 Ack=5192 Win=41856 Len=0 TSval=2262906232 TSecr=3235222273 31 1800.114763363 193.110.182.61 → 203.0.113.42 TLSv1.2 192 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 32 1800.115000416 203.0.113.42 → 193.110.182.61 TLSv1.2 117 Change Cipher Spec, Encrypted Handshake Message 33 1800.136070200 193.110.182.61 → 203.0.113.42 TLSv1.2 113 Application Data 34 1800.136155526 203.0.113.42 → 193.110.182.61 TLSv1.2 282 Application Data 35 1800.158854473 193.110.182.61 → 203.0.113.42 TLSv1.2 162 Application Data 36 1800.159254794 203.0.113.42 → 193.110.182.61 TLSv1.2 109 Application Data 37 1800.180286407 193.110.182.61 → 203.0.113.42 TLSv1.2 144 Application Data 38 1800.223005960 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5502 Ack=533 Win=65152 Len=0 TSval=3235222404 TSecr=2262906299 39 1802.230300244 203.0.113.42 → 193.110.182.61 TLSv1.2 146 Application Data 40 1802.251994333 193.110.182.61 → 203.0.113.42 TCP 2962 [TCP segment of a reassembled PDU] 41 1802.252034015 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5582 Ack=3429 Win=63616 Len=0 TSval=3235224433 TSecr=2262908371 42 1802.252279083 193.110.182.61 → 203.0.113.42 TLSv1.2 1295 Application Data 43 1802.252288316 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5582 Ack=4658 Win=64128 Len=0 TSval=3235224433 TSecr=2262908371 44 1802.272816060 193.110.182.61 → 203.0.113.42 TLSv1.2 833 Application Data, Application Data 45 1802.272827542 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5582 Ack=5425 Win=64128 Len=0 TSval=3235224453 TSecr=2262908392 46 1802.338807683 203.0.113.42 → 193.110.182.61 TLSv1.2 131 Application Data 47 1802.398968611 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=5425 Ack=5647 Win=44800 Len=0 TSval=2262908518 TSecr=3235224519 48 1863.257457500 193.110.182.61 → 203.0.113.42 TLSv1.2 101 Application Data 49 1863.257495688 203.0.113.42 → 193.110.182.61 TCP 66 25 → 10330 [ACK] Seq=5647 Ack=5460 Win=64128 Len=0 TSval=3235285438 TSecr=2262969376 50 1863.257654942 203.0.113.42 → 193.110.182.61 TLSv1.2 110 Application Data 51 1863.257721010 203.0.113.42 → 193.110.182.61 TLSv1.2 97 Encrypted Alert 52 1863.278242216 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [ACK] Seq=5460 Ack=5691 Win=44800 Len=0 TSval=2262969397 TSecr=3235285438 53 1863.278464176 193.110.182.61 → 203.0.113.42 TCP 66 10330 → 25 [RST, ACK] Seq=5460 Ack=5723 Win=44800 Len=0 TSval=2262969397 TSecr=3235285438

% tshark -O tls -r arz.pcap [...] Transport Layer Security TLSv1 Record Layer: Handshake Protocol: Client Hello Content Type: Handshake (22) Version: TLS 1.0 (0x0301) Length: 152 Handshake Protocol: Client Hello Handshake Type: Client Hello (1) Length: 148 Version: TLS 1.2 (0x0303) Random: 4575d1e7c93c09a564edc00b8b56ea6f5d826f8cfe78eb980c451a70a9c5123f GMT Unix Time: Dec 5, 2006 21:09:11.000000000 CET Random Bytes: c93c09a564edc00b8b56ea6f5d826f8cfe78eb980c451a70a9c5123f Session ID Length: 0 Cipher Suites Length: 26 Cipher Suites (13 suites) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d) Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c) Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d) Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c) Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035) Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f) Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff) [...] Transport Layer Security TLSv1.2 Record Layer: Handshake Protocol: Server Hello Content Type: Handshake (22) Version: TLS 1.2 (0x0303) Length: 89 Handshake Protocol: Server Hello Handshake Type: Server Hello (2) Length: 85 Version: TLS 1.2 (0x0303) Random: cf2ed24e3300e95e5f56023bf8b4e5904b862bb2ed8a5796444f574e47524401 GMT Unix Time: Feb 23, 2080 23:16:46.000000000 CET Random Bytes: 3300e95e5f56023bf8b4e5904b862bb2ed8a5796444f574e47524401 Session ID Length: 32 Session ID: 63d041b126ecebf857d685abd9d4593c46a3672e1ad76228f3eacf2164f86fb9 Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) [...]

In this network dump we see what cipher suites are offered, and the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 here is the Cipher Suite Name in IANA/RFC speak. Whis corresponds to the ECDHE-RSA-AES256-GCM-SHA384 in openssl speak (see Mozilla’s Mozilla’s cipher suite correspondence table), which we also saw in the postfix log.

Mission accomplished! :)

Now, if we’re interested in avoiding certain ciphers and increase security level, we can e.g. get rid of the SEED, CAMELLIA and all anonymous ciphers, and could accept only TLS v1.2 + v1.3, by further adjusting postfix’s main.cf:

smtpd_tls_ciphers = high smtpd_tls_exclude_ciphers = aNULL CAMELLIA smtpd_tls_mandatory_ciphers = high smtpd_tls_mandatory_protocols = TLSv1.2 TLSv1.3 smtpd_tls_protocols = TLSv1.2 TLSv1.3

Which would then gives us:

% testssl --cipher-per-proto -t=smtp mail.example.com:25 [...]

Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (IANA/RFC)

SSLv2 SSLv3 TLS 1 TLS 1.1 TLS 1.2 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 253 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 xc028 ECDHE-RSA-AES256-SHA384 ECDH 253 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 xc014 ECDHE-RSA-AES256-SHA ECDH 253 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA x9f DHE-RSA-AES256-GCM-SHA384 DH 2048 AESGCM 256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 253 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xccaa DHE-RSA-CHACHA20-POLY1305 DH 2048 ChaCha20 256 TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 xc0a3 DHE-RSA-AES256-CCM8 DH 2048 AESCCM8 256 TLS_DHE_RSA_WITH_AES_256_CCM_8 xc09f DHE-RSA-AES256-CCM DH 2048 AESCCM 256 TLS_DHE_RSA_WITH_AES_256_CCM x6b DHE-RSA-AES256-SHA256 DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 x39 DHE-RSA-AES256-SHA DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384 xc0a1 AES256-CCM8 RSA AESCCM8 256 TLS_RSA_WITH_AES_256_CCM_8 xc09d AES256-CCM RSA AESCCM 256 TLS_RSA_WITH_AES_256_CCM x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA xc051 ARIA256-GCM-SHA384 RSA ARIAGCM 256 TLS_RSA_WITH_ARIA_256_GCM_SHA384 xc053 DHE-RSA-ARIA256-GCM-SHA384 DH 2048 ARIAGCM 256 TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384 xc061 ECDHE-ARIA256-GCM-SHA384 ECDH 253 ARIAGCM 256 TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 253 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 xc027 ECDHE-RSA-AES128-SHA256 ECDH 253 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 xc013 ECDHE-RSA-AES128-SHA ECDH 253 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA x9e DHE-RSA-AES128-GCM-SHA256 DH 2048 AESGCM 128 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 xc0a2 DHE-RSA-AES128-CCM8 DH 2048 AESCCM8 128 TLS_DHE_RSA_WITH_AES_128_CCM_8 xc09e DHE-RSA-AES128-CCM DH 2048 AESCCM 128 TLS_DHE_RSA_WITH_AES_128_CCM xc0a0 AES128-CCM8 RSA AESCCM8 128 TLS_RSA_WITH_AES_128_CCM_8 xc09c AES128-CCM RSA AESCCM 128 TLS_RSA_WITH_AES_128_CCM x67 DHE-RSA-AES128-SHA256 DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 x33 DHE-RSA-AES128-SHA DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA xc050 ARIA128-GCM-SHA256 RSA ARIAGCM 128 TLS_RSA_WITH_ARIA_128_GCM_SHA256 xc052 DHE-RSA-ARIA128-GCM-SHA256 DH 2048 ARIAGCM 128 TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256 xc060 ECDHE-ARIA128-GCM-SHA256 ECDH 253 ARIAGCM 128 TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256 TLS 1.3 x1302 TLS_AES_256_GCM_SHA384 ECDH 253 AESGCM 256 TLS_AES_256_GCM_SHA384 x1303 TLS_CHACHA20_POLY1305_SHA256 ECDH 253 ChaCha20 256 TLS_CHACHA20_POLY1305_SHA256 x1301 TLS_AES_128_GCM_SHA256 ECDH 253 AESGCM 128 TLS_AES_128_GCM_SHA256

Don’t forget to also adjust the smpt_tls_* accordingly (for your sending side). For further information see the Postfix TLS Support documentation. Also check out options like tls_ssl_options (setting it to e.g. NO_COMPRESSION) and tls_preempt_cipherlist (setting it to yes would prefer the servers’ order of ciphers over clients).

Conclusions:

• no matter what you change in your mail server settings, be aware that the type of your SSL certificate also matters for what ciphers are offered and used
• there are mail servers out there that don’t support SSL certificates with ECDSA, using RSA for those ensure better compatibility (nowadays postfix supports parallel usage of ECDSA and RSA keys BTW, check out the smtpd_tls_eccert_file + smtpd_tls_eckey_file options)
• testssl is a very useful tool, especially with its –cipher-per-proto -t=smtp option to check SMTP servers
• if you’re uncertain what’s going on, consider capturing network data (tshark/tcpdump/… are your friends)
• review your postfix configuration and logs every now and then :)

Posted in Computer, Debian, English | 1 Comment »

What to expect from Debian/bookworm #newinbookworm

June 11th, 2023

Debian v12 with codename bookworm was released as new stable release on 10th of June 2023. Similar to what we had with #newinbullseye and previous releases, now it’s time for #newinbookworm!

I was the driving force at several of my customers to be well prepared for bookworm. As usual with major upgrades, there are some things to be aware of, and hereby I’m starting my public notes on bookworm that might be worth also for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.

Further readings

As usual start at the official Debian release notes, make sure to especially go through What’s new in Debian 12 + Issues to be aware of for bookworm.

Package versions

As a starting point, let’s look at some selected packages and their versions in bullseye vs. bookworm as of 2023-02-10 (mainly having amd64 in mind):

Linux Kernel

The bookworm release ships a Linux kernel based on version 6.1, whereas bullseye shipped kernel 5.10. As usual there are plenty of changes in the kernel area, including better hardware support, and this might warrant a separate blog entry, but to highlight some changes:

• a.out support is gone
• initial support for Rust
• lots of io_uring related improvements
• lots of BPF improvements
• support for Intel Software Guard eXtensions (SGX)
• ID mapping for mounted filesystems
• unprivileged overlayfs mounts and ID mapping in overlayfs
• NFS re-exporting support
• eager NFS writes with new writes=lazy/eager/wait mount options
• Landlock security module
• initial support for Apple M2
• new misc cgroup and new cgroup.kill file
• new memfd_secret(2) system call
• new NTFS file system implementation
• file system monitoring with fanotify
• lots of improvements around perf, including the new daemon, kwork and iostat commands, and JSON output option for stat

See Kernelnewbies.org for further changes between kernel versions.

Configuration management

puppet‘s upstream sadly still doesn’t provide packages for bookworm (see PA-4995), though Debian provides puppet-agent and puppetserver packages, and even puppetdb is back again, see release notes for further information.

ansible is also available and made it with version 2.14 into bookworm.

Prometheus stack

Prometheus server was updated from v2.24.1 to v2.42.0 and all the exporters that got shipped with bullseye are still around (in more recent versions of course).

Virtualization

docker (v20.10.24), ganeti (v3.0.2-3), libvirt (v9.0.0-4), lxc (v5.0.2-1), podman (v4.3.1), openstack (Zed), qemu/kvm (v7.2), xen (v4.17.1) are all still around.

Vagrant is available in version 2.3.4, also Vagrant upstream provides their packages for bookworm already.

If you’re relying on VirtualBox, be aware that upstream doesn’t provide packages for bookworm yet (see ticket 21524), but thankfully version 7.0.8-dfsg-2 is available from Debian/unstable (as of 2023-06-10) (VirtualBox isn’t shipped with stable releases since quite some time due to lack of cooperation from upstream on security support for older releases, see #794466).

rsync

rsync was updated from v3.2.3 to v3.2.7, and we got a few new options:

• --fsync: fsync every written file
• --old-dirs: works like –dirs when talking to old rsync
• --old-args: disable the modern arg-protection idiom
• --secluded-args, -s: use the protocol to safely send the args (replaces –protect-args option)
• --trust-sender: trust the remote sender’s file list

OpenSSH

OpenSSH was updated from v8.4p1 to v9.2p1, so if you’re interested in all the changes, check out the release notes between those version (8.5, 8.6, 8.7, 8.8, 8.9, 9.0, 9.1 + 9.2). Let’s highlight some notable new features:

• new system for restricting forwarding and use of keys added to ssh-agent(1), see SSH agent restriction for details)
• switched scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default (see release notes for v9.0 for details
• ssh(1): when prompting the user to accept a new hostkey, display any other host names/addresses already associated with the key
• ssh(1): allow UserKnownHostsFile=none to indicate that no known_hosts file should be used to identify host keys
• ssh(1): add a ssh_config KnownHostsCommand option that allows the client to obtain known_hosts data from a command in addition to the usual files
• ssh(1), sshd(8): add a RequiredRSASize directive to set a minimum RSA key length
• ssh(1): add a “host” line to the output of ssh -G showing the original hostname argument
• ssh-keygen -A (generate all default host key types) will no longer generate DSA keys
• ssh-keyscan(1): allow scanning of complete CIDR address ranges, e.g. ssh-keyscan 192.168.0.0/24

One important change you might wanna be aware of is that as of OpenSSH v8.8, RSA signatures using the SHA-1 hash algorithm got disabled by default, but RSA/SHA-256/512 AKA RSA-SHA2 gets used instead. OpenSSH has supported RFC8332 RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys will automatically use the stronger algorithm where possible. A good overview is also available at SSH: Signature Algorithm ssh-rsa Error.

Now tools/libraries not supporting RSA-SHA2 fail to connect to OpenSSH as present in bookworm. For example python3-paramiko v2.7.2-1 as present in bullseye doesn’t support RSA-SHA2. It tries to connect using the deprecated RSA-SHA-1, which is no longer offered by default with OpenSSH as present in bookworm, and then fails. Support for RSA/SHA-256/512 signatures in Paramiko was requested e.g. at #1734, and eventually got added to Paramiko and in the end the change made it into Paramiko versions >=2.9.0. Paramiko in bookworm works fine, and a backport by rebuilding the python3-paramiko package from bookworm for bullseye solves the problem (BTDT).

Misc unsorted

• new non-free-firmware component/repository (see Debian Wiki for details)
• support only the merged-usr root filesystem layout (see Debian Wiki for details)
• the asterisk package didn’t make it into bookworm (see #1031046)
• e2fsprogs: the breaking change related to metadata_csum_seed and orphan_file (see #1031325) was reverted with v1.47.0-2 for bookworm (also see #1031622 + #1030939)
• rsnapshot is back again (see #986709)
• crmadmin of pacemaker no longer interprets the timeout option (-t/–timeout) in milliseconds (as it used to be until v2.0.5), but as of v2.1.0 (and v2.1.5 is present in bookworm) it now interprets the argument as second by default

Thanks to everyone involved in the release, happy upgrading to bookworm, and let’s continue with working towards Debian/trixie. :)

Posted in Computer, Debian, English, Open Source | Comments Off on What to expect from Debian/bookworm #newinbookworm

HTU Bigband Konzert am 27.06.2023

June 2nd, 2023

Die HTU Bigband ist zur�ck! Am 27. Juni 2023 findet im Innenhof der TU Graz (Alte Technik, Rechbauerstra�e 12, 8010 Graz) das n�chste Konzert statt (bei Schlechtwetter geht es in den H�rsaal 2, der ebenfalls an der gleichen Adresse ist). Mit einem fulminanten Programm von Swing, �ber Soul, Funk, Latin bis Pop ist alles dabei – es gibt �ber 2 Stunden Musik vom Feinsten, und das Ganze bei freiem Eintritt.

F�r diejenigen mit Facebook-Account unter euch gibt es auch das passende Facebook-Event.

Ich bin als Schlagzeuger und Percussionist mit von der Partie und w�rde mich �ber bekannte Gesichter freuen, ich hoffe man sieht und h�rt sich! 8-)

Posted in Allgemein, Events | Comments Off on HTU Bigband Konzert am 27.06.2023

Vortrag: Debugging f�r Sysadmins @ GLT23

April 16th, 2023

Auf den Grazer Linuxtagen 2023 (GLT23) war ich als Referent mit einem Vortrag zum Thema “Debugging f�r Sysadmins” vertreten. In meinem Vortrag gibt es einen �berblick, welche Tools und Strategien rund ums Debugging in der Toolbox von Sysadmins nicht fehlen d�rfen.

Es gibt den Vortrag dank des wunderbaren c3voc-Teams bereits als Videomitschnitt online. Meine Vortragsfolien (1.2MB, PDF) stehen ebenfalls online zur Verf�gung. Viel Spa� beim Anschauen!

BTW: weil ich schon mehrfach gefragt wurde, den Vortrag gibt es auch in l�ngerer Workshop-Version, bei Interesse einfach bei mir melden.

Posted in Computer, Events | Comments Off on Vortrag: Debugging f�r Sysadmins @ GLT23