Planet CCC - Blogs and more around CCC, CCC-Hamburg and Attraktor

test 2 @ entropia Licensed to the public under https://creativecommons.org/licenses/by/4.0/ about this event: https://c3voc.de

Video:entropia-2-eng-Test_2_hd.mp4

A fundamental aspect of an OS security policy is the ability to manage mounting. That includes filesystem mounts, remounts, bind-mounts, and property changes. Contrary to popular believe Linux exposes only barebones policy hooks. This is about to change. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/J9TM7X/

Video:asg2026-441-eng-Managing_the_new_mount_api_sd.mp4

In this talk we will look at how binfmt and bpf can be integrated. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/8SCGQC/

Video:asg2026-440-eng-binfmt_bpf_sd.mp4

Varlink is slowly getting adopted in the desktop space. This brings a few new challenges, one of them being observability: D-Bus allows us to capture the entire system and session bus and then drill down on specific interactions in the system. Varlink on the other hand is just a protocol spoken over AF_UNIX sockets and thus has no bus and no global view of the system. The talk explores how we can use eBPF and extended attributes on sockets to give us back the observability we're used to with D-Bus. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/FKWEC8/

Video:asg2026-437-eng-Varlink_observability_sd.mp4

In this talk I'd like to provide a vision for accessing systemd hosts via local IPC APIs and remote RPC APIs. With the Varlinkification of systemd going full steam ahead, I'd like to explain what I think should be next for systemd and how things fit together with our focus on APIs. We'll talk about metrics/reports, Varlink in general, the Varlink HTTP proxy specifically, about security models, and zero-agent remote access, secure system parameterization and more. We'll talk about design patterns to follow, both for systemd components, and for other OS components, outside of it. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/RZ3PUB/

Video:asg2026-433-eng-An_API_for_systemd_Machines_sd.mp4

`systemd-sysupdate` provides a lot of incredibly useful primitives for doing system updates, but it is still mostly designed around old-school `SHA256SUMS` distribution and static transfer definitions. The need to have more autonomous systems for updating systems and adding components became quite clear when we started developing such systems at Amutable. This approach also dove-tailed nicely into developing an autonomous provisioning system using the same underlying mechanism, [leading us to build Quarry](https://amutable.com/blog/distributing-images-quarry) (which has been [recently open sourced](https://github.com/amutable-systems/quarry)). This talk will contain a discussion of the architecture we came up with, the pain points with integration with `systemd-sysupdate`, as well as some of the caveats that such an architecture has with generic distributions. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/BBJYFH/

Video:asg2026-431-eng-Modernising_Software_Distribution_with_Quarry_sd.mp4

For years, kexec was the standard way to skip lengthy firmware and bootloader delays at reboot. However, in modern cloud environments, even a fast reboot causes severe service degradation. Guests evacuation in virtualized environments or rebuild of massive in-memory caches for large databases is costly and disruptive. Kernel HandOver (KHO) and Live Update Orchestrator (LUO) introduce a complete framework for seamless kernel and userspace state preservation across upgrades. KHO provides the infrastructure to preserve kernel data structures and arbitrary memory regions across kexec. Physically contiguous memory areas managed by KHO let the incoming kernel to bootstrap itself without touching the preserved memory. On top of KHO, LUO provides userspace APIs for coordination of the transition and allows applications to preserve file descriptors across kexec. With initial support for memory file descriptors (memfd), the ecosystem is rapidly expanding to cover subsystems like PCI, VFIO, KVM, guestmemfd, and iommufd. Moreover, the ecosystem now features native systemd integration, leveraging the standard fdstore interface to pass resources across the kernel reboot. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/J8GNND/

Video:asg2026-430-eng-Seamless_Upgrades_with_KHO_LUO_and_systemd_sd.mp4

In this talk we'll cover the developments in kmod over the past few years. We'll wiz past the documentation and contribution updates, outline some of the performance improvements and discuss the latest APIs introduced. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/TTFUPB/

Video:asg2026-425-eng-Kmod_-_teaching_old_dog_new_tricks_sd.mp4

BuildStream is a tool for building / integrating software stacks. In a way, it has a similar goal to bitbake / yocto and Android's repo, but takes a completely different approach. It can be used to take software from various sources, build it with various buildsystems in a reproducible sandbox, and cache results for speedy rebuilds. mkosi is a tool for easily building customized OS images. It is generally used with distros that have a package manager such as dnf or apt. There is however recent development to add support for using it with BuildStream. In this talk I give a brief overview of Buildstream, and freedesktop-sdk which is a BuildStream project that can be used as a base to build your own operating system. While BuildStream on its own can build the whole thing, there are some advantages to using mkosi which I'll go over in this talk. I'll also go over the challenges faced moving GNOME OS to use mkosi for generating the image. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/DUNFF9/

Video:asg2026-423-eng-Building_image-based_operating_system_images_with_BuildStream_and_mkosi_sd.mp4

Delivering a Linux OS via OCI images unifies infrastructure management but often introduces severe network and state-reconciliation bottlenecks. This talk dives into the mechanics of solving these challenges by applying OS updates directly from OCI images to btrfs snapshots. We will explore how to drastically optimize these atomic upgrades using zstd-chunked compression, enabling systems to execute file-level delta updates via HTTP range requests instead of pulling full layers. Finally, we will demonstrate how to leverage snapper for seamless 3-way merges, safely reconciling upstream OS changes with local user configurations. The transition to image-based, immutable Linux systems fundamentally changes OS management, but it also introduces complex network and storage challenges that are often abstracted away. This session explores the architecture and technical workflows required to provision and upgrade an OS using OCI images, advanced compression, and copy-on-write filesystems. The foundation of this approach relies on mapping OCI images directly to btrfs snapshots. OCI image layers are pulled, unpacked, and directly applied as immutable btrfs snapshots, forming the read-only root volume of the system while maintaining distinct writable areas for persistent state. To eliminate the network bottleneck of pulling full OCI layers for minor patches, we will detail efficient delta updates with zstd-chunked OCI images. The OS agent identifies and downloads only the specific files that have changed between the current running image and the target image. We will explain the mechanics of using HTTP range requests to extract these payloads and write them directly into the target btrfs snapshot on the fly, drastically reducing update times and bandwidth consumption without requiring full local extraction. Finally, an immutable OS still requires localized configuration (e.g., in /etc), creating a challenge when the base OS image is updated. We will explore how to solve this using a 3-way merge strategy. In particular we will see how snapper is utilized to navigate btrfs snapshot trees and isolate the precise differences between the original base image, the user's localized state, and the incoming update. Having a clear picture of the differences in multiple directions allows us to define an algorithmic approach to safely merging these three states. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/E7L8GD/

Video:asg2026-417-eng-Delivering_the_OS_via_OCI_images_Btrfs_snapshots_delta_updates_and_3-way_merges_sd.mp4

KDE's new image-based Linux distribution embraces modern technologies whenever possible. We'll explore all the tools that contribute, and how they are being utilized to form the final product. From building the disk image to eventually updating it on the user's system. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/VXGVJF/

Video:asg2026-416-eng-KDE_Linux_Under_the_Hood_sd.mp4

A decade ago, container networking was in its infancy, relying heavily on complex bridges and iptables rules. Today, the landscape has fundamentally shifted towards highly performant, secure, and observable dataplanes powered by eBPF. In this talk, a core contributor to Cilium reflects on a decade of building container networking solutions - from the early days of Weave Net to modern eBPF-based architectures. We will explore the hard-fought engineering lessons from migrating away from traditional kernel bottlenecks and discuss how Cilium evolved into the most widely used container networking plugin (CNI) and a foundational platform for advanced networking research. Finally, we will look to the horizon to discuss what the next decade of Linux system networking holds. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/R7BEEK/

Video:asg2026-408-eng-Ten_Years_Building_Container_Networking_Lessons_Learned_and_What_the_Future_Might_Look_Like_sd.mp4

When a VM is confidential (CVM), the host cannot access its memory, enabling data-in-use protection. Hardware vendors like Intel (TDX) and AMD (SEV-SNP) provide the underlying technology to achieve this. The use case becomes more complex when a persistent disk is added to a CVM: the disk must also be encrypted, but how can we safely manage its encryption keys without trusting the host? We propose a mechanism to encrypt and decrypt a disk using remote attestation: the passphrase is stored in a Key Broker Service (KBS) that only releases it when the identity of the VM is verified. During the first boot, systemd-repart encrypts the root disk using a random key that is generated and stored in KBS. On subsequent boots, systemd-cryptsetup requests the same key to decrypt the disk. We present the changes we are working on to support this use case in components like KubeVirt, Trustee, and systemd-repart. We present PRs for Trustee, systemd-repart, and KubeVirt, along with a new LUKS token plugin that integrates systemd-repart and systemd-cryptsetup with KBS for key retrieval. This use case extends beyond KubeVirt and applies to any environment where confidential workloads require encrypted persistent storage, including bare metal deployments. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/TKKLMZ/

Video:asg2026-405-eng-Disk_encryption_using_attestation_for_Confidential_VMs_sd.mp4

[IncusOS](https://linuxcontainers.org/incus-os) is an immutable OS based on Debian 13 and built using mkosi. It's designed to be as secure as possible, actively relying on both UEFI Secure Boot and a TPM 2.0 module to provide strong boot security as well as encryption at rest through LUKS+TPM. Its whole purpose is to provide an ideal environment for running [Incus](https://linuxcontainers.org/incus/) with a focus on running on bare metal with pretty complex local and remote storage as well as a variety of networking options. Unlike most other Linux systems, IncusOS doesn't provide any local or remote shell access to the system. Instead it's entirely API driven with its management API being available through the Incus API itself. That management API allows for system configuration, update management, installation of additional components (systemd system extensions), configuring a variety of system services, ... But it also needs to handle a variety of fallback and recovery/patching mechanisms for when something goes wrong. This talk will cover why we built IncusOS, how we built it, its current state and what we're working towards in the near future. [Incus](https://linuxcontainers.org/incus/) is a modern container and virtual machine manager with native support for both system and application containers as well as virtual machines. It's designed to behave just like a private cloud, but one that you can run on your laptop or Raspberry Pi as well as on clusters of hundreds or thousands of servers. It's got support for a wide variety of local and remote storage options as well as both software defined and traditional networking. It has built-in clustering allowing for hundreds of servers to act as one with a consistent API and distributed database providing high availability and automated recovery. Incus is widely available through packages in most Linux distributions with IncusOS providing another method of deployment particularly targeted to those who don't want to have to manage yet another Linux system or those who want to perform large scale deployments with known identical systems. IncusOS can be deployed and managed at scale through [Operations Center](https://github.com/futurfusion/operations-center), allowing for hundreds of clusters to be deployed, updated and tracked in one place. Incus, IncusOS, Operations Center and generally everything related to Incus is Open Source and released under the Apache 2.0 license. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/ER3SSR/

Video:asg2026-402-eng-Building_IncusOS_sd.mp4

Image-based Linux systems have different storage requirements from traditional mutable systems as they rely on signed, immutable disk images for software delivery and separate writable filesystems for data. Technologies like systemd-homed require that multiple writable filesystems share the same backing storage. This thin provisioning is not something that is well supported by Linux today as filesystems are not aware of it. Another area for improvement is that many disk images often contain similar data that could be deduplicated. We will present a storage system prototype designed for the usecases of image based systems and our plans for improving thin provisioning on Linux. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/WSNAQL/

Video:asg2026-401-eng-OSPool_and_thin_volumes_Storage_for_Image-based_systems_sd.mp4

Confidential VM (CVM) instance types are becoming ubiquitous, with all major cloud providers offering them as a key component for achieving “zero trust.” While complete zero trust is currently more of a marketing term than a technical reality, we can acknowledge that CVM technologies help reduce the Trusted Computing Base (TCB) when utilized properly. But do we actually know how to use them effectively? When transitioning from a world of traditional, mutable operating systems, users often expect to gain security benefits simply by moving their existing workloads to new CVM instance types, perhaps with some level of attestation. Unfortunately, in many cases, this looks more like a role play in a “security theater”. Over the past few years, we have investigated what security benefits can truly be unlocked when mutable RHEL or Fedora workloads are deployed on confidential VMs, whether in the public cloud or on-premises. While immutable OSes offer distinct advantages in such scenarios, we pose the question: can traditional (mutable) OSes also benefit? Over the last year, we have contributed several features to systemd (and other components) that we believe can improve the situation. In particular, we are focused on preventing certain attack scenarios against a VM which is normally deployed from a publicly available cloud (marketplace) image. We have further ideas, and in this talk, we will share our vision for how these pieces can fit together. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/VNXPQS/

Video:asg2026-399-eng-Running_mutable_OS_images_on_Confidential_VMs_is_there_any_hope_sd.mp4

Enterprise Linux deployments lack a standardized, native enrollment flow. Current solutions rely on distro‑specific first‑boot mechanisms (systemd-firstboot, cloud‑init, realmd, ignition, kickstart and more) or external configuration management layered post‑deployment, resulting in poor feature sets w.r.t. modern features enabled by TPM2 or UEFI setting management. This talk proposes to explore minimal, vendor‑neutral set of APIs and protocols for OS enrollment, *analogous to* Windows Autopilot but designed for the Linux userspace stack and specifically for the ParticleOS ecosystem. We will examine a concrete model based on device identity (TPM, SMBIOS information), secure attestation, policy discovery, and management‑hint delivery, decoupled from any particular configuration management system. There might be a live demonstration presenting a reference implementation on NixOS communicating with a lightweight enrollment service over HTTP(S) leveraging a bunch of Varlink APIs built across the past years and presented at ASG. The primary outcome is a community‑reviewed specification draft for enrollment APIs, intended for integration into systemd and distro installers. The talk is not a proposal for a new tool but rather invite the attendees to think about the problem space around corporate enrollment for Linux distributions and how do we build native solutions for distributions interested by this idea. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/3RTJNA/

Video:asg2026-397-eng-Native_OS_enrollment_for_the_Linux_userspace_A_proposal_for_open_APIs_and_protocols_sd.mp4

When a machine/server/node is serviced and experiences a downtime, that results in a service interruption, customers or users tend to get quite sad. This talk will explore two recent additions to systemd that aim to make these service interruptions as minimal as possible. systemd v260 and v261 added two features which aim to let service owners minimize downtime. Firstly, when services use dm-verity images, profiling data showed that the largest bottleneck by far was loading the new images, which used to happen after the running service had already stopped during a unit restart. Since v260, when very specific conditions are met, this bottleneck has been removed from the critical path. Secondly, a common issue causing slowdowns for services is having to serialize and deserialize large amount of state from slow disks after a reboot due to an OS update. Since v261, systemd supports the kernel's Live Update Orchestration, and uses it to persist the content of the FD Store across a kexec reboot. Services can use a memfd for their runtime state, hand it over the systemd when they are stopped/restarted, and they will get it back and can mmap it and use it straight away on restart. With LUO support, services can rely on this even across a kexec, removing the need to use serialize and deserialize state to disk. This talk will explore both features, explaining how they work and how they can be used. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/XFZKN8/

Video:asg2026-393-eng-systemd_reducing_impact_of_servicing_interruptions_on_restart_reboot_sd.mp4

In this talk, we'll walk through the architecture of Amutable's OS image build system. We'll dive into how we mirror rpms from Fedora and orchestrate building those rpms from source, as well as building OS images with buck2, an open source monorepo build system. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/YSR3PP/

Video:asg2026-392-eng-Hold_My_Beer_Im_Building_100_Images_at_Once_sd.mp4

This talk will give an update on make NoNewPrivs the default on a Linux distribution. Running systems with “NoNewPrivs” by default is already possible without issues for many use cases, but there are still some cases where it doesn't work. As followup to last year's presentation “Accessing shadow records via varlink” this session provides an up-to-date status report on the progress made and the issues that remain. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/CG9K7H/

Video:asg2026-387-eng-Making_NoNewPrivs_the_default_sd.mp4

`systemd-sysupdate` provides a lot of incredibly useful primitives for doing system updates, but it is still mostly designed around old-school `SHA256SUMS` distribution and static transfer definitions. The need to have more autonomous systems for updating systems and adding components became quite clear when we started developing such systems at Amutable. This approach also dove-tailed nicely into developing an autonomous provisioning system using the same underlying mechanism, [leading us to build Quarry](https://amutable.com/blog/distributing-images-quarry) (which has been [recently open sourced](https://github.com/amutable-systems/quarry)). This talk will contain a discussion of the architecture we came up with, the pain points with integration with `systemd-sysupdate`, as well as some of the caveats that such an architecture has with generic distributions. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/BBJYFH/

Video:asg2026-431-eng-Modernising_Software_Distribution_with_Quarry_hd.mp4

This talk will give an update on make NoNewPrivs the default on a Linux distribution. Running systems with “NoNewPrivs” by default is already possible without issues for many use cases, but there are still some cases where it doesn't work. As followup to last year's presentation “Accessing shadow records via varlink” this session provides an up-to-date status report on the progress made and the issues that remain. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/CG9K7H/

Video:asg2026-387-eng-Making_NoNewPrivs_the_default_hd.mp4

What if we could run systemd services on top of virtual cpu/memory, to get hypervisor/cpu protection of memory pages? Demo of a very much work-in-progress proof-of-concept Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/7U77A3/

Video:asg2026-448-eng-systemd_services_as_microvms_sd.mp4

What's new in zbus https://zeenix.github.io/presentations-marp/LTs/zbus-6-whats-new.html Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/3C8PUC/

Video:asg2026-447-eng-zbus_60_sd.mp4

At Axis, we are always interested in standard technologies and how we can align with them. With that in mind, we have been studying the principles and ideas behind ParticleOS. In this session, we will build ParticleOS live with mkosi and boot it in a VM. We will look into the provisioning steps of such a machine and the changes in the OS image layout. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/L3KZRV/

Video:asg2026-445-eng-ParticleOS_in_action_sd.mp4

System extensions (sysexts) become part of the host's /usr, making them available to all users. But what if we want to run tightly coupled services without exposing their binaries to everyone else on the system? This lightning talk will show how to ship a real-world stack (kubelet, containerd, and runc) without making them accessible to the rest of the host. We will demonstrate how to use portable services and systemd capsules to allow these components to invoke each other and share some namespaces, all while keeping the host OS clean. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/K9GPKV/

Video:asg2026-436-eng-Couple_Services_Not_the_Host_Portable_Services_and_Capsules_sd.mp4

Recently systemd acquired functionality to download & invoke OCI containers directly. In this talk I'd like to explain where we are at with this, what we have, where the gaps are and where we are going, to make OCI containers a thing you can directly execute with systemd, without external tools. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/BWWVCN/

Video:asg2026-435-eng-systemd_OCI_sd.mp4

In recent systemd releases we substantially improved the tooling for provisioning and deploying operating systems with systemd. For example, systemd-sysinstall, systemd-sysupdate, credentials, systemd-firstboot, systemd-stub's parameterization logic, systemd-confext, systemd-imds, systemd-import all are relevant components for getting a node up and running with the right configuration, securely and robustly. In this talk I'd like to explain the current landscape, and how the various components matter in various scenarios, and how they fit together. I'll also discuss what's next, what's missing, and where we should be going with all this. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/NLRDPH/

Video:asg2026-434-eng-Provisioning_and_Deployment_Mechanisms_in_systemd_sd.mp4

The limitations of systemd-sysext/confext from the early days are mostly gone nowadays. The loading of extension images from the initrd means that generic content will behave as expected without any workarounds. New settings help making live reconfiguration work better as well. We will see how declarative configuration management with systemd-sysext/confext from git can be done with an OpenTofu/Terraform module. For image-based Linux binding configuration together with a pinned OS version is now possible through bootctl link. OS builders can ship /etc files through a default confext image. systemd-sysext/confext: https://www.freedesktop.org/software/systemd/man/devel/systemd-sysext.html bootctl: https://www.freedesktop.org/software/systemd/man/devel/bootctl.html Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/RMC879/

Video:asg2026-429-eng-An_update_on_systemd-sysext_confext_sd.mp4

Systemd can now be linked as a single static binary, which is enough to run a "full system" container that boots into a running manager. Transient units can then be used to do more things in the container. Thanks to systemd-nsresourced and systemd-mountfsd, this now works fully unprivileged and with no suid helpers. I want to show that this actually works and can be used for experimentation, development, and custom container images. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/XU9CJR/

Video:asg2026-427-eng-single-file_containers_with_statically_linked_systemd_sd.mp4

For a while, the standard way to run Podman containers under systemd was podman generate systemd. It worked, but the output was static and awkward to maintain. Quadlet replaced it — and most guides stop at "drop an INI file in a directory and your container starts." leaving the machinery a black box. This talk demystifies that black box, because the internals are exactly what you reach for when things break. Quadlet isn't a daemon. It's a systemd generator that runs at daemon-reload and turns your .container, .pod, and .network files into ordinary .service units. In session, we'll trace one stack example from end to end, watching a three-line container file expand into a full podman run command, and cross-unit references become injected After=/Wants= dependency edges with name-mangled volumes and networks. You'll leave knowing exactly what gets generated, why edits need a daemon-reload, how to dump & debug a failing unit yourself, and explain why keeping only the source files under version control is reliable. Less magic, more model. Quadlet isn't magic—it's a systemd generator. Watch a three-line .container file expand into a full service unit, and learn to dump, debug, and reason about what really gets generated. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/PXBVLE/

Video:asg2026-424-eng-Forget_podman_generate_systemd_How_Quadlet_Really_Works_sd.mp4

Using mkosi and apparmor.d as building blocks, this talk presents a prototype minimal Debian server. The goal is not to provide yet another Linux distribution, but to explore how hard it is to build personalized, hardened server images. On top of the usual mkosi-backed features such as UKIs, A/B updates and dm-verity, the project adds a whole-system AppArmor policy and an immutable `/etc`. Because older devices also matter, the prototype supports the Raspberry Pi 3 (UEFI) too, with minor changes to the security model. The talk presents the differences in design, attacker model, and implementation between the various targets. This talk walks through a prototype: a minimal, hardened Debian server built with `mkosi` and `apparmor.d`. It composes three layers of systemd and Debian tooling that are well understood alone but rarely combined, each reinforcing the others: - **Image-based OS:** `mkosi` magic. - **Whole-system MAC:** AppArmor policies that confine every process on the system. It also brings: - Multi-layered policy loading, fixing the AppArmor load-time issues that hurt small devices. - Immutable security policies. - Compatibility with systemd sysext/confext. - **Immutable `/etc`:** assembled from signed `systemd-confext` images, closing immutability's usual escape hatch. The same image targets both an x86-64 server and a Raspberry Pi 3 (UEFI), and that contrast is where it gets interesting. The two share a single build but diverge on attacker model and security guarantees: what an embedded device with no Secure Boot can promise versus a server, and where the implementation has to bend. We walk those differences, the limitations, and what would have to land upstream to make this kind of image easier. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/KJNYQ8/

Video:asg2026-422-eng-Immutable_Fully-Confined_Debian_Server_Images_Even_on_a_Raspberry_Pi_3_sd.mp4

Observability stacks have gotten good at CPU, memory, and network — but storage is still where the picture goes dark. When a workload stalls on I/O, device-level tools like iostat or blktrace tell you the disk was busy; they don't tell you which service caused it, or where in the stack the latency actually accrued: a slow fsync, writeback pressure in the page cache, a filesystem journal commit, or a saturated block queue. This talk follows a single write from the syscall boundary down to the block layer, using eBPF to instrument each handoff — VFS, the filesystem, the page cache, and block I/O submission and completion — without patching applications or running a custom kernel. We'll look at which attach points (tracepoints, fentry/fexit, kprobes) give honest per-layer latency, how to keep the measurement overhead from biasing the result, and how to attribute the latency back to the responsible cgroup or systemd service instead of an aggregate line on a dashboard. Examples range from fsync-heavy database commits to large sequential writes that look fine at the device level but stall well above it. Attendees will leave with a concrete method for turning storage from a black box into a traceable path — and an honest sense of where eBPF helps, and where it still can't see. Follow a single write from syscall to block layer with eBPF, measuring honest per-layer latency at each handoff and attributing storage stalls to the service that caused them. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/XHJX3N/

Video:asg2026-418-eng-From_fsync_to_Block_I_O_Tracing_Storage_with_eBPF_sd.mp4

Immutable, image-based Linux has gone mainstream — on Fedora. Nearly every atomic distro descends from the RPM/ostree lineage. Debian is conspicuously rare, with really only Vanilla OS and Endless OS to its name. There's a reason, and it's apt: Debian's package manager expects a writable root filesystem, so the moment you seal the rootfs, the native way to install and update software stops working. That's the wall every immutable Debian hits, and most climb it with Flatpak or containers. This talk takes a different route — Nix. Nix keeps everything in a self-contained, content-addressed /nix/store, so it never needs a writable rootfs. The base is built once with mkosi and frozen (apt isn't even installed) and Nix manages everything that changes after that, reproducibly. The result is an immutable Debian rootfs with the entire Nix package ecosystem available on top. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2026/talk/BEDGKF/

Video:asg2026-415-eng-Make_Debian_Immutable_with_mkosi_and_Nix_sd.mp4