Merge pull request #167 from Jip-Hop/develop

v1.4.0

NETWORKING.md

@@ -1,61 +0,0 @@
-# Jailmaker
-
-## Advanced Networking
-
-These are notes on advanced networking setup you may want to try. Contributions are welcome!
-
-### Bridge Networking
-
-As an alternative to the default host networking mode, you may want to connect to a bridge interface instead and let the jail obtain its IP address via DHCP (although you may have to be patient for up to 20 seconds after the jail started for networking to work, [assigning the IP address is somehow slow](https://github.com/Jip-Hop/jailmaker/issues/7)).
-
-[This YouTube video](https://www.youtube.com/watch?v=7clQw132w58) may be helpful when setting up the bridge interface. Note: You may lock yourself out... It may take several tries... TrueNAS is a bit picky when switching IP addresses and toggling DHCP. May be helpful to connect a monitor and keyboard to the NAS and use `/etc/netcli` to reset the networking interface. Kept bothering with "Register Default Gateway" warning... I just clicked Cancel.
-
-Add the `--network-bridge=br1 --resolv-conf=bind-host` systemd-nspawn flag when asked for `Additional flags` during jail creation, or set it post-creation by [editing](./README.md#edit-jail-config) the `SYSTEMD_NSPAWN_USER_ARGS` variable inside the `config` file.
-
-The TrueNAS host and the jail will be able to communicate with each other as if the jail was just another device on the LAN. It will use the same DNS servers as the TrueNAS host because the `--resolv-conf=bind-host` option bind mounts the `/etc/resolv.conf` file from the host inside the jail. If you want to use the DNS servers advertised via DHCP, then check [DNS via DHCP](#dns-via-dhcp).
-
-To configure a **static IP** with our bridge interface, we need to edit the `80-container-host0.network` file located in `/etc/systemd/network`. Change the `[Network]` section to look like this:
-
-```ini
-[Network]
-DHCP=false
-Address=192.168.0.12/24
-Gateway=192.168.0.1
-LinkLocalAddressing=no
-LLDP=yes
-EmitLLDP=customer-bridge
-```
-Then restart the `systemd-networkd` service and check your network configuration.
-
-```shell
-systemctl restart systemd-networkd
-systemctl status systemd-networkd
-ifconfig
-```
-
-
-### Macvlan Networking
-
-To setup Macvlan Networking you may follow the [Bridge Networking](#bridge-networking) section, but skip the setup of a bridge interface and use these flags instead: `--network-macvlan=eno1 --resolv-conf=bind-host`. By default the TrueNAS host and jail will not be able to communicate with each other via the network if Macvlan Networking mode is used. If that's required it would be better to use [Bridge Networking](#bridge-networking).
-
-### DNS via DHCP
-
-If you're not using host networking, and you're not using the `--resolv-conf=` in case of bridge/macvlan networking, then you have to configure the DNS servers to use.
-
-To get DNS servers via DHCP install and enable `resolvconf`.
-
-```shell
-# Only run this inside the jail!
-
-# Temporarily fix DNS resolution,
-# otherwise we can't install packages
-echo 'nameserver 8.8.8.8' > /etc/resolv.conf
-# On debian based distro
-apt update && apt -y install resolvconf
-```
-
-## References
-
-- [systemd-nspawn](https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html)- [Setting up Systemd-nspawn](https://www.cocode.se/linux/systemd_nspawn.html#orge360318)
-- [Debian Reference - Chapter 5. Network setup](https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_hostname_resolution)
-- [Disabling link-local addressing](https://jerrington.me/posts/2017-08-06-systemd-nspawn-disabling-link-local-addressing.html#disabling-link-local-addressing)

README.md

@@ -90,13 +90,15 @@ jlmkr start myjail
 
 ### List Jails
 
+See list of jails (including running, startup state, GPU passthrough, distro, and IP).
+
 ```shell
 jlmkr list
 ```
 
 ### Execute Command in Jail
 
-You may want to execute a command inside a jail, for example from a shell script or a CRON job. The example below executes the `env` command inside the jail.
+You may want to execute a command inside a jail, for example manually from the TrueNAS shell, a shell script or a CRON job. The example below executes the `env` command inside the jail.
 
 ```shell
 jlmkr exec myjail env
@@ -118,6 +120,8 @@ Once you've created a jail, it will exist in a directory inside the `jails` dir
 
 ### Remove Jail
 
+Delete a jail and remove it's files (requires confirmation).
+
 ```shell
 jlmkr remove myjail
 ```
@@ -136,6 +140,8 @@ jlmkr restart myjail
 
 ### Jail Shell
 
+Switch into the jail's shell.
+
 ```shell
 jlmkr shell myjail
 ```
@@ -148,6 +154,8 @@ jlmkr status myjail
 
 ### Jail Logs
 
+View a jail's logs.
+
 ```shell
 jlmkr log myjail
 ```
@@ -158,17 +166,21 @@ Expert users may use the following additional commands to manage jails directly:
 
 ## Networking
 
-By default the jail will have full access to the host network. No further setup is required. You may download and install additional packages inside the jail. Note that some ports are already occupied by TrueNAS SCALE (e.g. 443 for the web interface), so your jail can't listen on these ports. This is inconvenient if you want to host some services (e.g. traefik) inside the jail. To workaround this issue when using host networking, you may disable DHCP and add several static IP addresses (Aliases) through the TrueNAS web interface. If you setup the TrueNAS web interface to only listen on one of these IP addresses, the ports on the remaining IP addresses remain available for the jail to listen on.
+By default a jails will use the same networking namespace, with access to all (physical) interfaces the TrueNAS host has access to. No further setup is required. You may download and install additional packages inside the jail. Note that some ports are already occupied by TrueNAS SCALE (e.g. 443 for the web interface), so your jail can't listen on these ports.
 
-See [Advanced Networking](./NETWORKING.md) for more.
+Depending on the service this may be o.k. For example Home Assistant will bind to port 8123, leaving the 80 and 443 ports free from clashes for the TrueNAS web interface. You can then either connect to the service on 8123, or use a reverse proxy such as traefik.
+
+But clashes may happen if you want some services (e.g. traefik) inside the jail to listen on port 443. To workaround this issue when using host networking, you may disable DHCP and add several static IP addresses (Aliases) through the TrueNAS web interface. If you setup the TrueNAS web interface to only listen on one of these IP addresses, the ports on the remaining IP addresses remain available for the jail to listen on.
+
+See [the networking docs](./docs/network.md) for more advanced options (bridge and macvlan networking).
 
 ## Docker
 
-Using the [docker config template](./templates/docker/README.md) is recommended if you want to run docker inside the jail. You may of course manually install docker inside a jail. But keep in mind that you need to add `--system-call-filter='add_key keyctl bpf'` (or disable seccomp filtering). It is [not recommended to use host networking for a jail in which you run docker](https://github.com/Jip-Hop/jailmaker/issues/119). Docker needs to manage iptables rules, which it can safely do in its own networking namespace (when using [bridge or macvlan networking](./NETWORKING.md) for the jail).
+Using the [docker config template](./templates/docker/README.md) is recommended if you want to run docker inside the jail. You may of course manually install docker inside a jail. But keep in mind that you need to add `--system-call-filter='add_key keyctl bpf'` (or disable seccomp filtering). It is [not recommended to use host networking for a jail in which you run docker](https://github.com/Jip-Hop/jailmaker/issues/119). Docker needs to manage iptables rules, which it can safely do in its own networking namespace (when using [bridge or macvlan networking](./docs/network.md) for the jail).
 
 ## Documentation
 
-Additional documentation contributed by the community can be found in [the docs directory](./docs/).
+Additional documentation can be found in [the docs directory](./docs/) (contributions are welcome!).
 
 ## Comparison
 
@@ -178,16 +190,6 @@ TODO: write comparison between systemd-nspawn (without `jailmaker`), LXC, VMs, D
 
 The rootfs image `jlmkr.py` downloads comes from the [Linux Containers Image server](https://images.linuxcontainers.org). These images are made for LXC. We can use them with systemd-nspawn too, although not all of them work properly. For example, the `alpine` image doesn't work well. If you stick with common systemd based distros (Debian, Ubuntu, Arch Linux...) you should be fine.
 
-## Tips & Tricks
-
-### Colorized bash prompt
-
-To visually distinguish between a root shell inside the jail and a root shell outside the jail, it's possible to colorize the shell prompt. When using a debian jail with the bash shell, you may run the following command to get a yellow prompt inside the jail (will be activated the next time you run `jlmkr shell myjail`):
-
-```bash
-echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;33m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc
-```
-
 ## Filing Issues and Community Support
 
 When in need of help or when you think you've found a bug in jailmaker, [please start with reading this](https://github.com/Jip-Hop/jailmaker/discussions/135).

docs/README.md

@@ -1,3 +1,5 @@
-Welcome to the jailmaker wiki!
+# Jailmaker Docs
+
+Welcome to the Jailmaker Docs!
 
 Use the sidebar to navigate the topics.

docs/basicconfig.md

@@ -1,4 +1,8 @@
-# User Management
+# Jailmaker Docs
+
+Anything described on this page is completely optional. You do NOT need to do anything of this in order to start using jailmaker.
+
+## User Management
 The root user (also known as the superuser or su) can access any file, make system changes, and lots of room for security vulnerabilities.
 For this reason you should aspire to run services as a non-root user.
 
@@ -10,28 +14,54 @@ Where username can be anything, but should reflect the service/jail's name for d
 Then a password should be created as some commands require a non-blank password to be inserted:
 `passwd USERNAME`
 
-If you want the ability to run commands as root, add the user to the sudo group
+If you want the ability to run commands as root, add the user to the sudo group:
-`usermod -aG sudo USERNAME`
+
+```sh
+usermod -aG sudo USERNAME
+```
 
 This WILL require a non-blank password, and any command run with sudo will be run as root not as the user. But it saves time compared to switching users to root to install/change things then switching back.
 
 ### Switch to user
-`su -l USERNAME`
+
+```sh
+su -l USERNAME
+```
 
 ### Put a password on Root
-While logged in as root run `passwd`
 
-# Common tweaks
+While logged in as root run `passwd`.
+
+## Common Tweaks
+
 ### Update repository list 
-`sudo apt update`
+
+```sh
+sudo apt update
+```
 
 ### Install common services 
-`sudo apt install nano wget curl git`
+
+```sh
+sudo apt install nano wget curl git
+```
 
 ### Set Static IP
-See `Networking`
+
+See [Networking](./network.md)
+
+### Colorized bash prompt
+
+To visually distinguish between a root shell inside the jail and a root shell outside the jail, it's possible to colorize the shell prompt. When using a debian jail with the bash shell, you may run the following command **inside the jail** to get a yellow prompt inside the jail (will be activated the next time you run `jlmkr shell myjail`):
+
+```bash
+echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;33m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc
+```
 
 ### Install Docker
-```
+
+It's advised to use the [docker config template](../templates/docker/README.md). But you can install it manually like this as well:
+
+```sh
 apt install curl && cd /tmp && curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh && cd ~ && docker
 ```

docs/commands.md

@@ -1,35 +0,0 @@
-Create a jail
-`jlmkr create JAILNAME`
-
-Start a jail
-`jlmkr start JAILNAME`
-
-Stop a jail
-`jlmkr stop JAILNAME`
-
-Check jail status
-`jlmkr status JAILNAME`
-
-Delete a jail and remove it's files (requires confirmation)
-`jlmkr remove JAILNAME`
-
-See list of jails (including running, non running, distro, startup state, and IP)
-`jlmkr list`
-
-See list of running jails
-`machinectl list`
-
-Execute a command inside a jail from the TrueNAS shell
-`jlmkr exec JAILNAME COMMAND`
-
-Execute a bash command inside a jail from the TrueNAS shell
-`jlmkr exec JAILNAME bash -c 'BASHCOMMAND'`
-
-Switch into the jail's shell
-`machinectl shell JAILNAME`
-
-View a jail's logs
-`jlmkr log JAILNAME`
-
-Edit a jail's config
-`jlmkr edit JAILNAME`

docs/compatibility.md

@@ -1,12 +1,14 @@
-# TrueNAS Compatibility
+# Jailmaker Docs
+
+## TrueNAS Compatibility
 |   |   |
 |---|---|
 |TrueNAS Core|❌|
 |TrueNAS 22.12|✅|
 |TrueNAS 23.10|✅|
-|TrueNAS 24.04 nightly|✅|
+|TrueNAS 24.04|✅|
 
-# Distro Compatibility
+## Distro Compatibility
 |   |   |
 |---|---|
 |Debian 11 Bullseye|✅|

docs/network.md

@@ -1,24 +1,60 @@
-# Host Passthrough (Default network configuration)
+# Jailmaker Docs
-By default jails will use the same physical interface as the TrueNAS host. If a service attempts to bind to port 80 or 443, it will either fail or render both the service and TrueNAS unavailable. 
-### Flaws
-Depending on the service this may be ok, for example Home Assistant will bind to port 8123, leaving the 80 and 443 ports free from clashes for the TrueNAS web interface. You can then either connect to the service with the port, or use a reverse proxy such as [nginx](https://www.nginx.com/#).
-### Setup
-No configuration is necessary
 
-# MAC VLAN Virtual Interface
+## Host Networking
-Some services require the use of port 80 or 443, or would benefit from a separate IP. For these situations the easiest network configuration is the MAC VLAN configuration. This creates a virtual interface with its own separate randomly generated MAC address and IP.
+
-The default config uses DHCP by default, but can easily be set to a Static IP.
+[Notes on the default host networking are in the main README.md file](../README.md#networking).
-### Flaws
+
-Any services in the jail cannot communicate with the direct host (TrueNAS). The jail can communicate with any other jail or device on the network, besides TrueNAS. This may or not be a benefit (security) or disadvantage (no communication) depending on your service.
+## Bridge Networking
-### Setup
+
-Add the following argument to the "additional flags" prompt of jail creation or the "systemd_nspawn_user_arguments" line of the jail config file:
+As an alternative to the default host networking mode, you may want to connect to a bridge interface instead and let the jail obtain its IP address via DHCP.
+
+[![TrueNAS Scale: Setting up a Static IP and Network Bridge // Access NAS host from VM - YouTube Video](https://img.youtube.com/vi/uPkoeWUfiHU/0.jpg)<br>Watch on YouTube](https://www.youtube.com/watch?v=uPkoeWUfiHU "TrueNAS Scale: Setting up a Static IP and Network Bridge // Access NAS host from VM - YouTube Video")
+
+The above YouTube video may be helpful when setting up the bridge interface.
+
+### Bridge Flaws
+
+This type of interface takes much longer to set up both in complexity and wait time (you may have to be patient for up to 60 seconds after the jail started for networking to work, [assigning the IP address via DHCP is somehow slow](https://github.com/Jip-Hop/jailmaker/issues/7)). Furthermore, if the configuration is not correct it can render your TrueNAS inaccessible via ssh or the web interface, necessitating a reset using a keyboard and monitor plugged into the TrueNAS server and use `/etc/netcli` to reset the networking interface.
+
+### Bridge Setup
+
+Add the `--network-bridge=br1 --resolv-conf=bind-host` systemd-nspawn flag when asked for `Additional flags` during jail creation, or set it post-creation by [editing](./README.md#edit-jail-config) the `SYSTEMD_NSPAWN_USER_ARGS` variable inside the `config` file.
+
+The TrueNAS host and the jail will be able to communicate with each other as if the jail was just another device on the LAN. It will use the same DNS servers as the TrueNAS host because the `--resolv-conf=bind-host` option bind mounts the `/etc/resolv.conf` file from the host inside the jail. If you want to use the DNS servers advertised via DHCP, then check [DNS via DHCP](#dns-via-dhcp).
+
+### Bridge Static IP
+To configure a static IP with our bridge interface, we need to edit the  `/etc/systemd/network/80-container-host0.network` file. Change the [Network] section to look like this:
+
+```ini
+[Network]
+DHCP=false
+Address=192.168.0.12/24
+Gateway=192.168.0.1
+LinkLocalAddressing=no
+LLDP=yes
+EmitLLDP=customer-bridge
 ```
---network-macvlan=eno1 --resolv-conf=bind-host
+Then restart the `systemd-networkd` service and check your network configuration.
+
+```shell
+systemctl restart systemd-networkd
+systemctl status systemd-networkd
+ifconfig
 ```
 
-### Setting a Static IP
+## Macvlan Networking
-To set a Static IP you need to disable DHCP in the macvlan config file `/etc/systemd/network/mv-dhcp.network`
+
-You can do this with a network client like WinSCP by navigating into the jail's filesystem then the path above, or by using a text editing program like nano by running `nano /etc/systemd/network/mv-dhcp.network` in the jail's shell.
+Some services require the use of port 80 or 443, or would benefit from a separate IP. For these situations the easiest network configuration is the MAC VLAN configuration. This creates a virtual interface with its own separate randomly generated MAC address and IP. The default config uses DHCP by default, but can easily be set to a Static IP.
+
+### Macvlan Flaws
+Any services in the jail cannot communicate with the direct host (TrueNAS). The jail can communicate with any other jail or device on the network, besides TrueNAS or VMs hosted on TrueNAS. This may be a benefit (security) or disadvantage (no communication) depending on your service. If that's required it would be better to use [Bridge Networking](#bridge-networking).
+
+### Macvlan Setup
+
+Add the following argument to the "additional flags" prompt of jail creation or the "systemd_nspawn_user_arguments" line of the jail config file: `--network-macvlan=eno1 --resolv-conf=bind-host`. Where eno1 is the name of your physical network interface.
+
+### Macvlan Static IP
+To set a Static IP you need to disable DHCP in the macvlan config file `/etc/systemd/network/mv-dhcp.network`. You can do this with a network client like WinSCP by navigating into the jail's filesystem then the path above, or by using a text editing program like nano by running `nano /etc/systemd/network/mv-dhcp.network` in the jail's shell.
 
 The DHCP in [Network] needs to be set to false, an Address (static IP) needs to be added, a Gateway needs to be defined (e.g your router such as 192.168.0.1) and the entire DHCP section needs to be removed.
 
@@ -35,23 +71,24 @@ Gateway=192.168.X.X
 ```
 Then restart the network interface inside the jail `systemctl restart systemd-networkd` or restart the jail by running `jlmkr stop JAILNAME && jlmkr start JAILNAME` from the TrueNAS shell. Use `ifconfig` to verify the interface is up and has the correct IP.
 
-# Passthrough a TrueNAS Bridge Interface
+## DNS via DHCP
-By creating a network bridge in the TrueNAS Network page you can bridge the incoming physical network interface to a virtual interface that can be passed to the jail. This type of interface has the benefits of a MAC VLAN interface without the flaws (host to jail networking). Once working the virtual interface can either be assigned a static IP or obtain one automatically via DHCP.
-### Flaws
-This type of interface takes much longer to set up both in complexity and wait time as there is a current flaw in which HDCP can take between 10 seconds and a minute.
-Furthermore, if the configuration is not correct it can render your TrueNAS inaccessible via ssh, necessitating a reset using a keyboard and monitor plugged into the TrueNAS server.
-### Setup
-[TrueNAS Bridge interface guide](https://www.youtube.com/watch?v=7clQw132w58)
-May be helpful to connect a monitor and keyboard to the NAS and use /etc/netcli to reset the networking interface. Kept bothering with "Register Default Gateway" warning... I just clicked Cancel.
 
-Add the `--network-bridge=br1 --resolv-conf=bind-host` flag when asked for additional flags during jail creation, or set it post-creation by editing the `SYSTEMD_NSPAWN_USER_ARGS` variable inside the config file.
+If you're not using host networking, and you're not using the `--resolv-conf=` in case of bridge/macvlan networking, then you have to configure the DNS servers to use.
 
-### Static IP
+To get DNS servers via DHCP install and enable `resolvconf`.
-To configure a static IP with our bridge interface, we need to edit the  `/etc/systemd/network/80-container-host0.network` file. Change the [Network] section to look like this:
+
+```shell
+# Only run this inside the jail!
+
+# Temporarily fix DNS resolution,
+# otherwise we can't install packages
+echo 'nameserver 8.8.8.8' > /etc/resolv.conf
+# On debian based distro
+apt update && apt -y install resolvconf
 ```
-[Network]
+
-DHCP=false
+## References
-Address=192.168.X.XXX/24
+
-Gateway=192.168.X.X
+- [systemd-nspawn](https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html)- [Setting up Systemd-nspawn](https://www.cocode.se/linux/systemd_nspawn.html#orge360318)
-```
+- [Debian Reference - Chapter 5. Network setup](https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_hostname_resolution)
-Then restart the network interface inside the jail `systemctl restart systemd-networkd` or restart the jail by running `jlmkr stop JAILNAME && jlmkr start JAILNAME` from the TrueNAS shell. Use `ifconfig` to verify the interface is up and has the correct IP.
+- [Disabling link-local addressing](https://jerrington.me/posts/2017-08-06-systemd-nspawn-disabling-link-local-addressing.html#disabling-link-local-addressing)

docs/performance.md

@@ -1,3 +1,5 @@
+# Jailmaker Docs
+
 (Anecdotal from observations, actual measurements with resource monitor captures and wall power meter coming soon.)
 
 Kubernetes Server (TrueNAS Apps) with no apps installed:
@@ -16,6 +18,5 @@ Systemd-nspawn container (jailmaker) with 10 apps installed:
 * Idle on 7100T: ~4% / 8W
 * Idle on 10600K: ~0%
 
-
 Systemd-nspawn container (jailmaker) with 20 apps installed:
 * Idle on 10600K: ~1%

docs/storage.md

@@ -1,12 +1,14 @@
-# Default storage system
+# Jailmaker Docs
-When creating a jail, an entire Linux filesystem is created in the 'rootfs' folder within the jail's folder of the jailmaker directory E.g `/mnt/tank/vault/jailmaker/jails/jailname/rootfs`. No files from the TrueNAS host will be available. 
+
+## Default storage system
+When creating a jail, an entire Linux filesystem is created in the 'rootfs' folder within the jail's folder of the jailmaker directory. E.g. `/mnt/tank/vault/jailmaker/jails/jailname/rootfs`. No files from the TrueNAS host will be available. 
 
 Common locations for services are:
 `/home` for user accessible files
 `/var/www/` for webpages
 `/tmp` for temporary application data such as build files
 
-# Linking folders to TrueNAS folders
+## Linking folders to TrueNAS folders
 To allow file access by either the jail, another jail, or TrueNAS a bind can be made. A bind creates a link between two locations. Think of this as a portal, anything that goes in one side is visible from the other side and vice versa.
 
 Note that creating a file in the jail or TrueNAS will reflect in both binded locations, so be careful of overwrites and corruption.
@@ -21,7 +23,7 @@ And where `/jail/path/to/` is the folder you want those shared files accessible
 
 ### Example
 A use of this is making files available in a jail for it to use or serve, such as media files in Plex/Jellyfin:
-Example: `--bind='/mnt/tank/content/:/media'` will make any files inside the content dataset of the tank pool available inside the jail's /media folder. To visualise or test this you can copy some files to `/mnt/tank/content/` such as `media1.mp4`, `media2.mkv` and `photo.jpg`. Then change directory to that folder inside the jail `cd /media` and list files in that directory `ls -l` where those files should appear.
+Example: `--bind='/mnt/tank/content/:/media'` will make any files inside the content dataset of the tank pool available inside the jail's /media folder. To visualize or test this you can copy some files to `/mnt/tank/content/` such as `media1.mp4`, `media2.mkv` and `photo.jpg`. Then change directory to that folder inside the jail `cd /media` and list files in that directory `ls -l` where those files should appear.
 
 ### Warning
 Do not bind your TrueNAS system directories (`/root` `/mnt` `/dev` `/bin` `/etc` `/home` `/var` `/usr` or anything else in the root directory) to your jail as this can cause TrueNAS to lose permissions and render your TrueNAS system unusable.

docs/zfsmigration.md

@@ -1,24 +1,26 @@
-# ZFS Datasets Migration
+# Jailmaker Docs
+
+## ZFS Datasets Migration
 
 From version 1.1.4 ZFS Datasets support was added to jailmaker.
 By default starting in v1.1.4, jailmaker will create a separate dataset for each jail if possible. This allows the user to configure snapshots, rollbacks, replications etc.
 
 Jailmaker operates in dual-mode: it supports using both directories and datasets. If the 'jailmaker' directory is a dataset, it will use datasets, if it is a directory, it will use directories.
-___
-## Procedure to migrate from directories to ZFS Datasets
 
-### Stop all jails
+### Procedure to migrate from directories to ZFS Datasets
+
+#### Stop all jails
 
 `jlmkr stop jail1`
 
 `jlmkr stop jail2`
 etc..
 
-### Move/rename the 'jailmaker' directory
+#### Move/rename the 'jailmaker' directory
 
 `mv jailmaker orig_jailmaker`
 
-### Create the ZFS datasets for jailmaker
+#### Create the ZFS datasets for jailmaker
 
 Create all the required datasets via GUI or CLI.
 
@@ -44,8 +46,7 @@ zfs create mypool/jailmaker/jails/jail1
 zfs create mypool/jailmaker/jails/jail2
 ```
 
-
+#### Move the existing jail data into the newly created datasets
-### Move the existing jail data into the newly created datasets
 
 Now move all the jail data:
 
@@ -53,7 +54,7 @@ Now move all the jail data:
 
 Warning! It's important that both directories have the `/` at the end to make sure contents are copied correctly. Otherwise you may end up with `jailmaker/jailmaker`
 
-### Test everything works
+#### Test everything works
 
 If everything works, you should be able to use the `jlmkr` command directly. Try doing a `jlmkr list` to check if the jails are correctly recognized
 

jlmkr.py

@@ -4,7 +4,7 @@
 with full access to all files via bind mounts, \
 thanks to systemd-nspawn!"""
 
-__version__ = "1.3.0"
+__version__ = "1.4.0"
 
 __disclaimer__ = """USE THIS SCRIPT AT YOUR OWN RISK!
 IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS."""
@@ -43,15 +43,13 @@ docker_compatible=0
 # Turning off seccomp filtering improves performance at the expense of security
 seccomp=1
 
-# Add additional systemd-nspawn flags
+# Below you may add additional systemd-nspawn flags behind systemd_nspawn_user_args=
-# E.g. to mount host storage in the jail (--bind-ro for readonly):
+# To mount host storage in the jail, you may add: --bind='/mnt/pool/dataset:/home' 
-# --bind='/mnt/pool/dataset:/home' --bind-ro=/etc/certificates
+# To readonly mount host storage, you may add: --bind-ro=/etc/certificates
-# E.g. macvlan networking:
+# To use macvlan networking add: --network-macvlan=eno1 --resolv-conf=bind-host
-# --network-macvlan=eno1 --resolv-conf=bind-host
+# To use bridge networking add: --network-bridge=br1 --resolv-conf=bind-host
-# E.g. bridge networking:
+# Ensure to change eno1/br1 to the interface name you want to use
-# --network-bridge=br1 --resolv-conf=bind-host
+# To allow syscalls required by docker add: --system-call-filter='add_key keyctl bpf'
-# E.g. allow syscalls required by docker:
-# --system-call-filter='add_key keyctl bpf'
 systemd_nspawn_user_args=
 
 # Specify command/script to run on the HOST before starting the jail
@@ -73,10 +71,8 @@ post_stop_hook=
 distro=debian
 release=bookworm
 
-# Specify command/script to run IN THE JAIL before the first start
+# Specify command/script to run IN THE JAIL on the first start (once networking is ready in the jail)
 # Useful to install packages on top of the base rootfs
-# NOTE: this script will run in the host networking namespace and
-# ignores all systemd_nspawn_user_args such as bind mounts
 initial_setup=
 # initial_setup=bash -c 'apt-get update && apt-get -y upgrade'
 
@@ -545,55 +541,6 @@ def start_jail(jail_name):
 
     seccomp = config.my_getboolean("seccomp")
 
-    # Handle initial setup
-    initial_setup = config.my_get("initial_setup")
-
-    # Alternative method to setup on first boot:
-    # https://www.undrground.org/2021/01/25/adding-a-single-run-task-via-systemd/
-    # If there's no machine-id, then this the first time the jail is started
-    if initial_setup and not os.path.exists(
-        os.path.join(jail_rootfs_path, "etc/machine-id")
-    ):
-        initial_setup_file = None
-
-        if initial_setup.startswith("#!"):
-            # Write a script file and call that
-            initial_setup_file = os.path.abspath(
-                os.path.join(jail_path, ".initial_setup")
-            )
-            print(initial_setup, file=open(initial_setup_file, "w"))
-            stat_chmod(initial_setup_file, 0o700)
-            cmd = [
-                "systemd-nspawn",
-                "-q",
-                "-D",
-                jail_rootfs_path,
-                f"--bind-ro={initial_setup_file}:/root/initial_startup",
-                "/root/initial_startup",
-            ]
-        else:
-            # Run the command directly if it doesn't start with a shebang
-            cmd = [
-                "systemd-nspawn",
-                "-q",
-                "-D",
-                jail_rootfs_path,
-                *shlex.split(initial_setup),
-            ]
-
-        returncode = subprocess.run(cmd).returncode
-
-        # Cleanup the initial_setup_file
-        if initial_setup_file:
-            Path(initial_setup_file).unlink(missing_ok=True)
-
-        if returncode != 0:
-            eprint("Failed to run initial setup:")
-            eprint(initial_setup)
-            eprint()
-            eprint("Abort starting jail.")
-            return returncode
-
     systemd_run_additional_args = [
         f"--unit={SYMLINK_NAME}-{jail_name}",
         f"--working-directory=./{jail_path}",
@@ -605,6 +552,24 @@ def start_jail(jail_name):
         f"--directory={JAIL_ROOTFS_NAME}",
     ]
 
+    # The systemd-nspawn manual explicitly mentions:
+    # Device nodes may not be created
+    # https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html
+    # This means docker images containing device nodes can't be pulled
+    # https://github.com/moby/moby/issues/35245
+    #
+    # The solution is to use DevicePolicy=auto
+    # https://github.com/kinvolk/kube-spawn/pull/328
+    #
+    # DevicePolicy=auto is the default for systemd-run and allows access to all devices
+    # as long as we don't add any --property=DeviceAllow= flags
+    # https://manpages.debian.org/bookworm/systemd/systemd.resource-control.5.en.html
+    #
+    # We can now successfully run:
+    # mknod /dev/port c 1 4
+    # Or pull docker images containing device nodes:
+    # docker pull oraclelinux@sha256:d49469769e4701925d5145c2676d5a10c38c213802cf13270ec3a12c9c84d643
+
     if config.my_getboolean("docker_compatible"):
         eprint("WARNING: DEPRECATED OPTION")
         eprint(
@@ -704,23 +669,28 @@ def start_jail(jail_name):
             "--setenv=SYSTEMD_SECCOMP=0",
         ]
 
-    # The systemd-nspawn manual explicitly mentions:
+    initial_setup = False
-    # Device nodes may not be created
+
-    # https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html
+    # If there's no machine-id, then this the first time the jail is started
-    # This means docker images containing device nodes can't be pulled
+    if not os.path.exists(os.path.join(jail_rootfs_path, "etc/machine-id")) and (
-    # https://github.com/moby/moby/issues/35245
+        initial_setup := config.my_get("initial_setup")
-    #
+    ):
-    # The solution is to use DevicePolicy=auto
+        if not initial_setup.startswith("#!"):
-    # https://github.com/kinvolk/kube-spawn/pull/328
+            initial_setup = "#!/bin/sh\n" + initial_setup
-    #
+
-    # DevicePolicy=auto is the default for systemd-run and allows access to all devices
+        initial_setup_file_jailed_path = "/root/jlmkr-initial-setup"
-    # as long as we don't add any --property=DeviceAllow= flags
+        initial_setup_file_host_path = os.path.abspath(
-    # https://manpages.debian.org/bookworm/systemd/systemd.resource-control.5.en.html
+            jail_rootfs_path + initial_setup_file_jailed_path
-    #
+        )
-    # We can now successfully run:
+
-    # mknod /dev/port c 1 4
+        # Write a script file to call during initial setup
-    # Or pull docker images containing device nodes:
+        print(initial_setup, file=open(initial_setup_file_host_path, "w"))
-    # docker pull oraclelinux@sha256:d49469769e4701925d5145c2676d5a10c38c213802cf13270ec3a12c9c84d643
+        stat_chmod(initial_setup_file_host_path, 0o700)
+
+        # Ensure the jail init system is ready before we start the initial_setup
+        systemd_nspawn_additional_args += [
+            "--notify-ready=yes",
+        ]
 
     cmd = [
         "systemd-run",
@@ -755,6 +725,44 @@ def start_jail(jail_name):
             )
         )
 
+        return returncode
+
+    # Handle initial setup after jail is up and running (for the first time)
+    if initial_setup:
+        print("About to run the initial setup.")
+        print("Waiting for networking in the jail to be ready.")
+        print("Please wait (this may take 90s in case of bridge networking)...")
+        returncode = exec_jail(
+            jail_name,
+            [
+                "--",
+                "systemd-run",
+                f"--unit={os.path.basename(initial_setup_file_jailed_path)}",
+                "--quiet",
+                "--pipe",
+                "--wait",
+                "--service-type=exec",
+                "--property=After=network-online.target",
+                "--property=Wants=network-online.target",
+                initial_setup_file_jailed_path,
+            ],
+        )
+
+        # Cleanup the initial_setup_file_host_path
+        if initial_setup_file_host_path:
+            Path(initial_setup_file_host_path).unlink(missing_ok=True)
+
+        if returncode != 0:
+            eprint("Tried to run the following commands inside the jail:")
+            eprint(initial_setup)
+            eprint()
+            eprint(
+                f"""{RED}{BOLD}Failed to run initial setup... you may want to stop and remove the jail and try again.{NORMAL}"""
+            )
+            return returncode
+        else:
+            print(f"Done with initial setup of jail {jail_name}!")
+
     return returncode
 
 
@@ -922,11 +930,12 @@ def get_zfs_dataset(path):
     """
     Get ZFS dataset path.
     """
+
     def clean_field(field):
         # Put back spaces which were encoded
         # https://github.com/openzfs/zfs/issues/11182
-        return field.replace('\\040', ' ')
+        return field.replace("\\040", " ")
-    
+
     path = os.path.realpath(path)
     with open("/proc/mounts", "r") as f:
         for line in f:
@@ -1025,11 +1034,14 @@ def get_text_editor():
         if editor := os.environ.get(key):
             return shutil.which(editor)
 
-    return get_from_environ("VISUAL") \
+    return (
-        or get_from_environ("EDITOR") \
+        get_from_environ("VISUAL")
-        or shutil.which("editor") \
+        or get_from_environ("EDITOR")
-        or shutil.which("/usr/bin/editor") \
+        or shutil.which("editor")
+        or shutil.which("/usr/bin/editor")
         or "nano"
+    )
+
 
 def interactive_config():
     config = KeyValueParser()
@@ -1517,9 +1529,7 @@ def edit_jail(jail_name):
 
     jail_config_path = get_jail_config_path(jail_name)
 
-    returncode = subprocess.run(
+    returncode = subprocess.run([get_text_editor(), jail_config_path]).returncode
-        [get_text_editor(), jail_config_path]
-    ).returncode
 
     if returncode != 0:
         eprint(f"An error occurred while editing {jail_config_path}.")

templates/docker/README.md

@@ -2,4 +2,4 @@
 
 ## Setup
 
-Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/docker/config mydockerjail`.
+Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/docker/config mydockerjail`. If you want the `nvidia-container-toolkit` to be installed, ensure you set `gpu_passthrough_nvidia=1` when creating the jail.

templates/docker/config

@@ -6,10 +6,10 @@ seccomp=1
 
 # Use macvlan networking to provide an isolated network namespace,
 # so docker can manage firewall rules
-# Alternatively use --network-bridge=br1 instead of --network-macvlan
+# Alternatively use --network-macvlan=eno1 instead of --network-bridge
 # Ensure to change eno1/br1 to the interface name you want to use
 # You may want to add additional options here, e.g. bind mounts
-systemd_nspawn_user_args=--network-macvlan=eno1
+systemd_nspawn_user_args=--network-bridge=br1
     --resolv-conf=bind-host
     --system-call-filter='add_key keyctl bpf'
 
@@ -29,8 +29,8 @@ release=bookworm
 
 # Install docker inside the jail:
 # https://docs.docker.com/engine/install/debian/#install-using-the-repository
-# NOTE: this script will run in the host networking namespace and ignores
+# Will also install the NVIDIA Container Toolkit if gpu_passthrough_nvidia=1 during initial setup
-# all systemd_nspawn_user_args such as bind mounts
+# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
 initial_setup=#!/usr/bin/bash
     set -euo pipefail
 
@@ -43,8 +43,26 @@ initial_setup=#!/usr/bin/bash
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
     $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
     tee /etc/apt/sources.list.d/docker.list > /dev/null
+    
     apt-get update
     apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+    
+    # The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
+    if [ -f /usr/bin/nvidia-smi ]; then
+        curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey -o /etc/apt/keyrings/nvidia.asc
+        chmod a+r /etc/apt/keyrings/nvidia.asc
+        curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
+        sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia.asc] https://#g' | \
+        tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
+
+        apt-get update
+        apt-get install -y nvidia-container-toolkit
+
+        nvidia-ctk runtime configure --runtime=docker
+        systemctl restart docker
+    fi
+
+    docker info
 
 # You generally will not need to change the options below
 systemd_run_default_args=--property=KillMode=mixed

templates/incus/README.md

@@ -8,11 +8,10 @@
 
 Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/incus/config myincusjail`.
 
-Unfortunately incus doesn't want to install from the `initial_setup` script inside the config file. So we manually finish the setup by running the following after creating and starting the jail:
+We manually finish the setup by running the following after creating and starting the jail:
 
 ```bash
-jlmkr exec myincusjail bash -c 'apt-get -y install incus incus-ui-canonical &&
+jlmkr exec myincusjail bash -c 'incus admin init'
-    incus admin init'
 ```    
 
 Follow [First steps with Incus](https://linuxcontainers.org/incus/docs/main/tutorial/first_steps/).

templates/incus/config

@@ -3,15 +3,16 @@ startup=0
 gpu_passthrough_intel=0
 gpu_passthrough_nvidia=0
 # Turning off seccomp filtering improves performance at the expense of security
-seccomp=1
+# TODO: don't disable seccomp but specify which syscalls should be allowed
+seccomp=0
 
 # Use macvlan networking to provide an isolated network namespace,
 # so incus can manage firewall rules
-# Alternatively use --network-bridge=br1 instead of --network-macvlan
+# Alternatively use --network-macvlan=eno1 instead of --network-bridge
 # Ensure to change eno1/br1 to the interface name you want to use
 # You may want to add additional options here, e.g. bind mounts
 # TODO: don't use --capability=all but specify only the required capabilities
-systemd_nspawn_user_args=--network-macvlan=eno1
+systemd_nspawn_user_args=--network-bridge=br1
     --resolv-conf=bind-host
     --capability=all
     --bind=/dev/fuse
@@ -36,8 +37,6 @@ release=bookworm
 
 # Install incus according to:
 # https://github.com/zabbly/incus#installation
-# NOTE: this script will run in the host networking namespace and ignores
-# all systemd_nspawn_user_args such as bind mounts
 initial_setup=#!/usr/bin/bash
     set -euo pipefail
     apt-get update && apt-get -y install curl
@@ -54,6 +53,7 @@ initial_setup=#!/usr/bin/bash
 
     EOF'
     apt-get update
+    apt-get -y install incus incus-ui-canonical
 
 # You generally will not need to change the options below
 systemd_run_default_args=--property=KillMode=mixed

templates/lxd/README.md

@@ -8,20 +8,7 @@
 
 Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/lxd/config mylxdjail`.
 
-Unfortunately snapd doesn't want to install from the `initial_setup` script inside the config file. So we manually finish the setup by running the following after creating and starting the jail:
+We manually finish the setup by running the command below after creating and starting the jail. Choose the `dir` storage backend during `lxd init` and answer `yes` to "Would you like the LXD server to be available over the network?"
-
-```bash
-# Repeat listing the jail until you see it has an IPv4 address
-jlmkr list
-
-# Install packages
-jlmkr exec mylxdjail bash -c 'apt-get update &&
-    apt-get install -y --no-install-recommends snapd &&
-    snap install lxd'
-
-```
-
-Choose the `dir` storage backend during `lxd init` and answer `yes` to "Would you like the LXD server to be available over the network?"
 
 ```bash
 jlmkr exec mylxdjail bash -c 'lxd init &&

templates/lxd/config

@@ -3,11 +3,12 @@ startup=0
 gpu_passthrough_intel=0
 gpu_passthrough_nvidia=0
 # Turning off seccomp filtering improves performance at the expense of security
-seccomp=1
+# TODO: don't disable seccomp but specify which syscalls should be allowed
+seccomp=0
 
 # Use macvlan networking to provide an isolated network namespace,
 # so lxd can manage firewall rules
-# Alternatively use --network-bridge=br1 instead of --network-macvlan
+# Alternatively use --network-macvlan=eno1 instead of --network-bridge
 # Ensure to change eno1/br1 to the interface name you want to use
 # You may want to add additional options here, e.g. bind mounts
 # TODO: don't use --capability=all but specify only the required capabilities
@@ -34,12 +35,13 @@ pre_start_hook=#!/usr/bin/bash
 distro=ubuntu
 release=jammy
 
-# NOTE: this script will run in the host networking namespace and ignores
-# all systemd_nspawn_user_args such as bind mounts
 initial_setup=#!/usr/bin/bash
     set -euo pipefail
     # https://discuss.linuxcontainers.org/t/snap-inside-privileged-lxd-container/13691/8
     ln -sf /bin/true /usr/local/bin/udevadm
+    apt-get update
+    apt-get install -y --no-install-recommends snapd
+    snap install lxd
 
 # You generally will not need to change the options below
 systemd_run_default_args=--property=KillMode=mixed

templates/podman/README.md

@@ -46,6 +46,10 @@ usermod --del-subuids 0-4294967295 --del-subgids 0-4294967295 rootless
 # Set a specific range, so it fits inside the number of available UIDs
 usermod --add-subuids 65536-131071 --add-subgids 65536-131071 rootless
 
+# Add the required capabilities to the `newuidmap` and `newgidmap` binaries
+setcap cap_setuid+eip /usr/bin/newuidmap
+setcap cap_setgid+eip /usr/bin/newgidmap
+
 # Check the assigned range
 cat /etc/subuid
 # Check the available range
@@ -120,4 +124,7 @@ Resources mentioning `add_key keyctl bpf`
 Resources mentioning `@keyring`
 - https://github.com/systemd/systemd/issues/17606
 - https://github.com/systemd/systemd/blob/1c62c4fe0b54fb419b875cb2bae82a261518a745/src/shared/seccomp-util.c#L604
-`@keyring` also includes `request_key` but doesn't include `bpf`
+`@keyring` also includes `request_key` but doesn't include `bpf`
+Resources mentioning `cap_setuid+eip`, `cap_setgid+eip`, `newuidmap` and `newgidmap`
+- https://github.com/containers/podman/issues/2788#issuecomment-1016301663
+- https://github.com/containers/podman/issues/12637#issuecomment-996524341

templates/podman/config

@@ -6,10 +6,10 @@ seccomp=1
 
 # Use macvlan networking to provide an isolated network namespace,
 # so podman can manage firewall rules
-# Alternatively use --network-bridge=br1 instead of --network-macvlan
+# Alternatively use --network-macvlan=eno1 instead of --network-bridge
 # Ensure to change eno1/br1 to the interface name you want to use
 # You may want to add additional options here, e.g. bind mounts
-systemd_nspawn_user_args=--network-macvlan=eno1
+systemd_nspawn_user_args=--network-bridge=br1
     --resolv-conf=bind-host
     --system-call-filter='add_key keyctl bpf'
 
@@ -28,16 +28,9 @@ distro=fedora
 release=39
 
 # Install podman inside the jail
-# NOTE: this script will run in the host networking namespace and ignores
-# all systemd_nspawn_user_args such as bind mounts
 initial_setup=#!/usr/bin/bash
     set -euo pipefail
     dnf -y install podman
-    # Add the required capabilities to the `newuidmap` and `newgidmap` binaries
-    # https://github.com/containers/podman/issues/2788#issuecomment-1016301663
-    # https://github.com/containers/podman/issues/12637#issuecomment-996524341
-    setcap cap_setuid+eip /usr/bin/newuidmap
-    setcap cap_setgid+eip /usr/bin/newgidmap
 
 # You generally will not need to change the options below
 systemd_run_default_args=--property=KillMode=mixed