Merge pull request #167 from Jip-Hop/develop

v1.4.0
This commit is contained in:
Jip-Hop 2024-05-09 14:57:04 +02:00 committed by GitHub
commit bc8284edb8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
19 changed files with 299 additions and 302 deletions

View File

@ -1,61 +0,0 @@
# Jailmaker
## Advanced Networking
These are notes on advanced networking setup you may want to try. Contributions are welcome!
### Bridge Networking
As an alternative to the default host networking mode, you may want to connect to a bridge interface instead and let the jail obtain its IP address via DHCP (although you may have to be patient for up to 20 seconds after the jail started for networking to work, [assigning the IP address is somehow slow](https://github.com/Jip-Hop/jailmaker/issues/7)).
[This YouTube video](https://www.youtube.com/watch?v=7clQw132w58) may be helpful when setting up the bridge interface. Note: You may lock yourself out... It may take several tries... TrueNAS is a bit picky when switching IP addresses and toggling DHCP. May be helpful to connect a monitor and keyboard to the NAS and use `/etc/netcli` to reset the networking interface. Kept bothering with "Register Default Gateway" warning... I just clicked Cancel.
Add the `--network-bridge=br1 --resolv-conf=bind-host` systemd-nspawn flag when asked for `Additional flags` during jail creation, or set it post-creation by [editing](./README.md#edit-jail-config) the `SYSTEMD_NSPAWN_USER_ARGS` variable inside the `config` file.
The TrueNAS host and the jail will be able to communicate with each other as if the jail was just another device on the LAN. It will use the same DNS servers as the TrueNAS host because the `--resolv-conf=bind-host` option bind mounts the `/etc/resolv.conf` file from the host inside the jail. If you want to use the DNS servers advertised via DHCP, then check [DNS via DHCP](#dns-via-dhcp).
To configure a **static IP** with our bridge interface, we need to edit the `80-container-host0.network` file located in `/etc/systemd/network`. Change the `[Network]` section to look like this:
```ini
[Network]
DHCP=false
Address=192.168.0.12/24
Gateway=192.168.0.1
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge
```
Then restart the `systemd-networkd` service and check your network configuration.
```shell
systemctl restart systemd-networkd
systemctl status systemd-networkd
ifconfig
```
### Macvlan Networking
To setup Macvlan Networking you may follow the [Bridge Networking](#bridge-networking) section, but skip the setup of a bridge interface and use these flags instead: `--network-macvlan=eno1 --resolv-conf=bind-host`. By default the TrueNAS host and jail will not be able to communicate with each other via the network if Macvlan Networking mode is used. If that's required it would be better to use [Bridge Networking](#bridge-networking).
### DNS via DHCP
If you're not using host networking, and you're not using the `--resolv-conf=` in case of bridge/macvlan networking, then you have to configure the DNS servers to use.
To get DNS servers via DHCP install and enable `resolvconf`.
```shell
# Only run this inside the jail!
# Temporarily fix DNS resolution,
# otherwise we can't install packages
echo 'nameserver 8.8.8.8' > /etc/resolv.conf
# On debian based distro
apt update && apt -y install resolvconf
```
## References
- [systemd-nspawn](https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html)- [Setting up Systemd-nspawn](https://www.cocode.se/linux/systemd_nspawn.html#orge360318)
- [Debian Reference - Chapter 5. Network setup](https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_hostname_resolution)
- [Disabling link-local addressing](https://jerrington.me/posts/2017-08-06-systemd-nspawn-disabling-link-local-addressing.html#disabling-link-local-addressing)

View File

@ -90,13 +90,15 @@ jlmkr start myjail
### List Jails
See list of jails (including running, startup state, GPU passthrough, distro, and IP).
```shell
jlmkr list
```
### Execute Command in Jail
You may want to execute a command inside a jail, for example from a shell script or a CRON job. The example below executes the `env` command inside the jail.
You may want to execute a command inside a jail, for example manually from the TrueNAS shell, a shell script or a CRON job. The example below executes the `env` command inside the jail.
```shell
jlmkr exec myjail env
@ -118,6 +120,8 @@ Once you've created a jail, it will exist in a directory inside the `jails` dir
### Remove Jail
Delete a jail and remove it's files (requires confirmation).
```shell
jlmkr remove myjail
```
@ -136,6 +140,8 @@ jlmkr restart myjail
### Jail Shell
Switch into the jail's shell.
```shell
jlmkr shell myjail
```
@ -148,6 +154,8 @@ jlmkr status myjail
### Jail Logs
View a jail's logs.
```shell
jlmkr log myjail
```
@ -158,17 +166,21 @@ Expert users may use the following additional commands to manage jails directly:
## Networking
By default the jail will have full access to the host network. No further setup is required. You may download and install additional packages inside the jail. Note that some ports are already occupied by TrueNAS SCALE (e.g. 443 for the web interface), so your jail can't listen on these ports. This is inconvenient if you want to host some services (e.g. traefik) inside the jail. To workaround this issue when using host networking, you may disable DHCP and add several static IP addresses (Aliases) through the TrueNAS web interface. If you setup the TrueNAS web interface to only listen on one of these IP addresses, the ports on the remaining IP addresses remain available for the jail to listen on.
By default a jails will use the same networking namespace, with access to all (physical) interfaces the TrueNAS host has access to. No further setup is required. You may download and install additional packages inside the jail. Note that some ports are already occupied by TrueNAS SCALE (e.g. 443 for the web interface), so your jail can't listen on these ports.
See [Advanced Networking](./NETWORKING.md) for more.
Depending on the service this may be o.k. For example Home Assistant will bind to port 8123, leaving the 80 and 443 ports free from clashes for the TrueNAS web interface. You can then either connect to the service on 8123, or use a reverse proxy such as traefik.
But clashes may happen if you want some services (e.g. traefik) inside the jail to listen on port 443. To workaround this issue when using host networking, you may disable DHCP and add several static IP addresses (Aliases) through the TrueNAS web interface. If you setup the TrueNAS web interface to only listen on one of these IP addresses, the ports on the remaining IP addresses remain available for the jail to listen on.
See [the networking docs](./docs/network.md) for more advanced options (bridge and macvlan networking).
## Docker
Using the [docker config template](./templates/docker/README.md) is recommended if you want to run docker inside the jail. You may of course manually install docker inside a jail. But keep in mind that you need to add `--system-call-filter='add_key keyctl bpf'` (or disable seccomp filtering). It is [not recommended to use host networking for a jail in which you run docker](https://github.com/Jip-Hop/jailmaker/issues/119). Docker needs to manage iptables rules, which it can safely do in its own networking namespace (when using [bridge or macvlan networking](./NETWORKING.md) for the jail).
Using the [docker config template](./templates/docker/README.md) is recommended if you want to run docker inside the jail. You may of course manually install docker inside a jail. But keep in mind that you need to add `--system-call-filter='add_key keyctl bpf'` (or disable seccomp filtering). It is [not recommended to use host networking for a jail in which you run docker](https://github.com/Jip-Hop/jailmaker/issues/119). Docker needs to manage iptables rules, which it can safely do in its own networking namespace (when using [bridge or macvlan networking](./docs/network.md) for the jail).
## Documentation
Additional documentation contributed by the community can be found in [the docs directory](./docs/).
Additional documentation can be found in [the docs directory](./docs/) (contributions are welcome!).
## Comparison
@ -178,16 +190,6 @@ TODO: write comparison between systemd-nspawn (without `jailmaker`), LXC, VMs, D
The rootfs image `jlmkr.py` downloads comes from the [Linux Containers Image server](https://images.linuxcontainers.org). These images are made for LXC. We can use them with systemd-nspawn too, although not all of them work properly. For example, the `alpine` image doesn't work well. If you stick with common systemd based distros (Debian, Ubuntu, Arch Linux...) you should be fine.
## Tips & Tricks
### Colorized bash prompt
To visually distinguish between a root shell inside the jail and a root shell outside the jail, it's possible to colorize the shell prompt. When using a debian jail with the bash shell, you may run the following command to get a yellow prompt inside the jail (will be activated the next time you run `jlmkr shell myjail`):
```bash
echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;33m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc
```
## Filing Issues and Community Support
When in need of help or when you think you've found a bug in jailmaker, [please start with reading this](https://github.com/Jip-Hop/jailmaker/discussions/135).

View File

@ -1,3 +1,5 @@
Welcome to the jailmaker wiki!
# Jailmaker Docs
Welcome to the Jailmaker Docs!
Use the sidebar to navigate the topics.

View File

@ -1,4 +1,8 @@
# User Management
# Jailmaker Docs
Anything described on this page is completely optional. You do NOT need to do anything of this in order to start using jailmaker.
## User Management
The root user (also known as the superuser or su) can access any file, make system changes, and lots of room for security vulnerabilities.
For this reason you should aspire to run services as a non-root user.
@ -10,28 +14,54 @@ Where username can be anything, but should reflect the service/jail's name for d
Then a password should be created as some commands require a non-blank password to be inserted:
`passwd USERNAME`
If you want the ability to run commands as root, add the user to the sudo group
`usermod -aG sudo USERNAME`
If you want the ability to run commands as root, add the user to the sudo group:
```sh
usermod -aG sudo USERNAME
```
This WILL require a non-blank password, and any command run with sudo will be run as root not as the user. But it saves time compared to switching users to root to install/change things then switching back.
### Switch to user
`su -l USERNAME`
```sh
su -l USERNAME
```
### Put a password on Root
While logged in as root run `passwd`
# Common tweaks
While logged in as root run `passwd`.
## Common Tweaks
### Update repository list
`sudo apt update`
```sh
sudo apt update
```
### Install common services
`sudo apt install nano wget curl git`
```sh
sudo apt install nano wget curl git
```
### Set Static IP
See `Networking`
See [Networking](./network.md)
### Colorized bash prompt
To visually distinguish between a root shell inside the jail and a root shell outside the jail, it's possible to colorize the shell prompt. When using a debian jail with the bash shell, you may run the following command **inside the jail** to get a yellow prompt inside the jail (will be activated the next time you run `jlmkr shell myjail`):
```bash
echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;33m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc
```
### Install Docker
```
It's advised to use the [docker config template](../templates/docker/README.md). But you can install it manually like this as well:
```sh
apt install curl && cd /tmp && curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh && cd ~ && docker
```

View File

@ -1,35 +0,0 @@
Create a jail
`jlmkr create JAILNAME`
Start a jail
`jlmkr start JAILNAME`
Stop a jail
`jlmkr stop JAILNAME`
Check jail status
`jlmkr status JAILNAME`
Delete a jail and remove it's files (requires confirmation)
`jlmkr remove JAILNAME`
See list of jails (including running, non running, distro, startup state, and IP)
`jlmkr list`
See list of running jails
`machinectl list`
Execute a command inside a jail from the TrueNAS shell
`jlmkr exec JAILNAME COMMAND`
Execute a bash command inside a jail from the TrueNAS shell
`jlmkr exec JAILNAME bash -c 'BASHCOMMAND'`
Switch into the jail's shell
`machinectl shell JAILNAME`
View a jail's logs
`jlmkr log JAILNAME`
Edit a jail's config
`jlmkr edit JAILNAME`

View File

@ -1,12 +1,14 @@
# TrueNAS Compatibility
# Jailmaker Docs
## TrueNAS Compatibility
| | |
|---|---|
|TrueNAS Core|❌|
|TrueNAS 22.12|✅|
|TrueNAS 23.10|✅|
|TrueNAS 24.04 nightly|✅|
|TrueNAS 24.04|✅|
# Distro Compatibility
## Distro Compatibility
| | |
|---|---|
|Debian 11 Bullseye|✅|

View File

@ -1,24 +1,60 @@
# Host Passthrough (Default network configuration)
By default jails will use the same physical interface as the TrueNAS host. If a service attempts to bind to port 80 or 443, it will either fail or render both the service and TrueNAS unavailable.
### Flaws
Depending on the service this may be ok, for example Home Assistant will bind to port 8123, leaving the 80 and 443 ports free from clashes for the TrueNAS web interface. You can then either connect to the service with the port, or use a reverse proxy such as [nginx](https://www.nginx.com/#).
### Setup
No configuration is necessary
# Jailmaker Docs
# MAC VLAN Virtual Interface
Some services require the use of port 80 or 443, or would benefit from a separate IP. For these situations the easiest network configuration is the MAC VLAN configuration. This creates a virtual interface with its own separate randomly generated MAC address and IP.
The default config uses DHCP by default, but can easily be set to a Static IP.
### Flaws
Any services in the jail cannot communicate with the direct host (TrueNAS). The jail can communicate with any other jail or device on the network, besides TrueNAS. This may or not be a benefit (security) or disadvantage (no communication) depending on your service.
### Setup
Add the following argument to the "additional flags" prompt of jail creation or the "systemd_nspawn_user_arguments" line of the jail config file:
## Host Networking
[Notes on the default host networking are in the main README.md file](../README.md#networking).
## Bridge Networking
As an alternative to the default host networking mode, you may want to connect to a bridge interface instead and let the jail obtain its IP address via DHCP.
[![TrueNAS Scale: Setting up a Static IP and Network Bridge // Access NAS host from VM - YouTube Video](https://img.youtube.com/vi/uPkoeWUfiHU/0.jpg)<br>Watch on YouTube](https://www.youtube.com/watch?v=uPkoeWUfiHU "TrueNAS Scale: Setting up a Static IP and Network Bridge // Access NAS host from VM - YouTube Video")
The above YouTube video may be helpful when setting up the bridge interface.
### Bridge Flaws
This type of interface takes much longer to set up both in complexity and wait time (you may have to be patient for up to 60 seconds after the jail started for networking to work, [assigning the IP address via DHCP is somehow slow](https://github.com/Jip-Hop/jailmaker/issues/7)). Furthermore, if the configuration is not correct it can render your TrueNAS inaccessible via ssh or the web interface, necessitating a reset using a keyboard and monitor plugged into the TrueNAS server and use `/etc/netcli` to reset the networking interface.
### Bridge Setup
Add the `--network-bridge=br1 --resolv-conf=bind-host` systemd-nspawn flag when asked for `Additional flags` during jail creation, or set it post-creation by [editing](./README.md#edit-jail-config) the `SYSTEMD_NSPAWN_USER_ARGS` variable inside the `config` file.
The TrueNAS host and the jail will be able to communicate with each other as if the jail was just another device on the LAN. It will use the same DNS servers as the TrueNAS host because the `--resolv-conf=bind-host` option bind mounts the `/etc/resolv.conf` file from the host inside the jail. If you want to use the DNS servers advertised via DHCP, then check [DNS via DHCP](#dns-via-dhcp).
### Bridge Static IP
To configure a static IP with our bridge interface, we need to edit the `/etc/systemd/network/80-container-host0.network` file. Change the [Network] section to look like this:
```ini
[Network]
DHCP=false
Address=192.168.0.12/24
Gateway=192.168.0.1
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge
```
--network-macvlan=eno1 --resolv-conf=bind-host
Then restart the `systemd-networkd` service and check your network configuration.
```shell
systemctl restart systemd-networkd
systemctl status systemd-networkd
ifconfig
```
### Setting a Static IP
To set a Static IP you need to disable DHCP in the macvlan config file `/etc/systemd/network/mv-dhcp.network`
You can do this with a network client like WinSCP by navigating into the jail's filesystem then the path above, or by using a text editing program like nano by running `nano /etc/systemd/network/mv-dhcp.network` in the jail's shell.
## Macvlan Networking
Some services require the use of port 80 or 443, or would benefit from a separate IP. For these situations the easiest network configuration is the MAC VLAN configuration. This creates a virtual interface with its own separate randomly generated MAC address and IP. The default config uses DHCP by default, but can easily be set to a Static IP.
### Macvlan Flaws
Any services in the jail cannot communicate with the direct host (TrueNAS). The jail can communicate with any other jail or device on the network, besides TrueNAS or VMs hosted on TrueNAS. This may be a benefit (security) or disadvantage (no communication) depending on your service. If that's required it would be better to use [Bridge Networking](#bridge-networking).
### Macvlan Setup
Add the following argument to the "additional flags" prompt of jail creation or the "systemd_nspawn_user_arguments" line of the jail config file: `--network-macvlan=eno1 --resolv-conf=bind-host`. Where eno1 is the name of your physical network interface.
### Macvlan Static IP
To set a Static IP you need to disable DHCP in the macvlan config file `/etc/systemd/network/mv-dhcp.network`. You can do this with a network client like WinSCP by navigating into the jail's filesystem then the path above, or by using a text editing program like nano by running `nano /etc/systemd/network/mv-dhcp.network` in the jail's shell.
The DHCP in [Network] needs to be set to false, an Address (static IP) needs to be added, a Gateway needs to be defined (e.g your router such as 192.168.0.1) and the entire DHCP section needs to be removed.
@ -35,23 +71,24 @@ Gateway=192.168.X.X
```
Then restart the network interface inside the jail `systemctl restart systemd-networkd` or restart the jail by running `jlmkr stop JAILNAME && jlmkr start JAILNAME` from the TrueNAS shell. Use `ifconfig` to verify the interface is up and has the correct IP.
# Passthrough a TrueNAS Bridge Interface
By creating a network bridge in the TrueNAS Network page you can bridge the incoming physical network interface to a virtual interface that can be passed to the jail. This type of interface has the benefits of a MAC VLAN interface without the flaws (host to jail networking). Once working the virtual interface can either be assigned a static IP or obtain one automatically via DHCP.
### Flaws
This type of interface takes much longer to set up both in complexity and wait time as there is a current flaw in which HDCP can take between 10 seconds and a minute.
Furthermore, if the configuration is not correct it can render your TrueNAS inaccessible via ssh, necessitating a reset using a keyboard and monitor plugged into the TrueNAS server.
### Setup
[TrueNAS Bridge interface guide](https://www.youtube.com/watch?v=7clQw132w58)
May be helpful to connect a monitor and keyboard to the NAS and use /etc/netcli to reset the networking interface. Kept bothering with "Register Default Gateway" warning... I just clicked Cancel.
## DNS via DHCP
Add the `--network-bridge=br1 --resolv-conf=bind-host` flag when asked for additional flags during jail creation, or set it post-creation by editing the `SYSTEMD_NSPAWN_USER_ARGS` variable inside the config file.
If you're not using host networking, and you're not using the `--resolv-conf=` in case of bridge/macvlan networking, then you have to configure the DNS servers to use.
### Static IP
To configure a static IP with our bridge interface, we need to edit the `/etc/systemd/network/80-container-host0.network` file. Change the [Network] section to look like this:
To get DNS servers via DHCP install and enable `resolvconf`.
```shell
# Only run this inside the jail!
# Temporarily fix DNS resolution,
# otherwise we can't install packages
echo 'nameserver 8.8.8.8' > /etc/resolv.conf
# On debian based distro
apt update && apt -y install resolvconf
```
[Network]
DHCP=false
Address=192.168.X.XXX/24
Gateway=192.168.X.X
```
Then restart the network interface inside the jail `systemctl restart systemd-networkd` or restart the jail by running `jlmkr stop JAILNAME && jlmkr start JAILNAME` from the TrueNAS shell. Use `ifconfig` to verify the interface is up and has the correct IP.
## References
- [systemd-nspawn](https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html)- [Setting up Systemd-nspawn](https://www.cocode.se/linux/systemd_nspawn.html#orge360318)
- [Debian Reference - Chapter 5. Network setup](https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_hostname_resolution)
- [Disabling link-local addressing](https://jerrington.me/posts/2017-08-06-systemd-nspawn-disabling-link-local-addressing.html#disabling-link-local-addressing)

View File

@ -1,3 +1,5 @@
# Jailmaker Docs
(Anecdotal from observations, actual measurements with resource monitor captures and wall power meter coming soon.)
Kubernetes Server (TrueNAS Apps) with no apps installed:
@ -16,6 +18,5 @@ Systemd-nspawn container (jailmaker) with 10 apps installed:
* Idle on 7100T: ~4% / 8W
* Idle on 10600K: ~0%
Systemd-nspawn container (jailmaker) with 20 apps installed:
* Idle on 10600K: ~1%

View File

@ -1,12 +1,14 @@
# Default storage system
When creating a jail, an entire Linux filesystem is created in the 'rootfs' folder within the jail's folder of the jailmaker directory E.g `/mnt/tank/vault/jailmaker/jails/jailname/rootfs`. No files from the TrueNAS host will be available.
# Jailmaker Docs
## Default storage system
When creating a jail, an entire Linux filesystem is created in the 'rootfs' folder within the jail's folder of the jailmaker directory. E.g. `/mnt/tank/vault/jailmaker/jails/jailname/rootfs`. No files from the TrueNAS host will be available.
Common locations for services are:
`/home` for user accessible files
`/var/www/` for webpages
`/tmp` for temporary application data such as build files
# Linking folders to TrueNAS folders
## Linking folders to TrueNAS folders
To allow file access by either the jail, another jail, or TrueNAS a bind can be made. A bind creates a link between two locations. Think of this as a portal, anything that goes in one side is visible from the other side and vice versa.
Note that creating a file in the jail or TrueNAS will reflect in both binded locations, so be careful of overwrites and corruption.
@ -21,7 +23,7 @@ And where `/jail/path/to/` is the folder you want those shared files accessible
### Example
A use of this is making files available in a jail for it to use or serve, such as media files in Plex/Jellyfin:
Example: `--bind='/mnt/tank/content/:/media'` will make any files inside the content dataset of the tank pool available inside the jail's /media folder. To visualise or test this you can copy some files to `/mnt/tank/content/` such as `media1.mp4`, `media2.mkv` and `photo.jpg`. Then change directory to that folder inside the jail `cd /media` and list files in that directory `ls -l` where those files should appear.
Example: `--bind='/mnt/tank/content/:/media'` will make any files inside the content dataset of the tank pool available inside the jail's /media folder. To visualize or test this you can copy some files to `/mnt/tank/content/` such as `media1.mp4`, `media2.mkv` and `photo.jpg`. Then change directory to that folder inside the jail `cd /media` and list files in that directory `ls -l` where those files should appear.
### Warning
Do not bind your TrueNAS system directories (`/root` `/mnt` `/dev` `/bin` `/etc` `/home` `/var` `/usr` or anything else in the root directory) to your jail as this can cause TrueNAS to lose permissions and render your TrueNAS system unusable.

View File

@ -1,24 +1,26 @@
# ZFS Datasets Migration
# Jailmaker Docs
## ZFS Datasets Migration
From version 1.1.4 ZFS Datasets support was added to jailmaker.
By default starting in v1.1.4, jailmaker will create a separate dataset for each jail if possible. This allows the user to configure snapshots, rollbacks, replications etc.
Jailmaker operates in dual-mode: it supports using both directories and datasets. If the 'jailmaker' directory is a dataset, it will use datasets, if it is a directory, it will use directories.
___
## Procedure to migrate from directories to ZFS Datasets
### Stop all jails
### Procedure to migrate from directories to ZFS Datasets
#### Stop all jails
`jlmkr stop jail1`
`jlmkr stop jail2`
etc..
### Move/rename the 'jailmaker' directory
#### Move/rename the 'jailmaker' directory
`mv jailmaker orig_jailmaker`
### Create the ZFS datasets for jailmaker
#### Create the ZFS datasets for jailmaker
Create all the required datasets via GUI or CLI.
@ -44,8 +46,7 @@ zfs create mypool/jailmaker/jails/jail1
zfs create mypool/jailmaker/jails/jail2
```
### Move the existing jail data into the newly created datasets
#### Move the existing jail data into the newly created datasets
Now move all the jail data:
@ -53,7 +54,7 @@ Now move all the jail data:
Warning! It's important that both directories have the `/` at the end to make sure contents are copied correctly. Otherwise you may end up with `jailmaker/jailmaker`
### Test everything works
#### Test everything works
If everything works, you should be able to use the `jlmkr` command directly. Try doing a `jlmkr list` to check if the jails are correctly recognized

184
jlmkr.py
View File

@ -4,7 +4,7 @@
with full access to all files via bind mounts, \
thanks to systemd-nspawn!"""
__version__ = "1.3.0"
__version__ = "1.4.0"
__disclaimer__ = """USE THIS SCRIPT AT YOUR OWN RISK!
IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS."""
@ -43,15 +43,13 @@ docker_compatible=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1
# Add additional systemd-nspawn flags
# E.g. to mount host storage in the jail (--bind-ro for readonly):
# --bind='/mnt/pool/dataset:/home' --bind-ro=/etc/certificates
# E.g. macvlan networking:
# --network-macvlan=eno1 --resolv-conf=bind-host
# E.g. bridge networking:
# --network-bridge=br1 --resolv-conf=bind-host
# E.g. allow syscalls required by docker:
# --system-call-filter='add_key keyctl bpf'
# Below you may add additional systemd-nspawn flags behind systemd_nspawn_user_args=
# To mount host storage in the jail, you may add: --bind='/mnt/pool/dataset:/home'
# To readonly mount host storage, you may add: --bind-ro=/etc/certificates
# To use macvlan networking add: --network-macvlan=eno1 --resolv-conf=bind-host
# To use bridge networking add: --network-bridge=br1 --resolv-conf=bind-host
# Ensure to change eno1/br1 to the interface name you want to use
# To allow syscalls required by docker add: --system-call-filter='add_key keyctl bpf'
systemd_nspawn_user_args=
# Specify command/script to run on the HOST before starting the jail
@ -73,10 +71,8 @@ post_stop_hook=
distro=debian
release=bookworm
# Specify command/script to run IN THE JAIL before the first start
# Specify command/script to run IN THE JAIL on the first start (once networking is ready in the jail)
# Useful to install packages on top of the base rootfs
# NOTE: this script will run in the host networking namespace and
# ignores all systemd_nspawn_user_args such as bind mounts
initial_setup=
# initial_setup=bash -c 'apt-get update && apt-get -y upgrade'
@ -545,55 +541,6 @@ def start_jail(jail_name):
seccomp = config.my_getboolean("seccomp")
# Handle initial setup
initial_setup = config.my_get("initial_setup")
# Alternative method to setup on first boot:
# https://www.undrground.org/2021/01/25/adding-a-single-run-task-via-systemd/
# If there's no machine-id, then this the first time the jail is started
if initial_setup and not os.path.exists(
os.path.join(jail_rootfs_path, "etc/machine-id")
):
initial_setup_file = None
if initial_setup.startswith("#!"):
# Write a script file and call that
initial_setup_file = os.path.abspath(
os.path.join(jail_path, ".initial_setup")
)
print(initial_setup, file=open(initial_setup_file, "w"))
stat_chmod(initial_setup_file, 0o700)
cmd = [
"systemd-nspawn",
"-q",
"-D",
jail_rootfs_path,
f"--bind-ro={initial_setup_file}:/root/initial_startup",
"/root/initial_startup",
]
else:
# Run the command directly if it doesn't start with a shebang
cmd = [
"systemd-nspawn",
"-q",
"-D",
jail_rootfs_path,
*shlex.split(initial_setup),
]
returncode = subprocess.run(cmd).returncode
# Cleanup the initial_setup_file
if initial_setup_file:
Path(initial_setup_file).unlink(missing_ok=True)
if returncode != 0:
eprint("Failed to run initial setup:")
eprint(initial_setup)
eprint()
eprint("Abort starting jail.")
return returncode
systemd_run_additional_args = [
f"--unit={SYMLINK_NAME}-{jail_name}",
f"--working-directory=./{jail_path}",
@ -605,6 +552,24 @@ def start_jail(jail_name):
f"--directory={JAIL_ROOTFS_NAME}",
]
# The systemd-nspawn manual explicitly mentions:
# Device nodes may not be created
# https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html
# This means docker images containing device nodes can't be pulled
# https://github.com/moby/moby/issues/35245
#
# The solution is to use DevicePolicy=auto
# https://github.com/kinvolk/kube-spawn/pull/328
#
# DevicePolicy=auto is the default for systemd-run and allows access to all devices
# as long as we don't add any --property=DeviceAllow= flags
# https://manpages.debian.org/bookworm/systemd/systemd.resource-control.5.en.html
#
# We can now successfully run:
# mknod /dev/port c 1 4
# Or pull docker images containing device nodes:
# docker pull oraclelinux@sha256:d49469769e4701925d5145c2676d5a10c38c213802cf13270ec3a12c9c84d643
if config.my_getboolean("docker_compatible"):
eprint("WARNING: DEPRECATED OPTION")
eprint(
@ -704,23 +669,28 @@ def start_jail(jail_name):
"--setenv=SYSTEMD_SECCOMP=0",
]
# The systemd-nspawn manual explicitly mentions:
# Device nodes may not be created
# https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html
# This means docker images containing device nodes can't be pulled
# https://github.com/moby/moby/issues/35245
#
# The solution is to use DevicePolicy=auto
# https://github.com/kinvolk/kube-spawn/pull/328
#
# DevicePolicy=auto is the default for systemd-run and allows access to all devices
# as long as we don't add any --property=DeviceAllow= flags
# https://manpages.debian.org/bookworm/systemd/systemd.resource-control.5.en.html
#
# We can now successfully run:
# mknod /dev/port c 1 4
# Or pull docker images containing device nodes:
# docker pull oraclelinux@sha256:d49469769e4701925d5145c2676d5a10c38c213802cf13270ec3a12c9c84d643
initial_setup = False
# If there's no machine-id, then this the first time the jail is started
if not os.path.exists(os.path.join(jail_rootfs_path, "etc/machine-id")) and (
initial_setup := config.my_get("initial_setup")
):
if not initial_setup.startswith("#!"):
initial_setup = "#!/bin/sh\n" + initial_setup
initial_setup_file_jailed_path = "/root/jlmkr-initial-setup"
initial_setup_file_host_path = os.path.abspath(
jail_rootfs_path + initial_setup_file_jailed_path
)
# Write a script file to call during initial setup
print(initial_setup, file=open(initial_setup_file_host_path, "w"))
stat_chmod(initial_setup_file_host_path, 0o700)
# Ensure the jail init system is ready before we start the initial_setup
systemd_nspawn_additional_args += [
"--notify-ready=yes",
]
cmd = [
"systemd-run",
@ -755,6 +725,44 @@ def start_jail(jail_name):
)
)
return returncode
# Handle initial setup after jail is up and running (for the first time)
if initial_setup:
print("About to run the initial setup.")
print("Waiting for networking in the jail to be ready.")
print("Please wait (this may take 90s in case of bridge networking)...")
returncode = exec_jail(
jail_name,
[
"--",
"systemd-run",
f"--unit={os.path.basename(initial_setup_file_jailed_path)}",
"--quiet",
"--pipe",
"--wait",
"--service-type=exec",
"--property=After=network-online.target",
"--property=Wants=network-online.target",
initial_setup_file_jailed_path,
],
)
# Cleanup the initial_setup_file_host_path
if initial_setup_file_host_path:
Path(initial_setup_file_host_path).unlink(missing_ok=True)
if returncode != 0:
eprint("Tried to run the following commands inside the jail:")
eprint(initial_setup)
eprint()
eprint(
f"""{RED}{BOLD}Failed to run initial setup... you may want to stop and remove the jail and try again.{NORMAL}"""
)
return returncode
else:
print(f"Done with initial setup of jail {jail_name}!")
return returncode
@ -922,10 +930,11 @@ def get_zfs_dataset(path):
"""
Get ZFS dataset path.
"""
def clean_field(field):
# Put back spaces which were encoded
# https://github.com/openzfs/zfs/issues/11182
return field.replace('\\040', ' ')
return field.replace("\\040", " ")
path = os.path.realpath(path)
with open("/proc/mounts", "r") as f:
@ -1025,11 +1034,14 @@ def get_text_editor():
if editor := os.environ.get(key):
return shutil.which(editor)
return get_from_environ("VISUAL") \
or get_from_environ("EDITOR") \
or shutil.which("editor") \
or shutil.which("/usr/bin/editor") \
return (
get_from_environ("VISUAL")
or get_from_environ("EDITOR")
or shutil.which("editor")
or shutil.which("/usr/bin/editor")
or "nano"
)
def interactive_config():
config = KeyValueParser()
@ -1517,9 +1529,7 @@ def edit_jail(jail_name):
jail_config_path = get_jail_config_path(jail_name)
returncode = subprocess.run(
[get_text_editor(), jail_config_path]
).returncode
returncode = subprocess.run([get_text_editor(), jail_config_path]).returncode
if returncode != 0:
eprint(f"An error occurred while editing {jail_config_path}.")

View File

@ -2,4 +2,4 @@
## Setup
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/docker/config mydockerjail`.
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/docker/config mydockerjail`. If you want the `nvidia-container-toolkit` to be installed, ensure you set `gpu_passthrough_nvidia=1` when creating the jail.

View File

@ -6,10 +6,10 @@ seccomp=1
# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-macvlan=eno1
systemd_nspawn_user_args=--network-bridge=br1
--resolv-conf=bind-host
--system-call-filter='add_key keyctl bpf'
@ -29,8 +29,8 @@ release=bookworm
# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# NOTE: this script will run in the host networking namespace and ignores
# all systemd_nspawn_user_args such as bind mounts
# Will also install the NVIDIA Container Toolkit if gpu_passthrough_nvidia=1 during initial setup
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
initial_setup=#!/usr/bin/bash
set -euo pipefail
@ -43,9 +43,27 @@ initial_setup=#!/usr/bin/bash
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
if [ -f /usr/bin/nvidia-smi ]; then
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey -o /etc/apt/keyrings/nvidia.asc
chmod a+r /etc/apt/keyrings/nvidia.asc
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia.asc] https://#g' | \
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker
fi
docker info
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
--property=Type=notify

View File

@ -8,11 +8,10 @@
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/incus/config myincusjail`.
Unfortunately incus doesn't want to install from the `initial_setup` script inside the config file. So we manually finish the setup by running the following after creating and starting the jail:
We manually finish the setup by running the following after creating and starting the jail:
```bash
jlmkr exec myincusjail bash -c 'apt-get -y install incus incus-ui-canonical &&
incus admin init'
jlmkr exec myincusjail bash -c 'incus admin init'
```
Follow [First steps with Incus](https://linuxcontainers.org/incus/docs/main/tutorial/first_steps/).

View File

@ -3,15 +3,16 @@ startup=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1
# TODO: don't disable seccomp but specify which syscalls should be allowed
seccomp=0
# Use macvlan networking to provide an isolated network namespace,
# so incus can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
# TODO: don't use --capability=all but specify only the required capabilities
systemd_nspawn_user_args=--network-macvlan=eno1
systemd_nspawn_user_args=--network-bridge=br1
--resolv-conf=bind-host
--capability=all
--bind=/dev/fuse
@ -36,8 +37,6 @@ release=bookworm
# Install incus according to:
# https://github.com/zabbly/incus#installation
# NOTE: this script will run in the host networking namespace and ignores
# all systemd_nspawn_user_args such as bind mounts
initial_setup=#!/usr/bin/bash
set -euo pipefail
apt-get update && apt-get -y install curl
@ -54,6 +53,7 @@ initial_setup=#!/usr/bin/bash
EOF'
apt-get update
apt-get -y install incus incus-ui-canonical
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed

View File

@ -8,20 +8,7 @@
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/lxd/config mylxdjail`.
Unfortunately snapd doesn't want to install from the `initial_setup` script inside the config file. So we manually finish the setup by running the following after creating and starting the jail:
```bash
# Repeat listing the jail until you see it has an IPv4 address
jlmkr list
# Install packages
jlmkr exec mylxdjail bash -c 'apt-get update &&
apt-get install -y --no-install-recommends snapd &&
snap install lxd'
```
Choose the `dir` storage backend during `lxd init` and answer `yes` to "Would you like the LXD server to be available over the network?"
We manually finish the setup by running the command below after creating and starting the jail. Choose the `dir` storage backend during `lxd init` and answer `yes` to "Would you like the LXD server to be available over the network?"
```bash
jlmkr exec mylxdjail bash -c 'lxd init &&

View File

@ -3,11 +3,12 @@ startup=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1
# TODO: don't disable seccomp but specify which syscalls should be allowed
seccomp=0
# Use macvlan networking to provide an isolated network namespace,
# so lxd can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
# TODO: don't use --capability=all but specify only the required capabilities
@ -34,12 +35,13 @@ pre_start_hook=#!/usr/bin/bash
distro=ubuntu
release=jammy
# NOTE: this script will run in the host networking namespace and ignores
# all systemd_nspawn_user_args such as bind mounts
initial_setup=#!/usr/bin/bash
set -euo pipefail
# https://discuss.linuxcontainers.org/t/snap-inside-privileged-lxd-container/13691/8
ln -sf /bin/true /usr/local/bin/udevadm
apt-get update
apt-get install -y --no-install-recommends snapd
snap install lxd
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed

View File

@ -46,6 +46,10 @@ usermod --del-subuids 0-4294967295 --del-subgids 0-4294967295 rootless
# Set a specific range, so it fits inside the number of available UIDs
usermod --add-subuids 65536-131071 --add-subgids 65536-131071 rootless
# Add the required capabilities to the `newuidmap` and `newgidmap` binaries
setcap cap_setuid+eip /usr/bin/newuidmap
setcap cap_setgid+eip /usr/bin/newgidmap
# Check the assigned range
cat /etc/subuid
# Check the available range
@ -121,3 +125,6 @@ Resources mentioning `@keyring`
- https://github.com/systemd/systemd/issues/17606
- https://github.com/systemd/systemd/blob/1c62c4fe0b54fb419b875cb2bae82a261518a745/src/shared/seccomp-util.c#L604
`@keyring` also includes `request_key` but doesn't include `bpf`
Resources mentioning `cap_setuid+eip`, `cap_setgid+eip`, `newuidmap` and `newgidmap`
- https://github.com/containers/podman/issues/2788#issuecomment-1016301663
- https://github.com/containers/podman/issues/12637#issuecomment-996524341

View File

@ -6,10 +6,10 @@ seccomp=1
# Use macvlan networking to provide an isolated network namespace,
# so podman can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-macvlan=eno1
systemd_nspawn_user_args=--network-bridge=br1
--resolv-conf=bind-host
--system-call-filter='add_key keyctl bpf'
@ -28,16 +28,9 @@ distro=fedora
release=39
# Install podman inside the jail
# NOTE: this script will run in the host networking namespace and ignores
# all systemd_nspawn_user_args such as bind mounts
initial_setup=#!/usr/bin/bash
set -euo pipefail
dnf -y install podman
# Add the required capabilities to the `newuidmap` and `newgidmap` binaries
# https://github.com/containers/podman/issues/2788#issuecomment-1016301663
# https://github.com/containers/podman/issues/12637#issuecomment-996524341
setcap cap_setuid+eip /usr/bin/newuidmap
setcap cap_setgid+eip /usr/bin/newgidmap
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed