Update docs

This commit is contained in:
Jip-Hop 2024-05-28 20:55:46 +02:00
parent 07f405ef73
commit fbe988137f
8 changed files with 51 additions and 41 deletions

View File

@ -21,29 +21,9 @@ TrueNAS SCALE can create persistent Linux 'jails' with systemd-nspawn. This scri
- Optional: GPU passthrough (including nvidia GPU with the drivers bind mounted from the host) - Optional: GPU passthrough (including nvidia GPU with the drivers bind mounted from the host)
- Starting the jail with your config applied - Starting the jail with your config applied
## Security
Despite what the word 'jail' implies, jailmaker's intended use case is to create one or more additional filesystems to run alongside SCALE with minimal isolation. By default the root user in the jail with uid 0 is mapped to the host's uid 0. This has [obvious security implications](https://linuxcontainers.org/lxc/security/#privileged-containers). If this is not acceptable to you, you may lock down the jails by [limiting capabilities](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#Security_Options) and/or using [user namespacing](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#User_Namespacing_Options) or use a VM instead.
### Seccomp
Seccomp is a Linux kernel feature that restricts programs from making unauthorized system calls. This means that when seccomp is enabled there can be times where a process run inside a jail will be killed with the error "Operation not permitted." In order to find out which syscall needs to be added to the `--system-call-filter=` configuration you can use `strace`.
For example:
```
# /usr/bin/intel_gpu_top
Failed to initialize PMU! (Operation not permitted)
# strace /usr/bin/intel_gpu_top 2>&1 |grep Operation\ not\ permitted
perf_event_open({type=0x10 /* PERF_TYPE_??? */, size=PERF_ATTR_SIZE_VER7, config=0x100002, sample_period=0, sample_type=0, read_format=PERF_FORMAT_TOTAL_TIME_ENABLED|PERF_FORMAT_GROUP, precise_ip=0 /* arbitrary skid */, use_clockid=1, ...}, -1, 0, -1, 0) = -1 EPERM (Operation not permitted)
write(2, "Failed to initialize PMU! (Opera"..., 52Failed to initialize PMU! (Operation not permitted)
```
The syscall that needs to be added to the `--system-call-filter` option in the jlmkr config in this case would be `perf_event_open`. You may need to run strace multiple times.
Seccomp is important for security, but as a last resort can be disabled by setting `seccomp=0` in the jail config.
## Installation ## Installation
Beginning with 24.04 (Dragonfish), TrueNAS SCALE includes the systemd-nspawn containerization program in the base system. Technically there's nothing to install. You only need the `jlmkr.py` script file in the right place. [Instructions with screenshots](https://www.truenas.com/docs/scale/scaletutorials/apps/sandboxes/) are provided on the TrueNAS website. Start by creating a new dataset called `jailmaker` with the default settings (from TrueNAS web interface). Then login as the root user and download `jlmkr.py`. If you login as non-root user (e.g. as admin), **you must become root first** by executing `sudo su`. Beginning with 24.04 (Dragonfish), TrueNAS SCALE officially includes the systemd-nspawn containerization program in the base system. Technically there's nothing to install. You only need the `jlmkr.py` script file in the right place. [Instructions with screenshots](https://www.truenas.com/docs/scale/scaletutorials/apps/sandboxes/) are provided on the TrueNAS website. Start by creating a new dataset called `jailmaker` with the default settings (from TrueNAS web interface). Then login as the root user and download `jlmkr.py`.
```shell ```shell
cd /mnt/mypool/jailmaker cd /mnt/mypool/jailmaker
@ -53,6 +33,16 @@ chmod +x jlmkr.py
The `jlmkr.py` script (and the jails + config it creates) are now stored on the `jailmaker` dataset and will survive updates of TrueNAS SCALE. If the automatically created `jails` directory is also a ZFS dataset (which is true for new users), then the `jlmkr.py` script will automatically create a new dataset for every jail created. This allows you to snapshot individual jails. For legacy users (where the `jails` directory is not a dataset) each jail will be stored in a plain directory. The `jlmkr.py` script (and the jails + config it creates) are now stored on the `jailmaker` dataset and will survive updates of TrueNAS SCALE. If the automatically created `jails` directory is also a ZFS dataset (which is true for new users), then the `jlmkr.py` script will automatically create a new dataset for every jail created. This allows you to snapshot individual jails. For legacy users (where the `jails` directory is not a dataset) each jail will be stored in a plain directory.
### Alias
Optionally you may create a shell alias for the currently logged in (admin) user to conveniently run `jlmkr.py` without having to change into the `jailmaker` directory or specify the full absolute path. I suggest to create the `jlmkr` alias like this:
```shell
echo "alias jlmkr=\"sudo '/mnt/mypool/jailmaker/jlmkr.py'\"" > ~/.bashrc
```
Please replace `/mnt/mypool/jailmaker/` with the actual path to where you stored `jlmkr.py`. If you're using zsh instead of bash, then you should replace `.bashrc` in the command above with `.zshrc`. If you've created the alias, you may use it instead of `./jlmkr.py`.
## Usage ## Usage
### Create Jail ### Create Jail
@ -174,6 +164,26 @@ View a jail's logs.
Expert users may use the following additional commands to manage jails directly: `machinectl`, `systemd-nspawn`, `systemd-run`, `systemctl` and `journalctl`. The `jlmkr` script uses these commands under the hood and implements a subset of their functions. If you use them directly you will bypass any safety checks or configuration done by `jlmkr` and not everything will work in the context of TrueNAS SCALE. Expert users may use the following additional commands to manage jails directly: `machinectl`, `systemd-nspawn`, `systemd-run`, `systemctl` and `journalctl`. The `jlmkr` script uses these commands under the hood and implements a subset of their functions. If you use them directly you will bypass any safety checks or configuration done by `jlmkr` and not everything will work in the context of TrueNAS SCALE.
## Security
By default the root user in the jail with uid 0 is mapped to the host's uid 0. This has [obvious security implications](https://linuxcontainers.org/lxc/security/#privileged-containers). If this is not acceptable to you, you may lock down the jails by [limiting capabilities](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#Security_Options) and/or using [user namespacing](https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html#User_Namespacing_Options) or use a VM instead.
### Seccomp
Seccomp is a Linux kernel feature that restricts programs from making unauthorized system calls. This means that when seccomp is enabled there can be times where a process run inside a jail will be killed with the error "Operation not permitted." In order to find out which syscall needs to be added to the `--system-call-filter=` configuration you can use `strace`.
For example:
```
# /usr/bin/intel_gpu_top
Failed to initialize PMU! (Operation not permitted)
# strace /usr/bin/intel_gpu_top 2>&1 |grep Operation\ not\ permitted
perf_event_open({type=0x10 /* PERF_TYPE_??? */, size=PERF_ATTR_SIZE_VER7, config=0x100002, sample_period=0, sample_type=0, read_format=PERF_FORMAT_TOTAL_TIME_ENABLED|PERF_FORMAT_GROUP, precise_ip=0 /* arbitrary skid */, use_clockid=1, ...}, -1, 0, -1, 0) = -1 EPERM (Operation not permitted)
write(2, "Failed to initialize PMU! (Opera"..., 52Failed to initialize PMU! (Operation not permitted)
```
The syscall that needs to be added to the `--system-call-filter` option in the `jailmaker` config in this case would be `perf_event_open`. You may need to run strace multiple times.
Seccomp is important for security, but as a last resort can be disabled by setting `seccomp=0` in the jail config.
## Networking ## Networking
By default a jails will use the same networking namespace, with access to all (physical) interfaces the TrueNAS host has access to. No further setup is required. You may download and install additional packages inside the jail. Note that some ports are already occupied by TrueNAS SCALE (e.g. 443 for the web interface), so your jail can't listen on these ports. By default a jails will use the same networking namespace, with access to all (physical) interfaces the TrueNAS host has access to. No further setup is required. You may download and install additional packages inside the jail. Note that some ports are already occupied by TrueNAS SCALE (e.g. 443 for the web interface), so your jail can't listen on these ports.
@ -202,7 +212,7 @@ The rootfs image `jlmkr.py` downloads comes from the [Linux Containers Image ser
## Filing Issues and Community Support ## Filing Issues and Community Support
When in need of help or when you think you've found a bug in jailmaker, [please start with reading this](https://github.com/Jip-Hop/jailmaker/discussions/135). When in need of help or when you think you've found a bug in `jailmaker`, [please start with reading this](https://github.com/Jip-Hop/jailmaker/discussions/135).
## References ## References

View File

@ -52,7 +52,7 @@ See [Networking](./network.md)
### Colorized bash prompt ### Colorized bash prompt
To visually distinguish between a root shell inside the jail and a root shell outside the jail, it's possible to colorize the shell prompt. When using a debian jail with the bash shell, you may run the following command **inside the jail** to get a yellow prompt inside the jail (will be activated the next time you run `jlmkr shell myjail`): To visually distinguish between a root shell inside the jail and a root shell outside the jail, it's possible to colorize the shell prompt. When using a debian jail with the bash shell, you may run the following command **inside the jail** to get a yellow prompt inside the jail (will be activated the next time you run `./jlmkr.py shell myjail`):
```bash ```bash
echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;33m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;33m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc

View File

@ -91,7 +91,7 @@ DHCP=false
Address=192.168.X.XXX/24 Address=192.168.X.XXX/24
Gateway=192.168.X.X Gateway=192.168.X.X
``` ```
Then restart the network interface inside the jail `systemctl restart systemd-networkd` or restart the jail by running `jlmkr stop JAILNAME && jlmkr start JAILNAME` from the TrueNAS shell. Use `ifconfig` to verify the interface is up and has the correct IP. Then restart the network interface inside the jail `systemctl restart systemd-networkd` or restart the jail by running `./jlmkr.py stop JAILNAME && ./jlmkr.py start JAILNAME` from the TrueNAS shell. Use `ifconfig` to verify the interface is up and has the correct IP.
## DNS via DHCP ## DNS via DHCP

View File

@ -11,9 +11,9 @@ Jailmaker operates in dual-mode: it supports using both directories and datasets
#### Stop all jails #### Stop all jails
`jlmkr stop jail1` `./jlmkr.py stop jail1`
`jlmkr stop jail2` `./jlmkr.py stop jail2`
etc.. etc..
#### Move/rename the 'jailmaker' directory #### Move/rename the 'jailmaker' directory
@ -56,6 +56,6 @@ Warning! It's important that both directories have the `/` at the end to make su
#### Test everything works #### Test everything works
If everything works, you should be able to use the `jlmkr` command directly. Try doing a `jlmkr list` to check if the jails are correctly recognized If everything works, you should be able to use the `./jlmkr.py` command directly. Try doing a `./jlmkr.py list` to check if the jails are correctly recognized
You can also try creating a new jail and see that the dataset is created automatically. You can also try creating a new jail and see that the dataset is created automatically.

View File

@ -2,4 +2,4 @@
## Setup ## Setup
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/docker/config mydockerjail`. If you want the `nvidia-container-toolkit` to be installed, ensure you set `gpu_passthrough_nvidia=1` when creating the jail. Check out the [config](./config) template file. You may provide it when asked during `./jlmkr.py create` or, if you have the template file stored on your NAS, you may provide it directly by running `./jlmkr.py create --start --config /mnt/tank/path/to/docker/config mydockerjail`. If you want the `nvidia-container-toolkit` to be installed, ensure you set `gpu_passthrough_nvidia=1` when creating the jail.

View File

@ -6,17 +6,17 @@
## Setup ## Setup
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/incus/config myincusjail`. Check out the [config](./config) template file. You may provide it when asked during `./jlmkr.py create` or, if you have the template file stored on your NAS, you may provide it directly by running `./jlmkr.py create --start --config /mnt/tank/path/to/incus/config myincusjail`.
We manually finish the setup by running the following after creating and starting the jail: We manually finish the setup by running the following after creating and starting the jail:
```bash ```bash
jlmkr exec myincusjail bash -c 'incus admin init' ./jlmkr.py exec myincusjail bash -c 'incus admin init'
``` ```
Follow [First steps with Incus](https://linuxcontainers.org/incus/docs/main/tutorial/first_steps/). Follow [First steps with Incus](https://linuxcontainers.org/incus/docs/main/tutorial/first_steps/).
Then visit the Incus GUI inside the browser https://0.0.0.0:8443. To find out which IP address to use instead of 0.0.0.0, check the IP address for your jail with `jlmkr list`. Then visit the Incus GUI inside the browser https://0.0.0.0:8443. To find out which IP address to use instead of 0.0.0.0, check the IP address for your jail with `./jlmkr.py list`.
## Known Issues ## Known Issues

View File

@ -6,17 +6,17 @@
## Setup ## Setup
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/lxd/config mylxdjail`. Check out the [config](./config) template file. You may provide it when asked during `./jlmkr.py create` or, if you have the template file stored on your NAS, you may provide it directly by running `./jlmkr.py create --start --config /mnt/tank/path/to/lxd/config mylxdjail`.
We manually finish the setup by running the command below after creating and starting the jail. Choose the `dir` storage backend during `lxd init` and answer `yes` to "Would you like the LXD server to be available over the network?" We manually finish the setup by running the command below after creating and starting the jail. Choose the `dir` storage backend during `lxd init` and answer `yes` to "Would you like the LXD server to be available over the network?"
```bash ```bash
jlmkr exec mylxdjail bash -c 'lxd init && ./jlmkr.py exec mylxdjail bash -c 'lxd init &&
snap set lxd ui.enable=true && snap set lxd ui.enable=true &&
systemctl reload snap.lxd.daemon' systemctl reload snap.lxd.daemon'
``` ```
Then visit the `lxd` GUI inside the browser https://0.0.0.0:8443. To find out which IP address to use instead of 0.0.0.0, check the IP address for your jail with `jlmkr list`. Then visit the `lxd` GUI inside the browser https://0.0.0.0:8443. To find out which IP address to use instead of 0.0.0.0, check the IP address for your jail with `./jlmkr.py list`.
## Known Issues ## Known Issues

View File

@ -2,7 +2,7 @@
## Setup ## Setup
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create --start --config /mnt/tank/path/to/podman/config mypodmanjail`. Check out the [config](./config) template file. You may provide it when asked during `./jlmkr.py create` or, if you have the template file stored on your NAS, you may provide it directly by running `./jlmkr.py create --start --config /mnt/tank/path/to/podman/config mypodmanjail`.
## Rootless ## Rootless
@ -14,11 +14,11 @@ Check out the [config](./config) template file. You may provide it when asked du
Prerequisites: created a jail using the [config](./config) template file. Prerequisites: created a jail using the [config](./config) template file.
Run `jlmkr edit mypodmanjail` and add `--private-users=524288:65536 --private-users-ownership=chown` to `systemd_nspawn_user_args`. We start at UID 524288, as this is the [systemd range used for containers](https://github.com/systemd/systemd/blob/main/docs/UIDS-GIDS.md#summary). Run `./jlmkr.py edit mypodmanjail` and add `--private-users=524288:65536 --private-users-ownership=chown` to `systemd_nspawn_user_args`. We start at UID 524288, as this is the [systemd range used for containers](https://github.com/systemd/systemd/blob/main/docs/UIDS-GIDS.md#summary).
The `--private-users-ownership=chown` option will ensure the rootfs ownership is corrected. The `--private-users-ownership=chown` option will ensure the rootfs ownership is corrected.
After the jail has started run `jlmkr stop mypodmanjail && jlmkr edit mypodmanjail`, remove `--private-users-ownership=chown` and increase the UID range to `131072` to double the number of UIDs available in the jail. We need more than 65536 UIDs available in the jail, since rootless podman also needs to be able to map UIDs. If I leave the `--private-users-ownership=chown` option I get the following error: After the jail has started run `./jlmkr.py stop mypodmanjail && ./jlmkr.py edit mypodmanjail`, remove `--private-users-ownership=chown` and increase the UID range to `131072` to double the number of UIDs available in the jail. We need more than 65536 UIDs available in the jail, since rootless podman also needs to be able to map UIDs. If I leave the `--private-users-ownership=chown` option I get the following error:
> systemd-nspawn[678877]: Automatic UID/GID adjusting is only supported for UID/GID ranges starting at multiples of 2^16 with a range of 2^16 > systemd-nspawn[678877]: Automatic UID/GID adjusting is only supported for UID/GID ranges starting at multiples of 2^16 with a range of 2^16
@ -31,7 +31,7 @@ systemd_nspawn_user_args=--network-macvlan=eno1
--private-users=524288:131072 --private-users=524288:131072
``` ```
Start the jail with `jlmkr start mypodmanjail` and open a shell session inside the jail (as the remapped root user) with `jlmkr shell mypodmanjail`. Start the jail with `./jlmkr.py start mypodmanjail` and open a shell session inside the jail (as the remapped root user) with `./jlmkr.py shell mypodmanjail`.
Then inside the jail setup the new rootless user: Then inside the jail setup the new rootless user:
@ -61,7 +61,7 @@ exit
From the TrueNAS host, open a shell as the rootless user inside the jail. From the TrueNAS host, open a shell as the rootless user inside the jail.
```bash ```bash
jlmkr shell --uid 1000 mypodmanjail ./jlmkr.py shell --uid 1000 mypodmanjail
``` ```
Run rootless podman as user 1000. Run rootless podman as user 1000.
@ -97,7 +97,7 @@ Add `sysctl net.ipv4.ip_unprivileged_port_start=23` to the `pre_start_hook` insi
Install and enable cockpit: Install and enable cockpit:
```bash ```bash
jlmkr exec mypodmanjail bash -c "dnf -y install cockpit cockpit-podman && \ ./jlmkr.py exec mypodmanjail bash -c "dnf -y install cockpit cockpit-podman && \
systemctl enable --now cockpit.socket && \ systemctl enable --now cockpit.socket && \
ip a && ip a &&
ip route | awk '/default/ { print \$9 }'" ip route | awk '/default/ { print \$9 }'"
@ -108,7 +108,7 @@ Check the IP address of the jail and access the Cockpit web interface at https:/
If you've setup the `rootless` user, you may login with the password you've created earlier. Otherwise you'd have to add an admin user first: If you've setup the `rootless` user, you may login with the password you've created earlier. Otherwise you'd have to add an admin user first:
```bash ```bash
jlmkr exec podmantest bash -c 'adduser admin ./jlmkr.py exec podmantest bash -c 'adduser admin
passwd admin passwd admin
usermod -aG wheel admin' usermod -aG wheel admin'
``` ```