Add Incus template
This commit is contained in:
Jip-Hop
parent
ba74d5d3bd
commit
0d742e8a90
|
|
@ -1,37 +1,14 @@
|
|||
# Incus / LXD / LXC / KVM inside jail
|
||||
# Debian Incus Jail Template (LXD / LXC / KVM)
|
||||
|
||||
Check out the [config](./config) template file. You may provide it when asked during `jlmkr create` or, if you have the template file stored on your NAS, you may provide it directly by running `jlmkr create myincusjail /mnt/tank/path/to/incus/config`. Then check out [First steps with Incus](https://linuxcontainers.org/incus/docs/main/tutorial/first_steps/).
|
||||
|
||||
## Disclaimer
|
||||
|
||||
**These notes are a work in progress. Using Incus in this setup hasn't been extensively tested.**
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- TrueNAS SCALE 23.10 installed bare metal (not inside VM)
|
||||
- Jailmaker installed
|
||||
- Setup bridge networking (see Advanced Networking in the readme)
|
||||
|
||||
## Installation
|
||||
|
||||
Create a debian 12 jail and [install incus](https://github.com/zabbly/incus#installation). Also install the `incus-ui-canonical` package to install the web interface. Ensure the config file looks like the below:
|
||||
|
||||
Run `modprobe vhost_vsock` on the TrueNAS host.
|
||||
|
||||
```
|
||||
startup=0
|
||||
docker_compatible=1
|
||||
gpu_passthrough_intel=1
|
||||
gpu_passthrough_nvidia=0
|
||||
systemd_nspawn_user_args=--network-bridge=br1 --resolv-conf=bind-host --bind=/dev/fuse --bind=/dev/kvm --bind=/dev/vsock --bind=/dev/vhost-vsock
|
||||
# You generally will not need to change the options below
|
||||
systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0
|
||||
systemd_nspawn_default_args=--keep-unit --quiet --boot --bind-ro=/sys/module --inaccessible=/sys/module/apparmor
|
||||
```
|
||||
|
||||
Check out [First steps with Incus](https://linuxcontainers.org/incus/docs/main/tutorial/first_steps/).
|
||||
|
||||
## Create Ubuntu Desktop VM
|
||||
|
||||
Incus web GUI should be running on port 8443. Create new instance, call it `dekstop`, and choose the `Ubuntu jammy desktop virtual-machine ubuntu/22.04/desktop` image.
|
||||
Incus web GUI should be running on port 8443. Create new instance, call it `desktop`, and choose the `Ubuntu jammy desktop virtual-machine ubuntu/22.04/desktop` image.
|
||||
|
||||
## Bind mount / virtiofs
|
||||
|
||||
|
|
@ -75,41 +52,7 @@ root@incus:/home/test# dd if=/dev/random of=./test2.img bs=1G count=1 oflag=dsyn
|
|||
|
||||
## Create Ubuntu container
|
||||
|
||||
To be able to create unprivileged (rootless) containers with incus inside the jail, you need to increase the amount of UIDs available inside the jail. Please refer to the [Podman instructions](rootless_podman_in_rootless_jail.md) for more information. If you don't increase the UIDs you can only create privileged containers. You'd have to change `Privileged` to `Allow` in `Security policies` in this case.
|
||||
|
||||
## Canonical LXD install via snap
|
||||
|
||||
Installing the lxd snap is an alternative to Incus. But out of the box running `snap install lxd` will cause AppArmor issues when running inside a jailmaker jail on SCALE.
|
||||
|
||||
### Workaround 1: Disable AppArmor kernel module
|
||||
|
||||
[To my knowledge AppArmor is not uses on SCALE](https://github.com/truenas/charts/pull/428#issuecomment-1113936420). The AppArmor related packages aren't even installed.
|
||||
|
||||
Ensure to add --bind=/dev/fuse and ensure using bridge or macvlan networking:
|
||||
|
||||
```
|
||||
# On the host
|
||||
cat /sys/module/apparmor/parameters/enabled
|
||||
Y
|
||||
midclt call system.advanced.update '{"kernel_extra_options": "apparmor=0"}'
|
||||
reboot
|
||||
cat /sys/module/apparmor/parameters/enabled
|
||||
|
||||
# In Ubuntu jail
|
||||
apt update
|
||||
ln -s /bin/true /usr/local/bin/udevadm
|
||||
apt install -y --no-install-recommends snapd
|
||||
snap install lxd
|
||||
lxd init
|
||||
snap set lxd ui.enable=true
|
||||
systemctl reload snap.lxd.daemon
|
||||
|
||||
# Check out: https://example:8443
|
||||
```
|
||||
|
||||
### Workaround 2: inaccessible /sys/module/apparmor
|
||||
|
||||
If I don't want to mess with kernel parameters, I can trick the jail into thinking the apparmor module is not loaded by mounting over /sys/module/apparmor: `mount -v -r -t tmpfs -o size=50m test /sys/module/apparmor`. Then `snap install lxd` completes! Best way to do this is to add `--inaccessible=/sys/module/apparmor` to the systemd_nspawn_user_args.
|
||||
To be able to create unprivileged (rootless) containers with incus inside the jail, you need to increase the amount of UIDs available inside the jail. Please refer to the [Podman instructions](../podman/README.md) for more information. If you don't increase the UIDs you can only create privileged containers. You'd have to change `Privileged` to `Allow` in `Security policies` in this case.
|
||||
|
||||
## References
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,70 @@
|
|||
startup=0
|
||||
gpu_passthrough_intel=1
|
||||
gpu_passthrough_nvidia=0
|
||||
|
||||
# Use macvlan networking to provide an isolated network namespace,
|
||||
# so incus can manage firewall rules
|
||||
# Alternatively use --network-bridge=br1 instead of --network-macvlan
|
||||
# Ensure to change eno1/br1 to the interface name you want to use
|
||||
# You may want to add additional options here, e.g. bind mounts
|
||||
# TODO: don't use --capability=all but specify only the required capabilities
|
||||
systemd_nspawn_user_args=--network-macvlan=eno1
|
||||
--resolv-conf=bind-host
|
||||
--capability=all
|
||||
--bind=/dev/fuse
|
||||
--bind=/dev/kvm
|
||||
--bind=/dev/vsock
|
||||
--bind=/dev/vhost-vsock
|
||||
|
||||
# Script to run on the HOST before starting the jail
|
||||
# Load kernel module and config kernel settings required for incus
|
||||
pre_start_hook=#!/usr/bin/bash
|
||||
echo 'PRE_START_HOOK'
|
||||
echo 1 > /proc/sys/net/ipv4/ip_forward
|
||||
modprobe br_netfilter
|
||||
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
|
||||
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
|
||||
modprobe vhost_vsock
|
||||
|
||||
# Install incus according to:
|
||||
# https://github.com/zabbly/incus#installation
|
||||
# NOTE: this script will run in the host networking namespace and ignores
|
||||
# all systemd_nspawn_user_args such as bind mounts
|
||||
initial_setup=#!/usr/bin/bash
|
||||
mkdir -p /etc/apt/keyrings/
|
||||
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
|
||||
sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-stable.sources
|
||||
Enabled: yes
|
||||
Types: deb
|
||||
URIs: https://pkgs.zabbly.com/incus/stable
|
||||
Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
|
||||
Components: main
|
||||
Architectures: $(dpkg --print-architecture)
|
||||
Signed-By: /etc/apt/keyrings/zabbly.asc
|
||||
|
||||
EOF'
|
||||
apt-get update
|
||||
apt-get -y install incus incus-ui-canonical
|
||||
|
||||
# You generally will not need to change the options below
|
||||
systemd_run_default_args=--property=KillMode=mixed
|
||||
--property=Type=notify
|
||||
--property=RestartForceExitStatus=133
|
||||
--property=SuccessExitStatus=133
|
||||
--property=Delegate=yes
|
||||
--property=TasksMax=infinity
|
||||
--collect
|
||||
--setenv=SYSTEMD_NSPAWN_LOCK=0
|
||||
# TODO: check if the below 2 are required
|
||||
# --setenv=SYSTEMD_SECCOMP=0
|
||||
# --property=DevicePolicy=auto
|
||||
# TODO: add and use privileged flag?
|
||||
|
||||
systemd_nspawn_default_args=--keep-unit
|
||||
--quiet
|
||||
--boot
|
||||
--bind-ro=/sys/module
|
||||
--inaccessible=/sys/module/apparmor
|
||||
|
||||
# Used by jlmkr create
|
||||
initial_rootfs_image=debian bookworm
|
||||
Loading…
Reference in New Issue