4.1 KiB
Ubuntu LXD Jail Template
Disclaimer
Experimental. Using LXD in this setup hasn't been extensively tested and has known issues.
Setup
Check out the config template file. You may provide it when asked during ./jlmkr.py create or, if you have the template file stored on your NAS, you may provide it directly by running ./jlmkr.py create --start --config /mnt/tank/path/to/lxd/config mylxdjail.
We manually finish the setup by running the command below after creating and starting the jail. Choose the dir storage backend during lxd init and answer yes to "Would you like the LXD server to be available over the network?"
./jlmkr.py exec mylxdjail bash -c 'lxd init &&
snap set lxd ui.enable=true &&
systemctl reload snap.lxd.daemon'
Then visit the lxd GUI inside the browser https://0.0.0.0:8443. To find out which IP address to use instead of 0.0.0.0, check the IP address for your jail with ./jlmkr.py list.
Known Issues
Instance creation failed
LXD no longer has access to the LinuxContainers image server.
Failed getting remote image info: Failed getting image: The requested image couldn't be found for fingerprint "ubuntu/focal/desktop"
SCALE Virtual Machines
Using LXD in the jail will cause the following error when starting a VM from the TrueNAS SCALE web GUI:
[EFAULT] internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied 2024-02-16T14:40:14.886658Z qemu-system-x86_64: -accel kvm: failed to initialize kvm: Permission denied
A reboot will resolve the issue (until you start the LXD jail again).
ZFS Issues
If you create a new dataset on your pool (e.g. tank) called lxd from the TrueNAS SCALE web GUI and tell LXD to use it during lxd init, then you will run into issues. Firstly you'd have to run apt-get install -y --no-install-recommends zfsutils-linux inside the jail to install the ZFS userspace utils and you've have to add --bind=/dev/zfs to the systemd_nspawn_user_args in the jail config. By mounting /dev/zfs into this jail, it will have total control of the storage on the host!
But then SCALE doesn't seem to like the ZFS datasets created by LXD. I get the following errors when browsing the sub-datasets:
[EINVAL] legacy: path must be absolute
[EFAULT] Failed retreiving USER quotas for tank/lxd/virtual-machines
As long as you don't operate on these datasets in the SCALE GUI this may not be a real problem...
However, creating an LXD VM doesn't work with the ZFS storage backend (creating a container works though):
Failed creating instance from image: Could not locate a zvol for tank/lxd/images/1555b13f0e89bfcf516bd0090eee6f73a0db5f4d0d36c38cae94316de82bf817.block
Could this be the same issue as Instance creation failed?
More info
Refer to the Incus README as a lot of it applies to LXD too.
Ideas
Instead of installing snapd and lxd from the initial_setup script, it's possible to install the lxd-installer package instead, which is a wrapper to install the lxd snap on demand. This can be done on a rootfs which is not booted (e.g. in a chroot, dockerfile, or with systemd-nspawn without the --boot flag). Another option is to use a cloud variant image from linuxcontainers.org or other sources as I think they already include lxd. These images als come with cloud-init, which is a standardized way to customize a container (or VM) during the first boot. Benefit of using that would be a standardized way to customize the image rootfs. Downside could be more bloated images as it depends on Python and more.