74 lines
4.1 KiB
Markdown
74 lines
4.1 KiB
Markdown
# Ubuntu LXD Jail Template
|
|
|
|
## Disclaimer
|
|
|
|
**Experimental. Using LXD in this setup hasn't been extensively tested and has [known issues](#known-issues).**
|
|
|
|
## Setup
|
|
|
|
Check out the [config](./config) template file. You may provide it when asked during `./jlmkr.py create` or, if you have the template file stored on your NAS, you may provide it directly by running `./jlmkr.py create --start --config /mnt/tank/path/to/lxd/config mylxdjail`.
|
|
|
|
We manually finish the setup by running the command below after creating and starting the jail. Choose the `dir` storage backend during `lxd init` and answer `yes` to "Would you like the LXD server to be available over the network?"
|
|
|
|
```bash
|
|
./jlmkr.py exec mylxdjail bash -c 'lxd init &&
|
|
snap set lxd ui.enable=true &&
|
|
systemctl reload snap.lxd.daemon'
|
|
```
|
|
|
|
Then visit the `lxd` GUI inside the browser https://0.0.0.0:8443. To find out which IP address to use instead of 0.0.0.0, check the IP address for your jail with `./jlmkr.py list`.
|
|
|
|
## Known Issues
|
|
|
|
### Instance creation failed
|
|
|
|
[LXD no longer has access to the LinuxContainers image server](https://discuss.linuxcontainers.org/t/important-notice-for-lxd-users-image-server/18479).
|
|
|
|
```
|
|
Failed getting remote image info: Failed getting image: The requested image couldn't be found for fingerprint "ubuntu/focal/desktop"
|
|
```
|
|
|
|
### SCALE Virtual Machines
|
|
Using LXD in the jail will cause the following error when starting a VM from the TrueNAS SCALE web GUI:
|
|
|
|
```
|
|
[EFAULT] internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied 2024-02-16T14:40:14.886658Z qemu-system-x86_64: -accel kvm: failed to initialize kvm: Permission denied
|
|
```
|
|
|
|
A reboot will resolve the issue (until you start the LXD jail again).
|
|
|
|
### ZFS Issues
|
|
|
|
If you create a new dataset on your pool (e.g. `tank`) called `lxd` from the TrueNAS SCALE web GUI and tell LXD to use it during `lxd init`, then you will run into issues. Firstly you'd have to run `apt-get install -y --no-install-recommends zfsutils-linux` inside the jail to install the ZFS userspace utils and you've have to add `--bind=/dev/zfs` to the `systemd_nspawn_user_args` in the jail config. By mounting `/dev/zfs` into this jail, **it will have total control of the storage on the host!**
|
|
|
|
But then SCALE doesn't seem to like the ZFS datasets created by LXD. I get the following errors when browsing the sub-datasets:
|
|
|
|
```
|
|
[EINVAL] legacy: path must be absolute
|
|
```
|
|
|
|
```
|
|
[EFAULT] Failed retreiving USER quotas for tank/lxd/virtual-machines
|
|
```
|
|
|
|
As long as you don't operate on these datasets in the SCALE GUI this may not be a real problem...
|
|
|
|
However, creating an LXD VM doesn't work with the ZFS storage backend (creating a container works though):
|
|
|
|
```
|
|
Failed creating instance from image: Could not locate a zvol for tank/lxd/images/1555b13f0e89bfcf516bd0090eee6f73a0db5f4d0d36c38cae94316de82bf817.block
|
|
```
|
|
|
|
Could this be the same issue as [Instance creation failed](#instance-creation-failed)?
|
|
|
|
## More info
|
|
|
|
Refer to the [Incus README](../incus/README.md) as a lot of it applies to LXD too.
|
|
|
|
## Ideas
|
|
|
|
Instead of installing `snapd` and `lxd` from the `initial_setup` script, it's possible to install [the `lxd-installer` package](https://packages.ubuntu.com/noble/lxd-installer) instead, which is a wrapper to install the lxd snap on demand. This can be done on a rootfs which is not booted (e.g. in a chroot, dockerfile, or with `systemd-nspawn` without the `--boot` flag). Another option is to use [a cloud variant image from linuxcontainers.org](https://images.linuxcontainers.org) or [other sources](https://cloud-images.ubuntu.com/noble/current/) as I think they already include lxd. These images als come with `cloud-init`, which is a standardized way to customize a container (or VM) during the first boot. Benefit of using that would be a standardized way to customize the image rootfs. Downside could be more bloated images as it depends on Python and more.
|
|
|
|
## References
|
|
|
|
- [Running QEMU/KVM Virtual Machines in Unprivileged LXD Containers](https://dshcherb.github.io/2017/12/04/qemu-kvm-virtual-machines-in-unprivileged-lxd.html) |