git: 56ba04a704 - main - Handbook - Jails: Upgrade chapter
Sergio Carlavilla Delgado <[email protected]> Sun, 12 Jul 2026 15:47:22 +0000
| Newsgroups | gmane.os.freebsd.devel.cvs.doc |
|---|---|
| Message-ID | <[email protected]> |
The branch main has been updated by carlavilla: URL: https://cgit.FreeBSD.org/doc/commit/?id=56ba04a704e911e7f8f87be271f31f1dedc8f793 commit 56ba04a704e911e7f8f87be271f31f1dedc8f793 Author: Sergio Carlavilla Delgado <[email protected]> AuthorDate: 2026-07-12 15:45:35 +0000 Commit: Sergio Carlavilla Delgado <[email protected]> CommitDate: 2026-07-12 15:45:35 +0000 Handbook - Jails: Upgrade chapter * Improve top-level sections * Add NAT/port forwarding, administration from inside a jail, more jail options and jails with OpenZFS * Improve host network preparation, jail.conf and userland sources Approved by: arrowd@, bcr@ Differential Revision: D58173 --- .../content/en/books/handbook/jails/_index.adoc | 1348 ++++++++++++++++++-- 1 file changed, 1256 insertions(+), 92 deletions(-) diff --git a/documentation/content/en/books/handbook/jails/_index.adoc b/documentation/content/en/books/handbook/jails/_index.adoc index 49f38720b2..2d805fdaa2 100644 --- a/documentation/content/en/books/handbook/jails/_index.adoc +++ b/documentation/content/en/books/handbook/jails/_index.adoc @@ -80,9 +80,13 @@ This chapter covers: * The different ways to configure the network for a jail. * The jail configuration file. * How to create the different types of jail. +* How to configure network address translation for jails. * How to start, stop, and restart a jail. * The basics of jail administration, both from inside and outside the jail. * How to upgrade the different types of jail. +* How to limit the resources a jail can use. +* The additional per-jail security and feature options. +* How to delegate ZFS datasets to a jail. * An incomplete list of the different FreeBSD jail managers. [[jail-types]] @@ -116,10 +120,13 @@ Disadvantages of Thick Jails: [[thin-jails]] === Thin Jails -A thin jail shares the base system using OpenZFS snapshots or NullFS mounts from a template. -Only a minimal subset of base system is duplicated for each thin jail, resulting in less resource consumption compared to a thick jail. -However, this also means that thin jails have less isolation and independence compared to thick jails. -Changes in shared components could potentially affect multiple thin jails simultaneously. +A thin jail shares the base system from a template, using either OpenZFS snapshots or NullFS mounts. +Only a minimal subset of the base system is duplicated for each thin jail, resulting in less resource consumption compared to a thick jail. +The two mechanisms behave differently once the jails exist. +With the OpenZFS method, each thin jail is created as a clone of a template snapshot: an independent copy-on-write copy that shares unchanged blocks with the template but diverges as it is modified. +Later changes to the template do not propagate to existing clones, and each clone is maintained and updated individually, much like a thick jail. +With the NullFS method, each thin jail mounts the template read-only at runtime, so the base system is shared live and updating the template updates every jail that mounts it. +This shared, read-only base is also what gives NullFS thin jails less isolation and independence than thick jails, since changes in the shared components affect every jail using them simultaneously. In summary, a FreeBSD Thin Jail is a type of FreeBSD Jail that replicates a substantial portion, but not all, of the base system within the isolated environment. @@ -130,15 +137,16 @@ Since they share most of the base system, they consume less disk space and memor This makes it possible to run more jails on the same hardware without consuming excessive resources. * Faster Deployment: Creating and launching thin jails is generally faster compared to thick jails. This can be particularly advantageous when rapidly deploying multiple instances. -* Unified Maintenance: Since thin jails share the majority of their base system with the host system, updates and maintenance of common base system components (such as libraries and binaries) only need to be done once on the host. +* Unified Maintenance: With the NullFS method, where the thin jails share a single read-only copy of the base system, updates and maintenance of common base system components (such as libraries and binaries) only need to be done once on the template. This simplifies the maintenance process compared to maintaining an individual base system for each thick jail. +OpenZFS-clone thin jails do not share this benefit: because each clone is independent, updates must be applied to each jail individually, as with a thick jail. * Shared Resources: Thin jails can more easily share common resources such as libraries and binaries with the host system. This can potentially lead to more efficient disk caching and improved performance for applications within the jail. Disadvantages of Thin Jails: * Reduced Isolation: The primary disadvantage of thin jails is that they offer less isolation compared to thick jails. -Since they share a significant portion of the template's base system, vulnerabilities or issues affecting shared components could potentially impact multiple jails simultaneously. +With the NullFS method in particular, jails share a live copy of the template's base system, so vulnerabilities or issues affecting those shared components could potentially impact multiple jails simultaneously. * Security Concerns: The reduced isolation in thin jails could pose security risks, as a compromise in one jail might have a greater potential to affect other jails or the host system. * Dependency Conflicts: If multiple thin jails require different versions of the same libraries or software, managing dependencies can become complex. In some cases, this might require additional effort to ensure compatibility. @@ -150,7 +158,7 @@ In some cases, this might require additional effort to ensure compatibility. A service jail shares the complete filesystem tree directly with the host (the jail root path is [.filename]#/#) and as such can access and modify any file on the host, and shares the same user accounts with the host. By default it has no access to the network or other resources which are restricted in jails, but they can be configured to re-use the network of the host and to remove some of the jail-restrictions. The use case for service jails is automatic confinement of services/daemons inside a jail with minimal configuration, and without any knowledge of the files needed by such service/daemon. -Service jails exist since FreeBSD 15. +Service jails are available since FreeBSD 15.0. Advantages of Service Jails: @@ -168,7 +176,7 @@ Disadvantages of Service Jails: Most of the configuration of jails which is discussed below is not needed for service jails. To understand how jails work, it is recommended to understand those configuration possibilities. -The details about what is needed to configure a service jail is in crossref:jails[service-jails-config, Configuring service jails]. +The details about what is needed to configure a service jail are in the crossref:jails[service-jails-config, Service Jails] section. [[vnet-jails]] === VNET Jails @@ -217,15 +225,42 @@ To start jails when the system boots, run the following commands: With `jail_parallel_start`, all configured jails will be started in the background. ==== +By default every jail with an entry in the configuration files is started at boot. +To start only selected jails, list their names in `jail_list`: + +[source,shell] +.... +# sysrc jail_list="www db" +.... + +The names in `jail_list` are separated by spaces and must match the jail names defined in [.filename]#/etc/jail.conf# or [.filename]#/etc/jail.conf.d/#. +Only the listed jails are started at boot, and they are started in the order given, so any jail left out of `jail_list` has to be started by hand. +When `jail_list` is left empty, all jails defined in the configuration files are started. + +To stop the jails in the reverse of their `jail_list` order at shutdown, set `jail_reverse_stop`: + +[source,shell] +.... +# sysrc jail_reverse_stop="YES" +.... + +[NOTE] +==== +With `jail_parallel_start` enabled, jails start in the background and the `jail_list` order is no longer guaranteed. +When one jail must be running before another starts, express that relationship with the `depend` parameter in [.filename]#/etc/jail.conf# instead of relying on list order. +A jail's dependencies are created automatically before it starts, and stopped in the opposite order, regardless of `jail_list`. +The `depend` parameter is described in crossref:jails[jail-configuration-files,Jail Configuration Files]. +==== + [[jails-networking]] === Networking Networking for FreeBSD jails can be configured several different ways: Host Networking Mode (IP Sharing):: -In host networking mode, a jail shares the same networking stack as the host system. -When a jail is created in host networking mode it uses the same network interface and IP address. -This means that the jail does not have a separate IP address, and its network traffic is associated with the host's IP. +In host networking mode a jail shares the host's network stack instead of receiving its own. +Sharing the stack does not mean sharing the address: the common setup gives the jail its own dedicated `ip4.addr` (and `ip6.addr`) that man:jail[8] adds as an alias on the host interface, so the jail and the host answer on different addresses over the same stack. +Only a jail configured with `ip4 = inherit` shares the host's addresses wholesale and answers on the host's own address, gaining unrestricted access to every address the host holds. Virtual Networks (VNET):: Virtual Networks are a feature of FreeBSD jails that offer more advanced and flexible networking solutions than a basic networking mode like host networking. @@ -236,6 +271,74 @@ The netgraph system:: man:netgraph[4] is a versatile kernel framework for creating custom network configurations. It can be used to define how network traffic flows between jails and the host system and between different jails. +[[jail-ip-comes-from]] +==== Where a Jail's IP Address Comes From + +A shared-stack jail reaches the network through an address on a host interface. +The `interface` parameter names that interface, and for every address in `ip4.addr` and `ip6.addr` man:jail[8] manages the alias itself: as jail(8) puts it, "an alias for each address will be added to the interface before the jail is created, and will be removed from the interface after the jail is removed". +Nothing on the host has to pre-configure the address. + +The interface can also be given per address, so a single line ties an address to the interface that should carry it: + +[.programlisting] +.... +ip4.addr = "em0|192.168.1.151/24"; +.... + +When an address must persist independently of the jail — because another service on the host relies on it, or because the `interface` parameter is not used — assign it on the host as an rc.conf alias and leave it out of jail(8)'s management: + +[source,shell] +.... +# sysrc ifconfig_em0_aliases="inet 192.168.1.151/32" +.... + +The older, numbered form `ifconfig_em0_alias0="inet 192.168.1.151/32"` still works, but the `_aliases` form is preferred for new configurations because it avoids the fragile requirement that the numbered aliases form an unbroken `alias0`, `alias1`, `alias2` sequence. + +[[loopback-addresses-shared-ip-Jails]] +==== Loopback Addresses in Shared-IP Jails + +A shared-stack jail has no loopback interface of its own. +The FreeBSD kernel compensates for this: in a non-VNET jail, whenever `127.0.0.1` or `::1` is not itself one of the jail's own addresses, the kernel silently rewrites both `bind()` and `connect()` on the loopback address to the jail's first `ip4.addr` or `ip6.addr`. +This behavior is not documented in man:jail[8], but it matters, because it carries a security consequence. + +[WARNING] +==== +Do not assume that binding a service to `localhost` confines it to the jail. +In a shared-IP jail a daemon told to listen on `127.0.0.1` actually binds the jail's first address, which is frequently a LAN-facing one, and so becomes reachable from the network. +==== + +The standard mitigation is to give the jail a private loopback address and list it first. +Clone a dedicated loopback interface on the host and assign it an address in the `127.0.0.0/8` range (or a private RFC 1918 address): + +[source,shell] +.... +# sysrc cloned_interfaces="lo1" +# sysrc ifconfig_lo1="inet 127.0.1.1/32" +.... + +Then list that address first in the jail's `ip4.addr`, ahead of any routable address, so the loopback remap lands on the private address instead of a LAN-facing one. + +[[preparing-host-services]] +==== Preparing Host Services + +Each jail alias address shares the host's ports. +A host daemon bound to the wildcard address — shown as `*:port` in listener listings — is therefore listening on every jail alias too, and it occupies that port before a jail's own service can claim it. +Before starting any jails, audit what the host already listens on: + +[source,shell] +.... +# sockstat -4 -6 -l +.... + +Restrict each daemon that binds a wildcard address so it uses only the host's own address and leaves the jail addresses free: + +* man:syslogd[8]: set `syslogd_flags="-ss"` in [.filename]#/etc/rc.conf#. +A single `-s` already runs syslogd in secure mode so it does not listen for remote log messages; per syslogd(8), if `-s` "is specified twice, no network socket will be opened at all", which also stops syslogd from sending to remote hosts. +* man:sshd[8]: pin `ListenAddress` to the host's address in [.filename]#/etc/ssh/sshd_config# instead of leaving it on the default wildcard. +* man:ntpd[8] and man:sendmail[8]: bind them to the host address as well through their own configuration. + +For firewall-level control over which addresses a jail may use, see crossref:firewalls[firewalls,Firewalls]. + [[host-configuration-directories]] === Setting Up the Jail Directory Tree @@ -265,6 +368,9 @@ When using OpenZFS, execute the following commands to create datasets for these In this case, `zroot` was used for the parent dataset, but other datasets could have been used. ==== +Creating a separate child dataset per jail is what later enables per-jail snapshots, clones, quotas, and dataset delegation. +Because each jail directory is then its own dataset, OpenZFS can both cap and guarantee its disk use with the `quota` and `reservation` properties; see crossref:jails[jail-resource-limits,Jail Resource Limits]. + When using UFS, execute the following commands to create the directories: [source,shell] @@ -314,7 +420,7 @@ jailname { <.> # NETWORK ip4.addr = 192.168.1.151; <.> - ip6.addr = ::ffff:c0a8:197 <.> + ip6.addr = "2001:db8::151"; <.> interface = em0; <.> } .... @@ -328,21 +434,138 @@ A typical command to run is "/bin/sh /etc/rc.shutdown". <.> `allow.raw_sockets` - Allow creating raw sockets inside the jail. Setting this parameter allows utilities like man:ping[8] and man:traceroute[8] to operate inside the jail. <.> `exec.clean` - Run commands in a clean environment. -<.> `mount.devfs` - Mount a man:devfs[5] filesystem on the chrooted [.filename]#/dev# directory, and apply the ruleset in the devfs_ruleset parameter to restrict the devices visible inside the jail. +<.> `mount.devfs` - Mount a man:devfs[5] filesystem on the chrooted [.filename]#/dev# directory to restrict the devices visible inside the jail. +It applies the ruleset named by the `devfs_ruleset` parameter, or, when that parameter is left at its default of `0`, the `mount.devfs` fallback of ruleset 4, `devfsrules_jail`, which hides every host device except a minimal, jail-safe set (see crossref:jails[jail-conf-devfs, Devfs Rulesets]). <.> `host.hostname` - The hostname of the jail. <.> `path` - The directory which is to be the root of the jail. Any commands that are run inside the jail, either by jail or from man:jexec[8], are run from this directory. -<.> `ip4.addr` - IPv4 address. -There are two configuration possibilities for IPv4. -The first is to establish an IP or a list of IPs as has been done in the example. -The other is to use `ip4` instead and set the `inherit` value to inherit the host's IP address. -<.> `ip6.addr` - IPv6 address. -There are two configuration possibilities for IPv6. -The first is to establish an IP or a list of IPs as has been done in the example. -The other is to use `ip6` instead and set the `inherit` value to inherit the host's IP address. +<.> `ip4.addr` - One IPv4 address, or a comma-separated list of addresses, assigned to the jail. +Setting `ip4.addr` implies `ip4 = new`; `ip4` itself is not left at a standalone default. +The `ip4` parameter accepts three values: `new` restricts the jail to the addresses listed in `ip4.addr`, `inherit` gives the jail unrestricted access to every IPv4 address of the host, and `disable` turns off IPv4 in the jail entirely. +A jail with neither `ip4` nor `ip4.addr` set behaves like `inherit`, and `interface` and `ip4.addr` have no effect under `inherit`. +In a jail with several addresses, `ip4.saddrsel` lets the kernel choose the source address for outgoing connections on unbound sockets, while `ip4.nosaddrsel` forces the first address in the list. +<.> `ip6.addr` - One IPv6 address, or a comma-separated list of addresses, assigned to the jail. +The `ip6` parameter takes the same `new`, `inherit`, and `disable` values as `ip4`, with `ip6.saddrsel` and `ip6.nosaddrsel` controlling source-address selection. +Use `ip6 = disable` for an IPv4-only jail, or `ip4 = disable` for an IPv6-only one. <.> `interface` - A network interface to add the jail's IP addresses. Usually the host interface. +[NOTE] +==== +A jail has no virtual console: man:init[8] and man:getty[8] do not run inside it, so there is nothing to attach to interactively. +The `exec.consolelog` file captures only the standard output and standard error of the `exec.*` commands, that is, the output of the [.filename]#/etc/rc# startup and [.filename]#/etc/rc.shutdown# scripts. +It is not an interactive console; for that, use man:jexec[8] (see crossref:jails[access-jail, Access a Jail]) or run man:sshd[8] inside the jail. +The jail's own daemons send their later output to the jail's syslog. +==== + +[TIP] +==== +The `exec.consolelog` files grow without bound. +Rotate them with man:newsyslog[8] by adding a single glob entry, for example in [.filename]#/etc/newsyslog.conf.d/jail#: + +[.programlisting] +.... +/var/log/jail_console_*.log 640 7 100 * J +.... +==== + +[[global-settings-wildcards-variables]] +==== Global Settings, Wildcards and Variables + +Parameters set outside any jail block are wildcard defaults: man:jail.conf[5] applies them to every jail, exactly as if they were written in a `*` wildcard section. +Setting the shared parameters once keeps each jail block short. + +A name preceded by a dollar sign, and optionally enclosed in braces, is replaced with the value of that parameter or variable. +The built-in `${name}` expands to the jail's own name, and a custom variable is defined just like a parameter but with a leading dollar sign. + +The following [.filename]#/etc/jail.conf# defines a common header once and overrides only what differs per jail: + +[.programlisting] +.... +# Shared defaults applied to every jail +exec.start = "/bin/sh /etc/rc"; +exec.stop = "/bin/sh /etc/rc.shutdown"; +exec.consolelog = "/var/log/jail_console_${name}.log"; +exec.clean; +mount.devfs; +host.hostname = "${name}"; +path = "/usr/local/jails/containers/${name}"; +interface = em0; + +www { + ip4.addr = 192.168.1.151; +} + +db { + host.hostname = "database.example.com"; + ip4.addr = 192.168.1.152; +} +.... + +Here `www` inherits everything from the header and sets only its address, while `db` also overrides `host.hostname`. + +[[jail-conf-hooks]] +==== Command Hooks + +man:jail[8] runs a jail through a sequence of lifecycle hooks, and it matters whether each one runs in the host environment or inside the jail. +In creation order: + +`exec.prestart`:: Runs in the host environment before the jail is created. +`exec.created`:: Runs in the host environment after the jail exists but before any process starts inside it. +`exec.start`:: Runs inside the jail, typically `/bin/sh /etc/rc` to bring up its services. +`exec.poststart`:: Runs in the host environment after the jail has started. + +Shutdown runs the mirror image: + +`exec.prestop`:: Runs in the host environment before the jail is stopped. +`exec.stop`:: Runs inside the jail, typically `/bin/sh /etc/rc.shutdown`. +`exec.poststop`:: Runs in the host environment after the jail has stopped, to tear down whatever `exec.prestart` set up. + +A hook may be assigned more than one command by appending with `+=`: + +[.programlisting] +.... +exec.poststart += "logger jail ${name} started"; +.... + +`exec.timeout` caps how long man:jail[8] waits for a command to finish, and `exec.consolelog` captures the output of these commands. +The VNET section (crossref:jails[creating-vnet-jail, VNET Jails]) uses `exec.prestart` and `exec.poststop` to build and tear down each jail's network plumbing on the host. + +Where hooks order commands within a single jail, the `depend` parameter orders whole jails. +Writing `depend = otherjail;` tells man:jail[8] that this jail requires `otherjail`: the dependency is created first, up to the completion of its last `exec.poststart` command, before this jail is created, and the jails are torn down in the reverse order. + +[[jail-conf-devfs]] +==== Devfs Rulesets + +Every jail that sets `mount.devfs` gets a devfs ruleset that decides which device nodes appear under its [.filename]#/dev#, either the one named by the `devfs_ruleset` parameter or, when that parameter is unset, the `mount.devfs` default of ruleset 4. +man:devfs.rules[5] defines named rulesets in [.filename]#/etc/defaults/devfs.rules#. +Two of them are meant for jails: `devfsrules_jail`, ruleset 4, is the standard minimal device set for a jail, and `devfsrules_jail_vnet`, ruleset 5, adds [.filename]#/dev/pf# so a VNET jail can run its own firewall. +The VNET jail examples set `devfs_ruleset = 5` and the Linux jail example sets `devfs_ruleset = 4`; the classic and thin-jail examples set only `mount.devfs` and so use its default of ruleset 4. + +To expose an extra device, write a custom ruleset in [.filename]#/etc/devfs.rules# that includes one of the stock rulesets and unhides what it needs. +The following ruleset starts from `devfsrules_jail` and additionally unhides the man:bpf[4] devices: + +[.programlisting] +.... +[devfsrules_jail_bpf=100] +add include $devfsrules_jail +add path 'bpf*' unhide +.... + +Reload the rules, then reference the new ruleset by its number in the jail's configuration: + +[source,shell] +.... +# service devfs restart +.... + +[.programlisting] +.... +devfs_ruleset = 100; +.... + +Rulesets in [.filename]#/etc/devfs.rules# override same-numbered rulesets in [.filename]#/etc/defaults/devfs.rules#; otherwise the two files are effectively merged. + More information about configuration variables can be found in man:jail[8] and man:jail.conf[5]. [[classic-jail]] @@ -362,7 +585,7 @@ Execute the following command to download the userland: [source,shell,subs=attributes] .... -# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/{rel-latest}-RELEASE/base.txz -o /usr/local/jails/media/{rel-latest}-RELEASE-base.txz +# fetch https://download.freebsd.org/releases/amd64/amd64/{rel-latest}-RELEASE/base.txz -o /usr/local/jails/media/{rel-latest}-RELEASE-base.txz .... Once the download is complete, it will be necessary to extract the contents into the jail directory. @@ -427,6 +650,117 @@ Execute the following command to start the jail: More information on how to manage jails can be found in the section crossref:jails[jail-management, Jail Management]. +[[creating-jail-userland]] +=== Obtaining the Jail Userland + +The fetch-and-extract procedure shown above is the traditional way to populate a jail's root directory, but it is not the only one. +The same userland can be installed with man:bsdinstall[8], from base-system packages, or from a locally built source tree. + +man:bsdinstall[8] provides a dedicated `jail` target that sets up a new userland at the given directory for use with man:jail[8]. +It fetches and extracts the distribution sets without installing a kernel, skipping the disk partitioning and network steps of a normal installation, which makes it a one-command, scriptable alternative to the manual fetch and man:tar[1] sequence: + +[source,shell] +.... +# bsdinstall jail /usr/local/jails/containers/classic +.... + +The `jail` target installs the base distribution ([.filename]#base.txz#) by default. +On FreeBSD 15.0 and later man:bsdinstall[8] can also populate the root from base-system packages with its `pkgbase` target; the pkg-based method is described next. + +Starting with FreeBSD 15.0 the base system is also published as packages, so a jail userland can be installed entirely with man:pkg[8] from the `FreeBSD-base` repository. +Point man:pkg[8] at the jail's root directory with the global `--rootdir` option and install the jail variant of the base set: + +[source,shell] +.... +# pkg --rootdir /usr/local/jails/containers/classic install -r FreeBSD-base FreeBSD-set-base-jail +.... + +Use `FreeBSD-set-minimal-jail` instead of `FreeBSD-set-base-jail` for a minimal userland. +A jail installed this way is afterwards patched and upgraded with man:pkg[8] rather than man:freebsd-update[8]; see crossref:jails[jail-upgrade-pkgbase, Upgrading pkgbase Jails]. + +[NOTE] +==== +Base-system packages are a technology preview in FreeBSD 15.0. +The `FreeBSD-base` repository is defined in [.filename]#/etc/pkg/FreeBSD.conf# but disabled by default, so select it explicitly with `install -r FreeBSD-base`. +The target root directory and its [.filename]#var/cache/pkg# and [.filename]#var/db/pkg# subdirectories may need to be created first. +The traditional [.filename]#base.txz# distribution is still published for FreeBSD 15.0, so the fetch-and-extract method continues to work there as well. +==== + +A userland built from a source tree can be installed into a jail's root directory with the `DESTDIR` variable, which is the way to run a `-STABLE` or custom-built userland in a jail. +After building the world, install it and the remaining distribution files into the jail directory: + +[source,shell] +.... +# cd /usr/src +# make buildworld +# make installworld DESTDIR=/usr/local/jails/containers/classic +# make distribution DESTDIR=/usr/local/jails/containers/classic +.... + +`make distribution` installs the default configuration files under the jail's [.filename]#/etc# and is run only once, when populating the jail; repeating it during an update would overwrite the configuration inside the jail. +To update such a jail from source, follow the same sequence as for the host: run `installworld` with the same `DESTDIR`, merge the configuration files with `etcupdate -D` pointed at the jail's root directory, and remove obsolete files and libraries with `make delete-old delete-old-libs`, again with the same `DESTDIR`, rather than using man:freebsd-update[8]. +See crossref:cutting-edge[makeworld, Updating FreeBSD from Source] for the complete source-update procedure. + +[[classic-jail-clone]] +=== Cloning a Jail as a Template + +Once a thick jail is fully configured, with packages installed, services enabled, and local configuration in place, it can serve as a template image: a template from which many identical jails are deployed almost instantly. + +On OpenZFS this is done by snapshotting the configured jail's dataset and cloning it. +This requires the source jail to live on its own dataset, so create it with `zfs create` rather than the `mkdir` used in the classic-jail procedure above: + +[source,shell] +.... +# zfs create -p zroot/jails/containers/classic +.... + +Stop the jail before snapshotting so the image is consistent, then snapshot and clone the dataset once for each new jail: + +[source,shell] +.... +# service jail stop classic +# zfs snapshot zroot/jails/containers/classic@template +# zfs clone zroot/jails/containers/classic@template zroot/jails/containers/web1 +.... + +On a UFS host, copy the configured jail's directory tree instead: + +[source,shell] +.... +# cp -a /usr/local/jails/containers/classic /usr/local/jails/containers/web1 +.... + +Each clone needs its own entry in [.filename]#/etc/jail.conf#, or in [.filename]#jail.conf.d#, with a unique name, path, and IP address. +Because the `path` and `host.hostname` parameters in the example above are derived from the block name with `${name}`, only the block name and the address differ: + +[.programlisting] +.... +web1 { + # STARTUP/LOGGING + exec.start = "/bin/sh /etc/rc"; + exec.stop = "/bin/sh /etc/rc.shutdown"; + exec.consolelog = "/var/log/jail_console_${name}.log"; + + # PERMISSIONS + allow.raw_sockets; + exec.clean; + mount.devfs; + + # HOSTNAME/PATH + host.hostname = "${name}"; + path = "/usr/local/jails/containers/${name}"; + + # NETWORK + ip4.addr = 192.168.1.152; + interface = em0; +} +.... + +A template-image clone differs from a thin jail (crossref:jails[thin-jail, Thin Jails]). +A thin jail keeps sharing a read-only base and is maintained once at the template, whereas each template-image clone is a full, independently writable copy that is patched and upgraded on its own as described in crossref:jails[jail-upgrading, Jail Upgrading]. +An OpenZFS clone starts as a space-efficient copy-on-write copy that shares blocks with the snapshot, and it keeps a dependency on that origin snapshot until `zfs promote` severs it. +For a fully independent copy, or to replicate a jail to another pool or host, use `zfs send` piped to `zfs receive` instead of `zfs clone`. + [[thin-jail]] == Thin Jails @@ -440,7 +774,7 @@ such as being able to create them faster from snapshots or being able to update Due to the good integration between FreeBSD and OpenZFS it is very easy to create new Thin Jails using OpenZFS Snapshots. -To create a Thin Jail using OpenZFS Snapshots the first step is to create the jail directory tree by following the instructions in crossref:jails[host-configuration-directories, "Setting up the Jail Directory Tree"]. +To create a Thin Jail using OpenZFS Snapshots the first step is to create the jail directory tree by following the instructions in crossref:jails[host-configuration-directories, "Setting Up the Jail Directory Tree"]. Next, create a template. Templates will only be used to create new jails. @@ -457,7 +791,7 @@ Then execute the following command to download the userland: [source,shell,subs=attributes] .... -# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/{rel-latest}-RELEASE/base.txz -o /usr/local/jails/media/{rel-latest}-RELEASE-base.txz +# fetch https://download.freebsd.org/releases/amd64/amd64/{rel-latest}-RELEASE/base.txz -o /usr/local/jails/media/{rel-latest}-RELEASE-base.txz .... Once the download is complete, it will be necessary to extract the contents in the template directory by executing the following command: @@ -561,7 +895,7 @@ Then execute the following command to download the userland: [source,shell,subs=attributes] .... -# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/{rel-latest}-RELEASE/base.txz -o /usr/local/jails/media/{rel-latest}-RELEASE-base.txz +# fetch https://download.freebsd.org/releases/amd64/amd64/{rel-latest}-RELEASE/base.txz -o /usr/local/jails/media/{rel-latest}-RELEASE-base.txz .... Once the download is complete, it will be necessary to extract the contents in the template directory by executing the following command: @@ -685,7 +1019,7 @@ thinjail { } .... -Then the create the [.filename]#/usr/local/jails/thinjail-nullfs-base.fstab# file as follows: +Then create the [.filename]#/usr/local/jails/thinjail-nullfs-base.fstab# file as follows: [.programlisting,subs=attributes] .... @@ -693,6 +1027,11 @@ Then the create the [.filename]#/usr/local/jails/thinjail-nullfs-base.fstab# fil /usr/local/jails/containers/thinjail /usr/local/jails/thinjail-nullfs-base/skeleton nullfs rw 0 0 .... +The [.filename]#fstab# file controls how man:mount_nullfs[8] presents the base template and the skeleton inside the jail. +The base template is mounted read-only (`ro`) so that every NullFS jail shares one immutable copy of the base system. +The per-jail skeleton is mounted read-write (`rw`), which is where each jail keeps its own writable [.filename]#/etc#, [.filename]#/var#, [.filename]#/usr/local#, and other local directories. +Because all jails share that single read-only base template, updating the one template updates every NullFS jail at once, in contrast to independent OpenZFS clones which must each be updated separately. + Execute the following command to start the jail: [source,shell] @@ -701,7 +1040,7 @@ Execute the following command to start the jail: .... [[creating-vnet-jail]] -=== Creating a VNET Jail +== VNET Jails FreeBSD VNET Jails have their own distinct networking stack, including interfaces, IP addresses, routing tables, and firewall rules. @@ -719,6 +1058,14 @@ The output should be similar to the following: bridge0 .... +[WARNING] +==== +Assigning an IP address to an interface that is a member of a bridge is deprecated. +Assign the host's address to `bridge0` itself rather than to `em0`, exactly as the [.filename]#/etc/rc.conf# block below does. +Such configurations keep working for now: FreeBSD 15.0 logs a kernel warning when an interface that carries an address is added to a bridge, and refuses it with `EINVAL` only when the `net.link.bridge.member_ifaddrs` sysctl is changed from its default of `1` to `0`. +man:bridge[4] announces the removal of this sysctl in FreeBSD 16.0, after which addresses on bridge members will no longer be permitted. +==== + With the `bridge` created, it will be necessary to attach it to the `em0` interface and bring both of them up by executing the following commands: [source,shell] @@ -741,7 +1088,7 @@ For more information on bridging, see crossref:advanced-networking[network-bridg The next step is to create the jail as indicated above. -Either the crossref:jails[classic-jail, Classic Jail (Thick Jail)] procedure and the +Either the crossref:jails[classic-jail, Classic Jail (Thick Jail)] procedure or the crossref:jails[thin-jail, Thin Jails] procedure can be used. The only thing that will change is the configuration in the [.filename]#/etc/jail.conf# file. @@ -782,20 +1129,85 @@ vnet { exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up"; exec.start += "/sbin/ifconfig ${epair}b ${ip} up"; exec.start += "/sbin/route add default ${gateway}"; - exec.start += "/bin/sh /etc/rc"; - exec.stop = "/bin/sh /etc/rc.shutdown"; + exec.start += "/bin/sh /etc/rc"; + exec.stop = "/bin/sh /etc/rc.shutdown"; exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a"; exec.poststop += "/sbin/ifconfig ${epair}a destroy"; } .... -<.> Represents the IP of the Jail, it must be *unique*. +<.> A per-jail *unique* number used to derive both the epair device name (`epair${id}`) and the last octet of the jail's address (`192.168.1.${id}`); give each jail on the bridge a different value. <.> Refers to the bridge created previously. +The `vnet.interface` parameter names the interface that man:jail[8] moves into the jail's network stack once the jail is created, here the jail side of the epair, `${epair}b`. +That interface disappears from the host while the jail runs and is automatically released back to the host when the jail stops. +A physical network card or an SR-IOV virtual function can be handed to a jail the same way, giving it dedicated networking without a bridge or an epair. + +The `devfs_ruleset = 5` line applies the `devfsrules_jail_vnet` ruleset from [.filename]#/etc/defaults/devfs.rules#, which extends the default jail ruleset by exposing [.filename]#/dev/pf#. +A VNET jail can therefore run its own man:pf[4] firewall with `pf_enable="YES"` and a private [.filename]#pf.conf#, independent of the host. +See crossref:jails[jail-conf-devfs, Devfs Rulesets] for how rulesets are defined and crossref:firewalls[firewalls, Firewalls] for pf itself. + +[[jail-vnet-jib]] +=== Automating VNET Networking with jib + +Wiring up an epair and attaching it to the bridge by hand in `exec.prestart` and `exec.poststop`, as shown above, becomes tedious across many jails. +The base system ships a helper script, [.filename]#/usr/share/examples/jails/jib#, that performs the same plumbing automatically. + +Reference it from the jail's hooks instead of the five hand-rolled man:ifconfig[8] lines: + +[.programlisting] +.... + exec.prestart += "jib addm ${name} em0"; + exec.poststop += "jib destroy ${name}"; + + vnet.interface = "e0b_${name}"; +.... + +`jib addm` creates an epair whose host side is named `e0a_<name>` and whose jail side is `e0b_<name>`, then attaches the host side to a bridge that it creates automatically, named after the member interface (for `em0` the bridge is `em0bridge`). +Name that jail side in `vnet.interface`. +When the jail stops, `jib destroy` removes the epair. +The script also provides a `jib show` subcommand to list the interfaces it manages. + +[[jail-vnet-jng]] +=== Netgraph VNET Networking with jng + +The third networking mode mentioned in the crossref:jails[jails-networking, Networking] overview uses man:netgraph[4] instead of man:bridge[4] and epair. +The base system ships a second helper, [.filename]#/usr/share/examples/jails/jng#, that builds an `ng_bridge` connected to an `ng_eiface` for each jail: + +[.programlisting] +.... + exec.prestart += "jng bridge ${name} em0"; + exec.poststop += "jng shutdown ${name}"; + + vnet.interface = "ng0_${name}"; +.... + +`jng bridge` creates the netgraph nodes and a jail-side interface named `ng0_<name>`, which is named in `vnet.interface`; `jng shutdown` tears the topology down when the jail stops. +The result is functionally equivalent to the epair-and-bridge setup, but built on netgraph nodes rather than man:bridge[4]. + +[[jail-vnet-firewall]] +=== Running a Firewall and DHCP inside a VNET Jail + +Because `devfs_ruleset = 5` exposes [.filename]#/dev/pf#, a VNET jail can run its own firewall. +Enable it inside the jail exactly as on any host, with `pf_enable="YES"` in the jail's [.filename]#/etc/rc.conf# and its own [.filename]#/etc/pf.conf#. + +Ruleset 5 does not expose [.filename]#/dev/bpf#, however. +A VNET jail that obtains its address over DHCP, for example with `ifconfig_e0b_myjail="SYNCDHCP"` in its [.filename]#/etc/rc.conf#, runs man:dhclient[8], which needs bpf and therefore fails under ruleset 5. +Following the recipe in crossref:jails[jail-conf-devfs,Devfs Rulesets], define a custom ruleset in [.filename]#/etc/devfs.rules# that starts from `devfsrules_jail_vnet` and additionally unhides the bpf devices: + +[.programlisting] +.... +[devfsrules_jail_vnet_dhcp=101] +add include $devfsrules_jail_vnet +add path 'bpf*' unhide +.... + +After running `service devfs restart` on the host, point the jail at the new ruleset with `devfs_ruleset = 101;`. + [[creating-linux-jail]] -=== Creating a Linux Jail +== Linux Jails -FreeBSD can run Linux inside a jail using crossref:linuxemu[linuxemu,Linux Binary Compatibility] and man:debootstrap[8]. +FreeBSD can run a Linux userland inside a jail using crossref:linuxemu[linuxemu,Linux Binary Compatibility] and package:sysutils/debootstrap[]. Jails do not have a kernel. They run on the host's kernel. Therefore it is necessary to enable Linux Binary Compatibility in the host system. @@ -814,18 +1226,21 @@ Once enabled, it can be started without rebooting by executing the following com # service linux start .... -The next step will be to create a jail as indicated above, for example in -crossref:jails[creating-thin-jail-openzfs-snapshots, Creating a Thin Jail Using OpenZFS Snapshots], but *without* performing the configuration. -FreeBSD Linux jails require a specific configuration that will be detailed below. +The next step is to create a jail as indicated above, for example as in crossref:jails[creating-thin-jail-openzfs-snapshots, Creating a Thin Jail Using OpenZFS Snapshots], but *without* performing the configuration. +FreeBSD Linux jails require a specific configuration that is detailed below. -Create an empty [.filename]#/etc/jail.conf# to prevent failures when trying to stop the jail: +If the host does not already have an [.filename]#/etc/jail.conf# — for example on a system whose jails so far have been created only from the command line — create an empty one so that man:service[8] can stop the jail later: [source,shell] .... # touch /etc/jail.conf .... -Once the jail has been created as explained above, execute the following command to perform required configuration for the jail and start it: +The man:service[8] jail script reads [.filename]#/etc/jail.conf# when stopping a jail, so the stop command fails if the file does not exist. +If an [.filename]#/etc/jail.conf# is already present from an earlier section, this step is not needed. + +The following man:jail[8] command performs the required configuration for the jail and starts it directly, without a [.filename]#/etc/jail.conf# entry. +The `-c` flag creates a new jail and `-m` modifies an existing one; combined as `-cm`, man:jail[8] creates the jail if it does not yet exist and updates it if it does: [source,shell] .... @@ -834,7 +1249,7 @@ Once the jail has been created as explained above, execute the following command host.hostname="ubuntu.example.com" \ path="/usr/local/jails/containers/ubuntu" \ interface="em0" \ - ip4.addr="192.168.1.150" \ + ip4.addr="192.168.1.155" \ exec.start="/bin/sh /etc/rc" \ exec.stop="/bin/sh /etc/rc.shutdown" \ mount.devfs \ @@ -849,16 +1264,16 @@ Once the jail has been created as explained above, execute the following command enforce_statfs=1 .... -To access the jail, it will be necessary to install package:sysutils/debootstrap[]. - -Execute the following command to access the FreeBSD Linux jail: +Access the newly created jail as `root` with man:jexec[8]: [source,shell] .... # jexec -u root ubuntu sh .... -Inside the jail, execute the following commands to install man:pkg[7] and package:sysutils/debootstrap[] to prepare the Ubuntu environment: +Inside the jail, install package:sysutils/debootstrap[] and use it to build the Ubuntu userland under [.filename]#/compat/ubuntu#. +package:sysutils/debootstrap[] is the tool that builds the Linux userland; it is not required to enter the jail afterward. +Running man:pkg[7] for the first time bootstraps it automatically: [source,shell] .... @@ -866,14 +1281,19 @@ Inside the jail, execute the following commands to install man:pkg[7] and packag # debootstrap jammy /compat/ubuntu .... -When the process has finished and the message `Base system installed successfully` is displayed on the console, -it will be necessary to stop the jail from the host system by executing the following command: +`jammy` selects Ubuntu 22.04 LTS. +A newer LTS such as `noble` (Ubuntu 24.04) can be selected instead, provided it runs under the linuxulator. + +When the process has finished and the message `Base system installed successfully` is displayed on the console, stop the jail from the host system. +Because this jail is not enabled in [.filename]#/etc/rc.conf# and not yet listed in [.filename]#/etc/jail.conf#, use the `one`-prefixed man:service[8] action, which runs even for a jail that is not enabled: [source,shell] .... # service jail onestop ubuntu .... +The `onestart` and `onestop` actions are the counterparts of `start` and `stop` for jails that are not enabled through `jail_enable` or listed in `jail_list`. + Then add an entry in [.filename]#/etc/jail.conf# for the Linux jail: [.programlisting] @@ -909,6 +1329,13 @@ ubuntu { } .... +[TIP] +==== +The jail can also present a specific Linux kernel identity to the programs running inside it. +The `linux` parameter selects whether the jail inherits the host's Linux emulation environment (`inherit`) or gets its own (`new`), and `linux.osname`, `linux.osrelease`, and `linux.oss_version` set the OS name, release, and OSS version that Linux binaries see. +For example, `linux.osrelease=6.1.0` makes man:uname[1] and version-checking Linux programs inside the jail report that kernel release instead of the host-wide `compat.linux.osrelease` value. +==== + Then the jail can be started as usual with the following command: [source,shell] @@ -925,16 +1352,67 @@ The Ubuntu environment can be accessed using the following command: More information can be found in the chapter crossref:linuxemu[linuxemu,Linux Binary Compatibility]. +[[jail-nat]] +== NAT and Port Forwarding for Jails + +A jail that is given a private address — on a cloned man:lo[4] interface such as `lo1`, on a private bridge, or on a VNET epair on an internal network — has no route to the internet of its own. +The host must translate the jail's traffic to a routable address, and, to reach a service running inside such a jail from outside, redirect incoming connections to the jail. + +First, allow the host to forward packets between its interfaces. +Enable it at boot and immediately with the following commands: + +[source,shell] +.... +# sysrc gateway_enable="YES" +# sysctl net.inet.ip.forwarding=1 +.... + +For IPv6, use `ipv6_gateway_enable="YES"` and `net.inet6.ip6.forwarding=1`. + +Then configure Network Address Translation on the host with man:pf[4]. +The following [.filename]#/etc/pf.conf# gives every jail on the private `192.168.0.0/24` network outbound access through the host's external interface `em0`, and publishes a jailed web server on the host's port 80: + +[.programlisting] +.... +ext_if = "em0" + +# Translate outbound jail traffic to the host address +nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if) + +# Forward inbound web traffic to the jailed server +rdr on $ext_if proto tcp to port 80 -> 192.168.0.151 +.... + +The `rdr` target must be the jail's own address. + +Enable and start man:pf[4] with the following commands: + +[source,shell] +.... +# sysrc pf_enable="YES" +# service pf start +.... + +[NOTE] +==== +FreeBSD 15.0 adds the OpenBSD-style inline `nat-to` and `rdr-to` syntax, which attaches translation to a `pass` or `match` rule, for example `match out on $ext_if from 192.168.0.0/24 to any nat-to ($ext_if)`. +The classic `nat on` and `rdr on` rules shown above work on both FreeBSD 14.x and 15.x. +==== + +For the full treatment of man:pf[4] and man:ipfw[8], including complete rule sets and man:ipfw[8]-based NAT, see crossref:firewalls[firewalls,Firewalls]. + [[service-jails-config]] -=== Configuring Service Jails +== Service Jails +Service jails are available since FreeBSD 15.0. A service jail is configured completely via [.filename]#/etc/rc.conf# or man:sysrc[8]. The base system services are service jails ready. -They contain a config line which enables networking or lift other restrictions of jails. +They contain a config line which enables networking or lifts other restrictions of jails. Base system services which do not make sense to run inside jails are configured to not be started as a service jail, even if enabled in [.filename]#/etc/rc.conf#. -Some examples of such a service are services which want to mount or unmount something in the start of stop method, or only configure something like a route, or firewall, or the like. +Some examples of such a service are services which want to mount or unmount something in the start or stop method, or only configure something like a route, or firewall, or the like. -Third party services may or may not be service jails ready. To check if a service is service jail ready, the following command can be used: +Third party services may or may not be service jails ready. +To check if a service is service jail ready, the following command can be used: [source,shell] .... @@ -966,7 +1444,7 @@ If the `servicename_svcj` variable is changed, the service needs to be stopped b If it is not stopped, the rc framework will not detect the correct state of the service and will not be able to do what is requested. Service jails are managed only via man:rc.conf[5]/man:sysrc[8] and the man:service[8] command. -The jail utilities, like man:jls[8] as described in crossref:jails[jail-management,Jail Management] can be used to investigate the operation, but the man:jail[8] command is not supposed to be used to manage them. +The jail utilities, like man:jls[8] as described in crossref:jails[jail-management,Jail Management], can be used to investigate the operation, but the man:jail[8] command is not supposed to be used to manage them. [[jail-management]] == Jail Management @@ -988,7 +1466,7 @@ The output should be similar to the following: .... JID IP Address Hostname Path - 1 192.168.250.70 classic /usr/local/jails/containers/classic + 1 192.168.1.151 classic /usr/local/jails/containers/classic .... man:jls[8] supports the `--libxo` argument, which through the man:libxo[3] library allows other types of formats to be displayed, such as `JSON`, `HTML`, etc. @@ -1003,54 +1481,117 @@ For example, execute the following command to get the `JSON` output: The output should be similar to the following: .... -{"__version": "2", "jail-information": {"jail": [{"jid":1,"ipv4":"192.168.250.70","hostname":"classic","path":"/usr/local/jails/containers/classic"}]}} +{"__version": "2", "jail-information": {"jail": [{"jid":1,"ipv4":"192.168.1.151","hostname":"classic","path":"/usr/local/jails/containers/classic"}]}} .... -[[start-jail]] -=== Start, Restart, and Stop a Jail - -man:service[8] is used to start, reboot, or stop a jail on the host. - -For example, to start a jail, run the following command: +The default listing is only a small part of what man:jls[8] can report. +Add the `-v` flag for a verbose, multi-line summary of each jail that includes its state and its assigned cpuset ID: [source,shell] .... -# service jail start jailname +# jls -v .... -Change the `start` argument to `restart` or `stop` to perform other actions on the jail. - -[[destroy-jail]] -=== Destroy a Jail - -Destroying a jail is not as simple as stopping the jail using man:service[8] and removing the jail directory and [.filename]#/etc/jail.conf# entry. - -FreeBSD takes system security very seriously. -For this reason there are certain files that not even the root user can delete. -This functionality is known as File Flags. - -The first step is to stop the desired jail executing the following command: +Use `-h` to print a header line with the field names above the values of the requested parameters: [source,shell] .... -# service jail stop jailname +# jls -h jid name path .... -The second step is to remove these flags with man:chflags[1] by executing the following command, in which `classic` is the name of the jail to remove: +To query specific parameters of a single jail, name it with `-j` and list the parameters to print: [source,shell] .... -# chflags -R 0 /usr/local/jails/containers/classic +# jls -j classic path host.hostname ip4.addr .... -The third step is to delete the directory where the jail was: +The output should be similar to the following: -[source,shell] .... -# rm -rf /usr/local/jails/containers/classic +/usr/local/jails/containers/classic classic 192.168.1.151 .... -Finally, it will be necessary to remove the jail entry in [.filename]#/etc/jail.conf# or in [.filename]#jail.conf.d#. +Add `-d` to include jails that are shutting down, which are otherwise hidden while in the dying state: + +[source,shell] +.... +# jls -d +.... + +[[start-jail]] +=== Start, Restart, and Stop a Jail + +man:service[8] is used to start, reboot, or stop a jail on the host. + +For example, to start a jail, run the following command: + +[source,shell] +.... +# service jail start jailname +.... + +Change the `start` argument to `restart` or `stop` to perform other actions on the jail. + +Stopping a jail is more than killing the processes inside it. +When a jail is stopped, man:jail[8] first runs the commands in its `exec.stop` parameter — typically `/bin/sh /etc/rc.shutdown` — inside the jail so that its services shut down cleanly. +It then sends `SIGTERM` to any processes still running in the jail and waits up to `stop.timeout` seconds (10 by default) for them to exit. +Once that time has passed, the jail is removed, which kills whatever remains with `SIGKILL`. +If `stop.timeout` is set to zero, no `SIGTERM` is sent and the jail is removed immediately. +This is why a jail with a slow-stopping daemon can appear to hang for ten seconds while it stops; raise `stop.timeout` in [.filename]#/etc/jail.conf# for a jail that runs a database or another service that needs longer to shut down gracefully. + +[[destroy-jail]] +=== Destroy a Jail + +Destroying a jail is not as simple as stopping the jail using man:service[8] and removing the jail directory and [.filename]#/etc/jail.conf# entry. + +FreeBSD takes system security very seriously. +For this reason there are certain files that not even the root user can delete. +This functionality is known as File Flags. +On a jail whose files live on a UFS file system, these flags must be cleared before the jail directory can be removed. + +The first step is to stop the desired jail executing the following command: + +[source,shell] +.... +# service jail stop classic +.... + +The second step is to remove these flags with man:chflags[1] by executing the following command, in which `classic` is the name of the jail to remove: + +[source,shell] +.... +# chflags -R 0 /usr/local/jails/containers/classic +.... + +The third step is to delete the directory where the jail was: + +[source,shell] +.... +# rm -rf /usr/local/jails/containers/classic +.... + *** 769 LINES SKIPPED ***