MatteoRiondatoContributed by JailsjailsSynopsisThis chapter will provide an explanation of what &os; jails
are and how to use them. Jails, sometimes referred to as an
enhanced replacement of chroot environments,
are a very powerful tool for system administrators, but their basic
usage can also be useful for advanced users.After reading this chapter, you will know:What a jail is, and what purpose it may serve in &os;
installations.How to build, start, and stop a jail.The basics of jail administration, both from inside
and outside the jail.Other sources of useful information about jails are:The &man.jail.8; manual page. This is the full reference
of the jail utility — the
administrative tool which can be used in &os; to start, stop,
and control &os; jails.The mailing lists and their archives. The archives of the
&a.questions; and other mailing lists hosted by the
&a.mailman.lists; already contain a wealth of material for
jails. It should always be engaging to search the archives,
or post a new question to the &a.questions.name; mailing
list.Terms Related to JailsTo facilitate better understanding of parts of the &os; system
related to jails, their internals and the way they interact with
the rest of &os;, the following terms are used further in this
chapter:&man.chroot.8; (command)Utility, which uses &man.chroot.2; &os; system call to change
the root directory of a process and all its descendants.&man.chroot.2; (environment)The environment of processes running in
a chroot. This includes resources such as the part
of the file system which is visible, user and group IDs which are
available, network interfaces and other IPC mechanisms,
etc.&man.jail.8; (command)The system administration utility which allows launching of
processes within a jail environment.host (system, process, user, etc.)The controlling system of a jail environment. The host system
has access to all the hardware resources available, and can
control processes both outside of and inside a jail environment.
One of the important differences of the host system from a jail is
that the limitations which apply to superuser processes inside a
jail are not enforced for processes of the host system.hosted (system, process, user, etc.)A process, user or other entity, whose access to resources is
restricted by a &os; jail.IntroductionSince system administration is a difficult and perplexing
task, many powerful tools were developed to make life easier for
the administrator. These tools mostly provide enhancements of some sort
to the way systems are installed, configured and maintained.
Part of the tasks which an administrator is
expected to do is to properly configure the security of a system,
so that it can continue serving its real purpose, without allowing
security violations.One of the tools which can be used to enhance the security of
a &os; system are jails. Jails were
introduced in &os; 4.X by &a.phk;, but were greatly improved in
&os; 5.X to make them a powerful and flexible subsystem. Their
development still goes on, enhancing their usefulness, performance, reliability,
and security.What is a JailBSD-like operating systems have had &man.chroot.2; since the
time of 4.2BSD. The &man.chroot.8; utility can be used to
change the root directory
of a set of processes, creating a safe environment, separate
from the rest of the system. Processes created in the chrooted
environment can not access files or resources outside of it.
For that reason, compromising a service running in a chrooted
environment should not allow the attacker to compromise the
entire system. The &man.chroot.8; utility is good for easy
tasks, which do not require a lot of flexibility or complex and
advanced features. Since the inception of the
chroot concept, however, many ways have been found to escape from a
chrooted environment and, although they have been fixed in
modern versions of the &os; kernel, it was clear that
&man.chroot.2; was not the ideal solution for securing services.
A new subsystem had to be implemented.This is one of the main reasons why
jails were developed.Jails improve on the concept of the traditional
&man.chroot.2; environment, in several ways. In a traditional
&man.chroot.2; environment, processes are only limited in the
part of the file system they can access. The rest of the system
resources (like the set of system users, the running processes,
or the networking subsystem) are shared by the chrooted
processes and the processes of the host system. Jails expand
this model by virtualizing not only access to the file system,
but also the set of users, the networking subsystem of the &os;
kernel and a few other things. A more complete set of
fine-grained controls available for tuning the access of a
jailed environment is described in .A jail is characterized by four elements:A directory subtree — the starting point from
which a jail is entered. Once inside the jail, a process
is not permitted to escape outside of this subtree.
Traditional security issues which plagued the original
&man.chroot.2; design will not affect &os; jails.A hostname — the hostname which will be used
within the jail. Jails are mainly used for hosting network
services, therefore having a descriptive hostname for each
jail can really help the system administrator.An IP address — this will be
assigned to the jail and cannot be changed in any way during
the jail's life span. The IP address of a jail is usually an alias address
for an existing network interface, but this is not strictly necessary.A command — the path name of an executable to run
inside the jail. This is relative to the root directory of
the jail environment, and may vary a lot, depending on the
type of the specific jail environment.Apart from these, jails can have their own set of users and
their own root user. Naturally, the powers
of the root user are limited within the
jail environment and, from the point of view of the host system,
the jail root user is not an omnipotent user.
In addition, the root user of a jail is not
allowed to perform critical operations to the system outside of
the associated &man.jail.8; environment. More information
about capabilities and restrictions of the
root user will be discussed in below.Creating and Controlling JailsSome administrators divide jails into the following two types:
complete jails, which resemble a real &os; system,
and service jails, dedicated to one application or
service, possibly running with privileges. This is only a
conceptual division and the process of building a jail is not
affected by it. The &man.jail.8; manual page is quite clear about
the procedure for building a jail:&prompt.root; setenv D /here/is/the/jail
&prompt.root; mkdir -p $D
&prompt.root; cd /usr/src
&prompt.root; make buildworld
&prompt.root; make installworld DESTDIR=$D
&prompt.root; make distribution DESTDIR=$D
&prompt.root; mount -t devfs devfs $D/devSelecting a location for a jail is the best starting point.
This is where the jail will physically reside within the file system of the jail's host.
A good choice can be /usr/jail/jailname,
where jailname is the hostname
identifying the jail. The /usr/ file system usually has
enough space for the jail file system, which for complete jails is, essentially,
a replication of every file present in a default installation
of the &os; base system.If you have already rebuilt your userland using
make world or make buildworld,
you can skip this step and install your existing userland into the
new jail.This command will populate the directory subtree chosen
as jail's physical location on the file system with the
necessary binaries, libraries, manual pages and so on.The distribution target for
make installs every needed
configuration file. In simple words, it installs every installable file of
/usr/src/etc/ to the
/etc directory of the jail
environment:
$D/etc/.Mounting the &man.devfs.8; file system inside a jail is
not required. On the other hand, any, or almost any
application requires access to at least one device, depending
on the purpose of the given application. It is very important
to control access to devices from inside a jail, as improper
settings could permit an attacker to do nasty things in the
jail. Control over &man.devfs.8; is managed through rulesets
which are described in the &man.devfs.8; and
&man.devfs.conf.5; manual pages.Once a jail is installed, it can be started by using the
&man.jail.8; utility. The &man.jail.8; utility takes four
mandatory arguments which are described in the . Other arguments may be
specified too, e.g., to run the jailed process with the credentials of a specific
user. The argument depends on
the type of the jail; for a virtual system,
/etc/rc is a good choice, since it will
replicate the startup sequence of a real &os; system. For a
service jail, it depends on the service or
application that will run within the jail.Jails are often started at boot time and the &os;
rc mechanism provides an easy way to do
this.A list of the jails which are enabled to start at boot
time should be added to the &man.rc.conf.5; file:jail_enable="YES" # Set to NO to disable starting of any jails
jail_list="www" # Space separated list of names of jailsJail names in jail_list should
contain alphanumeric characters only.For each jail listed in jail_list, a
group of &man.rc.conf.5; settings, which describe the
particular jail, should be added:jail_www_rootdir="/usr/jail/www" # jail's root directory
jail_www_hostname="www.example.org" # jail's hostname
jail_www_ip="192.168.0.10" # jail's IP address
jail_www_devfs_enable="YES" # mount devfs in the jail
jail_www_devfs_ruleset="www_ruleset" # devfs ruleset to apply to jailThe default startup of jails configured in
&man.rc.conf.5;, will run the /etc/rc
script of the jail, which assumes the jail is a complete
virtual system. For service jails, the default startup
command of the jail should be changed, by setting the
jail_jailname_exec_start
option appropriately.For a full list of available options, please see the
&man.rc.conf.5; manual page.The /etc/rc.d/jail script can be used to
start or stop a jail by hand, if an entry for it exists in
rc.conf:&prompt.root; /etc/rc.d/jail start www
&prompt.root; /etc/rc.d/jail stop wwwA clean way to shut down a &man.jail.8; is not available at
the moment. This is because commands normally used to accomplish
a clean system shutdown cannot be used inside a jail. The best
way to shut down a jail is to run the following command from
within the jail itself or using the &man.jexec.8; utility from
outside the jail:&prompt.root; sh /etc/rc.shutdownMore information about this can be found in the &man.jail.8;
manual page.Fine Tuning and AdministrationThere are several options which can be set for any jail, and
various ways of combining a host &os; system with jails, to produce
higher level applications. This section presents:Some of the options available for tuning the behavior and
security restrictions implemented by a jail
installation.Some of the high-level applications for jail management,
which are available through the &os; Ports Collection, and can
be used to implement overall jail-based solutions.System tools for jail tuning in &os;Fine tuning of a jail's configuration is mostly done by
setting &man.sysctl.8; variables. A special subtree of sysctl
exists as a basis for organizing all the relevant options: the
security.jail.* hierarchy of &os; kernel
options. Here is a list of the main jail-related sysctls,
complete with their default value. Names should be
self-explanatory, but for more information about them, please
refer to the &man.jail.8; and &man.sysctl.8; manual
pages.security.jail.set_hostname_allowed:
1security.jail.socket_unixiproute_only:
1security.jail.sysvipc_allowed:
0security.jail.enforce_statfs:
2security.jail.allow_raw_sockets:
0security.jail.chflags_allowed:
0security.jail.jailed: 0These variables can be used by the system administrator of
the host system to add or remove some of
the limitations imposed by default on the
root user. Note that there are some
limitations which cannot be removed. The
root user is not allowed to mount or
unmount file systems from within a &man.jail.8;. The
root inside a jail may not load or unload
&man.devfs.8; rulesets, set firewall rules, or do many other
administrative tasks which require modifications of in-kernel
data, such as setting the securelevel of the
kernel.The base system of &os; contains a basic set of tools for
viewing information about the active jails, and attaching to a
jail to run administrative commands. The &man.jls.8; and
&man.jexec.8; commands are part of the base &os; system, and can be used
to perform the following simple tasks:Print a list of active jails and their corresponding
jail identifier (JID),
IP address, hostname and path.Attach to a running jail, from its host system, and run
a command inside the jail or perform administrative tasks inside the
jail itself. This is especially useful when the
root user wants to cleanly shut down a
jail. The &man.jexec.8; utility can also be used to start a
shell in a jail to do administration in it; for
example:&prompt.root; jexec 1 tcshHigh-level administrative tools in &os; Ports
CollectionAmong the many third-party utilities for jail administration,
one of the most complete and useful is sysutils/jailutils. It is a set of
small applications that contribute to &man.jail.8; management.
Please refer to its web page for more information.Application of JailsDanielGerzoContributed by Service JailsThis section is based upon an idea originally presented by
&a.simon; at , and an
updated article written by Ken Tom
locals@gmail.com. This section illustrates how
to set up a &os; system that adds an additional layer of
security, using the &man.jail.8; feature. It is also assumed
that the given system is at least running RELENG_6_0 and the
information provided earlier in this chapter has been well
understood.DesignOne of the major problems with jails is the management of
their upgrade process. This tends to be a problem because
every jail has to be rebuilt from scratch whenever it is
updated. This is usually not a problem for a single jail,
since the update process is fairly simple, but can be quite
time consuming and tedious if a lot of jails are
created.This setup requires advanced experience with &os; and
usage of its features. If the presented steps below look
too complicated, it is advised to take a look at a simpler
system such as sysutils/ezjail, which provides
an easier method of administering &os; jails and is not as
sophisticated as this setup.This idea has been presented to resolve such issues by
sharing as much as is possible between jails, in a safe way
— using read-only &man.mount.nullfs.8; mounts, so that
updating will be be simpler, and putting single services into
individual jails will become more attractive. Additionally,
it provides a simple way to add or remove jails as well as a
way to upgrade them.Examples of services in this context are: an
HTTP server, a DNS
server, a SMTP server, and so forth.The goals of the setup described in this section
are:Create a simple and easy to understand jail structure.
This implies not having to run a full
installworld on each and every jail.Make it easy to add new jails or remove existing
ones.Make it easy to update or upgrade existing
jails.Make it possible to run a customized &os;
branch.Be paranoid about security, reducing as much as
possible the possibility of compromise.Save space and inodes, as much as possible.As it has been already mentioned, this design relies
heavily on having a single master template which is read-only
(known as nullfs) mounted into each
jail and one read-write device per jail. A device can be a
separate physical disc, a partition, or a vnode backed
&man.md.4; device. In this example, we will use read-write
nullfs mounts.The file system layout is described in the following
list:Each jail will be mounted under the /home/j directory./home/j/mroot is
the template for each jail and the read-only partition for
all of the jails.A blank directory will be created for each jail under
the /home/j
directory.Each jail will have a /s directory, that will be
linked to the read-write portion of the system.Each jail shall have its own read-write system that is
based upon /home/j/skel.Each jailspace (read-write portion of each jail) shall
be created in /home/js.This assumes that the jails are based under the
/home partition. This
can, of course, be changed to anything else, but this change
will have to be reflected in each of the examples
below.Creating the TemplateThis section will describe the steps needed to create the
master template that will be the read-only portion for the
jails to use.It is always a good idea to update the &os; system to the
latest -RELEASE branch. Check the corresponding Handbook
Chapter
to accomplish this task. In the case the update is not
feasible, the buildworld will be required in order to be able
to proceed. Additionally, the sysutils/cpdup package will be
required. We will use the &man.portsnap.8; utility to
download the &os; Ports Collection. The Handbook Portsnap Chapter
is always good reading for newcomers.First, create a directory structure for the read-only
file system which will contain the &os; binaries for our
jails, then change directory to the &os; source tree and
install the read-only file system to the jail
template:&prompt.root; mkdir /home/j /home/j/mroot
&prompt.root; cd /usr/src
&prompt.root; make installworld DESTDIR=/home/j/mrootNext, prepare a &os; Ports Collection for the jails as
well as a &os; source tree, which is required for
mergemaster:&prompt.root; cd /home/j/mroot
&prompt.root; mkdir usr/ports
&prompt.root; portsnap -p /home/j/mroot/usr/ports fetch extract
&prompt.root; cpdup /usr/src /home/j/mroot/usr/srcCreate a skeleton for the read-write portion of the
system:&prompt.root; mkdir /home/j/skel /home/j/skel/home /home/j/skel/usr-X11R6 /home/j/skel/distfiles
&prompt.root; mv etc /home/j/skel
&prompt.root; mv usr/local /home/j/skel/usr-local
&prompt.root; mv tmp /home/j/skel
&prompt.root; mv var /home/j/skel
&prompt.root; mv root /home/j/skelUse mergemaster to install
missing configuration files. Then get rid of the extra
directories that mergemaster
creates:&prompt.root; mergemaster -t /home/j/skel/var/tmp/temproot -D /home/j/skel -i
&prompt.root; cd /home/j/skel
&prompt.root; rm -R bin boot lib libexec mnt proc rescue sbin sys usr devNow, symlink the read-write file system to the
read-only file system. Please make sure that the symlinks
are created in the correct s/ locations. Real
directories or the creation of directories in the wrong
locations will cause the installation to fail.&prompt.root; cd /home/j/mroot
&prompt.root; mkdir s
&prompt.root; ln -s s/etc etc
&prompt.root; ln -s s/home home
&prompt.root; ln -s s/root root
&prompt.root; ln -s ../s/usr-local usr/local
&prompt.root; ln -s ../s/usr-X11R6 usr/X11R6
&prompt.root; ln -s ../../s/distfiles usr/ports/distfiles
&prompt.root; ln -s s/tmp tmp
&prompt.root; ln -s s/var varAs a last step, create a generic
/home/j/skel/etc/make.conf with its
contents as shown below:WRKDIRPREFIX?= /s/portbuildHaving WRKDIRPREFIX set up this
way will make it possible to compile &os; ports inside
each jail. Remember that the ports directory is part of
the read-only system. The custom path for
WRKDIRPREFIX allows builds to be done
in the read-write portion of every jail.Creating JailsNow that we have a complete &os; jail template, we can
setup and configure the jails in
/etc/rc.conf. This example demonstrates
the creation of 3 jails: NS,
MAIL and WWW.Put the following lines into the
/etc/fstab file, so that the
read-only template for the jails and the read-write space
will be available in the respective jails:/home/j/mroot /home/j/ns nullfs ro 0 0
/home/j/mroot /home/j/mail nullfs ro 0 0
/home/j/mroot /home/j/www nullfs ro 0 0
/home/js/ns /home/j/ns/s nullfs rw 0 0
/home/js/mail /home/j/mail/s nullfs rw 0 0
/home/js/www /home/j/www/s nullfs rw 0 0Partitions marked with a 0 pass number are not
checked by &man.fsck.8; during boot, and partitions
marked with a 0 dump number are not backed up by
&man.dump.8;. We do not want
fsck to check
nullfs mounts or
dump to back up the read-only
nullfs mounts of the jails. This is why they are marked
with 0 0 in the last two columns of
each fstab entry above.Configure the jails in
/etc/rc.conf:jail_enable="YES"
jail_set_hostname_allow="NO"
jail_list="ns mail www"
jail_ns_hostname="ns.example.org"
jail_ns_ip="192.168.3.17"
jail_ns_rootdir="/usr/home/j/ns"
jail_ns_devfs_enable="YES"
jail_mail_hostname="mail.example.org"
jail_mail_ip="192.168.3.18"
jail_mail_rootdir="/usr/home/j/mail"
jail_mail_devfs_enable="YES"
jail_www_hostname="www.example.org"
jail_www_ip="62.123.43.14"
jail_www_rootdir="/usr/home/j/www"
jail_www_devfs_enable="YES"The reason why the
jail_name_rootdir
variable is set to /usr/home instead of
/home is that the
physical path of the /home directory on a
default &os; installation is /usr/home. The
jail_name_rootdir
variable must not be set to a path
which includes a symbolic link, otherwise the jails will
refuse to start. Use the &man.realpath.1; utility to
determine a value which should be set to this variable.
Please see the &os;-SA-07:01.jail Security Advisory for
more information.Create the required mount points for the read-only
file system of each jail:&prompt.root; mkdir /home/j/ns /home/j/mail /home/j/wwwInstall the read-write template into each jail. Note
the use of sysutils/cpdup, which helps to
ensure that a correct copy is done of each
directory:&prompt.root; mkdir /home/js
&prompt.root; cpdup /home/j/skel /home/js/ns
&prompt.root; cpdup /home/j/skel /home/js/mail
&prompt.root; cpdup /home/j/skel /home/js/wwwIn this phase, the jails are built and prepared to
run. First, mount the required file systems for each
jail, and then start them using the
/etc/rc.d/jail script:&prompt.root; mount -a
&prompt.root; /etc/rc.d/jail startThe jails should be running now. To check if they have
started correctly, use the &man.jls.8; command. Its output
should be similar to the following:&prompt.root; jls
JID IP Address Hostname Path
3 192.168.3.17 ns.example.org /home/j/ns
2 192.168.3.18 mail.example.org /home/j/mail
1 62.123.43.14 www.example.org /home/j/wwwAt this point, it should be possible to log onto each
jail, add new users or configure daemons. The
JID column indicates the jail
identification number of each running jail. Use the
following command in order to perform administrative tasks in
the jail whose JID is 3:&prompt.root; jexec 3 tcshUpgradingIn time, there will be a need to upgrade the system to a
newer version of &os;, either because of a security issue, or
because new features have been implemented which are useful
for the existing jails. The design of this setup provides an
easy way to upgrade existing jails. Additionally, it
minimizes their downtime, as the jails will be brought down
only in the very last minute. Also, it provides a way to roll
back to the older versions should any problems occur.The first step is to upgrade the host system in the
usual manner. Then create a new temporary read-only
template in /home/j/mroot2.&prompt.root; mkdir /home/j/mroot2
&prompt.root; cd /usr/src
&prompt.root; make installworld DESTDIR=/home/j/mroot2
&prompt.root; cd /home/j/mroot2
&prompt.root; cpdup /usr/src usr/src
&prompt.root; mkdir sThe installworld run creates
a few unnecessary directories, which should be
removed:&prompt.root; chflags -R 0 var
&prompt.root; rm -R etc var root usr/local tmpRecreate the read-write symlinks for the master file
system:&prompt.root; ln -s s/etc etc
&prompt.root; ln -s s/root root
&prompt.root; ln -s s/home home
&prompt.root; ln -s ../s/usr-local usr/local
&prompt.root; ln -s ../s/usr-X11R6 usr/X11R6
&prompt.root; ln -s s/tmp tmp
&prompt.root; ln -s s/var varThe right time to stop the jails is now:&prompt.root; /etc/rc.d/jail stopUnmount the original file systems:&prompt.root; umount /home/j/ns/s
&prompt.root; umount /home/j/ns
&prompt.root; umount /home/j/mail/s
&prompt.root; umount /home/j/mail
&prompt.root; umount /home/j/www/s
&prompt.root; umount /home/j/wwwThe read-write systems are attached to the read-only
system (/s) and
must be unmounted first.Move the old read-only file system and replace it with
the new one. This will serve as a backup and archive of the
old read-only file system should something go wrong. The
naming convention used here corresponds to when a new
read-only file system has been created. Move the original
&os; Ports Collection over to the new file system to save
some space and inodes:&prompt.root; cd /home/j
&prompt.root; mv mroot mroot.20060601
&prompt.root; mv mroot2 mroot
&prompt.root; mv mroot.20060601/usr/ports mroot/usrAt this point the new read-only template is ready, so
the only remaining task is to remount the file systems and
start the jails:&prompt.root; mount -a
&prompt.root; /etc/rc.d/jail startUse &man.jls.8; to check if the jails started correctly.
Do not forget to run mergemaster in each jail. The
configuration files will need to be updated as well as the
rc.d scripts.