path: root/en_US.ISO8859-1/articles/releng/article.xml

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V4.5-Based Extension//EN"
	"../../../share/xml/freebsd45.dtd" [
<!ENTITY art.re.pkgs '<ulink url="&url.articles.releng-packages;/article.html">The Release Engineering of Third Party Packages</ulink>'>
]>

<article lang='en'>
  <title>&os; Release Engineering</title>
  <articleinfo>

    <!-- This paper was presented at BSDCon Europe in Brighton, UK on
         November 11, 2001 -->
    <confgroup>
      <confdates>November 2001</confdates>
      <conftitle>BSDCon Europe</conftitle>
    </confgroup>

    <authorgroup>
      <author>
        <firstname>Murray</firstname>
        <surname>Stokely</surname>
        <authorblurb>
          <para>I've been involved in the development of &os; based products
          since 1997 at Walnut Creek CDROM, BSDi, and now Wind River Systems.
          &os;&nbsp;4.4 was the first official release of &os; that I played
          a significant part in.</para>
        </authorblurb>
        <affiliation>
          <address><email>murray@FreeBSD.org</email>
            <otheraddr><ulink url="http://www.FreeBSD.org/~murray/"></ulink></otheraddr>
          </address>
        </affiliation>
      </author>
    </authorgroup>

    <legalnotice id="trademarks" role="trademarks">
      &tm-attrib.freebsd;
      &tm-attrib.cvsup;
      &tm-attrib.intel;
      &tm-attrib.general;
    </legalnotice>

    <pubdate>$FreeBSD$</pubdate>

    <releaseinfo>$FreeBSD$</releaseinfo>

    <abstract>
      <para>This paper describes the approach used by the &os;
        release engineering team to make production quality releases
        of the &os; Operating System.  It details the methodology
        used for the official &os; releases and describes the tools
        available for those interested in producing customized &os;
        releases for corporate rollouts or commercial
        productization.</para>
    </abstract>

  </articleinfo>

<!-- Introduction -->
<sect1 id="introduction">
  <title>Introduction</title>

  <para>The development of &os; is a very open process.  &os; is
    comprised of contributions from thousands of people around the
    world.  The &os; Project provides anonymous
    <acronym>CVS</acronym>[1] access to the general public so that
    others can have access to log messages, diffs (patches) between
    development branches, and other productivity enhancements that
    formal source code management provides.  This has been a huge help
    in attracting more talented developers to &os;.  However, I
    think everyone would agree that chaos would soon manifest if write
    access was opened up to everyone on the Internet.  Therefore only
    a <quote>select</quote> group of nearly 300 people are given write
    access to the <acronym>CVS</acronym> repository.  These
    <emphasis>committers[5]</emphasis> are responsible for the bulk of
    &os; development.  An elected <emphasis>core-team[6]</emphasis>
    of very senior developers provides some level of direction over
    the project.</para>

  <para>The rapid pace of <systemitem
    class="osname">&os;</systemitem> development leaves little time
    for polishing the development system into a production quality
    release.  To solve this dilemma, development continues on two
    parallel tracks.  The main development branch is the
    <emphasis>HEAD</emphasis> or <emphasis>trunk</emphasis> of our CVS
    tree, known as <quote>&os;-CURRENT</quote> or
    <quote>-CURRENT</quote> for short.</para>

  <para>A more stable branch is maintained, known as
    <quote>&os;-STABLE</quote> or <quote>-STABLE</quote> for short.
    Both branches live in a master CVS repository in California and
    are replicated via <application
    class="software">CVSup</application>[2] to mirrors all over the
    world.  &os;-CURRENT[7] is the <quote>bleeding-edge</quote> of
    &os; development where all new changes first enter the system.
    &os;-STABLE is the development branch from which major releases
    are made.  Changes go into this branch at a different pace, and
    with the general assumption that they have first gone into
    &os;-CURRENT and have been thoroughly tested by our user
    community.</para>

  <para>In the interim period between releases, monthly snapshots are
    built automatically by the &os; Project build machines and made
    available for download from <systemitem
    class="resource">ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/</systemitem>.
    The widespread availability of binary release snapshots, and the
    tendency of our user community to keep up with -STABLE development
    with CVSup and <quote><command>make</command>
    <maketarget>world</maketarget></quote>[7] helps to keep
    &os;-STABLE in a very reliable condition even before the
    quality assurance activities ramp up pending a major
    release.</para>

  <para>Bug reports and feature requests are continuously submitted by
    users throughout the release cycle.  Problems reports are entered into our
    <application class="software">GNATS</application>[8] database
    through email, the &man.send-pr.1; application, or via the web
    interface provided at <ulink
    url="http://www.FreeBSD.org/send-pr.html"></ulink>.</para>

  <para>To service our most conservative users, individual release
    branches were introduced with &os;&nbsp;4.3.
    These release branches are created shortly before a final release
    is made.  After the release goes out, only the most critical
    security fixes and additions are merged onto the release branch.
    In addition to source updates via CVS, binary patchkits are
    available to keep systems on the
    <emphasis>RELENG_<replaceable>X</replaceable>_<replaceable>Y</replaceable></emphasis>
    branches updated.</para>

  <sect2>
    <title>What this article describes</title>

    <para>The following sections of this article describe:</para>

    <variablelist>
      <varlistentry>
	<term><xref linkend="release-proc"/></term>

	<listitem>
	  <para>The different phases of the release engineering process
	    leading up to the actual system build.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term><xref linkend="release-build"/></term>

	<listitem>
	  <para>The actual build process.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term><xref linkend="extensibility"/></term>

	<listitem>
	  <para>How the base release may be extended by third parties.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term><xref linkend="lessons-learned"/></term>

	<listitem>
	  <para>Some of the lessons learned through the release of &os;&nbsp;4.4.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term><xref linkend="future"/></term>

	<listitem>
	  <para>Future directions of development.</para>
	</listitem>
      </varlistentry>
    </variablelist>
  </sect2>
</sect1>

<!-- Release Process -->
<sect1 id="release-proc">
  <title>Release Process</title>

  <para>New releases of &os; are released from the -STABLE branch
    at approximately four month intervals.  The &os; release
    process begins to ramp up 45 days before the anticipated release
    date when the release engineer sends an email to the development
    mailing lists to remind developers that they only have 15 days to
    integrate new changes before the code freeze.  During this time,
    many developers perform what have become known as <quote>MFC
    sweeps</quote>.  <acronym>MFC</acronym> stands for <quote>Merge
    From CURRENT</quote> and it describes the process of merging a
    tested change from our -CURRENT development branch to our -STABLE
    branch.</para>

  <sect2>
    <title>Code Review</title>

    <para>Thirty days before the anticipated release, the source
      repository enters a <quote>code slush</quote>.  During this
      time, all commits to the -STABLE branch must be approved by the
      &a.re;.  The kinds of changes that are allowed during this 15 day
      period include:</para>

    <itemizedlist>
      <listitem>
        <para>Bug fixes.</para>
      </listitem>

      <listitem>
        <para>Documentation updates.</para>
      </listitem>

      <listitem>
        <para>Security-related fixes of any kind.</para>
      </listitem>

      <listitem>
        <para>Minor changes to device drivers, such as adding new Device
        IDs.</para>
      </listitem>

      <listitem>
        <para>Any additional change that the release engineering team feels
        is justified, given the potential risk.</para>
      </listitem>
    </itemizedlist>

    <para>After the first 15 days of the code slush, a
      <emphasis>release candidate</emphasis> is released for
      widespread testing and the code enters a <quote>code
      freeze</quote> where it becomes much harder to justify new
      changes to the system unless a serious bug-fix or security issue
      is involved.  During the code freeze, at least one release
      candidate is released per week, until the final release is
      ready.  During the days leading to the final release, the
      release engineering team is in constant communication with the
      security-officer team, the documentation maintainers, and the
      port maintainers, to ensure that all of the
      different components required for a successful release are
      available.</para>
  </sect2>

  <sect2>
    <title>Final Release Checklist</title>

    <para>When several release candidates have been made available for
      widespread testing and all major issues have been resolved, the
      final release <quote>polishing</quote> can begin.</para>

    <sect3 id="rel-branch">
      <title>Creating the Release Branch</title>

      <para>As described in the introduction, the
        <literal>RELENG_<replaceable>X</replaceable>_<replaceable>Y</replaceable></literal>
        release branch is a relatively new addition to our release
        engineering
        methodology.  The first step in creating this branch is to
        ensure that you are working with the newest version of the
        <literal>RELENG_<replaceable>X</replaceable></literal> sources
        that you want to branch <emphasis>from</emphasis>.</para>

      <screen>/usr/src&prompt.root; <userinput>cvs update -rRELENG_4 -P -d</userinput></screen>

      <para>The next step is to create a branch point
        <emphasis>tag</emphasis>, so that diffs against the start of
        the branch are easier with CVS:</para>

      <screen>/usr/src&prompt.root; <userinput>cvs rtag -rRELENG_4 RELENG_4_8_BP src</userinput></screen>

      <para>And then a new branch tag is created with:</para>

      <screen>/usr/src&prompt.root; <userinput>cvs rtag -b -rRELENG_4_8_BP RELENG_4_8 src</userinput></screen>

      <note>
        <para><emphasis>The
          <literal>RELENG_<replaceable>*</replaceable></literal> tags
          are restricted for use by the CVS-meisters and release
          engineers.</emphasis></para>
      </note>

      <sidebar>
        <para>A <quote><emphasis>tag</emphasis></quote> is CVS
        vernacular for a label that identifies the source at a specific point
        in time.  By tagging the tree, we ensure that future release builders
        will always be able to use the same source we used to create the
        official &os; Project releases.</para>
      </sidebar>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-head" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os; Development Branch</phrase>
        </textobject>
      </mediaobject>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-releng3" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os;&nbsp;3.x STABLE Branch</phrase>
        </textobject>
      </mediaobject>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-releng4" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os;&nbsp;4.x STABLE Branch</phrase>
        </textobject>
      </mediaobject>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-releng5" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os;&nbsp;5.x STABLE Branch</phrase>
        </textobject>
      </mediaobject>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-releng6" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os;&nbsp;6.x STABLE Branch</phrase>
        </textobject>
      </mediaobject>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-releng7" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os;&nbsp;7.x STABLE Branch</phrase>
        </textobject>
      </mediaobject>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-releng8" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os;&nbsp;8.x STABLE Branch</phrase>
        </textobject>
      </mediaobject>

      <mediaobject>
        <imageobject>
          <imagedata fileref="branches-releng9" align="center"/>
        </imageobject>

        <textobject>
          <phrase>&os;&nbsp;9.x STABLE Branch</phrase>
        </textobject>
      </mediaobject>
    </sect3>

    <sect3 id="versionbump">
      <title>Bumping up the Version Number</title>

      <para>Before the final release can be tagged, built, and
        released, the following files need to be modified to reflect
        the correct version of &os;:</para>

      <itemizedlist>
        <listitem>
          <para><filename>doc/en_US.ISO8859-1/books/handbook/mirrors/chapter.xml
          </filename></para>
        </listitem>

        <listitem>
          <para><filename>doc/en_US.ISO8859-1/books/porters-handbook/book.xml
          </filename></para>
        </listitem>

        <listitem>
          <para><filename>doc/share/xml/freebsd.ent</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/Makefile.inc1</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/UPDATING</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/gnu/usr.bin/groff/tmac/mdoc.local</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/release/Makefile</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/release/doc/en_US.ISO8859-1/share/xml/release.dsl</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/release/doc/share/examples/Makefile.relnotesng</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/release/doc/share/xml/release.ent</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/share/examples/cvsup/standard-supfile</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/sys/conf/newvers.sh</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/sys/sys/param.h</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/usr.sbin/pkg_install/add/main.c</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/docs/man.xml</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/cgi/ports.cgi</filename></para>
        </listitem>

        <listitem>
          <para><filename>ports/Tools/scripts/release/config</filename></para>
        </listitem>

      </itemizedlist>

      <para>The release notes and errata files also need to be adjusted for the
      new release (on the release branch) and truncated appropriately
      (on the stable/current branch):</para>

      <itemizedlist>
        <listitem>
          <para><filename>src/release/doc/en_US.ISO8859-1/relnotes/common/new.xml
          </filename></para>
        </listitem>

        <listitem>
          <para><filename>src/release/doc/en_US.ISO8859-1/errata/article.xml
          </filename></para>
        </listitem>
      </itemizedlist>

      <para><application>Sysinstall</application> should be updated to note
        the number of available ports and the amount of disk space required
	for the Ports Collection[4].  This information is currently kept in
	<filename>src/usr.sbin/sysinstall/dist.c</filename>.</para>

      <para>After the release has been built, a number of file should
        be updated to announce the release to the world.</para>

      <itemizedlist>
        <listitem>
          <para><filename>doc/share/images/articles/releng/branches-releng<replaceable>X</replaceable>.pic</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/share/xml/advisories.xml</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/share/xml/includes.release.xml</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/share/xml/includes.release.xsl</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/releases/*</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/releng/index.xml</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/news/news.xml</filename></para>
        </listitem>

        <listitem>
          <para><filename>www/en/search/web.atoz</filename></para>
        </listitem>

        <listitem>
          <para><filename>src/share/misc/bsd-family-tree</filename></para>
        </listitem>

      </itemizedlist>

    </sect3>

    <sect3 id="versionbump-major">
      <title>Preparing a new major release branch
        (RELENG_<replaceable>X</replaceable>)</title>

      <para>When a new major release branch, such as
        <literal>RELENG_6</literal> is branched from HEAD, some
	additional files must be updated before releases can be made
	from this new branch.</para>

      <itemizedlist>
	<listitem>
          <para><filename>src/share/examples/cvsup/stable-supfile</filename>
- must be updated to point to the new -STABLE branch, when
applicable.</para>
        </listitem>
      </itemizedlist>

    </sect3>

    <sect3>
      <title>Creating Release Tags</title>

      <para>When the final release is ready, the following command
        will create the <literal>RELENG_4_8_0_RELEASE</literal>
        tag.</para>

      <screen>/usr/src&prompt.root; <userinput>cvs rtag -rRELENG_4_8 RELENG_4_8_0_RELEASE src</userinput></screen>

      <para>The Documentation and Ports managers are responsible for
        tagging the respective trees with the <literal>RELEASE_4_8_0</literal>
        tag.</para>

      <para>Occasionally, a last minute fix may be required
        <emphasis>after</emphasis> the final tags have been created.
        In practice this is not a problem, since <acronym>CVS</acronym>
        allows tags to be manipulated with <command>cvs
        tag -d <replaceable>tagname filename</replaceable></command>.
        It is very important that any last minute changes be tagged
        appropriately as part of the release.  &os; releases must
        always be reproducible.  Local hacks in the release
        engineer's environment are not acceptable.</para>
    </sect3>
  </sect2>
</sect1>

<!-- Release Building -->
<sect1 id="release-build">
  <title>Release Building</title>

  <para>&os; <quote>releases</quote> can be built by anyone with a
    fast machine and access to a source repository. (That should be
    everyone, since we offer anonymous CVS! See The Handbook for
    details.)  The <emphasis>only</emphasis> special requirement is
    that the &man.md.4; device must be available. If the
    device is not loaded into your kernel, then the kernel module
    should be automatically loaded when &man.mdconfig.8; is executed
    during the boot media creation phase.  All of the tools necessary
    to build a release are available from the CVS repository in
    <filename>src/release</filename>.  These tools aim to provide a
    consistent way to build &os; releases.  A complete release can
    actually be built with only a single command, including the
    creation of <acronym>ISO</acronym> images suitable for burning to
    CDROM, installation floppies, and an FTP install directory.  This
    command is aptly named <command>make
    release</command>.</para>

  <sect2>
    <title><command>make release</command></title>

    <para>To successfully build a release, you must first populate
      <filename>/usr/obj</filename> by running <command>make
      world</command> or simply
      <command>make
      buildworld</command>. The release
      target requires several variables be set properly to build a
      release:</para>

    <itemizedlist>
      <listitem>
        <para><makevar>CHROOTDIR</makevar> - The directory to be used as the
        chroot environment for the entire release build.</para>
      </listitem>

      <listitem>
        <para><makevar>BUILDNAME</makevar> - The name of the release to be
        built.</para>
      </listitem>

      <listitem>
        <para><makevar>CVSROOT</makevar> - The location of a CVS Repository.
        </para>
      </listitem>

      <listitem>
        <para><makevar>RELEASETAG</makevar> - The CVS tag corresponding to the
        release you would like to build.</para>
      </listitem>
    </itemizedlist>

     <para>If you do not already have access to a local CVS
       repository, then you may mirror one with <ulink
       url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/synching.html#CVSUP">CVSup</ulink>.
       The supplied supfile,
       <filename>/usr/share/examples/cvsup/cvs-supfile</filename>, is
       a useful starting point for mirroring the CVS
       repository.</para>

     <para>If <makevar>RELEASETAG</makevar> is omitted, then the
       release will be built from the <literal>HEAD</literal> (aka -CURRENT) branch.
       Releases built from this branch are normally referred to as
       <quote>-CURRENT snapshots</quote>.</para>

     <para>There are many other variables available to customize the
       release build. Most of these variables are documented at the
       top of <filename>src/release/Makefile</filename>. The exact
       command used to build the official &os;&nbsp;4.7 (x86) release
       was:</para>

     <screen><command>make <literal>release CHROOTDIR=/local3/release \
       BUILDNAME=4.7-RELEASE \
       CVSROOT=/host/cvs/usr/home/ncvs \
       RELEASETAG=RELENG_4_7_0_RELEASE</literal>
       </command>
     </screen>

     <para>The release <filename>Makefile</filename> can be broken down into several distinct
       steps.</para>

    <itemizedlist>
      <listitem>
        <para>Creation of a sanitized system environment in a separate
	directory hierarchy with <quote><command>make
	<literal>installworld</literal></command></quote>.
        </para>
      </listitem>

      <listitem>
        <para>Checkout from CVS of a clean version of the system source,
        documentation, and ports into the release build hierarchy.</para>
      </listitem>

      <listitem>
        <para>Population of <filename>/etc</filename> and
        <filename>/dev</filename> in the chrooted
        environment.</para>
      </listitem>

      <listitem>
        <para>chroot into the release build hierarchy, to make it harder for
        the outside environment to taint this build.</para>
      </listitem>

      <listitem>
        <para><command>make world</command>
        in the chrooted environment.</para>
      </listitem>

      <listitem>
        <para>Build of Kerberos-related binaries.</para>
      </listitem>

      <listitem>
        <para>Build <filename>GENERIC</filename> kernel.</para>
      </listitem>

      <listitem>
        <para>Creation of a staging directory tree where the binary
        distributions will be built and packaged.</para>
      </listitem>

      <listitem>
        <para>Build and installation of the documentation toolchain needed to
        convert the documentation source (SGML) into HTML and text documents
        that will accompany the release.</para>
      </listitem>

      <listitem>
        <para>Build and installation of the actual documentation
        (user manuals, tutorials, release notes, hardware compatibility lists,
        and so on.)</para>
      </listitem>

      <listitem>
        <para>Build of the <quote>crunched</quote> binaries used for
        installation floppies.</para>
      </listitem>

      <listitem>
        <para>Package up distribution tarballs of the binaries and sources.
        </para>
      </listitem>

      <listitem>
        <para>Create the boot media and a <quote>fixit</quote> floppy.</para>
      </listitem>

      <listitem>
        <para>Create FTP installation hierarchy.</para>
      </listitem>

      <listitem>
        <para><emphasis>(optionally)</emphasis> Create ISO images for
        CDROM/DVD media.</para>
      </listitem>
    </itemizedlist>

    <para>For more information about the release build infrastructure,
      please see &man.release.7;.</para>

    <note><para>It is important to remove any site-specific settings
      from <filename>/etc/make.conf</filename>.  For example, it would
      be unwise to distribute binaries that were built on a system
      with <varname>CPUTYPE</varname> set to a specific
      processor.</para></note>

  </sect2>

  <sect2>
    <title>Contributed Software (<quote>ports</quote>)</title>

    <para>The <ulink url="http://www.FreeBSD.org/ports">&os; Ports
      collection</ulink> is a collection of over &os.numports;
      third-party software packages available for &os;. The &a.portmgr;
      is responsible for maintaining a consistent ports tree that can be used
      to create the binary packages that accompany official &os;
      releases.</para>

    <para>The release engineering activities for our collection of
      third-party packages is beyond the scope of this document.  A
      separate article, &art.re.pkgs;, covers this topic
      in depth.</para>

  </sect2>

  <sect2>
    <title>Release ISOs</title>

    <para>Starting with &os;&nbsp;4.4, the &os; Project decided to
      release all four ISO images that were previously sold on the
      <emphasis>BSDi/Wind River Systems/FreeBSD Mall</emphasis>
      <quote>official</quote> CDROM distributions. Each of the four
      discs must contain a <filename>README.TXT</filename> file that
      explains the contents of the disc, a
      <filename>CDROM.INF</filename> file that provides meta-data for
      the disc so that &man.sysinstall.8; can validate and use the
      contents, and a <filename>filename.txt</filename> file that
      provides a manifest for the disc. This
      <emphasis>manifest</emphasis> can be created with a simple
      command:</para>

    <screen>/stage/cdrom&prompt.root; <userinput>find . -type f | sed -e 's/^\.\///' | sort > filename.txt</userinput></screen>

    <para>The specific requirements of each CD are outlined below.</para>

    <sect3>
      <title>Disc 1</title>

      <para>The first disc is almost completely created by
        <command>make
        release</command>. The only changes
        that should be made to the <filename>disc1</filename> directory are the addition of
        a <filename>tools</filename> directory, and as many popular
        third party software packages as will fit on the disc. The
        <filename>tools</filename> directory contains software that allow users to create
        installation floppies from other operating systems. This disc
        should be made bootable so that users of modern PCs do not
        need to create installation floppy disks.</para>

      <para>If a custom kernel of &os; is to be included, then
        &man.sysinstall.8; and &man.release.7; must be updated to
        include installation instructions. The relevant code is contained
        in <filename>src/release</filename> and <filename>src/usr.sbin/sysinstall</filename>.
        Specifically, the file <filename>src/release/Makefile</filename>, and
        <filename>dist.c</filename>, <filename>dist.h</filename>,
        <filename>menus.c</filename>, <filename>install.c</filename>, and
        <filename>Makefile</filename> will need to be updated under
        <filename>src/usr.sbin/sysinstall</filename>.  Optionally, you may choose
        to update <filename>sysinstall.8</filename>.</para>

    </sect3>

    <sect3>
      <title>Disc 2</title>

      <para>The second disc is also largely created by <command>make
        release</command>. This disc contains a <quote>live
        filesystem</quote> that can be used from &man.sysinstall.8; to
        troubleshoot a &os; installation. This disc should be
        bootable and should also contain a compressed copy of the CVS
        repository in the <filename>CVSROOT</filename> directory and
        commercial software demos in the <filename>commerce</filename>
        directory.</para>
    </sect3>

    <sect3>
      <title>Discs 3 and 4</title>

      <para>The remaining two discs contain additional software
        packages for &os;. The packages should be clustered so that
        a package and all of its <emphasis>dependencies</emphasis> are
        included on the same disc.  More information about the
        creation of these discs is provided in the &art.re.pkgs;
        article.</para>
    </sect3>

    <sect3>
      <title>Multi-volume support</title>

      <para><application>Sysinstall</application> supports multiple
        volume package installations.  This requires that each disc
        have an <filename>INDEX</filename> file containing all of the
        packages on all volumes of a set, along with an extra field
        that indicates which volume that particular package is on.
        Each volume in the set must also have the
        <literal>CD_VOLUME</literal> variable set in the
        <filename>cdrom.inf</filename> file so that sysinstall can
        tell which volume is which.  When a user attempts to install a
        package that is not on the current disc, sysinstall will
        prompt the user to insert the appropriate one.</para>
    </sect3>
  </sect2>
</sect1>

<!-- Distribution -->
<sect1 id="distribution">
  <title>Distribution</title>

  <sect2 id="dist-ftp">
    <title>FTP Sites</title>

  <para>When the release has been thoroughly tested and packaged for
    distribution, the master FTP site must be updated.  The official
    &os; public FTP sites are all mirrors of a master server that
    is open only to other FTP sites.  This site is known as
    <hostid>ftp-master</hostid>.  When the release is ready, the
    following files must be modified on <hostid>ftp-master</hostid>:</para>

  <variablelist>
    <varlistentry>
      <term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/</filename></term>
      <listitem>
        <para>The installable FTP directory as output from <command>make
          release</command>.</para>
      </listitem>
    </varlistentry>

    <varlistentry>
      <term><filename>/pub/FreeBSD/ports/<replaceable>arch</replaceable>/packages-<replaceable>X.Y</replaceable>-release/</filename></term>
      <listitem><para>The complete package build for this
        release.</para></listitem>
    </varlistentry>

    <varlistentry>
      <term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/tools</filename></term>
      <listitem><para>A symlink to
      <filename>../../../tools</filename>.</para></listitem>
    </varlistentry>

    <varlistentry>
      <term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/packages</filename></term>
      <listitem><para>A symlink to
      <filename>../../../ports/<replaceable>arch</replaceable>/packages-<replaceable>X.Y</replaceable>-release</filename>.</para></listitem>
    </varlistentry>

    <varlistentry>
      <term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/ISO-IMAGES/<replaceable>X.Y</replaceable>/<replaceable>X.Y</replaceable>-RELEASE-<replaceable>arch</replaceable>-*.iso</filename></term>
      <listitem><para>The ISO images.  The <quote>*</quote> is
	<filename>disc1</filename>, <filename>disc2</filename>, etc.
	Only if there is a <filename>disc1</filename> and there is an
	alternative first installation CD (for example a
	stripped-down install with no windowing system) there may
	be a <filename>mini</filename> as well.</para>
      </listitem>
    </varlistentry>
  </variablelist>

  <para>For more information about the distribution mirror
    architecture of the &os; FTP sites, please see the <ulink
    url="&url.articles.hubs;/">Mirroring &os;</ulink> article.</para>

  <para>It may take many hours to two days after updating
    <hostid>ftp-master</hostid> before a majority of the Tier-1 FTP
    sites have the new software depending on whether or not a package
    set got loaded at the same time.  It is imperative that the release
    engineers coordinate with the &a.mirror-announce; before announcing the general
    availability of new software on the FTP sites.  Ideally
    the release package set should be loaded at least four
    days prior to release day.  The release bits should be
    loaded between 24 and 48 hours before the planned release
    time with <quote>other</quote> file permissions turned off.
    This will allow the mirror sites to download it but the
    general public will not be able to download it from the mirror
    sites.  Mail should be sent to &a.mirror-announce; at the time
    the release bits get posted saying the release has been staged
    and giving the time that the mirror sites should begin allowing
    access.  Be sure to include a time zone with the
    time, for example make it relative to GMT.</para>
  </sect2>

  <sect2 id="dist-cdrom">
    <title>CD-ROM Replication</title>

    <para>Coming soon: Tips for sending &os; ISOs to a replicator
      and quality assurance measures to be taken.</para>
  </sect2>

</sect1>

<!-- Extensibility -->
<sect1 id="extensibility">
  <title>Extensibility</title>

  <para>Although &os; forms a complete operating system, there is
    nothing that forces you to use the system exactly as we have
    packaged it up for distribution. We have tried to design the
    system to be as extensible as possible so that it can serve as a
    platform that other commercial products can be built on top
    of. The only <quote>rule</quote> we have about this is that if you
    are going to distribute &os; with non-trivial changes, we
    encourage you to document your enhancements! The &os; community
    can only help support users of the software we provide. We
    certainly encourage innovation in the form of advanced
    installation and administration tools, for example, but we cannot
    be expected to answer questions about it.</para>

  <sect2>
    <title>Creating Customized Boot floppies</title>

    <para>Many sites have complex requirements that may require
      additional kernel modules or userland tools be added to the
      installation floppies.  The <quote>quick and dirty</quote> way
      to accomplish this would be to modify the staging directory of
      an existing <command>make release</command> build hierarchy:</para>

    <itemizedlist>
      <listitem>
        <para>Apply patches or add additional files inside the chroot
          release build directory.</para>
      </listitem>

      <listitem>
        <para><command>rm
        ${CHROOTDIR}/usr/obj/usr/src/release/release.[59]</command></para>
      </listitem>

      <listitem>
        <para>rebuild &man.sysinstall.8;, the kernel, or whatever
          parts of the system your change affected.</para>
      </listitem>

      <listitem>
        <para><command>chroot ${CHROOTDIR} ./mk floppies
        </command></para>
      </listitem>

    </itemizedlist>

    <para>New release floppies will be located in
      <filename>${CHROOTDIR}/R/stage/floppies</filename>.</para>

    <para>Alternatively, the
      <filename>boot.flp</filename> make
      target can be called, or the filesystem
      creating script,
      <filename>src/release/scripts/doFS.sh</filename>, may be invoked
      directly.</para>

    <para>Local patches may also be supplied to the release build by
      defining the <makevar>LOCAL_PATCH</makevar> variable in <command>make
      release</command>.
    </para>
  </sect2>

  <sect2>
    <title>Scripting <command>sysinstall</command></title>

    <para>The &os; system installation and configuration tool,
      &man.sysinstall.8;, can be scripted to provide automated installs
      for large sites. This functionality can be used in conjunction
      with &intel; PXE[12] to bootstrap systems from the network, or
      via custom boot floppies with a sysinstall script.  An example
      sysinstall script is available in the CVS tree as
      <filename>src/usr.sbin/sysinstall/install.cfg</filename>.</para>
  </sect2>
</sect1>

<!-- Lessons Learned -->
<sect1 id="lessons-learned">
  <title>Lessons Learned from &os;&nbsp;4.4</title>

  <para>The release engineering process for 4.4 formally began on
    August 1st, 2001. After that date all commits to the
    <literal>RELENG_4</literal> branch of &os; had to be explicitly
    approved by the &a.re;.  The first
    release candidate for the x86 architecture was released on August
    16, followed by 4 more release candidates leading up to the final
    release on September 18th.  The security officer was very involved
    in the last week of the process as several security issues were
    found in the earlier release candidates. A total of over
    <emphasis>500</emphasis> emails were sent to the &a.re; in
    little over a month.</para>

  <para>Our user community has made it very clear that the security
    and stability of a &os; release should not be sacrificed for
    any self-imposed deadlines or target release dates.  The &os;
    Project has grown tremendously over its lifetime and the need for
    standardized release engineering procedures has never been more
    apparent. This will become even more important as &os; is
    ported to new platforms.</para>
</sect1>

<!-- Future Directions -->
<sect1 id="future">
  <title>Future Directions</title>

  <para>It is imperative for our release engineering activities to
    scale with our growing userbase. Along these lines we are working
    very hard to document the procedures involved in producing &os;
    releases.</para>

  <itemizedlist>
    <listitem>
      <para><emphasis>Parallelism</emphasis> - Certain portions of the
        release build are actually <quote>embarrassingly
        parallel</quote>.  Most of the tasks are very I/O&nbsp;intensive,
        so having multiple high-speed disk drives is actually more important than
        using multiple processors in speeding up the <command>make
        release</command> process.  If multiple disks are used for
        different hierarchies in the &man.chroot.2;
        environment, then the CVS checkout of the <filename>ports</filename> and <filename>doc</filename> trees
        can be happening simultaneously as the <command>make
        world</command> on another disk.  Using a
        <acronym>RAID</acronym> solution (hardware or software) can
        significantly decrease the overall build time.</para>
    </listitem>

    <listitem>
      <para><emphasis>Cross-building releases</emphasis> - Building
        IA-64 or Alpha release on x86 hardware? <command>make
        TARGET=ia64 release</command>.
      </para>
    </listitem>

    <listitem>
      <para><emphasis>Regression Testing</emphasis> - We need better
        automated correctness testing for &os;.</para>
    </listitem>

    <listitem>
      <para><emphasis>Installation Tools</emphasis> - Our installation
        program has long since outlived its intended life span.
        Several projects are under development to provide a more
        advanced installation mechanism.  The libh project was one
        such project that aimed to provide an intelligent new package
        framework and GUI installation program.</para>
    </listitem>
  </itemizedlist>

</sect1>

<!-- Acknowledgements -->
<sect1 id="ackno">
<title>Acknowledgements</title>

  <para>I would like to thank Jordan Hubbard for giving me the
    opportunity to take on some of the release engineering
    responsibilities for &os;&nbsp;4.4 and also for all of his work
    throughout the years making &os; what it is today.  Of course
    the release would not have been possible without all of the
    release-related work done by &a.asami;, &a.steve;, &a.bmah;, &a.nik;,
    &a.obrien;, &a.kris;, &a.jhb; and the rest of the &os; development
    community.  I would also like to thank &a.rgrimes;, &a.phk;, and others
    who worked on the release engineering tools in the very early days
    of &os;.  This article was influenced by release engineering
    documents from the CSRG[13], the NetBSD Project[10], and John
    Baldwin's proposed release engineering process notes[11].</para>
</sect1>

<!-- Reference / Biblio Section -->
<sect1 id="biblio">
  <title>References</title>
  <para>[1] CVS - Concurrent Versions System
  <ulink url="http://www.cvshome.org"></ulink></para>

  <para>[2] CVSup - The CVS-Optimized General Purpose Network File Distribution
  System <ulink url="http://www.polstra.com/projects/freeware/CVSup"></ulink>
  </para>

  <para>[3] <ulink url="http://pointyhat.FreeBSD.org"></ulink></para>

  <para>[4] &os; Ports Collection
  <ulink url="http://www.FreeBSD.org/ports"></ulink></para>

  <para>[5] &os; Committers <ulink
  url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/contributors/staff-committers.html"></ulink>
  </para>

  <para>[6] &os; Core Team
  <ulink url="&url.base;/administration.html#t-core"></ulink></para>

  <para>[7] &os; Handbook
  <ulink url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook"></ulink>
  </para>

  <para>[8] GNATS: The GNU Bug Tracking System
  <ulink url="http://www.gnu.org/software/gnats"></ulink>
  </para>

  <para>[9] &os; PR Statistics
  <ulink url="http://www.FreeBSD.org/prstats/index.html"></ulink></para>

  <para>[10] NetBSD Developer Documentation: Release Engineering
  <ulink url="http://www.NetBSD.org/developers/releng/index.html"></ulink>
  </para>

  <para>[11] John Baldwin's &os; Release Engineering Proposal
  <ulink url="http://people.FreeBSD.org/~jhb/docs/releng.txt"></ulink>
  </para>

  <para>[12] PXE Jumpstart Guide
  <ulink
  url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/pxe/index.html"></ulink>
  </para>

  <para>[13] Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic:
  <ulink url="http://docs.FreeBSD.org/44doc/papers/releng.html">
<emphasis>The Release Engineering of 4.3BSD</emphasis></ulink>
  </para>
</sect1>
</article>