path: root/en_US.ISO8859-1/articles/releng-packages/article.xml

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V4.5-Based Extension//EN"
	"../../../share/xml/freebsd45.dtd">

<article lang='en'>
  <title>FreeBSD Release Engineering for Third Party Software
    Packages</title>
  <articleinfo>
    <authorgroup>
      <author>
        <firstname>Steve</firstname>
        <surname>Price</surname>
        <affiliation>
          <address><email>steve@FreeBSD.org</email></address>
        </affiliation>
      </author>
    </authorgroup>

    <legalnotice id="trademarks" role="trademarks">
      &tm-attrib.freebsd;
      &tm-attrib.intel;
      &tm-attrib.xfree86;
      &tm-attrib.general;
    </legalnotice>

    <pubdate>$FreeBSD$</pubdate>

    <releaseinfo>$FreeBSD$</releaseinfo>

    <abstract>
      <para>This paper describes the approach used by the FreeBSD
        ports management team to produce a high quality package set
        suitable for official FreeBSD release media.  This document is
        a work in progress, but eventually it will cover the process
        used to build a clean package set on the FreeBSD.org <quote>Ports
        Cluster</quote>, how to configure any other set of machines as a
        ports cluster, how to split up the packages for the release
        media, and how to verify that a package set is
        consistent.</para>
    </abstract>

  </articleinfo>

  <!-- Introduction

  <sect1 id="introduction">
    <title>Introduction</title>

    <para><emphasis>Coming Soon</emphasis></para>

  </sect1>

-->
  <sect1 id="portbuild">
    <title>Building packages from the Ports Collection</title>

    <para>The <ulink url="http://www.FreeBSD.org/ports">FreeBSD Ports
      collection</ulink> is a collection of over &os.numports;
      third-party software packages available for FreeBSD. The &a.portmgr;
      is responsible for maintaining a consistent ports tree that can be used
      to create the binary packages that accompany a given FreeBSD
      release.</para>

    <sect2>
      <title>The Ports Cluster</title>

      <para>In order to provide a consistent set of third-party
        packages for FreeBSD releases, every port is built in a
        separate chroot environment, starting with an empty
        <filename>/usr/local</filename> and
        <filename>/usr/X11R6</filename>. The requisite dependencies
        are installed as packages before the build proceeds. This
        enforces <emphasis>consistency</emphasis> in the package build
        process. By starting the package build in a pristine
        environment, we can assure that the package metadata (such as
        required dependencies) is accurate.  This way, we will never
        generate packages that might work on some systems and not on
        others depending on what software was previously
        installed.</para>

      <para>The <quote>Ports Cluster</quote> for the x86 architecture
        currently consists of a master node (Dual &pentium; III 733MHz)
        and 8 slave nodes (&pentium; III 800MHz) to do the actual
        package builds.  With this configuration, a complete package
        build takes over 24 hours.  These machines are co-located with
        the other FreeBSD Project equipment at Yahoo's corner of
        Exodus in Santa Clara, CA.</para>

      <para>The <quote>Ports Cluster</quote> for the Alpha
        architecture consists of 7 PWS 500A machines donated by Compaq
        and also co-located with Yahoo's facilities.</para>
    </sect2>
  </sect1>

  <sect1>
    <title>The Package Split</title>

    <para>For FreeBSD 4.4 over 4.1 gigabytes of packages were created.
      This causes a problem for CDROM distributions because we would
      like to ship as many packages as possible without making the
      user insert another disc to satisfy dependencies. The solution
      is to create <quote>clusters</quote> of like packages with
      similar dependencies and group these onto specific discs.  This
      section describes the software and methodology used to create
      those package sets for the official FreeBSD release
      discs.</para>

    <para>The scripts and other files needed to produce a package
      split can be found in the CVS tree in
      <filename>ports/Tools/scripts/release</filename>.
      Copy this directory to a machine that has enough free disk
      space to hold 2 to 3 times the size of the package set that you
      wish to split.</para>

    <para>The following scripts are present in this directory:</para>

    <variablelist>
      <varlistentry>
        <term><filename>config</filename></term>

        <listitem><para>This file contains the free space on each disc
          and whether packages, distfiles, or both are allowed on any
          given disc.  The first column is the disc name.  It must be
	  of the form <literal>disc[0-9a-z]</literal>.  Currently it is set up
	  to allow for 10 discs (4 for the release set and 6 for the toolkit).
	  There is an implied extra disc called <quote>scratch</quote> where
	  all of the remaining distfiles/packages land if they do not fit
          elsewhere.  The second column can be either a 1 or 0, where 1
          says that it is okay to place packages on this disc.  The
          third column works the same way, but it controls
          whether distfiles are placed on this disc.  The last column
          denotes the number of bytes of free space on a
          disc.</para></listitem>
        </varlistentry>

      <varlistentry>
        <term><filename>doit.sh</filename></term>

        <listitem><para>This is the workhorse.  Once you have all the
          files in place and things properly configured this script
          directs the process of splitting packages.  Beware it is
          interactive so you need to keep an eye on it as it runs.
          More details on what happens in this script will
          follow.</para></listitem>
      </varlistentry>

      <varlistentry>
        <term><filename>checkdeps.pl</filename></term>

        <listitem><para>Makes sure all packages dependencies are
	  satisfied given an <filename>INDEX</filename> file and a directory
	  of packages.</para></listitem>
      </varlistentry>

      <varlistentry>
        <term><filename>oneshot.pl</filename></term>

        <listitem><para>This is where all the magic (and I use that
          term loosely as it is mostly just a brute force approach)
          happens.  Given a list of required packages for each disc
          and a set of packages/distfiles this is the script that
          places a package or distfile on a disc along with all of its
          dependencies.</para></listitem>
      </varlistentry>

      <varlistentry>
        <term><filename>print-cdrom-packages.sh</filename></term>

        <listitem><para>This file is a copy of
          <filename>src/release/scripts/print-cdrom-packages.sh</filename>
          from the release you are working on.</para></listitem>
      </varlistentry>

      <varlistentry>
        <term><filename>scrubindex.pl</filename></term>

	<listitem><para>This script removes lines from an
	  <filename>INDEX</filename> file for packages that are not present.
	  It also removes the &xfree86; dependencies.  NOTE: you will need to
	  tweak the value of the <varname>xdep</varname> variable to make sure
	  the version number is correct.</para></listitem>
      </varlistentry>

      <varlistentry>
        <term><filename>setup.sh</filename></term>

        <listitem><para>This is a helper script that I use on the
          ports building cluster to grab a copy of the ports tree and the
          matching set of the packages/distfiles.</para></listitem>
      </varlistentry>
    </variablelist>

    <para>Here is a checklist of things you will need to check or
      configure before going any further.</para>

    <orderedlist>
      <listitem><para>Edit <filename>config</filename> to denote the
        number of discs you have, their sizes, and whether you want
        them want to contain packages, distfiles, both, or
        neither.</para></listitem>

      <listitem><para>Make sure you remove the <varname>gen</varname>
	directory if there is an old one laying around.  This directory
	contains working files that will only be valid for the current
        split.</para></listitem>

      <listitem><para>On your first pass through a split it is best to
        fake the copying of packages and distfiles.  This will save
        both time and disk space while you do a couple of trial runs to
        make sure things fit, etc.  In the
	<filename>oneshot.pl</filename> set the <varname>fake</varname>
	variable to 1 and instead of actually copying the files it will
	&man.touch.1; them.  Be sure you turn this off or set
	<varname>fake</varname> to 0 before you give the resultant discs to
	the person that will be mastering the discs otherwise they will get a
	directory full of zero-sized files.</para></listitem>

      <listitem><para>Make sure you have a recent copy of the
        <filename>print-cdrom-packages.sh</filename> and that it is
        from the correct release.</para></listitem>

      <listitem><para>Check to make sure the &xfree86; dependency in
        <filename>scrubindex.pl</filename> has the correct
        version number.  You will also need to make sure this value is
        correct in <filename>doit.sh</filename> as
        well.</para></listitem>
    </orderedlist>

    <para>Next you will need to get a copy of the ports tree, packages,
      and distfiles from a recent build on the package cluster.  See
      the <filename>setup.sh</filename> for a working example
      but essentially here is what needs to be done.</para>

    <orderedlist>

      <listitem><para>Grab a copy of <filename>ports.tar.gz</filename>
        and extract it into the <filename>ports</filename> directory alongside
        <filename>doit.sh</filename> and the
        <filename>scripts</filename> directory.</para></listitem>

      <listitem><para>Remove the packages/distfiles directories or
        symlinks.  Bento has these as symlinks and you will have mixed
        results if you do not get rid of them before
        proceeding.</para></listitem>

      <listitem><para>Create a new ports/packages directory and copy
        the package set from the package building
        cluster.</para></listitem>

      <listitem><para>Create a new ports/distfiles directory and copy
        the distfiles from the package building cluster.  NOTE: if you
        do not want any distfiles simply create the directory and leave
        it empty.  This directory must be present even if it does not
        contain anything.</para></listitem>
    </orderedlist>

    <para>Now we are finally ready for the fun task of actually
      splitting the packages.  You start the processing by running
      <command>./doit.sh</command>.  Here is what it does the first
      time you run it.</para>

    <orderedlist>
      <listitem><para>Create a list of the restricted (can not be on the
        master FTP site) ports.</para></listitem>

      <listitem><para>Asks you if you would like to remove the restricted
        ports.  Most of the time you will want to answer (y)es
        here.</para></listitem>

      <listitem><para>Create a list of the packages/distfiles that
        can not be put on the discs.</para></listitem>

      <listitem><para>Asks you if you would like to remove the
        non-cdromable packages/distfiles.  Most of the time you will
        want to answer (y)es here.</para></listitem>

      <listitem><para>Copies the <filename>INDEX</filename> from the
	<filename>ports</filename> directory to the <filename>gen</filename>
	directory.  In doing so it removes the lines for ports
        where the packages do not exist.  It also checks to make sure
        that all of the required dependency packages are
        present.</para></listitem>

      <listitem><para>Create a list of packages that are required on
        each disc.</para></listitem>

      <listitem><para>Asks you if you would like to populate the discs.
        After populating each disc it will check for missing
        dependencies, scrub the <filename>INDEX</filename> file, and create the
        <filename>CHECKSUM.MD5</filename> file.</para></listitem>

      <listitem><para>Check to make sure the required packages made it
        on each disc and gives you a summary of the sizes of each
        disc.</para></listitem>
    </orderedlist>

    <para>After going through this the first time if you are lucky
      enough that all of the required packages built and fit on each
      disc.  All you need to do is set <varname>fake</varname> to 0 in
      <filename>oneshot.pl</filename> and re-run
      <command>./doit.sh</command>.  The second and subsequent times
      around it will skip steps 1-5 above.  If you want to re-run any
      of those steps refer to <filename>doit.sh</filename> for which files
      need to be removed to not short-circuit those steps.  If you want to
      repeat all of these steps then the easiest way is to <command>rm -rf
      gen</command>.</para>

    <para>Upon successful completion the packages/distfiles will be in
      the <filename>disc*</filename> directories and the leftover will
      be in the <filename>scratch</filename> directory.</para>

    <para>What to do if things go wrong?  Here is some common gotchas
      and workarounds.</para>

    <variablelist>
      <varlistentry>
        <term>Missing required packages</term>

        <listitem><para>This is a pretty common occurrence.  You will
          either need to wait for a new set of packages where the
          missing packages were built or get someone to re-start the
	  package build for you.  <emphasis>Do not</emphasis> attempt to build
	  the missing packages on your own machine and add them into the fray.
          While you might be able to get away with this if you are
          extremely careful the vast majority of the time you will miss
          some little detail and the simple process of adding a
          package could make hundreds of others come up mysteriously
          broken.</para></listitem>
      </varlistentry>

      <varlistentry>
        <term>Required packages will not fit</term>

        <listitem><para>This happens on occasion too and is relatively
          easy to fix.  Simply edit
	  <filename>print-cdrom-packages.sh</filename> to move
          packages around until they fit.  Yes this is an iterative
	  process and one of the reasons why you should enable
	  <varname>fake</varname> in <filename>oneshot.pl</filename> until you
	  have gotten things the way you want them.  Re-run
          <command>./doit.sh</command> after you made your
          adjustments.</para></listitem>
      </varlistentry>

      <varlistentry>
        <term>Required packages not on the right (or any) disc</term>

        <listitem><para>This usually means you did not add them to
          <filename>print-cdrom-packages.sh</filename> or you put them
          on the wrong disc.  This script is the gospel by which this
          whole process determines where a package must be.  If you
          want to force a package to land on a particular disc this is
          the only way to ensure that it will
          happen.</para></listitem>
      </varlistentry>
    </variablelist>

    <para>If you get completely stuck and can not figure out why things
      are borked or how to fix them then email &a.steve; for
      assistance.</para>

  </sect1>

</article>