diff options
Diffstat (limited to 'en_US.ISO8859-1/articles/fbsd-from-scratch/article.xml')
| -rw-r--r-- | en_US.ISO8859-1/articles/fbsd-from-scratch/article.xml | 599 |
1 files changed, 0 insertions, 599 deletions
diff --git a/en_US.ISO8859-1/articles/fbsd-from-scratch/article.xml b/en_US.ISO8859-1/articles/fbsd-from-scratch/article.xml deleted file mode 100644 index c3cb6d8e81..0000000000 --- a/en_US.ISO8859-1/articles/fbsd-from-scratch/article.xml +++ /dev/null @@ -1,599 +0,0 @@ -<?xml version="1.0" encoding="iso-8859-1"?> -<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V5.0-Based Extension//EN" - "http://www.FreeBSD.org/XML/share/xml/freebsd50.dtd" [ -<!ENTITY scratch.ap "<application xmlns='http://docbook.org/ns/docbook'>FreeBSD From Scratch</application>"> -]> -<article xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en"> - <info><title>FreeBSD From Scratch</title> - - - <author><personname><firstname>Jens</firstname><surname>Schweikhardt</surname></personname><affiliation> - <address><email>schweikh@FreeBSD.org</email></address> - </affiliation></author> - <copyright> - <year>2002,2003,2004,2008</year> - <holder>Jens Schweikhardt</holder> - </copyright> - - <legalnotice xml:id="trademarks" role="trademarks"> - &tm-attrib.freebsd; - &tm-attrib.adobe; - &tm-attrib.general; - </legalnotice> - - <pubdate>$FreeBSD$</pubdate> - - <releaseinfo>$FreeBSD$</releaseinfo> - - <abstract> - <para>This article describes my efforts at &scratch.ap;: a fully - automated installation of a customized &os; system compiled from - source, including compilation of all your favorite ports and - configured to match your idea of the perfect system. If you - think <command>make world</command> is a wonderful concept, - &scratch.ap; extends it to <command>make evenmore</command>.</para> - </abstract> - </info> - - <sect1 xml:id="introduction"> - <title>Introduction</title> - - <para>Have you ever upgraded your system with <command>make world</command>? - There is a problem if you have only one system on your disks. If - the <buildtarget>installworld</buildtarget> fails partway through, - you are left with a broken system that might not even boot any - longer. Or maybe the <buildtarget>installworld</buildtarget> runs - smoothly but the new kernel does not boot. Then it is time to - reach for the Fixit CD and dig for those backups you have taken - half a year ago.</para> - - <para>I believe in the <quote>wipe your disks when upgrading systems</quote> - paradigm. Wiping disks, or rather partitions, makes sure there is no - old cruft left lying around, something which most upgrade procedures - just do not care about. But wiping the partitions means you have to - also recompile/reinstall all your ports and packages and then - redo all your carefully crafted configuration tweaks. - If you think that this task should be automated as well, read on.</para> - </sect1> - - <sect1 xml:id="why"> - <title>Why would I (not) want &scratch.ap;?</title> - - <para>This is a legitimate question. We have - <application>sysinstall</application> and the well known way to - compile the kernel and the userland tools.</para> - - <para>The problem with <application>sysinstall</application> is - that it is severely limited in what, where and how it can install.</para> - - <itemizedlist> - <listitem> - <para>It is normally used to install pre-built distribution sets and - packages from some other source (CD, DVD, FTP). It cannot install - the result of a <literal>make buildworld</literal>.</para> - </listitem> - - <listitem> - <para>It cannot install a second system under a directory - in a running system.</para> - </listitem> - - <listitem> - <para>It cannot install in <application>Vinum</application> - or <application>ZFS</application> partitions.</para> - </listitem> - - <listitem> - <para>It cannot compile ports, only install precompiled packages.</para> - </listitem> - - <listitem> - <para>It is hard to script or to make arbitrary post-installation - changes.</para> - </listitem> - - <listitem> - <para>Last but not least, <application>sysinstall</application> - is semi-officially at its End-Of-Life.</para> - </listitem> - </itemizedlist> - - <para>The well known way to build and install the world, as - described in <link xlink:href="http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html">the - Handbook</link>, by default replaces - the existing system. Only the kernel and modules are saved. - System binaries, headers and a lot of other files are overwritten; - obsolete files are still present and can cause surprises. If the - upgrade fails for any reason, it may be hard or even impossible to - restore the previous state of the system.</para> - - <para>&scratch.ap; solves all these problems. The strategy is - simple: use a running system to install a new system under an empty - directory tree, while new partitions are mounted appropriately - in that tree. Many config files can be copied to the appropriate - place and &man.mergemaster.8; can take care of those that cannot. - Arbitrary post-configuration of the new system can be - done from within the old system, up to the point where you can - chroot to the new system. In other words, we go through three - stages, where each stage consists of either running a shell - script or invoking <command>make</command>:</para> - - <orderedlist> - <listitem> - <para><filename>stage_1.sh</filename>: - Create a new bootable system under an empty directory and merge - or copy as many files as are necessary. - Then boot the new system.</para> - </listitem> - - <listitem> - <para><filename>stage_2.sh</filename>: - Install desired ports.</para> - </listitem> - - <listitem> - <para><filename>stage_3.mk</filename>: - Do post-configuration for software installed in previous stage.</para> - </listitem> - </orderedlist> - - <para>Once you have used &scratch.ap; to build a second system and - found it works satisfactorily for a couple of weeks, you can then - use it again to reinstall the original system. From now on, whenever - you feel like an update is in order, you simply toggle the - partitions you want to wipe and reinstall.</para> - - <para>Maybe you have heard of or even tried <link xlink:href="http://www.linuxfromscratch.org/">Linux From Scratch</link>, - or LFS for short. LFS also describes how to build and install a - system from scratch in empty partitions using a running system. - The focus in LFS seems to be to show the role of each system - component (such as kernel, compiler, devices, shell, terminal database, - etc) and the details of each component's installation. - &scratch.ap; does not go into that much detail. My goal is to - provide an automated and complete installation, not explaining all - the gory details that go on under the hood when making the world. - In case you want to explore &os; at this level of detail, start - looking at <filename>/usr/src/Makefile</filename> and follow the - actions of a <command>make buildworld</command>.</para> - - <para>There are also downsides in the approach taken by &scratch.ap; - that you should bear in mind.</para> - - <!-- XXX: A nice idea would be to write stage_2.sh using a jail - that runs into the newly installed world from stage_1. Having - properly set up a network address as the jail's primary IP - address, it might even be possible to build ports in a chroot - without uninstalling anything from the 'host' system. But - keep in mind that even jails run on the 'host' kernel. --> - - <itemizedlist> - <listitem> - <para>While compiling the ports during stage two the system can - not be used for its usual duties. If you run a production server - you have to consider the downtime caused by stage two. The ports - compiled by <filename>stage_2.conf.default</filename> below require - about 8 hours (of which 4 hours are due to - <application>OpenOffice.org</application>) to build on a contemporary - system. If you prefer to install packages instead of ports, - you can significantly reduce the downtime to about 10 minutes.</para> - </listitem> - </itemizedlist> - </sect1> - - <sect1 xml:id="prerequisites"> - <title>Prerequisites</title> - - <para>For going the &scratch.ap; way, you need to have:</para> - - <itemizedlist> - <listitem> - <para>A running &os; system with sources and a ports tree.</para> - </listitem> - - <listitem> - <para>At least one unused partition where the new system will be - installed.</para> - </listitem> - - <listitem> - <para>Experience with running &man.mergemaster.8;. Or at least no fear - doing so.</para> - </listitem> - - <listitem> - <para>If you have no or only a slow link to the Internet: the distfiles - for your favorite ports.</para> - </listitem> - - <listitem> - <para>Basic knowledge of shell scripting with the Bourne shell, - &man.sh.1;.</para> - </listitem> - - <listitem> - <para>Finally, you should also be able to tell your boot - loader how to boot the new system, either interactively, or - by means of a config file.</para> - </listitem> - </itemizedlist> - </sect1> - - <sect1 xml:id="stage1"> - <title>Stage One: System Installation</title> - - <para>The first version of this article used a single shell script - for stage one where all your customization had to be done by editing - the script. After valuable user feedback I have decided to - separate the code and data in the scripts. This allows to have - different configuration data sets to install different systems - without changing any of the code scripts.</para> - - <para>The code script for stage one is - <filename>stage_1.sh</filename> and when run with exactly one - argument, like</para> - - <informalexample> - <screen>&prompt.root; <userinput>./stage_1.sh <replaceable>default</replaceable></userinput></screen> - </informalexample> - - <para>will read its configuration from - <filename>stage_1.conf.default</filename> and write a log to - <filename>stage_1.log.default</filename>.</para> - - <para>Further below you find my <filename>stage_1.conf.default</filename>. - You need to customize it in various places to match your idea of the - <quote>perfect system</quote>. I have tried to extensively comment - the places you should adapt. The configuration script must provide - four shell functions, <command>create_file_systems</command>, - <command>create_etc_fstab</command>, <command>copy_files</command> - and <command>all_remaining_customization</command> (in case it - matters: this is also the sequence in which they will be called - from <filename>stage_1.sh</filename>).</para> - - <para>The points to ponder are:</para> - - <itemizedlist> - <listitem> - <para>Partition layout.</para> - - <para>I do not subscribe to the idea of a single huge partition - for the whole system. My systems generally have at least - one partition for - <filename>/</filename>, - <filename>/usr</filename> and - <filename>/var</filename> with - <filename>/tmp</filename> symlinked to - <filename>/var/tmp</filename>. - In addition I share the file systems for - <filename>/home</filename> (user homes), - <filename>/home/ncvs</filename> (&os; CVS repository replica), - <filename>/usr/ports</filename> (the ports tree), - <filename>/src</filename> (various checked out src trees) and - <filename>/share</filename> (other shared data without the need - for backups, like the news spool).</para> - </listitem> - - <listitem> - <para>Luxury items.</para> - - <para>What you want immediately after booting the new system and - even before starting stage two. The reason for not simply - chrooting to the new system during stage one and installing - all my beloved ports is that in theory and in practice there - are bootstrap and consistency issues: stage one has your old - kernel running, but the chrooted environment consists of new - binaries and headers. If the new binaries use a new system - call, these binaries will die with <literal>SIGSYS, Bad - system call</literal>, because the old kernel does not have - that system call. I have seen other issues when I tried - building <package>lang/perl5.8</package>.</para> - - </listitem> - </itemizedlist> - - <para>Before you run <filename>stage_1.sh</filename> make sure - you have completed the usual tasks in preparation for - <command>make installworld installkernel</command>, like:</para> - - <itemizedlist> - <listitem> - <para>configured your kernel config file</para> - </listitem> - - <listitem> - <para>successfully completed <command>make buildworld</command></para> - </listitem> - - <listitem> - <para>successfully completed <command>make buildkernel - KERNCONF=<replaceable>whatever</replaceable></command></para> - </listitem> - </itemizedlist> - - <para>When you run <filename>stage_1.sh</filename> for the first - time, and the config files copied from your running system to the - new system are not up-to-date with respect to what is under - <filename>/usr/src</filename>, <command>mergemaster</command> will - ask you how to proceed. I recommend merging the changes. If you get - tired of going through the dialogues you can simply update the files - on your <emphasis>running</emphasis> system once (Only if this is an - option. You probably do not want to do this if one of your systems - runs <literal>-STABLE</literal> and the other - <literal>-CURRENT</literal>. The changes may be incompatible). - Subsequent <command>mergemaster</command> invocations will detect - that the RCS version IDs match those under - <filename>/usr/src</filename> and skip the file.</para> - - <para>The <filename>stage_1.sh</filename> script will stop at the - first command that fails (returns a non-zero exit status) due to - <command>set -e</command>, so you cannot overlook errors. It will - also stop if you use an unset environment variable, probably due - to a typo. You should correct any errors in your version of - <filename>stage_1.conf.default</filename> before you go on.</para> - - <para>In <filename>stage_1.sh</filename> we invoke - <command>mergemaster</command>. Even if none of the files requires a - merge, it will display and ask at the end</para> - - <screen>*** Comparison complete -*** Saving mtree database for future upgrades - -Do you wish to delete what is left of /var/tmp/temproot.stage1? [no] <userinput>no</userinput></screen> - - <para>Please answer <literal>no</literal> or just hit - <keycap>Enter</keycap>. The reason is that <command>mergemaster</command> - will have left a few zero sized files below - <filename>/var/tmp/temproot.stage1</filename> which will be copied to the - new system later (unless already there).</para> - - <para>After that <command>mergemaster</command> will list the files it - installed and ask if the new <filename>login.conf</filename> should be - generated:</para> - -<screen>*** You chose the automatic install option for files that did not - exist on your system. The following were installed for you: - /newroot/etc/defaults/rc.conf - ... - /newroot/COPYRIGHT - -*** You installed a new aliases file into /newroot/etc/mail, but - the newaliases command is limited to the directories configured - in sendmail.cf. Make sure to create your aliases database by - hand when your sendmail configuration is done. - -*** You installed a login.conf file, so make sure that you run - '/usr/bin/cap_mkdb /newroot/etc/login.conf' - to rebuild your login.conf database - - Would you like to run it now? y or n [n]</screen> - - <para>The answer does not matter since <filename>stage_1.sh</filename> will - run &man.cap.mkdb.1; for you in any case.</para> - - <para>Here is the author's <link xlink:href="stage_1.conf.default"><filename>stage_1.conf.default</filename></link>, - which you need to modify substantially. The comments give you - enough information what to change.</para> - -<programlisting><xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="stage_1.conf.default" parse="text"/></programlisting> - - <para>Download <link xlink:href="stage_1.conf.default"><filename>stage_1.conf.default</filename> - </link>.</para> - - <para>Running this script installs a system that when booted - provides:</para> - <itemizedlist> - <listitem> - <para>Inherited users and groups.</para> - </listitem> - <listitem> - <para>Firewalled Internet connectivity over Ethernet.</para> - </listitem> - <listitem> - <para>Correct time zone and NTP.</para> - </listitem> - <listitem> - <para>Some more minor configuration, like - <filename>/etc/ttys</filename> and - <command>inetd</command>.</para> - </listitem> - </itemizedlist> - - <para>Other areas are prepared for configuration, but will not work - until stage two is completed. For example we have copied files to - configure printing and X11. Printing however is likely to need - applications not found in the base system, like &postscript; - utilities. X11 will not run before we have compiled the server, - libraries and programs.</para> - </sect1> - - <sect1 xml:id="stage2"> - <title>Stage Two: Ports Installation</title> - - <note> - <para>It is also possible to install the (precompiled) - packages at this stage, instead of compiling ports. In this case, - <filename>stage_2.sh</filename> would be nothing more than a list of - <command>pkg_add</command> commands. I trust you know how to write - such a script. Here we concentrate on the more flexible and - traditional way of using the ports.</para> - </note> - - <para>The following <filename>stage_2.sh</filename> script is how I - install my favorite ports. It can be run any number of times and - will skip all ports that are already installed. It supports the - <emphasis>dryrun</emphasis> option (<option>-n</option>) to just - show what would be done. You run it like <filename>stage_1.sh</filename> - with exactly one argument to denote a config file, e.g.</para> - - <informalexample> - <screen>&prompt.root; <userinput>./stage_2.sh <replaceable>default</replaceable></userinput></screen> - </informalexample> - - <para>which will read the list of ports from - <filename>stage_2.conf.default</filename>.</para> - - <para>The list of ports consists of lines with two or more space - separated words: the category and the port, optionally followed by - an installation command that will compile and install the port - (default: <command>make install BATCH=yes < /dev/null</command>). - Empty lines and lines - starting with # are ignored. Most of the time it suffices to only - name category and port. A few ports however can be fine tuned by - specifying <command>make</command> variables, e.g.:</para> - - <programlisting>www mozilla make WITHOUT_MAILNEWS=yes WITHOUT_CHATZILLA=yes install</programlisting> - - <para>In fact you can specify arbitrary shell commands, so you are - not restricted to simple <command>make</command> invocations:</para> - - <programlisting>java jdk16 echo true > files/license.sh; make install BATCH=yes < /dev/null -print acroread8 yes accept | make install PAGER=ls -x11-fonts gnu-unifont make install && mkfontdir /usr/local/lib/X11/fonts/local -news inn-stable CONFIGURE_ARGS="--enable-uucp-rnews --enable-setgid-inews" make install</programlisting> - - <para>The first two lines are examples how you can handle ports - asking you to accept a licence. Note how the line for - <package>news/inn-stable</package> is an example - for a one-shot shell variable assignment to - <literal>CONFIGURE_ARGS</literal>. The port - <filename>Makefile</filename> will use this as an initial value - and augment some other essential args. The difference to - specifying a <application>make</application> variable on the command line - with</para> - - <programlisting>news inn-stable make CONFIGURE_ARGS="--enable-uucp-rnews --enable-setgid-inews" install</programlisting> - - <para>is that the latter will override instead of augment. It depends on - the particular port which method you want.</para> - - <para>Be careful that your ports do not use an interactive install, i.e. - they should not try to read from stdin other than what you explicitly - give them on stdin. If they do, they will read the next line(s) from - your list of ports in the here-document and get confused. If - <filename>stage_2.sh</filename> mysteriously skips a port or stops - processing, this is likely the reason.</para> - - <para>Below is <filename>stage_2.conf.default</filename>. A log file named - <filename>LOGDIR/category+port</filename> is created for each port - it actually installs.</para> - -<programlisting><xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="stage_2.conf.default" parse="text"/></programlisting> - - <para>Download <link xlink:href="stage_2.conf.default"><filename>stage_2.conf.default</filename></link>.</para> - </sect1> - - <sect1 xml:id="stage3"> - <title>Stage Three</title> - - <para>You have installed your beloved ports during stage two. Some - ports require a little bit of configuration. This is what stage three, - the post-configuration is for. I could have integrated this - post-configuration at the end of the <filename>stage_2.sh</filename> - script. However, I think there is a conceptual difference between - installing a port and modifying its out-of-the-box configuration - that warrants a separate stage.</para> - - <para>I have chosen to implement stage three as a - <filename>Makefile</filename> because this allows easy selection of - what you want to configure simply by running:</para> - - <informalexample> - <screen>&prompt.root; <userinput>make -f stage_3.mk <replaceable>target</replaceable></userinput></screen> - </informalexample> - - <para>As with <filename>stage_2.sh</filename> make sure you have - <filename>stage_3.mk</filename> available after booting the new - system, either by putting it on a shared partition or copying it - somewhere on the new system.</para> - </sect1> - - <sect1 xml:id="limitations"> - <title>Limitations</title> - - <para>The automated installation of a port may prove difficult if it - is interactive and does not support <command>make BATCH=YES - install</command>. For a few ports the interaction is nothing more - than typing <literal>yes</literal> when asked to accept some license. - If the answer is read from standard input, simply pipe the - appropriate answers to the installation command (e.g. <command>yes | - make install</command>. For other ports you need to investigate - where exactly the interactive command is located and deal with it - appropriately. See the examples above for - <package>print/acroread8</package> and - <package>java/jdk16</package>.</para> - - <para>You should also be aware of upgrade issues for config files. - In general you do not know when and if the format or contents of a - config file changes. A new group may be added to - <filename>/etc/group</filename>, or <filename>/etc/passwd</filename> - may gain another field. All of this has happened in the past. Simply - copying a config file from the old to the new system may be enough - most of the time, but in these cases it was not. If you update a - system the canonical way (by overwriting the old files) you are - expected to use <command>mergemaster</command> to deal with changes - where you effectively want to merge your local config with - potentially new items. Unfortunately, <command>mergemaster</command> - is only available for base system files, not for anything installed - by ports. Some third party software seems to be especially designed - to keep me on my toes by changing the config file format every - fortnight. To detect such silent changes, I keep a copy of the - modified config files in the same place where I keep - <filename>stage_3.mk</filename> and compare the result with a - <application>make</application> rule, e.g. for - <application>apache</application>'s <filename>httpd.conf</filename> - in target <command>config_apache</command> with</para> - -<programlisting> -@if ! cmp -s /usr/local/etc/apache2/httpd.conf httpd.conf; then \ - echo "ATTENTION: the httpd.conf has changed. Please examine if"; \ - echo "the modifications are still correct. Here is the diff:"; \ - diff -u /usr/local/etc/apache2/httpd.conf httpd.conf; \ -fi -</programlisting> - - <para>If the diff is innocuous I can make the message go away with - <command>cp /usr/local/etc/apache2/httpd.conf - httpd.conf</command>.</para> - - <para>I have used &scratch.ap; several times to update a - <literal>7-CURRENT</literal> to <literal>7-CURRENT</literal> and - <literal>8-CURRENT</literal> to <literal>8-CURRENT</literal>, i.e. - I have never tried to install a <literal>8-CURRENT</literal> from - a <literal>7-CURRENT</literal> system or vice versa. Due to the - number of changes between different major release numbers I would - expect this process to be a bit more involved. Using &scratch.ap; - for upgrades within the realm of a <literal>STABLE</literal> branch - should work painlessly (although I have not yet tried it.)</para> - - </sect1> - - <sect1 xml:id="files"> - <title>The Files</title> - - <para>Here are the three files you need beside the config files - already shown above.</para> - - <para>This is the <link xlink:href="stage_1.sh"><filename>stage_1.sh</filename></link> - script, which you should not need to modify.</para> - -<programlisting><xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="stage_1.sh" parse="text"/></programlisting> - - <para>Download <link xlink:href="stage_1.sh"><filename>stage_1.sh</filename></link>.</para> - - <para>This is the <link xlink:href="stage_2.sh"><filename>stage_2.sh</filename></link> - script. You may want to modify the variables at the - beginning.</para> - -<programlisting><xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="stage_2.sh" parse="text"/></programlisting> - - <para>Download <link xlink:href="stage_2.sh"><filename>stage_2.sh</filename></link>.</para> - - <para>This is my <link xlink:href="stage_3.mk"><filename>stage_3.mk</filename></link> to - give you an idea how to automate all reconfiguration.</para> - -<programlisting><xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="stage_3.mk" parse="text"/></programlisting> - - <para>Download <link xlink:href="stage_3.mk"><filename>stage_3.mk</filename></link>.</para> - </sect1> - -</article> |