- Rename .sgml files to .xml

- Reflect the rename in referencing files

Approved by:	doceng (implicit)

1 files changed, 498 insertions, 0 deletions

diff --git a/en_US.ISO8859-1/books/developers-handbook/testing/chapter.xml b/en_US.ISO8859-1/books/developers-handbook/testing/chapter.xml
new file mode 100644
index 0000000000..9470863c45
--- /dev/null
+++ b/en_US.ISO8859-1/books/developers-handbook/testing/chapter.xml
@@ -0,0 +1,498 @@
+<?xml version="1.0" encoding="iso-8859-1" standalone="no"?>
+<!--
+     The FreeBSD Documentation Project
+
+     $FreeBSD$
+-->
+
+<chapter id="testing">
+  <title>Regression and Performance Testing</title>
+
+  <para>Regression tests are used to exercise a particular bit of the
+    system to check that it works as expected, and to make sure that
+    old bugs are not reintroduced.</para>
+
+  <para>The &os; regression testing tools can be found in the &os;
+    source tree in the directory <filename
+      class="directory">src/tools/regression</filename>.</para>
+
+  <section id="testing-micro-benchmark">
+    <title>Micro Benchmark Checklist</title>
+
+    <para>This section contains hints for doing proper
+      micro-benchmarking on &os; or of &os; itself.</para>
+
+    <para>It is not possible to use all of the suggestions below every
+      single time, but the more used, the better the benchmark's
+      ability to test small differences will be.</para>
+
+    <itemizedlist>
+      <listitem>
+	<para>Disable <acronym>APM</acronym> and any other kind of
+	  clock fiddling (<acronym>ACPI</acronym> ?).</para>
+      </listitem>
+
+      <listitem>
+	<para>Run in single user mode.  E.g., &man.cron.8;, and other
+	  daemons only add noise.  The &man.sshd.8; daemon can also
+	  cause problems.  If ssh access is required during testing
+	  either disable the SSHv1 key regeneration, or kill the
+	  parent <command>sshd</command> daemon during the
+	  tests.</para>
+      </listitem>
+
+      <listitem>
+	<para>Do not run &man.ntpd.8;.</para>
+      </listitem>
+
+      <listitem>
+	<para>If &man.syslog.3; events are generated, run
+	  &man.syslogd.8; with an empty
+	  <filename>/etc/syslogd.conf</filename>, otherwise, do not
+	  run it.</para>
+      </listitem>
+
+      <listitem>
+	<para>Minimize disk-I/O, avoid it entirely if possible.</para>
+      </listitem>
+
+      <listitem>
+	<para>Do not mount file systems that are not needed.</para>
+      </listitem>
+
+      <listitem>
+	<para>Mount <filename class="directory">/</filename>,
+	  <filename class="directory">/usr</filename>, and any other
+	  file system as read-only if possible.  This removes atime
+	  updates to disk (etc.) from the I/O picture.</para>
+      </listitem>
+
+      <listitem>
+	<para>Reinitialize the read/write test file system with
+	  &man.newfs.8; and populate it from a &man.tar.1; or
+	  &man.dump.8; file before every run.  Unmount and mount it
+	  before starting the test.  This results in a consistent file
+	  system layout.  For a worldstone test this would apply to
+	  <filename class="directory">/usr/obj</filename> (just
+	  reinitialize with <command>newfs</command> and mount).  To
+	  get 100% reproducibility, populate the file system from a
+	  &man.dd.1; file (i.e.: <command>dd
+	    if=<filename>myimage</filename> of=<filename
+	      class="devicefile">/dev/ad0s1h</filename>
+	    bs=1m</command>)</para>
+      </listitem>
+
+      <listitem>
+	<para>Use malloc backed or preloaded &man.md.4;
+	  partitions.</para>
+      </listitem>
+
+      <listitem>
+	<para>Reboot between individual iterations of the test, this
+	  gives a more consistent state.</para>
+      </listitem>
+
+      <listitem>
+	<para>Remove all non-essential device drivers from the kernel.
+	  For instance if USB is not needed for the test, do not put
+	  USB in the kernel.  Drivers which attach often have timeouts
+	  ticking away.</para>
+      </listitem>
+
+      <listitem>
+	<para>Unconfigure hardware that are not in use.  Detach disks
+	  with &man.atacontrol.8; and &man.camcontrol.8; if the disks
+	  are not used for the test.</para>
+      </listitem>
+
+      <listitem>
+	<para>Do not configure the network unless it is being tested,
+	  or wait until after the test has been performed to ship the
+	  results off to another computer.</para>
+
+	<para>If the system must be connected to a public network,
+	  watch out for spikes of broadcast traffic.  Even though it
+	  is hardly noticeable, it will take up CPU cycles.  Multicast
+	  has similar caveats.</para>
+      </listitem>
+
+      <listitem>
+	<para>Put each file system on its own disk.  This minimizes
+	  jitter from head-seek optimizations.</para>
+      </listitem>
+
+      <listitem>
+	<para>Minimize output to serial or VGA consoles.  Running
+	  output into files gives less jitter.  (Serial consoles
+	  easily become a bottleneck.)  Do not touch keyboard while
+	  the test is running, even <keycap>space</keycap> or
+	  <keycap>back-space</keycap> shows up in the numbers.</para>
+      </listitem>
+
+      <listitem>
+	<para>Make sure the test is long enough, but not too long.  If
+	  the test is too short, timestamping is a problem.  If it is
+	  too long temperature changes and drift will affect the
+	  frequency of the quartz crystals in the computer.  Rule of
+	  thumb: more than a minute, less than an hour.</para>
+      </listitem>
+
+      <listitem>
+	<para>Try to keep the temperature as stable as possible around
+	  the machine.  This affects both quartz crystals and disk
+	  drive algorithms.  To get real stable clock, consider
+	  stabilized clock injection.  E.g., get a OCXO + PLL, inject
+	  output into clock circuits instead of motherboard xtal.
+	  Contact &a.phk; for more information about this.</para>
+      </listitem>
+
+      <listitem>
+	<para>Run the test at least 3 times but it is better to run
+	  more than 20 times both for <quote>before</quote> and
+	  <quote>after</quote> code.  Try to interleave if possible
+	  (i.e.: do not run 20 times before then 20 times after), this
+	  makes it possible to spot environmental effects.  Do not
+	  interleave 1:1, but 3:3, this makes it possible to spot
+	  interaction effects.</para>
+
+	<para>A good pattern is: <literal>bababa{bbbaaa}*</literal>.
+	  This gives hint after the first 1+1 runs (so it is possible
+	  to stop the test if it goes entirely the wrong way), a
+	  standard deviation after the first 3+3 (gives a good
+	  indication if it is going to be worth a long run) and
+	  trending and interaction numbers later on.</para>
+      </listitem>
+
+      <listitem>
+	<para>Use &man.ministat.1;
+	  to see if the numbers are significant.  Consider buying
+	  <quote>Cartoon guide to statistics</quote> ISBN:
+	  0062731025, highly recommended, if you have forgotten or
+	  never learned about standard deviation and Student's
+	  T.</para>
+      </listitem>
+
+      <listitem>
+	<para>Do not use background &man.fsck.8; unless the test is a
+	  benchmark of background <command>fsck</command>.  Also,
+	  disable <varname>background_fsck</varname> in
+	  <filename>/etc/rc.conf</filename> unless the benchmark is
+	  not started at least 60+<quote><command>fsck</command>
+	    runtime</quote> seconds after the boot, as &man.rc.8;
+	  wakes up and checks if <command>fsck</command> needs to run
+	  on any file systems when background <command>fsck</command>
+	  is enabled.  Likewise, make sure there are no snapshots
+	  lying around unless the benchmark is a test with
+	  snapshots.</para>
+      </listitem>
+
+      <listitem>
+	<para>If the benchmark show unexpected bad performance, check
+	  for things like high interrupt volume from an unexpected
+	  source.  Some versions of <acronym>ACPI</acronym> have been
+	  reported to <quote>misbehave</quote> and generate excess
+	  interrupts.  To help diagnose odd test results, take a few
+	  snapshots of <command>vmstat -i</command> and look for
+	  anything unusual.</para>
+      </listitem>
+
+      <listitem>
+	<para>Make sure to be careful about optimization parameters
+	  for kernel and userspace, likewise debugging.  It is easy to
+	  let something slip through and realize later the test was
+	  not comparing the same thing.</para>
+      </listitem>
+
+      <listitem>
+	<para>Do not ever benchmark with the
+	  <literal>WITNESS</literal> and <literal>INVARIANTS</literal>
+	  kernel options enabled unless the test is interested to
+	  benchmarking those features.  <literal>WITNESS</literal> can
+	  cause 400%+ drops in performance.  Likewise, userspace
+	  &man.malloc.3; parameters default differently in -CURRENT
+	  from the way they ship in production releases.</para>
+      </listitem>
+    </itemizedlist>
+  </section>
+
+  <section id="testing-tinderbox">
+    <title>The &os; Source Tinderbox</title>
+
+    <para>The source Tinderbox consists of:</para>
+
+    <itemizedlist>
+      <listitem>
+	<para>A build script, <filename>tinderbox</filename>, that
+	  automates checking out a specific version of the &os; source
+	  tree and building it.</para>
+      </listitem>
+      <listitem>
+	<para>A supervisor script, <filename>tbmaster</filename>, that
+	  monitors individual Tinderbox instances, logs their output,
+	  and emails failure notices.</para>
+      </listitem>
+      <listitem>
+	<para>A <acronym>CGI</acronym> script named
+	  <filename>index.cgi</filename> that reads a set of tbmaster
+	  logs and presents an easy-to-read <acronym>HTML</acronym>
+	  summary of them.</para>
+      </listitem>
+      <listitem>
+	<para>A set of build servers that continually test the tip of
+	  the most important &os; code branches.</para>
+      </listitem>
+      <listitem>
+	<para>A webserver that keeps a complete set of Tinderbox logs
+	  and displays an up-to-date summary.</para>
+      </listitem>
+    </itemizedlist>
+
+    <para>The scripts are maintained and were developed by &a.des;,
+      and are now written in Perl, a move on from their original
+      incarnation as shell scripts.  All scripts and configuration
+      files are kept in <ulink
+	url="http://www.freebsd.org/cgi/cvsweb.cgi/projects/tinderbox/">/projects/tinderbox/</ulink>.</para>
+
+    <para>For more information about the tinderbox and tbmaster
+      scripts at this stage, see their respective man pages:
+      tinderbox(1) and tbmaster(1).</para>
+
+    <section>
+      <title>The <filename>index.cgi</filename> Script</title>
+
+      <para>The <filename>index.cgi</filename> script generates the
+	<acronym>HTML</acronym> summary of tinderbox and tbmaster
+	logs.  Although originally intended to be used as a
+	<acronym>CGI</acronym> script, as indicated by its name, this
+	script can also be run from the command line or from a
+	&man.cron.8; job, in which case it will look for logs in the
+	directory where the script is located.  It will automatically
+	detect context, generating <acronym>HTTP</acronym> headers
+	when it is run as a <acronym>CGI</acronym> script.  It
+	conforms to <acronym>XHTML</acronym> standards and is styled
+	using <acronym>CSS</acronym>.</para>
+
+      <para>The script starts in the <function>main()</function> block
+	by attempting to verify that it is running on the official
+	Tinderbox website.  If it is not, a page indicating it is not
+	an official website is produced, and a <acronym>URL</acronym>
+	to the official site is provided.</para>
+
+      <para>Next, it scans the log directory to get an inventory of
+	configurations, branches and architectures for which log files
+	exist, to avoid hard-coding a list into the script and
+	potentially ending up with blank rows or columns.  This
+	information is derived from the names of the log files
+	matching the following pattern:</para>
+
+      <programlisting>tinderbox-$config-$branch-$arch-$machine.{brief,full}</programlisting>
+
+      <para>The configurations used on the official Tinderbox build
+	servers are named for the branches they build.  For example,
+	the <literal>releng_8</literal> configuration is used to build
+	<literal>RELENG_8</literal> as well as all still-supported
+	release branches.</para>
+
+      <para>Once all of this startup procedure has been successfully
+	completed, <function>do_config()</function> is called for each
+	configuration.</para>
+
+      <para>The <function>do_config()</function> function generates
+	<acronym>HTML</acronym> for a single Tinderbox
+	configuration.</para>
+
+      <para>It works by first generating a header row, then iterating
+	over each branch build with the specified configuration,
+	producing a single row of results for each in the following
+	manner:</para>
+
+      <itemizedlist>
+	<listitem>
+	  <para>For each item:</para>
+	  <itemizedlist>
+	    <listitem>
+	      <para>For each machine within that architecture:</para>
+	      <itemizedlist>
+		<listitem>
+		  <para>If a brief log file exists, then:</para>
+		  <itemizedlist>
+		    <listitem>
+		      <para>Call <function>success()</function> to
+			detemine the outcome of the build.</para>
+		    </listitem>
+		    <listitem>
+		      <para>Output the modification size.</para>
+		    </listitem>
+		    <listitem>
+		      <para>Output the size of the brief log file with
+			a link to the log file itself.</para>
+		    </listitem>
+		    <listitem>
+		      <para>If a full log file also exists,
+			then:</para>
+		      <itemizedlist>
+			<listitem>
+			  <para>Output the size of the full log file
+			    with a link to the log file itself.</para>
+			</listitem>
+		      </itemizedlist>
+		    </listitem>
+		  </itemizedlist>
+		</listitem>
+		<listitem>
+		  <para>Otherwise:</para>
+		  <itemizedlist>
+		    <listitem>
+		      <para>No output.</para>
+		    </listitem>
+		  </itemizedlist>
+		</listitem>
+	      </itemizedlist>
+	    </listitem>
+	  </itemizedlist>
+	</listitem>
+      </itemizedlist>
+
+      <para>The <function>success()</function> function mentioned
+	above scans a brief log file for the string <quote>tinderbox
+	  run completed</quote> in order to determine whether the
+	build was successful.</para>
+
+      <para>Configurations and branches are sorted according to their
+	branch rank.  This is computed as follows:</para>
+
+      <itemizedlist>
+	<listitem>
+	  <para><literal>HEAD</literal> and <literal>CURRENT</literal>
+	    have rank 9999.</para>
+	</listitem>
+	<listitem>
+	  <para><literal>RELENG_<replaceable>x</replaceable></literal>
+	    has rank <replaceable>xx</replaceable>99.</para>
+	</listitem>
+	<listitem>
+	  <para><literal>RELENG_<replaceable>x</replaceable>_<replaceable>y</replaceable></literal>
+	    has rank <replaceable>xxyy</replaceable>.</para>
+	</listitem>
+      </itemizedlist>
+
+      <para>Tthis means that <literal>HEAD</literal> always ranks
+	highest, and <literal>RELENG</literal> branches are ranked in
+	numerical order, with each <literal>STABLE</literal> branch
+	ranking higher than the release branches forked off of it.
+	For instance, for &os;&nbsp;8, the order from highest to
+	lowest would be:</para>
+
+      <itemizedlist>
+	<listitem>
+	  <para><literal>RELENG_8</literal> (branch rank 899).</para>
+	</listitem>
+	<listitem>
+	  <para><literal>RELENG_8_3</literal> (branch rank
+	    803).</para>
+	</listitem>
+	<listitem>
+	  <para><literal>RELENG_8_2</literal> (branch rank
+	    802).</para>
+	</listitem>
+	<listitem>
+	  <para><literal>RELENG_8_1</literal> (branch rank
+	    801).</para>
+	</listitem>
+	<listitem>
+	  <para><literal>RELENG_8_0</literal> (branch rank
+	    800).</para>
+	</listitem>
+      </itemizedlist>
+
+      <para>The colors that Tinderbox uses for each cell in the table
+	are defined by <acronym>CSS</acronym>.  Successful builds are
+	displayed with green text; unsuccessful builds are displayed
+	with red text.  The color fades as time passes since the
+	corresponding build, with every half an hour bringing the
+	color closer to grey.</para>
+    </section>
+
+    <section>
+      <title>Official Build Servers</title>
+
+      <para>The official Tinderbox build servers are hosted by <ulink
+	  url="http://www.sentex.ca">Sentex Data
+	  Communications</ulink>, who also host the <ulink
+	  url="http://www.freebsd.org/projects/netperf/cluster.html">&os;
+	  Netperf Cluster</ulink>.</para>
+
+      <para>Three build servers currently exist:</para>
+
+      <para><emphasis>freebsd-current.sentex.ca</emphasis>
+	builds:</para>
+
+      <itemizedlist>
+	<listitem>
+	  <para><literal>HEAD</literal> for amd64, arm, i386,
+	    i386/pc98, ia64, mips, powerpc, powerpc64, and sparc64,
+	    which takes approximately four hours.</para>
+	</listitem>
+	<listitem>
+	  <para><literal>RELENG_9</literal> and supported
+	    9.<replaceable>X</replaceable> branches for amd64, arm,
+	    i386, i386/pc98, ia64, mips, powerpc, powerpc64, and
+	    sparc64, which takes approximately three and a half
+	    hours.</para>
+	</listitem>
+      </itemizedlist>
+
+      <para><emphasis>freebsd-stable.sentex.ca</emphasis>
+	builds:</para>
+
+      <itemizedlist>
+	<listitem>
+	  <para><literal>RELENG_8</literal> and supported
+	    8.<replaceable>X</replaceable> branches for amd64, i386,
+	    i386/pc98, ia64, mips, powerpc and sparc64, and each
+	    branch takes approximately six hours.</para>
+	</listitem>
+      </itemizedlist>
+
+      <para><emphasis>freebsd-legacy.sentex.ca</emphasis>
+	builds:</para>
+
+      <itemizedlist>
+	<listitem>
+	  <para><literal>RELENG_7</literal> and supported
+	    7.<replaceable>X</replaceable> branches for amd64, i386,
+	    i386/pc98, ia64, powerpc, and sparc64, and each branch
+	    takes approximately three hours.</para>
+	</listitem>
+      </itemizedlist>
+    </section>
+
+    <section>
+      <title>Official Summary Site</title>
+
+      <para>Summaries and logs from the official build servers are
+	available online at <ulink
+	  url="http://tinderbox.FreeBSD.org">http://tinderbox.FreeBSD.org</ulink>,
+	hosted by &a.des; and set up as follows:</para>
+
+      <itemizedlist>
+	<listitem>
+	  <para>A &man.cron.8; job checks the build servers at regular
+	    intervals and downloads any new log files using
+	    &man.rsync.1;.</para>
+	</listitem>
+	<listitem>
+	  <para>Apache is set up to use <filename>index.cgi</filename>
+	    as <literal>DirectoryIndex</literal>.</para>
+	</listitem>
+	<listitem>
+	  <para>A <ulink
+	      url="https://www.varnish-cache.org/">Varnish</ulink>
+	    instance in front of Apache ensures that
+	    <filename>index.cgi</filename> does not need to run more
+	    than once every two minutes.</para>
+	</listitem>
+      </itemizedlist>
+    </section>
+  </section>
+</chapter>