path: root/en_US.ISO8859-1/articles/wp-toolbox/article.xml

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V4.5-Based Extension//EN"
	"../../../share/xml/freebsd45.dtd" [
<!ENTITY ports		"Ports Collection">
<!ENTITY os.ports	"&os; &ports;">
<!ENTITY frisbee	"<application>Frisbee</application>">
<!ENTITY ghost		"<application>Ghost</application>">
<!ENTITY nessus		"<application>Nessus</application>">
]>

<article lang='en'>
  <title>Creating a Software Testing Environment Using &os;</title>
  <articleinfo>

    <authorgroup>
      <author>
	<firstname>Chris</firstname>
	<surname>McMahon</surname>
	<affiliation>
	  <address><email>christopher.mcmahon@gmail.com</email></address>
	</affiliation>
      </author>
    </authorgroup>

    <legalnotice id="trademarks" role="trademarks">
      &tm-attrib.freebsd;
      &tm-attrib.intel;
      &tm-attrib.microsoft;
      &tm-attrib.symantec;
      &tm-attrib.general;
    </legalnotice>

    <pubdate>$FreeBSD$</pubdate>

    <releaseinfo>$FreeBSD$</releaseinfo>

    <abstract><para>A slightly altered version of this paper was first
      published in the Pacific Northwest Software Quality Conference
      Proceedings, 2005.</para></abstract>

  </articleinfo>


  <sect1 id="overview">
    <title>Overview</title>

    <para>From late 2003 until early 2005, I was a tester in an
      all-&windows; environment.  Although unlikely on the face of it,
      &os; became a valuable test tool platform in that context.
      &os; contains useful and powerful applications for any tester
      in any environment.</para>

    <para>Unlike Linux, &os; is a single monolithic project, rather
      than a collection of disparate parts assembled into a
      distribution.  And the most attractive part of &os; for a
      software tester is the &os.ports;&mdash;a very large,
      managed set of software applications with a single, simple, and
      uniform installation procedure.</para>

    <para>This paper describes several software test tools from the
      &os.ports; that I used to test software and
      systems in an all-Windows environment.</para>

  </sect1>

  <sect1 id="challenge">
    <title>The Challenge</title>

    <para>Software testing environments are radically more complex
      than software development environments.  Interconnected systems
      to test, network entities, databases, and filesystems present
      challenges to testers that developers can, for the most part, mock
      out and essentially ignore.  Software testers need more tools,
      and more complex tools, than do software developers.</para>

    <para>On the other hand, software development tools are much more
      highly evolved than software testing tools.  There is no Eclipse,
      or IntelliJ, or even Visual Studio aimed at testing.  Testers
      struggle and scratch to find tools appropriate to their test
      environments and appropriate to their Systems Under Test (SUTs).</para>

    <para>Every test environment is different.  Finding appropriate
      test tools is challenging.  And when a tester has identified a
      set of useful tools, introducing them to the test environment
      and integrating them with the test environment is a challenge.</para>

  </sect1>

  <sect1 id="freebsd-solution">
    <title>The &os; Solution</title>

    <sect2 id="freebsd-intro">
      <title>Introduction</title>

      <para>The set of tools available with the &os; Operating
        System is amazing.  The &os; <ulink
        url="http://www.freebsd.org/ports">&ports;</ulink>
        contains more than thirteen thousand separate applications,
        all of which have a standard installation procedure and
        conform to a set of guidelines that make them reliable without
        the need to manage dependencies, appropriate versions, and all
        of the other problems that affect even the most well-managed
        Linux distribution or the various versions of Microsoft
        Windows.  The monolithic nature of &os; and the &os.ports;
	removes much of the trouble of integrating
        tools with the test environment, regardless of the OS under
        which the SUT runs.  &os; is a highly evolved server
        environment, and contains so many reliable applications, that
        every tester should consider adding a &os; machine (or
        several) to their test environment.</para>

      <para>Of course, all of the applications available in the
        &os.ports; will not be appropriate for any single
        test environment. Some of the obvious choices for software and
        systems testing are the six hundred or so system utilities,
        the more than one thousand network tools, and the fifty-odd
        benchmarking tools. Whether your test environment is Windows,
        UNIX, Linux, Mac OS, &os; itself, or some combination of
        any of them, &os; and the &os.ports; is a
        great place to look first.</para>

    </sect2>

    <sect2 id="freebsd-ports">
      <title>How To Use The Ports System</title>

      <para>Installing an application from the &os.ports; is a simple matter of: </para>

      <screen>&prompt.root; <userinput>cd /usr/ports/foo</userinput>
&prompt.root; <userinput>make install</userinput></screen>

      <para>and the system does the rest.  It reports build status and
        test status, and installs all the relevant documentation as
        well.  This aspect of &os; is very attractive to a tester,
        who typically is pressed for time!</para>
    </sect2>

    <sect2 id="freebsd-testing">
      <title>&os; For Testing</title>

      <para>The test environment should be more stable than the SUT.
        Once the tester decides to use the tools available on &os;,
        &os;'s long record of reliability makes it an easy choice
        for a test tools platform.</para>

      <para>My own introduction to &os; occurred when I was hired
        by a major vendor of large-scale network security video
        services to be their network-testing person in an all-Windows
        environment.  My first assignment was to replace the obsolete,
        buggy, disk imaging system.  I chose to do that with an Open
        Source disk imaging system called <ulink
	  url="http://www.cs.utah.edu/flux/papers/frisbee-usenix03-base.html">&frisbee;</ulink>
        which was implemented originally on &os;.  I built the
        system&mdash;a feature-for-feature replacement for an expensive
        proprietary system&mdash;but we never actually used it in our
        production system.</para>

      <para>In the meantime, I had discovered the &os.ports;
        and started to use some of those tools for testing;
        and I had discovered the power of disk imaging with &frisbee;,
        especially for smoke testing and installation testing; and
        &os; became a permanent part of my test lab.  The test lab
        I built, and the &os; systems I created still exist, and
        still provide value to the testers there.</para>

    </sect2>

    <sect2 id="freebsd-collab">
      <title>&os; For Collaboration:  Twiki</title>

      <para>A wiki is a simple set of web pages to allow many users to
        share information and collaborate on any sort of documents.
        <ulink url="http://twiki.org">Twiki</ulink> is a wiki fine-tuned for
        document management.  It has built-in version control and
        security implemented at the user/password level.  I used it
        for requirements management and traceability.</para>

      <para>The company I worked for managed their requirements in a
        proprietary tool with a truly byzantine hierarchical
        structure.  Although the tool was fine for generating
        requirements, it was horrible for testing.  Also
        unfortunately, the tool supported very few export mechanisms.
        From the proprietary tool, I had to:</para>

      <itemizedlist>
        <listitem><simpara>Export to Access</simpara></listitem>
	<listitem><simpara>Export from Access to Excel</simpara></listitem>
	<listitem><simpara>Export from Excel to a delimited file</simpara></listitem>
	<listitem><simpara>Import the delimited file to the wiki by means of a Perl script</simpara></listitem>
      </itemizedlist>

      <para>And in Twiki I was able to implement version control,
        traceability, and test coverage, all features missing from the
        proprietary tool.  This gave me a remarkable ability to trace
        requirements coverage with my tests.  Having imported a
        complex hierarchy of sets of requirements, I could use Twiki's
        simple markup features to show each requirement as tested,
        passed, or failed.  I could attach test documentation to wiki
        pages, and since I had already scripted the process, it was
        easy to update the requirements as they changed.  And Twiki
        itself handled version control for such updates.</para>

      <para>As with all of the examples in this paper, installing
        Twiki on &os; is fairly simple.  It takes just a few
        minutes on a &os; system. However, if you want to use Twiki
        on a Microsoft Windows platform, I strongly suggest you read
        the Twiki documentation extremely carefully.  I know someone
        who installed Twiki on Windows, and it took him several days.
        Twiki on Windows requires not only knowledge of Windows, but
        also deep knowledge of Cygwin and Perl.</para>

      <para>Furthermore, at one point in the project I had to migrate
        my wiki from a machine running &os;&nbsp;4.8 to one running
        &os;&nbsp;5.3.  The migration consisted merely of installing
        Twiki on &os;&nbsp;5.3; using <command>tar</command> on the &os;&nbsp;4.8
        machine to gather all of the Twiki data files specific to my
        testing; FTPing the gathered files to the new &os;&nbsp;5.3
        machine; and untarring the file. The complete set of Twiki
        documents migrated with no issues or problems at all.  That is
        the power of a unified system like &os;.</para>
    </sect2>

    <sect2 id="freebsd-frisbee">
      <title>&os; For Disk Imaging: Frisbee</title>

      <para>A disk imaging system is a mechanism for saving and
        restoring all of the data on a physical disk.  The most
        popular commercial system for doing this is probably the
        product &ghost;&trade; from Symantec.</para>

      <para>The &frisbee; enterprise disk imaging system mentioned above
        had a lot of features I never implemented in the test lab.
        Using &frisbee; and an Open Source tool called <application>PXELINUX</application>, I was
        able to:</para>

      <itemizedlist>
        <listitem><simpara>Boot the Windows client machines from the network</simpara></listitem>
	<listitem><simpara>Make an image of the client</simpara></listitem>
	<listitem><simpara>Update the BIOS on the client</simpara></listitem>
	<listitem><simpara>Lay down an existing image on the client machine</simpara></listitem>
	<listitem><simpara>Make a set of restore CDs for the client</simpara></listitem>
      </itemizedlist>

      <para>In the test lab, I only needed to boot from the &frisbee;&nbsp;CD,
	make an image, or lay down an image on the client machine.
        Both &frisbee; and proprietary imaging systems allow the user to
        image individual drives on the client, but I never had a need
        to do this.</para>

      <para>Installation testing was a large part of my duties at the
        company where I used &os;.  To do this testing, I would
        typically use &frisbee; to make an image of a machine containing
        only a Windows OS, install the SUT, and run a smoke test.  The
        smoke test typically left the test machine in a very bad
        state.  But instead of having to painstakingly clean up the
        mess left by the failed installation, I simply re-imaged the
        machine in question with the bare OS image and started over.
        A typical re-image containing only the Windows OS and a few
        test tools took less than three minutes.  Using &frisbee;, we
        could run smoke tests on about six builds per day; before
        &frisbee;, we could run smoke tests on about three builds per
        week.</para>

      <para>Of course, &ghost; or other proprietary tools also image
        machines quickly under these circumstances: once you buy the
        tool, license the software, install it on an appropriate
        server, and configure it properly.  I prefer &frisbee; to &ghost;
        because: &frisbee; is marginally faster; &frisbee; is very easy to
        install on &os;; and &frisbee; is very efficient.  Adding a
        couple of small Perl scripts to the normal &frisbee;
        distribution gave me an imaging environment tailored for the
        test lab.</para>

      <para>I also used &frisbee; to preserve the state of a machine
        after I had uncovered particularly complex defects.  That is,
        if it took a large effort (many steps and/or a long duration
        of time) to demonstrate a defect, I could make an image of the
        machine at the point at which the defect was visible, and
        restore the image at will to demonstrate the defect to
        developers or managers.</para>
    </sect2>

    <sect2 id="freebsd-nessus">
      <title>&os; Security Testing:  &nessus;</title>

      <para>Whenever you have more than one entity on a network, and
        whenever you expose a server to the wider Internet, security
        of the machine itself is always a concern.  <ulink
        url="http://www.nessus.org">&nessus;</ulink> is an Open Source
        remote vulnerability scanner for security and penetration
        testing that consistently is rated among the top products of
        its type throughout the security industry.</para>

      <para>&nessus; probes a remote machine over the network for
        security vulnerabilities.  It does a port scan, finds which
        ports are open, and investigates the software that has those
        ports open for a huge number of security risks, for all major
        OSs.  It generates detailed reports in a number of formats
        that anyone can understand.  The number of security probes
        available in the default installation of &nessus; is very large,
        but sophisticated security and penetration testers take
        advantage of NASL, the &nessus; Attack Scripting Language, to
        craft their own attacks using &nessus;' available features.</para>

      <para>Of interest is that, while &nessus; is a free download for
        UNIX-like systems (and is available in the &ports; of
        &os;), it is available on Windows only as a commercial
        product from a company called <emphasis>Tenable</emphasis>.  The Tenable product is
        <application>NeWT</application>, <quote>Nessus on Windows Technology</quote>.</para>
    </sect2>

    <sect2 id="freebsd-network">
      <title>&os; Network Tools</title>

      <para>&os; is most widely used as a robust server platform.
        It follows, then, that tools related to network analysis and
        performance will be highly evolved on &os;.  Here is a
        brief description of network diagnostic tools that I found
        invaluable in testing in a networked environment.</para>

      <para>From the name, one would assume that <ulink
        url="http://www.ntop.org">ntop</ulink> emulates the functions of
        the UNIX &man.top.1; command, but for the network
        rather than for the local machine.  Perhaps the first version
        did; currently, <application>ntop</application> is capable of providing detailed
        information about a huge number of hosts and their status and
        activities on the network.</para>

      <para>For testing, two features I found very powerful: at a high
        level, <application>ntop</application> shows the amount of network
        traffic on the entire network segment minute-by-minute,
        hour-by-hour, and day-by-day in a graphical format.  Also,
        <application>ntop</application> shows information about recent connections between
        individual hosts on the network.</para>

      <para>It is easy to see traffic trends on the network as they
        are occurring; also, if something anomalous appears, <application>ntop</application>
        records detailed information about network connections between
        hosts, including the ports over which the connection happened.
        This was critically important when analyzing software
        issues. If <application>ntop</application> showed a period of time for which traffic was
        particularly high, I would find out which host was generating
        the traffic. I would examine the software running on that
        host, over that port. Often it was a new build with a
        bug.</para>

      <para><ulink url="http://ettercap.sourceforge.net">Ettercap</ulink> is
        a tool for ARP poisoning which can also decipher passwords on
        the fly and corrupt IP traffic by means of a Man In The Middle
        (MITM) attack. However, I used <application>Ettercap</application> as a performance tool.
        In my test labs, all of my &os; machines ran on discarded
        hardware, Pentium II processors.  I found that when I used
        <application>Ettercap</application> to sniff traffic between two hosts, the lack of
        processing power caused <application>Ettercap</application> on the slow MITM machine to
        start dropping packets, making it look to the client machine
        in the SUT as if there was interference or other trouble on
        the network.  And by varying the load on the &os; machine,
        I could in fact control the number of packets being dropped:
        running <application>Ettercap</application> and the UNIX <command>yes</command> utility
        caused 100% packet loss.</para>

      <para>This was my most creative use of a &os; tool for
        testing.  In a more straightforward application, any time a
        tester needs to eavesdrop on traffic between two hosts on a
        network, <application>Ettercap</application> is an excellent choice because of its power
        and ease of use.</para>

      <para>Perl gets a special mention here because Perl's network
        utilities are outstandingly good compared to other
        languages. Perl <literal>Net::*</literal> modules and
        <literal>IO::Socket::*</literal> modules are robust and
        powerful&mdash;but they often fail to compile on Windows.  It is
        the ease of use of Perl's network utilities on &os; that
        gets Perl the mention in this section.</para>

      <para>I use Perl's network utilities to impersonate network
        clients and servers for test purposes.  On one occasion, I was
        required to test software that was a client to an interface on
        the <emphasis>New York Stock Exchange</emphasis>.  Unfortunately, the NYSE test
        server was down about nine days out of ten.  I wrote a little
        network server in Perl to emulate simple functions of the NYSE
        server in order to test the client software.</para>

      <para>On another occasion, I had to test some functions on a web
        server.  It was not difficult to write a simple Perl HTTP
        client to validate that the server was functioning properly.</para>

      <para>I have also used Perl to validate the output from a server
        sending to a multicast address.  I wrote a simple Perl
        multicast client on &os; to monitor the traffic on a
        multicast address.  Lincoln Stein's excellent
        <literal>IO::Socket::Multicast</literal> module made it easy.
        (Note: I never got <literal>IO::Socket::Multicast</literal> to
        compile on Windows.  I tried it on Windows 2000 and on Windows
        XP.)</para>
    </sect2>
  </sect1>

  <sect1 id="conclusion">
    <title>Conclusion</title>

    <para>I used tools from the &os.ports; in four
      areas: in the network, where the operating system has very
      little impact on how software behaves; for remote security
      testing and performance testing in order to manipulate remote
      machines over the network, regardless of the operating system;
      for disk imaging of Windows, Linux, and &os; machines; and on
      the webserver, where Twiki was my collaboration tool of choice.</para>

    <para>Because the installation procedure for all of these tools is
      standard, I spent relatively little time installing and
      configuring my tools.  Because all the tools were hosted on a
      single platform, I had all of my configuration and diagnostic
      information located in just a few places.  I kept all of my
      potentially dangerous security tools on a single machine, which
      made my presence on the network tolerable to the company's
      network management staff.  And the compatibility between &os;
      versions made it fairly simple to upgrade and to manage multiple
      &os; machines.  And of course, I could rely on the
      correctness of my test results, because the system itself is so
      reliable.</para>

    <para>I have tried using Linux in a similar way, but my experience
      is that package management quickly becomes tedious if not
      overwhelming.  The &os.ports; handled that for me.
      And many of these tools are simply not available on Microsoft
      Windows.  And when they (or their equivalents) are available,
      their cost, both financial and in terms of overhead, was simply
      too high.</para>

    <para>&os;'s simple installation procedures and robust &ports;
      makes it easy to experiment with the huge number of
      tools available.  I often find myself browsing the &ports;
      looking for interesting applications to install, just
      to see how they work.  (I found <application>Ettercap</application> by browsing the &ports;,
      a tool that became very useful very quickly.)  It
      became clear that the more tools I used on &os;, the more
      economical became the management of those tools.</para>

    <para>The next time you need to reach into your toolbox for some
      sophisticated, reliable, and powerful testing tools, I hope you
      find them in &os;.</para>
  </sect1>

<!--
<sect1>
  <title>Biography</title>

  <para>Chris McMahon has been for nearly a decade a tester of library
    software, telecom, stock trading, and networked video systems.  He
    has published a number of articles aimed at technical testers, but
    his true career is at being an enthusiastic beginner.  Chris works
    for ThoughtWorks, Inc. as a tester, and can be reached at
    christopher.mcmahon@gmail.com.</para>
</sect1>
-->

</article>