path: root/en/handbook/README

This will be the location of the DocBook version of the FreeBSD Handbook,
which will eventually obsolete the version currently in doc/handbook/.

Interested parties should examine

  <URL:http://www.nothing-going-on.demon.co.uk/FreeBSD/docbook-migration.html>

and get in touch with Nik Clayton (either to nik@FreeBSD.ORG or via the
FreeBSD-doc mailing list) if they have specific questions.

All the scripts mentioned here can also be downloaded by doing to

  <URL:http://www.freebsd.org/~nik/script_name>

for example,

  <URL:http://www.freebsd.org/~nik/entity-cdata.pl>

  
   ------------------------------------------------------------------------
    The Handbook is midway through the conversion process. It will almost
		certainly not convert to other formats cleanly
   ------------------------------------------------------------------------


				   Actions

  This list explains what's been done so far, so the Japanese team can
  track my changes. All actions took place on freefall.

  1. Initial conversion to DocBook

     Checked out a copy of the doc repository to ~/cvs/. Then used 2 scripts
     to convert the handbook to its initial DocBook format. The 2 scripts are
     2docbook.sh and entity-cdata.pl, both of which can be found in ~nik/bin/.

     2docbook.sh calls entity-cdata.pl as necessary.

         % cd ~/cvs/doc/handbook
	 % 2docbook.sh

     This created handbook-db.sgml in ~/cvs/doc/handbook. This file contains
     syntactically valid (but quite ugly) SGML. This file was then moved to
     the doc/en/handbook directory and renamed to handbook.sgml. The
     conversion process left a few spurious changes in the old handbook files
     which I don't want to commit, so I removed them and updated the
     repository.

     The new file was then committed.

         % mv handbook-db.sgml ~/cvs/doc/en/handbook/handbook.sgml
	 % rm *.sgml
	 % cvs update
	 % cd ~/cvs/doc/en/handbook
	 % cvs add handbook.sgml
	 % cvs commit

  2. handbook.sgml was loaded into XEmacs 20.30 (straight from the ports
     collection) and sgml-mode was turned on. My .emacs file contains the
     following hook:

         (add-hook 'sgml-mode-hook
              (function
               (lambda()
                 (setq sgml-omittag nil)
                 (setq sgml-indent-data t))))

     This configures psgml to not omit any tags that the DTD lists as
     omittable, and to indent data in the same way that markup is indented.

     The following function was pasted into the *scratch* buffer, and then
     "M-x eval-current-buffer" was run.

         (defun sgml-indent-buffer
            "Indents the current buffer, one line at a time"
            (interactive "*")
            (save-excursion
              (goto-char (point-min))
              (while (= (forward-line 1) 0)
                (sgml-indent-or-tab))))

     In the handbook.sgml buffer, the point was placed on the first character
     of the first line, and "M-x sgml-indent-buffer" was run.

     The changes were then committed.

  3. Refilled the Handbook -- this rewraps the lines as necessary. This was
     done by placing the point on the first <book> tag, and running
     "M-x sgml-fill-element".

     This takes about 10 minutes to run.

     It also reformats some sections that should not be reformatted, including
     examples of text on the screen, PGP key blocks and so on. They will be
     fixed in a later commit.

  4. Removed spurious markup. The conversion process has left a lot of

         <para></para>

     entries in the handbook, and they need to be removed. There are a
     number of places this happens, and the rules are slightly different
     each time.

     For example,

     ====================================================================
       Original markup                      Changed markup
       --------------------------------------------------------------

       <listitem>                            <listitem>
         <para></para>                         <para>A real paragraph</para>

	 <para>A real paragraph</para>

       --------------------------------------------------------------

         <para>A real paragraph</para>         <para>A real paragraph</para>
	                                     </listitem>
         <para></para> 
	 
       </listitem>       

       --------------------------------------------------------------

         <para>A real paragraph</para>       <para>A real paragraph</para>

	 <para></para>                       <para>Another paragraph</para>

	 <para>Another paragraph</para>
     ====================================================================

     Notice the last example. It's not enough to simply put together a
     regexp that matches (all whitespace)<para></para>(allwhitespace)
     and removes it, since that would leave you with

       <para>A realparagraph</para>
       <para>Another paragraph</para>

     In the end I got bored of trying to write this using Emacs regexps,
     and knocked together a quick Perl script to do it. It's ~nik/bin/para.pl
     on freefall.

  5. Got halfway through looking for filenames, and marking them up as such.

     There are a lot ( :-( ) of filenames in the Handbook. The conversion
     process did a pretty good job of marking them as <filename>...</filename>
     but it wasn't perfect.

     I'm halfway through (line 16704) going through the Handbook, eyeballing
     each line and changing things like <emphasis remap="tt">...</emphasis>
     to <filename>...</filename> where appropriate.

     The remainder will follow tomorrow evening.

  6. Finished the first sweep marking up filenames.

     If it looked like a filename (but wasn't a command for the user to type
     in) it's been marked up with <filename>...</filename>.

     If it had already been marked up as <filename>...</filename> but wasn't
     a filename, the markup was changed to <emphasis remap="tt">...</emphasis>

     PSGML and Xemacs are very useful for this, using "C-c =" to change
     existing markup.
     
     Synchronising with changes 5 and 6 will involve examining the diffs
     and changing by hand I'm afraid. It could not be automated.

  7. Start replacing `` and '' with <quote> and </quote>. Don't change        
     things indiscriminately, but look at the context to see if the change is
     appropriate. There are still many `` and '' occurences which should be
     changed to some other element.

     This was done using a regexp search/replace, looking for the regexp

	 ``\([^']\)''

     and replacing with 

	 <quote>\1</quote>

     Not all the `` '' pairs were changed, since in some cases they delimit
     filenames, options and so on.

  8. As with change 7, but replace with <command> ... </command> as
     necessary.

  9. Remove the `` and '' from options. 

     ``<option>...</option>'' becomes <option>...</option>

 10. Converted appropriate occurences of 

         <emphasis remap=tt>...</emphasis>

     to
   
         <filename>...</filename>

 11. As above, but changing to <command>...</command>. Modified the Emacs
     regexp slightly to search for 

         <emphasis[ \n\t]+remap=tt>\([^<]+\)</emphasis>

     which matches elements spread over two lines.

 12. Looked for explanatory notes in the text (typically prefixed by "note",
     "Note:" or "<para><blockquote><para><emphasis role=bf>Note:</emphasis>"
     and marked them up as 'note' elements.

     This change involves markup changes *and* text changes. This is because
     text like

         <para>Note: The foo file is only used once, and can be deleted.</para>

     became

         <note>
           <para>The foo file is only used once, and can be deleted.</para>
         </note>
 
 13. Look for text marked up as an acronym and alter as necessary. The
     automatic conversion tended to mark any string of upper case letters
     as acronyms, which is not always right.

     The difference between an <acronym> and <abbrev> is subtle -- in a 
     nutshell, an acronym is pronounceble, an abbreviation isn't.

 14. Another sweep for "`" and "``" (and their closing equivalents), 
     replacing them with the right markup (since most of the time they're
     used to 'delimit' filenames or options from the surrounding text.

     The only quotes left now are either around items for which I'm not 100%
     sure which element to use, or in literal blocks as part of commands the
     user types in.

 15. Look for double quotes not used in attributes and alter to the 
     appropriate markup (or remove as necessary). A useful Emacs regexp
     when doing the search replace is

         \([^=]\)"\([^ \t\n]+[^"]+\)"\([^>]\)

     and replace with

         \1<quote>\2</quote>\3

     or whatever the replacement element is.

     Converted '"' into <quote>, <literal>, <command>, <application>,
     <filename>, <emphasis>, <option> or removed it as neccessary.

 16. A general cleanup to get it to validate. The original conversion 
     process left some <sect?>'s with just a title, which is invalid,
     they must contain a <para> or similar element.

     Also fixed a couple of typos in the tags. The document should now 
     validate, save for the undefined external entitites.

 17. Created a new FreeBSD Doc. Project DTD in the ../../sgml directory.
     Changed the declaration at the top of the handbook to use this new
     DTD.

 18. Yet more things that should be filenames marked up as such.

 19. Use the new <hostid> element to mark up hostnames, IP addresses and
     such. The markup choice is as follows.

        <hostid>...</hostid> is a simple hostname.
        <hostid role="ipaddr">...</hostid> is an IP address.
	<hostid role="domainname">...</hostid> is a domain name.
	<hostid role="fqdn">...</hostid> is a fully qualified domain name.
	<hostid role="netmask">...</hostid> is a netmask.
        <hostid role="mac">...</hostid> is a network card MAC address.

     These might migrate to being separate elements in the future. However,
     if they do then changing the markup can be done automatically.

 20. Convert <emphasis remap=it>...</emphasis> to plain <emphasis> in some
     cases. I'm pretty certain that all the <emphasis>...</emphasis>
     markup is correct now, which makes searching for markup that does
     need changing much easier. 

 21. Replace the last few occurences of curly quoted items (`` and '') 
     with the right markup.

 22. Almost the last lot. I missed a diff I'd done at home. There's a
     section in the handbook that talks about kernel options, where the
     quoted options are quoted with `` and ''. Fix them so that standard
     double quotes are used (so they can cut-n-pasted).
 
 23. Start working on <emphasis remap=bf>...</emphasis>

     Convert the first lot to <command>...</command>

 24. Fixed manual page references to use the right markup, which is

       <citerefentry>
         <refentrytitle>page_name</refentrytitle>
         <manvolnum>number</manvolnum>
       </citerefentry>

     Did this with a regexp search for

       \([a-z-_\.]+\)(\([1-9]\))

     and replacing with

       <citerefentry><refentrytitle>\1</refentrytitle><manvolnum>\2</manvolnum>

     Since most of the page references had <command>, <emphasis>, or
     <ulink> elements wrapped around them, you then have to sweep through the
     file looking for "><cite" and using C-c C-k to kill the markup 
     immediately before and after.

 25. <emphasis remap=..>...</emphasis> -> <literal>...</literal>

 26. <emphasis remap=..>...</emphasis> -> <makevar>...</makevar>

 27. <emphasis remap=..>...</emphasis> -> <maketarget>...</maketarget>
 
 28. Fix up some uses of <screen> and the use of <emphasis> elements within
     and near it. Most of the time this consisted of replacing the <emphasis>
     with <replaceable> or <userinput>.

 29. Fixed up more references to manpages that used <ulink> to use 
     <citerefentry>. These were missed at step 24 because they didn't 
     include a section number. No references to man.cgi now exist in  
     handbook.sgml.

 30. Create two entities, prompt.root and prompt.user. Use these anywhere
     the OS prompt is displayed, depending on whether the user should be
     a normal user or root.

     Also markup other prompts (e.g., the DOS prompt C:\> that occurs in
     some places) as <prompt>s.

 31. Reviewing the use of <informalexample> and <screen>.

     In some cases <informalexample> wasn't appropriate, and the markup was
     changed to <programlisting> or other.

     In some cases there were spurious <para> elements before and after the
     <informalexample>. These were removed.

     Reformatted text within <screen> elements because the whitespace *is*
     significant.

     Added <prompt> and <userinput> elements within <screen> where necessary.

     If I spotted inappropriate use of markup within the immediate vicinity
     of the <informalexample> elements then I fixed that (mostly the use of
     <emphasis remap="...">).

     This is part one of these changes -- there's a load of them, and this
     goes up to line 11,284 or thereabouts, roughly one third of the way
     through.

 32. Continuing the work from the previous commit. This takes us up to
     the beginning of Chapter 16, "The Cutting Edge".

 33. Finished sweep. If it's white space sensitive (examples, program 
     listings, PGP signatures...) the white space is now correct. I may
     have missed one or two on the way, I'll catch them later.

 34. Removed repeated spaces from the end of stops,  . , ! ? : ;

     Some parts of the handbook had single spaces after stops, some had double
     or triple. While the typographical convention for monospaced fonts may
     be to use double spaces after them, that doesn't apply here. TeX will
     ignore them, as will HTML. If we need for a plain text version of the 
     Handbook then the stylesheet / conversion mechanism can insert them
     as necessary.

     Searching for

         _\([;:!\.\?,]\)  +_

     in Emacs and replacing with

         _\1 _

     (ignore the '_', they're just to delineate the regexps) does the job
     quite nicely. However, you can't do this everywhere, since some of the
     double spaces might be in program listings or other literal sections
     (e.g., the BSD Copyright), so you need to sit and bounce on the 'y' or
     'n' key as appropriate for each occurence of a stop.

 35. Some paragraphs have leading space(s). E.g.,

        <para>  There is some leading space here.</para>

     Get rid of it, doing an emacs search/replace for

        <para> +\([^ ]\)

     and replacing with

        <para>\1

     This can be done globally.

 36. A lot of </para> tags have leading whitespace before them. Remove it. Do
     this (in Emacs) by searching for 

        \s-+</para>

     and replacing with

        </para>

     Do this for all occurences *except* where the element immediately before
     the </para> is one of <itemizedlist>, <orderedlist>, <variablelist>,
     <procedure>. The <para>...</para> wrapping these elements is mostly
     redundant, and will be removed later.

 37. With the agreement of the Japanese team, a change in the way I'm doing
     things.

     I'm now working through from the beginning of the handbook to end,
     correcting as I go. I'll commit in chunks of 5,000 lines (or 
     thereabouts).

     Most of the changes fall into the following categories.

       * <emphasis remap=bf> --> <emphasis>

       * Spurious <para>s around <*list>s deleted (but not reformatted)

         "C-c -" in Emacs SGML mode (when the point is on an element starting
         or end tags) will delete that element's starting or end tags.

       * Marked smileys with <!-- smiley --> for possible future deletion

       * Deleting <emphasis>, around

         <term><emphasis>...</emphasis></term> -> <term>...</term>

       * Fine tuning markup choices in some cases

         - <filename>C:</filename>  ->  <devicename>C:</devicename>

       * Extra <note>s here and there.

       * Some <*list>s to <procedure> (and <listitem>s to <step>)

       * ASCII emphasis converted to <emphasis> 

         i.e.,   do it like *this*  ->  do it like <emphasis>this</emphasis>

       * <symbol> -> <replaceable>

     There are very few whitespace changes, although a few have probably 
     cropped up. The vast majority of the whitespace changes will happen in
     one megacommit, hopefully some time next week. 

 38. As above, to line 11490.

 39. . . . to line 15126 . . . 

 40. . . . to line 20370 . . .

 41. . . . to line 24997 . . .

 42. Brief interruption, small changes to keep it validating.

 43. . . . to line 30118 . . .

 44. . . . to line 35973 . . .

 45. . . . to end of file!

 46. <emphasis remap=..> -> <emphasis>
     <literal remap=..>  -> <literal>
     <command remap=..>  -> <command>

     Or deleted <emphasis ..> altogether in some cases.

     More redundant <para>..</para>'s removed.

 47. Removed white space after <title> and before </title>. Use two search and
     replaces

         \s-+</title>  ->  </title>
         <title>\s-+   ->  <title>

 48. Use the correct ISO entities for dashes. According to a TeX manual I have
     kicking around here,

      daughter-in-law, X-rated    = hyphen      =  -
      pages 13--67                = en-dash     = &ndash;
      yes---or no?                = em-dash     = &mdash;
      0, 1, and -1                = minus sign  = &minus;

 49. Step 1. Find <ulink url="mailto:...">...</ulink>  and change the <ulink>
     to <email>.

     Can't do this globally. Some of the links are odd (i.e,. the link
     is not their e-mail address but is their name, eg

       <ulink url="mailto:nik@freebsd.org">Nik Clayton</ulink>

     which would turn to

       <email>Nik Clayton</email>

     which isn't very useful. Ignore these ones, and do the others.
     (i.e., the ones that look like

       <ulink url="mailto:nik@freebsd.org">nik@freebsd.org</ulink>

     This Emacs regexp does the job.

         Search for: <ulink\s-+url="mailto[^>]+>\([^<]+\)</ulink>  
       Replace with: <email>\1</email>

     Step 2. A lot of the <email>...</email> sets will have '<' and '>' 
     embedded in them (as entities). These can be removed, since the stylesheet
     will add them;

         Search for: <email>&lt;\([^&]+\)&gt;</email>
       Replace with: <email>\1</email>

     Step 3. The trick now is to turn

       <ulink url="mailto:nik@freebsd.org">Nik Clayton</ulink>

     into

       Nik Clayton <email>nik@freebsd.org</email>

     This step could (possibly) have been done first, and then steps
     1 and 2 could be done globally. I haven't done this because of
     concerns about the ordering of names within languages. This 
     transformation is fairly simple in English, I've no idea what 
     it's like in Japanese.

        Search for: <ulink\s-+url="mailto:\([^"]+\)">\([^<]+\)</ulink>
      Replace with: \2 <email>\1</email>

     Step 4. Remove leading and trailing spaces that may have slipped in

        Search for: <email>\s-+
      Replace with: <email>

        Search for: \s-+</email>
      Replace with: </email>

 50. <abbrev> -> <acronym> in some cases.

 51. Fixup erroneous or extraneous <para>...</para> elements.

 52. <foo
       id="bar">
       ...

     changed to

     <foo id="bar">
       ...

     Before people complain that "Hang on, now you can't find out what the 
     allocated ID values are with a simple 'grep'" I'll say that's not a 
     problem. I plan to introduce a target in the Makefile (probably 
     something like 'handbook.id' which will automatically generate this
     list doing a proper SGML parse.

 53. Where kernel options ("options INET" for example) are listed, wrap them
     in <literal>...</literal>.