Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8)

and geom_uncompress(4):

1. mkuzip(8):

 - Proper support for eliminating all-zero blocks when compressing an
   image. This feature is already supported by the geom_uzip(4) module
   and CLOOP format in general, so it's just a matter of making mkuzip(8)
   match. It should be noted, however that this feature while it sounds
   great, results in very slight improvement in the overall compression
   ratio, since compressing default 16k all-zero block produces only 39
   bytes compressed output block, which is 99.8% compression ratio. With
   typical average compression ratio of amd64 binaries and data being
   around 60-70% the difference between 99.8% and 100.0% is not that
   great further diluted by the ratio of number of zero blocks in the
   uncompressed image to the overall number of blocks being less than
   0.5 (typically). However, this may be important from performance
   standpoint, so that kernel are not spinning its wheels decompressing
   those empty blocks every time this zero region is read. It could also
   be important when you create huge image mostly filled with zero
   blocks for testing purposes.

 - New feature allowing to de-duplicate output image. It turns out that
   if you twist CLOOP format a bit you can do that as well. And unlike
   zero-blocks elimination, this gives a noticeable improvement in the
   overall compression ratio, reducing output image by something like
   3-4% on my test UFS2 3GB image consisting of full FreeBSD base system
   plus some of the packages (openjdk, apache etc), about 2.3GB worth of
   file data (800+MB compressed). The only caveat is that images created
   with this feature "on" would not work on older versions of FeeBSDxi
   kernel, hence it's turned off by default.

 - provide options to control both features and document them in manual
   page.

 - merge in all relevant LZMA compression support from the mkulzma(8),
   add new option to select between both.

 - switch license from ad-hoc beerware into standard 2-clause BSD.

2. geom_uzip(4):

 - implement support for de-duplicated images;

 - optimize some code paths to handle "all-zero" blocks without reading
   any compressed data;

 - beef up manual page to explain that geom_uzip(4) is not limited only
   to md(4) images. The compressed data can be written to the block
   device and accessed directly via magic of GEOM(4) and devfs(4),
   including to mount root fs from a compressed drive.

 - convert debug log code from being compiled in conditionally into
   being present all the time and provide two sysctls to turn it on or
   off. Due to intended use of the module, it can be used in
   environments where there may not be a luxury to put new kernel with
   debug code enabled. Having those options handy allows debug issues
   without as much problem by just having access to serial console or
   network shell access to a box/appliance. The resulting additional
   CPU cycles are just few int comparisons and branches, and those are
   minuscule when compared to data decompression which is the main
   feature of the module.

 - hopefully improve robustness and resiliency of the geom_uzip(4) by
   performing some of the data validation / range checking on the TOC
   entries and rejecting to attach to an image if those checks fail.

 - merge in all relevant LZMA decompression support from the
   geom_uncompress(4), enable automatically when appropriate format is
   indicated in the header.

 - move compilation work into its own worker thread so that it does not
   clog g_up. This allows multiple instances work in parallel utilizing
   smp cores.

 - document new knobs in the manual page.

Reviewed by:		adrian
MFC after:		1 month
Differential Revision:	https://reviews.freebsd.org/D5333

1 files changed, 82 insertions, 8 deletions

diff --git a/share/man/man4/geom_uzip.4 b/share/man/man4/geom_uzip.4
index 0d36322121e6..4dbab524cca3 100644
--- a/share/man/man4/geom_uzip.4
+++ b/share/man/man4/geom_uzip.4
@@ -30,7 +30,7 @@
 .Os
 .Sh NAME
 .Nm geom_uzip
-.Nd "GEOM based compressed disk images"
+.Nd "GEOM based compressed disk images and partitions"
 .Sh SYNOPSIS
 To compile this driver into the kernel,
 place the following line in your
@@ -51,7 +51,7 @@ The
 framework provides support for compressed read-only
 disk images.
 This allows significant storage savings at the expense of
-a little CPU time on each read.
+a some CPU time on each read.
 Data written in the GEOM label area allows
 .Nm
 to detect compressed images which have been created with
@@ -63,17 +63,53 @@ creates a unique
 .Pa md#.uzip
 device for each image.
 .Pp
+.Nm
+is not limited to supporting only
+.Xr md 4
+images.
+The image can also reside on a block device.
+.Pq For example, a disk, USB flash drive,  DVD-ROM, etc.
+The appropriate device node will appear with the
+.Pa .uzip
+suffix.
+.Bd -literal -offset indent
+# gpart show da0
+=>      0  7833600  da0  BSD  (3.7G)
+        0  2097152    1  freebsd-ufs  (1.0G)
+  2097152  5736448       - free -  (2.7G)
+# gpart add -t freebsd-ufs -s 1G da0
+da0b added
+# dd if=/tmp/20160217_dcomp_zcomp.uzip bs=256k of=/dev/da0b
+3190+1 records in
+3190+1 records out
+836331008 bytes transferred in 111.021489 secs (7533055 bytes/sec)
+# fsck -t ffs /dev/da0b.uzip
+** /dev/da0b.uzip (NO WRITE)
+** Last Mounted on /mnt
+** Phase 1 - Check Blocks and Sizes
+** Phase 2 - Check Pathnames
+** Phase 3 - Check Connectivity
+** Phase 4 - Check Reference Counts
+** Phase 5 - Check Cyl groups
+97455 files, 604242 used, 184741 free (2349 frags, 22799 blocks,
+   0.3% fragmentation)
+# mount -o ro /dev/da0b.uzip /mnt
+# df /dev/da0b.uzip
+Filesystem     1K-blocks    Used  Avail Capacity  Mounted on
+/dev/da0b.uzip   3155932 2416968 738964    77%    /mnt
+.Ed
+.Pp
 The
 .Nm
-device is subsequently used by the
+device is subsequently used by
 .Fx
-kernel to access the disk images.
+kernel to access the uncompressed data.
 The
 .Nm
 driver does not allow write operations to the underlying disk image.
 To check which
-.Xr md 4
-devices match a given
+.Dq providers
+match a given
 .Nm
 device:
 .Bd -literal -offset indent
@@ -83,13 +119,44 @@ Providers:
 1. Name: md1.uzip
    Mediasize: 22003712 (21M)
    Sectorsize: 512
-   Mode: r1w0e1
 Consumers:
 1. Name: md1
    Mediasize: 9563648 (9.1M)
    Sectorsize: 512
-   Mode: r1w0e1
+
+Geom name: da0b.uzip
+Providers:
+1. Name: da0b.uzip
+   Mediasize: 3355443200 (3.1G)
+   Sectorsize: 512
+Consumers:
+1. Name: da0b
+   Mediasize: 1073741824 (1.0G)
+   Sectorsize: 512
 .Ed
+.Pp
+.Nm
+allows mounting the root file system from a compressed disk partition by
+setting the
+.Dv vfs.root.mountfrom
+tunable.
+See
+.Xr loader.conf 5
+for details.
+.Sh DIAGNOSTICS
+Several flags are provided for tracing
+.Nm
+I/O operations and TOC parsing via the following sysctls.
+.Bl -tag -width indent
+.It Va kern.geom.uzip.debug
+Log level.
+Zero disables logging.
+Higher values enable more verbose debug logging for
+.Nm .
+Supported levels are from 0 (no logging) to 4 (maximum amount of logging).
+.It Va kern.geom.uzip.debug_block
+Log operations involving compressed cluster number.
+.El
 .Sh SEE ALSO
 .Xr GEOM 4 ,
 .Xr md 4 ,
@@ -101,5 +168,12 @@ The
 .Nm
 driver was written by
 .An Max Khon Aq Mt fjoe@FreeBSD.org .
+The block de-duplication code as well as some
+.Nm
+driver optimizations have been contributed by
+.An Maxim Sobolev Aq Mt sobomax@FreeBSD.org .
+The LZMA decompression support and CLOOP 3.0 support have been added by
+.An Aleksandr Rybalko Aq Mt ray@FreeBSD.org .
+.Pp
 This manual page was written by
 .An Ceri Davies Aq Mt ceri@FreeBSD.org .