diff options
author | Enji Cooper <ngie@FreeBSD.org> | 2020-03-28 01:14:37 +0000 |
---|---|---|
committer | Enji Cooper <ngie@FreeBSD.org> | 2020-03-28 01:14:37 +0000 |
commit | dc60c918217e1392aa238dbc75719268c5a2b63a (patch) | |
tree | e975271df916ee08d5cf74016899f55079d0ccfd /contrib/kyua/doc | |
parent | 07ca59522d61ce1b9e794031abc39db353e64ab3 (diff) | |
download | src-dc60c918217e1392aa238dbc75719268c5a2b63a.tar.gz src-dc60c918217e1392aa238dbc75719268c5a2b63a.zip |
Follow up to r359385
Actually add the generated manpages to unbreak the build.
MFC with: r359385
Notes
Notes:
svn path=/head/; revision=359386
Diffstat (limited to 'contrib/kyua/doc')
-rw-r--r-- | contrib/kyua/doc/kyua-about.1 | 95 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-config.1 | 59 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-db-exec.1 | 199 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-db-migrate.1 | 182 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-debug.1 | 398 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-help.1 | 64 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-list.1 | 232 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-report-html.1 | 253 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-report-junit.1 | 237 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-report.1 | 307 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua-test.1 | 498 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua.1 | 400 | ||||
-rw-r--r-- | contrib/kyua/doc/kyua.conf.5 | 141 | ||||
-rw-r--r-- | contrib/kyua/doc/kyuafile.5 | 407 |
14 files changed, 3472 insertions, 0 deletions
diff --git a/contrib/kyua/doc/kyua-about.1 b/contrib/kyua/doc/kyua-about.1 new file mode 100644 index 000000000000..a6cf61dce602 --- /dev/null +++ b/contrib/kyua/doc/kyua-about.1 @@ -0,0 +1,95 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd May 20, 2015 +.Dt KYUA-ABOUT 1 +.Os +.Sh NAME +.Nm "kyua about" +.Nd Shows detailed authors, license, and version information +.Sh SYNOPSIS +.Nm +.Op Ar authors | license | version +.Sh DESCRIPTION +The +.Sq about +command provides generic information about the +.Xr kyua 1 +tool. +In the default synopsis form (no arguments), the information printed +includes: +.Bl -enum +.It +The name of the package, which is +.Sq kyua . +.It +The version number, which is +.Sq 0.13 . +.It +License information. +.It +Authors information. +.It +A link to the project web site. +.El +.Pp +You can customize the information printed by this command by specifying +the desired topic as the single argument to the command. +This can be one of: +.Bl -tag -width authorsXX +.It Ar authors +Displays the list of authors and contributors only. +.It Ar license +Displays the license information and the list of copyrights. +.It Ar version +Displays the package name and the version number in a format that is +compatible with the output of GNU tools that support a +.Fl -version +flag. +Use this whenever you have to query the version number of the package. +.El +.Sh FILES +The following files are read by the +.Nm +command: +.Bl -tag -width XX +.It Pa /usr/share/doc/kyua/AUTHORS +List of authors (aka copyright holders). +.It Pa /usr/share/doc/kyua/CONTRIBUTORS +List of contributors (aka individuals that have contributed to the project). +.It Pa /usr/share/doc/kyua/LICENSE +License information. +.El +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/contrib/kyua/doc/kyua-config.1 b/contrib/kyua/doc/kyua-config.1 new file mode 100644 index 000000000000..9c13ce06505e --- /dev/null +++ b/contrib/kyua/doc/kyua-config.1 @@ -0,0 +1,59 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd September 9, 2012 +.Dt KYUA-CONFIG 1 +.Os +.Sh NAME +.Nm "kyua config" +.Nd Inspects the values of the loaded configuration +.Sh SYNOPSIS +.Nm +.Op Ar variable1 .. variableN +.Sh DESCRIPTION +The +.Nm +command provides a way to list all defined configuration variables and +their current values. +.Pp +This command is intended to help you in resolving the values of the +configuration variables without having to scan over configuration files. +.Pp +In the default synopsis form (no arguments), the command prints all +configuration variables. +If any arguments are provided, the command will only print the +requested variables. +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if any of the specified configuration +variables does not exist. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/contrib/kyua/doc/kyua-db-exec.1 b/contrib/kyua/doc/kyua-db-exec.1 new file mode 100644 index 000000000000..519a4abbfa29 --- /dev/null +++ b/contrib/kyua/doc/kyua-db-exec.1 @@ -0,0 +1,199 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-DB-EXEC 1 +.Os +.Sh NAME +.Nm "kyua db-exec" +.Nd Executes a SQL statement in a results file +.Sh SYNOPSIS +.Nm +.Op Fl -no-headers +.Op Fl -results-file Ar file +.Ar statement +.Sh DESCRIPTION +The +.Nm +command provides a way to execute an arbitrary SQL statement within the +database. +This command is mostly intended to aid in debugging, but can also be used to +extract information from the database when the current interfaces do not +provide the desired functionality. +.Pp +The input database must exist. +It makes no sense to use +.Nm +on a nonexistent or empty database. +.Pp +The +.Nm +command takes one or more arguments, all of which are concatenated to form +a single SQL statement. +Once the statement is executed, +.Nm +prints the resulting table on the screen, if any. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -no-headers +Avoids printing the headers of the table in the output of the command. +.It Fl -results-file Ar path , Fl s Ar path +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to operate on. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically load the latest results file from the current test suite. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq LATEST +Requests the load of the latest results file available for the test suite rooted +at the current directory. +.It Directory +Requests the load of the latest results file available for the test suite rooted +at the given directory. +.It Test suite name +Requests the load of the latest results file available for the given test suite. +.It Results identifier +Requests the load of a specific results file. +.It Explicit file name (aka everything else) +Load the specified results file. +.El +.Pp +See +.Sx Results files +for more details. +.El +.Ss Results files +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Results files contain, as their name implies, the results of the execution of a +test suite. +Each test suite executed by +.Xr kyua-test 1 +generates a new results file, and such results files can be loaded later on by +inspection commands such as +.Xr kyua-report 1 +to analyze their contents. +.Pp +Results files support identifier-based lookups and also path name lookups. +The differences between the two are described below. +.Pp +The default naming scheme for the results files provides simple support for +identifier-based lookups and historical recording of test suite runs. +Each results file is given an identifier derived from the test suite that +generated it and the time the test suite was run. +Kyua can later look up results files by these fields. +.Pp +The identifier follows this pattern: +.Bd -literal -offset indent +\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt +.Ed +.Pp +where +.Sq test_suite +is the path to the root of the test suite that was run with all slashes replaced +by underscores and +.Sq YYYYMMDD-HHMMSS-uuuuuu +is a timestamp with microsecond resolution. +.Pp +When using the default naming scheme, results files are stored in the +.Pa ~/.kyua/store/ +subdirectory and each file holds a name of the form: +.Bd -literal -offset indent +~/.kyua/store/results.\*(Ltidentifier\*(Gt.db +.Ed +.Pp +Results files are simple SQLite databases with the schema described in the +.Pa /usr/share/kyua/store/schema_v?.sql +files. +For details on the schema, please refer to the heavily commented SQL file. +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if the SQL statement is invalid or fails +to run. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-test 1 diff --git a/contrib/kyua/doc/kyua-db-migrate.1 b/contrib/kyua/doc/kyua-db-migrate.1 new file mode 100644 index 000000000000..1fbc9a92bcc2 --- /dev/null +++ b/contrib/kyua/doc/kyua-db-migrate.1 @@ -0,0 +1,182 @@ +.\" Copyright 2013 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-DB-MIGRATE 1 +.Os +.Sh NAME +.Nm "kyua db-migrate" +.Nd Upgrades the schema of an existing results file +.Sh SYNOPSIS +.Nm +.Op Fl -results-file Ar file +.Sh DESCRIPTION +The +.Nm +command migrates the schema of an existing database to the latest +version implemented in +.Xr kyua 1 . +.Pp +This operation is not reversible. +However, a backup of the database is created in the same directory where the +database lives. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -results-file Ar path , Fl s Ar path +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to operate on. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically load the latest results file from the current test suite. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq LATEST +Requests the load of the latest results file available for the test suite rooted +at the current directory. +.It Directory +Requests the load of the latest results file available for the test suite rooted +at the given directory. +.It Test suite name +Requests the load of the latest results file available for the given test suite. +.It Results identifier +Requests the load of a specific results file. +.It Explicit file name (aka everything else) +Load the specified results file. +.El +.Pp +See +.Sx Results files +for more details. +.El +.Ss Results files +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Results files contain, as their name implies, the results of the execution of a +test suite. +Each test suite executed by +.Xr kyua-test 1 +generates a new results file, and such results files can be loaded later on by +inspection commands such as +.Xr kyua-report 1 +to analyze their contents. +.Pp +Results files support identifier-based lookups and also path name lookups. +The differences between the two are described below. +.Pp +The default naming scheme for the results files provides simple support for +identifier-based lookups and historical recording of test suite runs. +Each results file is given an identifier derived from the test suite that +generated it and the time the test suite was run. +Kyua can later look up results files by these fields. +.Pp +The identifier follows this pattern: +.Bd -literal -offset indent +\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt +.Ed +.Pp +where +.Sq test_suite +is the path to the root of the test suite that was run with all slashes replaced +by underscores and +.Sq YYYYMMDD-HHMMSS-uuuuuu +is a timestamp with microsecond resolution. +.Pp +When using the default naming scheme, results files are stored in the +.Pa ~/.kyua/store/ +subdirectory and each file holds a name of the form: +.Bd -literal -offset indent +~/.kyua/store/results.\*(Ltidentifier\*(Gt.db +.Ed +.Pp +Results files are simple SQLite databases with the schema described in the +.Pa /usr/share/kyua/store/schema_v?.sql +files. +For details on the schema, please refer to the heavily commented SQL file. +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if the migration fails. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/contrib/kyua/doc/kyua-debug.1 b/contrib/kyua/doc/kyua-debug.1 new file mode 100644 index 000000000000..6e3516a167f1 --- /dev/null +++ b/contrib/kyua/doc/kyua-debug.1 @@ -0,0 +1,398 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-DEBUG 1 +.Os +.Sh NAME +.Nm "kyua debug" +.Nd Executes a single test case with facilities for debugging +.Sh SYNOPSIS +.Nm +.Op Fl -build-root Ar path +.Op Fl -kyuafile Ar file +.Op Fl -stdout Ar path +.Op Fl -stderr Ar path +.Ar test_case +.Sh DESCRIPTION +The +.Nm +command provides a mechanism to execute a single test case bypassing some +of the Kyua infrastructure and allowing the user to poke into the execution +behavior of the test. +.Pp +The test case to run is selected by providing a test filter, described below in +.Sx Test filters , +that matches a single test case. +The test case is executed and its result is printed as the last line of the +output of the tool. +.Pp +The test executed by +.Nm +is run under a controlled environment as described in +.Sx Test isolation . +.Pp +At the moment, the +.Nm +command allows the following aspects of a test case execution to be +tweaked: +.Bl -bullet +.It +Redirection of the test case's stdout and stderr to the console (the +default) or to arbitrary files. +See the +.Fl -stdout +and +.Fl -stderr +options below. +.El +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -build-root Ar path +Specifies the build root in which to find the test programs referenced +by the Kyuafile, if different from the Kyuafile's directory. +See +.Sx Build directories +below for more information. +.It Fl -kyuafile Ar file , Fl k Ar file +Specifies the Kyuafile to process. +Defaults to +.Pa Kyuafile +file in the current directory. +.It Fl -stderr Ar path +Specifies the file to which to send the standard error of the test +program's body. +The default is +.Pa /dev/stderr , +which is a special character device that redirects the output to +standard error on the console. +.It Fl -stdout Ar path +Specifies the file to which to send the standard output of the test +program's body. +The default is +.Pa /dev/stdout , +which is a special character device that redirects the output to +standard output on the console. +.El +.Pp +For example, consider the following Kyua session: +.Bd -literal -offset indent +$ kyua test +kernel/fs:mkdir -> passed +kernel/fs:rmdir -> failed: Invalid argument + +1/2 passed (1 failed) +.Ed +.Pp +At this point, we do not have a lot of information regarding the +failure of the +.Sq kernel/fs:rmdir +test. +We can run this test through the +.Nm +command to inspect its output a bit closer, hoping that the test case is +kind enough to log its progress: +.Bd -literal -offset indent +$ kyua debug kernel/fs:rmdir +Trying rmdir('foo') +Trying rmdir(NULL) +kernel/fs:rmdir -> failed: Invalid argument +.Ed +.Pp +Luckily, the offending test case was printing status lines as it +progressed, so we could see the last attempted call and we can know match +the failure message to the problem. +.Ss Build directories +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Em Build directories +(or object directories, target directories, product directories, etc.) is +the concept that allows a developer to keep the source tree clean from +build products by asking the build system to place such build products +under a separate subtree. +.Pp +Most build systems today support build directories. +For example, the GNU Automake/Autoconf build system exposes such concept when +invoked as follows: +.Bd -literal -offset indent +$ cd my-project-1.0 +$ mkdir build +$ cd build +$ ../configure +$ make +.Ed +.Pp +Under such invocation, all the results of the build are left in the +.Pa my-project-1.0/build/ +subdirectory while maintaining the contents of +.Pa my-project-1.0/ +intact. +.Pp +Because build directories are an integral part of most build systems, and +because they are a tool that developers use frequently, +.Nm +supports build directories too. +This manifests in the form of +.Nm +being able to run tests from build directories while reading the (often +immutable) test suite definition from the source tree. +.Pp +One important property of build directories is that they follow (or need to +follow) the exact same layout as the source tree. +For example, consider the following directory listings: +.Bd -literal -offset indent +src/Kyuafile +src/bin/ls/ +src/bin/ls/Kyuafile +src/bin/ls/ls.c +src/bin/ls/ls_test.c +src/sbin/su/ +src/sbin/su/Kyuafile +src/sbin/su/su.c +src/sbin/su/su_test.c + +obj/bin/ls/ +obj/bin/ls/ls* +obj/bin/ls/ls_test* +obj/sbin/su/ +obj/sbin/su/su* +obj/sbin/su/su_test* +.Ed +.Pp +Note how the directory layout within +.Pa src/ +matches that of +.Pa obj/ . +The +.Pa src/ +directory contains only source files and the definition of the test suite +(the Kyuafiles), while the +.Pa obj/ +directory contains only the binaries generated during a build. +.Pp +All commands that deal with the workspace support the +.Fl -build-root Ar path +option. +When this option is provided, the directory specified by the +option is considered to be the root of the build directory. +For example, considering our previous fake tree layout, we could invoke +.Nm +as any of the following: +.Bd -literal -offset indent +$ kyua debug --kyuafile=src/Kyuafile --build-root=obj +$ cd src && kyua debug --build-root=../obj +.Ed +.Ss Test filters +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +A +.Em test filter +is a string that is used to match test cases or test programs in a test suite. +Filters have the following form: +.Bd -literal -offset indent +test_program_name[:test_case_name] +.Ed +.Pp +Where +.Sq test_program_name +is the name of a test program or a subdirectory in the test suite, and +.Sq test_case_name +is the name of a test case. +.Ss Test isolation +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +The test programs and test cases run by +.Nm +are all executed in a deterministic environment. +This known, clean environment serves to make the test execution as +reproducible as possible and also to prevent clashes between tests that may, +for example, create auxiliary files with overlapping names. +.Pp +For plain test programs and for TAP test programs, the whole test program +is run under a single instance of the environment described in this page. +For ATF test programs (see +.Xr atf 7 ) , +each individual test case +.Em and +test cleanup routine are executed in separate environments. +.Bl -tag -width XX +.It Process space +Each test is executed in an independent processes. +Corollary: the test can do whatever it wants to the current process (such +as modify global variables) without having to undo such changes. +.It Session and process group +The test is executed in its own session and its own process group. +There is no controlling terminal attached to the session. +.Pp +Should the test spawn any children, the children should maintain the same +session and process group. +Modifying any of these settings prevents +.Nm +from being able to kill any stray subprocess as part of the cleanup phase. +If modifying these settings is necessary, or if any subprocess started by +the test decides to use a different process group or session, it is the +responsibility of the test to ensure those subprocesses are forcibly +terminated during cleanup. +.It Work directory +The test is executed in a temporary directory automatically created by the +runtime engine. +Corollary: the test can write to its current directory +without needing to clean any files and/or directories it creates. +The runtime engine takes care to recursively delete the temporary directories +after the execution of a test case. +Any file systems mounted within the temporary directory are also unmounted. +.It Home directory +The +.Va HOME +environment variable is set to the absolute path of the work directory. +.It Umask +The value of the umask is set to 0022. +.It Environment +The +.Va LANG , +.Va LC_ALL , +.Va LC_COLLATE , +.Va LC_CTYPE , +.Va LC_MESSAGES , +.Va LC_MONETARY , +.Va LC_NUMERIC +and +.Va LC_TIME +variables are unset. +.Pp +The +.Va TZ +variable is set to +.Sq UTC . +.Pp +The +.Va TMPDIR +variable is set to the absolute path of the work directory. +This is to prevent the test from mistakenly using a temporary directory +outside of the automatically-managed work directory, should the test use the +.Xr mktemp 3 +familiy of functions. +.It Process limits +The maximum soft core size limit is raised to its corresponding hard limit. +This is a simple, best-effort attempt at allowing tests to dump core for +further diagnostic purposes. +.It Configuration varibles +The test engine may pass run-time configuration variables to the test program +via the environment. +The name of the configuration variable is prefixed with +.Sq TEST_ENV_ +so that a configuration variable of the form +.Sq foo=bar +becomes accessible in the environment as +.Sq TEST_ENV_foo=bar . +.El +.Sh EXIT STATUS +The +.Nm +command returns 0 if the test case passes or 1 if the test case fails. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyuafile 5 diff --git a/contrib/kyua/doc/kyua-help.1 b/contrib/kyua/doc/kyua-help.1 new file mode 100644 index 000000000000..2c4f2bc3859e --- /dev/null +++ b/contrib/kyua/doc/kyua-help.1 @@ -0,0 +1,64 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd September 9, 2012 +.Dt KYUA-HELP 1 +.Os +.Sh NAME +.Nm "kyua help" +.Nd Shows usage information +.Sh SYNOPSIS +.Nm +.Op Ar command +.Sh DESCRIPTION +The +.Nm +command provides interactive help on all supported commands and options. +If, for some reason, you happen to spot a discrepancy in the output of this +command and this document, the command is the authoritative source of +information. +.Pp +If no arguments are provided, the command prints the list of common options +and the list of supported subcommands. +.Pp +If the +.Ar command +argument is provided to, this single argument is the name of a valid +subcommand. +In that case, +.Nm +prints a textual description of the command, the list of common options and +the list of subcommand-specific options. +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 diff --git a/contrib/kyua/doc/kyua-list.1 b/contrib/kyua/doc/kyua-list.1 new file mode 100644 index 000000000000..155dccfff71c --- /dev/null +++ b/contrib/kyua/doc/kyua-list.1 @@ -0,0 +1,232 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-LIST 1 +.Os +.Sh NAME +.Nm "kyua list" +.Nd Lists test cases and their metadata +.Sh SYNOPSIS +.Nm +.Op Fl -build-root Ar path +.Op Fl -kyuafile Ar file +.Op Fl -verbose +.Ar test_case1 Op Ar .. test_caseN +.Sh DESCRIPTION +The +.Nm +command scans all the test programs and test cases in a test suite (as +defined by a +.Xr kyuafile 5 ) +and prints a list of all their names, optionally accompanied by any metadata +properties they have. +.Pp +The optional arguments to +.Nm +are used to select which test programs or test cases to run. +These are filters and are described below in +.Sx Test filters . +.Pp +This command must be run within a test suite or a test suite must be +provided with the +.Fl -kyuafile +flag. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -build-root Ar path +Specifies the build root in which to find the test programs referenced +by the Kyuafile, if different from the Kyuafile's directory. +See +.Sx Build directories +below for more information. +.It Fl -kyuafile Ar path , Fl k Ar path +Specifies the Kyuafile to process. +Defaults to a +.Pa Kyuafile +file in the current directory. +.It Fl -verbose , Fl v +Prints metadata properties for every test case. +.El +.Ss Build directories +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Em Build directories +(or object directories, target directories, product directories, etc.) is +the concept that allows a developer to keep the source tree clean from +build products by asking the build system to place such build products +under a separate subtree. +.Pp +Most build systems today support build directories. +For example, the GNU Automake/Autoconf build system exposes such concept when +invoked as follows: +.Bd -literal -offset indent +$ cd my-project-1.0 +$ mkdir build +$ cd build +$ ../configure +$ make +.Ed +.Pp +Under such invocation, all the results of the build are left in the +.Pa my-project-1.0/build/ +subdirectory while maintaining the contents of +.Pa my-project-1.0/ +intact. +.Pp +Because build directories are an integral part of most build systems, and +because they are a tool that developers use frequently, +.Nm +supports build directories too. +This manifests in the form of +.Nm +being able to run tests from build directories while reading the (often +immutable) test suite definition from the source tree. +.Pp +One important property of build directories is that they follow (or need to +follow) the exact same layout as the source tree. +For example, consider the following directory listings: +.Bd -literal -offset indent +src/Kyuafile +src/bin/ls/ +src/bin/ls/Kyuafile +src/bin/ls/ls.c +src/bin/ls/ls_test.c +src/sbin/su/ +src/sbin/su/Kyuafile +src/sbin/su/su.c +src/sbin/su/su_test.c + +obj/bin/ls/ +obj/bin/ls/ls* +obj/bin/ls/ls_test* +obj/sbin/su/ +obj/sbin/su/su* +obj/sbin/su/su_test* +.Ed +.Pp +Note how the directory layout within +.Pa src/ +matches that of +.Pa obj/ . +The +.Pa src/ +directory contains only source files and the definition of the test suite +(the Kyuafiles), while the +.Pa obj/ +directory contains only the binaries generated during a build. +.Pp +All commands that deal with the workspace support the +.Fl -build-root Ar path +option. +When this option is provided, the directory specified by the +option is considered to be the root of the build directory. +For example, considering our previous fake tree layout, we could invoke +.Nm +as any of the following: +.Bd -literal -offset indent +$ kyua list --kyuafile=src/Kyuafile --build-root=obj +$ cd src && kyua list --build-root=../obj +.Ed +.Ss Test filters +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +A +.Em test filter +is a string that is used to match test cases or test programs in a test suite. +Filters have the following form: +.Bd -literal -offset indent +test_program_name[:test_case_name] +.Ed +.Pp +Where +.Sq test_program_name +is the name of a test program or a subdirectory in the test suite, and +.Sq test_case_name +is the name of a test case. +.Sh EXIT STATUS +The +.Nm +command returns 0 on success or 1 if any of the given test case filters +does not match any test case. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyuafile 5 diff --git a/contrib/kyua/doc/kyua-report-html.1 b/contrib/kyua/doc/kyua-report-html.1 new file mode 100644 index 000000000000..ffe0430a6d22 --- /dev/null +++ b/contrib/kyua/doc/kyua-report-html.1 @@ -0,0 +1,253 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-REPORT-HTML 1 +.Os +.Sh NAME +.Nm "kyua report-html" +.Nd Generates an HTML report with the results of a test suite run +.Sh SYNOPSIS +.Nm +.Op Fl -force +.Op Fl -output Ar path +.Op Fl -results-file Ar file +.Op Fl -results-filter Ar types +.Sh DESCRIPTION +The +.Nm +command provides a simple mechanism to generate HTML reports of the +execution of a test suite. +The command processes a results file and then populates a directory with +multiple HTML and supporting files to describe the results recorded in that +results file. +.Pp +The HTML output is static and self-contained, so it can easily be served by +any simple web server. +The command expects the target directory to not exist, because it would +overwrite any contents if not careful. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -force +Forces the deletion of the output directory if it exists. +Use care, as this effectively means a +.Sq rm -rf . +.It Fl -output Ar directory +Specifies the target directory into which to generate the HTML files. +The directory must not exist unless the +.Fl -force +option is provided. +The default is +.Pa ./html . +.It Fl -results-file Ar path , Fl s Ar path +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to operate on. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically load the latest results file from the current test suite. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq LATEST +Requests the load of the latest results file available for the test suite rooted +at the current directory. +.It Directory +Requests the load of the latest results file available for the test suite rooted +at the given directory. +.It Test suite name +Requests the load of the latest results file available for the given test suite. +.It Results identifier +Requests the load of a specific results file. +.It Explicit file name (aka everything else) +Load the specified results file. +.El +.Pp +See +.Sx Results files +for more details. +.It Fl -results-filter Ar types +Comma-separated list of the test result types to include in the report. +The ordering of the values is respected so that you can determine how you +want the list of tests to be shown. +.Pp +The valid values are: +.Sq broken , +.Sq failed , +.Sq passed , +.Sq skipped +and +.Sq xfail . +If the parameter supplied to the option is empty, filtering is suppressed +and all result types are shown in the report. +.Pp +The default value for this flag includes all the test results except the +passed tests. +Showing the passed tests by default clutters the report with too much +information, so only abnormal conditions are included. +.El +.Ss Results files +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Results files contain, as their name implies, the results of the execution of a +test suite. +Each test suite executed by +.Xr kyua-test 1 +generates a new results file, and such results files can be loaded later on by +inspection commands such as +.Xr kyua-report 1 +to analyze their contents. +.Pp +Results files support identifier-based lookups and also path name lookups. +The differences between the two are described below. +.Pp +The default naming scheme for the results files provides simple support for +identifier-based lookups and historical recording of test suite runs. +Each results file is given an identifier derived from the test suite that +generated it and the time the test suite was run. +Kyua can later look up results files by these fields. +.Pp +The identifier follows this pattern: +.Bd -literal -offset indent +\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt +.Ed +.Pp +where +.Sq test_suite +is the path to the root of the test suite that was run with all slashes replaced +by underscores and +.Sq YYYYMMDD-HHMMSS-uuuuuu +is a timestamp with microsecond resolution. +.Pp +When using the default naming scheme, results files are stored in the +.Pa ~/.kyua/store/ +subdirectory and each file holds a name of the form: +.Bd -literal -offset indent +~/.kyua/store/results.\*(Ltidentifier\*(Gt.db +.Ed +.Pp +Results files are simple SQLite databases with the schema described in the +.Pa /usr/share/kyua/store/schema_v?.sql +files. +For details on the schema, please refer to the heavily commented SQL file. +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +.Ss Workflow with results files +If one runs the following command twice in a row: +.Bd -literal -offset indent +kyua test -k /usr/tests/Kyuafile +.Ed +.Pp +the two executions will generate two different files with names like: +.Bd -literal -offset indent +~/.kyua/store/results.usr_tests.20140731-150500-196784.db +~/.kyua/store/results.usr_tests.20140731-151730-997451.db +.Ed +.Pp +Taking advantage of the default naming scheme, the following commands would all +generate a report for the results of the +.Em latest +execution of the test suite: +.Bd -literal -offset indent +cd /usr/tests && kyua report-html +cd /usr/tests && kyua report-html --results-file=LATEST +kyua report-html --results-file=/usr/tests +kyua report-html --results-file=usr_tests +kyua report-html --results-file=usr_tests.20140731-151730-997451 +.Ed +.Pp +But it is also possible to explicitly load data for older runs or from +explicitly-named files: +.Bd -literal -offset indent +kyua report-html \\ + --results-file=usr_tests.20140731-150500-196784 +kyua report-html \\ + --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db +.Ed +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report 1 , +.Xr kyua-report-junit 1 diff --git a/contrib/kyua/doc/kyua-report-junit.1 b/contrib/kyua/doc/kyua-report-junit.1 new file mode 100644 index 000000000000..1a0ae831c8c7 --- /dev/null +++ b/contrib/kyua/doc/kyua-report-junit.1 @@ -0,0 +1,237 @@ +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-REPORT-JUNIT 1 +.Os +.Sh NAME +.Nm "kyua report-junit" +.Nd Generates a JUnit report with the results of a test suite run +.Sh SYNOPSIS +.Nm +.Op Fl -output Ar path +.Op Fl -results-file Ar file +.Sh DESCRIPTION +The +.Nm +command provides a simple mechanism to generate JUnit reports of the +execution of a test suite. +The command processes a results file and then generates a single XML file +that complies with the JUnit XSchema. +.Pp +The JUnit output is static and self-contained, so it can easily be plugged +into any continuous integration system, like Jenkins. +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -output Ar directory +Specifies the file into which to store the JUnit report. +.It Fl -results-file Ar path , Fl s Ar path +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to operate on. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically load the latest results file from the current test suite. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq LATEST +Requests the load of the latest results file available for the test suite rooted +at the current directory. +.It Directory +Requests the load of the latest results file available for the test suite rooted +at the given directory. +.It Test suite name +Requests the load of the latest results file available for the given test suite. +.It Results identifier +Requests the load of a specific results file. +.It Explicit file name (aka everything else) +Load the specified results file. +.El +.Pp +See +.Sx Results files +for more details. +.El +.Ss Caveats +Because of limitations in the JUnit XML schema, not all the data collected by +Kyua can be properly represented in JUnit reports. +However, because test data are extremely useful for debugging purposes, the +.Nm +command shovels these data into the JUnit output. +In particular: +.Bl -bullet +.It +The test case metadata values are prepended to the test case's standard error +output. +.It +Test cases that report expected failures as their results are recorded as +passed. +The fact that they failed as expected is recorded in the test case's standard +error output along with the corresponding reason. +.El +.Ss Results files +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Results files contain, as their name implies, the results of the execution of a +test suite. +Each test suite executed by +.Xr kyua-test 1 +generates a new results file, and such results files can be loaded later on by +inspection commands such as +.Xr kyua-report 1 +to analyze their contents. +.Pp +Results files support identifier-based lookups and also path name lookups. +The differences between the two are described below. +.Pp +The default naming scheme for the results files provides simple support for +identifier-based lookups and historical recording of test suite runs. +Each results file is given an identifier derived from the test suite that +generated it and the time the test suite was run. +Kyua can later look up results files by these fields. +.Pp +The identifier follows this pattern: +.Bd -literal -offset indent +\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt +.Ed +.Pp +where +.Sq test_suite +is the path to the root of the test suite that was run with all slashes replaced +by underscores and +.Sq YYYYMMDD-HHMMSS-uuuuuu +is a timestamp with microsecond resolution. +.Pp +When using the default naming scheme, results files are stored in the +.Pa ~/.kyua/store/ +subdirectory and each file holds a name of the form: +.Bd -literal -offset indent +~/.kyua/store/results.\*(Ltidentifier\*(Gt.db +.Ed +.Pp +Results files are simple SQLite databases with the schema described in the +.Pa /usr/share/kyua/store/schema_v?.sql +files. +For details on the schema, please refer to the heavily commented SQL file. +.Sh EXIT STATUS +The +.Nm +command always returns 0. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +.Ss Workflow with results files +If one runs the following command twice in a row: +.Bd -literal -offset indent +kyua test -k /usr/tests/Kyuafile +.Ed +.Pp +the two executions will generate two different files with names like: +.Bd -literal -offset indent +~/.kyua/store/results.usr_tests.20140731-150500-196784.db +~/.kyua/store/results.usr_tests.20140731-151730-997451.db +.Ed +.Pp +Taking advantage of the default naming scheme, the following commands would all +generate a report for the results of the +.Em latest +execution of the test suite: +.Bd -literal -offset indent +cd /usr/tests && kyua report-junit +cd /usr/tests && kyua report-junit --results-file=LATEST +kyua report-junit --results-file=/usr/tests +kyua report-junit --results-file=usr_tests +kyua report-junit --results-file=usr_tests.20140731-151730-997451 +.Ed +.Pp +But it is also possible to explicitly load data for older runs or from +explicitly-named files: +.Bd -literal -offset indent +kyua report-junit \\ + --results-file=usr_tests.20140731-150500-196784 +kyua report-junit \\ + --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db +.Ed +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report 1 , +.Xr kyua-report-html 1 diff --git a/contrib/kyua/doc/kyua-report.1 b/contrib/kyua/doc/kyua-report.1 new file mode 100644 index 000000000000..7ea24ef6d734 --- /dev/null +++ b/contrib/kyua/doc/kyua-report.1 @@ -0,0 +1,307 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-REPORT 1 +.Os +.Sh NAME +.Nm "kyua report" +.Nd Generates reports with the results of a test suite run +.Sh SYNOPSIS +.Nm +.Op Fl -output Ar path +.Op Fl -results-file Ar file +.Op Fl -results-filter Ar types +.Op Fl -verbose +.Op Ar test_filter1 .. test_filterN +.Sh DESCRIPTION +The +.Nm +command parses a results file and generates a user-friendly, plaintext +report for user consumption on the terminal. +By default, these reports only display a summary of the execution of the full +test suite to highlight where problems may lie. +.Pp +The output of +.Nm +can be customized to display full details on all executed test cases. +Additionally, the optional arguments to +.Nm +are used to select which test programs or test cases to display. +These are filters and are described below in +.Sx Test filters . +.Pp +Reports generated by +.Nm +are +.Em not intended to be machine-parseable . +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -output Ar path +Specifies the path to which the report should be written to. +The special values +.Pa /dev/stdout +and +.Pa /dev/stderr +can be used to specify the standard output and the standard error, +respectively. +.It Fl -results-file Ar path , Fl s Ar path +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to operate on. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically load the latest results file from the current test suite. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq LATEST +Requests the load of the latest results file available for the test suite rooted +at the current directory. +.It Directory +Requests the load of the latest results file available for the test suite rooted +at the given directory. +.It Test suite name +Requests the load of the latest results file available for the given test suite. +.It Results identifier +Requests the load of a specific results file. +.It Explicit file name (aka everything else) +Load the specified results file. +.El +.Pp +See +.Sx Results files +for more details. +.It Fl -results-filter Ar types +Comma-separated list of the test result types to include in the report. +The ordering of the values is respected so that you can determine how you +want the list of tests to be shown. +.Pp +The valid values are: +.Sq broken , +.Sq failed , +.Sq passed , +.Sq skipped +and +.Sq xfail . +If the parameter supplied to the option is empty, filtering is suppressed +and all result types are shown in the report. +.Pp +The default value for this flag includes all the test results except the +passed tests. +Showing the passed tests by default clutters the report with too much +information, so only abnormal conditions are included. +.It Fl -verbose +Prints a detailed report of the execution. +In addition to all the information printed by default, verbose reports +include the runtime context of the test suite run, the metadata of each +test case, and the verbatim output of the test cases. +.El +.Ss Results files +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Results files contain, as their name implies, the results of the execution of a +test suite. +Each test suite executed by +.Xr kyua-test 1 +generates a new results file, and such results files can be loaded later on by +inspection commands such as +.Xr kyua-report 1 +to analyze their contents. +.Pp +Results files support identifier-based lookups and also path name lookups. +The differences between the two are described below. +.Pp +The default naming scheme for the results files provides simple support for +identifier-based lookups and historical recording of test suite runs. +Each results file is given an identifier derived from the test suite that +generated it and the time the test suite was run. +Kyua can later look up results files by these fields. +.Pp +The identifier follows this pattern: +.Bd -literal -offset indent +\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt +.Ed +.Pp +where +.Sq test_suite +is the path to the root of the test suite that was run with all slashes replaced +by underscores and +.Sq YYYYMMDD-HHMMSS-uuuuuu +is a timestamp with microsecond resolution. +.Pp +When using the default naming scheme, results files are stored in the +.Pa ~/.kyua/store/ +subdirectory and each file holds a name of the form: +.Bd -literal -offset indent +~/.kyua/store/results.\*(Ltidentifier\*(Gt.db +.Ed +.Pp +Results files are simple SQLite databases with the schema described in the +.Pa /usr/share/kyua/store/schema_v?.sql +files. +For details on the schema, please refer to the heavily commented SQL file. +.Ss Test filters +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +A +.Em test filter +is a string that is used to match test cases or test programs in a test suite. +Filters have the following form: +.Bd -literal -offset indent +test_program_name[:test_case_name] +.Ed +.Pp +Where +.Sq test_program_name +is the name of a test program or a subdirectory in the test suite, and +.Sq test_case_name +is the name of a test case. +.Sh EXIT STATUS +The +.Nm +command returns 0 if no filters were specified or if all filters match one +or more test cases. +If any filter fails to match any test case, the command returns 1. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +.Ss Workflow with results files +If one runs the following command twice in a row: +.Bd -literal -offset indent +kyua test -k /usr/tests/Kyuafile +.Ed +.Pp +the two executions will generate two different files with names like: +.Bd -literal -offset indent +~/.kyua/store/results.usr_tests.20140731-150500-196784.db +~/.kyua/store/results.usr_tests.20140731-151730-997451.db +.Ed +.Pp +Taking advantage of the default naming scheme, the following commands would all +generate a report for the results of the +.Em latest +execution of the test suite: +.Bd -literal -offset indent +cd /usr/tests && kyua report +cd /usr/tests && kyua report --results-file=LATEST +kyua report --results-file=/usr/tests +kyua report --results-file=usr_tests +kyua report --results-file=usr_tests.20140731-151730-997451 +.Ed +.Pp +But it is also possible to explicitly load data for older runs or from +explicitly-named files: +.Bd -literal -offset indent +kyua report \\ + --results-file=usr_tests.20140731-150500-196784 +kyua report \\ + --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db +.Ed +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report-html 1 , +.Xr kyua-report-junit 1 diff --git a/contrib/kyua/doc/kyua-test.1 b/contrib/kyua/doc/kyua-test.1 new file mode 100644 index 000000000000..54bfe4866b81 --- /dev/null +++ b/contrib/kyua/doc/kyua-test.1 @@ -0,0 +1,498 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd October 13, 2014 +.Dt KYUA-TEST 1 +.Os +.Sh NAME +.Nm "kyua test" +.Nd Runs tests +.Sh SYNOPSIS +.Nm +.Op Fl -build-root Ar path +.Op Fl -kyuafile Ar file +.Op Fl -results-file Ar file +.Op Ar test_filter1 .. test_filterN +.Sh DESCRIPTION +The +.Nm +command loads a test suite definition from a +.Xr kyuafile 5 , +runs the tests defined in it, and records the results into a new results +file. +By default, all tests in the test suite are executed but the optional +arguments to +.Nm +can be used to select which test programs or test cases to run. +These are filters and are described below in +.Sx Test filters . +.Pp +Every test executed by +.Nm +is run under a controlled environment as described in +.Sx Test isolation . +.Pp +The following subcommand options are recognized: +.Bl -tag -width XX +.It Fl -build-root Ar path +Specifies the build root in which to find the test programs referenced by +the Kyuafile, if different from the Kyuafile's directory. +See +.Sx Build directories +below for more information. +.It Fl -kyuafile Ar path , Fl k Ar path +Specifies the Kyuafile to process. +Defaults to a +.Pa Kyuafile +file in the current directory. +.It Fl -results-file Ar path , Fl s Ar path +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Specifies the results file to create. +Defaults to +.Sq LATEST , +which causes +.Nm +to automatically generate a new results file for the test run. +.Pp +The following values are accepted: +.Bl -tag -width XX +.It Sq NEW +Requests the automatic generation of a new results filename based on the test +suite being run and the current time. +.It Explicit filename (aka everything else) +Store the results file where indicated. +.El +.Pp +See +.Sx Results files +for more details. +.El +.Pp +You can later inspect the results of the test run in more detail by using +.Xr kyua-report 1 +or you can execute a single test case with debugging functionality by using +.Xr kyua-debug 1 . +.Ss Build directories +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Em Build directories +(or object directories, target directories, product directories, etc.) is +the concept that allows a developer to keep the source tree clean from +build products by asking the build system to place such build products +under a separate subtree. +.Pp +Most build systems today support build directories. +For example, the GNU Automake/Autoconf build system exposes such concept when +invoked as follows: +.Bd -literal -offset indent +$ cd my-project-1.0 +$ mkdir build +$ cd build +$ ../configure +$ make +.Ed +.Pp +Under such invocation, all the results of the build are left in the +.Pa my-project-1.0/build/ +subdirectory while maintaining the contents of +.Pa my-project-1.0/ +intact. +.Pp +Because build directories are an integral part of most build systems, and +because they are a tool that developers use frequently, +.Nm +supports build directories too. +This manifests in the form of +.Nm +being able to run tests from build directories while reading the (often +immutable) test suite definition from the source tree. +.Pp +One important property of build directories is that they follow (or need to +follow) the exact same layout as the source tree. +For example, consider the following directory listings: +.Bd -literal -offset indent +src/Kyuafile +src/bin/ls/ +src/bin/ls/Kyuafile +src/bin/ls/ls.c +src/bin/ls/ls_test.c +src/sbin/su/ +src/sbin/su/Kyuafile +src/sbin/su/su.c +src/sbin/su/su_test.c + +obj/bin/ls/ +obj/bin/ls/ls* +obj/bin/ls/ls_test* +obj/sbin/su/ +obj/sbin/su/su* +obj/sbin/su/su_test* +.Ed +.Pp +Note how the directory layout within +.Pa src/ +matches that of +.Pa obj/ . +The +.Pa src/ +directory contains only source files and the definition of the test suite +(the Kyuafiles), while the +.Pa obj/ +directory contains only the binaries generated during a build. +.Pp +All commands that deal with the workspace support the +.Fl -build-root Ar path +option. +When this option is provided, the directory specified by the +option is considered to be the root of the build directory. +For example, considering our previous fake tree layout, we could invoke +.Nm +as any of the following: +.Bd -literal -offset indent +$ kyua test --kyuafile=src/Kyuafile --build-root=obj +$ cd src && kyua test --build-root=../obj +.Ed +.Ss Results files +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Results files contain, as their name implies, the results of the execution of a +test suite. +Each test suite executed by +.Xr kyua-test 1 +generates a new results file, and such results files can be loaded later on by +inspection commands such as +.Xr kyua-report 1 +to analyze their contents. +.Pp +Results files support identifier-based lookups and also path name lookups. +The differences between the two are described below. +.Pp +The default naming scheme for the results files provides simple support for +identifier-based lookups and historical recording of test suite runs. +Each results file is given an identifier derived from the test suite that +generated it and the time the test suite was run. +Kyua can later look up results files by these fields. +.Pp +The identifier follows this pattern: +.Bd -literal -offset indent +\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt +.Ed +.Pp +where +.Sq test_suite +is the path to the root of the test suite that was run with all slashes replaced +by underscores and +.Sq YYYYMMDD-HHMMSS-uuuuuu +is a timestamp with microsecond resolution. +.Pp +When using the default naming scheme, results files are stored in the +.Pa ~/.kyua/store/ +subdirectory and each file holds a name of the form: +.Bd -literal -offset indent +~/.kyua/store/results.\*(Ltidentifier\*(Gt.db +.Ed +.Pp +Results files are simple SQLite databases with the schema described in the +.Pa /usr/share/kyua/store/schema_v?.sql +files. +For details on the schema, please refer to the heavily commented SQL file. +.Ss Test filters +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +A +.Em test filter +is a string that is used to match test cases or test programs in a test suite. +Filters have the following form: +.Bd -literal -offset indent +test_program_name[:test_case_name] +.Ed +.Pp +Where +.Sq test_program_name +is the name of a test program or a subdirectory in the test suite, and +.Sq test_case_name +is the name of a test case. +.Ss Test isolation +.\" Copyright 2014 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +The test programs and test cases run by +.Nm +are all executed in a deterministic environment. +This known, clean environment serves to make the test execution as +reproducible as possible and also to prevent clashes between tests that may, +for example, create auxiliary files with overlapping names. +.Pp +For plain test programs and for TAP test programs, the whole test program +is run under a single instance of the environment described in this page. +For ATF test programs (see +.Xr atf 7 ) , +each individual test case +.Em and +test cleanup routine are executed in separate environments. +.Bl -tag -width XX +.It Process space +Each test is executed in an independent processes. +Corollary: the test can do whatever it wants to the current process (such +as modify global variables) without having to undo such changes. +.It Session and process group +The test is executed in its own session and its own process group. +There is no controlling terminal attached to the session. +.Pp +Should the test spawn any children, the children should maintain the same +session and process group. +Modifying any of these settings prevents +.Nm +from being able to kill any stray subprocess as part of the cleanup phase. +If modifying these settings is necessary, or if any subprocess started by +the test decides to use a different process group or session, it is the +responsibility of the test to ensure those subprocesses are forcibly +terminated during cleanup. +.It Work directory +The test is executed in a temporary directory automatically created by the +runtime engine. +Corollary: the test can write to its current directory +without needing to clean any files and/or directories it creates. +The runtime engine takes care to recursively delete the temporary directories +after the execution of a test case. +Any file systems mounted within the temporary directory are also unmounted. +.It Home directory +The +.Va HOME +environment variable is set to the absolute path of the work directory. +.It Umask +The value of the umask is set to 0022. +.It Environment +The +.Va LANG , +.Va LC_ALL , +.Va LC_COLLATE , +.Va LC_CTYPE , +.Va LC_MESSAGES , +.Va LC_MONETARY , +.Va LC_NUMERIC +and +.Va LC_TIME +variables are unset. +.Pp +The +.Va TZ +variable is set to +.Sq UTC . +.Pp +The +.Va TMPDIR +variable is set to the absolute path of the work directory. +This is to prevent the test from mistakenly using a temporary directory +outside of the automatically-managed work directory, should the test use the +.Xr mktemp 3 +familiy of functions. +.It Process limits +The maximum soft core size limit is raised to its corresponding hard limit. +This is a simple, best-effort attempt at allowing tests to dump core for +further diagnostic purposes. +.It Configuration varibles +The test engine may pass run-time configuration variables to the test program +via the environment. +The name of the configuration variable is prefixed with +.Sq TEST_ENV_ +so that a configuration variable of the form +.Sq foo=bar +becomes accessible in the environment as +.Sq TEST_ENV_foo=bar . +.El +.Sh EXIT STATUS +The +.Nm +command returns 0 if all executed test cases pass or 1 if any of the +executed test cases fails or if any of the given test case filters does not +match any test case. +.Pp +Additional exit codes may be returned as described in +.Xr kyua 1 . +.Sh EXAMPLES +.Ss Workflow with results files +If one runs the following command twice in a row: +.Bd -literal -offset indent +kyua test -k /usr/tests/Kyuafile +.Ed +.Pp +the two executions will generate two different files with names like: +.Bd -literal -offset indent +~/.kyua/store/results.usr_tests.20140731-150500-196784.db +~/.kyua/store/results.usr_tests.20140731-151730-997451.db +.Ed +.Pp +Taking advantage of the default naming scheme, the following commands would all +generate a report for the results of the +.Em latest +execution of the test suite: +.Bd -literal -offset indent +cd /usr/tests && kyua report +cd /usr/tests && kyua report --results-file=LATEST +kyua report --results-file=/usr/tests +kyua report --results-file=usr_tests +kyua report --results-file=usr_tests.20140731-151730-997451 +.Ed +.Pp +But it is also possible to explicitly load data for older runs or from +explicitly-named files: +.Bd -literal -offset indent +kyua report \\ + --results-file=usr_tests.20140731-150500-196784 +kyua report \\ + --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db +.Ed +.Sh SEE ALSO +.Xr kyua 1 , +.Xr kyua-report 1 , +.Xr kyuafile 5 diff --git a/contrib/kyua/doc/kyua.1 b/contrib/kyua/doc/kyua.1 new file mode 100644 index 000000000000..21cf46e4396e --- /dev/null +++ b/contrib/kyua/doc/kyua.1 @@ -0,0 +1,400 @@ +.\" Copyright 2011 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd May 12, 2015 +.Dt KYUA 1 +.Os +.Sh NAME +.Nm kyua +.Nd Testing framework for infrastructure software +.Sh SYNOPSIS +.Nm +.Op Fl -config Ar file +.Op Fl -logfile Ar file +.Op Fl -loglevel Ar level +.Op Fl -variable Ar name=value +.Ar command +.Op Ar command_options +.Op Ar command_arguments +.Sh DESCRIPTION +.Em If you are here looking for details on how to run the test suite in +.Pa /usr/tests +.Em ( or +.Pa /usr/tests ) , +.Em please start by reading the +.Xr tests 7 +.Em manual page that should be supplied by your system . +.Pp +Kyua is a testing framework for infrastructure software, originally +designed to equip BSD-based operating systems with a test suite. +This means that Kyua is lightweight and simple, and that Kyua integrates well +with various build systems and continuous integration frameworks. +.Pp +Kyua features an expressive test suite definition language, a safe +runtime engine for test suites and a powerful report generation engine. +.Pp +Kyua is for both developers and users, from the developer applying a +simple fix to a library to the system administrator deploying a new +release on a production machine. +.Pp +Kyua is able to execute test programs written with a plethora of testing +libraries and languages. +The test program library of choice is ATF, which +.Nm Ns 's +design originated from. +However, framework-less test programs and TAP-compliant test programs can also +be executed through +.Nm +.Ss Overview +As can be observed in the synopsis, the interface of +.Nm +implements a common subcommand-based interface. +The arguments to the tool specify, in this order: a set of common options +that all the commands accept, a required +.Ar command +name that specifies what +.Nm +should do, and +a set of possibly-optional +.Ar command_options +and +.Ar command_arguments +that are specific to the chosen command. +.Pp +The following options are recognized by all the commands. +Keep in mind that these must always be specified before the command name. +.Bl -tag -width XX +.It Fl -config Ar path , Fl c Ar path +Specifies the configuration file to process, which must be in the format +described in +.Xr kyua.conf 5 . +The special value +.Sq none +explicitly disables the loading of any configuration file. +.Pp +Defaults to +.Pa ~/.kyua/kyua.conf +if it exists, otherwise to +.Pa /etc/kyua/kyua.conf +if it exists, +or else to +.Sq none . +.It Fl -logfile Ar path +Specifies the location of the file to which +.Nm +will log run time events useful for postmortem debugging. +.Pp +The default depends on different environment variables as described in +.Sx Logging , +but typically the file will be stored within the user's home directory. +.It Fl -loglevel Ar level +Specifies the maximum logging level to record in the log file. +See +.Sx Logging +for more details. +.Pp +The default is +.Sq info . +.It Fl -variable Ar name=value , Fl v Ar name=value +Sets the +.Ar name +configuration variable to +.Ar value . +The values set through this option have preference over the values set in the +configuration file. +.Pp +The specified variable can either be a builtin variable or a test-suite +specific variable. +See +.Xr kyua.conf 5 +for more details. +.El +.Pp +The following commands are generic and do not have any relation to the execution +of tests or the inspection of their results: +.Bl -tag -width reportXjunitXX -offset indent +.It Ar about +Shows general program information. +See +.Xr kyua-about 1 . +.It Ar config +Inspects the values of the configuration variables. +See +.Xr kyua-config 1 . +.It Ar db-exec +Executes an arbitrary SQL statement on a results file and prints the +resulting table. +See +.Xr kyua-db-exec 1 . +.It Ar help +Shows usage information. +See +.Xr kyua-help 1 . +.El +.Pp +The following commands are used to generate reports based on the data previously +recorded in a results file: +.Bl -tag -width reportXjunitXX -offset indent +.It Ar report +Generates a plaintext report. +Combined with its +.Fl -verbose +flag and the ability to only display specific test cases, this command can also +be used to debug test failures post-facto on the console. +See +.Xr kyua-report 1 . +.It Ar report-html +Generates an HTML report. +See +.Xr kyua-report-html 1 . +.It Ar report-junit +Generates a JUnit report. +See +.Xr kyua-report-junit 1 . +.El +.Pp +The following commands are used to interact with a test suite: +.Bl -tag -width reportXjunitXX -offset indent +.It Ar debug +Executes a single test case in a controlled environment for debugging purposes. +See +.Xr kyua-debug 1 . +.It Ar list +Lists test cases defined in a test suite by a +.Xr kyuafile 5 +and, optionally, displays their metadata. +See +.Xr kyua-list 1 . +.It Ar test +Runs tests defined in a test suite by a +.Xr kyuafile 5 . +See +.Xr kyua-test 1 . +.El +.Ss Logging +.Nm +has a logging facility that collects all kinds of events at run time. +These events are always logged to a file so that the log is available when +it is most needed: right after a non-reproducible problem happens. +The only way to disable logging is by sending the log to +.Pa /dev/null . +.Pp +The location of the log file can be manually specified with the +.Fl -logfile +option, which applies to all commands. +If no file is explicitly specified, the location of the log files is chosen in +this order: +.Bl -enum -offset indent +.It +.Pa ${HOME}/.kyua/logs/ +if +.Va HOME +is defined. +.It +.Pa ${TMPDIR}/ +if +.Va TMPDIR +is defined. +.It +.Pa /tmp/ . +.El +.Pp +And the default naming scheme of the log files is: +.Sq <progname>.<timestamp>.log . +.Pp +The messages stored in the log file have a level (or severity) attached to +them. +These are: +.Bl -tag -width warningXX -offset indent +.It error +Fatal error messages. +The program generally terminates after these, either in a clean manner or by +crashing. +.It warning +Non-fatal error messages. +These generally report a condition that must be addressed but the application +can continue to run. +.It info +Informational messages. +These tell the user what the program was doing at a general level of +operation. +.It debug +Detailed informational messages. +These are often useful when debugging problems in the application, as they +contain lots of internal details. +.El +.Pp +The default log level is +.Sq info +unless explicitly overridden with +.Fl -loglevel . +.Pp +The log file is a plain text file containing one line per log record. +The format of each line is as follows: +.Bd -literal -offset indent +timestamp entry_type pid file:line: message +.Ed +.Pp +.Ar entry_type +can be one of: +.Sq E +for an error, +.Sq W +for a warning, +.Sq I +for an informational message and +.Sq D +for a debug message. +.Ss Bug reporting +If you think you have encountered a bug in +.Nm , +please take the time to let the developers know about it. +This will ensure that the bug is addressed and potentially fixed in the next +Kyua release. +.Pp +The first step in reporting a bug is to check if there already is a similar +bug in the database. +You can check what issues are currently in the database by going to: +.Bd -literal -offset indent +https://github.com/jmmv/kyua/issues/ +.Ed +.Pp +If there is no existing issue that describes an issue similar to the +one you are experiencing, you can open a new one by visiting: +.Bd -literal -offset indent +https://github.com/jmmv/kyua/issues/new/ +.Ed +.Pp +When doing so, please include as much detail as possible. +Among other things, explain what operating system and platform you are running +.Nm +on, what were you trying to do, what exact messages you saw on the screen, +how did you expect the program to behave, and any other details that you +may find relevant. +.Pp +Also, please include a copy of the log file corresponding to the problem +you are experiencing. +Unless you have changed the location of the log files, you can most likely +find them in +.Pa ~/.kyua/logs/ . +If the problem is reproducible, it is good idea to regenerate the log file +with an increased log level so as to provide more information. +For example: +.Bd -literal -offset indent +$ kyua --logfile=problem.log --loglevel=debug \\ + [rest of the command line] +.Ed +.Sh ENVIRONMENT +The following variables are recognized and can be freely tuned by the end user: +.Bl -tag -width COLUMNSXX +.It Va COLUMNS +The width of the screen, in number of characters. +.Nm +uses this to wrap long lines. +If not present, the width of the screen is determined from the terminal +stdout is connected to, and, if the guessing fails, this defaults to infinity. +.It Va HOME +Path to the user's home directory. +.Nm +uses this location to determine paths to configuration files and default log +files. +.It Va TMPDIR +Path to the system-wide temporary directory. +.Nm +uses this location to place the work directory of test cases, among other +things. +.Pp +The default value of this variable depends on the operating system. +In general, it is +.Pa /tmp . +.El +.Pp +The following variables are also recognized, but you should not need to set them +during normal operation. +They are only provided to override the value of built-in values, which is useful +when testing +.Nm +itself: +.Bl -tag -width KYUAXCONFDIRXX +.It Va KYUA_CONFDIR +Path to the system-wide configuration files for +.Nm . +.Pp +Defaults to +.Pa /etc/kyua . +.It Va KYUA_DOCDIR +Path to the location of installed documentation. +.Pp +Defaults to +.Pa /usr/share/doc/kyua . +.It Va KYUA_MISCDIR +Path to the location of the installed miscellaneous scripts and data +files provided by +.Nm . +.Pp +Defaults to +.Pa /usr/share/kyua/misc . +.It Va KYUA_STOREDIR +Path to the location of the installed store support files; e.g., the +directory containing the SQL database schema. +.Pp +Defaults to +.Pa /usr/share/kyua/store . +.El +.Sh FILES +.Bl -tag -width XXXX +.It Pa ~/.kyua/store/ +Default location for the results files. +.It Pa ~/.kyua/kyua.conf +User-specific configuration file. +.It Pa ~/.kyua/logs/ +Default location for the collected log files. +.It Pa /etc/kyua/kyua.conf +System-wide configuration file. +.El +.Sh EXIT STATUS +.Nm +returns 0 on success, 1 on a controlled error condition in the given +subcommand, 2 on a general unexpected error and 3 on a usage error. +.Pp +The documentation of the subcommands in the corresponding manual pages only +details the difference between a successful exit (0) and the detection of a +controlled error (1). +Even though when those manual pages do not describe any other exit statuses, +codes above 1 can be returned. +.Sh SEE ALSO +.Xr kyua.conf 5 , +.Xr kyuafile 5 , +.Xr atf 7 , +.Xr tests 7 +.Sh AUTHORS +For more details on the people that made +.Nm +possible and the license terms, run: +.Bd -literal -offset indent +$ kyua about +.Ed diff --git a/contrib/kyua/doc/kyua.conf.5 b/contrib/kyua/doc/kyua.conf.5 new file mode 100644 index 000000000000..0961043e9649 --- /dev/null +++ b/contrib/kyua/doc/kyua.conf.5 @@ -0,0 +1,141 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd February 20, 2015 +.Dt KYUA.CONF 5 +.Os +.Sh NAME +.Nm kyua.conf +.Nd Configuration file for the kyua tool +.Sh SYNOPSIS +.Fn syntax "int version" +.Pp +Variables: +.Va architecture , +.Va platform , +.Va test_suites , +.Va unprivileged_user . +.Sh DESCRIPTION +The configuration of Kyua is a simple collection of key/value pairs called +configuration variables. +There are configuration variables that have a special meaning to the runtime +engine implemented by +.Xr kyua 1 , +and there are variables that only have meaning in the context of particular +test suites. +.Pp +Configuration files are Lua scripts. +In their most basic form, their whole purpose is to assign values to +variables, but the user has the freedom to implement any logic he desires +to compute such values. +.Ss File versioning +Every +.Nm +file starts with a call to +.Fn syntax "int version" . +This call determines the specific schema used by the file so that future +backwards-incompatible modifications to the file can be introduced. +.Pp +Any new +.Nm +file should set +.Fa version +to +.Sq 2 . +.Ss Runtime configuration variables +The following variables are internally recognized by +.Xr kyua 1 : +.Bl -tag -width XX -offset indent +.It Va architecture +Name of the system architecture (aka processor type). +.It Va parallelism +Maximum number of test cases to execute concurrently. +.It Va platform +Name of the system platform (aka machine type). +.It Va unprivileged_user +Name or UID of the unprivileged user. +.Pp +If set, the given user must exist in the system and his privileges will be +used to run test cases that need regular privileges when +.Xr kyua 1 +is executed as root. +.El +.Ss Test-suite configuration variables +Each test suite is able to recognize arbitrary configuration variables, and +their type and meaning is specific to the test suite. +Because the existence and naming of these variables depends on every test +suite, this manual page cannot detail them; please refer to the documentation +of the test suite you are working with for more details on this topic. +.Pp +Test-suite specific configuration variables are defined inside the +.Va test_suites +dictionary. +The general syntax is: +.Bd -literal -offset indent +test_suites.<test_suite_name>.<variable_name> = <value> +.Ed +.Pp +where +.Va test_suite_name +is the name of the test suite, +.Va variable_name +is the name of the variable to set, and +.Va value +is a value. +The value can be a string, an integer or a boolean. +.Sh FILES +.Bl -tag -width XX +.It /usr/share/examples/kyua/kyua.conf +Sample configuration file. +.El +.Sh EXAMPLES +The following +.Nm +shows a simple configuration file that overrides a bunch of the built-in +.Xr kyua 1 +configuration variables: +.Bd -literal -offset indent +syntax(2) + +architecture = 'x86_64' +platform = 'amd64' +.Ed +.Pp +The following is a more complex example that introduces the definition of +per-test suite configuration variables: +.Bd -literal -offset indent +syntax(2) + +-- Assign built-in variables. +unprivileged_user = '_tests' + +-- Assign test-suite variables. All of these must be strings. +test_suites.NetBSD.file_systems = 'ffs ext2fs' +test_suites.X11.graphics_driver = 'vesa' +.Ed +.Sh SEE ALSO +.Xr kyua 1 diff --git a/contrib/kyua/doc/kyuafile.5 b/contrib/kyua/doc/kyuafile.5 new file mode 100644 index 000000000000..380f4cef2acd --- /dev/null +++ b/contrib/kyua/doc/kyuafile.5 @@ -0,0 +1,407 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd July 3, 2015 +.Dt KYUAFILE 5 +.Os +.Sh NAME +.Nm Kyuafile +.Nd Test suite description files +.Sh SYNOPSIS +.Fn atf_test_program "string name" "[string metadata]" +.Fn current_kyuafile +.Fn fs.basename "string path" +.Fn fs.dirname "string path" +.Fn fs.exists "string path" +.Fn fs.files "string path" +.Fn fs.is_absolute "string path" +.Fn fs.join "string path" "string path" +.Fn include "string path" +.Fn plain_test_program "string name" "[string metadata]" +.Fn syntax "int version" +.Fn tap_test_program "string name" "[string metadata]" +.Fn test_suite "string name" +.Sh DESCRIPTION +A test suite is a collection of test programs and is represented by a +hierarchical layout of test binaries on the file system. +Any subtree of the file system can represent a test suite, provided that it +includes one or more +.Nm Ns s , +which are the test suite definition files. +.Pp +A +.Nm +is a Lua script whose purpose is to describe the structure of the test +suite it belongs to. +To do so, the script has access to a collection of special functions provided +by +.Xr kyua 1 +as described in +.Sx Helper functions . +.Ss File versioning +Every +.Nm +file starts with a call to +.Fn syntax "int version" . +This call determines the specific schema used by the file so that future +backwards-incompatible modifications to the file can be introduced. +.Pp +Any new +.Nm +file should set +.Fa version +to +.Sq 2 . +.Ss Test suite definition +If the +.Nm +registers any test programs, +the +.Nm +must define the name of the test suite the test programs belong to by using the +.Fn test_suite +function at the very beginning of the file. +.Pp +The test suite name provided in the +.Fn test_suite +call tells +.Xr kyua 1 +which set of configuration variables from +.Xr kyua.conf 5 +to pass to the test programs at run time. +.Ss Test program registration +A +.Nm +can register test programs by means of a variety of +.Fn *_test_program +functions, all of which take the name of a test program and a set of +optional metadata properties that describe such test program. +.Pp +The test programs to be registered must live in the current directory; in +other words, the various +.Fn *_test_program +calls cannot reference test programs in other directories. +The rationale for this is to force all +.Nm +files to be self-contained, and to simplify their internal representation. +.Pp +.Em ATF test programs +are those that use the +.Xr atf 7 +libraries. +They can be registered with the +.Fn atf_test_program +table constructor. +This function takes the +.Fa name +of the test program and a collection of optional metadata settings for all +the test cases in the test program. +Any metadata properties defined by the test cases themselves override the +metadata values defined here. +.Pp +.Em Plain test programs +are those that return 0 on success and non-0 on failure; in general, most test +programs (even those that use fancy unit-testing libraries) behave this way and +thus also qualify as plain test programs. +They can be registered with the +.Fn plain_test_program +table constructor. +This function takes the +.Fa name +of the test program, an optional +.Fa test_suite +name that overrides the global test suite name, and a collection of optional +metadata settings for the test program. +.Pp +.Em TAP test programs +are those that implement the Test Anything Protocol. +They can be registered with the +.Fn tap_test_program +table constructor. +This function takes the +.Fa name +of the test program and a collection of optional metadata settings for the +test program. +.Pp +The following metadata properties can be passed to any test program definition: +.Bl -tag -width XX -offset indent +.It Va allowed_architectures +Whitespace-separated list of machine architecture names allowed by the test. +If empty or not defined, the test is allowed to run on any machine +architecture. +.It Va allowed_platforms +Whitespace-separated list of machine platform names allowed by the test. +If empty or not defined, the test is allowed to run on any machine +platform. +.It Va custom.NAME +Custom variable defined by the test where +.Sq NAME +denotes the name of the variable. +These variables are useful to tag your tests with information specific to +your project. +The values of such variables are propagated all the way from the tests to the +results files and later to any generated reports. +.Pp +Note that if the name happens to have dashes or any other special characters +in it, you will have to use a special Lua syntax to define the property. +Refer to the +.Sx EXAMPLES +section below for clarification. +.It Va description +Textual description of the test. +.It Va is_exclusive +If true, indicates that this test program cannot be executed along any other +programs at the same time. +Test programs that affect global system state, such as those that modify the +value of a +.Xr sysctl 8 +setting, must set themselves as exclusive to prevent failures due to race +conditions. +Defaults to false. +.It Va required_configs +Whitespace-separated list of configuration variables that the test requires +to be defined before it can run. +.It Va required_disk_space +Amount of available disk space that the test needs to run successfully. +.It Va required_files +Whitespace-separated list of paths that the test requires to exist before +it can run. +.It Va required_memory +Amount of physical memory that the test needs to run successfully. +.It Va required_programs +Whitespace-separated list of basenames or absolute paths pointing to executable +binaries that the test requires to exist before it can run. +.It Va required_user +If empty, the test has no restrictions on the calling user for it to run. +If set to +.Sq unprivileged , +the test needs to not run as root. +If set to +.Sq root , +the test must run as root. +.It Va timeout +Amount of seconds that the test is allowed to execute before being killed. +.El +.Ss Recursion +To reference test programs in another subdirectory, a different +.Nm +must be created in that directory and it must be included into the original +.Nm +by means of the +.Fn include +function. +.Pp +.Fn include +may only be called with a relative path and with at most one directory +component. +This is by design: Kyua uses the file system structure as the layout of the +test suite definition. +Therefore, each subdirectory in a test suite must include its own +.Nm +and each +.Nm +can only descend into the +.Nm Ns s +of immediate subdirectories. +.Pp +If you need to source a +.Nm +located in disjoint parts of your file system namespace, you will have to +create a +.Sq shadow tree +using symbolic links and possibly helper +.Nm Ns s +to plug the various subdirectories together. +See the +.Sx EXAMPLES +section below for details. +.Pp +Note that each file is processed in its own Lua environment: there is no +mechanism to pass state from one file to the other. +The reason for this is that there is no such thing as a +.Dq top-level +.Nm +in a test suite: the user has to be able to run the test suite from any +directory in a given hierarchy, and this execution must not depend on files +that live in parent directories. +.Ss Top-level Kyuafile +Every system has a top directory into which test suites get installed. +The default is +.Pa /usr/tests . +Within this directory live test suites, each of which is in an independent +subdirectory. +Each subdirectory can be provided separately by independent third-party +packages. +.Pp +Kyua allows running all the installed test suites at once in order to +provide comprehensive cross-component reports. +In order to do this, there is a special file in the top directory that knows +how to inspect the subdirectories in search for other Kyuafiles and include +them. +.Pp +The +.Sx FILES +section includes more details on where this file lives. +.Ss Helper functions +The +.Sq base , +.Sq string , +and +.Sq table +Lua modules are fully available in the context of a +.Nm . +.Pp +The following extra functions are provided by Kyua: +.Bl -tag -width XX -offset indent +.It Ft string Fn current_kyuafile +Returns the absolute path to the current +.Nm . +.It Ft string Fn fs.basename "string path" +Returns the last component of the given path. +.It Ft string Fn fs.dirname "string path" +Returns the given path without its last component or a dot if the path has +a single component. +.It Ft bool Fn fs.exists "string path" +Checks if the given path exists. +If the path is not absolute, it is relative to the directory containing the +.Nm +in which the call to this function occurs. +.It Ft iterator Fn fs.files "string path" +Opens a directory for scanning of its entries. +The returned iterator yields an entry on each call, and the entry is simply +the filename. +If the path is not absolute, it is relative to the directory containing the +.Nm +in which the call to this function occurs. +.It Ft is_absolute Fn fs.is_absolute "string path" +Returns true if the given path is absolute; false otherwise. +.It Ft join Fn fs.join "string path" "string path" +Concatenates the two paths. +The second path cannot be absolute. +.El +.Sh FILES +.Bl -tag -width XX +.It Pa /usr/tests/Kyuafile . +Top-level +.Nm +for the current system. +.It Pa /usr/share/examples/kyua/Kyuafile.top . +Sample file to serve as a top-level +.Nm . +.El +.Sh EXAMPLES +The following +.Nm +is the simplest you can define. +It provides a test suite definition and registers a couple of different test +programs using different interfaces: +.Bd -literal -offset indent +syntax(2) + +test_suite('first') + +atf_test_program{name='integration_test'} +plain_test_program{name='legacy_test'} +.Ed +.Pp +The following example is a bit more elaborate. +It introduces some metadata properties to the test program definitions and +recurses into a couple of subdirectories: +.Bd -literal -offset indent +syntax(2) + +test_suite('second') + +plain_test_program{name='legacy_test', + allowed_architectures='amd64 i386', + required_files='/bin/ls', + timeout=30} + +tap_test_program{name='privileged_test', + required_user='root'} + +include('module-1/Kyuafile') +include('module-2/Kyuafile') +.Ed +.Pp +The syntax to define custom properties may be not obvious if their names +have any characters that make the property name not be a valid Lua identifier. +Dashes are just one example. +To set such properties, do something like this: +.Bd -literal -offset indent +syntax(2) + +test_suite('FreeBSD') + +plain_test_program{name='the_test', + ['custom.FreeBSD-Bug-Id']='category/12345'} +.Ed +.Ss Connecting disjoint test suites +Now suppose you had various test suites on your file system and you would +like to connect them together so that they could be executed and treated as +a single unit. +The test suites we would like to connect live under +.Pa /usr/tests , +.Pa /usr/local/tests +and +.Pa ~/local/tests . +.Pp +We cannot create a +.Nm +that references these because the +.Fn include +directive does not support absolute paths. +Instead, what we can do is create a shadow tree using symbolic links: +.Bd -literal -offset indent +$ mkdir ~/everything +$ ln -s /usr/tests ~/everything/system-tests +$ ln -s /usr/local/tests ~/everything/local-tests +$ ln -s ~/local/tests ~/everything/home-tests +.Ed +.Pp +And then we create an +.Pa ~/everything/Kyuafile +file to drive the execution of the integrated test suite: +.Bd -literal -offset indent +syntax(2) + +test_suite('test-all-the-things') + +include('system-tests/Kyuafile') +include('local-tests/Kyuafile') +include('home-tests/Kyuafile') +.Ed +.Pp +Or, simply, you could reuse the sample top-level +.Nm +to avoid having to manually craft the list of directories into which to +recurse: +.Bd -literal -offset indent +$ cp /usr/share/examples/kyua/Kyuafile.top ~/everything/Kyuafile +.Ed +.Sh SEE ALSO +.Xr kyua 1 |