.\"
.\" Copyright (c) 2009-2019 Dell EMC Isilon http://www.isilon.com/
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice(s), this list of conditions and the following disclaimer as
.\"    the first lines of this file unmodified other than the possible
.\"    addition of one or more copyright notices.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice(s), this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY
.\" EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
.\" DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
.\" CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
.\" DAMAGE.
.\"
.Dd June 6, 2019
.Dt FAIL 9
.Os
.Sh NAME
.Nm DEBUG_FP ,
.Nm KFAIL_POINT_CODE ,
.Nm KFAIL_POINT_CODE_FLAGS ,
.Nm KFAIL_POINT_CODE_COND ,
.Nm KFAIL_POINT_ERROR ,
.Nm KFAIL_POINT_EVAL ,
.Nm KFAIL_POINT_DECLARE ,
.Nm KFAIL_POINT_DEFINE ,
.Nm KFAIL_POINT_GOTO ,
.Nm KFAIL_POINT_RETURN ,
.Nm KFAIL_POINT_RETURN_VOID ,
.Nm KFAIL_POINT_SLEEP_CALLBACKS ,
.Nm fail_point
.Nd fail points
.Sh SYNOPSIS
.In sys/fail.h
.Fn KFAIL_POINT_CODE "parent" "name" "code"
.Fn KFAIL_POINT_CODE_FLAGS "parent" "name" "flags" "code"
.Fn KFAIL_POINT_CODE_COND "parent" "name" "cond" "flags" "code"
.Fn KFAIL_POINT_ERROR "parent" "name" "error_var"
.Fn KFAIL_POINT_EVAL "name" "code"
.Fn KFAIL_POINT_DECLARE "name"
.Fn KFAIL_POINT_DEFINE "parent" "name" "flags"
.Fn KFAIL_POINT_GOTO "parent" "name" "error_var" "label"
.Fn KFAIL_POINT_RETURN "parent" "name"
.Fn KFAIL_POINT_RETURN_VOID "parent" "name"
.Fn KFAIL_POINT_SLEEP_CALLBACKS "parent" "name" "pre_func" "pre_arg" "post_func" "post_arg" "code"
.Sh DESCRIPTION
Fail points are used to add code points where errors may be injected
in a user controlled fashion.
Fail points provide a convenient wrapper around user-provided error
injection code, providing a
.Xr sysctl 9
MIB, and a parser for that MIB that describes how the error
injection code should fire.
.Pp
The base fail point macro is
.Fn KFAIL_POINT_CODE
where
.Fa parent
is a sysctl tree (frequently
.Sy DEBUG_FP
for kernel fail points, but various subsystems may wish to provide
their own fail point trees), and
.Fa name
is the name of the MIB in that tree, and
.Fa code
is the error injection code.
The
.Fa code
argument does not require braces, but it is considered good style to
use braces for any multi-line code arguments.
Inside the
.Fa code
argument, the evaluation of
.Sy RETURN_VALUE
is derived from the
.Fn return
value set in the sysctl MIB.
.Pp
Additionally,
.Fn KFAIL_POINT_CODE_FLAGS
provides a
.Fa flags
argument which controls the fail point's behaviour.
This can be used to e.g., mark the fail point's context as non-sleepable,
which causes the
.Sy sleep
action to be coerced to a busy wait.
The supported flags are:
.Bl -ohang -offset indent
.It FAIL_POINT_USE_TIMEOUT_PATH
Rather than sleeping on a
.Fn sleep
call, just fire the post-sleep function after a timeout fires.
.It FAIL_POINT_NONSLEEPABLE
Mark the fail point as being in a non-sleepable context, which coerces
.Fn sleep
calls to
.Fn delay
calls.
.El
.Pp
Likewise,
.Fn KFAIL_POINT_CODE_COND
supplies a
.Fa cond
argument, which allows you to set the condition under which the fail point's
code may fire.
This is equivalent to:
.Bd -literal
	if (cond)
		KFAIL_POINT_CODE_FLAGS(...);

.Ed
See
.Sx SYSCTL VARIABLES
below.
.Pp
The remaining
.Fn KFAIL_POINT_*
macros are wrappers around common error injection paths:
.Bl -inset
.It Fn KFAIL_POINT_RETURN parent name
is the equivalent of
.Sy KFAIL_POINT_CODE(..., return RETURN_VALUE)
.It Fn KFAIL_POINT_RETURN_VOID parent name
is the equivalent of
.Sy KFAIL_POINT_CODE(..., return)
.It Fn KFAIL_POINT_ERROR parent name error_var
is the equivalent of
.Sy KFAIL_POINT_CODE(..., error_var = RETURN_VALUE)
.It Fn KFAIL_POINT_GOTO parent name error_var label
is the equivalent of
.Sy KFAIL_POINT_CODE(..., { error_var = RETURN_VALUE; goto label;})
.El
.Pp
You can also introduce fail points by separating the declaration,
definition, and evaluation portions.
.Bl -inset
.It Fn KFAIL_POINT_DECLARE name
is used to declare the
.Sy fail_point
struct.
.It Fn KFAIL_POINT_DEFINE parent name flags
defines and initializes the
.Sy fail_point
and sets up its
.Xr sysctl 9 .
.It Fn KFAIL_POINT_EVAL name code
is used at the point that the fail point is executed.
.El
.Sh SYSCTL VARIABLES
The
.Fn KFAIL_POINT_*
macros add sysctl MIBs where specified.
Many base kernel MIBs can be found in the
.Sy debug.fail_point
tree (referenced in code by
.Sy DEBUG_FP ) .
.Pp
The sysctl variable may be set in a number of ways:
.Bd -literal
  [<pct>%][<cnt>*]<type>[(args...)][-><more terms>]
.Ed
.Pp
The <type> argument specifies which action to take; it can be one of:
.Bl -tag -width ".Dv return"
.It Sy off
Take no action (does not trigger fail point code)
.It Sy return
Trigger fail point code with specified argument
.It Sy sleep
Sleep the specified number of milliseconds
.It Sy panic
Panic
.It Sy break
Break into the debugger, or trap if there is no debugger support
.It Sy print
Print that the fail point executed
.It Sy pause
Threads sleep at the fail point until the fail point is set to
.Sy off
.It Sy yield
Thread yields the cpu when the fail point is evaluated
.It Sy delay
Similar to sleep, but busy waits the cpu.
(Useful in non-sleepable contexts.)
.El
.Pp
The <pct>% and <cnt>* modifiers prior to <type> control when
<type> is executed.
The <pct>% form (e.g. "1.2%") can be used to specify a
probability that <type> will execute.
This is a decimal in the range (0, 100] which can specify up to
1/10,000% precision.
The <cnt>* form (e.g. "5*") can be used to specify the number of
times <type> should be executed before this <term> is disabled.
Only the last probability and the last count are used if multiple
are specified, i.e. "1.2%2%" is the same as "2%".
When both a probability and a count are specified, the probability
is evaluated before the count, i.e. "2%5*" means "2% of the time,
but only 5 times total".
.Pp
The operator -> can be used to express cascading terms.
If you specify <term1>-><term2>, it means that if <term1> does not
.Ql execute ,
<term2> is evaluated.
For the purpose of this operator, the
.Fn return
and
.Fn print
operators are the only types that cascade.
A
.Fn return
term only cascades if the code executes, and a
.Fn print
term only cascades when passed a non-zero argument.
A pid can optionally be specified.
The fail point term is only executed when invoked by a process with a
matching p_pid.
.Sh EXAMPLES
.Bl -tag -width Sy
.It Sy sysctl debug.fail_point.foobar="2.1%return(5)"
21/1000ths of the time, execute
.Fa code
with RETURN_VALUE set to 5.
.It Sy sysctl debug.fail_point.foobar="2%return(5)->5%return(22)"
2/100ths of the time, execute
.Fa code
with RETURN_VALUE set to 5.
If that does not happen, 5% of the time execute
.Fa code
with RETURN_VALUE set to 22.
.It Sy sysctl debug.fail_point.foobar="5*return(5)->0.1%return(22)"
For 5 times, return 5.
After that, 1/1000th of the time, return 22.
.It Sy sysctl debug.fail_point.foobar="0.1%5*return(5)"
Return 5 for 1 in 1000 executions, but only 5 times total.
.It Sy sysctl debug.fail_point.foobar="1%*sleep(50)"
1/100th of the time, sleep 50ms.
.It Sy sysctl debug.fail_point.foobar="1*return(5)[pid 1234]"
Return 5 once, when pid 1234 executes the fail point.
.El
.Sh AUTHORS
.An -nosplit
This manual page was written by
.Pp
.An Matthew Bryan Aq Mt matthew.bryan@isilon.com
and
.Pp
.An Zach Loafman Aq Mt zml@FreeBSD.org .
.Sh CAVEATS
It is easy to shoot yourself in the foot by setting fail points too
aggressively or setting too many in combination.
For example, forcing
.Fn malloc
to fail consistently is potentially harmful to uptime.
.Pp
The
.Fn sleep
sysctl setting may not be appropriate in all situations.
Currently,
.Fn fail_point_eval
does not verify whether the context is appropriate for calling
.Fn msleep .
You can force it to evaluate a
.Sy sleep
action as a
.Sy delay
action by specifying the
.Sy FAIL_POINT_NONSLEEPABLE
flag at the point the fail point is declared.