hwpmc(4): document debugging options

The debug options for hwpmc are not documented in detail anywhere, and
setting it up was error-prone the first time I had to figure it out (and
each time I've had to remember it). Add some explanation of the required
options and describe the kern.hwpmc.debugflags sysctl format.

Reviewed by:	emaste
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D40545

(cherry picked from commit 5fc97cc32553c4073c7dafd67a2d489b8af8563e)
(cherry picked from commit 89240e0ab60a29a47c9af9f952a09a7f2e1bb72c)

2 files changed, 143 insertions, 3 deletions

diff --git a/share/man/man4/hwpmc.4 b/share/man/man4/hwpmc.4
index 3a4be77085bc..bf9cb5e9e5e3 100644
--- a/share/man/man4/hwpmc.4
+++ b/share/man/man4/hwpmc.4
@@ -1,10 +1,12 @@
 .\" Copyright (c) 2003-2008 Joseph Koshy
-.\" Copyright (c) 2007 The FreeBSD Foundation
-.\" All rights reserved.
+.\" Copyright (c) 2007,2023 The FreeBSD Foundation
 .\"
 .\" Portions of this software were developed by A. Joseph Koshy under
 .\" sponsorship from the FreeBSD Foundation and Google, Inc.
 .\"
+.\" Portions of this documentation were written by Mitchell Horne
+.\" under sponsorship from the FreeBSD Foundation.
+.\"
 .\" Redistribution and use in source and binary forms, with or without
 .\" modification, are permitted provided that the following conditions
 .\" are met:
@@ -56,6 +58,17 @@ Alternatively, to compile this driver into the kernel:
 .Bd -ragged -offset indent
 .Cd "device hwpmc"
 .Ed
+.Pp
+To enable debugging features
+.Po see
+.Sx DEBUGGING
+.Pc :
+.Bd -ragged -offset indent
+.Cd "options KTR"
+.Cd "options KTR_COMPILE=(KTR_SUBSYS)"
+.Cd "options KTR_MASK=(KTR_SUBSYS)"
+.Cd "options HWPMC_DEBUG"
+.Ed
 .Sh DESCRIPTION
 The
 .Nm
@@ -527,6 +540,126 @@ The value for tunable
 .Va kern.hwpmc.nsamples
 was negative or greater than 65535.
 .El
+.Sh DEBUGGING
+The
+.Nm
+module can be configured to record trace entries using the
+.Xr ktr 4
+interface.
+This is useful for debugging the driver's functionality, primarily during
+development.
+This debugging functionality is not enabled by default, and requires
+recompiling the kernel and
+.Nm
+module after adding the following to the kernel config:
+.Bd -literal -offset indent
+.Cd options KTR
+.Cd options KTR_COMPILE=(KTR_SUBSYS)
+.Cd options KTR_MASK=(KTR_SUBSYS)
+.Cd options HWPMC_DEBUG
+.Ed
+.Pp
+This alone is not enough to enable tracing; one must also configure the
+.Va kern.hwpmc.debugflags
+.Xr sysctl 8
+variable, which provides fine-grained control over which types of events are
+logged to the trace buffer.
+.Pp
+.Nm
+trace events are grouped by 'major' and 'minor' flag types.
+The major flag names are as follows:
+.Pp
+.Bl -tag -width "sampling" -compact -offset indent
+.It cpu
+CPU events
+.It csw
+Context switch events
+.It logging
+Logging events
+.It md
+Machine-dependent/class-dependent events
+.It module
+Miscellaneous events
+.It owner
+PMC owner events
+.It pmc
+PMC management events
+.It process
+Process events
+.It sampling
+Sampling events
+.El
+.Pp
+The minor flags for each major flag group can vary.
+The individual minor flag names are:
+.Bd -ragged -offset indent
+allocaterow,
+allocate,
+attach,
+bind,
+config,
+exec,
+exit,
+find,
+flush,
+fork,
+getbuf,
+hook,
+init,
+intr,
+linktarget,
+mayberemove,
+ops,
+read,
+register,
+release,
+remove,
+sample,
+scheduleio,
+select,
+signal,
+swi,
+swo,
+start,
+stop,
+syscall,
+unlinktarget,
+write
+.Ed
+.Pp
+The
+.Va kern.hwpmc.debugflags
+variable is a string with a custom format.
+The string should contain a space-separated list of event specifiers.
+Each event specifier consists of the major flag name, followed by an equal sign
+(=), followed by a comma-separated list of minor event types.
+To track all events for a major group, an asterisk (*) can be given instead of
+minor event names.
+.Pp
+For example, to trace all allocation and release events, set
+.Va debugflags
+as follows:
+.Bd -literal -offset indent
+kern.hwpmc.debugflags="pmc=allocate,release md=allocate,release"
+.Ed
+.Pp
+To trace all events in the process and context switch major flag groups:
+.Bd -literal -offset indent
+kern.hwpmc.debugflags="process=* csw=*"
+.Ed
+.Pp
+To disable all trace events, set the variable to an empty string.
+.Bd -literal -offset indent
+kern.hwpmc.debugflags=""
+.Ed
+.Pp
+Trace events are recorded by
+.Xr ktr 4 ,
+and can be inspected at run-time using the
+.Xr ktrdump 8
+utility, or at the
+.Xr ddb 4
+prompt after a panic with the 'show ktr' command.
 .Sh ERRORS
 A command issued to the
 .Nm
@@ -752,8 +885,11 @@ operation is not being monitored by
 .Xr kenv 1 ,
 .Xr pmc 3 ,
 .Xr pmclog 3 ,
+.Xr ddb 4 ,
+.Xr ktr 4 ,
 .Xr kgmon 8 ,
 .Xr kldload 8 ,
+.Xr ktrdump 8 ,
 .Xr pmccontrol 8 ,
 .Xr pmcstat 8 ,
 .Xr sysctl 8 ,
diff --git a/sys/sys/pmc.h b/sys/sys/pmc.h
index 12583e67b062..153d24ca250f 100644
--- a/sys/sys/pmc.h
+++ b/sys/sys/pmc.h
@@ -1080,7 +1080,11 @@ extern struct pmc_driverstats pmc_stats;
 
 #define	__pmcdbg_used		/* unused variable annotation */
 
-/* debug flags, major flag groups */
+/*
+ * Debug flags, major flag groups.
+ *
+ * Please keep the DEBUGGING section of the hwpmc(4) man page in sync.
+ */
 struct pmc_debugflags {
 	int	pdb_CPU;
 	int	pdb_CSW;