dwatch: add nine diagnostic modules; grow errno, io, proc, sched - src

diff options

author	Devin Teske <dteske@FreeBSD.org>	2026-07-28 23:32:57 +0000
committer	Devin Teske <dteske@FreeBSD.org>	2026-07-28 23:32:57 +0000
commit	434283fda99e89af20c3fe95fde427624ae96d04 (patch)
tree	d9f71fd0c3339ee607bc60257ae2d2abeca95b80
parent	14f5fd2395e3f1c3fadeee8b95756c142f7dcb0b (diff)

dwatch: add nine diagnostic modules; grow errno, io, proc, sched

Grow the module collection to answer, each with a single command, the first questions asked when diagnosing a sick system: why is my application stalling, where is the kernel fighting over locks, what file could it not find, why is this process getting EPERM, what killed my process, will that fatal signal actually leave a core behind, what was my process stuck on, where is my kernel memory going, who is creating or entering jails, is the network slow because TCP is resending, how long did my thread wait to run, and is the disk itself slow. Every module keeps to the house style: invocation-name overloading through hard links, predicate-only D with inline lookup tables (no if-statements), and stable providers only (syscall, proc, sched, io, dtmalloc, and the lockstat, vfs, priv, and mib SDT providers), so the modules remain drop-in compatible with older releases (the one documented exception is noted below). No kernel changes: new and extended profiles under cddl/usr.sbin/dwatch/libexec plus one libdtrace inline table (priv.d). slow (slow-fsync, slow-open, slow-read, slow-syscall, slow-write, or any slow-NAME by new link) records syscall entry timestamps in thread-local storage and prints, at return, any call whose latency meets a threshold (DWATCH_SLOW_MS, default 100), naming the syscall, the elapsed time to the microsecond, and any errno returned. The bare profile watches a curated set of filesystem-related calls expected to be fast; slow-syscall watches everything; unrecognized invocation names fall through to syscall::NAME:return with the matching entry probe derived mechanically from the return probe list. lock (lock-adaptive, lock-block, lock-lockmgr, lock-rw, lock-spin, lock-sx, lock-thread) rides the dtrace_lockstat(4) block and spin probes, printing the held-off thread (free from the standard event tag), the holdoff duration, the lock class, the lo_name of the lock through a single cast of arg0 to struct lock_object (the first member of every kernel lock), and reader/writer intent on the probes that report it. Holdoffs shorter than DWATCH_LOCK_MS (default 1; 0 shows everything) are suppressed. namei (namei-enoent, namei-entry, namei-failure) records the pathname at vfs:namei:lookup:entry and reports it with the result at return. Unlike the vop_lookup profile, which reconstructs paths from the name cache one component at a time, this sees the whole path exactly as the process requested it. namei-enoent hunts file-not-found storms -- the single most common use of truss(1) -- without stopping the victim. priv (priv-err, priv-ok) watches priv_check(9) verdicts, naming the exact privilege denied -- something no syscall tracer can see, because by the time EPERM surfaces the priv(9) value is gone. The number is decoded by priv_string[], a new libdtrace inline table in the errno.d and signal.d tradition, mechanically generated from sys/priv.h (247 entries) and installed to /usr/lib/dtrace where dtrace(1) auto-loads it; on older releases it is a drop-in file like the module itself. coredump (coredump-top) watches for delivery of signals whose default action produces a core, per the SIGPROP_CORE entries of the sigproptbl in kern_sig.c, and renders a verdict the same way and in the same order the kernel will decide it: ignored or caught per the target's struct sigacts, then the coredump() gauntlet of kern.coredump, kern.sugid_coredump vs P_SUGID, procctl(2) PROC_TRACE_CTL, and RLIMIT_CORE -- the sysctl knobs read live through kernel globals. Where a coredump-worthy signal will produce no core, the verdict says precisely which policy ate it. coredump-top maintains a cumulative catalog of coredump-worthy signals by process and signal, refreshed every 3 seconds in the style of systop; combine the event profile with `-O cmd' to capture state as each event occurs. hang (hang-top) pairs sched:::sleep with sched:::wakeup through a tid-keyed timestamp array and prints, as each thread wakes, any sleep that meets a threshold (DWATCH_HANG_MS, default 1000), naming the sleeper in the details and the waker in the standard event tag. This is the blocking the slow module structurally cannot see: a syscall that never returns never reports its latency, while hang reports the moment the wait ends, with the full duration. hang-top maintains a cumulative catalog of long sleeps by process (count and maximum) in the style of coredump-top. jail (jail-attach, jail-get, jail-remove, jail-set) watches the jail management plane -- jail(2), jail_set(2), jail_get(2), jail_attach(2), and jail_remove(2) -- naming the operation, the jail id (taken from the entry argument for attach/remove, from the return value for the others), and any errno. Complements the dwatch `-j jail' filter, which scopes any profile to processes inside one jail; this watches who manipulates jails, from any jail or none. dtmalloc (dtmalloc-top, or any dtmalloc-NAME by new link) rides the dtmalloc provider (one malloc and one free probe per malloc(9) type). The event profile prints allocations and frees meeting a size threshold (DWATCH_MALLOC_MIN, default 65536) -- who is allocating huge kernel buffers. dtmalloc-top maintains a running catalog of net bytes and outstanding allocation balance by type, sorted by net bytes so leak suspects rise: a type that climbs without bound while the system is in steady state is the suspect. The catalog reflects activity since the watch began, and is honest about caches holding what they allocate. mib (tcp-retransmit, or any mib-NAME by new link) rides the per-counter mib SDT probes of the network stack. The tcp-retransmit profile curates the counters that signal send-path congestion or loss -- data packet retransmissions, unnecessary retransmissions, retransmit timer expirations, and connections dropped by retransmit exhaustion -- decoded through an inline description table, answering "is this network slow because TCP is resending?" as events with process context rather than netstat(1) deltas. NB: the mib probes exist only in kernels built with options KDTRACE_MIB_SDT (default in -CURRENT via std.debug); the module documents this and dtrace(1) refuses the script elsewhere, making the dependency self-announcing. Four existing modules gain personalities. proc grows proc-signal-fatal, filtering signal-send to signals whose default disposition terminates the receiver, most-notably including kernel-generated SIGSEGV/SIGBUS/SIGILL/SIGFPE that no kill(2) watcher will ever see. errno now reads its invocation name: errno-NAME shows only syscalls returning that errno, where NAME is a symbolic name from errno.d or a number; links are installed for errno-EACCES, errno-ECAPMODE, errno-ENOENT, errno-ENOTCAPABLE, and errno-EPERM (the latter pairs covering capsicum(4) capability-mode violations), and any other errno needs only a new link. sched grows sched-latency, recording a timestamp at sched:::enqueue keyed by tid and printing at sched:::on-cpu any run-queue wait meeting a threshold (DWATCH_SCHED_MS, default 10) -- the literal measurement of scheduler delay on a system with idle CPU that still feels sluggish. io grows io-slow, pairing io:::start with io:::done through a bio-keyed timestamp array and printing any request that meets a threshold (DWATCH_IO_MS, default 100), naming the device, command, size, and elapsed time; watched against zvols and a pool's leaf vdevs this brackets where in a ZFS stack the time is going, without touching unstable providers. Document all of the above plus the DWATCH_HANG_MS, DWATCH_IO_MS, DWATCH_LOCK_MS, DWATCH_MALLOC_MIN, DWATCH_SCHED_MS, and DWATCH_SLOW_MS knobs in dwatch(1). All 46 new invocation names were exercised through `dwatch -d' with a profile-path sandbox emulating the installed hard links: every one sources cleanly and emits the intended D -- probe selection per alias, entry/return and sleep/wakeup pairing through thread-local and global associative arrays, threshold and mask predicates picking up their knobs, aggregation clauses and printa column layout in the -top profiles, multi-line predicate rendering, and `-t' correctly displacing each module's default test were verified by inspection of the generated scripts. Invocations untouched by this pass generate D identical to their previous output. Modules pass sh -n, fit 80 columns, and dwatch.1 passes mandoc -Tlint with no new warnings. A validation harness performing a `dwatch -e' compile per profile against the live kernel globs every staged profile for runs wherever the dtrace device is present. Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D58093

Diffstat

-rw-r--r--

cddl/lib/libdtrace/Makefile

-rwxr-xr-x

cddl/lib/libdtrace/mkpriv

119

-rw-r--r--

cddl/usr.sbin/dwatch/dwatch.1

105

-rw-r--r--

cddl/usr.sbin/dwatch/libexec/Makefile

-rw-r--r--