path: root/sys/sys
diff options
authorAlexander Motin <mav@FreeBSD.org>2021-07-29 01:18:50 +0000
committerAlexander Motin <mav@FreeBSD.org>2021-07-29 02:00:29 +0000
commitaefe0a8c32d370f2fdd0d0771eb59f8845beda17 (patch)
tree27afafabe2d1c41df94233241336df76c938dfa0 /sys/sys
parent7cbf1de38e06663c76f4f075db31ea25f429f1b3 (diff)
Refactor/optimize cpu_search_*().
Remove cpu_search_both(), unused for many years. Without it there is less sense for the trick of compiling common cpu_search() into separate cpu_search_lowest() and cpu_search_highest(), so split them completely, making code more readable. While there, split iteration over children groups and CPUs, complicating code for very small deduplication. Stop passing cpuset_t arguments by value and avoid some manipulations. Since MAXCPU bump from 64 to 256, what was a single register turned into 32-byte memory array, requiring memory allocation and accesses. Splitting struct cpu_search into parameter and result parts allows to even more reduce stack usage, since the first can be passed through on recursion. Remove CPU_FFS() from the hot paths, precalculating first and last CPU for each CPU group in advance during initialization. Again, it was not a problem for 64 CPUs before, but for 256 FFS needs much more code. With these changes on 80-thread system doing ~260K uncached ZFS reads per second I observe ~30% reduction of time spent in cpu_search_*(). MFC after: 1 month
Diffstat (limited to 'sys/sys')
1 files changed, 2 insertions, 0 deletions
diff --git a/sys/sys/smp.h b/sys/sys/smp.h
index a971ffbbd91b..cee1199015a7 100644
--- a/sys/sys/smp.h
+++ b/sys/sys/smp.h
@@ -81,6 +81,8 @@ struct cpu_group {
struct cpu_group *cg_child; /* Optional children groups. */
cpuset_t cg_mask; /* Mask of cpus in this group. */
int32_t cg_count; /* Count of cpus in this group. */
+ int32_t cg_first; /* First cpu in this group. */
+ int32_t cg_last; /* Last cpu in this group. */
int16_t cg_children; /* Number of children groups. */
int8_t cg_level; /* Shared cache level. */
int8_t cg_flags; /* Traversal modifiers. */