iflib: Use a bounded buf_ring for simple_txHEADmain

Implement buf_ring/drbr deferred transmit in iflib. This is intended
to allow the new simpler code path to replace mp_ring. This patch
makes the simple_tx outperform mp_ring by a wide margin when CPU is
the bottleneck (eg, cannot fill the NIC). See graphs at:
https://people.freebsd.org/~gallatin/mpring_vs_simple_tx

Note that the buf ring is used for contention, not capacity. Eg,
it is used as a place for contending threads to put packets without
waiting for a mutex. It is not designed to act as a software ring
on top of the hardware descriptors provided by the underlying NIC
driver. "stranded packets" are exceedingly rare due to the fact that
if there is enough load to use the buf_ring, there will probably be
more load coming that can be a drainer. Not scheduling a gtask to
drain is intentional, and we really on the timer as a fallback.

One thing I noticed while developing this patch is that a simple mutex
with no deferral generally outperformed both mp_ring and drbr at high
levels of contention for the same queue. This is inherent in a
bounded MPSC queue where multiple producers are contending on claiming
ring entries. So I came up with the idea of bounding the number of
producers such that the deferral ring would devolve to a mutex when
contention was high.

Identifying the crossover point in a general way was hard. On
different machines, the point between a mutex and a deferral ring was
very different and also depended on the placement of the producers. I
eventually realized that on the large AMD EPYC servers that I was
testing on, the crossover point generally coincided with the work
spilling into another CCX. So I developed an approach where we limit
the number of producers by limiting simultanious producers to the same
L3 AND by bounding the number of simultanious producers.

This limit can be adjusted via the sysctl net.iflib.max_producers.

The number of packets drained from the deferral ring is limited
by net.iflib.simple_drain_quota. The intent is to process just
enough packets in the gtaskq context so as to make space to
allow threads to make progress. The task also uses a trylock
so as to avoid blocking, waiting for a thread to drain.

This change also enables ALTQ support for simple tx.

Reviewed by: kbowling
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D58901

0 files changed, 0 insertions, 0 deletions