Bug Fixes:
* Verify that an ENA ring is in netmap only in native mode

Minor Changes:
* Move parenthesis to correct place in switch
* Add comment
* Reorder define

Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55698
Sponsored by: Amazon, Inc.

(cherry picked from commit 96c5eaf0ac6b98d0832e1037d672064de43a7e00)

netmap operates in two modes:
1) Emulated - netmap handling is done by the network stack, the
NIC driver operates transparently to netmap.
2) Native - netmap management is done by the NIC driver.

When checking whether a specific ENA ring is running in netmap
mode, only the following checks were done:
1. IFCAP_NETMAP - Check whether netmap capability is enabled on
the device.
2. NKR_NETMAP_ON - Check whether netmap is actively using this
ring.

The above checks implied that the netmap mode is native and the
ENA driver needs to handle the netmap logic.
The code was missing an explicit check on whether native mode
is actually on (NAF_NATIVE).
This led to a case where though emulated mode was used and
a netmap application was turned on, the ENA driver still managed
netmap logic partially and caused missing buffers and lack of
refill as part of the datapath.

Note: Enabling netmap emulated mode is insufficient and there's
a need to load a netmap program in order to trigger this use-case.

Add an explicit check of whether NAF_NATIVE mode is set.

The issue was reported in [1].

[1]: https://github.com/amzn/amzn-drivers/issues/361

Fixes: 358bcc4c6cde ("Add support for ENA NETMAP partial initialization")
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55697
Sponsored by: Amazon, Inc.

(cherry picked from commit 97e84c587d6f86aa883720296449b380adcf6915)

1. Move parenthesis to correct place in switch and fix include order
2. Add comment at the end of an ifdef for clarity
3. Change include order.

Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55696
Sponsored by: Amazon, Inc.

(cherry picked from commit 2667a8454cff5896c7b467c78cd4ace5ad40f5eb)

Changes since 2.8.0:

Bug Fixes:
* Fix LLQ normal width misconfiguration
* Check for errors when detaching children first, not last

Minor Changes:
* Remove \n from sysctl description

Approved by: cperciva (mentor)
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D50041

(cherry picked from commit 59b30c1a864ee8a22c2e9912301cb88674f714c9)

Patch 0a33c047a443 introduced new values to
hw.ena.force_large_llq_header. The default value of 2 means no
preference, while 0 and 1 act as the previous false and true
respectively, which allowed forcefully setting regular or large LLQ.

There are 2 ways to force the driver to select regular LLQ:

1. Setting hw.ena.force_large_llq_header = 0 via sysctl.
2. Turning on ena express, which makes the recommendation by the FW to
   be regular LLQ.

When the device supports large LLQ but the driver is forced to
regular LLQ, llq_config->llq_ring_entry_size_value is never initialized
and since it is a variable allocated on the stack, it stays garbage.

Since this variable is involved in calculating max_entries_in_tx_burst,
it could cause the maximum burst size to be zero. This causes the driver
to ignore the real maximum burst size of the device, leading to driver
resets in devices that have a maximum burst size (Nitro v4 and on. see
[1] for more information).

In case the garbage value is 0, the calculation of
max_entries_in_tx_burst divides by 0 and causes kernel panic.

The patch modifies the logic to take into account all use-cases and
ensure that the relevant fields are properly initialized.

[1]: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-nitro-instances.html

Fixes: 0a33c047a443 ("ena: Support LLQ entry size recommendation from device")
Approved by: cperciva (mentor)
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D50040

(cherry picked from commit 56c45700f2ae15755358f2da8266247613c564df)

sysctl(8) prints a newline after the description, no need for this extra
newline.

MFC after:	1 week

(cherry picked from commit f2233ac33ab64f9bb03370c097af97f26dd0fca1)
(cherry picked from commit 1f6936723e1051e92fb9b658f5f6fa13c2001cee)

Features:
* Add support for device request reset message over AENQ
* Support LLQ entry size recommendation from device
* Support max large LLQ depth from the device
* Expand PHC infrastructures
* Configuration notification support

Bug Fixes:
* Fix leaking ifmedia resources on detach
* Fix netmap socket chain unmapping issue
* Properly reinit netmap structs upon sysctl changes
* Correctly count missing TX completions

Minor Changes:
* Add reset reason for corrupted TX/RX completion descriptors
* Add reset reason for missing admin interrupts
* Improve reset reason statistics
* Update licenses

Approved by: cperciva (mentor)
Sponsored by: Amazon, Inc.

(cherry picked from commit ce4cc746bb4171a131ab9099947a500c0de18ff4)

ifmedia_add() allocates an ifmedia_entry during ena_attach.
Current code doesn't release this memory during ena_detach()

This commit calls ifmedia_removeall() to properly free the
allocated memory during ena_detach().

Also, in case ena_attach fails, we need to detach ifmedia
which was allocated within ena_setup_ifnet().

This bug was first described in:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278100

Reviewed by: zlei
Approved by: cperciva (mentor)
Sponsored by: Amazon, Inc.

(cherry picked from commit 449496eb28daec8d5b852fa4be1e337c2957345c)

Large LLQ depth size is currently calculated by dividing the maximum
possible size of LLQ by 2.
In newer paltforms, starting from r8g the size of BAR2,
which contains LLQ, will be increased, and the maximum depth of
wide LLQ will be set according to a value set by the device, instead of
hardcoded division by 2.

The new value will be stored by the device in max_wide_llq_depth field
for drivers that expose ENA_ADMIN_LLQ_FEATURE_VERSION_1 or higher to
the device.

There is an assumption that max_llq_depth >= max_wide_llq_depth, since
they both use the same bar, and if it is possible to have a wide LLQ
of size max_wide_llq_depth, it is possible to have a normal LLQ of the
same size, since it will occupy half of the space.

Also moved the large LLQ case calculation of max_tx_queue_size
before its rounddown.

Approved by: cperciva (mentor)
Sponsored by: Amazon, Inc.

(cherry picked from commit d0419551d96c8f995bdf6388a8e69684be33f9b5)

This commit adds support for receiving LLQ entry size recommendation
from the device. The driver will use the recommended entry size, unless
the user specifically chooses to use regular or large LLQ entry.

Also added enum ena_llq_header_size_policy_t and llq_plociy field in
order to support the new feature.

Approved by: cperciva (mentor)
Sponsored by: Amazon, Inc.

(cherry picked from commit b1c38df05d79c81ee1e9fd0942774820a4ffcb63)