mlx5: support DevX UMEM page size bitmap

rdma-core's mlx5dv_devx_umem_reg_ex() passes the page sizes the caller
can use in MLX5_IB_ATTR_DEVX_UMEM_REG_PGSZ_BITMAP.  Add the attribute and
choose the UMEM page size, page offset and MTT count the way Linux commit
7610ab57de56 ("RDMA/mlx5: Allow larger pages in DevX umem") does.

Linux looks at the DMA list only.  The UMEM is mapped at an IOVA equal to
the DMA address of its first byte, ib_umem_find_best_pgsz() picks the
largest page size of the bitmap that this mapping allows, and
devx_umem_find_best_pgsize() halves it while the first DMA address is not
aligned to it or the length is not a multiple of it, stopping at
PAGE_SIZE, even if that is below every size in the bitmap.

The FreeBSD OFED layer has none of the ib_umem helpers this relies on,
so local copies of the current Linux ones are added.
ib_umem_find_best_pgoff() and ib_umem_dma_offset() come from Linux
commit b045db62f6f6 ("RDMA/mlx5: Use ib_umem_find_best_pgoff() for
SRQ"), ib_umem_num_dma_blocks() from Linux commit a665aca89a41
("RDMA/umem: Split ib_umem_num_pages() into ib_umem_num_dma_blocks()").
The ib_umem_find_best_pgsz() logic includes Linux commit 486055f5e09d
("RDMA/core: Fix best page size finding when it can cross SG entries"),
without which no page size above 4K would ever be found, as FreeBSD's
ib_umem_get() maps every page as an SG entry of its own, and Linux
commit 09ea6837a043 ("RDMA/umem: Be careful about boundary conditions in
ib_umem_find_best_pgsz()").

Without the attribute the bitmap defaults to GENMASK_ULL(63, 12), as in
Linux.  The attribute is read with uverbs_copy_from(), FreeBSD's
uverbs_get_const_default() works on an s64 and rejects any value with bit
63 set, the default included.  The page size is kept in a u64, Linux
returns it as an unsigned int and so fails registrations that would use
pages of 4G or more.

Reviewed by:  kib
Sponsored by: Nvidia networking
MFC after:    1 month

0 files changed, 0 insertions, 0 deletions