net: Remove vlan metadata on pcp / vlan encapsulation

For oubound traffic, the flag M_VLANTAG is set in mbuf packet header to
indicate the underlaying interface do hardware VLAN tag insertion if
capable, otherwise the net stack will do 802.1Q encapsulation instead.

Commit 868aabb4708d introduced per-flow priority which set the priority ID
in the mbuf packet header. There's a corner case that when the driver is
disabled to do hardware VLAN tag insertion, and the net stack do 802.1Q
encapsulation, then it will result double tagged packets if the driver do
not check the enabled capability (hardware VLAN tag insertion).

Unfortunately some drivers, currently known cxgbe(4) re(4) ure(4) igc(4)
and vmx(4), have this issue. From a quick review for other interface
drivers I believe a lot more drivers have the same issue. It makes more
sense to fix in net stack than to try to change every single driver.

PR:		270736
Reviewed by:	kp
Approved by:	re (gjb)
Fixes:	868aabb4708d Add IP(V6)_VLAN_PCP to set 802.1 priority per-flow
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D39499

(cherry picked from commit b22aae410bc7e4e9a6b43e556dc34be72deadb65)

1 files changed, 5 insertions, 2 deletions

diff --git a/sys/net/if_ethersubr.c b/sys/net/if_ethersubr.c
index 63ea5e7591c2..ab274eeb88bf 100644
--- a/sys/net/if_ethersubr.c
+++ b/sys/net/if_ethersubr.c
@@ -446,9 +446,11 @@ ether_set_pcp(struct mbuf **mp, struct ifnet *ifp, uint8_t pcp)
 	struct ether_header *eh;
 
 	eh = mtod(*mp, struct ether_header *);
-	if (ntohs(eh->ether_type) == ETHERTYPE_VLAN ||
-	    ntohs(eh->ether_type) == ETHERTYPE_QINQ)
+	if (eh->ether_type == htons(ETHERTYPE_VLAN) ||
+	    eh->ether_type == htons(ETHERTYPE_QINQ)) {
+		(*mp)->m_flags &= ~M_VLANTAG;
 		return (true);
+	}
 
 	qtag.vid = 0;
 	qtag.pcp = pcp;
@@ -1463,6 +1465,7 @@ ether_8021q_frame(struct mbuf **mp, struct ifnet *ife, struct ifnet *p,
 			if_printf(ife, "unable to prepend 802.1Q header");
 			return (false);
 		}
+		(*mp)->m_flags &= ~M_VLANTAG;
 	}
 	return (true);
 }