August 4, 2017
by Pablo Narváez
Last night, I was working on my VXLAN lab and wanted to make sure the VM connections were setup correctly. I opted to use the Link Layer Discovery Protocol or LLDP on the switches to validate the network assignment (net-x) for each VM .
To my surprise, the “show lldp neighbors” command displayed no information about the neighboring devices.
spine01# show lldp neighbors LLDP neighbors: -------------------------------------
To make sure VMs were exchanging LLDP frames, I verified LLDP was enabled globally. However, when looking at the output of “show lldp traffic” I noticed that LLDP frames were being sent out of the bridge interfaces but not received.
spine01#show lldp LLDP transmit interval : 30 seconds LLDP transmit holdtime : 120 seconds LLDP reinitialization delay : 2 seconds LLDP Management Address VRF : default Enabled optional TLVs: Port Description System Name System Description System Capabilities Management Address (best) IEEE802.1 Port VLAN ID IEEE802.3 Link Aggregation IEEE802.3 Maximum Frame Size Port Tx Enabled Rx Enabled Et1 Yes Yes Et2 Yes Yes Et3 Yes Yes Ma1 Yes Yes spine01# !
When searching for a reasonable explanation on this behavior, I found a really interesting post here:
“LLDP frames have reserved destination MAC address 01-80-C2-00-00-0E, which by default are not forwarded by 802.1d-compliant bridges.”
It turns out that Linux Bridge does not forward certain Layer-2 traffic and filters some reserved multicast addresses by default so it can be compliant with the 802.1AB/802.1D standards which makes perfect sense for an open standards-based bridge.
Take a look at the diagram below.
Linux Bridge on KVM
As you can see, the links between two VMs are, in fact, interfaces connected to some Linux Bridge ports. What we need to do is modify the Linux Bridge behavior so it doesn’t filter the LLDP frames that “intercepts” from VMs.
NOTE: For the sake of testing, we are going to break the standards by overriding the default behavior of Linux Bridge. Hope you can live with that!
It’s also important to understand why we have to deal with Linux Bridge: Every time a virtual network is added on virt-manager (please see me previous posts for details) a Linux Bridge is created automatically.
In my previous posts, when I used virt-manager to assign a virtual network (net-x) to some vNIC, we were actually attaching the vNIC to a Linux Bridge (virbr) port.
Let’s see the Linux Bridge instances on the host that were created by virt-manager.
As with any regular 802.1D Ethernet bridge, Linux Bridge uses Spanning-Tree (STP) as a loop prevention mechanism. By default, STP is enabled on Linux Bridge but I turned it off on each device.
NOTE: In production, you should NEVER disable STP. On routed interfaces STP will not run but that does not mean that you can remove it from the box.
How do we actually allow the transparent forwarding of LLDP frames through Linux Bridge?
First, check the content of the group fwd_mask:
$ cat /sys/class/net/brX/bridge/group_fwd_mask 0x0
The Linux group fwd_mask was introduced to make the bridge forward link local group addresses. The point of this group is so that users can have non-standard bridging behavior.
Notice the “0x0” value. In order to pass LLDP frames through Linux Bridge we need to change this attribute value . To do so, type the following command:
$ echo 16384 > /sys/class/net/brX/bridge/group_fwd_mask
Where “brX” is the name of the bridge you want to modify.
This is it, once you change the default sysfs attribute value from “0x0” to 16384 (0x4000 in hex) the bridge will transparently forward LLDP frames. Just remember this has to be done on a per-bridge basis, so you have to apply this command to any bridge that you want to modify.
As an alternative method to set the attribute value, you can edit the group fwd_mask:
$ sudo nano /sys/class/net/brX/bridge/group_fwd_mask
Just replace the “0x0” for either “16384” or its hex value of “0x4000” (you just need to type one of them).
Finally, you can verify the operation of LLDP with “show lldp”, “show lldp neighbors” and “show lldp traffic”.
The method described above applies to Linux Bridge. If you are using Open vSwitch (OVS) the procedure is quite different and will depend on the OVS flavor that you have. Here is an example for OVS on Pica8 PicOS.