VMWare ESXi networking and nested virtualization20 Jan 2017
Recently I installed an Ubuntu LXD node in a virtual machine, running on VMWare ESXi. Since I wanted the containers to be directly accessible via the network, I configured an Ethernet bridge connected to the server’s network interface. To my surprise, after I got an Ubuntu container up and running, I quickly realized that no traffic intended for the container (that did not originate on the LXD node) actually managed to reach it.
Pinging the local router from the container and running tcpdump on both the router and the LXD node, I observed that ICMP echo request packets were reaching the router, but the responses were not visible on the bridge. As it turns out, this is due to an ESXi vSwitch security setting, called promiscuous mode.
An ESXi vSwitch is already aware of the effective (set by the guest OS) MAC addresses of all VM network interface cards attached to it and it does not perform MAC address learning (tracking which MAC address is seen on which port). Thus it forwards incoming frames only to the known physical addresses and drops the rest. When promiscuous mode is set to Accept, the vSwitch starts behaving more like a regular one and forwards all frames appropriately.
There is another security setting related to nested virtualization, which is called forged transmit. When it is set to Reject, the switch drops Ethernet frames with a source address that does not correspond to the originating network card physical address. The goal of this feature is to prevent MAC address spoofing from compromised virtual machines. In my case, this was set to Accept and frames were being forwarded properly, but I am not aware whether this is the default setting or not, so I believe it is worth mentioning.