Use an extended bridge using OpenVSwitch and VXLan over internal network

Classic bridges are local only, with OpenVSwitch and automatic VXLan tunelling if you have a private network between your two servers you can have a bridge on each one linked.
They will have the same subnet, and servers from one side could reach the other without issue.
It's possible to switch from brctl to ovs without issues since there is no required config in the interfaces side of the containers, only setup the bridge and use it.


The eth1vmbr0 or vmbr2 is in fact transparent, you don't add it in the bridge, it's juste “transparently” used by VXLan (because you use the tunnel over the private network).

OpenVSwitch bridges are not compatible with brctl, you should use ovs-vsctl, like

ovs-vsctl show

Requirements

Here we are assuming:

  • Server 1 PRIVATE LAN IP: 192.168.1.4
  • Server 2 PRIVATE LAN IP: 192.168.1.5
  • Bridge name on each server: vmbr0
  • Extended Bridge network: 10.0.0.0
  • Server 1 BRIDGE IP: 10.0.0.1
  • Server 2 BRIDGE IP: 10.0.0.2

Blah

apt install openvswitch-switch openvswitch-common

Create an OpenVSWitch bridge on each server:

ovs-vsctl add-br vmbr0

Config on server1:

/etc/network/interfaces
auto vmbr0
iface vmbr0 inet static
	address 10.0.0.1
	netmask 255.255.255.0
	ovs_type OVSBridge
	post-up ovs-vsctl add-port vmbr0 vxlan1 -- set Interface vxlan1 type=vxlan options:remote_ip=192.168.1.5

For server2:

/etc/network/interfaces
auto vmbr0
iface vmbr0 inet static
	address 10.0.0.2
	netmask 255.255.255.0
	ovs_type OVSBridge
	post-up ovs-vsctl add-port vmbr0 vxlan1 -- set Interface vxlan1 type=vxlan options:remote_ip=192.168.1.4

Up the network on each:

ifup vmbr0


You may need to reboot to load OpenVSwitch kernel modules.
And you should be able to

ping 10.0.0.2

from server 1 and server 1 from server 2.
You can get OpenVSwitch status config by using:

server1:~# ovs-vsctl show
03edd856-b35a-4c2d-b283-1dfc28ab7abb
    Bridge "vmbr0"
        Port "vmbr0"
            Interface "vmbr0"
                type: internal
        Port "vxlan1"
            Interface "vxlan1"
                type: vxlan
                options: {remote_ip="192.168.1.5"}
        Port "veth2ES9B5"
            Interface "veth2ES9B5"
    ovs_version: "2.3.0"

LXC Notes

LXC Uses brctl and brctl isn't compatible with OpenVSwitch, here is the configuration needed to use the new ovs bridge:

/etc/lxc/ifup
#!/bin/bash
BRIDGE='vmbr0'
ovs-vsctl --may-exist add-br $BRIDGE
ovs-vsctl --if-exists del-port $BRIDGE $5
ovs-vsctl --may-exist add-port $BRIDGE $5
/etc/lxc/ifdown
#!/bin/bash
ovsBr='vmbr0'
ovs-vsctl --if-exists del-port ${ovsBr} $5

In the CT config:

/var/lib/lxc/derpy/config
lxc.network.type = veth
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.script.up = /etc/lxc/ifup
lxc.network.script.down = /etc/lxc/ifdown
lxc.network.ipv4 = 10.0.0.111/24
lxc.network.ipv4.gateway = 10.0.0.100