Use an extended bridge using OpenVSwitch and VXLan over internal network

Summary

Classic bridges are local only, with OpenVSwitch and automatic VXLan tunelling if you have a private network between your two servers you can have a bridge on each one linked.

They will have the same subnet, and servers from one side could reach the other without issue.

It's possible to switch from brctl to ovs without issues since there is no required config in the interfaces side of the containers, only setup the bridge and use it.

Drawing

The eth1<->//vmbr0 or vmbr2// is in fact transparent, you don't add it in the bridge, it's juste "transparently" used by VXLan (because you use the tunnel over the private network).

Notes

OpenVSwitch bridges are not compatible with brctl, you should use ovs-vsctl, like ovs-vsctl show

Setup

Requirements

Here we are assuming:

• Server 1 PRIVATE LAN IP: 192.168.1.4
• Server 2 PRIVATE LAN IP: 192.168.1.5
• Bridge name on each server: vmbr0
• Extended Bridge network: 10.0.0.0
• Server 1 BRIDGE IP: 10.0.0.1
• Server 2 BRIDGE IP: 10.0.0.2

Blah

apt install openvswitch-switch openvswitch-common

Create an OpenVSWitch bridge on each server:

ovs-vsctl add-br vmbr0

Config on server1, file /etc/network/interfaces:

auto vmbr0
iface vmbr0 inet static
    address 10.0.0.1
    netmask 255.255.255.0
    ovs_type OVSBridge
    post-up ovs-vsctl add-port vmbr0 vxlan1 -- set Interface vxlan1 type=vxlan options:remote_ip=192.168.1.5

For server2, file /etc/network/interfaces:

auto vmbr0
iface vmbr0 inet static
    address 10.0.0.2
    netmask 255.255.255.0
    ovs_type OVSBridge
    post-up ovs-vsctl add-port vmbr0 vxlan1 -- set Interface vxlan1 type=vxlan options:remote_ip=192.168.1.4

Up the network on each: ifup vmbr0

You may need to reboot to load OpenVSwitch kernel modules.

And you should be able to ping 10.0.0.2 from server 1 and server 1 from server 2.

You can get OpenVSwitch status config by using:

server1:~# ovs-vsctl show
03edd856-b35a-4c2d-b283-1dfc28ab7abb
    Bridge "vmbr0"
        Port "vmbr0"
            Interface "vmbr0"
                type: internal
        Port "vxlan1"
            Interface "vxlan1"
                type: vxlan
                options: {remote_ip="192.168.1.5"}
        Port "veth2ES9B5"
            Interface "veth2ES9B5"
    ovs_version: "2.3.0"

LXC Notes

LXC Uses brctl and brctl isn't compatible with OpenVSwitch, here is the configuration needed to use the new ovs bridge, file /etc/lxc/ifup:

#!/bin/bash

BRIDGE='vmbr0'
ovs-vsctl --may-exist add-br $BRIDGE
ovs-vsctl --if-exists del-port $BRIDGE $5
ovs-vsctl --may-exist add-port $BRIDGE $5

And file /etc/lxc/ifdown:

#!/bin/bash

ovsBr='vmbr0'
ovs-vsctl --if-exists del-port ${ovsBr} $5

In the CT config, /var/lib/lxc/derpy/config:

lxc.network.type = veth
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.script.up = /etc/lxc/ifup
lxc.network.script.down = /etc/lxc/ifdown
lxc.network.ipv4 = 10.0.0.111/24
lxc.network.ipv4.gateway = 10.0.0.100