Metallb

Openshift

Networking

K8s

Nmstate

[OCP4][Experimental] Metallb multiple vlan on secondary interface

What I would like to achieve is using MetalLB in L2 mode with a single physical interface but multiple VLANs, where each VLAN is mapped to its own MetalLB IP address pool.

Prerequitsite

• OpenShift cluster (tested on 4.20)
• Metallb operator
• NMstate operator
• OpenShift node with 1 more interface added (trunk)

Configured OpenShift with routingViaHost to enable routing decisions to be made at the host level.

oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig": {"routingViaHost": true} }}}}' --type=merge

oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}' --type=merge

Configured node network

Attach interface ens35 and use NMstate operator to configured sub interface for each vlan 2025,2028

• 172.16.25.0/24 -> vlan 2025
• 172.16.28.0/24 -> vlan 2028

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: worker-1-secondary
spec:
  desiredState:
    interfaces:
    - name: ens35
      type: ethernet
      state: up
    - name: ens35.2025
      type: vlan
      state: up
      vlan:
        base-iface: ens35
        id: 2025
      ipv4:
        address:
          - ip: 172.16.25.218
            prefix-length: 24
        enabled: true
      ipv6:
        enabled: false
    - name: ens35.2028
      type: vlan
      state: up
      vlan:
        base-iface: ens35
        id: 2028
      ipv4:
        address:
          - ip: 172.16.28.218
            prefix-length: 24
        enabled: true
      ipv6:
        enabled: false
  nodeSelector:
    kubernetes.io/hostname: bmwrk-01

The node will have the interfaces and routes configured as shown below.

Metallb L2 mode configured

• create IP Pool

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ip-addresspool-2025
  namespace: metallb-system
spec:
  addresses:
  - 172.16.25.208-172.16.25.210
  autoAssign: true
  avoidBuggyIPs: false
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ip-addresspool-2028
  namespace: metallb-system
spec:
  addresses:
  - 172.16.28.208-172.16.28.210
  autoAssign: true
  avoidBuggyIPs: false

• create L2advertisement

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-adv-2025
  namespace: metallb-system
spec:
  interfaces:
    - ens35.2025
  ipAddressPools:
    - ip-addresspool-2025
  nodeSelectors:
    - matchLabels:
        kubernetes.io/hostname: bmwrk-01
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-adv-2028
  namespace: metallb-system
spec:
  interfaces:
    - ens35.2028
  ipAddressPools:
    - ip-addresspool-2028
  nodeSelectors:
    - matchLabels:
        kubernetes.io/hostname: bmwrk-01

l2-adv-2025:

• Announces IPs from the pool ip-addresspool-2025
• Advertises only onens35.2025 (vlan 2025)

l2-adv-2028:

• Same concept but for pool ip-addresspool-2028
• Advertises only on ens35.2028 (vlan 2028)

Create sample application and service

I use my own sample application here https://github.com/Surote/simple-fastapi

and create service

apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.io/ip-allocated-from-pool: ip-addresspool-2025
  name: fast-localtime-check-svc
  labels:
    app: fast-localtime-check
spec:
  selector:
    app: fast-localtime-check
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.io/ip-allocated-from-pool: ip-addresspool-2028
  name: fast-localtime-check-svc-2028
  labels:
    app: fast-localtime-check
spec:
  selector:
    app: fast-localtime-check
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
  type: LoadBalancer

Two services are configured — one for VLAN 2025 and another for VLAN 2028 — both pointing to the same application.

The service will pick up an IP from IP pool specified in service annotation.

Connection Testing

https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html-single/networking_operators/index#nw-metallb-layer2_about-metallb-and-metallb-operator

The client used for testing needs to be on the same subnet as the VIP assigned by MetalLB. Otherwise, the node will not send traffic back to the source due to asymmetric routing. To resolve this, follow the solution here. https://access.redhat.com/solutions/7117243

testing for vlan 2028

testing for vlan 2025

Disclaimer

This is intended solely for testing purposes and is not officially affiliated with or supported by Red Hat solutions. The information provided herein is for educational and testing use only.