Distributed Overlay Configuration

This section describes how to deploy a distributed overlay on top of an existing large campus underlay.

A three-tier network referred to as Seattle is used to illustrate this process.

Some steps require manual CLI configuration of switches. Complete these steps using the MultiEdit feature of Central UI groups.

Prepare the Underlay

Network Review

This section assumes a large campus deployment has been created, which is consistent with the preceding VSG procedures. The image and table below represent the topology and IP scheme.

Device fabric personas to be assigned are shown in bold. The access aggregation switches perform only underlay functions.

The following table lists the IP subnets of the underlay networks.

The following table lists the IP subnets used to deploy the distributed overlay. Best practice is to segment wired and wireless traffic onto separate networks.

Migrate Central Groups

All Aruba CX switches to be included in the fabric must be in the same Central UI group.

Gateways and access points can be in separate fabric groups; however, this procedure illustrates all devices in the same group. In this example, a group called CP-SEA-FAB is created. Refer to the Create New Groups procedure as needed.

To move fabric devices from existing UI or template groups to the new fabric group:

Step 1 On the left menu, select Organization.

Step 2 Select the Groups tile.

Step 3 In the Groups table, select + (Add Group).

Step 4 On the Add Group page, assign the following settings, and click Next.

• Name: CP-SEA-FAB

• Group will contain: check-mark all 3

Step 5 On the Add Group page, assign the following settings, and click Add.

• Architecture for access points and gateways in this group: ArubaOS 10
• Network role of the access points in this group: Campus/Branch
• Network role of the gateways in this group: Mobility
• Type of switches used in this group: AOS-CX only

Step 6 With the Retain CX-Switch configuration option enabled, move devices into the CP-SEA-FAB group following this procedure.

Loopback Configuration

The switches should already have loopback0 as part of the earlier OSPF configuration process.

Loopback1 must be configured on all devices participating in the fabric. It will also be advertised into OSPF. Loopback1 serves as the source interface for the VXLAN overlay. Switches assigned as VSX pairs use the same loopback1 IP address.

Step 1 Configure loopback1 on access switches.

interface loopback 1
  ip address 10.10.10.50/32
  ip ospf 1 area 0.0.0.0

Step 2 Configure “loopback 1” on the core, service aggregation, and WAN aggregation switches.

interface loopback 1
  ip address 10.10.10.52/32

Note: The NetConductor fabric wizard currently requires “loopback 0” to be allocated for router IDs and “loopback 1” to be allocated for VXLAN VTEPs.
Alternate interface numbers are not supported at this time.

Configure Transit VLAN & OSPF Configuration

A transit VLAN is configured to extend Layer 3 connectivity from the aggregation to access switches. OSPF is configured to advertise the newly configured loopback interfaces.

Step 1 Create new transit VLANs on the aggregation switches and add them to the MC-LAGs connected to the access switches.

vlan 901
    vsx-sync
    description Interconnect_VLAN for Access AGG1-ACC1
vlan 902
    vsx-sync
    description Interconnect_VLAN for Access AGG1-ACC2
interface lag 11 multi-chassis
     description ACCESS-1
     vlan trunk 3,6,13,15,901
interface lag 12 multi-chassis
     description ACCESS-2
     vlan trunk 3,6,13,15,901
interface vlan 901
    ip mtu 9198
    vsx active-forwarding
    ip address 172.18.110.97/29
    ip ospf 1 area 0.0.0.0
interface vlan 902
    ip mtu 9198
    vsx active-forwarding
    ip address 172.18.110.105/29
    ip ospf 1 area 0.0.0.0

Step 2 Create new transit VLANs on the access switches and add them to the LAGs connected to the aggregation switches.

vlan 901
    description Interconnect_VLAN for Access AGG1
interface lag 11
    mtu 9198
    no shutdown
    description AGGREGATION UPLINKS
    no routing
    vlan trunk native 1
    vlan trunk allowed 3,6,13,15,901
    lacp mode active
interface vlan 901
    ip mtu 9198
    ip address 172.18.110.99/29
    ip ospf 1 area 0.0.0.0
    no ip ospf passive
router ospf 1
    router-id [loopback 0 IP]
    max-metric router-lsa include-stub on-startup
    passive-interface default
    graceful-restart restart-interval 660
    redistribute connected
    area 0.0.0.0

MTU Configuration

VXLAN requires a larger frame size than the 1500-byte MTU enabled by default on most network devices. Jumbo Frame MTU sizing must be enabled on all interfaces of devices in the fabric and all underlay links carrying overlay traffic.

Step 1 Configure the MTU to 9198 bytes on Layer 2 interfaces.

interface 1/1/32
	mtu 9198

Step 2 Configure the IP MTU & MTU to 9198 bytes on routed interfaces

interface 1/1/2
    no shutdown
    mtu 9198
    ip mtu 9198

Note: Routed ports must have both the Layer 2 and Layer 3 MTU commands set.

Configure Role Policy

Note: Customers must contact their Aruba Account Manager to be added to the allow-list for the Global Policy Manager feature.

Roles and role policies, are provisioned at a global level and apply to all fabrics. This procedure uses the example roles:

• EMPLOYEE
• CONTRACTOR

The following example role-to-role policy will prevent employees and contractors from communicating.

Create Roles

Use this procedure to create the employee and contractor roles:

Step 1 In the dropdown, select the Global group and select Security on the left menu.

Step 2 Select the Client Roles page.

Step 3 Create a new role by clicking the + (plus sign) in the upper right corner of the table.

Step 4 In the Create new role window, assign the following settings, and click Save.

• Name : EMPLOYEE
• Description: <insert optional role description>
• Policy Identifier: <use default value>
• Allow default role to source role permissions for wired clients: <selected>

Note: The Allow default role to source role permissions for wired clients option creates policy rules which allow clients assigned the role to send and recieve ARP packets and traffic from outside the fabric.

Step 5 Repeat steps 3 and 4 to create the CONTRACTOR role and enter an optional description.

Define Role to Role Policy

Use this procedure to create a policy to prevent the EMPLOYEE and CONTRACTOR roles from communicating.

Step 1 Mouse-over the CONTRACTOR row and click the edit (pencil) icon on the right.

Step 2 In the PERMISSIONS edit window for the CONTRACTOR role, click the edit (pencil) icon at the the top right of the PERMISSIONS table. The Assign Permissions window appears.

Step 3 On the Assign Permissions window, assign the following settings and click Save.

• CONTRACTOR (self):
  • Allow Source to Destination: Checked
  • Allow Destination to Source: Checked
• EMPLOYEE:
  • Allow Source to Destination: Unchecked
  • Allow Destination to Source: Unchecked

Step 4 Repeat steps 1-3 for the EMPLOYEE role using the following settings.

• CONTRACTOR
  • Allow Source to Destination: Unchecked

  • Allow Destination to Source: Unchecked

  • EMPLOYEE (self):
    • Allow Source to Destination: Checked
    • Allow Destination to Source: Checked

Configure Wireless Integration

Wireless gateways form static VXLAN tunnels with the stub border to provide connectivity to the fabric and map the VLANs to VNIs. Jumbo frames are enabled on the gateway interfaces, static VXLAN tunnels are configured, and roles are created.

Prepare Gateway

Refer to Configuring Wireless Group Settings to assist with:

• Enabling jumbo frame processing

Refer to Configure Gateway VLANs to assist with:

• Creating VLAN 400
• Verifying jumbo frames on the port channel

Create Fabric SSID

Refer to Configuring Wireless Access to assist with:

• Creating an SSID named SSID-SEACP-01 that authenticates to ClearPass.

Update AOS Gateway Security Policies

By default, AOS 10 blocks all traffic to or from a role. Policy changes are required to allow traffic for a new role after it is created.

Use this procedure to configure these policies at the gateway cluster level:

Step 1 Navigate to the Central UI group with the gateways. Select Devices, then Gateways, and click Configure.

Step 2 Select Security, then Roles. Click the + (plus sign) and add the following roles:

• EMPLOYEE
• CONTRACTOR

Step 3 On the Roles page, select CONTRACTOR. Scroll down to the policy section and click the + (plus sign) .

Step 4 In the Add Policy window, assign the following settings and click Save Settings.

• Add an existing policy: selected

• Create a new policy: unselected

• Policy type: Session

• Policy name: allowall

• Position: leave blank

Step 5 Repeat steps 1-4 for the EMPLOYEE role.

Note: Though not shown in the UI, the role-to-role policy is enforced above the allowall policy. This procedure will change in a future release when gateways support the default role.

Configure Static VXLAN Tunnel

Use the following procedure to configure stub VXLAN tunnels between the gateway(s) and stub switches:

Step 1 In the Global dropdown, select the switch group. In this example, the group is CP-SEA-FAB.

1. On the left menu, select Devices.

   

2. Select Gateways, then select Config.

   

3. Click the Interface tab and click VXLAN Tunnels. Click the + (plus sign) in the lower left.

   

4. On the Add VXLAN Tunnel page, assign the following settings:

   • IP Version: IPv4

   • VXLAN Tunnel Source: VLAN

   • VLAN Interface: 10

   • Virtual tunnel end point (vtep) peer IP: 10.10.10.105

   • Enable tunnel admin state: Checked

   • Enable global policy identifier (gpid): Checked

   

Note: The 10.10.10.105 IP address above is the loopback1 IP address shared between the VSX pair devices acting as stub VTEPs.

Step 5 Click the + (plus sign) in VLAN/VNI mapping, assign the following settings, and click OK.

• VLAN ID: 400

• Virtual network identifier: 400

Deploy the Fabric

Use the fabric wizard to deploy an overlay fabric. Follow the procedures below to provision the VXLAN tunnels, EVPN control plane, VRFs, fabric VLANs, and Anycast Gateways.

Create The Fabric

Step 1 In the Global dropdown, select the switch group. In this example, the group is CP-SEA-FAB.

Step 2 On the left menu, select Devices.

Step 3 Select Switches, then select Config.

Step 4 Under Routing, select Fabrics.

Step 5 On the Fabrics table, click the + (plus sign) at the top right.

Step 6 In the Create a New Fabric workflow, click Name Fabric, assign the following settings, and click Next.

• Fabric Name: Seattle

• BGP AS Number : Use default

Step 7 On the Add Devices page, select each access switch and use the Assign selected devices to window to assign the Edge persona. Click Apply.

Step 8 Repeat steps 6 and 7 for the RR, Border, and Stub device personas, then click Next.

Note: Click Apply after each persona selection or the assignment will not apply.

Step 9 On the Add Overlay Network page, leave the default overlay network. Click Next.

Step 10 On the Stub Tunnels to Gateway page, click the + (plus sign) at the top right of the table.

Step 11 In the Tunnels table, assign the following settings. Click outside the new row to continue.

• Switch: SW-SEACP-STUB-01
• Gateway List IP: 10.10.0.68, 10.10.0.70

Note: Gateway IPs must match the VXLAN Tunnel Source configured on the AOS-10 Gateways.

Step 12 Repeat steps 10 and 11 for additional stub switches. Click Next.

Step 13 Review the Summary page for accuracy. Return to previous pages and make corrections, if needed. Click Save.

Create the Fabric Segments

Use this procedure to create VLAN associated segments within the fabric. In the following example, a wired segment is applied to all Edge devices and a wireless segment is applied to the Stub devices.

Step 1 Expand the Seattle fabric, then click the New Segment icon.

Step 1 On the Overlay Network & VLAN page of the New Segment workflow, assign the following settings, using the + (plus sign) to add DHCP servers, and click Next.

• Overlay Network: overlay_network
• VLAN Name: Users
• VLAN ID: 100
• DHCP Server: 10.2.120.98, 10.2.120.99
• DHCP Server VRF: default, default
• IPv4 Version: IPv4
• Default Gateway IP: 10.10.1.1
• Subnet Mask: 24

Step 3 On the Roles page, select both roles, then click Next.

Step 4 On the Devices page, select the Edge devices, then click Next.

Step 5 Review the Summary page for accuracy, then click Save.

Step 6 Repeat steps 1-5 to create a wireless VLAN, on stub devices only, using the values listed below.

• Overlay Network: overlay_network

• VLAN Name: Users

• VLAN ID: 400

• DHCP Server: 10.2.120.98

• IPv4 Version: IPv4

• Default Gateway IP: 10.10.4.1

• Subnet Mask: 24

Note: Fabric DHCP requests will be sourced from loopback1.

Configure External Connectivity

After a fabric has been provisioned, it is typically connected to an external network via the border leaf switches. There are multiple approaches to creating this connection. The following procedure establishes layer 3 connectivity between a VRF on a border leaf and an upstream device using OSPF.

Apply the following configuration to the border leaf:

Step 1 Configure a prefix-list that will match /32 host routes.

   ip prefix-list HOST_ROUTES seq 10 permit 0.0.0.0/0 ge 32

Step 2 Configure a route-map that will deny /32 route advertisements. This will prevent EVPN type-2 host prefixes from being propagated outside the fabric.

   route-map BORDER_ROUTING deny seq 10
        match ip address prefix-list HOST_ROUTES
   route-map BORDER_ROUTING permit seq 20

Step 3 Configure a new OSPF process in the overlay VRF using a unique router-id.

   router ospf 2 vrf overlay_network
       router-id 10.10.10.254
       redistribute bgp route-map BORDER_ROUTING
       area 0.0.0.0

Note: The router-id in the new OSPF process must be different from the router-id used in the OSPF process for the underlay. Fabric routes from the BGP process on the border leaf are redistributed into the new OSPF process, which advertises the routes to the upstream neighbor. Apply the route-map to the redistribution statement to block host routes from being advertised.

Step 4 Add a VLAN in the overlay VRF to support an OSPF adjacency with the upstream device.

   vlan 600
       name OVERLAY-TRANSIT
   interface vlan 600
       vrf attach overlay_network
       description OVERLAY-TRANSIT
       ip mtu 9198
       ip address 172.18.110.60/31
       ip ospf 2 area 0.0.0.0
       ip ospf network point-to-point

Step 5 Add the new VLAN to the trunk from the border leaf to the upstream device.

   interface 1/1/11
       vlan trunk allowed 600

Step 6 Create a static route on the border with a next hop IP address of the upstream device.

   ip route 0.0.0.0/0 172.18.110.45 vrf overlay_network

Step 7 Redistribute the static route into the overlay EVPN.

   router bgp 65001
   !
       vrf overlay_network
           address-family ipv4 unicast
               redistribute static

ClearPass Integration

RADIUS-based authentication is required on all edge ports participating in the fabric. ClearPass is the recommended solution.

Modify the services as needed to ensure that ClearPass returns a role.

For more information about configuring Radius on switches, refer to the Configure RADIUS and UBT section.

Verification

The steps below illustrate how to verify the functionality of a distributed fabric deployment. Central provides a remote console capability that allows for CLI access on any managed switch. Refer to the Verify OSPF Operation section for a more detailed overview.

Verify Underlay

Step 1 In a Remote Console window, type the command show ip ospf neighbors, and press ENTER. Confirm the state is FULL with all appropriate OSPF peers.

Step 2 In a Remote Console window, type the command show ip route, and press ENTER. Confirm that all loopback0 and 1 /32 routes are listed.

Verify Overlay

The EVPN verification below is recommended for all fabric switches. VXLAN verification is recommended for the Edge, Border, and Stub devices. Endpoint verification is recommended for Edge switches.

Step 1 In a Remote Console window, type the command show bgp all summary, and press ENTER. Confirm that BGP peering is active between the route reflectors and all fabric devices.

Step 2 In a Remote Console window, type the commandshow evpn evi, and press ENTER. Verify the EVPN configuration and operational state.

Step 3 In a Remote Console window, type the command show bgp l2vpn evpn, and press ENTER. Verify EVPN overlay routes.

Step 3 In a Remote Console window, type the command show bgp l2vpn evpn, and press ENTER.

Step 5 In a Remote Console window, type the command show evpn mac-ip, and press ENTER. Ensure overlay MAC/IP address information is learned from EVPN

Step 6 In a Remote Console window, type the command show interface vxlan 1, and press ENTER. Verify VXLAN tunnels are established.

Step 6 In a Remote Console window, type the command show port-access clients, and press ENTER. Verify the authentication state of an endpoint and confirm proper role assignment.

Step 7 In a Remote Console window, type the command show port-access gbp, and press ENTER. Verify configured GBP policies are applied.

Verify Gateways

The VXLAN verification below confirms the operational state of the static VXLAN tunnel and is recommended for all gateways.

Step 1 In the Global dropdown, select the switch group. In this example, the group is CP-SEA-FAB.

Step 2 On the left menu, select Devices.

Step 3 Select Gateways, then select Clusters.

Step 4 In the Name column, click the name of the fabric connected cluster.

Step 5 Select the Tunnels page.

Step 6 In the Tunnels table, find the stub switch VTEP address in the Destination Device column, and confirm that the Status column indicates “Up”.