AFC Multifabric Configuration

Combining multiple EVPN fabrics into a single overlay allows sharing Layer 2 and Layer 3 reachability between data center pods at the same site as well as more distant data center locations.

Developing an effective multifabric strategy supports growth, location diversity, and disaster recovery. Aruba Fabric Composer (AFC) can simplify the configuration process and automate building a multifabric EVPN-VXLAN overlay.

Multiple data center fabrics and locations can be combined into a single overlay topology. This guide focuses on the common usage of combining two data center locations.

Additional considerations are required when adding three or more fabrics or when multiple fabrics are present at a single site which co-exist with fabrics at remote locations. Details on supporting these configurations can be found in the AOS-CX EVPN VXLAN Guide in the Aruba Support Portal.

The primary data center fabric in this guide is named RSVDC-FB1, which was previously created. The new fabric established in a colocation facility is named RSVCO-FB2.

Table of contents

• AFC Multifabric Configuration
  • Second Fabric Guidance
    • VXLAN Data Plane Configuration
    • Underlay WAN IP
  • Configure Multifabric Underlay Connectivity
    • WAN Underlay Connectivity
    • Assign WAN IP Addresses
    • Configure Underlay Routing
  • Configure Overlay Control Plane
    • Multifabric VXLAN Tunnel Requirements
    • Join Fabric Overlay Control Planes Together
    • Extend Layer 3 Reachability Across Fabrics
    • Enable External Layer 3 Multifabric Advertisements
      • Create Route Map
      • Apply Route Map to EVPN Peering
    • Extend Layer 2 Reachability Across Fabrics

Second Fabric Guidance

Use the process outlined in the AFC EVPN-VXLAN Configuration guide to create a second fabric. The same AFC instance can be used to build the new fabric, if IP connectivity can be established between AFC and the new fabric location with a latency of 50 ms or less. A second AFC instance can be deployed and linked to the first instance, if latency exceeds this value.

The diagram below illustrates the physical topology of the second fabric used in this guide. The new fabric is located at a colocation facility and identified as RSVCO-FB2.

The RSVCO-FB2 fabric uses unique IP numbering for loopbacks and VTEPs. The following IP ranges are used to build the second fabric.

The following network elements are logically contiguous between fabrics.

VXLAN Data Plane Configuration

The PROD-DC-VRF is Layer 3, extended across the RSVDC and RSVCO data center fabrics. VLANs 101 and 102 are Layer 2, extended between the fabrics. To allow a contiguous overlay for network segments extended between fabrics, VXLAN VNI values in each data center must agree. The same L3 VXLAN VNI value for PROD-DC-VRF must be defined in both fabrics.

To assign the same L2 VXLAN VNIs to both fabrics, assign the same Base L2VNI value in the EVPN CONFIGURATION wizard.

Underlay WAN IP

Border leaf switches typically establish underlay route peerings between fabrics, but other switch members of a fabric can perform this function.

IP addresses must be assigned to WAN interfaces in both fabrics. When Layer 2 WAN services or dark fiber are used, IP addresses are assigned for peering directly between switches at each site. The following IP addresses are reserved for underlay interfaces connecting to the sample Layer2 metro Ethernet circuit: 10.255.6.0/29. When using Layer 3 WAN services, IP addresses are assigned to allow direct peering with the service provider.

Configure Multifabric Underlay Connectivity

A multifabric underlay serves the same purpose as the underlay within a single fabric. It shares IP loopbacks to enable MP-BGP EVPN peerings and VTEP reachability.

WAN Underlay Connectivity

Several methods can be used to establish connectivity between fabrics at different sites. Dark fiber and most metro Ethernet services support jumbo frame capabilities. Fragmentation of VXLAN encapsulated traffic is not supported. The WAN path between fabrics must accommodate an increase of 50 bytes in MTU over the encapsulated traffic. A WAN path MTU of 1600 is recommended.

This guide uses a Layer 2 metro Ethernet service with multi-port customer premise equipment (CPE). The border leaf switches for each fabric use a single physical connection to the CPE at their respective locations.

The diagram below illustrates the sample connectivity between data center locations. RSVDC Fabric 1 is the fabric created in the AFC EVPN-VXLAN Configuration guide. RSVCO Fabric 2 is the second fabric located in the colocation facility.

Improved WAN resiliency can be realized by using multiple CPE devices at each location, by provisioning a second metro Ethernet circuit, or by using multiple dark fiber links.

Assign WAN IP Addresses

A single IP subnet is used over the Layer 2 metro Ethernet service. IP addresses are assigned to physical switch interfaces. Routed interfaces are used to avoid Layer 2 loops.

Step 1 Select Configuration > Routing > VRF on the top menu.

Step 2 Select RSVDC-FB1 in the Fabric menu, click the • • • symbol next to default, and select IP Interfaces.

Step 3 On the lower ACTIONS menu of the IP INTERFACES tab, select Add.

Step 4 On the Interface Type page, assign the following values:

• Type: RPI
• Switch: RSVDC-FB1-LF1-1
• Port/LAG: 1/1/13

Step 5 On the IPv4 Addresses page, enter the WAN IP address for the Primary IPv4 Network Address. Click NEXT.

Step 6 On the Name page, enter a Name and Description, then click NEXT.

Step 7 On the Summary page, verify the interface settings and click APPLY.

Step 8 Repeat the procedure to assign IP addresses to the physical WAN interface of each border leaf.

Note: Click VRF in the current Configuration / Routing / VRF / default display in the upper left or in the left navigation pane to return to the VRF window. Select RSVCO-FB2 in the Fabric menu to assign IP addresses in the second fabric.

Configure Underlay Routing

External BGP (eBGP) using the IPv4 address-family is used to share IP loopback and VTEP reachability. The diagram below illustrates the eBGP IPv4 sessions established to share loopback and VTEP reachability information.

Step 1 Select Configuration > Routing > BGP on the top menu.

Step 2 Select RSVDC-FB1 in the Fabric menu. Click the • • • symbol next to default and select Switches.

Step 3 Click the • • • symbol next to RSVDC-FB1-LF1-1 and select Neighbors.

Step 4 On the lower ACTIONS menu, select Add.

Step 5 On the Settings page, enter the following non-default values and click NEXT.

• Neighbor AS Number: 65002
• IP Address: 10.255.6.3
• Enable Bidirectional Forwarding Detection (BFD) Fall Over: < checked >

Step 6 On the Name page, enter a Name and Description, then click NEXT.

Step 7 On the Summary page, verify the BGP settings and click APPLY.

Step 8 Repeat the procedure to configure underlay eBGP peerings on the remaining border leaf switches in the RSVDC-FB1 fabric.

Step 9 Repeat the procedure to configure underlay eBGP peerings on the border leaf switches in the RSVCO-FB2 fabric. In step 2, Select RSVCO-FB2 in the Fabric menu to configure BGP peerings in the second fabric.

Step 10 Click BGP in the left navigation pane and select RSVDC-FB1 for the Fabric field.

Step 11 Click the • • • symbol next to default and select Neighbors Summary.

Step 12 In the Address Family column filter, select IPv4 Unicast. Click the Apply table filters (arrow) icon.

Step 13 Verify that each BGP session displays Established in the State column.

Configure Overlay Control Plane

The overlay control plane uses MP-BGP EVPN advertisements to share host MAC and IP reachability across both fabrics. The diagram below illustrates the eBGP EVPN sessions established to share overlay reachability information.

Each site contains one border leader VTEP. Border leader VTEP switches establish eBGP EVPN address-family peerings with border leaders in other sites. A full mesh of peerings is established between sites.

When more than one fabric is present in a single site, the border leader VTEP switches also establish eBGP EVPN address-family peerings with border leaf switches of each fabric in the same site.

Note: The border leader is a control plane role that optimizes the number of eBGP sessions required to share EVPN reachability information between sites, when any individual site contains multiple fabrics. The multifabric VXLAN data plane still establishes a full mesh of VXLAN tunnels between between border leaf VTEPs.

Multifabric VXLAN Tunnel Requirements

Overlay host traffic within a fabric only requires encapsulation in a single VXLAN tunnel, because a full mesh of tunnels is established between VTEPs. In a multifabric environment, a full mesh of tunnels between all leaf switches is not present. Traffic between fabrics is enabled using a VXLAN tunnel between the border leaf switches of each fabric. In this model, overlay host traffic between fabrics may traverse multiple VXLAN tunnels: VXLAN tunnels internal to a fabric and the inter-fabric VXLAN tunnel.

Join Fabric Overlay Control Planes Together

The AFC EVPN VXLAN Multi-Fabric wizard configures eBGP EVPN peerings between the border leader switches of each fabric. Supporting prefix lists, route maps, OSPF redistributions, and BGP redistributions also are configured. VXLAN forwarding between iBGP and eBGP learned VXLAN tunnels is also enabled. These peerings share overlay host and prefix reachability between fabrics and are established between border leader loopback addresses.

AFC adds configuration to share the complete set of VTEP IPs in both fabrics to assist with troubleshooting.

Step 1 On the left navigation pane, click EVPN VXLAN Multi-Fabric.

Note: If the left pane no longer displays routing options, select Configuration > Routing > EVPN VXLAN Multi-Fabric on the top menu.

Step 2 Verify that RSVDC-FB1 is selected in the Fabric menu. On the ACTIONS menu, select Add.

Step 3 On the Name page, enter a Name and Description, then click NEXT.

Step 4 On the Settings page, select the border leader VSX pair in the Border Leader field, verify that both border leader switches are listed in the L3 Connect field, and click NEXT.

Step 5 On the Remote Fabrics page, select Local AFC in the AFC Site field and RSVCO-FB2 in the Fabric field. Verify that the auto-populated values for AS Number, Remote Border Leader Address, and Secondary Remote Border Leader Address are correct. Click Add.

Note: More than one fabric can be added on this page. Fabrics managed by remote AFC instances can be selected when the remote AFC instance is configured as an AFC Site on the local AFC application.

Step 6 Click NEXT.

Step 7 On the Summary page, verify the multifabric BGP settings and click APPLY.

Step 8 Repeat the procedure to configure eBGP EVPN peerings in the second fabric. At step 2, select RSVCO-FB2 on the Fabric menu.

Step 9 In the menu bar at the top right of the AFC display, click the CLI Commands icon and select Show Commands.

Step 10 On the CLI Command Processor page, enter the following values, then click RUN.

• Switches: RSVDC-FB1-LF1-1, RSVDC-FB1-LF1-1, RSVCO-FB2-LF1-1, RSVCO-FB2-LF1-2

• Commands: show bgp l2vpn evpn summary

Step 11 Verify that the output for each switch displays the newly configured peers with an Established state.

The eBGP EVPN route peerings establish the control plane mechanism to share overlay reachability between fabrics. Additional steps are necessary to install IP prefixes and MAC addresses between fabrics.

Extend Layer 3 Reachability Across Fabrics

When using EVPN, MP-BGP EVPN route-type 5 advertisements share IP prefix reachability for a VRF. Route targets included in these advertisements control how Layer 3 IP addresses are installed in IP forwarding tables. Route targets defined during the creation of each fabric are typically used only to share information locally. New VRF route-targets are defined to share IP reachability between fabrics.

Step 1 On the left navigation pane, click VRF.

Note: If the left pane no longer displays routing options, select Configuration > Routing > VRF on the top menu.

Step 2 Select RSVDC-FB1 in the Fabric field. Click the radio button next to PROD-DC-VRF. On the ACTIONS menu, select Edit.

Step 3 Click the the ROUTE TARGETS page heading. On the page, enter the values below and click ADD.

• Route Target Mode: Both
• Route Target Ext-Community: 1:100001
• Address Family: EVPN

Note: It is best practice to use a route target between fabrics that is distinct from the route target within a fabric to provide flexibility in controlling the installation of overlay IP prefixes.

Step 4 Click Apply.

Step 5 Repeat the procedure to assign the new route target to the RSVCO-FB2 PROD-DC VRF. Select RSVCO-FB2 in the Fabric field to begin.

Enable External Layer 3 Multifabric Advertisements

The AFC default configuration applies a route map that permits only local fabric prefixes to be shared across fabrics. The AFC created route map named to-border-leaders must be modified, if IP prefixes learned in the overlay from outside the fabric must be shared.

In this sample deployment, a campus summary prefix learned in PROD-DC-VRF of the RSVDC-FB1 fabric is shared with PROD-DC-VRF in the RSVCO-FB2 fabric. The default route learned in the same VRF also is shared as a backup default route if Internet connectivity in the colocation facility fails.

A new route map is created for the RSVDC-FB1 border leaders and applied to its eBGP EVPN peerings to allow advertising of routes learned from outside the fabric.

Create Route Map

A new route map is created to allow local fabric reachability and enable advertising of campus-learned IP prefixes from the RSVDC-FB1 border leaders to the RSVCO-FB2 fabric. The route map ALLOWED-EXT-AS defined in the initial EVPN-VXLAN configuration is re-used in this procedure.

Step 1 Click the ROUTE MAPS tab. On the ACTIONS menu, select Add.

Step 2 On the Name page, enter a Name and Description, then click NEXT.

Step 3 On the Scope page, select the RSVDC-FB1 border leaders in the Switches field and click NEXT.

Step 4 On the ACTIONS menu, select Add.

Step 5 On the Settings page, enter the following values and click NEXT

• Sequence: 10
• Description: permit local overlay advertisements
• Action: Permit

Step 6 On the Match Attributes page, enter the following values and click NEXT.

• Attributes: Match AS Path List
• Match AS Path List: local-fabric

Step 7 On the Set Attributes page, click NEXT.

Step 8 On the Summary page, verify the multifabric BGP settings and click APPLY.

Step 9 Repeat steps 11 to 15 to add a second route map entry with the values below.

• Sequence: 20
• Description: permit campus/firewall prefixes
• Action: Permit
• Attributes: Match AS Path List
• Match AS Path List: ALLOWED-EXT-AS

Step 10 Click NEXT.

Step 11 On the Summary page, verify the route map settings and click APPLY.

Apply Route Map to EVPN Peering

Step 1 On the menu bar at the top right of the AFC display, click the CLI Commands icon and select Show Commands.

Step 2 Select RSVDC-FB1 in the Fabric field and both border leaf switches in the Switch field.

Step 3 In the switch configuration window, scroll to the address-family l2vpn evpn stanza in the BGP configuration section. Set the border-leaders peer group’s outbound route map to RSVDC-to-borders.

Step 4 Click the RSVDC-FB1-LF1-2 tab. Set the border-leaders peer group’s outbound route map to RSVDC-to-borders. Click VALIDATE ALL.

Step 5 A success message verifies the configuration is valid. Click APPLY ALL.

Note: If configuration errors are present on a switch, a red error icon appears on the switch tab and the configuration errors are highlighted in red. Correct the errors and validate the configuration again. It is possible an incomplete spanning-tree configuration was unintentionally created in the AFC guided setup process. If spanning-tree config-name is in the config with no name parameter and no other spanning-tree configuration present, it is safe to delete the line.

Step 6 Success messages verify a configuration checkpoint was created and the configuration changes were applied.

Step 7 On the RSVDC-FB1-LF1-1 switch CLI, clear the EVPN BGP sessions to the second fabric to apply the new route map policy.

clear bgp 10.250.3.5 clear bgp 10.250.7

Step 8 Repeat step 7 on the RSVDC-FB1-LF1-2 switch.

Step 9 On the menu bar at the top right of the AFC display, click the CLI Commands icon and select Show Commands.

Step 10 On the CLI Command Processor page, enter the following values, then click RUN.

• Switches: RSVCO-FB2-LF1-1, RSVCO-FB2-LF1-1, RSVCO-FB2-LF2, RSVCO-FB2-LF3

• Commands: show ip route vrf PROD-DC-VRF

Step 11 Verify that each leaf switch in the RSVCO fabric learns the 10.0.0.0/12 campus summary route and the 10.5.50.0/24 prefixes that are present only on CX 10000 switches in the RSVDC-FB1 fabric.

Extend Layer 2 Reachability Across Fabrics

When using EVPN, host reachability advertisements include a route-target to inform remote VTEPs of the VLAN associated with the host MAC address advertisement. VLAN route-targets can be generated automatically within a fabric using an iBGP EVPN peering. The eBGP EVPN peering between fabrics requires an explicitly defined route target for each VLAN. This additional route target controls when MAC advertisements shared between fabrics are installed in local MAC address tables.

Step 1 On the Configuration menu, select Routing > EVPN, then click the EVPN MULTI SITE tab.

Note: If the left pane no longer displays routing options, select Configuration > Routing > EVPN on the top menu.

Step 2 With RSVDC-FB1 selected in the Fabric menu, click the ACTIONS menu and select Add.

Step 3 On the Fabrics page, select both fabrics and click NEXT.

Step 4 On the VLANs page, enter the VLAN IDs that share Layer 2 reachability information across both fabrics.

Step 5 On the Route Targets page, enter the following values and click ADD.

• Route Target Type: NN:VNI
• Administrative Number: 1

Step 6 Click NEXT.

Step 7 On the Summary page, verify the route target settings and click APPLY.

Note: If the fabrics are managed by different AFC instances, the EVPN wizard must be run for each instance.

Step 8 On the menu bar at the top right of the AFC display, click the CLI Commands icon and select Show Commands.

Step 9 On the CLI Command Processor page, enter the following values, then click RUN.

• Fabrics: RSVDC-FB1, RSVCO-FB2

• Commands: show mac-address-table

Note: VXLAN-learned MAC address entries include the VTEP IP in parentheses in the Port column.

Step 10 Verify that each switch in both fabrics learns MAC addresses from the other fabric (the MAC address entires are displayed with a VTEP IP of the border leader in the other fabric).

At the completion of this procedure, a multifabric EVPN is established between the RSVDC-FB1 and RSVCO-FB2 fabrics.

---