Link Search Menu Expand Document
calendar_month 07-Mar-24

Two-Tier Core

Configure Two-Tier core switches as a VSX pair for Layer 2 aggregation of the data center access switches, IP data center services, and routing to the main campus.

Table of contents

Configure Core VSX ISL Interface

To establish a VSX relationship between the core switches, create a link aggregation (LAG) interface for assignment as the VSX data plane’s inter-switch link (ISL). The LAG can be defined at the Central UI group level when using the same ports for the VSX ISL on both core switches.

Step 1 On the left Aruba Central menu, click the current context, then click the data center core switch group name in the Groups column.

Note: The current context in the screenshot above is Global.

Step 2 On the left navigation menu, click Devices.

Step 3 At the upper right of the Switches pane, click Config.

Step 4 In the Interfaces tile, click Ports & Link Aggregations

Step 5 Scroll to the right of the Ports & Link Aggregations table, and click the + (plus sign) at the upper right.

Step 6 On the Add LAG page, enter the following values and click ADD:

  • Name: lag256
  • Description: VSX-ISL-LAG
  • Port Members: 1/1/31, 1/1/32
  • Speed Duplex: <no value> (default)
  • VLAN Mode: trunk
  • Native VLAN: 1 (default)
  • Allowed VLANs: <no value> (default)
  • Admin Up: checked
  • Aggregation Mode: LACP Active

Step 7 In the Ports & Link Aggregations table’s title row, click ← (left arrow) to return to the main configuration page.

Configure Routing VLAN

In this topology, a VLAN is created for routing traffic to upstream external networks. The same VLAN is used to establish a routed transit path between the core switches using the VSX ISL. If one of the core switches loses its external network peering, external reachability information is learned from the ISL and external hosts are still reachable from the other core switch.

Note: When more than one VRF is present, a VLAN per VRF is created. This sample topology uses only the default VRF, so only one VLAN is created.

Step 1 In the Bridging tile, click VLANs.

Step 2 In the upper right of the VLANs table, click the + (plus sign).

Step 3 On the Add VLAN page, enter the following field values, then click ADD.

  • ID: 4000

  • Name: CORE-ROUTING

  • Description: <no value> (default)

  • Admin Up: checked (default)

  • Voice: unchecked (default)

Step 4 In the VLANs table’s title row, click ← (left arrow) to return to the main configuration page.

Spanning Tree

Multi-chassis link aggregations (MC-LAGs) provide the primary loop prevention mechanism in a Two-Tier architecture. When configured on both core and access switches, MC-LAGs allow loop-free forwarding on all inter-switch links simultaneously in both directions, .

MC-LAGs provide efficient, hash-based load balancing with better performance than individually mapped VLANs to Multiple Spanning-Tree (MST) instances.

Spanning-tree (STP) is configured as a backup loop prevention mechanism in case of operator cabling errors when connecting hosts to top of rack switches.

Setting the spanning-tree priority to 0 ensures that the core VSX pair of switches is the STP root.

Step 1 In the Bridging tile, click Loop Prevention.

Step 2 In the Loop Prevention window, set the following Spanning Tree values, then click SAVE.

  • Priority: 0
  • Region: RSVDC

Enter MultiEdit Configuration

The Central UI interface provides simplified access to most common switch configuration features. MultiEdit is a tool in the Central UI for CX switches that enables configuration of any CX feature using CLI syntax. MultiEdit provides syntax checking, colorization, and command completion.

For complete details on using MultiEdit, refer to the Editing Configuration on AOS-CX section of Central online help.

The text configuration snippets in the following steps are intended for copying and pasting into MultiEdit. To prevent potential copy/paste errors, scroll to the bottom of the configuration, create a new line, then paste the new configuration lines. MultiEdit automatically positions new lines in the correct configuration context.

Step 1 At the upper left of the Switches pane, click the MultiEdit enable slider.

Step 2 Click both core switches in the Devices lists, then click EDIT CONFIG.

Note: When using the Central MultiEditor, it is beneficial to save small sets of configuration at a time. This reduces the volume of configuration that must be inspected when errors occur and makes troubleshooting configuration elements faster.

Configure Core Switch VSX

The core switches are configured as a VSX pair to support Layer 2 multi-chassis link aggregation (MC-LAG) to the access layer switches. The previously defined LAG is assigned as the VSX data path inter-switch link (ISL). The out-of-band (OOB) mgmt interface is used for VSX keepalives to maximize the number of ports available to connect access switches.

Step 1 Enter the initial VSX configuration.

vsx
	system-mac 02:00:00:00:10:00
	inter-switch-link lag 256
	role primary
	keepalive peer 172.16.117.102 source 172.16.117.101 vrf mgmt

Note: When the mgmt vrf is specified, the keepalive peer addresses are the IPs assigned to the out-of-band management interfaces. When using DHCP IP address assignments on the OOB management network, DHCP reservations must be created for VSX paired switches to avoid future keepalive failures.

Step 2 Mouse-over the role value of primary to display the values for each individual switch, then right-click.

Step 3 In the Modify Parameters window, click primary under RSVDC-CORE1-2, select secondary from the menu, then click SAVE CHANGES.

Note: Hover the mouse over the per-switch values to display a switch’s assigned value.

Step 4 Modify the VSX keepalive peer and source parameters by right-clicking on the values.

Switchpeersource
RSVDC-CORE1-1172.16.117.102172.16.117.101
RSVDC-CORE1-2172.16.117.101172.16.117.102

Step 5 Assign a description and maximum MTU value to the VSX ISL physical interfaces.

interface 1/1/31
    description VSX-ISL
    mtu 9198
interface 1/1/32
    description VSX-ISL
    mtu 9198

Configure Core Switch MC-LAGs

Step 1 Create MC-LAG interfaces for connecting to redundant top-of-rack access switches and upstream firewalls.

interface lag 1 multi-chassis
    description RACK-1
    no shutdown
    no routing
    vlan trunk native 1
    vlan trunk allowed all
    lacp mode active
    lacp fallback
    spanning-tree root-guard
interface lag 2 multi-chassis
    description RACK-2
    no shutdown
    no routing
    vlan trunk native 1
    vlan trunk allowed all
    lacp mode active
    lacp fallback
    spanning-tree root-guard
interface lag 101 multi-chassis
    description EXT-FW1-1
    no shutdown
    no routing
    vlan trunk native 1
    vlan trunk allowed 4000
    lacp mode active
    lacp fallback
    spanning-tree root-guard
interface lag 102 multi-chassis
    description EXT-FW1-2
    no shutdown
    no routing
    vlan trunk native 1
    vlan trunk allowed 4000
    lacp mode active
    lacp fallback
    spanning-tree root-guard

Note: MC-LAG interfaces can scope trunked VLANs only to those required for a specific downstream rack. Tagging all VLANs on all core-to-access MC-LAGs supports ubiquitous host mobility across all racks within the Two-Tier structure and reduces the administrative overhead of maintaining VLAN assignments per rack.

Step 2 Assign physical interfaces to the MC-LAGs..

interface 1/1/1
    description RSVDC-ACCESS1-1
    no shutdown
    mtu 9198
    lag 1
interface 1/1/2
    description RSVDC-ACCESS1-2
    no shutdown
    mtu 9198
    lag 1
interface 1/1/3
    description RSVDC-ACCESS2-1
    no shutdown
    mtu 9198
    lag 2
interface 1/1/4
    description RSVDC-ACCESS2-2
    no shutdown
    mtu 9198
    lag 2
interface 1/1/29
    description EXT-FW1-1
    no shutdown
    mtu 9198
    lag 101
interface 1/1/30
    description EXT-FW1-2
    no shutdown
    mtu 9198
    lag 102

Step 3 Remove the following configuration line from interfaces configured above: vlan access 1

Note: Interface LAG assignments and VLAN access statements cannot be assigned to an interface simultaneously. An error occurs when saving the MultiEdit configuration if the vlan access 1 statement is not removed.

Configure Routing Services

In a Two-Tier architecture, the core switches provide IP gateways to downstream hosts, route traffic between data center hosts in different subnets, and route traffic between the data center and external networks.

Configure Routing Components

This sample deployment connects to a campus network via an active/passive pair of upstream firewalls. It is common to place firewalls between the data center and a campus network for granular policy enforcement. When using an active/passive redundant firewall pair, IP assignments are used only by the currently active firewall. Access control lists (ACLs) also can be used to augment firewall policy or in place of a firewall, when policy complexity does not require a dedicated appliance.

OSPF is the most common protocol used between Two-Tier data center core switches and external networks. Alternatively, BGP can be configured.

In the diagram below, OSPF adjacencies are formed between the two core switches and between each core switch and RSVDC-FW1-1, which is the active member of the firewall cluster pair. Under normal operating conditions, RSVDC-FW1-2 does not participate in routing or traffic forwarding. When RSVDC-FW1-2 becomes the active member of the firewall cluster, the OSPF sessions are moved and traffic is forwarded to RSVDC-FW1-2.

Note: When more than one VRF is present, each VRF maintains its own set of OSPF peerings over a unique VLAN on the same physical links.

VSX combines two separate switches into a single, logical Layer 2 switch. Layer 3 functions remain independent. If an external link fails between one of the core switches and the upstream firewall, the same VLAN configured for external network reachability provides a routed transit path over the VSX ISL. An OSPF adjacency is configured between the core switches, which shares external network reachability between them.

Step 1 Set the switch profile.

profile l3-agg

Note: The available profile options are platform dependent. Selecting a profile optimizes switch hardware resources for its role in the network. It is recommended to assign the l3-agg profile to CX 8325 and CX 10000 core switches. CX 8360 switches should use their default aggregation-leaf profile. CX 9300 switches should use their default leaf profile.

Step 2 Create the OSPF process.

router ospf 1
    router-id 10.250.12.1
    passive-interface default
    area 0.0.0.0

Step 3 Mouse-over the OSPF router ID values 10.250.12.1, right-click to set per switch values, set the router-id of RSVDC-CORE-1-2 to 10.250.12.2, and click SAVE CHANGES.

Step 4 Create core switch loopback interfaces. The loopback IP should be the same value assigned to the OSPF router-id.

interface loopback 0
	ip address 10.250.12.1/32
	ip ospf 1 area 0.0.0.0

Step 5 Mouse-over the loopback 0 ip address value of 10.250.12.1, right-click to set per switch values, set the ip address of RSVDC-CORE1-2 to 10.250.12.2/32, and click SAVE CHANGES.

Step 6 Configure the external/transit VLAN SVI.

interface vlan4000
    description CORE-ROUTING-SVI
    ip mtu 9198
    ip address 10.255.12.1/29
    ip ospf 1 area 0.0.0.0
    no ip ospf passive

Step 7 Mouse-over the transit VLAN ip address value of 10.255.12.1, right-click to set per switch values, set the ip address of RSVDC-CORE1-2 to 10.255.12.2/29, and click SAVE CHANGES.

Configure Host VLAN SVIs

Step 1 Configure VLAN switched virtual interfaces (SVIs) for data center host VLANs. Core switches provide the default gateway to downstream data center hosts. An active gateway IP and MAC address are configured for each VLAN to allow both core switches to represent the same IP gateway.

interface vlan101
    description PROD-WEB-SVI
    ip mtu 9198
    ip address 10.12.101.2/24
    active-gateway ip mac 02:00:0a:01:65:01
    active-gateway ip 10.12.101.1
    ip ospf 1 area 0.0.0.0
interface vlan102
    description PROD-DB-SVI
    ip mtu 9198
    ip address 10.12.102.2/24
    active-gateway ip mac 02:00:0a:01:65:01
    active-gateway ip 10.12.102.1
    ip ospf 1 area 0.0.0.0

Note: The sample active gateway MAC address associated with the virtual IP sets the locally administered bit to “1” and embeds a hexadecimal representation of the active gateway IP in the last four octets.

Step 2 Mouse-over the VLAN 101 ip address value of 10.12.101.2, right-click to set per switch values, set the ip address of RSVDC-CORE1-2 to 10.12.101.3, and click SAVE CHANGES.

Step 3 Mouse-over the VLAN 102 ip address value of 10.12.102.2, right-click to set per switch values, set the ip address of RSVDC-CORE1-2 to 10.12.102.3, and click SAVE CHANGES.

Step 4 At the bottom right of the MultiEdit Configuration window, click SAVE.

Two-Tier Multicast

The ESP Two-Tier Data Center uses Protocol Independent Multicast—Sparse-Mode (PIM-SM) to distribute multicast source information and establish interface forwarding state. A centralized Rendezvous Point (RP) registers and distributes multicast sources throughout the network. The data center core switches perform multicast routing and typically point to an RP already established for campus usage to advertise data center sources and learn about campus sources. The Bootstrap Router (BSR) mechanism for PIM elects and learns the active RP.

Internet Group Messaging Protocol (IGMP) is enabled on data center host interfaces (routed-only interfaces and VLAN/SVI interfaces) to identify multicast listeners. When a host is interested in received traffic for a multicast group, it sends an IGMP join message.

Step 1 Enable PIM routing on the core switches.

router pim
    enable
    active-active

Step 2 Enable PIM-SM on external and data center host VLANs.

interface vlan4000
    ip pim-sparse enable
interface vlan101
    ip pim-sparse enable
interface vlan102
    ip pim-sparse enable

Step 3 Enable IGMP on VLAN SVI interfaces for data center hosts.

interface vlan 101
    ip igmp enable
interface vlan 102
    ip igmp enable

Verify Operational State

Step 1 On the left navigation menu, click Tools.

Step 2 On the Tools menu at the top, click the Commands tab.

Step 3 Click the Available Devices dropdown, select both data center core switches, then click elsewhere on the page.

Step 4 In the Categories list, click All Category. Enter vsx in the commands list filter, click show vsx status, then click Add >.

Step 5 Add the following additional commands to the Selected Commands list.

  • show lacp interfaces
  • show ip ospf interface all-vrfs
  • show ip route all-vrfs
  • show spanning-tree mst detail
  • show ip pim
  • show ip igmp
  • show ntp status

Step 6 At the lower left of the Commands pane, click RUN.

Step 7 Scroll down to review the CLI command output for each switch. Verify key results for each command.

  • show vsx status
    • ISL channel: In-Sync
    • ISL mgmt channel: operational
    • Config Sync Status: In-Sync
    • Device Role: set to primary and secondary on corresponding switches
    • Other VSX attributes display equal values for both VSX members

  • show lacp interfaces
    • Both Actor and Partner have corresponding interfaces for each MC-LAG.
    • All Actor interfaces have a Forwarding State of “up” for all host facing MC-LAGs and the upstream core switch facing MC-LAGs.
    • All Actor and Partner interfaces have a state of “ALFNCD”.

  • show ip ospf interface all-vrfs
    • All interfaces display Area “0.0.0.0” and Process “1”.
    • VLAN 4000 State/Type is set to a valid value of DR, BDR or DR-other.
    • VLAN 4000 DR and BDR values are populated with IP addresses on the link.

  • show ip route all-vrfs
    • Verify that a default route is learned from the OSPF protocol and installed in the route table with the upstream firewall as the next hop.
    • Verify that valid campus routes are learned from the OSPF protocol.

  • show spanning-tree mst detail
    • Verify that the Bridge Address and Root Address values are the same.
    • Verify that all LAG interfaces have a Role of “Designated” and State of “Forwarding”.

  • show ip pim
    • Verify that PIM Status is “Enabled”.

  • show ip igmp
    • Verify that each host VLAN has an interface stanza with the Querier IP field populated with an IP address of one of the corresponding core switch VLAN SVIs.

  • show ntp status
    • Verify that NTP Server is populated with a configured NTP server IP address.
    • Verify that the Time Accuracy field is populated.