Abstract
Use this Techdoc to help you to configure primary and backup Shared Ethernet Adapters for load sharing in the Virtual I/O Server.
Table of contents:
Introduction
Requirements
Functionality
Simple test scenario from scratch
- Setup
- How do I know that load sharing is working and how do I know which VLANs are bridged by each SEA?
- Some outputs in disaster cases Enhanced test scenario (adding additional trunk adapter dynamically to the SEA)
- Setup
- Some outputs in disaster cases Switching from SEA Load Sharing to SEA failover
Switching from SEA failover to SEA Load Sharing
FAQs
References
Introduction:
The first implementation to fulfil redundancy requirements for Power System with network virtualization was the introduction of NIB (Network Interface Backup, s. Figure 1)
Figure 1: Network Interface Backup (NIB)
The client LPAR is connected to two separate VLANs. VLAN tagging cannot be supported in this case is because you can’t have one (virtual) switch with two connections.
To get around this problem, a different configuration needs to be used, namely, Shared Ethernet Adapter (SEA) Failover (s. Figure 2)
Figure 2: SEA Failover
The Shared Ethernet Adapter failover configuration provides redundancy only by configuring a backup Shared Ethernet Adapter on a different Virtual I/O Server.
This backup Shared Ethernet Adapter is in the standby mode and can be used only if the primary Shared Ethernet Adapter fails.
Hence, the bandwidth of the backup Shared Ethernet Adapter is not used. When using very expensive 10 Gb/s ethernet adapter the utilization of both, primary and backup adapter, would be highly appreciated. This can be obtained with use of SEA together with virtual switches (s. Figure 3).
Figure 3: Multiple Virtual Ethernet Switches
A significant benefit to this design is that both Virtual I/O Servers can be active at the same time.
Half of the clients could be configured to use Virtual I/O Server 1 and the other half to use Virtual I/O Server 2 as their primary paths. Each client would failover to its respective secondary path in the case that its primary path was lost. So the customer's investment in hardware is more effectively utilized.
If you need more detailed information, refer to the doc "Using Virtual Switches in PowerVM to Drive Maximum Value of 10 Gb Ethernet" (http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101752)
Different to SEA with vswitch mode is the SEA with load sharing configuration, where the SEA algorithm itself decides which VLAN is bridged by the primary or the backup SEA (s. Figure 4).
Figure 4: SEA with Load Sharing
Functionality
In the Shared Ethernet Adapter failover with load sharing configuration, the primary and the backup Shared Ethernet Adapters negotiate the set of virtual local area network (VLAN) IDs that they are responsible for bridging. After successful negotiation, each Shared Ethernet Adapter bridges the assigned trunk adapters and the associated VLANs. Thus, both the primary and the backup Shared Ethernet Adapter bridge the workload for their respective VLANs. If a failure occurs, the active Shared Ethernet Adapter bridges all trunk adapters and the associated VLANs. This action helps to avoid disruption in network services (s. Figure 4).
Requirements
--> Both of primary and backup Virtual I/O Servers are at Version 2.2.1.0, or later.
--> Two or more trunk adapters are configured for the primary and backup SEA pair.
--> Load Sharing mode must be enabled on both primary and backup SEA pair.
--> The virtual local area network (VLAN) definitions of the trunk adapters are
identical between the primary and backup SEA pair.
--> You need to set the same priority to all trunk adapters under one SEA. The
primary and backup priority definitions are set at the SEA level, not at trunk
adapters level.
Simple test scenario from scratch
If you already have traditional SEA failover configured refer to section “How to switch from ha_mode to sharing mode”.
For maintainability we only have tagged traffic in our szenario.
After doing below steps your setup will look like this:
Figure 5: SEA Load Sharing Scenario
| p72vio1: | p72vio2: |
| State: PRIMARY_SH Bridge Mode: Partial VID shared: 1 10 11 High Availability Mode: Sharing Priority: 1 | State: BACKUP_SH Bridge Mode: Partial VID shared: 2 12 13 High Availability Mode: Sharing Priority: 2 |
If the ha_mode is set to "sharing" on both the primary and the backup SEA but the state of the SEA is "PRIMARY" or "BACKUP" as opposed to "PRIMARY_SH" or "BACKUP_SH" something went wrong.
An error must have occurred which forced SEA to go back to non-sharing state even though Load sharing is enabled. Try to enable Load sharing again by running a chdev command on the backup SEA device. Check the SEA state again. If the SEA has recovered from its error, then it must be operating in Load sharing mode again. If the SEA is still not operating in Load sharing mode, then further detail investigation is required.
Setup
1.) Create virtual ethernet adapters on p72vio1
HMC --> select p72vio1 --> select Configuration --> select Manage Profiles
--> select the correct profile
--> Actions --> Edit --> Select Virtual Adapters
--> Actions --> Create Virtual Adapter --> Ethernet Adapter
Create first trunk adapter on p72vio1:
Create second trunk adapter on p72vio1:
Create Control Channel on p72vio1:
2.) Create virtual ethernet adapters on p72vio2
HMC --> select p72vio2 --> select Configuration --> select Manage Profiles
--> select the correct profile
--> Actions --> Edit --> Select Virtual Adapters
--> Actions --> Create Virtual Adapter --> Ethernet Adapter
Create first Trunk Adapter on p72vio2:
Create second Trunk Adapter on p72vio2:
Create Control Channel on p72vio2:
3.) Power on both VIO servers
p72vio1:
# lsdev -Cc adapter |grep Ethernet
ent0 Available Logical Host Ethernet Port (lp-hea)
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
ent2 Available Virtual I/O Ethernet Adapter (l-lan)
ent3 Available Virtual I/O Ethernet Adapter (l-lan)
lhea0 Available Logical Host Ethernet Adapter (l-hea)
# entstat -d ent1 |grep Trunk
Trunk Adapter: True
# entstat -d ent2 |grep Trunk
Trunk Adapter: True
# entstat -d ent1 |grep VLAN
Port VLAN ID: 1
VLAN Tag IDs: 10 11
# entstat -d ent2 |grep VLAN
Port VLAN ID: 2
VLAN Tag IDs: 12 13
p72vio2:
# lsdev -Cc adapter |grep Ethernet
ent0 Available Logical Host Ethernet Port (lp-hea)
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
ent2 Available Virtual I/O Ethernet Adapter (l-lan)
ent3 Available Virtual I/O Ethernet Adapter (l-lan)
lhea0 Available Logical Host Ethernet Adapter (l-hea)
# entstat -d ent1 |grep Trunk
Trunk Adapter: True
# entstat -d ent2 |grep Trunk
Trunk Adapter: True
# entstat -d ent1 |grep VLAN
Port VLAN ID: 1
VLAN Tag IDs: 10 11
# entstat -d ent2 |grep VLAN
Port VLAN ID: 2
VLAN Tag IDs: 12 13
On HMC:
hscroot@p7hmc:~>
lshwres -r virtualio --rsubtype eth -m p72 --level lpar |grep p72vio1
lpar_name=p72vio1,lpar_id=1,slot_num=2,state=1,is_required=0, is_trunk=1,trunk_priority=1,ieee_virtual_eth=1, port_vlan_id=1,vswitch=ETHERNET0,"addl_vlan_ids=10,11",mac_addr=761FA3934802,allowed_os_mac_addrs=all,qos_priority=none
lpar_name=p72vio1,lpar_id=1,slot_num=3,state=1,is_required=0, is_trunk=1,trunk_priority=1,ieee_virtual_eth=1, port_vlan_id=2,vswitch=ETHERNET0,"addl_vlan_ids=12,13",mac_addr=761FA3934803,allowed_os_mac_addrs=all,qos_priority=none
lpar_name=p72vio1,lpar_id=1,slot_num=4,state=1,is_required=0, is_trunk=0,ieee_virtual_eth=0,port_vlan_id=99,vswitch=ETHERNET0,addl_vlan_ids=,mac_addr=761FA3934804,allowed_os_mac_addrs=all,qos_priority=none
hscroot@p7hmc:~>
lshwres -r virtualio --rsubtype eth -m p72 --level lpar |grep p72vio2
lpar_name=p72vio2,lpar_id=2,slot_num=2,state=1,is_required=0, is_trunk=1,trunk_priority=2,ieee_virtual_eth=1, port_vlan_id=1,vswitch=ETHERNET0,"addl_vlan_ids=10,11",mac_addr=761FA4F6A602,allowed_os_mac_addrs=all,qos_priority=none
lpar_name=p72vio2,lpar_id=2,slot_num=3,state=1,is_required=0, is_trunk=1,trunk_priority=2,ieee_virtual_eth=1, port_vlan_id=2,vswitch=ETHERNET0,"addl_vlan_ids=12,13",mac_addr=761FA4F6A603,allowed_os_mac_addrs=all,qos_priority=none
lpar_name=p72vio2,lpar_id=2,slot_num=4,state=1,is_required=0, is_trunk=0,ieee_virtual_eth=0,port_vlan_id=99,vswitch=ETHERNET0,addl_vlan_ids=,mac_addr=761FA4F6A604,allowed_os_mac_addrs=all,qos_priority=none
4.) Creating SEA with Load Sharing mode
If all the Load sharing criteria are satisfied, then Load sharing can be enabled by setting the ha_mode attribute of SEA device to "sharing". This value must be set on primary SEA first before it is set for the backup SEA, because the backup SEA initiates the request for load sharing. If this sequence is not followed (means not activated on the primary first), then a chdev must occur on the backup SEA for Load sharing to work.
- When the backup SEA is configured in Load sharing mode it initiates the request for Load sharing by sending a special packet over the control channel to the primary SEA.
- The packet contains the list of VLANs the backup SEA proposes to take over for bridging.
- When the primary SEA receives the request for sharing it verifies the request and grants the request if it meets the sharing criteria.
- After the primary SEA grants the request, it switches over to sharing mode and sends an ACK packet to backup SEA via the control channel.
- When the backup SEA receives the ACK packet it switches to sharing mode and starts bridging for VLANs it had proposed to bridge.
- From then on, heartbeats are exchanged between the primary and the backup SEA in load sharing mode.
- Different to the traditional SEA failover mode, both the primary and the backup SEAs send and receive heartbeats.
Note that these heartbeats are in addition to heartbeats sent by the primary SEA in traditional SEA failover mode
- When either side (primary or backup) fails to receive load sharing heartbeats for a predetermined period of time, it is assumed that the other SEA has encountered a problem and it falls back to non-sharing mode and starts bridging traffic for all VLANs.
- To restart Load sharing, a chdev must occur on the backup SEA for the backup SEA to reinitiate Load sharing request.
On p72vio1:
$ mkvdev -sea ent0 -vadapter ent1,ent2 -default ent1 -defaultid 1 -attr ha_mode=sharing ctl_chan=ent3
ent4 Available
en4
et4
On p72vio2:
$ mkvdev -sea ent0 -vadapter ent1,ent2 -default ent1 -defaultid 1 -attr ha_mode=sharing ctl_chan=ent3
ent4 Available
en4
et4
How do I know that load sharing is working and how do I know which VLANs are bridged by each SEA?
--> Check the value of ha_mode attribute of the SEA device. It must be set to
"sharing" on both the primary SEA and the backup SEA
--> Run entstat -all (as padmin) or entstat -d (as root) for SEA device. If the SEA
is operating in Load sharing state, then its state must be either PRIMARY_SH or
BACKUP_SH
--> Run the entstat on the SEA device and look for values for VID Bridged. These are
the VLANs bridged by the SEA while SEA is operating in Load sharing mode.
On p72vio1:
# lsattr -El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode sharing High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
…
State: PRIMARY_SH
Bridge Mode: Partial
VID shared: 1 10 11
High Availability Mode: Sharing
Priority: 1
…
On p72vio2:
# lsattr -El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode sharing High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
…
State: BACKUP_SH
Bridge Mode: Partial
VID shared: 2 12 13
High Availability Mode: Sharing
Priority: 2
…
Some outputs in disaster cases
Shutdown primary vio server (p72vio1) | |
| p72vio1 (primary): | p72vio2 (backup): |
 | # errpt E136EAFA 0112173112 I H ent4 BECOME PRIMARY … # entstat –d ent4 … State: PRIMARY Bridge Mode: All High Availability Mode: Sharing Priority: 2 … |
Power on primary vio server (p72vio1) | |
| p72vio1 (primary): | p72vio2 (backup): |
| # errpt … E136EAFA 0112105212 I H ent4 BECOME PRIMARY … # entstat –d ent4 … State: PRIMARY_SH Bridge Mode: Partial VID shared: 1 10 11 High Availability Mode: Sharing Priority: 1 … | # errpt … 40D97644 0112174812 I H ent4 BECOME BACKUP … # entstat –d ent4 … State: BACKUP_SH Bridge Mode: Partial VID shared: 2 12 13 High Availability Mode: Sharing Priority: 2 … |
Shutdown backup vio server (p72vio2) | |
| p72vio1 (primary): | p72vio2 (backup): |
| # errpt … # entstat –d ent4 … State: PRIMARY Bridge Mode: All High Availability Mode: Sharing Priority: 1 … | |
Power on backup vio server (p72vio2) | |
| p72vio1 (primary): | p72vio2 (backup): |
| # errpt … # entstat –d ent4 … State: PRIMARY_SH Bridge Mode: Partial VID shared: 1 10 11 High Availability Mode: Sharing Priority: 1 … | On backup vio server: # errpt … 40D97644 0112180112 I H ent4 BECOME BACKUP … # entstat –d ent4 … State: BACKUP_SH Bridge Mode: Partial VID shared: 2 12 13 High Availability Mode: Sharing Priority: 2 … |
Enhanced test scenario (adding additional trunk adapter dynamically to the SEA)
Figure 6: Enhanced SEA Load Sharing Scenario
| p72vio1: | p72vio2: |
| State: PRIMARY_SH Bridge Mode: Partial VID shared: 3 4 33 44 333 High Availability Mode: Sharing Priority: 1 | State: BACKUP_SH Bridge Mode: Partial VID shared: 1 2 10 11 12 13 High Availability Mode: Sharing Priority: 2 |
Setup
1.) create additional trunk adapter on the vio server with dlpar
HMC --> select p72vio1 --> select Dynamic Logical Partitioning Select Virtual
Adapter
--> Actions --> Create Virtual Adapter --> Ethernet Adapter
Create additional trunk adapter on p72vio1:
1.) create additional trunk adapter on the vio server with dlpar
HMC --> select p72vio2 --> select Dynamic Logical Partitioning Select Virtual
Adapter
--> Actions --> Create Virtual Adapter --> Ethernet Adapter
Create additional trunk adapter on p72vio2:
Create additional trunk adapter on p72vio2:
3.) add adapter to the profile
(starting with HMC 7.7.3 the existing profile can be overwritten,
previously you needed to specify new profile name )
On both vio server (p72vio1 and p72vio2):
HMC --> select Configuration --> seclect Save Current Configuration Overwrite
Existing Profile
4.) run cfgmgr (or cfgdev as padmin) on both vio server (p72vio1 and p72vio2):
# cfgmgr
# lsdev -Cc adapter |grep Ethernet
ent0 Available Logical Host Ethernet Port (lp-hea)
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
ent2 Available Virtual I/O Ethernet Adapter (l-lan)
ent3 Available Virtual I/O Ethernet Adapter (l-lan)
ent4 Available Shared Ethernet Adapter
ent5 Available Virtual I/O Ethernet Adapter (l-lan)
ent6 Available Virtual I/O Ethernet Adapter (l-lan)
lhea0 Available Logical Host Ethernet Adapter (l-hea)
new virual adapter ent5 and ent6
5.) Add the new trunk adapter to the SEA
On p72vio1:
$ chdev -dev ent4 -attr virt_adapters=ent1,ent2,ent5,ent6
ent4 changed
On p72vio2:
$ chdev -dev ent4 -attr virt_adapters=ent1,ent2,ent5,ent6
ent4 changed
How do I know that load sharing is working and how do I know which VLANs are bridged by each SEA?
On p72vio1:
# lsattr –El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode sharing High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
...
State: PRIMARY_SH
Bridge Mode: Partial
VID shared: 3 4 33 44 333
High Availability Mode: Sharing
Priority: 1
...
On p72vio2:
# lsattr -El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode sharing High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
...
State: BACKUP_SH
Bridge Mode: Partial
VID shared: 1 2 10 11 12 13
High Availability Mode: Sharing
Priority: 2
...
Some outputs in disaster cases
Shutdown primary vio server (p72vio1) | |
| p72vio1 (primary): | p72vio2 (backup): |
| # errpt … E136EAFA 0113115612 I H ent4 BECOME PRIMARY … # entstat –d ent4 … State: PRIMARY Bridge Mode: All High Availability Mode: Sharing Priority: 2 … |
Power on primary vio server (p72vio1) | |
| p72vio1 (primary): | p72vio2 (backup): |
| # errpt: … E136EAFA 0113050512 I H ent4 BECOME PRIMARY … # entstat –d ent4 … State: PRIMARY_SH Bridge Mode: Partial VID shared: 3 4 33 44 333 High Availability Mode: Sharing Priority: 1 … | # errpt: … 40D97644 0113120112 I H ent4 BECOME BACKUP … # entstat –d ent4 … State: BACKUP_SH Bridge Mode: Partial VID shared: 1 2 10 11 12 13 High Availability Mode: Sharing Priority: 2 … |
Switching from SEA Load Sharing to SEA failover
Switching to auto or to load sharing, you have to change ha_mode on the primary SEA first before changing ha_mode on the backup SEA.
On p72vio1 (Primary):
$ chdev -dev ent4 -attr ha_mode=auto
ent4 changed
# lsattr -El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode auto High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
State: PRIMARY
Bridge Mode: All
High Availability Mode: Auto
Priority: 1
On p72vio2 (Backup):
$ chdev -dev ent4 -attr ha_mode=auto
ent4 changed
# lsattr -El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode auto High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
Limbo Packets: 0
State: BACKUP
Bridge Mode: None
High Availability Mode: Auto
Priority: 2
Switching from SEA failover to SEA Load Sharing
On p72vio1 (Primary):
$ chdev -dev ent4 -attr ha_mode=sharing
ent4 changed
# lsattr -El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode sharing High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
State: PRIMARY
Bridge Mode: All
High Availability Mode: Sharing
Priority: 1
On p72vio2 (Backup):
$ chdev -dev ent4 -attr ha_mode=sharing
ent4 changed
# lsattr -El ent4
...
ctl_chan ent3 Control Channel adapter for SEA failover True
...
ha_mode sharing High Availability Mode True
...
pvid 1 PVID to use for the SEA device True
pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ...
real_adapter ent0 Physical adapter associated with the SEA True
...
virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)
# entstat –d ent4
State: BACKUP_SH
Bridge Mode: Partial
VID shared: 1 2 10 11 12 13
High Availability Mode: Sharing
Priority: 2
On p72vio1 (Primary):
State: PRIMARY_SH
Bridge Mode: Partial
VID shared: 3 4 33 44 333
High Availability Mode: Sharing
Priority: 1
FAQs
1) What is the algorithm that the VIOS pair uses to determine which VIOS should "own" the VID's?
VIOS divides the number of trunk adapters by 2. One VIOS bridges all VLANs of one half, and the other VIOS does bridging for VLANs of the other half.
2) As time goes on will the list of VID's change for each VIOS? (related to my algorithm question).
Not if the SEA configuration remains the same.
3) In load sharing mode is there any potential for broadcast storm if either of the VIOS are rebooted? (one at a time).
No.
4) Before rebooting each VIOS (one at a time) do I need to switch it back to "auto" mode?
No.
5) If I want to rmdev the SEA can I do this when it's in "sharing" mode on either VIOS? Based on what I've read I'd have to do this on the Primary SEA VIOS (with the lowest priority) first. In "auto" mode I can run rmdev against the SEA on either VIOS.
Yes.
No comments:
Post a Comment