Tag Archives: virtual chassis

Configuring a Virtual Chassis on QFX5100

Reading Time: 2 minutes

When configuring a 2 member Virtual Chassis using 2xQFX5100, there is a slight difference compared to EX Series but it’s very much similar. As this is the case, I thought I’d do a quick post (And it doubles up as documentation writing for work lol). The QFX doesn’t have dedicated VC ports or a VC module, so with this in mind you’ll have to use either 10GB SPF+ port(s) or 40GB QSFP(s) ports to connect the switches together. The method is same as Configuring Virtual Chassis on EX switch using VCEP ports, the one difference is that with the QFX you can do the entire configuration with the VCEP port pre-connected, but this wasn’t the case with EX Series. However, similarly, it’s recommended that you have Backup Routing Engine (RE) or Linecard powered off if you’re using the preprovisioned method like I am 🙂

Let’s get cracking 😀

In one of my previous post, Configuring Virtual Chassis on Juniper EX Series, it’s recommended that you have the following commands set before processing with configuring a Virtual Chassis:

set system commit synchronize
set chassis redundancy graceful-switchover
set routing-options nonstop-routing
set protocols layer2-control nonstop-bridging
Note
The QFX non-bridging command is under the protocols layer2-control stanza NOT ethernet-switching-options stanza as on the EX Series

Once these have been committed, you can configure the virtual-chassis stanza.

[email protected]> show configuration virtual-chassis | display set 
set virtual-chassis preprovisioned
set virtual-chassis no-split-detection
set virtual-chassis member 0 role routing-engine
set virtual-chassis member 0 serial-number TA3715110057
set virtual-chassis member 1 role routing-engine
set virtual-chassis member 1 serial-number TA3715110028

Just like on EX Series, you have to set the VC-Ports on the Master Routing Engine for them to know those ports are being used as the Virtual Chassis interconnects

[email protected]> request virtual-chassis vc-port set pic-slot 0 port 48 
[email protected]> request virtual-chassis vc-port set pic-slot 0 port 50

You can power up (the Backup RE), having completed the entire configuration on the Master RE. Once the Backup has booted, as done on the Master, you have to set the 40GB QSFPs ports as the VC-Ports

[email protected]> request virtual-chassis vc-port set pic-slot 0 port 48 
[email protected]> request virtual-chassis vc-port set pic-slot 0 port 50

Once this is done, the Virtual Chassis is created and you’ll be kicked out of the Backup RE and will have to log back into the switch, where you will be logged into the Master Routing Engine. To verify that everything is working as expected you can run the commands show virtual-chassis vc-port and show virtual-chassis

show virtual-chassis vc-portshow virtual-chassis
[email protected]> show virtual-chassis vc-port    
fpc0:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
0/48        Configured          5    Up           40000        1   vcp-255/0/48
0/50        Configured          5    Up           40000        1   vcp-255/0/50

fpc1:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
0/48        Configured          5    Up           40000        0   vcp-255/0/48
0/50        Configured          5    Up           40000        0   vcp-255/0/50
[email protected]> show virtual-chassis 

Preprovisioned Virtual Chassis
Virtual Chassis ID: 1165.24bd.5581
Virtual Chassis Mode: Enabled
                                                Mstr           Mixed Route Neighbor List
Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
0 (FPC 0)  Prsnt    TA3715110057 qfx5100-48s-6q 129   Master*      N  VC   1  vcp-255/0/48
                                                                           1  vcp-255/0/50
1 (FPC 1)  Prsnt    TA3715110028 qfx5100-48s-6q 129   Backup       N  VC   0  vcp-255/0/48
                                                                           0  vcp-255/0/50

And you’re done! As I said, it’s very similar to configuring a Virtual Chassis on EX Series, except for a couple of small changes that could throw someone off if they didn’t know!

For more in-depth detail you can check Juniper’s TechLibrary page

Share this:
Share

Virtual Chassis Upgrade with Minimal Downtime

Reading Time: 6 minutes

At work we were looking to do a firmware upgrade of our junos going from 12.3 to 13.2X and we got a few VC switches. The plan was to use the NSSU method so that we didn’t get any downtime however, when doing testing I would kick off the NSSU and the backup member would upgrade, reboot and come up as expected:

{master:0}
[email protected]> ...p/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz    
Chassis ISSU Check Done
[Dec 18 04:32:13]:ISSU: Validating Image
[Dec 18 04:32:41]:ISSU: Preparing Backup RE
[Dec 18 04:32:42]: Installing image on other FPC's along with the backup

[Dec 18 04:32:42]: Checking pending install on fpc1
[Dec 18 04:33:41]: Pushing bundle to fpc1
NOTICE: Validating configuration against mchassis-install.tgz.
NOTICE: Use the 'no-validate' option to skip this if desired.
WARNING: A reboot is required to install the software
WARNING:     Use the 'request system reboot' command immediately
[Dec 18 04:34:42]: Completed install on fpc1
[Dec 18 04:34:53]: Backup upgrade done
[Dec 18 04:34:53]: Rebooting Backup RE

Rebooting fpc1
[Dec 18 04:34:54]:ISSU: Backup RE Prepare Done
[Dec 18 04:34:54]: Waiting for Backup RE reboot

After an hour of looking at this on the master, I consoled into the backup to see what had booted and was up, and I clearly had an issue. I aborted the NSSU and checked to see what was going; the backup member had upgraded and had connected with the master:

{master:0}
[email protected]> show version 
fpc0:
--------------------------------------------------------------------------
Hostname: EX4200-A
Model: ex4200-48t
JUNOS Base OS boot [12.3R5.7]
JUNOS Base OS Software Suite [12.3R5.7]
JUNOS Kernel Software Suite [12.3R5.7]
JUNOS Crypto Software Suite [12.3R5.7]
JUNOS Online Documentation [12.3R5.7]
JUNOS Enterprise Software Suite [12.3R5.7]
JUNOS Packet Forwarding Engine Enterprise Software Suite [12.3R5.7]
JUNOS Routing Software Suite [12.3R5.7]
JUNOS Web Management [12.3R5.7]
JUNOS FIPS mode utilities [12.3R5.7]

fpc1:
--------------------------------------------------------------------------
Hostname: EX4200-A
Model: ex4200-48t
JUNOS EX  Software Suite [13.2X51-D35.3]
JUNOS FIPS mode utilities [13.2X51-D35.3]
JUNOS Online Documentation [13.2X51-D35.3]
JUNOS EX 4200 Software Suite [13.2X51-D35.3]
JUNOS Web Management [13.2X51-D35.3]

I thought this was very odd so I checked the logs to see if anything was out of the norm and saw that VCP ports had come up however, the attempts to backup member had timed out :/

show log messages output
[email protected]> show log messages | last 100    
Dec 18 04:44:33  EX4200-A /kernel: tcp_timer_rexmt: Dropping socket connection due to error: 65
Dec 18 04:44:36  EX4200-A last message repeated 4 times
Dec 18 05:01:30  EX4200-A chassism[1280]: cm_ff_ifd_disable: fast failover disabled for internal-0/26
Dec 18 05:01:30  EX4200-A chassism[1280]: cm_ff_ifd_disable: fast failover disabled for internal-0/27
Dec 18 05:01:30  EX4200-A vccpd[1282]: ifl vcp-0.32768 set up, ifl flags 0, flags 1
Dec 18 05:01:30  EX4200-A vccpd[1282]: interface vcp-0 came up
Dec 18 05:01:30  EX4200-A chassism[1280]: cm_ff_ifd_disable: fast failover disabled for internal-1/26
Dec 18 05:01:30  EX4200-A vccpd[1282]: ifl vcp-1.32768 set up, ifl flags 0, flags 1
Dec 18 05:01:30  EX4200-A vccpd[1282]: interface vcp-1 came up
Dec 18 05:01:30  EX4200-A chassism[1280]: cm_ff_ifd_disable: fast failover disabled for internal-1/27
Dec 18 05:01:30  EX4200-A vccpd[1282]: Member 0, interface vcp-1.32768 came up
Dec 18 05:01:30  EX4200-A vccpd[1282]: Member 0, interface vcp-0.32768 came up
Dec 18 05:01:30  EX4200-A vccpd[1282]: Member 1, interface vcp-1.32768 came up
Dec 18 05:01:30  EX4200-A vccpd[1282]: Member 1, interface vcp-0.32768 came up
Dec 18 05:01:36  EX4200-A chassism[1280]: cm_ff_vcp_port_add: fast failover received VCP port add on dev 0 port 26
Dec 18 05:01:36  EX4200-A chassism[1280]: cm_ff_vcp_port_add: fast failover received VCP port add on dev 0 port 27
Dec 18 05:01:36  EX4200-A chassism[1280]: cm_ff_vcp_port_add: fast failover received VCP port add on dev 1 port 26
Dec 18 05:01:36  EX4200-A chassism[1280]: cm_ff_vcp_port_add: fast failover received VCP port add on dev 1 port 27
Dec 18 05:01:36  EX4200-A chassism[1280]: CM_CHANGE: Member 0->0, Mode M->M, 0M 1B, GID 0, Master Unchanged, Members Changed
Dec 18 05:01:36  EX4200-A chassism[1280]: CM_CHANGE: 0M 1B
Dec 18 05:01:36  EX4200-A chassism[1280]: CM_CHANGE: Signaling license service
Dec 18 05:01:36  EX4200-A chassism[1280]: mvlan_member_change_add: member id 1 (my member id 0, my role 1)
Dec 18 05:01:36  EX4200-A chassism[1280]: mvlan_ifl_create: Creating ifl, name bme0, subunit 32770
Dec 18 05:01:36  EX4200-A chassism[1280]: mvlan_rts_ifl_op: IFL idx is 8 is created
Dec 18 05:01:39  EX4200-A chassisd[1298]: CHASSISD_VERSION_MISMATCH: Version mismatch:   chassisd message version 2   FPC 1 message version 2   local IPC version $Revision: 590540 $   remote IPC version $Revision: 653007 $
Dec 18 05:01:42  EX4200-A license-check[1331]: LICENSE: copy to /config/license from fpc0:/config/.license_priv/
Dec 18 05:01:42  EX4200-A license-check[1331]: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -Ji fpc0:/config/.license_priv/ /config/license : rcp: /config/.license_priv/: No such file or directory
Dec 18 05:01:42  EX4200-A license-check[1331]: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -Ji fpc1:/config/.license_priv/ /config/license : rcp: /config/.license_priv/: No such file or directory
Dec 18 05:01:42  EX4200-A license-check[1331]: copy from member 0 failed
Dec 18 05:01:42  EX4200-A license-check[1331]: LICENSE: copy to /config/license from fpc1:/config/.license_priv/
Dec 18 05:01:42  EX4200-A license-check[1331]: copy from member 1 failed
Dec 18 05:01:50  EX4200-A bdbrepd: Subscriber Management is ready for GRES
Dec 18 05:01:52  EX4200-A license-check[1331]: LICENSE: copy to /config/license from fpc0:/config/.license_priv/
Dec 18 05:01:52  EX4200-A license-check[1331]: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -Ji fpc0:/config/.license_priv/ /config/license : rcp: /config/.license_priv/: No such file or directory
Dec 18 05:01:52  EX4200-A license-check[1331]: copy from member 0 failed
{...}
Dec 18 05:02:39  EX4200-A chassisd[1298]: CHASSISD_FRU_ONLINE_TIMEOUT: fpc_online_timeout: attempt to bring FPC 1 online timed out
Dec 18 05:03:39  EX4200-A chassisd[1298]: CHASSISD_FRU_ONLINE_TIMEOUT: fpc_online_timeout: attempt to bring FPC 1 online timed out
Dec 18 05:03:39  EX4200-A chassisd[1298]: CHASSISD_FRU_UNRESPONSIVE: Error for FPC 1: attempt to bring online timed out; restarted it
Dec 18 05:03:39  EX4200-A chassisd[1298]: CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 1 offline: Restarting unresponsive board
Dec 18 05:03:39  EX4200-A chassisd[1298]: CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(1)
Dec 18 05:03:39  EX4200-A chassisd[1298]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4200-48T, 8 POE @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
Dec 18 05:03:40  EX4200-A chassisd[1298]: CHASSISD_VERSION_MISMATCH: Version mismatch:   chassisd message version 2   FPC 1 message version 2   local IPC version $Revision: 590540 $   remote IPC version $Revision: 653007 $

It was Friday and I had a planned upgrade for the following week, so I didn’t have the time to raise a JTAC case (which I should have probably done but that could come later). With this in mind I thought I should be able to manually failover the Routing-Engines and upgrade each member the same way without all of the magic of the NSSU:

NSSU Note
It took longer than expected to do this testing and I had to cancel my change. I found out that currently (as in when this was written) you can’t use NSSU to upgrade from 12.3 to any higher versions. This explained why everything was breaking and giving me issues. After raising this with our Technical Account Manager at Juniper, he provided details on What version Of Junos supports NSSU on EX Series.

Soooooooo this is what this post will be about, the success or failure of manually failing over a VC with minimal downtime 🙂

Let’s get cracking!

I was using 2x EX4200 with JUNOS 12.3R5.7; it’s the same setup I had in my previous Virtual Chassis post. I used the preprovisioned method of stacking the switches, and had the following VC specific configuration applied:

show routing-optionsshow chassisshow virtual-chassis
[email protected]# show routing-options 
nonstop-routing;
static {
    route 0.0.0.0/0 {
        next-hop 10.1.0.1;
        no-readvertise;
    }
}
[email protected]# show chassis 
redundancy {
    graceful-switchover;
}
[email protected]# show virtual-chassis 
preprovisioned;
no-split-detection;
member 0 {
    role routing-engine;
    serial-number BP0214340104;
}
member 1 {
    role routing-engine;
    serial-number BP0215090120;
}
fast-failover {
    ge;
    xe;
}

It’s important to make sure you have nonstop-routing, graceful-switchover and no-split-detection configured without these or you will most likely get a split brain affect and that’s not a good thing!

I’ve got a VM connected to both switches in LACP bond configured

[email protected]> show lldp neighbors 
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/2.0         ae1.0               00:0c:29:4f:26:bb   eth1               km-vm1              
ge-1/0/2.0         ae1.0               00:0c:29:4f:26:bb   eth2               km-vm1

and I have the VM pinging it default gateway (192.31.1.1), which is the l3-interface on the switch

[email protected]:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.31.1.1      0.0.0.0         UG    0      0        0 bond0
10.1.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.31.1.0      0.0.0.0         255.255.255.0   U     0      0        0 bond0

Now everything is sorted, let’s try some stuff!

As the VM is dual connected to both members, I’ll shutdown the interfaces and the VCP ports of backup switch, upgrade it and then do the same on the master switch. In essence, I’ll be breaking the VC to upgrade each switch individually. I’ll be running a continuous ping from the VM switch and will be able see if any packets are dropped during this work.

I start with the backup member. I have to disable the data and break the virtual chassis by disabling the VCP ports. I had to copy over the junos package from member 0 to member 1, as I’d have no access to member 0 once the virtual chassis had been broken.

[email protected]> file copy /tmp/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz fpc1:/tmp/

This will copy the package from the member 0 to member 1. Confirmed by entering the shell cli and checking the /tmp folder on member 1

{backup:1}
[email protected]> start shell 
[email protected]:BK:1% cd /tmp/
[email protected]:BK:1% ls -la
total 234744
drwxrwxrwt   3 root  wheel           512 Dec 18 14:57 .
drwxr-xr-x  23 root  wheel           512 Dec 18 04:06 ..
-rw-r--r--   1 root  wheel            92 Dec 18 12:13 .clnpkg.LCK
-rw-r--r--   1 root  wheel            92 Dec 18 12:13 .pkg.LCK
drwxrwxr-x   2 root  operator        512 Dec 18 12:10 .snap
-rw-r--r--   1 root  wheel     120120669 Dec 18 14:58 jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz
-rw-r--r--   1 root  wheel           393 Dec 18 12:10 partitions.spec
[email protected]:BK:1% exit

Next disable the member 1 port, in my case ge-1/0/2, deactivate interfaces ge-1/0/2

[email protected]# run show interfaces ge-1/0/2          
Physical interface: ge-1/0/2, Administratively down, Physical link is Down

The server dropped 3 packets, which is acceptable to most; so far so good. Next I disabled the VCP on the member 1 and member 0 and then console onto member 1.

[email protected]> request virtual-chassis vc-port set interface vcp-0 member 1 disable 
  [email protected]> request virtual-chassis vc-port set interface vcp-1 member 1 disable
  [email protected]> request virtual-chassis vc-port set interface vcp-0 disable 
  [email protected]> request virtual-chassis vc-port set interface vcp-1 disable

On member 1, it automatically took mastership and doesn’t member 0 anymore

{master:1}
[email protected]> show virtual-chassis status 

Preprovisioned Virtual Chassis
Virtual Chassis ID: e8a9.d27b.0f05
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  NotPrsnt BP0214340104 ex4200-48t
1 (FPC 1)  Prsnt    BP0215090120 ex4200-48t 129  Master*      N

The server is still pinging along, so now we can upgrade the backup member as if it was a standalone device. We’ll run request system software add /tmp/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz reboot validate reboot

Once member 1 rebooted I had to wait for a bit as it was looking for the master (due to the preprovisioned config) and it initial booted as a linecard however, it changed back to master after I entered the operational mode.

Next I enabled the member 1 port, activate interfaces ge-1/0/2

Comment
When I went to commit the change it took an awfully long time to activate the interface however, with a bit of patience the interface did come back up….. Eventually! Patience is the Key!

To double check and confirm it was up, I checked the lldp neighbor

[email protected]> show interfaces ge-1/0/2   
Physical interface: ge-1/0/2, Enabled, Physical link is Up
{master:1}
[email protected]> show lldp neighbors 
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-1/0/2.0         ae1.0               00:0c:29:4f:26:bb   eth2               km-vm1

Now disable the member 0 port, in my case ge-0/0/2, deactivate interfaces ge-0/0/2

The Server had dropped 47 packets after the interface was disabled. This was most likely due to the convergence time for the LACP bond and the port going down, and this is shown in the log messages

Logs
Dec 18 17:01:19  EX4200-A /kernel: Percentage memory available(19)less than threshold(20 %)- 14
Dec 18 17:01:50  EX4200-A dcd[5164]: ae0 : Warning: aggregated-ether-options link-speed no kernel value! default to  0
Dec 18 17:01:50  EX4200-A dcd[5164]: check_prot: p_ae NULL, ifdp->ifdp_type is 25 ifdp_ifname ae1
Dec 18 17:01:50  EX4200-A mgd[3713]: UI_CHILD_EXITED: Child exited: PID 5164, status 1, command '/sbin/dcd'
Dec 18 17:03:27  EX4200-A mgd[3713]: UI_COMMIT: User 'root' requested 'commit' operation (comment: none)
Dec 18 17:03:34  EX4200-A /kernel: Percentage memory available(19)less than threshold(20 %)- 15
Dec 18 17:04:07  EX4200-A dcd[5205]: ae0 : Warning: aggregated-ether-options link-speed no kernel value! default to  0
Dec 18 17:04:07  EX4200-A dcd[5205]: ae1 : Warning: aggregated-ether-options link-speed no kernel value! default to  0
Dec 18 17:04:12  EX4200-A lldpd[1326]: UI_CONFIGURATION_ERROR: Process: lldpd, path: , statement: , Configuration database open failure: Database is already open
Dec 18 17:04:13  EX4200-A mgd[3713]: UI_DBASE_LOGOUT_EVENT: User 'root' exiting configuration mode
Dec 18 17:04:52  EX4200-A dcd[1297]: ae0 : aggregated-ether-options link-speed set to kernel value of  10000000000
Dec 18 17:04:52  EX4200-A dcd[1297]: ae1 : Warning: aggregated-ether-options has no childern! link-speed set to  0
Dec 18 17:04:52  EX4200-A /kernel: ae_bundlestate_ifd_change: bundle ae1: bundle IFD minimum links not met 0 < 1
Dec 18 17:04:52  EX4200-A mib2d[1304]: SNMP_TRAP_LINK_DOWN: ifIndex 655, ifAdminStatus up(1), ifOperStatus down(2), ifName ae1
Dec 18 17:04:52  EX4200-A /kernel: GENCFG: op 22 (Sflow) failed; err 1 (Unknown)
Dec 18 17:04:52  EX4200-A /kernel: drv_ge_misc_handler: ifd:135  new address:cc:e1:7f:2b:82:85
Dec 18 17:04:53  EX4200-A mib2d[1304]: SNMP_TRAP_LINK_DOWN: ifIndex 708, ifAdminStatus up(1), ifOperStatus down(2), ifName ae1.0
Dec 18 17:04:53  EX4200-A mib2d[1304]: SNMP_TRAP_LINK_DOWN: ifIndex 506, ifAdminStatus down(2), ifOperStatus down(2), ifName ge-0/0/2

With the server passing traffic over member 1, I could upgrade member 0 which was the same as before request system software add /tmp/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz reboot validate reboot

Same as member 1, it came back up after its reboot but the switch took an age to find the master and just as long to commit the activation of interface ge-0/0/2! Extreme Patience’s Needed!

Confirmation of the link is up and I have lldp neighbor

{master:0}
[email protected]> show lldp neighbors 
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/2.0         ae1.0               00:0c:29:4f:26:bb   eth1               km-vm1
{master:0}
[email protected]> show interfaces ge-0/0/2   
Physical interface: ge-0/0/2, Enabled, Physical link is Up

Having both members are on the same code as expected:

{master:0}
[email protected]> show version 
fpc0:
--------------------------------------------------------------------------
Hostname: EX4200-A
Model: ex4200-48t
JUNOS EX  Software Suite [13.2X51-D35.3]
JUNOS FIPS mode utilities [13.2X51-D35.3]
JUNOS Online Documentation [13.2X51-D35.3]
JUNOS EX 4200 Software Suite [13.2X51-D35.3]
JUNOS Web Management [13.2X51-D35.3]

{master:1}
[email protected]> show version 
fpc1:
--------------------------------------------------------------------------
Hostname: EX4200-A
Model: ex4200-48t
JUNOS EX  Software Suite [13.2X51-D35.3]
JUNOS FIPS mode utilities [13.2X51-D35.3]
JUNOS Online Documentation [13.2X51-D35.3]
JUNOS EX 4200 Software Suite [13.2X51-D35.3]
JUNOS Web Management [13.2X51-D35.3]

To get them joined together into the virtual chassis I enabled the VCP ports on member 0 and hoped this would bring them back together with no issues (He says!!!)

{master:0}
[email protected]> request virtual-chassis vc-port set interface vcp-0    
{master:0}
[email protected]> request virtual-chassis vc-port set interface vcp-1

To finish off, I ran the command request system snapshot slice alternate all-members to make sure the backup partition image was consistent with the primary

And finally everything is complete! I confirmed the virtual-chassis, firmware version, lldp neighbors and Upgraded the Backup Partition! Never forget to do this!

show virtual-chassisshow versionshow lldp neighborsrequest system snapshot slice alternate
[email protected]> show virtual-chassis    

Preprovisioned Virtual Chassis
Virtual Chassis ID: e8a9.d27b.0f05
Virtual Chassis Mode: Enabled
                                                Mstr           Mixed Route Neighbor List
Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214340104 ex4200-48t     129   Master*      N  VC   1  vcp-0      
                                                                           1  vcp-1      
1 (FPC 1)  Prsnt    BP0215090120 ex4200-48t     129   Backup       N  VC   0  vcp-0      
                                                                           0  vcp-1

[email protected]> show version              
fpc0:
--------------------------------------------------------------------------
Hostname: EX4200-A
Model: ex4200-48t
JUNOS EX  Software Suite [13.2X51-D35.3]
JUNOS FIPS mode utilities [13.2X51-D35.3]
JUNOS Online Documentation [13.2X51-D35.3]
JUNOS EX 4200 Software Suite [13.2X51-D35.3]
JUNOS Web Management [13.2X51-D35.3]

fpc1:
--------------------------------------------------------------------------
Hostname: EX4200-A
Model: ex4200-48t
JUNOS EX  Software Suite [13.2X51-D35.3]
JUNOS FIPS mode utilities [13.2X51-D35.3]
JUNOS Online Documentation [13.2X51-D35.3]
JUNOS EX 4200 Software Suite [13.2X51-D35.3]
JUNOS Web Management [13.2X51-D35.3]

[email protected]> show lldp neighbors 
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/2.0         ae1.0               00:0c:29:4f:26:bb   eth1               km-vm1              
ge-1/0/2.0         ae1.0               00:0c:29:4f:26:bb   eth2               km-vm1
[email protected]> request system snapshot slice alternate  
fpc0:
--------------------------------------------------------------------------
Formatting alternate root (/dev/da0s1a)...
Copying '/dev/da0s2a' to '/dev/da0s1a' .. (this may take a few minutes)
The following filesystems were archived: /

fpc1:
--------------------------------------------------------------------------
Formatting alternate root (/dev/da0s2a)...
Copying '/dev/da0s1a' to '/dev/da0s2a' .. (this may take a few minutes)
The following filesystems were archived: /

From the running pings:

--- 192.31.1.1 ping statistics ---
9365 packets transmitted, 9234 received, +42 errors, 1% packet loss, time 9377278ms
rtt min/avg/max/mdev = 0.771/1.162/11.807/0.370 ms, pipe 3
[email protected]:~$

There was 1% packet over the whole time of the test (156 minutes), working out as a 93.77 second outage which isn't too bad. Considering this was the first time I tried this method I’ll be going over it again because it took far too long, but overall this method works!

I also messed about with the different types of bonding methods available:

With the round-robin or bond-type 0, the switch was configured as two access ports and I saw high packet loss during the testing.

--- 192.31.1.1 ping statistics ---
6106 packets transmitted, 3125 received, 48% packet loss, time 6128448ms
rtt min/avg/max/mdev = 0.814/1.484/902.641/16.131 ms

This was due to the nature of the round-robin bonding method.

Round-robin policy to transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.

With the active-backup or bond-type 1, the switch was configured as two access ports and I saw no packet loss during the testing. A sight difference when using active-backup (as expected to be honest) when you check the lldp neighbors is that you’ll only see one interface up at a time.

This is due to the nature of the bond-type

Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch.
Ping OutputLLDP Difference
--- 192.31.1.1 ping statistics ---
2905 packets transmitted, 2892 received, 0% packet loss, time 2908023ms
rtt min/avg/max/mdev = 0.846/1.214/20.269/0.758 ms
[email protected]> show lldp neighbors 
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/2.0         -                   00:0c:29:4f:26:bb   eth1               km-vm1              
vme.0              -                   00:19:06:cd:8f:80   GigabitEthernet1/0/36 oob-sw0-10.lab      
xe-0/1/0.0         ae0.0               78:fe:3d:46:2a:c0   xe-0/0/2.0         EX4500

Having got a method that worked, the tabs below show some of the methods I tried and failed on. Looking back on some of the methods, the two methods I used were never going to work however, this is why you have a lab and it’s always good to see things for yourself to see if you can troubleshoot your way out! With all that being said I’ve actually picked up a few things I didn’t know, so this was a good exercise!

Tester Method #1
Upgrade the member 1 then see if you failover routing-engine from member 0 to member 1. The issue that could arise is that the routing-engine will not failover as the 2 switches will be on different version of code and the VC will not join back up as backup routing-engine.

I started the upgrade on member 1 running:

request system software add /tmp/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz member 1 reboot

Once the upgrade had completed, I checked the virtual chassis and as I thought member 1 didn’t join back into the VC as backup routing-engine

[email protected]> show virtual-chassis    

Preprovisioned Virtual Chassis
Virtual Chassis ID: e8a9.d27b.0f05
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214340104 ex4200-48t 129  Master*      N  1  vcp-0      
                                                                 1  vcp-1      
1 (FPC 1)  Inactive BP0215090120 ex4200-48t 129  Linecard     N  0  vcp-0      
                                                                 0  vcp-1

And the logs show the mismatch of code and timeout of member 1 rejoining the VC. This method is out, but then it was expected to be honest

Tester Method #2
Upgrade using NSSU method and when it gets stuck see if you can abort and failover. This method sounds like it’s a bit of a hack and won’t work however, we’re in the lab so it doesn’t matter, and if it works then yaaay!

I ran the command to kick of the NSSU

request system software nonstop-upgrade /tmp/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz
Chassis ISSU Check Done
[Dec 18 08:41:50]:ISSU: Validating Image
[Dec 18 08:42:20]:ISSU: Preparing Backup RE
[Dec 18 08:42:21]: Installing image on other FPC's along with the backup

[Dec 18 08:42:21]: Checking pending install on fpc1
[Dec 18 08:43:21]: Pushing bundle to fpc1
NOTICE: Validating configuration against mchassis-install.tgz.
NOTICE: Use the 'no-validate' option to skip this if desired.
WARNING: A reboot is required to install the software
WARNING:     Use the 'request system reboot' command immediately
[Dec 18 08:44:23]: Completed install on fpc1
[Dec 18 08:44:34]: Backup upgrade done
[Dec 18 08:44:34]: Rebooting Backup RE

Rebooting fpc1
[Dec 18 08:44:34]:ISSU: Backup RE Prepare Done
[Dec 18 08:44:34]: Waiting for Backup RE reboot

Having a console into member 1, I can see that member 1 has joined the VC cluster

{backup:1}
[email protected]> show virtual-chassis 

Preprovisioned Virtual Chassis
Virtual Chassis ID: e8a9.d27b.0f05
Virtual Chassis Mode: Enabled
                                                Mstr           Mixed Route Neighbor List
Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214340104 ex4200-48t     129   Master       N  VC   1  vcp-0      
                                                                           1  vcp-1      
1 (FPC 1)  Prsnt    BP0215090120 ex4200-48t     129   Backup*      N  VC   0  vcp-0      
                                                                           0  vcp-1 

I aborted the NSSU on the member 0 console screen and tried to failover the routing-engine. However, when I aborted the NSSU it took over an hour to get the operational prompt and once I got to the operational prompt, the VC cluster had detached and was back to Master and Linecard. This makes sense now as the switches are out of the NSSU process, and it will just go back seeing 2 mismatched Junos versions

{master:0}
[email protected]> show virtual-chassis 

Preprovisioned Virtual Chassis
Virtual Chassis ID: e8a9.d27b.0f05
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214340104 ex4200-48t 129  Master*      N  1  vcp-0      
                                                                 1  vcp-1      
1 (FPC 1)  Inactive BP0215090120 ex4200-48t 129  Linecard     N  0  vcp-0      
                                                                 0  vcp-1

This method is out (but then this was expected)

Side Notes
Note 1Note 1.5Rollback Firmware Upgrade
If you want to change it you can release the routing-engine mastership by running the command request chassis routing-engine master release, but by using this command you will get an outage as the PFE will switchover and as you can not have non-stop routing and graceful switchover, both configured under routing-options stanza an outage will happen.
Additionally, whatever config changes you made as the members are separated will not be kept if you switchover the PFE. I saw that when I enabled interface ge-1/0/2 on member 1, but when the PFE was switchover it become inactive.
[email protected]> request system software rollback member 1 reboot 
fpc1:
--------------------------------------------------------------------------
Junos version '12.3R5.7' will become active at next reboot
Rebooting ...
shutdown: [pid 1280]
Shutdown NOW!

Then once member 1 has rebooted, I checked to make sure it is present into the virtual chassis

[email protected]> show virtual-chassis                                

Preprovisioned Virtual Chassis
Virtual Chassis ID: e8a9.d27b.0f05
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214340104 ex4200-48t 129  Master*      N  1  vcp-0      
                                                                 1  vcp-1      
1 (FPC 1)  Prsnt    BP0215090120 ex4200-48t 129  Backup       N  0  vcp-0      
                                                                 0  vcp-1
Share this:
Share

Juniper EX Virtual Chassis Part 2

Reading Time: 2 minutes

I’ve already written a post on how to create a Virtual Chassis by using the 1/10GB uplink modules. If you have a switch in production and want to add another switch for additional ports or redundancy, you can easily create a virtual chassis. This time I’ll be using the dedicated VC ports and cables and adding a new switch to a production switch.

I’ll be using the preprovisioned method, and before I do any virtual chassis configuration I’ll need to add some features to the master member to minimize failover times:

set system commit synchronize
set chassis redundancy graceful-switchover
set routing-options nonstop-routing
set ethernet-switching-options nonstop-bridging

Having added these features, we can now configure preprovisioned virtual chassis onto the master switch, which will become member 0. Because this is only a 2 member VC, I’ve added the no-split-detection command as recommended by Juniper, and to help with the failover times fast-failover on all ports ge/xe that have been enabled.

set virtual-chassis preprovisioned
set virtual-chassis no-split-detection
set virtual-chassis member 0 role routing-engine
set virtual-chassis member 0 serial-number BP0214340104
set virtual-chassis member 1 role routing-engine
set virtual-chassis member 1 serial-number BP0215090120
set virtual-chassis fast-failover ge
set virtual-chassis fast-failover xe

For now, that’s everything on the master member. On the new switch (member 1), you need to clear all config from the switch and set the root password to allow you to commit your changes:

root> edit 
Entering configuration mode
 
{master:0}[edit]
root# delete 
This will delete the entire configuration
Delete everything under this level? [yes,no] (no) yes 
{master:0}[edit]
root# set system root-authentication plain-text-password    
New password:
Retype new password:
root# commit 
configuration check succeeds
commit complete

You need to ensure there are no past virtual chassis configurations, and you can do this by entering the shell cli of the switch and removing anything in the vchassis folder:

root> start shell 
[email protected]:RE:0% rm -rf /config/vchassis/*
[email protected]:RE:0% cd /config/vchassis/
[email protected]:RE:0% ls -la
total 8
drwxr-xr-x  2 root  wheel  512 Sep 13 07:26 .
drwxr-xr-x  5 root  wheel  512 Sep 13 06:57 ..
[email protected]:RE:0% exit
exit

Now you will need to power off the backup member for at least a minute, to ensure that the other switch is elected as master.

After the minute, patch the VC-cable into the dedicated VCP-Ports at the back of the chassis and power on the backup switch. Once member 1 has booted you will be able to verify the new member by running: show virtual-chassis status

[email protected]> show virtual-chassis status     
 
Preprovisioned Virtual Chassis
Virtual Chassis ID: f1a1.ca8e.bbba
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214340104 ex4200-48t 129  Master*      N  1  vcp-0      
                                                                 1  vcp-1      
1 (FPC 1)  Prsnt    BP0215090120 ex4200-48t 129  Backup       N  0  vcp-0      
                                                                 0  vcp-1  

And you can verify the health of the VCP ports by running: show virtual-chassis vc-port

[email protected]> show virtual-chassis vc-port    
fpc0:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           1    Up           32000        1   vcp-0  
vcp-1       Dedicated           2    Up           32000        1   vcp-1  
 
fpc1:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           1    Up           32000        0   vcp-0  
vcp-1       Dedicated           2    Up           32000        0   vcp-1  
Share this:
Share

Configuring Virtual Chassis on Juniper EX Series

Reading Time: 3 minutes

You have two ways of configuring a Virtual Chassis (VC) on the Juniper EX Series. One method would be use VCP cables that will connect the two EX switches together (which will be written about shortly). The issue with VCP cables is that they are limited to 5 metres in length, which can be an issue if you wanted to create a VC and your switches were more than 5m apart or some other reason why it would be difficult to use VCP cables.

With that in mind, you are able to define port(s) (either copper or if you’re using an uplink module with an SFP fibre) that can connect the switches together and have them as the VC link. These ports are known as VCEP ports. Although you have the advantage of distance by using VCEP ports, the disadvantage is that: 1. You lose out on front port capacity and 2. The bandwidth between the switches is much lower. Using a dedicated VC cable gives you 64Gbps throughput per cable (max of 2 VC cables per switch) whereas using VCEP, if you’re using single copper or a 1Gbps fibre SFP+ 2Gbps (full duplex throughput) or 20Gbps (full duplex throughput) if you’re using a single 10Gbps link from the uplink module.

This page will detail how to configure VCEP Virtual Chassis.

Note: I was using 2 x 1Gbps interfaces from a fibre uplink module.

If you are using uplink modules you need to define these ports before setting the Virtual Chassis. You need to define whether you will be using the module as a 10G uplink or as a 1G uplink.

1. Show chassis hardware (this shows that the module is seen by the switch, however, it sees it as a 10G module. Where we are using 1G SPF the switch doesn’t see the SPF modules as shown under PIC 1)

Before the module has been defined:

root> show chassis hardware 
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                BP0214030110      EX4200-48T
Routing Engine 0 REV 23   750-033063   BP0214030110      EX4200-48T, 8 POE
FPC 0            REV 23   750-033063   BP0214030110      EX4200-48T, 8 POE
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           48x 10/100/1000 Base-T
  PIC 1          REV 07   711-026017   CH0214081797      2x 10GE SFP+
    Xcvr 0       REV 02   NON-SFP+     PQN4G6N           SFP-SX
    Xcvr 2       REV 02   NON-SFP+     PQN4S6U           SFP-SX
Power Supply 0   REV 05   740-020957   AT0513484229      PS 320W AC
Fan Tray

We need to define the module as 1G module.

2. Set chassis fpc 0 pic 1 sfpplus pic-mode 1g / set chassis fpc 1 pic 1 sfpplus pic-mode 1g

Now that the module has been defined we can check that the switch is now seeing the additional interfaces and the the module is now 4x 1G SFP+ ports compared to the previous 2x 10G SPF+

3. Show chassis hardware

After the module has been defined:

root> show chassis hardware 
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                BP0214030110      EX4200-48T
Routing Engine 0 REV 23   750-033063   BP0214030110      EX4200-48T, 8 POE
FPC 0            REV 23   750-033063   BP0214030110      EX4200-48T, 8 POE
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           48x 10/100/1000 Base-T
  PIC 1          REV 07   711-026017   CH0214081797      4x GE SFP+
    Xcvr 0       REV 02   740-011613   PQN4G6N           SFP-SX
    Xcvr 2       REV 02   740-011613   PQN4S6U           SFP-SX
Power Supply 0   REV 05   740-020957   AT0513484229      PS 320W AC
Fan Tray

4. Now the module has been defined we can set the VC-port on both switches. We make this request in Operational mode.

request virtual-chassis vc-port set pic-slot 1 port 2

5. We set VC as recommended by Juniper as having 2 Routing-Engines (RE) before setting a Linecard. The members are defined by the serial number of each device within the VC. So during the configure if you use show chassis hardware, you will be able to get the serial-number. Now we can set the virtual chassis:

set virtual-chassis preprovisioned
set virtual-chassis member 0 role routing-engine
set virtual-chassis member 0 serial-number xxxxxxxxxxx
set virtual-chassis member 1 role routing-engine
set virtual-chassis member 1 serial-number xxxxxxxxxxx

After this has been set and committed on both switches the Backup Member (member 1) will reset back to the login prompt screen (I was consoled into both and I assumed the worse, but the member 0 was acting as expected after a commit and-quit). To check everything was working I checked the virtual-chassis and vc-port:

root> show virtual-chassis vc-port 
fpc0:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           1    Down         32000
vcp-1       Dedicated           2    Down         32000
1/2         Configured         -1    Up           1000         1   vcp-255/1/2

fpc1:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           1    Down         32000
vcp-1       Dedicated           2    Down         32000
1/2         Configured         -1    Up           1000         0   vcp-255/1/2


root> show virtual-chassis 
Preprovisioned Virtual Chassis
Virtual Chassis ID: f1ec.bc12.8f6b
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214030110 ex4200-48t 129  Master*      N  1  vcp-255/1/2
1 (FPC 1)  Prsnt    BP0211489184 ex4200-48t 129  Backup       N  0  vcp-255/1/2
Share this:
Share