Tag Archives: high availability (HA)

Juniper EX Virtual Chassis Part 2

I’ve already written a post on how to create a Virtual Chassis by using the 1/10GB uplink modules. If you have a switch in production and want to add another switch for additional ports or redundancy, you can easily create a virtual chassis. This time I’ll be using the dedicated VC ports and cables and adding a new switch to a production switch.

I’ll be using the preprovisioned method, and before I do any virtual chassis configuration I’ll need to add some features to the master member to minimize failover times:

set system commit synchronize
set chassis redundancy graceful-switchover
set routing-options nonstop-routing
set ethernet-switching-options nonstop-bridging

Having added these features, we can now configure preprovisioned virtual chassis onto the master switch, which will become member 0. Because this is only a 2 member VC, I’ve added the no-split-detection command as recommended by Juniper, and to help with the failover times fast-failover on all ports ge/xe that have been enabled.

set virtual-chassis preprovisioned
set virtual-chassis no-split-detection
set virtual-chassis member 0 role routing-engine
set virtual-chassis member 0 serial-number BP0214340104
set virtual-chassis member 1 role routing-engine
set virtual-chassis member 1 serial-number BP0215090120
set virtual-chassis fast-failover ge
set virtual-chassis fast-failover xe

For now, that’s everything on the master member. On the new switch (member 1), you need to clear all config from the switch and set the root password to allow you to commit your changes:

root> edit 
Entering configuration mode
 
{master:0}[edit]
root# delete 
This will delete the entire configuration
Delete everything under this level? [yes,no] (no) yes 
{master:0}[edit]
root# set system root-authentication plain-text-password    
New password:
Retype new password:
root# commit 
configuration check succeeds
commit complete

You need to ensure there are no past virtual chassis configurations, and you can do this by entering the shell cli of the switch and removing anything in the vchassis folder:

root> start shell 
[email protected]:RE:0% rm -rf /config/vchassis/*
[email protected]:RE:0% cd /config/vchassis/
[email protected]:RE:0% ls -la
total 8
drwxr-xr-x  2 root  wheel  512 Sep 13 07:26 .
drwxr-xr-x  5 root  wheel  512 Sep 13 06:57 ..
[email protected]:RE:0% exit
exit

Now you will need to power off the backup member for at least a minute, to ensure that the other switch is elected as master.

After the minute, patch the VC-cable into the dedicated VCP-Ports at the back of the chassis and power on the backup switch. Once member 1 has booted you will be able to verify the new member by running: show virtual-chassis status

[email protected]> show virtual-chassis status     
 
Preprovisioned Virtual Chassis
Virtual Chassis ID: f1a1.ca8e.bbba
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    BP0214340104 ex4200-48t 129  Master*      N  1  vcp-0      
                                                                 1  vcp-1      
1 (FPC 1)  Prsnt    BP0215090120 ex4200-48t 129  Backup       N  0  vcp-0      
                                                                 0  vcp-1  

And you can verify the health of the VCP ports by running: show virtual-chassis vc-port

[email protected]> show virtual-chassis vc-port    
fpc0:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           1    Up           32000        1   vcp-0  
vcp-1       Dedicated           2    Up           32000        1   vcp-1  
 
fpc1:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
vcp-0       Dedicated           1    Up           32000        0   vcp-0  
vcp-1       Dedicated           2    Up           32000        0   vcp-1  
Share this:
Share

Configuring a 802.3ad Bonded Interface Ubuntu (NIC Teaming)

Messing about in the lab configuring 802.3ad LACP bundled interfaces between switches and I wanted to see how easy (or hard) it would be to create a bonded interface on a server. I’ve got an Ubuntu 14.04LTS VM and 3 NICs available, so eth1 and eth2 were told they will become one 😀

NOTE
Please make sure you are either doing this via ILO/KVM or have a management interface I like have, as you are making network changes and you could lock yourself out of your server, if it goes horribly wrong!

Let’s get cracking!

Firstly, I configured the switch as 802.3ad LACP aggregated interface and set the interfaces to apart of the aggregated interface:

{master:0}[edit interfaces]
[email protected]# show  
ge-0/0/2 {
    description "km-vm1 1GB";
    enable;
    ether-options {
        802.3ad ae1;
    }
}
ge-0/0/3 {
    description "km-vm1 eth2 1GB";
    enable;
    ether-options {
        802.3ad ae1;
    }
}
ae1 {
    aggregated-ether-options {
        lacp {
            active;                     
            periodic fast;
        }
    }
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members v10;
            }
        }
    }
}

Server wise, check that the NICs can be configured as an 802.3ad bond, as when I’m using LACP method of bonding, you need to ensure that the NICs support ethtool.

By running ethtool {interface} , if a link is detected then you’re good to go:

[email protected]:~$ ethtool eth1
Settings for eth1:
	Supported ports: [ TP ]
	Supported link modes:   1000baseT/Full 
	                        10000baseT/Full 
	Supported pause frame use: No
	Supports auto-negotiation: No
	Advertised link modes:  Not reported
	Advertised pause frame use: No
	Advertised auto-negotiation: No
	Speed: 10000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: off
	MDI-X: Unknown
Cannot get wake-on-lan settings: Operation not permitted
	Link detected: yes

[email protected]:~$ ethtool eth2
Settings for eth2:
	Supported ports: [ TP ]
	Supported link modes:   1000baseT/Full 
	                        10000baseT/Full 
	Supported pause frame use: No
	Supports auto-negotiation: No
	Advertised link modes:  Not reported
	Advertised pause frame use: No
	Advertised auto-negotiation: No
	Speed: 10000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: off
	MDI-X: Unknown
Cannot get wake-on-lan settings: Operation not permitted
	Link detected: yes

I needed to install ifenslave package, as this package is used to attach and detach NICs to a bonding interface

sudo apt-get install ifenslave

Once that has been installed, the kernel module file needs to be edited to include bonding before creating a bonded interface:

sudo nano /etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

lp
rtc
bonding

Once that is saved, manually load the module:

sudo modprobe bonding

Next edit the interfaces into a bond sudo nano /etc/network/interfaces

auto eth1
iface eth1 inet manual
    bond-master bond0

auto eth2
iface eth2 inet manual
    bond-master bond0

auto bond0
iface bond0 inet static
    # For jumbo frames, change mtu to 9000
    mtu 1500
    address 192.31.1.2
    netmask 255.255.255.0
    network 192.31.1.0
    broadcast 192.31.1.255
    gateway 192.31.1.1
    bond-miimon 100
    bond-downdelay 200 
    bond-updelay 200 
    bond-mode 4
    bond-slaves none
Bond Configuration Details
Bond-MiimonBond-DowndelayBond-UpdelayBond-ModeBond-Slaves
Specifies the MII link monitoring frequency in milliseconds. This determines how often the link state of each slave is inspected for link failures
Specifies the time, in milliseconds, to wait before disabling a slave after a link failure has been detected.
Specifies the time, in milliseconds, to wait before enabling a slave after a link recovery has been detected.
Specifies what mode of NIC bonding configured. There’s 7 mode:

  • Mode 0 – balance-rr
  • Mode 1 – active-backup
  • Mode 2 – balance-xor
  • Mode 3 – broadcast
  • Mode 4 – 802.3ad
  • Mode 5 – balance-tlb
  • Mode 6 – balance-alb

For more in-depth details on bonding modes and Linux Ethernet Bonding visit Kernel.org white paper documentation

Defines all the interfaces that will be in the bond. My example has none because I had defined them with bond-master

Save and Exit, then you need to do network restart or reboot the server for the change to take effect.

Once the reboot/restart has completed you should be sorted. You can check this by running the commands ifconfig

[email protected]:~$ ifconfig 
bond0     Link encap:Ethernet  HWaddr 00:0c:29:4f:26:c5  
          inet addr:192.31.1.2  Bcast:192.31.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:26c5/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:150 errors:0 dropped:5 overruns:0 frame:0
          TX packets:446 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:14381 (14.3 KB)  TX bytes:53888 (53.8 KB)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:4f:26:bb  
          inet addr:10.1.0.137  Bcast:10.1.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:26bb/64 Scope:Link
          inet6 addr: 2001:41c1:4:8040:20c:29ff:fe4f:26bb/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:304 errors:0 dropped:0 overruns:0 frame:0
          TX packets:127 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:26921 (26.9 KB)  TX bytes:24900 (24.9 KB)

eth1      Link encap:Ethernet  HWaddr 00:0c:29:4f:26:c5  
          inet6 addr: fe80::20c:29ff:fe4f:26c5/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:24 errors:0 dropped:1 overruns:0 frame:0
          TX packets:216 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4155 (4.1 KB)  TX bytes:26653 (26.6 KB)

eth2      Link encap:Ethernet  HWaddr 00:0c:29:4f:26:c5  
          inet6 addr: fe80::20c:29ff:fe4f:26c5/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:126 errors:0 dropped:4 overruns:0 frame:0
          TX packets:230 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10226 (10.2 KB)  TX bytes:27235 (27.2 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:64 errors:0 dropped:0 overruns:0 frame:0
          TX packets:64 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:5696 (5.6 KB)  TX bytes:5696 (5.6 KB)

or cat /proc/net/bonding/bond0

[email protected]:~$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
	Aggregator ID: 1
	Number of ports: 2
	Actor Key: 33
	Partner Key: 2
	Partner Mac Address: cc:e1:7f:2b:82:80

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:4f:26:c5
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:4f:26:cf
Aggregator ID: 1
Slave queue ID: 0

By using cat /proc/net/bonding/bond0 you can also check if a link in the bond has failed as the Link Failure Count would increase.

And thats how you can configure 802.3ad Bonded Interface 🙂

Share this:
Share

Disabling a SRX Chassis Cluster

My final post on SRX Chassis Clustering, if you been with me from the start, it has been emotional 😀 haha

If you wanted to disable chassis cluster and have the SRX firewall back as a standalone devices, you will need to run the following command. From Operational mode:

{primary:node0}
[email protected]_SRX220_Top> set chassis cluster disable reboot

(If you remember from the first post, this was the first command I used)

You will get a message, saying the chassis cluster has been disabled and the device is going to reboot. Once the reboot has completed you will have your SRX back to standalone device!

As straightforward as that!!!

I hope that you have enjoyed my series of posts SRX Chassis Clustering process. For myself, at the time of writing, this was the first time I had ever done chassis cluster! If you have any comments, questions or feedback, drop a comment as im all ears!

Cheers 😀

For a greater insight and further in-depth understanding and knowledge on Chassis Clustering on SRX Series, I would recommend having read of the Juniper Security Device documentation

Share this:
Share

Upgrading a SRX Chassis Cluster

In my previous post, I had successfully failed over the redundancy groups on the cluster using Manual Failover and Interface Failure methods. This post will look into the methods that can be used, when upgrading a SRX Chassis Cluster.

Testing Information
i)I had the scp latest recommended version of Junos (12.1X44-D45.2) onto both Node0 and Node1. The package is located under the /var/tmp file. You can get this folder via cli. From Operation Mode start shell then cd /var/tmp
ii) I will have rolling pings from trust <--> untrust zones in separate terminal windows, so I can see when the outage starts and will be timing the length
iii) All command will run from Node0, unless stated otherwise

You have two methods of updating a SRX Cluster:

Method A (Individual Node upgrades)

Disclaimer
Using this method of chassis cluster upgrade, as a SERVICE DISRUPTION of 3-5 minutes minimum. You will need to ensure that you have considered the business impact of this method of upgrade.

This method can also be used for downgrading Junos, as well as upgrading and has no Junos version limitation. With this method you will be simply upgrading both individual nodes at the same time. As I have already uploaded the Junos image onto both nodes. I will need to run the command on BOTH Node0 and Node1 from Operational Mode

{primary:node0}
[email protected]_SRX220_Top> request system software add /var/tmp/junos-srxsme-12.1X44-D45.2-domestic.tgz
{secondary:node1}
[email protected]_SRX220_Top> request system software add /var/tmp/junos-srxsme-12.1X44-D45.2-domestic.tgz

Once they have been added, you will need to reboot both Nodes simultaneously. You can use request system reboot node all from Node0

After the reboot, you will need to update the backup image of Junos on both Nodes, to have a consistent primary and backup image.

Method B (In Service Software Upgrades)

Before I begin, with in-service updates, Juniper have two types of in-service upgrade. For the High-End Data Centre SRX models SRX1400, SRX3400, SRX5600 and SRX5800 will use In-Service Software Upgrade (ISSU) and the Small/Medium Branch SRX models SRX100, SRX110, SRX220, SRX240 and SRX650 will use In-Band Cluster Upgrade (ICU). Although the commands are near enough the same; the pre-upgrade requirement, service impacts and the minimum Junos firmware version that supporting in-service upgrades are different.

As I’m using 2x SRX220H2 model firewalls, I will be upgrading via ICU. When I get chance to upgrade a High-End SRX model, I will update the post with my findings :p

Even before you consider using the ISSU/ICU method, I am telling you (no recommendation here!!) to check the Juniper page Limitation on ISSU and ICU. The page will confirm what version of Junos is supported by ISSU/ICU and (more importantly) services that are not supported by ISSU/ICU. In essence, you will need to change if/what services you are running on your SRX cluster to see if they are supported. If they are not supported then you are told DO NOT perform an upgrade with this method.

With that out of the way and if you have checked that your cluster is fully supported (firmware and service) by ISSU/ICU you can proceed with the pre-checks 😀

Pre-Upgrade Checks ICU
Junos VersionNo-sync optionDowngrade Method?Disk Space
You will need to be running Junos version 11.2R2 minimum. This can be checked by running show version on both Nodes.
ICU is available with the no-sync options only. The no-sync option disables the flow state from syncing with the second node when it boots with the new Junos image.
You CAN NOT use ICU to downgrade Junos to version lower than 11.2R2
You will need to check the disk space available in the /var/tmp file on the SRX. From Operational Mode start shell then enter the command df -h and you will get disk spaces available.

Having confirmed all the pre-checks are good, we can proceed with the upgrade. It is important to note that during an ICU, there WILL BE A SERVICE DISRUPTION! will be approximately 30 seconds with no-sync option. During this 30 seconds traffic will be dropped and flow session will be lost. You will need to keep this in mind, if you are doing this upgrade in-hours or you need to have a good record on your flow session for any reason.

To start the upgrade, we need to run request system software in-service-upgrade /path/to/package no-sync

{primary:node0}
[email protected]_SRX220_Top> request system software in-service-upgrade /var/tmp/junos-srxsme-12.1X44-D45.2-domestic.tgz no-sync
ICU Console observations
RebootingUpgrade OrderNode0 to Node1 failover processEnd Host View Point
It is important to note that during the ICU process, you won’t need do any manual reboots, all the reboots are automated within the process

WARNING: in-service-upgrade shall reboot both the nodes
         in your cluster. Please ignore any subsequent 
         reboot request message
Once the process has started Node1 is upgraded first:

Node1 is upgraded first
ISSU: start downloading software package on secondary node
Pushing bundle to node1
{.......}
JUNOS 12.1X44-D45.2 will become active at next reboot
WARNING: A reboot is required to load this software correctly
WARNING:     Use the 'request system reboot' command
WARNING:         when software installation is complete
Saving state for rollback ...
ISSU: failover all redundancy-groups 1...n to primary node
Successfully reset all redundancy-groups priority back to configured priority.
Successfully reset all redundancy-groups priority back to configured priority.
Initiated manual failover for all redundancy-groups to node0
Redundancy-groups-0 will not failover and the primaryship remains unchanged.
ISSU: rebooting Secondary Node
Shutdown NOW!
[pid 13353]
ISSU: Waiting for secondary node node1 to reboot.
ISSU: node 1 went down
ISSU: Waiting for node 1 to come up
It takes few minutes for node0 to reboot after node1 comes back online if you have console connection on both SRXs, you will need to be patient before aborting the upgrade. If you have rolling ping going for each nodes fxp interface you will when the node0 is about to reboot as node1 pings will return. Once node1 is up and booted, Node0 will start to reboot.

ISSU: node 1 came up
ISSU: secondary node node1 booted up.
Shutdown NOW!
From hitting enter to having both firewalls upgraded it had taken 22:45min. Although the documentation said were will be an outage of 30 seconds the rolling ping between trust <--> untrust shows that there was no packet-loss and only 6 packets out of 1600 transmitted weren’t received. (Saying that, for my testing I was unable to get live flow session information.)

root> ping 172.16.0.2 routing-instance trust 
--- 172.16.0.2 ping statistics ---
1600 packets transmitted, 1594 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.720/2.640/13.673/0.652 ms
--------------------------------------------------------------------------
root> ping 192.168.0.2 routing-instance untrust
--- 192.168.0.2 ping statistics ---
1600 packets transmitted, 1594 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.838/2.535/13.669/0.681 ms
To verify that the upgrade has been successful, we can run the commands show version

{secondary:node0}
[email protected]_SRX220_Top> show version 
node0:
--------------------------------------------------------------------------
Hostname: lab_SRX220_Top
Model: srx220h2
JUNOS Software Release [12.1X44-D45.2]

node1:
--------------------------------------------------------------------------
Hostname: lab_SRX220_Top
Model: srx220h2
JUNOS Software Release [12.1X44-D45.2]

And show chassis cluster status, to see that chassis status is as expected

[email protected]_SRX220_Top> show chassis cluster status 
Cluster ID: 1 
Node                  Priority          Status    Preempt  Manual failover

Redundancy group: 0 , Failover count: 0
    node0                   100         secondary      no       no  
    node1                   1           primary        no       no  

Redundancy group: 1 , Failover count: 1
    node0                   100         primary        yes      no  
    node1                   1           secondary      yes      no 

We can see that we are running the upgraded version of Junos. As expected Redundancy Group 0 is primary on Node1 and Redundancy Group 1 is primary on Node0. As discussed in my previous post, with preempt enabled Redundancy Group 1 will automatically failover to Node0, once it is available. We will have to do a manual failover of redundancy group 0 back to Node0 from Node1 and we will need to upgrade the backup image of Junos to have a consistent primary and backup image.

If you had a case where you had to abort the ICU process you will need to run request system software abort in-service-upgrade on the primary node. It is important to note, if you do use the abort command, you will put the cluster into an inconsistent state, where the secondary node will be running a newer version of Junos to the Primary node. To recover the cluster into a consistent state you will need to do the following all on the secondary node:

Recovering from an Inconsistent State
1. You will need to abort the upgrade: request system software abort in-service-upgrade
2. Rollback to the older version of Junos, that will be on the primary node request system software rollback node {node-id}
3. Perform a reboot of the node request system reboot

**UPDATE 29/4/2015**
Lucky enough, as I was finishing up this series of posts, my colleague had finished working on the SRX1400 we have in our lab! So I was able to run testing on doing ISSU upgrade on High End SRX Series device 😀 Happy Days!!!

SRX1400 testing differences
1. The SRX1400 doesn’t have any routing protocols, I will not need to configure graceful restart.
2. I will be upgrading from 12.1X44-D40.2 to 12.1X46-D10.2
3. The topology will be the same, however the IP addressing will be different. Trust will be 192.168.13.0/24 and Untrust will be 172.31.13.0/24
Pre-Upgrade Checks ISSU
Junos VersionDowngrade Method?RoutingRedundancy GroupsRedundancy Group 0
You will need to check to see, if the version of Junos code supports ISSU. This can be checked by running show version on both Nodes. You will need to be using Junos version 9.6 and later
ISSU DOES NOT support firmware downgrade!
Juniper recommend that a graceful restart for routing protocols be enabled Before starting an ISSU
Manually failing over all redundancy groups to one active only (For my example, as I have a active/backup setup, you won’t need to change anything. However, if you have active/active setup, you will need to change you configuration changes)
Once the upgrade has been completed you will need to Manual Failover Redundancy Group 0 back to Node0 (see Failover on SRX cluster pt1)

To start the upgrade, firstly all the redundancy groups need to fail over to one active node. As I have an active/backup setup, all my redundancy groups are on node0

{primary:node0}
[email protected]_be-rtr0-h3> show chassis cluster status        
Cluster ID: 1 
Node                  Priority          Status    Preempt  Manual failover

Redundancy group: 0 , Failover count: 3
    node0                   100         primary        no       no  
    node1                   99          secondary      no       no  

Redundancy group: 1 , Failover count: 5
    node0                   100         primary        yes      no  
    node1                   99          secondary      yes      no

To be the upgrade process, we need to run request system software in-service-upgrade /path/to/package reboot

Important note
Unlike with the ICU upgrade process, you have to enter the option reboot to confirm that you want a reboot after. If you don’t use the option reboot, the command will fail. This only applies to the High End SRX devices, SRX1400, SRX3400, SRX3600, SRX5600 and SRX5800.
ISSU Console observations
Patience neededNode1 FailoverEnd Host View Point
It does take quite a while from this point before more output will come from the console on node0, so you will need to be patience.

Validation succeeded
failover all RG 1+ groups to node 0 
Initiated manual failover for all redundancy-groups to node0
Redundancy-groups-0 will not failover and the primaryship remains unchanged.
ISSU: Preparing Backup RE
Pushing bundle to node1
Once Node1 is up and you see the output below

ISSU: Backup RE Prepare Done
Waiting for node1 to reboot.
node1 booted up.
Waiting for node1 to become secondary
node1 became secondary.
Waiting for node1 to be ready for failover
ISSU: Preparing Daemons

It takes around 5-10mins before you see anymore output to say the upgrade process is still going on! Again you will need to be patient as this does take its time!

Secondary node1 ready for failover.
{.......}
Failing over all redundancy-groups to node1
ISSU: Preparing for Switchover
Initiated failover for all the redundancy groups to node1
Waiting for node1 take over all redundancy groups
From hitting enter to having both firewalls upgraded it had taken 30:18min. The rolling ping between trust <--> untrust shows that they was no packet-loss and only 2 packets out of 3639 transmitted weren’t received. (As like before, unfortunately I was unable to get live flow session information)

root> ping 172.31.13.2 routing-instance trust 
--- 172.31.13.2 ping statistics ---
1818 packets transmitted, 1817 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.769/3.080/44.226/3.536 ms
--------------------------------------------------------------------------
root> ping 192.168.13.2 routing-instance untrust 
--- 192.168.13.2 ping statistics ---
1821 packets transmitted, 1820 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.831/3.071/44.524/3.244 ms

To verify that the upgrade has been successful, we can run the commands show version

{secondary:node0}
[email protected]_be-rtr0-h3> show version 
node0:
--------------------------------------------------------------------------
Hostname: lab_be-rtr0-h3
Model: srx1400
JUNOS Software Release [12.1X46-D10.2]

node1:
--------------------------------------------------------------------------
Hostname: lab_be-rtr0-i3
Model: srx1400
JUNOS Software Release [12.1X46-D10.2]

And show chassis cluster status, to see that chassis status is as expected

{secondary:node0}
[email protected]_be-rtr0-h3> show chassis cluster status 
Cluster ID: 1 
Node                  Priority          Status    Preempt  Manual failover

Redundancy group: 0 , Failover count: 0
    node0                   100         secondary      no       no  
    node1                   99          primary        no       no  

Redundancy group: 1 , Failover count: 1
    node0                   100         primary        yes      no  
    node1                   99          secondary      yes      no 

We can see that we are running the upgraded version of Junos. As expected Redundancy Group 0 is primary on Node1 and Redundancy Group 1 is primary on Node0. As discussed in my previous post, with preempt enabled Redundancy Group 1 will automatically failover to Node0, once it is available. We will have to do a manual failover of redundancy group 0 back to Node0 from Node1 and we will need to upgrade the backup image of Junos to have a consistent primary and backup image.

Unexpected output
During the reboot and manual failover of redundancy group 0 on Node0, I had got the output below on my console terminal

Message from [email protected]_be-rtr0-h3 at Apr 29 12:26:40  ...
lab_be-rtr0-h3 node0.fpc1.pic0 PFEMAN: Shutting down , PFEMAN Resync aborted! No peer info on reconnect or master rebooted?  

Message from [email protected]_be-rtr0-h3 at Apr 29 12:26:40  ...
lab_be-rtr0-h3 node0.cpp0 RDP: Remote side closed connection: rdp.(17825794:13321).(serverRouter:chassis)

[email protected]_be-rtr0-i3> Apr 29 12:27:04 init: can not access /usr/sbin/ipmid: No such file or directory

Message from [email protected]_be-rtr0-i3 at Apr 29 12:27:05  ...
lab_be-rtr0-i3 node1.cpp0 RDP: Remote side closed connection: rdp.(34603010:33793).(serverRouter:pfe) 

Message from [email protected]_be-rtr0-i3 at Apr 29 12:27:05  ...
lab_be-rtr0-i3 node1.cpp0 RDP: Remote side closed connection: rdp.(34603010:33792).(serverRouter:chassis) 

Message from [email protected]_be-rtr0-i3 at Apr 29 12:27:17  ...
lab_be-rtr0-i3 node1.cpp0 RDP: Remote side reset connection: rdp.(34603010:33794).(primaryRouter:1008) 

Message from [email protected]_be-rtr0-i3 at Apr 29 12:27:18  ...
lab_be-rtr0-i3 node1.cpp0 RDP: Remote side reset connection: rdp.(34603010:33795).(primaryRouter:1007)

I had raised this with Juniper and they sent this article. The article confirms that the error messages are expected if you are connected via the console or fxp0 interface. “The above mentioned messages, which are generated on the console session, states that the routing-engine [control plane(RG0)] has become active on the other node….These messages are due to the following syslog user configuration: system syslog user *.

You can stop this error by deactivating system syslog user *.

Note: It is recommended by Juniper for you keep the ‘syslog user (‘any emergency’)’ configuration and ignore these informational messages, as they might show certain useful information to the user.

Phew that was a lot of work and quite a bit to take in there!! Time for a break, (a drink or 6 lol)

My next post will be the last post in the SRX Chassis Cluster Series (sad times 🙁 ). It will be nice simple one on how to disable chassis cluster!

Share this:
Share

Juniper SRX Failover Testing Part 2

Having completed a manual failover of the redundancy groups in my previous post, this test go through the process of what would have happen if we had a link fault.

Test B (Interface Failure)

In my first post creating srx cluster, I had configured Interface Monitoring. Interface monitoring can be used to trigger a failover in the event link status on an interface goes down. For this test I will be disconnecting interface ge-0/0/1, once this has been disconnected we should see that redundancy group 1 failover to Node1 from Node0.

We will check the status of the cluster and the interfaces before proceed:

Chassis Cluster StatusChassis Cluster Interfaces
{primary:node0}[email protected]_SRX220_Top# run show chassis cluster status 
Cluster ID: 1 
Node                  Priority          Status    Preempt  Manual failover

Redundancy group: 0 , Failover count: 7
    node0                   100         primary        no       no  
    node1                   1           secondary      no       no  

Redundancy group: 1 , Failover count: 33
    node0                   100         primary        yes      no  
    node1                   1           secondary      yes      no
{primary:node0}
[email protected]_SRX220_Top> show chassis cluster interfaces    
Control link status: Up

Control interfaces: 
    Index   Interface        Status
    0       fxp1             Up    

Fabric link status: Up

Fabric interfaces: 
    Name    Child-interface    Status
                               (Physical/Monitored)
    fab0    ge-0/0/5           Up   / Up  
    fab0   
    fab1    ge-3/0/5           Up   / Up  
    fab1   

Redundant-ethernet Information:     
    Name         Status      Redundancy-group
    reth0        Up          1                
    reth1        Up          1                
   
Redundant-pseudo-interface Information:
    Name         Status      Redundancy-group
    lo0          Up          0                

Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    ge-0/0/2          255       Up        1   
    ge-3/0/2          255       Up        1   
    ge-3/0/1          255       Up        1   
    ge-0/0/1          255       Up        1   

{primary:node0}

As everything is as expected, I have disconnected interface ge-0/0/1 and by running the same commands we are able to see that the link failure has been detected by running show chassis cluster interface command

{primary:node0}[edit]
[email protected]_SRX220_Top# run show chassis cluster interfaces       
Control link status: Up

Control interfaces: 
    Index   Interface        Status
    0       fxp1             Up    

Fabric link status: Up

Fabric interfaces: 
    Name    Child-interface    Status
                               (Physical/Monitored)
    fab0    ge-0/0/5           Up   / Up  
    fab0   
    fab1    ge-3/0/5           Up   / Up  
    fab1   

Redundant-ethernet Information:     
    Name         Status      Redundancy-group
    reth0        Up          1                
    reth1        Up          1                
   
Redundant-pseudo-interface Information:
    Name         Status      Redundancy-group
    lo0          Up          0                

Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    ge-0/0/2          255       Up        1   
    ge-3/0/2          255       Up        1   
    ge-3/0/1          255       Up        1   
    ge-0/0/1          255       Down      1

And running show chassis cluster status we can see that redundancy group 1 has failed over from Node0 to Node1.

{primary:node0}[edit]
[email protected]_SRX220_Top# run show chassis cluster status            
Cluster ID: 1 
Node                  Priority          Status    Preempt  Manual failover

Redundancy group: 0 , Failover count: 7
    node0                   100         primary        no       no  
    node1                   1           secondary      no       no  

Redundancy group: 1 , Failover count: 34
    node0                   0           secondary      yes      no  
    node1                   1           primary        yes      no

As I have configured preempt on redundancy group 1, once the link (ge-0/0/1) is reconnected, it will automatically fail back to Node0

Chassis Cluster InterfaceChassis Cluster Status
{primary:node0}[edit]
[email protected]_SRX220_Top# run show chassis cluster interfaces    
Control link status: Up

Control interfaces: 
    Index   Interface        Status
    0       fxp1             Up    

Fabric link status: Up

Fabric interfaces: 
    Name    Child-interface    Status
                               (Physical/Monitored)
    fab0    ge-0/0/5           Up   / Up  
    fab0   
    fab1    ge-3/0/5           Up   / Up  
    fab1   

Redundant-ethernet Information:     
    Name         Status      Redundancy-group
    reth0        Up          1                
    reth1        Up          1                
   
Redundant-pseudo-interface Information:
    Name         Status      Redundancy-group
    lo0          Up          0                

Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    ge-0/0/2          255       Up        1   
    ge-3/0/2          255       Up        1   
    ge-3/0/1          255       Up        1   
    ge-0/0/1          255       Up        1
{primary:node0}[edit]
[email protected]_SRX220_Top# run show chassis cluster status        
Cluster ID: 1 
Node                  Priority          Status    Preempt  Manual failover

Redundancy group: 0 , Failover count: 7
    node0                   100         primary        no       no  
    node1                   1           secondary      no       no  

Redundancy group: 1 , Failover count: 35
    node0                   100         primary        yes      no  
    node1                   1           secondary      yes      no

My next post in this series, will demonstrate the methods of upgrading Junos version on a SRX cluster.

Share this:
Share