Upgrading Dual Routing Engine Juniper MX Series

Reading Time: 4 minutes

In one of my previous post, I explained how you would go about upgrading a Juniper EX switch. I said whenever I got the chance to upgrade a MX Series Router, I’ll get something noted down…… *raises hands* today is the day! As I’ve said in a few posts, there has been a lot of change and now team is now getting access to the Core Juniper MX Series Routers. As part of this increased access, one of our first tasks is it upgrade JunOS from 12.3R5.7 to 14.1R6.4. With most, if not, all MX Series above the MX80, they will come with two Routing Engines (RE), and both are independent of each other. This is being the case, when upgrading a MX; you will need to upgrade each RE by individually.

This post will go over what you will need to do upgrade an MX Router, in my setup I’ll be upgrading a Juniper MX480 Router and I’ll be doing the upgrade via the console port on each Routing Engine.

To link two Routing Engines together, you would need to apply similar configuration to what I used:

set groups re0 system host-name re0-mx480
set groups re0 interfaces fxp0 unit 0 family inet address x.x.x.x/x
set groups re1 system host-name re1-mx480
set groups re1 interfaces fxp0 unit 0 family inet address x.x.x.x/x
set apply-groups re1
set apply-groups re0

set chassis redundancy graceful-switchover
set routing-options nonstop-routing
set system commit synchronize

With that all cleared up.. Let’s get cracking πŸ™‚

Pre Works

Upload the new firmware version to wherever you normally keep it them. Currently, we would normally upload the package into the /var/tmp folder on the device in question

[[email protected] ~]$ scp jinstall-14.3R6.4-domestic-signed.tgz re0-mx480:/var/tmp

After just saying how you to link the two REs together, for an upgrade, you will need to disable graceful-switchover and nonstop-routing. Skipping this step can potentially result in the control plane and forwarding plane having two different JUNOS versions, which can cause a number of potential issues!

deactivate chassis redundancy graceful-switchover
deactivate routing-options nonstop-routing

Upgrade Process

Having disabled both graceful-switchover and nonstop-routing, log onto the Backup RE, either by console or from the Master RE run the command request routing-engine login re1. Once on the Backup RE, you will need to run the command request system software validate add /var/tmp/xxx reboot.

[email protected]> request system software validate add /var/tmp/jinstall-14.1R6.4-domestic-signed.tgz reboot
NOTE
If you’re like us and save the new firmware package to the local device, when you run the software add command DO NOT set what RE has the package stored. If you do you add the package’s location once the upgrade is completed, on one of the RE, it will delete the image from the device!
Additionally you had requested a session from re0 to re1 to connect to Backup RE, once the RE reboots, you get this message and get booted off

[email protected]>                                                                                
*** FINAL System shutdown message from [email protected] ***                 

System going down IMMEDIATELY                                                  

                                                                               
rlogin: connection closed

If you have console access, you can watch the upgrade ticking along, if you don’t, you can confirm the Backup RE is up and running by using the command show chassis routing-engine, it will show the status and hardware stats for both Routing Engines.

show chassis routing-engine output
[email protected]> show chassis routing-engine    
Routing Engine status:
  Slot 0:
    Current state                  Master
    Election priority              Master (default)
    Temperature                 31 degrees C / 87 degrees F
    CPU temperature             37 degrees C / 98 degrees F
    DRAM                      3584 MB (3584 MB installed)
    Memory utilization          20 percent
    CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     4 percent
      Interrupt                  0 percent
      Idle                      96 percent
    Model                          RE-S-2000
    Serial ID                      9012021718
    Start time                     2016-03-22 13:14:37 GMT
    Uptime                         3 hours, 38 minutes, 16 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       0.01       0.01       0.00
Routing Engine status:
  Slot 1:
    Current state                  Backup
    Election priority              Backup (default)
    Temperature                 33 degrees C / 91 degrees F
    CPU temperature             38 degrees C / 100 degrees F
    DRAM                      3584 MB (4096 MB installed)
    Memory utilization          16 percent
    CPU utilization:
      User                       0 percent
      Background                 0 percent
      Kernel                     0 percent
      Interrupt                  0 percent
      Idle                     100 percent
    Model                          RE-S-2000
    Serial ID                      9012022174
    Start time                     2016-03-22 16:47:56 GMT
    Uptime                         4 minutes, 45 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       0.34       0.47       0.23

Having upgraded Backup RE, to reduce the downtime and service impact, you can failover the Master Routing Engine, so that the Backup becomes the new Master Routing Engine. This is an manual process by running the command, from the current Master RE, request chassis routing-engine master switch. This WILL cause a brief outage as the PFE is reset and the new firmware is loaded.

[email protected]> request chassis routing-engine master switch    
warning: Traffic will be interrupted while the PFE is re-initialized
Toggle mastership between routing engines ? [yes,no] (no) yes 

Resolving mastership...
Complete. The other routing engine becomes the master.

You can confirm by running show chassis routing-engine again on RE0

AFTER failing over Routing Engine
[email protected]> show chassis routing-engine 
Routing Engine status:
  Slot 0:
    Current state                  Backup
    Election priority              Master (default)
    Temperature                 32 degrees C / 89 degrees F
    CPU temperature             39 degrees C / 102 degrees F
    DRAM                      3584 MB (3584 MB installed)
    Memory utilization          16 percent
    CPU utilization:
      User                       2 percent
      Background                 0 percent
      Kernel                     1 percent
      Interrupt                  0 percent
      Idle                      97 percent
    Model                          RE-S-2000
    Serial ID                      9012021718
    Start time                     2016-03-22 13:14:37 GMT
    Uptime                         3 hours, 50 minutes, 7 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       0.21       0.07       0.02
Routing Engine status:
  Slot 1:
    Current state                  Master
    Election priority              Backup (default)
    Temperature                 33 degrees C / 91 degrees F
    CPU temperature             41 degrees C / 105 degrees F
    DRAM                      3584 MB (4096 MB installed)
    Memory utilization          22 percent
    CPU utilization:
      User                      43 percent
      Background                 0 percent
      Kernel                    28 percent
      Interrupt                  0 percent
      Idle                      29 percent
    Model                          RE-S-2000
    Serial ID                      9012022174
    Start time                     2016-03-22 16:47:56 GMT
    Uptime                         16 minutes, 42 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       4.71       1.11       0.46

Having failed over the RE now, all that needed is to repeat the same command as before request system software validate add /var/tmp/xxx reboot to install the new firmware on RE0

[email protected]> request system software add /var/tmp/jinstall-14.1R6.4-domestic-signed.tgz reboot

Once the Routing-Engine has upgraded you need to re-enable graceful-switchover and non-routing, first, you will see why in moment.

activate chassis redundancy graceful-switchover
activate routing-options nonstop-routing

After commit synchronising those changes, you will need to set RE0 back to the Master Routing Engine. This will be done by running request chassis routing-engine master switch from RE1 now.

[email protected]>request chassis routing-engine master switch 
Toggle mastership between routing engines ? [yes,no] (no) yes 

Resolving mastership...
Complete. The other routing engine becomes the master.

{backup}
[email protected]> 

Now that Graceful Switchover is enable when you run the command you wont see the same warning about traffic being disrupted, this is because Graceful Switchover preserves interface and kernel information allowing the PFE to continue forwarding packets, even though one of the REs is unavailable.

NOTE
More detail Graceful Switchover can be found on Juniper’s TechLibrary

We will need to save a backup of the currently running and active file system by issuing the command request system snapshot on both primary as well as backup REs

[email protected]> request system snapshot 
Doing the initial labeling...
Verifying compatibility of destination media partitions...
Running newfs (899MB) on hard-disk media  / partition (ad2s1a)...
Running newfs (100MB) on hard-disk media  /config partition (ad2s1e)...
Copying '/dev/ad0s1a' to '/dev/ad2s1a' .. (this may take a few minutes)
Copying '/dev/ad0s1e' to '/dev/ad2s1e' .. (this may take a few minutes)
The following filesystems were archived: / /config

Finally, confirm the code version by running show version invoke-on all-routing-engine. I’ve used the commands show version and show version invoke-on other-routing-engine just because I can put the two outputs into a table-like thing and it looks neater in a post :p

show version RE0show version RE1
{master}
[email protected]> show version          
Hostname: RE0-MX480-02
Model: mx480
Junos: 14.1R6.4
JUNOS Base OS boot [14.1R6.4]
JUNOS Base OS Software Suite [14.1R6.4]
JUNOS Packet Forwarding Engine Support (M/T/EX Common) [14.1R6.4]
JUNOS Packet Forwarding Engine Support (MX Common) [14.1R6.4]
JUNOS platform Software Suite [14.1R6.4]
JUNOS Runtime Software Suite [14.1R6.4]
JUNOS Online Documentation [14.1R6.4]
JUNOS Services AACL Container package [14.1R6.4]
JUNOS AppId Services [14.1R6.4]
JUNOS Services Application Level Gateways [14.1R6.4]
JUNOS Services Captive Portal and Content Delivery Container package [14.1R6.4]
JUNOS Border Gateway Function package [14.1R6.4]
JUNOS Services HTTP Content Management package [14.1R6.4]
JUNOS IDP Services [14.1R6.4]
JUNOS Services LL-PDF Container package [14.1R6.4]
JUNOS Services Jflow Container package [14.1R6.4]
JUNOS Services MobileNext Software package [14.1R6.4]
JUNOS Services Mobile Subscriber Service Container package [14.1R6.4]
JUNOS Services PTSP Container package [14.1R6.4]
JUNOS Services NAT [14.1R6.4]
JUNOS Services RPM [14.1R6.4]           
JUNOS Services Stateful Firewall [14.1R6.4]
JUNOS Voice Services Container package [14.1R6.4]
JUNOS Services Crypto [14.1R6.4]
JUNOS Services SSL [14.1R6.4]
JUNOS Services IPSec [14.1R6.4]
JUNOS py-base-i386 [14.1R6.4]
JUNOS Kernel Software Suite [14.1R6.4]
JUNOS Crypto Software Suite [14.1R6.4]
JUNOS Routing Software Suite [14.1R6.4]
{master}
[email protected]> show version invoke-on other-routing-engine 
re1:
--------------------------------------------------------------------------
Hostname: RE1-MX480-02
Model: mx480
Junos: 14.1R6.4
JUNOS Base OS boot [14.1R6.4]
JUNOS Base OS Software Suite [14.1R6.4]
JUNOS Packet Forwarding Engine Support (M/T/EX Common) [14.1R6.4]
JUNOS Packet Forwarding Engine Support (MX Common) [14.1R6.4]
JUNOS platform Software Suite [14.1R6.4]
JUNOS Runtime Software Suite [14.1R6.4]
JUNOS Online Documentation [14.1R6.4]
JUNOS Services AACL Container package [14.1R6.4]
JUNOS Services Application Level Gateways [14.1R6.4]
JUNOS AppId Services [14.1R6.4]
JUNOS Services Captive Portal and Content Delivery Container package [14.1R6.4]
JUNOS Border Gateway Function package [14.1R6.4]
JUNOS Services HTTP Content Management package [14.1R6.4]
JUNOS Services Jflow Container package [14.1R6.4]
JUNOS IDP Services [14.1R6.4]
JUNOS Services LL-PDF Container package [14.1R6.4]
JUNOS Services MobileNext Software package [14.1R6.4]
JUNOS Services Mobile Subscriber Service Container package [14.1R6.4]
JUNOS Services NAT [14.1R6.4]           
JUNOS Services RPM [14.1R6.4]
JUNOS Services PTSP Container package [14.1R6.4]
JUNOS Services Stateful Firewall [14.1R6.4]
JUNOS Voice Services Container package [14.1R6.4]
JUNOS Services SSL [14.1R6.4]
JUNOS Services Crypto [14.1R6.4]
JUNOS Services IPSec [14.1R6.4]
JUNOS py-base-i386 [14.1R6.4]
JUNOS Kernel Software Suite [14.1R6.4]
JUNOS Crypto Software Suite [14.1R6.4]
JUNOS Routing Software Suite [14.1R6.4]

And with that, we have an upgraded Dual Routing Engine MX Series router! πŸ˜€ Yay! I’ll most likely, now that I’ve got the access, mess about with ISSU upgrade on MX next so keep an eye for that one!

Reference

Configuring Dual Routing Engines MX Series
Procedure to Upgrade JUNOS on a Dual Routing Engine System
Understanding Graceful Switchover (GRES)

The following two tabs change content below.

Keeran Marquis

Network Engineer
Keeran Marquis is a Network Engineer. His main goal is to learn everything within the Networking field, pick up a little bit of scripting, be a poor man sysadmin and share whatever he knows! All Posts are his own views, opinions and experiences, no guarantees they will work for you but point you in the right direction πŸ™‚
Share this:
Share

6 thoughts on “Upgrading Dual Routing Engine Juniper MX Series”

  1. Mike

    Great post.
    I just have one Question.

    I did exactly everything you did here, but when I deactivated/deleted the non-stop routing graceful-switchover, I deleted the commit synchronize command as well.

    and when I was completely done, I enabled the 3 commands:
    set chassis redundancy graceful-switchover
    set routing-options nonstop-routing
    set system commit synchronize

    Now after finishing the backup RE first, I went to master and switched it using:
    # request chassis routing-engine master switch.

    When I did this I had an outage for 1 to 2 minutes…. does deleting the “system commit synchronize” has anything to do with the outage? would keeping the synchronization minimize or eliminate the outage? or the outage would happen no matter what?

    I ask because in your article you mentioned to deactivate the non-stop routing and graceful-switchover only and you did not deactivate the synchronization at all.

    1. Tomas Jensen

      Hi Mike,

      FPC/MPSs micro code is updated as well and to do that these slots will reboot.
      Newer MPC’s are running a Linux kernel onboard and this is downloaded from RE via tftp. You can in fact see this if you are on the console.

      BR
      Tomas Jensen.

  2. Mohsin Farooq

    Hi Keeran,

    i am trying to update from Model: mx480 Junos: 15.1F7.3 to Junos 16.1R4-S4.3 but full image is junos-vmhost-install-mx-x86-64-16.1R4-S4.3.tgz

    below is current running version on re0 & re1
    Hostname: re0-mx480
    Model: mx480
    Junos: 15.1F7.3
    JUNOS OS Kernel 64-bit [20170209.344539_builder_stable_10]
    JUNOS OS libs [20170209.344539_builder_stable_10]
    JUNOS OS runtime [20170209.344539_builder_stable_10]
    JUNOS OS time zone information [20170209.344539_builder_stable_10]
    JUNOS OS libs compat32 [20170209.344539_builder_stable_10]
    JUNOS OS 32-bit compatibility [20170209.344539_builder_stable_10]
    JUNOS py base [20170302.065919_builder_junos_151_f7]
    JUNOS OS crypto [20170209.344539_builder_stable_10]
    JUNOS network stack and utilities [20170302.065919_builder_junos_151_f7]
    JUNOS modules [20170302.065919_builder_junos_151_f7]
    JUNOS libs compat32 [20170302.065919_builder_junos_151_f7]
    JUNOS runtime [20170302.065919_builder_junos_151_f7]
    JUNOS platform support [20170302.065919_builder_junos_151_f7]
    JUNOS libs [20170302.065919_builder_junos_151_f7]
    JUNOS mtx Data Plane Crypto Support [20170302.065919_builder_junos_151_f7]
    JUNOS daemons [20170302.065919_builder_junos_151_f7]
    JUNOS Voice Services Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services SSL [20170302.065919_builder_junos_151_f7]
    JUNOS Services Stateful Firewall [20170302.065919_builder_junos_151_f7]
    JUNOS Services RPM [20170302.065919_builder_junos_151_f7]
    JUNOS Services PTSP Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services NAT [20170302.065919_builder_junos_151_f7]
    JUNOS Services Mobile Subscriber Service Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services MobileNext Software package [20170302.065919_builder_junos_151_f7]
    JUNOS Services LL-PDF Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services Jflow Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services IPSec [20170302.065919_builder_junos_151_f7]
    JUNOS IDP Services [20170302.065919_builder_junos_151_f7]
    JUNOS Services HTTP Content Management package [20170302.065919_builder_junos_151_f7]
    JUNOS Services Crypto [20170302.065919_builder_junos_151_f7]
    JUNOS Services Captive Portal and Content Delivery Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services COS [20170302.065919_builder_junos_151_f7]
    JUNOS Border Gateway Function package [20170302.065919_builder_junos_151_f7]
    JUNOS AppId Services [20170302.065919_builder_junos_151_f7]
    JUNOS Services Application Level Gateways [20170302.065919_builder_junos_151_f7]
    JUNOS Services AACL Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Packet Forwarding Engine Support (wrlinux) [20170302.065919_builder_junos_151_f7]
    JUNOS Packet Forwarding Engine Support (MX/EX92XX Common) [20170302.065919_builder_junos_151_f7]
    JUNOS Packet Forwarding Engine Support (M/T Common) [20170302.065919_builder_junos_151_f7]
    JUNOS Online Documentation [20170302.065919_builder_junos_151_f7]
    JUNOS FIPS mode utilities [20170302.065919_builder_junos_151_f7]

    re1:
    ————————————————————————–
    Hostname: re1-mx480
    Model: mx480
    Junos: 15.1F7.3
    JUNOS OS Kernel 64-bit [20170209.344539_builder_stable_10]
    JUNOS OS libs [20170209.344539_builder_stable_10]
    JUNOS OS runtime [20170209.344539_builder_stable_10]
    JUNOS OS time zone information [20170209.344539_builder_stable_10]
    JUNOS OS libs compat32 [20170209.344539_builder_stable_10]
    JUNOS OS 32-bit compatibility [20170209.344539_builder_stable_10]
    JUNOS py base [20170302.065919_builder_junos_151_f7]
    JUNOS OS crypto [20170209.344539_builder_stable_10]
    JUNOS network stack and utilities [20170302.065919_builder_junos_151_f7]
    JUNOS modules [20170302.065919_builder_junos_151_f7]
    JUNOS libs compat32 [20170302.065919_builder_junos_151_f7]
    JUNOS runtime [20170302.065919_builder_junos_151_f7]
    JUNOS platform support [20170302.065919_builder_junos_151_f7]
    JUNOS libs [20170302.065919_builder_junos_151_f7]
    JUNOS mtx Data Plane Crypto Support [20170302.065919_builder_junos_151_f7]
    JUNOS daemons [20170302.065919_builder_junos_151_f7]
    JUNOS Voice Services Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services SSL [20170302.065919_builder_junos_151_f7]
    JUNOS Services Stateful Firewall [20170302.065919_builder_junos_151_f7]
    JUNOS Services RPM [20170302.065919_builder_junos_151_f7]
    JUNOS Services PTSP Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services NAT [20170302.065919_builder_junos_151_f7]
    JUNOS Services Mobile Subscriber Service Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services MobileNext Software package [20170302.065919_builder_junos_151_f7]
    JUNOS Services LL-PDF Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services Jflow Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services IPSec [20170302.065919_builder_junos_151_f7]
    JUNOS IDP Services [20170302.065919_builder_junos_151_f7]
    JUNOS Services HTTP Content Management package [20170302.065919_builder_junos_151_f7]
    JUNOS Services Crypto [20170302.065919_builder_junos_151_f7]
    JUNOS Services Captive Portal and Content Delivery Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Services COS [20170302.065919_builder_junos_151_f7]
    JUNOS Border Gateway Function package [20170302.065919_builder_junos_151_f7]
    JUNOS AppId Services [20170302.065919_builder_junos_151_f7]
    JUNOS Services Application Level Gateways [20170302.065919_builder_junos_151_f7]
    JUNOS Services AACL Container package [20170302.065919_builder_junos_151_f7]
    JUNOS Packet Forwarding Engine Support (wrlinux) [20170302.065919_builder_junos_151_f7]
    JUNOS Packet Forwarding Engine Support (MX/EX92XX Common) [20170302.065919_builder_junos_151_f7]
    JUNOS Packet Forwarding Engine Support (M/T Common) [20170302.065919_builder_junos_151_f7]
    JUNOS Online Documentation [20170302.065919_builder_junos_151_f7]
    JUNOS FIPS mode utilities [20170302.065919_builder_junos_151_f7]

    [email protected]> request system software add validate /var/tmp/junos-vmhostall-mx-x86-64-16.1R4-S4.3.tgz reboot
    Verified junos-vmhost-install-mx-x86-64-16.1R4-S4.3 signed by PackageProductionEc_2017 method ECDSA256+SHA256
    ERROR: junos-vmhost-install-16.1R4-S4.3: sysctl: ‘none’ not ‘!none’
    NOTICE: ‘pending’ set will be activated at next reboot…
    Rebooting. Please wait …
    shutdown: [pid 9535]
    Shutdown NOW!

    *** FINAL System shutdown message from [email protected] ***

    System going down IMMEDIATELY

    Stopping cron.
    Waiting for PIDS: 3888.
    .
    Feb 13 10:44:37 jlaunchd: routing-socket-proxy (PID 3950) terminate signal 15 sent
    Feb 13 10:44:41 jlaunchd: mspd (PID 3966) exited with status=0 Normal Exit
    Waiting (max 60 seconds) for system process `vnlru’ to stop…done
    Waiting (max 60 seconds) for system process `bufdaemon’ to stop…done
    Waiting (max 60 seconds) for system process `syncer’ to stop…
    Syncing disks, vnodes remaining…0 0 0 done
    All buffers synced.
    Uptime: 15m42s
    besw0: 4 Broadcom SDK kernel threads killed

    ==============================================
    i remember i have updated to this version before using command request VMHOST software add /var/tm********** But that command dosent exist anymore. can you please help. thanks

    1. Keeran Marquis Post Author

      Hi Moshin

      With the new VHOST they have a different junos architecture compared to older code versions.

      I have done an upgrade on the new architecture (well on MXs i should say) but if its the same as the QFX then you should be able to run ISSU on each RE and shouldn’t drop packets, however you will need to check that as im not 100% on it

      Cheers

      Keeran

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.