Archive

Category Archives for "The Data Plumber"

Migration Strategy: Moving From MPLS/LDP to Segment Routing

MPLS core networks that use Label Distribution Protocol (LDP) are common in SP core networks and have served us well. So, the thought of pulling the guts out of the core is pretty daunting and invites the question why you would want to perform open-heart surgery on such critical infrastructure.   This article attempts to explain the benefits that would accrue from such a move and gives a high-level view of a migration strategy.

Why Do I Need Segment Routing?

  • Simplicity:   LDP was invented as a label distribution protocol for MPLS because nobody wanted to go back to the standards bodies to re-invent OSPF or IS-IS so that they could carry labels.  A pragmatic decision, but one that results in networks having to run two protocols.  Two protocols means twice the complexity.  
    Segment Routing simplifies things by allowing you to turn off LDP.  Instead it carries label (or Segment ID) information in extensions to the IGP.  This then leaves you with only IS-IS or OSPF to troubleshoot.  As Da Vinci reportedly said, ‘simplicity is the ultimate sophistication’. 

  • Scale:  LDP scales, but for fast convergence RSVP-TE is often used to tunnel LDP across a core. RSVP requires core routers to Continue reading

3 Stumbling Blocks for Network Engineers Adopting Ansible

Ansible, ansible, ansible seems to be all we hear these days. There are lots of resources out there all trying to convince us this is the new way get stuff done. The reality is quite different – adoption of tools like this is slow in the networking world, and making the move is hard for command-line devotees.

Here are the three main problems I encountered in my adoption of Ansible as a modern way to manage devices:

1. Most network devices don’t support Python

Ansible is derived from the systems world, and is only latterly coming to be used for managing network devices. It is often said that Ansible is agentless, but when managing a Linux host (for example) the control machine pushes the Ansible playbook to that host and executes it there. In effect, *Python* is the agent.

Most network devices don’t have on-box Python, so when using Ansible against a router or a switch you have to have ‘connection: local’ in your playbook:





---
name: Get info
hosts: all
roles:
Juniper.junos # Invokes the Junos Ansible module
connection: local # Tells it to run locally
gather_facts: no

What this does is run the playbook using the local Continue reading

Pulling Configs from Cisco NSO using curl and json2yaml.py

We’re using Cisco NSO in our lab at the moment to provision L3VPNs across multi-vendor environments as part of a demo. Just noting down a few things here for future reference:

You can use the curl (command-line URL) utility to query NSO’s API and retrieve the configuration of a device it knows about. You probably know that NSO syncs a device’s config locally, so this will be a way to retrieve the device config that NSO knows about – if the device is out of sync, of course this won’t quite be the latest:

curl -u username:password -H "Accept: application/vnd.yang.data+json" http://192.168.8.172:8080/api/config/devices/device/CPE-3/config | json2yaml

I’ll break this down:

curl -u username:password – provides the username and password of your NSO installation

-H “Accept: application/vnd.yang.data+json” – specifies an HTTP header to send in the request. In this instance, we are saying that we are expecting a JSON response. Alternatively you could specify ‘vnd.yang.data+xml’ to receive an XML response.

http://192.168.8.172:8080/api/config/devices/device/CPE-3/config – this is the API request we are making. CPE-3 is the device we are requesting the configuration of.

| json2yaml – this pipes the JSON response through a python module Continue reading

Testing notes: simulating link failure by filtering BFD packets

In some testing I am doing, I need to prove that BFD can be used with iBGP to tell the BGP protocol when there is an interruption.  This will enable BGP to be brought down much faster than if regular BGP timers are used.

To make this easier to do, I used a firewall filter on one of the two routers to filter out BFD but accept all other packets:
Single-hop BFD (i.e. across a link) uses UDP 3784, while multi-hop BFD uses 4784.  Since my BFD sessions are configured between loopbacks, it is this latter type I need to filter.

In the example below, CORE1 is a BGP client of CORE2, which is the route-reflector.

The following was configured on the routers to bring up the BFD session (I am only showing one side – you can figure out the mirror of this yourself I think):

[edit protocols bgp group CORE neighbor 10.0.0.6]
      bfd-liveness-detection {
          minimum-receive-interval 300;
          multiplier 3;
          transmit-interval {
              minimum-interval 100;
          }
      }

When the remote side was done, the session came up:


[email protected]> show bfd session
Dec 28 17:17:10
                               Detect Transmit
Address       State Interface  Time   Interval  Multiplier
10.0. Continue reading

Juniper RADIUS-delivered switching filters

I’ve been experimenting with getting RADIUS to deploy switching filters to Juniper switches recently, as part of a reference architecture demo.  The concept is called REACH2020 and combines network virtualisation with the ability to identify network users and devices so that categories of user can be put into different virtual networks.   This leaves the firewall that connects the virtual networks together as a convenient single point of control.

Anyway, back to the matter in hand.  It turns out there’s a limit to the length of switching filter you can send a Juniper EX.

In this case, I am using Aruba Clearpass 6.6 to send some RADIUS attributes to a Juniper EX4300 switch using Junos 17.4.     What I need to do is send a web portal address that a connecting client will be redirected to, and a switching filter so that they can’t go anywhere other than the portal. The switching filter is required as far as I can tell – if you just send the portal address, Junos ignores the RADIUS attribute.

An alternative way of achieving this is to configure your centralised web authentication (CWA) web portal on every edge switch, but since RADIUS Continue reading

SSH tunnelling from OSX

In certain situations, I need to connect to a remote network via a bastion host but only have SSH available to me.   To get around installing some kind of VPN gateway, the easiest thing to do is to create an SSH tunnel.

In this situation, what happens is that you set up local port-forwarding.   Local port numbers are forwarded to the remote host via an SSH tunnel to the intermediate host.

The command format to do this on OSX is:

ssh -C -L <LOCAL-PORT>:<REMOTE-IP>:<REMOTE-PORT> <USERNAME>@<BASTION-HOST>

So for example, I can forward local port 1000 on my Mac to the remote device’s port 23 using this command (sudo has to be used for local ports lower than 1024):

sudo ssh -C -L 1000:10.200.0.1:23 [email protected]

Once authenticated, I can open another terminal window and type:

telnet localhost 1000

and hey-presto, I get connected to a remote router via telnet through an encrypted SSH tunnel!   The -C parameter is for compression and isn’t actually necessary.

If you need to create a number of local port-forwards, this can be done in a config file instead.   By default SSH reads /etc/ssh/sshd_config for system-wide parameters, and also Continue reading

FTP and Telnet removed from OSX High Sierra (10.13.1)

For those of us that often have to use console servers to connect over IP to serial ports of devices, the removal of telnet from High Sierra is a bit of a pain in the bum.   Here are two things you can do:

Use the ‘nc’ command to connect in exactly the same way as you used to do at the command-line with telnet.  For example:   nc <IP address> <Port Number>

nc

SFTP is good and I use it wherever I can, but sometimes you come across some old kit that can’t support SSH or SFTP, so you just need those old tools.   An alternative is to do this:

  1. Enter Time Machine
  2. Look for a backup taken from before your upgrade.  You can
  3. If you’re not using the time-machine interface, you can find your backup here:  /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/
  4. In the appropriate backup subdirectory, look in usr/bin and you should find the telnet and ftp executable files.
  5. Copy these to your machine in /usr/local/bin

 

 


Storm control on a QFX VCF

There don’t seem to be many operational commands in Junos to tell you what’s going on with regard to Storm Control.   Here’s all I could find – let me know if you know of more:

In the lab, I configured this storm control profile:

{master:1}
[email protected]> show configuration forwarding-options
storm-control-profiles TAT-StormControl {
     all {
         bandwidth-level 1000;
     }
     action-shutdown;
}

This was then configured on ae2, which is a trunk interface towards the Ixia tester:

{master:1}
[email protected]> show configuration interfaces ae2 unit 0
 family ethernet-switching {
     interface-mode trunk;
     storm-control TAT-StormControl;
 }

 

Unfortunately there’s no ‘show forwarding-options storm-control’ type command to see what interfaces have storm control configured.   I can’t find any other command that shows this info either.

So I generate 3Mbps of traffic to ff:ff:ff:ff:ff:ff from my Ixia into the VCF and the port goes disabled immediately according to the logs:

Dec 15 12:57:23 VCF l2ald[3261]: L2ALD_ST_CTL_IN_EFFECT: ae2.0: storm control in effect on the port
Dec 15 12:57:23 VCF l2ald[3261]: L2ALD_ST_CTL_DISABLED: ae2.0: storm control disabled port
Dec 15 12:57:23 VCF l2cpd[1814]: Root bridge in routing-instance 'default' changed from 4096:b0:a8:6e:0a:bd:41 to 32768:dc:38:e1:5f:c4:02
Dec 15 12:57:23 VCF mib2d[3271]: SNMP_TRAP_LINK_DOWN: ifIndex 526, ifAdminStatus up(1), ifOperStatus down(2), ifName ae2

Continue reading

Creating a mixed-mode Virtual Chassis Fabric (VCF)

In order to mix EX switches and QFX switches in the same VCF, you need to enable mixed-mode.   This requires all members of the VCF to reboot unfortunately:

{master:1}
[email protected]> request virtual-chassis mode fabric mixed
fpc0:
--------------------------------------------------------------------------
Mode set to 'Fabric with mixed devices'. (Reboot required)

fpc2:
--------------------------------------------------------------------------
Mode set to 'Fabric with mixed devices'. (Reboot required)

fpc3:
--------------------------------------------------------------------------
Mode set to 'Fabric with mixed devices'. (Reboot required)

fpc1:
--------------------------------------------------------------------------
WARNING, Virtual Chassis Fabric mode enabled without a valid software license.
 Please contact Juniper Networks to obtain a valid Virtual Chassis Fabric License.

Mode set to 'Fabric with mixed devices'. (Reboot required)

{master:1}
[email protected]>

Once you’ve cabled up your QSFP ports between the EX4300 you are adding and the QFX spines, you need to do the following:

Enable the VCF port on the QFX spine:

request virtual-chassis vc-port set pic-slot 0 port 48

 

 

 


Updating to JDK 1.8 or 1.9 on Ubuntu

Trying to install OpenDaylight Nitrogen needs JDK 1.8 or later.

Needless to say, I’ve not go the right version on my Ubuntu 16.04 server – it reports 1.7.  Also needless to say, installing it isn’t a simple matter of adding the software through apt-get because the repository appears to be broken or empty (at the time of writing).  I was hoping to get away with doing this:

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java9-installer

Anyway, the last part failed with:

Connecting to download.oracle.com (download.oracle.com)|104.86.110.251|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-10-20 14:05:08 ERROR 404: Not Found.download failed
Oracle JDK 9 is NOT installed.
dpkg: error processing package oracle-java9-installer (--configure):
 subprocess installed post-installation script returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)

So instead I downloaded it from here:  http://www.oracle.com/technetwork/java/javase/downloads/jdk9-downloads-3848520.html

Unpacked the tarball with this:

cd /opt
tar -xvzf jdk-9.0.1_linux-x64_bin.tar.gz

Finally updated my environment variables to tell it where the JDK is:

 export JAVA_HOME=/opt/jdk-9.0.1/
  Continue reading

Shrubbery.net TACACS+ daemon and Junos

Axians Professional Services normally recommends using RADIUS authentication to our customers, but one of our customers uses TACACS.  We did some type-approval testing of new Junos release for them recently and had to set up a TACACS+ daemon in the lab to make sure authentication still worked following the upgrade.

Shrubbery.net very helpfully provide a TACACS+ implementation that you can download to a Linux host for this purpose, but the documentation is a bit light on their website, and what you find using Google is naturally somewhat Cisco-specific.  So here are some notes on getting a basic setup going with Shrubbery’s tac_plus daemon and Junos.  Maybe this will help someone else.

 

Create the Junos Config

First, you need to set up the Junos side of things.  You need to do three things:

  1. Create a tacplus-server in the config
  2. Add tacplus to the authentication-order
  3. Create a ‘local user’ account that has no password, but gives anyone belonging to it the appropriate privileges.

 

Create the TACACS server

[email protected]> show configuration system tacplus-server | display set

set system tacplus-server 192.168.3.237 port 49
set system tacplus-server 192.168.3.237 secret testing123
set system tacplus-server 192.168.3. Continue reading

Restoring Space 15.2 data to 16.1

The upgrade from Space platform 15.2 to 16.1 is one of the worst procedures I’ve seen in quite a while.   It is complicated because the underlying CentOS is being upgraded at the same time, so I guess that’s part of the reason, but still, it could be a lot slicker and better tested.

In summary, you have to apply a couple of patches, the second of which backs your 15.2 data up somewhere else – ideally over SCP to a remote server.  You then shut down your 15.2 VM, install a fresh 16.1 VM with the same IP addresses, and restore the data to it.

Sounds easy, but the 16.1 installation part can generally only be done by the customer’s VMware admin because it needs console access.  So you’ve got to rely on them following lots of instructions quite well.

Recently a customer experienced some kind of failure in the restoration part, leaving me with a fresh installed 16.1, but no data.  I SSHed on to the VM and could see the standard menu, but wasn’t offered any option to attempt another restore.   After digging around for a while, I found Continue reading

Issuing Junos Commands Using Ansible raw Module

If you want to issue something quick on a lot of devices, you don’t need to write a whole Ansible playbook to do that.  In fact you don’t really need the Junos module installed.

Ansible expects there to be Python on the managed device.  As you can read in this PacketPushers blog, it pushes the module out to the device and tries to execute it there.  Junos is going to get on-box Python at some point, but right now that’s roadmap (or SOPD if you must).

Suppose you want to find out what version of software you have on a your lab device, here’s a quick way to do that.

$ ansible 192.168.30.20 -m raw -a "show version" -u username -k
SSH password:
192.168.30.20 | SUCCESS | rc=0 >>
fpc0:
--------------------------------------------------------------------
Model: ex2200-24t-4g
JUNOS Base OS boot [12.3R12.4]
JUNOS Base OS Software Suite [12.3R12.4]
JUNOS Kernel Software Suite [12.3R12.4]
JUNOS Crypto Software Suite [12.3R12.4]
JUNOS Online Documentation [12.3R12.4]
JUNOS Enterprise Software Suite [12.3R12.4]
JUNOS Packet Forwarding Engine Enterprise Software Suite [12.3R12.4]
JUNOS Routing Software Suite [12.3R12.4]
JUNOS Web Management  Continue reading

Ansible and Junos Notes

I’m working on a project to push out configs to Juniper devices and upgrade them if necessary.  In the first instance I thought about writing it all in Python, but there’s really no need because quite a lot of legwork has already been done for you in the form of ‘PyEz’ and the Junos Ansible core modules.

Juniper give you a few examples to get you started, but don’t really explain what each of the lines in the YAML file does, but I guess they expect you to figure that out.  Below are a few notes on things I discovered – perhaps obvious to some, but they might help someone else.

‘No module named jnpr.junos’ When Running Ansible

In the examples Juniper give, they don’t tell you that the Ansible module ‘Juniper.junos’ relies on a Python module called ‘jnpr.junos’.  (It is mentioned elsewhere if you look for it.)

So if you’ve done an ‘ansible-galaxy install Juniper.junos’ you could be forgiven for thinking that you’ve downloaded the modules you need.  You then gaily go on to have a crack at the example given above, but get this error:

$ ansible-playbook juniper-test.yml

PLAY [Get info] *********************************************************

TASK  Continue reading

JBAS011469 Error in Junos Space

Just went onto a customer’s Junos Space/Security Director installation to discover that their SRX5800 was showing as ‘out of sync’.    I tried to do a ‘Resynchronize with Network’ from the Device Operations menu, but this failed with the following error:

Error while reading config from device: <devicename> javax.persistence. TransactionRequiredException: JBAS011469: Transaction is required to perform this operation (either use a transaction of extended persistence context)
jbas011469

JBAS011469 error in Space 15.2

Unfortunately (like most Space-related errors) there’s nothing about this in Juniper’s knowledgebase – the only hit I found on a Google search was a similar error, but with a different cause that had been fixed.   So I thought I’d put this here in case it helps anyone.  

The solution in the end was to update the DMI schema for the device. The SRX5800 had been upgraded from 12.1R3.5 to 12.3X48-D30.7 but the schema within Space had not.  I downloaded the appropriate schema, and now the device re-syncs just fine.


Setting up vSRX and Space in VMware Fusion

I’m going to be travelling a bit in the near future and wanted an easy, laptop-based Juniper device and Junos Space instance to mess with. I’ve recently made some headway with CLI configlets and wanted to build on what I’ve got working.

I already run VMware Fusion on the Mac in order to run Ubuntu and Windows, so I figured that was the best thing to use. What I wanted was two VMs that could see each other and be accessed from the host machine, but without them being dependent on the host’s interfaces being up. Wifi access would be chargeable, and wired impossible – without making an ethernet loopback plug to bring the Mac’s interface up artifically, that is. So that ruled out both bridged and NAT type connections. What I needed was some kind of internal network within the host – Fusion seems to call this a ‘Private to my Mac’ connection.

My version of Fusion is 7.1.2 – it doesn’t seem to be the ‘Pro’ version, but I can’t confirm this. For some reason, the help isn’t vey helpful. You do some Googling and VMware Workstation information comes up, or help pages about what appear to be Continue reading

Junos Space Log Collector – Utilities

The Juniper documentation on log collector is a bit sparse to be honest, and once it is installed, SSHing to it doesn’t seem to produce a configuration menu any more.  In order to change its config, there are some scripts, but I had to dig around for them:

[[email protected] bin]# ls
adhoc.py disableExport.sh logcollectorWatchdog.py selfhealingES.py
agentScript.sh elasticDiskAllocation.py logcollectorWatchdog.pyc selfhealingES.pyc
agentUtilityScript.sh elasticDiskRollover.sh logcolmon.py startService.sh
bashUtils.sh enableExport.sh logcolmon.pyc stopService.sh
cleanZipLogs.sh generateReponse.pl lsStatisticsupdate.sh subsequentBootupdate.sh
collectSystemLogs.sh getMountLocation.sh monitorPacketDrop.sh support-diagnostics.sh
configureMailSetup.sh getRebootDetails.pl mountNfs.sh syslogForwardToggle.sh
configureNameServer.sh getSystemInfo networkScript.sh updateEtcHosts.sh
configureNode.sh getZipLogs.pl resizeFS.sh updateIndexerip.sh
configureNtp.sh initConf.pl resourceMonitoring validateIpAddress.sh
configureTimeZone.sh loadFirewal.sh rootWrapper whiteList.sh

[[email protected] bin]#

They are in this directory:

[[email protected] bin]# pwd
/opt/jnpr/bin
[[email protected] bin]#

 

An important thing to be sure of is that log collector does not have two interfaces – it should have only eth0.   If it gets an IP address on eth1, you might find that logging does not work.  This is probably because it received a DHCP address on eth1, Continue reading

EVPN – the basics

A great EVPN overview and detailed walkthrough from a colleague of mine… Nice and clear.

PACKETS AND STUFF

So I decided to take a deep dive into eVPN, I’ll mostly be looking into VLAN-aware bundling, as per RFC 7432 – and mostly because I think this will fit more closely, with the types of deployments most of the customers are used to – good old IRB interfaces and bridge-tables!

As everyone knows, VPLS has been available for many years now and it’s pretty widely deployed, most of the customers I see have some flavour of VPLS configured on their networks and use it to good effect – so why eVPN? what’s the point in introducing a new technology if the current one appears to work fine.

The reality is that multipoint layer-2 VPNs (VPLS) were never quite as polished as layer-3 VPNs, when layer-3 VPNs were first invented they became, and still are the in many cases the “go to” technology for layer-3 connectivity across MPLS networks, and…

View original post 4,444 more words


Forgotten ‘maintenance’ password for Junos Space

The maintenance users password can be reset in Junos Space if you still have access via the CLI:

  • SSH to the Space host
  • Log in as the admin user
  • Choose the debug option on the menu (6 or 7, depending on whether this is a VM or an appliance).  Just press the number, not the number followed by return!
  • Put in the admin user’s password again.  You’re now in the Centos shell.
  • Issue the command ‘htpasswd -sb /var/www/maintenance/maintPW maintenance <newpassword>

 

Simple as that…   I was never sure why an additional maintenance password was required as well as the admin user and the GUI super user password.  Makes it a pain to keep a record of, but there you go – presumably there’s a good reason.


1 2 3 4