Joel Knight

Author Archives: Joel Knight

Five Functional Facts about TACACS+ in ISE 2.0

The oft-requested and long awaited arrival of TACACS+ support in Cisco’s Identity Services Engine (ISE) is finally here starting in version 2.0. I’ve been able to play with this feature in the lab and wanted to blog about it so that existing ISE and ACS (Cisco’s Access Control Server, the long-time defacto TACACS+ server) users know what to expect.

Below are five facts about how TACACS+ works in ISE 2.0.

Continue reading

Speaking Notes: The Data Center Network Evolution

I will be presenting at the Cisco Connect Canada tour in Edmonton and Calgary on November 3rd and 5th, respectively. My presentation is about that three letter acronym that everyone loves to hate: SDN :-)

I will talk about SDN in general terms and describe what it really means; what we’re really doing in the network when we say that it’s “software defined”. No unicorns or fairy tales here, just engineering.

Next I’ll talk about three areas where Cisco is introducing programmability into its data center solutions:

  • Application Centric Infrastructure
  • Virtual Topology System
  • Open NX-OS

Below are the notes I made for myself while researching these topics and preparing for the presentation. At the bottom of this post is a Q&A section with some frequently asked questions.

Continue reading

How a Cisco SE Navigates Cisco.com

At the time that I’m writing this I’ve been working at Cisco for just over 3 years as a Systems Engineer. Prior to that I worked for multiple Cisco customers and was heavily involved in Cisco technologies. I know what a monster cisco.com is and how hard it can be to find what you’re looking for.

Since starting at Cisco, the amount of time I’ve spent on cisco.com has shot up dramatically. Add to that studying for my CCIE and it goes up even more. In fact, cisco.com is probably the number 1 or 2 site I visit on a daily basis (in close competition with Google/searching).

After spending all this time on the site and given how vast the site is and how hard it can be to find that specific piece of information you’re looking for, I’m writing this post as an aid to help other techies, like myself, use the site more effectively.

Layout of this Post

This post is structured to follow (part of) Cisco’s network design lifecycle as a way to help you parse this post later on when you need a quick reference. The sections are:

When a Port Channel Member Link Goes Down

Mohamed Anwar asked the following question on my post “4 Types of Port Channels and When They’re Used“.

“I need a clarification, where if a member link fails, what will happen to the traffic already sent over that link ? Is there any mechanism to notify the upper layer about the loss and ask it to resend ? How this link failure will be handled for data traffic and control traffic ?”
–Mohamed Anwar

I think his questions are really important because he hits on two really key aspects of a failure event: what happens in the data plane and what happens in the control plane.

A network designer needs to bear both of these aspects in mind as part of their design. Overlooking either aspect will almost always open the network up to additional risk.

I think it’s well understood that port channels add resiliency in the data plane (I cover some of that in the previous article). What may not be well understood is that port channels also contribute to a stable control plane! I’ll talk about that below. I’ll also address Mohamed’s question about what happens to traffic on the failed link.

Control Plane

The control Continue reading

The Correct Mask for a PE’s Loopback0

As I’ve written about previously (The Importance of BGP NEXT_HOP in L3VPNs), the BGP NEXT_HOP attribute is key to ensuring end to end connectivity in an MPLS L3VPN. In the other article, I examine the different forwarding behavior of the network based on which of the egress PE’s IP addresses is used as the NEXT_HOP. In this article I’ll look at the subnet mask that’s associated with the NEXT_HOP and the differences in forwarding behavior when the mask is configured to different values.

There is a lot of (mis-)information on the web stating that the PE’s loopback address — which, as I explain in the previous article, should always be used as the NEXT_HOP — must have a /32 mask. This is not exactly true. I think this is an example of some information that has been passed around incorrectly, and without proper context, and is now taken as a rule. I’ll explain more about this further on in the article.

Example Network

Here’s the example network:

MPLS_PE_Loopback_MaskNote that R2 and R7 are the PEs and they each have a /24 mask on their loopback0 interfaces. The PEs are peering via their loopbacks. OSPF is running between R2, Continue reading

Packets of Interest (2015-07-24)

I’ve been doing a lot of reading and video watching on securing industrial control and automation systems (ICAS) (sometimes referred to as SCADA systems) so this POI has a few links related to that and ends with a link to an editorial piece about privacy and why privacy matters to us all.

SCADA and ICS for Security Experts: How to avoid Cyberdouchery (Blackhat 2010)

This is a funny but also educational and truthful presentation by James Arlen that every IT person needs to watch if they intent to work with and gain any credibility with their counterparts in Operations Technology (OT).

Digital Bond Quickdraw SCADA IDS Signatures

https://www.digitalbond.com/tools/quickdraw/

https://github.com/digitalbond/quickdraw

Quickdraw is a set of IDS/IPS signatures for Snort (and other IDS/IPS software that understands the Snort rule language) that deals specifically with ICAS protocols such as DNP3, Modbus/TCP, and EtherNet/IP. The rules appear to be generic in nature and not focused on any particular ICAS vendor equipment.

Digital Bond also wrote Snort preprocessors for DNP3, EtherNet/IP, and Modbus/TCP which some of the rules depend on. I tried browsing through Digital Bond’s diffs to Snort 2.8.5.3 but they are very hard to read because the Continue reading

The Importance of BGP NEXT_HOP in L3VPNs

In an MPLS network with L3VPNs, it’s very easy for the NEXT_HOP attribute of a VPN route to look absolutely correct but be very wrong at the same time. In a vanilla IP network, the NEXT_HOP can point to any IP address that gets the packets moving in the right direction towards the ultimate destination. In an MPLS network, the NEXT_HOP must get the packets moving in the right direction but it must also point to the exact right address in order for traffic to successfully reach the destination.

The reason it has to be exact is because IOS only assigns MPLS labels to the next hop address and not to each individual VPN route. So when an ingress PE needs to forward a packet from a CE across the MPLS network, the PE finds the label associated with the NEXT_HOP address and uses that as the outer label to get the packet to the egress PE.

Since each NEXT_HOP has a different label, that means each NEXT_HOP is reachable through a different Label Switched Path (LSP). Different LSPs can, and likely will, forward traffic differently through the network.

An MPLS label identifies a Forwarding Equivalence Class (FEC). A FEC is Continue reading

Packets of Interest (2015-06-19)

It’s been a while since I’ve done a POI so here we go.

The Mystery of Duqu 2.0: a sophisticated cyberespionage actor returns

https://securelist.com/blog/research/70504/the-mystery-of-duqu-2-0-a-sophisticated-cyberespionage-actor-returns/

Kaspersky Lab found this new variant of the Duqu malware in their own network. They wrote a paper based on their analysis of this new malware. It fascinates me how sophisticated these software packages are and how much effort the threat actors put into them.

Diffie-Hellman Key Exchange

Diffie-Hellman (DH) is the world’s first public key crypto system. It’s used in everything from secure browsing, to secure shell. This video visually demonstrates how the Diffie-Hellman key exchange works. The best part is that you don’t need to know anything about crypto to follow along.

Passphrases That You Can Memorize – But That Even the NSA Can’t Guess

https://firstlook.org/theintercept/2015/03/26/passphrases-can-memorize-attackers-cant-guess/

Use this informative guide to generate secure, human-memorizable passphrases that are suitable for protecting your private PGP key, your private SSH key, and your master key for your password safe.

Encrypting Your Laptop Like You Mean It

https://firstlook.org/theintercept/2015/04/27/encrypting-laptop-like-mean/

A well written article about encrypting one’s laptop. Covers topics such as what disk encryption does and does not protect against, attacks against disk encryption, and Continue reading

BRKDCT-2333 – Data Center Network Failure Detection

Presenter: Arkadiy Shapiro, Manager Technical Marketing (Nexus 2000 – 7000) @ArkadiyShapiro

You could say I’m obsessed with BFD –Arkadiy

The focus on this session is around failure detection (not reconergence, protocol tuning, etc). This session will not go over user-driven failure detection methods (ping, traceroutes, etc).

Fast failure detection is the key to fast convergence.

Routing convergence steps:

  1. Detect
  2. Propagate (tell my neighbors)
  3. Process (routing recalc, SPF, DUAL, etc)
  4. Update (update RIB/FIB, program hardware tables)

Failure detection tools: a layered approach: Layer 1, 2, MPLS, 3, application.

Interconnect options:

  • Point to point – failure detection is really easy here; event driven; fast
  • Layer 3 with Layer 1 (DWDM) bump in the wire
  • Layer 3 with Layer 2 (ethernet) bump in the wire
  • Layer 3 with Layer 3 (firewall/router) bump in the wire

Think about this: moving to higher speeds (1G -> 10G -> 40G -> beyond) means that more data is lost as you move to higher speeds without changing the failure detection/reconvergence characteristics of the network. 1 second reconvergence time at 1G is way different than 1 second at 40G.

Be aware: ISSU may not support aggressive timers on various protocols. Another reason to be wary of timer cranking.

Continue reading

BRKSEC-2137 – Snort Implementation in Cisco Products

Presenter: Eric Kostlan, Technical Marketing Engineer, Cisco Security Technologies Group

 

Above all, Snort is a community –Eric

Snort stats

  • over 4 million downloads
  • nearly 500,000 registered users

Snort was created in 1998 (!!). Sourcefire founded in 2001.

The Snort engine

  • Packet sniffer (DAQ)
  • Packet decoder
  • Preprocessors
  • Detection engine
  • Output module

DAQ – packet acquisition library(ies?). Snort leverages this to pull packets off the wire (Snort doesn’t have its own built-in packet capture abilities). DAQ provides a form of abstraction between the Snort engine and the hardware where the bits are flowing. DAQ – Data AcQusition. DAQ modes: inline, passive or read from file.

Packet decoder – look for header anomalies, look for weird TCP flags, much more. Generator id (GID) is 116 for the packet decoder. Decodes Layer  and Layer 3 protocols with a focus on TCP/IP suite.

Preprocessors – apply to Layer 3, 4, and 7 protocols. “Protocol decoders”. Normalizes traffic. Major preprocessors: frag3 (reassembly), stream5 (reconstruct TCP streams), http_inspect (normalizes http traffic), protocol decoders (telnet, ftp, smtp, so on).

Detection engine – various performance settings (eg, how long to spend on regex). Two components: rule builder and inspection component. Rule builder: assembles the rules into Continue reading

BRKSEC-2139: Advanced Malware Protection

Presenter: Eric Howard, Techincal Marketing Engineer

Why aren’t we stopping all the malware???


The term “APT” has become the boogey man of cyber security. :-)

You don’t need to know squat about writing malware in order to launch malware

  • Malware rentals
  • Malware as a Service (swipe CC, pay bitcoin)

Why aren’t we stopping all the malware?

  1. To solve the malware problem is to follow a very involved, multi-step process. Not every step can be automated; humans are needed (analysis, triage, more). This makes the process expensive, too.
  2. There’s no silver bullet

Product does not solve the issue. Process is required, too. Ideally, good process backed by good product.

If you knew you were going to be compromised, would you do security differently? — Marty Roesch, Cheif Architect, Cisco Security, founder of Sourcefire

Do security different:

  • Plan A – Prevention: shore up the environment; dig a bigger moat, build thicker walls
  • Plan B – Retrospection: track system behaviors without regard for disposition (ie, do this for everything, not just known malware but also “known good” and “unknown”)

Plan A

  • 1-to-1 signatures: like anti-virus; also hashes; AV vendors only enable 8-10% of their rules; AMP cloud runs all sigs all the time; Continue reading

BRKCRS-3900: NBase-T and the Evolution of Ethernet

Presenters: Dave Zacks, Distinguished Engineer; Peter Zones, Principle Engineer

History has been: 10x performnce increase at 3x the cost. 40Gb broke that model –> 100Gb PHYs were very expensive; industry needed/wanted an intermediate step.

Ethernet has a really strong roadmap and will continue to evolve for a very long time. Roadmap: http://www.ethernetalliance.org/roadmap/

  • 25Gb – direct server connect (Twinax)
  • 40GBase-T (Cat 8 cable!)
  • 2.5/5G – N-BaseT
  • 400Gb
  • More

SERDES

  • Serializer/deserializer
  • Turns bits on the wire into bytes and vise-versa
  • 40Gb Ethernet based on 4x10Gb SERDES

100m is the sweet spot for copper cable lengths. Why? CSMA/CD and also electrical wiring, placement of wiring closets just make 100m the right fit.

Cisco Mgig

  • PoE/PoE+/UPoE
  • Standards compliant
  • Investment protection (existing cable plant)
  • Supports 100M but not 10M; (had to drop something as far as standards and nobody uses 10M anymore really)

802.11ac Wave 2

  • Max PHY rate: 6.8Gbps (in absolute best conditions)
  • More likely 3-ish Gb/s
  • Point: it’s more than 1Gbs

Cisco Mgig products:

  • 4500E line card
  • New 3850 models with Mgig ports
  • New compact 3560CX with 2x Mgig ports

Between 2003 and 2014, approx 70 billion meters of Cat 5e and Cat 6 cabling were sold

Continue reading

BRKSEC-2010: Emerging Threats – The State of Cyber Security

Presenter: Craig Williams (@security_craig) – Sr Technical Leader / Security Outreach Manager, Cisco TALOS

I’m from Talos. We love to stop bad guys.

 
Talos by the numbers:

  • 1.1 million incoming malware samples per day
  • 1.5 billion Sender Base reputation queries per day

Talos has a serious amount of data. For serious.

Data is key. It allows generation of real threat intel.

We basically have a bottomless pit of data

Talos vuln dev team:

  • Looking for ways to programmatically find 0-days
  • Takes this research and feeds it back into Cisco to a) make Cisco products more secure and b) generate sigs and threat intel to protect customers

With ransomware, you’re basically funding the malware underground.

Malvertizing:

  • Malicious ads which redirect user to malware and then infects them
  • Kyle & Stan campaign dynamically generated a new .exe every time it was downloaded; prevented matching on the file hash; Cisco AMP can stay on the bleeding edge of this
  • blogs.cisco.com/security/talos/kyle-and-stan

Destructive/Wiper Malware:

  • Targets your data
  • Not just file data, but also seen targetting network devices and wiping their configs
  • Cryptolocker 2.0: uses TOR for C&C; encrypted binary to avoid hash fingerprinting; anti-VM check
  • Cryptolocker 3.0: still Continue reading

BRKARC-2032 – Designing for Secure Convergence of Enterprise and PCNs

BRKARC-2032 – Designing for Secure Convergence of Enterprise and Process Control Networks

Presenter: Chuck Stickney, Cisco SE

Handful of OT folks in the room; majority IT.

Convergence Benefits

  • Simplification (common protocols)
  • Reduced Cost
  • Pervasive enablement of features and services


PCN vs Enterprise

  • PCN: peer-to-peer, publish/subscribe model; application defines communication parameters; strict time sync
  • Enterprise: three-tier architecture; session oriented; many-to-one (centralized apps)
  • PCN: short, high-volume messages; localized traffic; delay/jitter sensitive; unreliable transmission; no out of order messages, no retransissions; similar to voice/video (these are problems that IT has solved for years)
  • Enterprise: large messages; remote traffic; delay tolerant; reliable, connection oriented; retransmission, re-ordering

“Layer 2, Layer 3″ are not terms that OT folks understand. IT folks: speak a language your OT folks can understand.

PCN Characteristics

  • Proprietary protocols (Modbus, Profibus, DeviceNet)
  • Incompatibility between systems (connectors, cabling, signals) (think: Ethernet vs Token Ring)
  • Industrial Ethernet: a common data link layer using standard 802.3 components (EtherNet/IP, Modbus/TCP, Profinet)
  • Ethernet/IP: Rockwell; uses Common Industrial Protocol (CIP); implicit, real-time (UDP, mcast port 2222); explicit, non-time critical (tcp port 44818)
  • Profinet: Siemens; IO and non-realtime; IO is Layer 2 only where app layer directly interfaces with MAC layer bypassing layers 3 – 6; non-real time Continue reading

BRKARC-3004 APIC-EM Controller Workflow and Use Cases

Presenter: Markus Harbek, CCIE, CCDE

 
Who knows what SDN stands for?

  • Still Don’t kNow
  • Still Does Nothing
  • Schnitzel Dinner Night


APIC – Application Policy Infrastructure Controller

  • Data center
  • n9000s
  • Focus on application network profile. SLA, Security, QOS, load balancing
  • Application intent

UCI – User Centric Infrastructure

APIC-EM – APIC Enterprise Module

  • Catalyst, ISR, N7k, n6k, n5k, WLAN
  • Focus on user, things, network profile, QoS, security, SLA, device
  • Application intent

Eventually, APIC and APIC-EM will have a common policy model so they can share policies across DC and enterprise. They will not integrate directly but will talk to a common policy orchestrator.

APIC-EM is really focussed on brownfield deployments because the assumption is that customers already have networks up and running hat APIC-EM needs to integrate into. APIC-EM won’t cconfigure OSPF and STP today, things like that, because they’re more than likely already running.

Imperative Control

  • Baggage handlers at an airport follow sequences of simple, basic instructions

Declarative control

  • ATC tells where to take off from but not how to fly the plane
  • ATC tells the “what”
  • Pilot figures out the “how” part
  • In the network, this would be like the admin wanting segmentation between tenants, controller decides which technology Continue reading

BRKSEC-3005 – An IoT Security Model for Securing IT-OT Assets

Presenter: Jeff Schutt – Cybersecurity Solutions Architect (Jeff works in Adv Services in the IoT team)

Full Title: An IoT Security Model & Architecture for Securing Cyber-Physical and IT-OT Converged Assets

Mix of IT/OT folks in the room. 

How do we do physical security?

  • Protect the perimeter
  • Detect breaches
  • Situational awareness (<< THIS!)
  • Forensics

How do we do cybersecurity?

  • Same principles!
  • Just different tools

IT landscape

  • Systems approach
  • Requirements dominated by business data focus
  • Time horizon: driven by Moore’s law and high tech product cycles
  • Scale: 1000s
  • Security: built into protocols (IPsec, TLS)

OT landscape

  • Requirements dominated by needs of physical systems
  • Time horizon driven by capital equipment life; complete lifecycle determined and managed by engineers
  • Scale: few; 10s – 100s
  • Security: No access to outside systems; insecure protocols

With IT and OT convergence, ther’s no way people are going to lose their jobs. We all have too much to do for anyone to be redundant. Additionally, there is a well-known shortage of skilled workers in this area.

Security awareness and training: a combination of people, process, and technology.

“Airgap security” does not address “people, process and technology”. Airgap is NOT security (on its own). Airgap is not Continue reading

BRKIOT-2109 – Connecting Oil & Gas Pipelines

Presenter: Konrad Reszka, IoT Vertical Solutions Group Engineering Lead

Given a chance, how many people in this room would volunteer to be a meteorologies in San Diego?


Inflection point between 2009 and 2010 where the number of connected devices began to out number the connected people. 50 billion “things” by 2020. And this doesn’t include phones and tablets. It’s other smart devices.

Shift in dominant endpoints: from consumers (people) to devices (like sensors and such). This shift demands changes in the network to support this growth.

Cisco + Schneider Electric joint functional reference model for connected pipelines.

  • Modular approach
  • Pick the pieces you want
  • ISA99 model
  • Modern approach, such as virtualization
  • Forthcoming reference model with Cisco + Rockwell

Isolate your enterprise network from the operations network.

  • Industrial DMZ at level 3.5 (in the ISA99 model)
  • “Pull the plug” if need be and airgap the OT network from the enterprise network
  • Makes compliance/audits esier

In the erm… pipeline:

  • Connected Pipelines Cisco Validated Design
  • Schneider Electric TVDA (their version of a CVD)
  • Both docs are being co-written by Cisco and Schneider

Had to leave session halfway through due to an overlapping MtE session.

Copyright Joel Knight. All Rights Reserved.
www.packetmischief.ca

DEVNET-1001 – Coding 101

How to Call REST APIs from a REST Client and Python

Presenter: Matt (didn’t catch last name, sorry)


I was late to this session because of wonderful San Diego traffic :-/

A walk-through of using the REST API on APIC-EM.

http://learninglabs.cisco.com – sample code, docs

Postman – plugin for Chrome browser to craft, send, receive API commands over HTTP using a nice graphical interface. Helpful for building and testing queries and also viewing the raw output from the controller that you’re querying. Is there an equivalent for Firefox?

APIC-EM docs fully cover the API. Methods, variables, etc.

“Requests” library in Python – simplifies the CRUD operations in Python.

When you’re in the lab, verifying the SSL cert of your controller (in your code) might be optional. Don’t bring that into prod code. Get a proper cert and have your code validate the cert.

Other references:

  • http://developer.cisco.com
  • http://learnpythonthehardway.org/book
  • http://api.jquery.com
  • http://codeacademy.com/tracks/python
Copyright Joel Knight. All Rights Reserved.
www.packetmischief.ca

My CLUS 2015 Schedule

I’m lucky enough to be heading to Cisco Live in San Diego this year to host customers from my area. When I’m not with a customer during the day I plan on attending these sessions:

Monday

  • Coding 101: How to Call REST APIs from a REST Client and Python
  • IoT Solutions – Connecting Oil and Gas Pipelines
  • An IoT Security Model & Architecture for Securing Cyber-Physical and IT-OT Converged Assets
  • Keynote

Tuesday

  • APIC-EM: Controller Workload and Use Cases
  • Designing for the Secure Convergence of Enterprise and Process Control Networks
  • Emerging Threats – The State of Cyber Security

Wednesday

  • IWAN Customer Case Study
  • Industrial Keynote – IoE and the IT Mindset Shift – The Evolution of the IT Career
  • Ethernet Evolving – Ethernet at New Speeds, Deterministic Networking, and Power over Everything!
  • Advanced Malware Protection

Thursday

  • Snort Implementation in Cisco Products
  • Cloud Consumption in North America
  • No Lights, No Power, No Service? – Defending IoT
  • Closing Keynote

My main themes for picking sessions were industrial connectivity (due to the customer base I cover) and cyber security with a sprinkling of strategically chosen sessions to fill the gaps.

I plan on blogging my notes from each session soon after the session ends. Continue reading

MPLS “No Label” vs “Pop Label”

I like MPLS. And I don’t necessarily mean as a solution to solve a problem, but as something to configure in the lab. It’s fun to build things that do something when you’re done. Setting up OSPF or EIGRP and being able to traceroute across routers is meh. But configuring MPLS with all the associated technologies — an IGP, LDP, MP-BGP, — and then getting all of them working in unison… when you get the traceroute working, it’s rewarding.

Here’s something to keep an eye out for when you’re troubleshooting MPLS: An LFIB entry (that is, the Label Forwarding Information Base) that states “No Label” versus one that states “Pop Label”. These mean very different things and can be the difference between a working Label Switched Path (LSP) and a non-working LSP.

The Topology

Here’s the topology I’m working with:

MPLS-no-label-vs-pop-label-topology

Click to Enlarge

 

R21 and R8 are Customer Edge (CE) routers and they communicate through the MPLS network in the “BRANCHES” VRF. R4 and R7 are Provider Edge (PE) routers and R1, R5, and R6 are Provider (P) routers.

Working State

I’m going to examine traffic going from R8’s 10.1.8.8 address and destined to R21’s 10. Continue reading

1 2 3 4 5 6