Jay Swan

Author Archives: Jay Swan

Using Bro DNS Logging for Network Management

I was recently asked if someone in our desktop support group could get alerted when certain laptops connected to the corporate network. We have a lot of employees who work at industrial locations and rarely connect their machines to our internal networks, so the support group likes to take those rare opportunities to do management tasks that aren't otherwise automated.

The two mechanisms that came to mind for alerting on these events are DHCP address assignment, and DNS autoregistration. While we do send DHCP logs to a central archive, the process of alerting on a frequently changing list of hostnames would be somewhat cumbersome. I have been looking for ways to use Bro for network management tasks, so this seemed like a natural use case.

We already had Bro instances monitoring DNS traffic for two of our central DNS servers. I don't fully understand how Windows DNS autoregistration works, but from looking at the Bro logs, it appears that the DHCP server sends a DNS SOA query to the central DNS servers containing the hostname of the device to which it assigns a lease.

I wanted to learn how to use the input framework in Bro 2.2, so I Continue reading

New Year Resolution: Code Cleanup

I enjoyed Ethan Banks' post on New Year's Thoughts: Start with Documentation, so I thought I'd write about what I'm doing this week: code cleanup. Over the last couple of years I've written a decent amount of code to automate mundane network management tasks. As quick one-off hacks have turned into things that I actually depend on, I've noticed a lot of ugliness that I want to fix.

Everything here is assuming Python as the language of choice:
  • For scripts that send email, check to make sure the list of mail receivers is up-to-date.
  • Look for those nasty embedded IP addresses and replace them with DNS names.
  • Change from old-style open(FILE)/close(FILE) constructs to with open(FILE) as f constructs.
  • Get rid of "pickles" for persistent structured data storage. Pickles are a form of native Python object serialization that are quick and convenient, but have a lot of potential problems. I've mostly used Python's native SQLite3 library to store data in simple persistent databases, but occasionally I just use plain text files.
  • Look for repetitive code in the main script logic and try to move it into functions or classes where possible. For example, I had several scripts that were building Continue reading

Handy Tshark Expressions

Tshark is the CLI version of Wireshark, and it's amazing. I'm going to start collecting some of my favorite tshark one-liners here. Check back often.

Find All Unique Filenames Referenced in SMB2
tshark -r file.pcap -Tfields -e ip.src -e ip.dst -e text smb2 | grep -oP "GUID handle File: .*?," | sort | uniq | awk -F: '{print $2}' | sed 's/,//'

Notes:
You don't actually need to include the ip.src and ip.dst fields, since they're not extracted by the grep command. I include them in case I want to do an ad-hoc grep for an IP address during the analysis process. Another way to do the same thing would be to modify the display filter to look only for certain addresses, e.g.:

tshark -r file.pcap -Tfields -e text smb2 and ip.addr==1.1.1.1 | grep -oP "GUID handle File: .*?," | sort | uniq | awk -F: '{print $2}' | sed 's/,//'

How to Tell if TCP Payloads Are Identical

I was working on a problem today in which vendor tech support was suggesting that a firewall was subtly modifying TCP data payloads. I couldn't find any suggestion of this in the firewall logs, but seeing as how I've seen that vendor's firewall logs lie egregiously in the past, I wanted to verify it independently.

I took a packet capture from both hosts involved in the conversation and started thinking about how to see if the data sent by the server was the same as the data received by the client. I couldn't just compare the capture files themselves, because elements like timestamps, TTLs, and IP checksums would be different.

After a bunch of fiddling around, I came up with the idea of using tshark to extract the TCP payloads for each stream in the capture file and hash the results. If the hashes matched, the TCP payloads were being transferred unmodified. Here are the shell commands to do this:

tshark -r server.pcap -T fields -e tcp.stream | sort -u | sed 's/r//' | xargs -i tshark -r server.pcap -q -z follow,tcp,raw,{} | md5sum
2cfe2dbb5f6220f29ff8aff82f7f68f5 *-

You then run exactly the same commands on the "client.pcap" file Continue reading

Java is to JavaScript as Car is to Carpet – a Beginner’s Guide

Some recent discussions at work have led me to the surprising realization that lots of people working in IT don't understand that Java and JavaScript are almost completely unrelated to each other. This is actually a fairly important misunderstanding to correct:  it leads to wasted troubleshooting efforts, such as downgrading or upgrading Windows Java installations in response to browser JavaScript errors.

I found the title of this blog entry in a StackOverflow post: "Java is to JavaScript as Car is to Carpet". That's pretty much it, in a nutshell. For the record, the only things that Java and JavaScript have in common are:
  1. They are both programming languages.
  2. The word "Java".
  3. Both came out of the web technology explosion of the early 1990s.
  4. Both are frequently encountered in the context of web browsers.
Java is a compiled programming language that was originally developed with a major goal of allowing similar or identical codebases to run on different platforms without needing to be recompiled. It does this by compiling to "bytecode" rather than platform-specific machine code, which then typically runs inside a so-called "Java Virtual Machine". Java was originally developed and controlled by Sun Microsystems (now Oracle), but it has Continue reading

Understanding Flow Export Terminology

The variety of terms used to describe network flow export technologies and components can be pretty confusing. Just last year I wrote a post on web usage tracking and NetFlow that is already a bit obsolete, so here's an attempt to explain some of the newer terms and capabilities in use today.

NetFlow Version 5
NetFlow v5 is sort of the least common denominator in flow technologies. Almost all vendors and devices that support a flow export technology will do NetFlow v5. Because it's only capable of exporting information about packet fields up to layer 4, however, it's not flexible enough to use for analytics that require information about the application layer. NetFlow v5 tracks only the following data:
  • Source interface
  • Source and destination IP address
  • Layer 4 protocol
  • TCP flags
  • Type of Service
  • Egress interface
  • Packet count
  • Byte count
  • BGP origin AS
  • BGP peer AS
  • IP next hop
  • Source netmask
  • Destination netmask
Netflow Version 9
Netflow v9 was Cisco's first attempt at defining an extensible flow export format, defined in RFC 3954 back in 2004. It provides a flexible format for building customizable flow export records that contain a wide variety of information types. Many of the goals for Continue reading

Convert Hex to Decimal in IOS

Lots of IOS commands produce output in hex that I sometimes want to convert to decimal. Common ones for me are stuff like:

show ip cache flow
show ip flow top-talkers

and various debug commands. For example:

Router#sh ip cache flow | i Fa1/0.6
Fa0/1.6  10.5.188.158   Tu101*  10.5.24.15  06 0DA7 6D61  345

I have no idea what the port numbers in columns six and seven are. Fortunately, if the IOS device has the TCL or Bash shells available, we can quickly convert them.

Method 1: Tcl Shell

Most routers have the Tcl shell available:

Router#tclsh
Router(tcl)#puts [expr 0xda7]
3495

Router(tcl)#puts [expr 0x6d61]
28001

You could write a callable Tcl script to make this permanently available from normal EXEC mode too.

Method 2: Bash Shell

The Bash shell came out in one of the early IOS 15.0 versions, so you may or may not have it available. You need to explicitly enable it by entering "shell processing full" in global configuration mode.

Router#sh run | i shell
shell processing full < required to enable Bash in IOS 15+
Router#printf "%d" 0xda7
3495
Router#printf "%d" 0x6d61
28001


Building a Ghetto WAN Emulation Network

I wanted a way to do some controlled tests of WAN acceleration products, using a production network. You can buy or rent commercial WAN emulators, but for my purposes it seemed like an improvised solution would suffice. I had a couple of Cisco 2800 routers, a switch, and an ESXi box in my lab that I could press into service, so I built a test network that looks like this:


R1 acts like the WAN router at a branch site. It has a QoS policy with a "shape average" statement on its "WAN" interface to change the bandwidth to whatever we want to test.

R2 simply NATs the test traffic onto an IP address in the production network, since I didn't feel like configuring a new production subnet just for the test.

The ESXi box is where the fun part lives: I created two vSwitches and connected one physical NIC to each. I then spun up a simple Ubuntu 12.04 VM with eth0 and eth1 connected to each of the two vSwitches, giving me a separate network connected to each Cisco router. I then enabled routing on the Linux VM and created the appropriate static routes to enable the Continue reading

Friday Distraction: Who’s Leaking >/24 to Global BGP?

[It occurred to me after finishing this that I should have done everything based on ASN, but play time is over for the day...]

An interesting conversation with my friend @denise_donohue led to this question: what providers are leaking prefixes longer than /24 to the global Internet?

Following my continuing theme of "fun stuff you can do by combining IOS and Bash", I ran a two step process via one of my BGP routers to get the answer:

$ ssh routername 'show ip bgp prefix-list GT24' > /tmp/gt24.txt

$ grep "^*" /tmp/gt24.txt | awk '{print $1}' | sed 's/*>i//g' | awk -F. '{OFS=".";print $1,$2 ".0.0"}'  | sort -u | xargs -i whois {} | grep netname | sort -u

Here's the breakdown:

 Extract just valid BGP prefixes from the router output:

grep "^*" /tmp/gt24.txt | awk '{print $1}'
 
Extract just the prefix itself and substitute ".0.0" for the last two octets, normalizing to the parent /16, then remove duplicates:

| awk '{print $1}' | sed 's/*>i//g' | awk -F. '{OFS=".";print $1,$2 ".0.0"}'  | sort -u

Send those prefixes one-by-one to the "whois" command, extract the "netname" field, Continue reading

Quick Tip: Improvised File Transfer

The Python module "SimpleHTTPServer" is traditionally a quick and dirty way to test web code without the overhead of installing a full webserver. You can also use it as a quick way to transfer files between systems, if you have Python available:

jswan@ubuntu:~$ python -m SimpleHTTPServer 8080
Serving HTTP on 0.0.0.0 port 8080 ...

This causes Python to make all the files in the current directory available over HTTP on port 8080. From the client system, you can then use a browser, curl, or wget to transfer the file.

No, it's not secure, and yes, it may violate data exfiltration policies (and as an aside, Bro detects this by default). But I've used it fairly often to move files between Windows and Mac or Linux in situations where SCP isn't available and I don't feel like setting up a fileshare.

I've also used this in conjunction with the "time" command to test the effects of latency on various network protocols, and as an improvised way to test WAN optimization software.


An Operational Reason for Knowing Trivia

I've been largely out of touch with the IT certification scene lately, but I'm sure that people are still complaining incessantly about the fact that they need to memorize "trivia" in order to pass certification tests. Back when I was teaching Cisco classes full-time, my certification-oriented students were particularly bitter about this. Of course, this is a legitimate debate and the definition of "trivia" varies from person to person.

When I saw this article about CloudFlare's world-wide router meltdown, however, I immediately felt a bit smug about all those hours spent learning and teaching about packet-level trivia. If you don't want to read the article, here's the tl;dr:

  • their automated DDoS detection tool detected an attack against a customer using packets sized in the 99,000 byte range.
  • their ops staff pushed rules to their routers to drop those packets
  • their routers crashed and burned
So at this point you should be saying what some of the commenters did: huh? IP packets have a maximum size of 65,536 bytes, because the length field is 16 bits long.

In order for this meltdown to happen, they had to have a compounded series of errors:

Baby Bro, Part 3: Containers and Loops

Bro has four main container types, which I'm going to cover in somewhat nontraditional order:
  • tables
  • sets
  • vectors
  • records
Tables
A table is a collection of indexed key-value pairs: the same idea is referred to as a dictionary, associative array, or hash table in other languages. Here's a simple example that pairs letters with their place in the alphabet:


1
2
3
4
5
event bro_init()
    {   
    local letters = table([1] = "a", [2] = "b", [3] = "c");
    print letters;
    }

Running it, we get this:

jswan@so12a:~/bro$ bro tables.bro
{
[3] = c,
[1] = a,
[2] = b
}

 Note that the output isn't in the same order as the script; in Bro, like in most other languages, hash tables are unordered.

Iterating over a table with a "for" loop returns the key, again like other languages:


1
2
3
4
5
6
7
8
9
event bro_init()
    {
    local letters = table([1] = "a", [2] = "b", [3] = "c");
    
    for (key in letters)
        {
        print letters[key];
        }
    }

And the output:

jswan@so12a:~/bro$ bro tables.bro
Continue reading

Cisco Ironport WSA with WCCP and IP Spoofing

Recently I had to set up a transparent proxy with the Cisco Ironport Web Security Appliance (WSA) using WCCP on a Catalyst 6500 with a Sup720, with IP spoofing and web cache ACLs enabled. Like with many technologies, this turned out to be pretty simple but I couldn't find it documented all in one place. Perfect blog fodder!

The network topology looked like this (simplified, but not by much):



Normally when you set up a transparent proxy with WCCP, the IP address of the proxy server is used as the source of the HTTP requests. The problem in this topology is that I wanted the real source address of the client to appear in the firewall logs. The IP spoofing feature on the WSA allows this to happen, but it requires configuring bidirectional WCCP redirection on the Cat6k. If this had been a Cisco ASA firewall, we could have enabled WCCP there and saved some trouble, but in this case the network was using a firewall from another vendor that didn't support WCCP.

One important thing to realize about WCCP on the Catalyst 6500 with the Sup720 is that WCCP egress redirection is done with software switching rather than in Continue reading

Baby Bro, Part 2: Conditionals, Address Types

Bro has native types for addresses and networks, making it much easier to work with network data. Today's Baby Bro script shows global variable definition, the use of the address and subnet types, and a simple conditional:



 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# declaring global variables
# no need to put quotes around addr or subnet variable definitions
global ipv4_host:addr = 1.1.1.1; 
global ipv4_net:subnet = 1.1.0.0/16;
event bro_init()
    {   
    if (ipv4_host in ipv4_net)
        {   
        # addr and subnet types are autoconverted to strings with fmt 
        print fmt("%s is in network %s",ipv4_host,ipv4_net);
        }   
    else
        {   
        print fmt("host %s is not in network %s",ipv4_host,ipv4_net);
        }   
   }

Running this from the CLI, we get the expected output:

jswan@so12a:~/bro$ bro addr_net_types.bro
1.1.1.1 is in network 1.1.0.0/16

Bro also has several interesting built-in functions for working with network data that we'll explore in upcoming posts. For now, we'll take a look at the mask_addr function, which allows you to use Bro as an improvised subnet calculator. You can run a Bro Continue reading

Baby Bro, Part 1: Functions Etc.

[Note: Blogger seems to have done something nasty to my new blog template, so it's back to the old one at least temporarily]

Here's my first "Baby Bro" post. Before getting into using Bro scripting for its intended use of network traffic analysis, I wanted to figure out how to accomplish basic tasks common to most programming languages:

  • Functions
  • Common types and variable definitions
  • Loops
  • Conditionals
  • Iteration of container types
  • Basic string and arithmetic operations
This is the kind of stuff that many programmers can figure out instantly by looking at a language reference sheet, but I think it helps the rest of us to have explicit examples.

I'm not sure if I'll get through all of them in this series, but here's a start: a main dish of functions, with a side of string formatting and concatenation.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
  1 # "add one" is the function name
  2 # (i:int) is the variable and type passed into the function
  3 # the final "int" is the type returned by the return statement
  4 function add_one(received_value:int Continue reading

Basic Bro Language References

Finding simple examples of Bro language features is somewhat difficult: the scripts that come packaged with Bro are written by experts in the language and are quite idiomatic.

Here are some of the basic Bro language references I've found so far. In upcoming blog posts, I'll show some "Baby Bro" that is even more basic than these examples.

From Ryesecurity by Scott Runnels(@srunnels):
Solving Network Forensic Challenges with Bro, Part 1
Solving Network Forensic Challenges with Bro, Part 2
Solving Network Forensic Challenges with Bro, Part 3
Logging YouTube Titles with Bro

Justin Azoff's (@JustinAzoff) Bro Presentation on Github

The Official Bro Workshop 2011 Pages

The Bro Language Cheat Sheet

Changes!

Yee-haw! New blog template!

Trying to figure out how to do source code highlighting in a way that doesn't suck or rely on external JavaScript hosting.
1 2