My Federal Communication Conniption

The Political Problem 

The President of the United States called for the FCC to reclassify ISPs under Title II of the Communications Act as “common carriers.” 

Your telephone company is a common carrier. It is illegal for, say, Telephone Company A to degrade service quality for calls to your grandmother, who uses Telephone Company B, or charge you more to connect to Telephone Company B. 

What’s problematic is that FCC chairman, Tom Wheeler, is avoiding Title II regulation. And as Wheeler was a former lobbyist for the telecommunications industry, President Obama knew that Wheeler would probably not be for reclassifying ISPs as common carriers when he appointed him chair of the FCC back in November of 2013. 

In fact, Wheeler is opposing Obama’s proposals. Naturally. Instead of putting the ISPs under Title II of the Communications Act of 1934, he wants to classify them under the Section 706 of the Telecommunications Act of 1996.  However, courts have ruled that the FCC doesn’t have regulatory oversight of Section 706. 

So, the President publically says one should classify ISPs as common carriers, but took the Continue reading

SDN Deployments/Worries Rise Among Service Providers

SDN Deployments/Worries Rise Among Service Providers

For the second consecutive year, Packet Design surveyed attendees of the SDN/MPLS International Conference in Washington, D.C. about SDN adoption, business drivers, and concerns. The answers – mostly from service providers – were very interesting compared to the 2013 survey. Overall, use of SDN in production networks is up, but so are concerns about everything from industry standards to management skills to tools. Here’s a rundown of the results (also summarized in our announcement). 

Production SDNs Increase 

More than half of this year’s survey respondents (53 percent) said they have some production SDN deployed, compared to only 19 percent of survey respondents in 2013. Of those, about 42 percent have up to 25 percent of their networks SDN-enabled. Only 11 percent indicated having between 26 and 75 percent of their networks as production SDN. 

Business Agility, New Services Up as Top Business Drivers 

The number of survey respondents who consider “agility to respond faster to business demands” as their number one driver jumped by 150 percent (65 percent in 2014 vs. only 26 percent in 2013). Nearly Continue reading

SDN Analytics and Orchestration from the 17th Annual SDN/MPLS Conference

SDN Analytics & Orchestration from the 17th Annual SDN/MPLS Conference

Last week at the SDN/MPLS [1] conference in Washington, D.C., large service providers, research organizations and academia, and equipment manufacturers from around the world gathered to hear about the latest SDN/NFV developments. Cengiz Alaettinoglu, Packet Design’s CTO, contributed his insights and experience by presenting at the conference on “SDN Analytics: Bridging Overlay and Underlay Networks.” His premise is that underlay routing issues will impact overlay network performance, thus creating the need for SDN analytics to correlate the two and provide management visibility. 

Figure 1. SDN Analytics can correlate the impact of underlay network issues on overlay performance.

The Best Presentations on SDN Analytics and Wide Area Orchestration at SDN/MPLS 2014

The Best Presentations on SDN Analytics and Wide Area Orchestration at SDN/MPLS 2014

I attended the SDN/MPLS conference in Washington, D.C. last week, where I presented on the importance of analytics for WAN SDN application bandwidth scheduling and the need for even richer analytics when looking at the data center, network edge and WAN SDN holistically. In my presentation I highlighted the importance of accurate traffic demand matrices and the need to consider failures when selecting paths, so that the network can survive them without creating congestion. I was not the only one talking about WAN orchestration and analytics.

One of the most interesting presentations in my opinion was by Douglas Freimuth of IBM. Douglas presented his work titled “Orchestrated Bandwidth-on-Demand for Cloud Services.” It is a collaboration between IBM, Ciena, and AT&T. They carried out the work in a laboratory test bed.

In the test bed, there were three data centers (Los Angeles, New York and Chicago) running OpenStack. When VM workload in the Los Angeles data center exceeded a threshold, some of the VMs were moved to the New York data center to reduce the load. Continue reading

The Care and Feeding of a High Maintenance Network

The Care and Feeding of a High Maintenance Network

A network is an organic creation. The minute it’s born, when all new core and edge connections are made and routing is turned up, things begin to change. Many changes are self-driven due to unexpected interactions: Equal Cost Paths (ECMPs), Asymmetric Paths, etc. Other changes are due to the random nature of the Internet and are readily noticeable at the peering points into the newborn network.

Some people think that once the switch is turned on things will just work as designed. I’ve found that is rarely the case. Networks need care and feeding. Tools to check on the processing capacity, resource consumption, and well being of the network and its individual elements are required.

For the monitoring aspect of this “care and feeding,” simple SNMP tools may be used. They are perfectly adequate for tracking and graphing CPU rates, available memory and throughput for connections between network elements. However, when it comes to understanding the network’s routing and traffic patterns, using SNMP-based tools is rarely the best method.

Today’s dynamic IP networks require visibility into what’s happening Continue reading

Is November 12 D-Day for 512k prefixes?

Is November 12 D-Day for 512k prefixes?

During his presentation on October 7th at NANOG on the trends for prefixes over the past half-decade, Jim Cowie, chief scientist at Dyn Research, found something interesting: We may be reaching “512K day” as soon as next month.

Back in 2009, a typical IPv4 routing table contained only 269k entries. Today, in 2014, it is around 471k, and it is projected to be 519k in 2015… if not sooner. See the chart below from Cowie’s presentation.

Many older but still-in-use Cisco routers can only handle 512k border gateway protocol (BGP) routing entries in their TCAM memory. Indeed, a major outage occurred on August 12th of this year, when an accidental de-aggregation of 20k prefixes pushed the consensus routing table size over that 512k limit.

Was this an aberration, or an industry wake-up call to the fact that we are rapidly approaching a persistent 512k number of BGP entries? The chart above certainly suggests that latter. Cowie predicts that as the consensus BGP routing table hits 512k organically, we'll have a much larger and longer outage as everyone works to upgrade or replace Continue reading

First Impressions of the OpenDaylight Helium Release

First Impressions of the OpenDaylight Helium Release

OpenDaylight introduced Helium last month, which is a major upgrade to its open source SDN controller. Here’s my take on its features, shortcomings and strengths from an engineering perspective. 

Helium is the second software release from the OpenDaylight project and includes the following notable enhancements and new features: 

1. Support for the Apache Karaf container to allow network operators to customize installation using only the features they want
2. A new user interface, dlux (the OpenDayLight User eXperience) for client-side MVW (Model View Whatever)
3. Group Base Policy for an application-centric way of expressing policy
4. Accounting, authorization and authentication for enhanced security
5. The BGPCEP project now supports provisioning segment routing through PCEP 

With Helium there are no base, virtualization or service provider editions. Instead, bundles are supported through karaf features. Unlike the first (Hydrogen) release, Helium is targeted towards production environments. 

Out of the Box Experience 

After downloading the pre-built zip file, it is straightforward to run the karaf container with the ‘bin/karaf’ script. The ‘help’ command in the console provides a list of karaf commands Continue reading

The Current State of SDN Protocols

The Current State of SDN Protocols

In last week’s blog post I started outlining the various standards needed to make SDN a reality. Here is more detail about the relevant protocols and the IETF’s progress on each one. 

OpenFlow has emerged as a Layer 2 software defined networking (SDN) southbound protocol. Similarly, Path Computation Element Protocol (PCEP), BGP Link State Distribution (BGP-LS), and NetConf/YANG are becoming the de-facto SDN southbound protocols for Layer 3. The problem is that these protocols are stuck in various draft forms that are not interoperable, which limits the industry’s SDN progress. 

Path Computation Element Protocol (PCEP) 

PCEP is used for communicating Label Switched Paths (LSPs) between a Path Computation Client (PCC) and a Path Computation Element (PCE). PCEP has been in use since 2006. The stateful [draft-ietf-pce-stateful-pce] and PCE-initiated LSP [draft-ietf-pce-pce-initiated-lsp] extensions were added more recently and enable PCEP use for SDN deployments. The IETF drafts for both extensions have not yet advanced to “Proposed Standard” status after more than two years. 

Because the drafts went through many significant revisions, vendors are struggling to keep up with the Continue reading

Status Quo of Open Source SDN and IETF Standardization

Status Quo of Open Source SDN and IETF Standardization

Open source SDN projects like OpenDaylight are offering an open platform for network programmability to network elements such as routers and switches. They rely on standards-based approaches to provide true multi-vendor support. The IETF (and its various working groups) is moving slowly towards creating the standards needed for widespread SDN development and adoption.

OpenDaylight uses topology models, described in YANG, that are getting standardized in the NETMOD and I2RS IETF working groups. It also uses emerging protocol extensions for PCEP and BGP-LS that are being discussed in the PCE and BGP IETF working groups. OpenDaylight, with more than 200 developers, just last week issued Helium, its second software release.

On the other hand, the IETF is progressing more slowly on standardization. PCEP extensions to support statefullness and PCE-initiated LSP are still in draft form. Major routing vendors are finding it difficult to match up the draft updates to their release cycles. As a result of this, customers are challenged to pair up the draft version supported by their routing software with the controller version.

This slowness can cause other issues. Continue reading

Duty for Reporting: Is IT a Servant or a Steward

Duty for Reporting: Is IT a Servant or a Steward?

There’s an interesting “Ask Slashdot” thread from Slashdot user “MrWHO,” an IT guy who wonders why his clients preferred to receive PDF reports delivered via e-mail instead of signing into the dashboard. After all, dashboards are useful for up-to-the-minute, at-a-glance information (if they’re designed well), and they can also be used to sift through historical data. A report becomes obsolete soon after it is created (though it is far easier to show a PDF-based report to a person who does not have access to the dashboard), so it’s fair to ask why some business people prefer it.

The strange thing that I found was that many, many people were telling MrWHO, essentially, to shut up and stop complaining about the state of affairs. That IT is supposed to serve the clients and if they want reports, they get reports.

And I get that IT’s job is to support the business – or the client – but here’s my question: Does IT serve the business, or is it a steward of the business?

This is getting into some splitting of linguistic hairs, but Continue reading

Necessity of Analytics and Monitoring in the SDN Era

Necessity of Monitoring and Analytics in the SDN Era

A recent SDNCentral article about the Five Habits of Highly Effective SDN Startups asserts that achieving success in the SDN landscape will require creating focused products that solve real-world problems. In addition, the article emphasizes the need to build a sales channel with great partners, market strategically but be lean and mean, and adopt a slow and steady route into the SDN world. 

The article quoted the new landscape of SDN as we know it. Take a look at the diagram below. The building blocks range from controllers to network operating systems to monitoring and analytics. 

As the above diagram illustrates, monitoring and analytics are key. With SDN, the network self-adapts to the new demands of the application, and without visibility into these changes it’s very hard to say if an SDN application is doing the right things to your network. Understanding the programmable events that happen in real time requires mature analytics technology that can correlate service delivery to physical and virtual resource states. 

At Packet Design, we have been working on a solution for SDN analytics Continue reading

Close to the wire: How route analytics can help prevent BGP-caused outages

Close to the wire: How route analytics can help prevent BGP-caused outages

At around 3:00 a.m. Eastern Daylight Time on August 13th, Internet users started reporting slow connectivity and intermediate outages. This impacted many large networks and hosting providers including eBay, Comcast, and Time-Warner.

The problem was that some older Cisco routers have a default limit of 512k border gateway protocol (BGP) routing entries in their TCAM memory. Normally, routing tables typically have around 500k entries, so there’s a little bit of a buffer. But BGP prefix aggregation for a major service provider’s systems temporarily failed. The service provider quickly fixed the problem on their end, but not before 15,000 new prefixes were sent to the global routing table, surpassing that 512k limit. 

There is a work-around for these routers to increase the maximum size for the routing tables, but one has to wonder why these routers were running so close to maximum to begin with. In short, there is clearly a need for a larger margin of error. 

The August 13th event highlights one of the reasons that route analytics are more important than ever. With the visibility Continue reading

Network Neutrality Is a Political, Not Technical, Problem

Network Neutrality is a Political, Not Technical, Problem

We've mentioned Network Neutrality several times before on the Knetwork Knowledge Blog, but I wanted to take another look at it since it's back in the news with Wednesday's planned protests by "BattleForTheNet.com" - an artificial "Internet Slowdown" that will create symbolic "loading" symbols and artificially slow down page loading. Participating websites include Kickstarter, Reddit, Foursquare, Vimeo, Namecheap, and others. 

Packet Design has differing opinions on the issue of network neutrality. This is a bit surprising when you consider network neutrality as a technical issue, because you would expect that the engineering and mathematics would speak for themselves. It should be relatively easy to prove, from a technological standpoint, whether a neutral or particular non-neutral Internet scheme would be "better." 

But the minute you ask "better for whom?" you start to realize that network neutrality is not a technical problem. It is a political problem that happens to involve technology. 

As our CTO Cengiz Alaettinoglu said in "Hot Potatoes and Network Neutrality," BGP and IGP routing delivers packets to the next autonomous system (AS) in the route Continue reading

Framing SDN as Network as a Service (NaaS)

Framing SDN as Network as a Service (NAAS)

Tom Nolle absolutely nails the real promise of SDN in his latest blog post – Should SDN be About OpenDaylight and not OpenFlow? – which is essentially to create Network as a Service (NaaS). Readers of the Knetwork Knowledge blog will know that we have been advocating for some time that SDN is a lot more than just the separation of the network’s control and data planes, and that OpenFlow is “merely” a mechanism (not the only one) for SDN controllers to pass forwarding instructions to the underlying infrastructure. Our industry often gets lost in the technology details and misses the point, which in this case is about creating malleable network infrastructures that flex efficiently with business demands. The really interesting, valuable, and (yes) hard work is to supply the controllers with the intelligence they need to make smart infrastructure changes.   

And equally important is the recognition that we have to be able to deliver NaaS with existing network gear: A forklift upgrade to support new southbound protocols is not an option. We also need to be open to the notion Continue reading

How Route Analytics Help Detect BGP Route Hijacking

How Route Analytics Detect BGP Route Hijacking

Previously, I have talked about BGP route hijacking as a security threat and various techniques being developed to secure it. In this blog entry, I will talk about how route analytics technology can help detect BGP route hijacking in the meantime. 

There are two instances of route hijacking that need detecting. The first is when one of your prefixes is being hijacked; that is, someone is redirecting your traffic elsewhere and you are the victim. The second is when someone passes you a hijacked route; that is, you are being used as an instrument to hijack someone else. Route Analytics can help with both of these cases. However, the data sources that are needed for the analysis are different. 

When your routes are being hijacked, you cannot look at the data that is in your BGP routers in the majority of the cases. Because of the way BGP AS_path attribute works, these routes will contain your AS number and therefore, BGP will not pass them back to your routers in order to avoid loops. However, if you have access to external BGP sessions Continue reading

Q and A with Neela Jacques, OpenDaylight Executive Director

Q&A with Neela Jacques, OpenDaylight Executive Director

As OpenDaylight makes progress towards spurring adoption of SDN and NFV via an open platform, we asked Executive Director Neela Jacques his latest thoughts on the project’s current status, the state of SDN management, and what’s next.   

1. For people who may not be familiar with OpenDaylight, what is your mission?
OpenDaylight is an open source project that is creating a common, open platform for SDN and NFV. We’re a community of developers uniting competitors to work collaboratively to overcome networking’s toughest challenge -- technology fragmentation and duplication. By creating an open codebase for SDN and NFV, OpenDaylight is a vehicle for vendors to build their unique products, service and support offerings on top of a common, core set of technologies.   

2. Do you feel like the move toward open SDN has reached a critical mass? When and how do you see that happening?
In less than 15 months since we formed, OpenDaylight has grown to include 39 member companies and more than 220 developers that are working to unify the networking industry around a common, open, standard code base. Continue reading

SDN a Mixed Bag for U.S. Enterprises

SDN a Mixed Bag for U.S. Enterprises

Juniper Networks recently surveyed 400 enterprise IT “decision makers” in government, education, financial services and healthcare about their SDN adoption plans. The results were split: While almost 53 percent have plans to deploy SDN, the other half (48 percent) has no plans to adopt the technology. 

Nearly three-quarters of those who plan to implement SDN say they will do so within the next year. Their motivations are the perceived SDN benefits of improved network performance and efficiency (26 percent), simplified network operations (19 percent), and cost savings on operations (13 percent). The survey does not delve into how much of these enterprise networks will be SDN-enabled. Indeed, 63 percent of those surveyed said business networks in the next five years will be a mix of software-defined and traditional. 

The gap between the perceived benefits and reality on the ground may be inhibiting SDN deployments. The survey respondents cited cost (50 percent), difficulty integrating with existing systems (35 percent), security concerns (34 percent), and lack of skills from existing employees (28 percent) as the top challenges to SDN adoption. 

In Continue reading

Multicast Rising

Multicast Rising

Multicast hasn’t been a hot topic in networking in recent years, but that may be changing with last week’s announcements by both AT&T and Verizon that they will launch LTE multicast in 2015. Verizon plans to start embedding the technology in phones in the fourth quarter of this year and commercially launch the service in 2015. AT&T will also begin to roll out multicast capabilities next year. 

According to Verizon CFO Fran Shammo, multicast is “…the pivotal point that starts to change the way content is delivered over a mobile handset which opens up content into the wireless world." 

As Humberto Saabedra explains in an article for PhoneNews.com: “LTE Multicast allows the same content to be sent to a large number of subscribers at the same time, resulting in a more efficient use of network resources than each user requesting the same content and then having the content individually streamed to each user.” 

Currently, organizations use multicast for multimedia distribution, desktop imaging, market trading data distribution, broadcast video, online education, and other purposes where data must be delivered simultaneously to multiple receivers. Packet Continue reading

Thwarting BGP Route Hijacking with SDN as a Catalyst

Thwarting BGP Route Hijacking with SDN as a Catalyst

Following up on my last post about the security vulnerabilities in BGP, the IETF has taken two efforts to fix them. Back in 1995, the Routing Policy System Working Group was formed (I have chaired this working group, and many in the community, including folks from service providers and address registry operators, contributed). We have standardized a language called Routing Policy Specification Language (RPSL[ref]), and a security model (RP-SEC [ref]).

Network operators, both service providers and enterprises, would register their authorized routes (by chain of trust starting from the Internet Assigned Numbers Authority), and the neighbor ASs they pass these routes to. Given the state of the art in 1994, the security credentials (authentication as well as authorization) would be checked at the time of registration. We then wrote a tool that read these validated policy specifications and generated router configurations that would be immune to these kinds of attacks. Unfortunately, RPSL adoption has been low (more on this later).

IETF recently took another effort in its Secure Inter-Domain Routing Working Group (SIDR). The technology developed there can check the security credentials in-band Continue reading

BGP Security Vulnerabilities a Growing Concern

BGP Security Vulnerabilities a Growing Concern

Border Gateway Protocol (BGP), the protocol that connects different networks together, was not designed with security in mind. It is easy to take down portions of the Internet by announcing illegitimate routes to those parts (referred to as route hijacking). A classic example of this attack is a widely popularized incident a few years ago by a Pakistani service provider. The Pakistan government wanted to block YouTube internally. The service providers there injected a BGP route for YouTube and directed YouTube traffic to nowhere. This route somehow leaked outside of Pakistan, and was carried by many service providers across the Internet. This resulted, in effect, in YouTube’s removal from the Internet. 

These incidents, many not as high-profile as the YouTube incident, are routine and go back as far as I can remember. The first incident I am aware of is a dial-up Internet provider in Florida taking down the MIT network in the pre-1994, non-commercial era Internet. Early on, these incidents were results of honest configuration mistakes or fat fingering of wrong BGP configuration knobs. 

As we all know, the days of Internet innocence Continue reading