OUTAGES: They Suck, But Let’s Share the After-Action Reports

“It’s impossible to solve significant problems using the same level of knowledge that created them!” –Albert Einstein Outages happen- it’s a simple fact of running any type of system, be it network, server, application, aviation, nuclear, etc.  Urs Hölzle, a Distinguished Fellow at Google and it’s first vice president of engineering, plainly states it this […]

Author information

Andrew Gallo

Senior Information Systems Engineer

Andrew Gallo is a Washington, DC based Senior Information Systems Engineer
and Network Architect, responsible for design and implementation of the
enterprise network for a large university.

Areas of specialization include the University's wide area connections,
including a 150 kilometer DWDM ring, designing a multicampus routing
policy, and business continuity planning for two online datacenters.

Andrew started during the internet upswing of the mid to late 90s
installing and terminating fiber. As his career progressed, he has had
experience with technologies from FDDI to ATM, and all speeds of Ethernet,
including a recent deployment of several metro area 100Gbps circuits.

Focusing not only on data networks, Andrew has experience in traditional
TDM voice, VoIP, and real-time, unified collaboration technologies.

Areas of interest include optical transport, network virtualization and
software defined networking, and network science and Continue reading

Great News for Networkers: Embrane’s 2014 Is Off to a Great Start!

The hype surrounding SDN (Software-Defined Networking) and now NFV (Network Function Virtualization) is widely known.  As one of the first to enter the market of innovative network solutions, I have seen new players enter (and exit) while others have re-positioned themselves to try to get a share of the pie. I’m proud to say that we have remained committed to the vision we laid out at the beginning and are delivering real products to real customers that execute on that vision.

While championing the advantages and benefits that SDN enables, in many ways we steered clear of the hype around Layer 2 solutions created to solve problems that customers don’t have. We listened to customers that craved solutions that are disruptive, but not destructive.  We focused on delivering evolutionary solutions that provided new tools to the right people rather than forcing unnecessary organization and infrastructure changes; and our customers rewarded us with their investments.

Today, we are announcing our support for OpFlex, a new open, standards-based protocol that provides a unique mechanism to enable a network controller such as Cisco’s Application Policy Infrastructure Controller (APIC) to transfer abstract policies to a set of “smart” devices capable of directly rendering Continue reading

Response: Best Of Interop 2014 Winners Unveiled

Well deserved Overall Best of Interop  (in my opinion) for OpenDaylight Hydrogen release. As the IEEE and IETF fail deliver on innovation we are turning to open source for real progress & change in networking. While the Open Daylight Hydrogen release is a bit rough, it deserves the Best of Interop award for the reasons […]

The post Response: Best Of Interop 2014 Winners Unveiled appeared first on EtherealMind.

The Value of a Microsecond

While perusing vendor datasheets, have you ever questioned the inclusion of seemingly insignificant latency specifications? Take a look at Arista's line-up, for instance. Their 7500 series chassis lists a port-to-port latency of up to 13 microseconds (that's thirteen thousandths of a millisecond) whereas their "ultra-low latency" 7150 series switches provide sub-microsecond latency.

Arista_7150_series.png

But who cares? Both values can be roughly translated as "zero" for us wetware-powered humans. (For reference, 8,333 microseconds pass in the time it takes your shiny new 120 Hz HDTV to complete one screen refresh.) So, does anyone really care about such obscenely low latency?

For a certain few organizations involved in high-frequency stock trading, those shaved microseconds can add up to billions of dollars in profit. The New York Times recently published an article titled The Wolf Hunters of Wall Street by Michael Lewis, which reveals how banks have leveraged low network latency to manipulate stock prices in open markets. (Thanks to @priscillaoppy for the tip!)

The increments of time involved were absurdly small: In theory, the fastest travel time, from Katsuyama’s desk in Manhattan to the BATS exchange in Weehawken, N.J., was about two milliseconds, and the slowest, from Continue reading

Fibre Channel: The Heart of New SDN Solutions

From Juniper to Cisco to VMware, companies are spouting up new SDN solutions. Juniper’s Contrail, Cisco’s ACI, VMware’s NSX, and more are all vying to be the next generation of data center networking. What is surprising, however, is what’s at the heart of these new technologies.

Is it VXLAN, NVGRE, Openflow? Nope. It’s Fibre Channel.

Seriously.

If you think about it, it makes sense. Fibre Channel has been doing fabrics since before we ever called Ethernet fabrics, well, fabrics. And this isn’t the first time that Fibre Channel has shown up in unusual places. There’s a version of Fibre Channel that runs inside certain airplanes, including jet fighters like the F-22.

1_FW_F-22_Raptor_participates_in_Red_Flag

Keep the skies safe from FCoE (sponsored by the Evaluator Group)

New generation of switches have been capable of Data Center Bridging (DCB), which enables Fibre Channel over Ethernet. These chips are also capable of doing native Fibre Channel So rather than build complicated VPLS fabrics or routed networks, various data center switching companies are leveraging the inherent Fibre Channel capabilities of the merchant silicon and building Fibre Channel-based underlay networks to support an IP-based overlay.

Buffer-to-buffer (B2B) credit system and losslessness of Fibre Channel, plus the new 32/128 Continue reading

Route Analytics in the Age of SDN – Now, More than Ever.

Route Analytics in the Age of SDN - Now, More than Ever.


by Brian Boyko, Technology Commentator - April 1, 2014

One of the biggest problems that may delay widespread SDN adoption is not a problem of coding or engineering, but one of poetry.

Network management is about logic and reason, but our nomenclature was written by poets. Even the word “networking” calls to the image a “netting work” - like you would find on a hammock or trapeze artist's safety net. We talk about network “pipes,” and conjure throughput like liquid water flowing through Roman aqueducts or modern PVC piping. We think of the “flow” of data through those pipes, though the only “fluid” is the movement of electrons – if that.

Metaphors that are inelegant or uninformative (“Information Superhighway” for example) fail.

Which is why SDN can be hard to envision. Intellectually, we know that it is about separating the control of where data is sent and the hardware that actually sends it. But what metaphor can we conjure that makes sense to explain this concept?

This is a problem for poets.

One metaphor that was recently used in IT Business Edge was the idea of “relying Continue reading

FCC Eases Restrictions on 5GHz UNII-1 Band

The FCC just made an unofficial news release of pending action that was approved at today's commission hearing which eases restrictions on the existing 5 GHz UNII-1 frequency band (5.150-5.250 GHz, Wi-Fi channels 36-48).

To quote from the news release (emphasis added by me):
By its action the Commission significantly increased the utility of the 100 megahertz of spectrum, and streamlined existing rules and equipment authorization procedures for devices throughout the 5 GHz band.
This ruling makes the following changes to the UNII-1 band:
  • Removes restriction on indoor use. Now the UNII-1 band can be used outdoors as well. This will allow use of the band by WISPs and for outdoor hotspot deployments which are rapidly growing with the support of telecommunications and cable operators.

  • Increases the allowed power output. The power output had been lower in UNII-1 than other 5 GHz UNII bands due to indoor-only use (50mW IR, 200mW EIRP). Exact details of the new power limits have not been released, but it is fair to assume the UNII-1 band will have similarly capable power output as UNII-2/2e bands (250mW IR, 1W EIRP) the UNII-3 band (1W IR, 4W EIRP). (Update after reviewing Commissioner Rosenworcel's Continue reading

The Packet Pushers At Dell’s Interop Booth Tuesday & Wednesday @ 3:45pm

The Dell networking team has asked the Packet Pushers to co-host a couple of Interop sessions at their booth. Greg Ferro and Ethan Banks will be at the Dell booth on Tuesday, 1-April and Wednesday, 2-April at 3:45pm. In the Tuesday session, we’ll be discussing the new Dell Z9500. In the Wednesday session, we’ll discuss […]

Author information

Ethan Banks

Ethan Banks, CCIE #20655, has been managing networks for higher ed, government, financials and high tech since 1995. Ethan co-hosts the Packet Pushers Podcast, which has seen over 2M downloads and reaches over 10K listeners. With whatever time is left, Ethan writes for fun & profit, studies for certifications, and enjoys science fiction. @ecbanks

The post The Packet Pushers At Dell’s Interop Booth Tuesday & Wednesday @ 3:45pm appeared first on Packet Pushers Podcast and was written by Ethan Banks.

Beware the Ides of March: Subsea Cable Cut Trend Continues

Earlier this month, the International Cable Protection Committee, a submarine cable advisory group, held their annual plenary in Dubai. One question that they could have considered is: Why do so many submarine cables get cut in the February/March timeframe? In this blog, we’ll look back at the last three years and the submarine cable industry’s own version of March Madness.

2012

Two years ago in February 2012, we saw a rash of closely-timed submarine cable cuts, causing Internet disruptions extending into March. In one incident, three cables were simultaneously severed in the Red Sea on February 17th, and then a fourth was damaged on the 25th off the coast of Kenya. The fourth cable was the TEAMS (The East African Marine System) cable systems, which runs from Mombasa to Fujairah, UAE.

We detailed the impact of the TEAMS cable break here, noting the resilience of many East African providers, who had purchased redundant capacity on the other two East African submarine cables: EASSy and SEACOM. The TEAMS cable would experience a second cut just weeks after it was repaired, which led TEAMS to threaten a lawsuit against the Kenya Ports Authority (KPA) over the repeated damage caused by Continue reading

Encrypting sensitive data is a must; new key management tools make it easy

Encryption is one of the best ways to prevent the type of terrible headaches that many high-profile companies have experienced with stolen data. Even if experienced hackers are able to penetrate a system, having the data encrypted can mean that nothing useful is taken.But encryption can place a high burden on a network and its users. It's not too difficult to encrypt data, but providing access to protected files for authorized users while keeping everyone else away is extremely complicated. Small organizations may be able to manage encryption schemes manually, but for anything beyond a few dozen users, the task of doing that without help becomes monumentally difficult.To read this article in full or to leave a comment, please click here(Insider Story)

Show 185 – Vendor Product Management

How do vendors decide to make network products ? What is the process and thinking behind what happens inside the wall of the vendor ? Today, Greg is join by Omar Sultan from Cisco to talk broadly about how vendors make big decisions

Author information

Greg Ferro

Greg Ferro is a Network Engineer/Architect, mostly focussed on Data Centre, Security Infrastructure, and recently Virtualization. He has over 20 years in IT, in wide range of employers working as a freelance consultant including Finance, Service Providers and Online Companies. He is CCIE#6920 and has a few ideas about the world, but not enough to really count.

He is a host on the Packet Pushers Podcast, blogger at EtherealMind.com and on Twitter @etherealmind and Google Plus.

The post Show 185 – Vendor Product Management appeared first on Packet Pushers Podcast and was written by Greg Ferro.

Custom DHCP server configuration on Opengear

For the last few years I've run my home DHCP service on a virtual private server at AWS.

This was not a great idea. It's a pain to resolve issues with my Internet service when those issues cause my laptop to stop getting an IP address assigned because the path between my house and the DHCP server has been interrupted.

The service is at Amazon because I wanted to purge "server" like things from the house, but its clear that I needed to bring DHCPd back home. I started investigating moving the service to one of my Opengear ACM5000 units, which is always running anyway because it keeps tabs on my generator and home security system, sends me text messages about interesting events and whatnot.

The Opengear web UI doesn't offer too many DHCP service configuration options, but I didn't expect that to be a problem. One of the things I love about Opengear is that most anywhere you look, the baked-in scripts and configuration elements can be replaced with user-supplied versions of those things.

I'd expected to find something like include /etc/config/dhcpd-user.conf in the automagically-generated DHCPd configuration, but it wasn't there.

So, I strings-ed every file in Continue reading

Healthy Paranoia Show 23: Phone Phreaking, Hacking and Fraud, Oh My!

That’s right listeners, you’re not in Kansas anymore! It’s time to follow that Yellow Brick Road to another episode of Healthy Paranoia. Today, we’ll be discussing phone phreaking, hacking and fraud, oh my! So we’re off to see Wizard, the Wonderful Wizard of VoIP security, Patrick McNeil. Joining me over the rainbow for this trip […]

Author information

Mrs. Y

Snarkitecht at Island of Misfit Toys

Mrs. Y is a recovering Unix engineer working in network security. Also the host of Healthy Paranoia and official nerd hunter. She likes long walks in hubsites, traveling to security conferences and spending time in the Bat Cave. Sincerely believes that every problem can be solved with a "for" loop. When not blogging or podcasting, can be found using up her 15 minutes in the Twittersphere or Google+ as @MrsYisWhy.

The post Healthy Paranoia Show 23: Phone Phreaking, Hacking and Fraud, Oh My! appeared first on Packet Pushers Podcast and was written by Mrs. Y.

Turkish Internet Censorship Takes a New Turn

Internet censorship in Turkey took a new and ominous turn yesterday. In order to better seal off access to social media sites like YouTube and Twitter, the incumbent TurkTelecom began hijacking the IP address space of public DNS resolvers like those of Google. This allows TurkTelecom servers to masquerade as Google DNS servers, returning whatever answers they want. Under normal circumstances, such queries would have been destined for servers outside the country, which is how Turkish users were circumventing the ban on YouTube imposed earlier this week. However, now local users of these global DNS services are surreptitiously redirected to alternate providers within TurkTelekom. You can see this route redirection for yourself, here and here.

Recap

Turkey’s 25th and current Prime Minister, Recep Tayyip Erdoğan, has publicly and repeatedly expressed his dislike of social media, instructing various sites to be blocked. The current attempt to curtail this important medium began on March 21st via DNS poisoning of Twitter by Turkish ISPs, probably trying to implement the government-mandated ban in a minimally invasive way.

But Continue reading

Turkey Hijacking IP addresses for popular Global DNS providers

At BGPmon we see numerous BGP hijacks every single day, some are interesting because of the size and scale of the hijack or as we’ve seen today because of the targeted hijacked prefixes. 
It all started last weekend when the Turkish president ordered the censorship of twitter.com. This started with a block of twitter by returning false twitter IP addresses by Turk Telekom DNS servers.  Soon users in Turkey discovered that changing DNS providers to Google DNS or OpenDNS was a good method of bypassing the censorship.
But as of around 9am UTC today (Saturday March 29) this changed when Turk Telekom started to hijack the IP address for popular free and open DNS providers such as Google’s 8.8.8.8, OpenDNS’ 208.67.222.222 and Level3’s 4.2.2.2.

BGP hijack
Using the Turk Telekom looking glass we can see that AS9121 (Turk Telekom) has specific /32 routes for these IP addresses. Since this is the most specific route possible for an IPv4 address, this route will always be selected and the result is that traffic for this IP address is sent to this new bogus route.

Turk Telekom route server displaying the hijacked route

Turk Telekom route server displaying the Continue reading

Network Test Automation with Ansible

In the last post, I talked about how Ansible could be used for various forms of network automation.  In the comments, Michael asked if Ansible could also be used for network test automation and verification.  Since I’m just starting to explore Ansible, I figured why not try it out.  The short answer is, it’s possible.  Let’s take a look at an example proving this out.
I just built a playbook that verifies a router, or group of routers, has reachability to a defined list of destinations.  Maybe you are deploying a new site? Maybe a new host?  Is the route being sent correctly to all sites?  Now, a playbook like this can be used to ensure every site (from the router) has reachability to the new site, subnet, or host.  You define the destination.  Cool, right?

Update: Before reading anymore, it's recommended to read the first post on Ansible for Networking.

Let’s do a quick walk through.

Here is the playbook.
As you can see, there is not a lot to it.  This playbook that we are looking at, called ‘pinger’, is using a module called ‘auto_pinger’.  Continue reading

Over Layer Versus Under Layer

Do you feel like you are in data center acronym soup these days?   I sure feel it, and I think sometimes tech-speak can help mask the real driver for change. In the data center, we are striving for a new model.  The idea of real time resource allocation and reallocation, the ideal organism that responds perfectly to every request and oh, did I mention resiliency in the whole stack for instant recovery from any fault. Wow, that would be great!  I think we have a ways to go. For now, the latest craze is to add the word virtualization to each topic.

Why is that?  I think it is because virtualization has helped us learn that you can decouple the hardware and software and create layers of abstraction that lead to better systems.  And here “better” could be lower power / cooling and space utilization, or it could the idea that a virtual machine (VM) can be your 18 wheeler, or container ship, and move the application or data anywhere you want, to help in that resource allocation / re allocation or resiliency story I mentioned above.

Now if we look on the network side, Continue reading

When Good NICs Do Bad Things: A Blast of IPv6 Multicast Listener Discovery Queries

This is the write-up of a recent event we experienced on our network.  This will be combination of a  journal of symptoms, troubleshooting steps taken, and a brief overview of the environment and platforms involved. This isn’t a forensic analysis of the cause or of different behaviors in various environments.  Rather, it’s meant to be […]

Author information

Andrew Gallo

Senior Information Systems Engineer

Andrew Gallo is a Washington, DC based Senior Information Systems Engineer
and Network Architect, responsible for design and implementation of the
enterprise network for a large university.

Areas of specialization include the University's wide area connections,
including a 150 kilometer DWDM ring, designing a multicampus routing
policy, and business continuity planning for two online datacenters.

Andrew started during the internet upswing of the mid to late 90s
installing and terminating fiber. As his career progressed, he has had
experience with technologies from FDDI to ATM, and all speeds of Ethernet,
including a recent deployment of several metro area 100Gbps circuits.

Focusing not only on data networks, Andrew has experience in traditional
TDM voice, VoIP, and real-time, unified collaboration technologies.

Areas of interest include optical transport, network virtualization and
software defined networking, and network science and graph Continue reading

Internets of Interest for 28th March 2014

  Collection of useful, relevant or just fun places on the Internets for 28th March 2014 and a bit commentary about what I’ve found interesting about them: Brocade – Brocade Extends Ethernet Fabric Leadership – Brocade hasn’t given up on the Campus. This announcement from February talks about their switch platform, most interesting is that […]

The post Internets of Interest for 28th March 2014 appeared first on EtherealMind.