Arun, Author at NetworkingNexus.net

Arun

Author Archives: Arun

Understanding the Container Network Interface (CNI)

Container Network Interface (CNI) is a framework and an interface that abstracts away the network stack from the container runtime. This abstraction helps separate container management and networking thereby providing different players in the market, target the two problems independently. CNI is one of the two frameworks that make this abstraction possible, the other being the Container Network Model (CNM) that Docker implements as libNetwork. When Docker developed libNetwork (based on CNM) for docker container networking, CoreOS came up with a competing model for rkt (pronounced rocket) their container runtime. CNI being the network interface of choice for Kubernetes, the most popular container orchestration platform, it has gained popularity. CNI is now part of the CNCF consortium.

The CNI"nterface" is composed of two components - NetPlugin and the IPAM plugin. While NetPlugin is responsible for setting up network plumbing i.e. conduit between the container & the host, the IPAM plugin is responsible for IP address allocation. Since these two are designed to be pluggable, we can pick-and-choose plugins from different sources/vendors for each of them. CNI takes in a JSON configuration file from the container runtime or from an orchestrator like Kubernetes, that specifies the choice for each Continue reading

VMWare Networking to Openstack Networking to AWS Networking

Infrastructure and management of infrastructure has come a long way in the past few years. Buzzwords today (2016-17) are Private clouds, Public clouds and more recently Hybrid clouds and containers (docker, kubernetes, et.al.). Datacenter design is also changing rapidly with some companies expanding their server footprint building massive private clouds while others reduce them by adopting hybrid cloud strategies. Networking also has be re-thought and reworked by both the public cloud providers and their customers who move towards a hybrid cloud approach. One major area of investment by cloud adopters is to mimic and apply network policies and topologies present in a private data center onto one or more of public cloud providers.

Understanding the networking constructs in Openstack, VMware and AWS will help in making these networking design decisions. I will try to compare and equate network constructs in these three cloud technologies below:

	Openstack	Amazon AWS	VMware (VSwitch / DVSwitch)
Virtual Network Continue reading

How To: Setting up VPN (IPSec tunnel) to an AWS VPC

Recently I've had to set up an VPN tunnel (IPSec) between a private data center and a private Amazon VPC. There are two ways to set this up:

Amazon's VPN connection service that uses the customer gateway and virtual private gateway.

Using a VPN appliance that acts as a gateway terminating IPSec tunnel.

This post will describe method 1. Note that Amazon's VPN offering costs extra money like most other services.

AWS side configuration

Create a Virtual Private Gateway. This does not take any settings except a tag/name

Create a Customer Gateway.

Make sure the Customer Gateway mimic’s your external / gateway router in your infrastructure. (WAN IP). Select BGP or non-BGP according to your router config.

Create a new VPC, say 10.0.0.0/16

Connect the Virtual Private Gateway to this VPC. (VPG -> Attach VPC -> Select your vpc)

Open the route table for this VPC and enable route propagation (VPC -> Route table -> Route Propataion -> Yes)

Create new VPN

Choose specific VPG to associate along with Customer Gateway. You can create a Customer Gateway when creating a VPN if you haven't already done step 1).
Set routing options. Dynamic if your gateway router Continue reading

Container Namespaces – How to add networking to a docker container

I've discussed how we can network a docker container directly with the host's networking stack bypassing docker0, the default bridge docker creates for you. That method involves asking docker to create a port on a user defined bridge and from the inside configuring the container to ask for an IP by DHCP. A more advanced way of achieving this is to bring up a docker container without networking and later configure the stack out-of-band of docker. This approach is one of the methods used by Calico for example to network containers and I've spoken about that here.

Today, lets deep dive into adding interfaces to a container manually and in-turn gain some insight into how all of this works. Since this discussion is going to revolve around network namespaces I assume you have some background in that area. If you are new to the concept of namespaces and network namespaces, I recommend reading this.

Step 1: We will first bring up a docker container without networking. From docker docs, using the --network none when running a docker container leaves out container interface creation for that docker instance. Although docker skips network interface creation it brings up the container with Continue reading

How To: Setting up a GRE or VXLAN tunnel on Linux

This is a guide to setting up VXLAN/GRE tunnels between two linux machines, be it virtual or physical. There are so many guides and reference pages on the internet but I couldn't find one that is comprehensive yet simple. Most popular method of setting up tunnels in linux seems to be using OpenvSwitch. There are a few interesting posts [redhat][Brent Sailsbury's blog][David Mahler - Youtube] out there that describe tunnel setup using OVS but few that leverages linux network stack directly. The closest I could find were a kernel reference page for VXLAN and a presentation from LinuxCon2013.

In this post both methods - OpenvSwitch & only linux networking will be discussed. When we speak about VXLAN we normally discuss multicast groups and how endpoints (VTEP) learn and populate both the fdb (forwarding database entry) and the mdb (multicast group database entry). This article describes "not using" a multicast group and dynamic learning when setting the tunnel up natively, i.e. without OVS. Both methods require a fairly newer kernel (3.7 and greater) and needs the "vxlan" kernel module to be loaded. lsmod can be used to check if this Continue reading

Container Namespaces – Deep dive into container networking

Of late, there have been various Open source projects to manage networking for containers. Docker implemented "libnetwork". I've written in the past about using Calico with Docker containers. Debugging and low level tweaks to performance isn't easy without an in-depth understanding of how the network stack works for a container. This post tries to explain network namespaces and its role in container networking using default networking that comes out-of-box with docker.

As you probably already know containers use namespaces to isolate resources and rate limit their use. Linux's network namespaces are used to glue container processes and the host networking stack. Docker spawns a container in the containers own network namespace (use the CLONE_NEWNET flag defined in sched.h when calling the clone system call to create a new network namespace for the subprocess) and later on runs a veth pair (a cable with two ends) between the container namespace and the host network stack. If you are new to network namespaces this blog post by Scott gives a quick overview and serves as a good 101 refresher if you are already familiar with these concepts but haven't used them for a while.

Now lets see how to Continue reading

IP Networking – generational shift in the industry to pure-L3 network stack

After years and years of innovation, the networking industry is changing. IPv6 is now slowly being adopted - both by interest and need. With all of IPv4 addresses having been exhausted it is now a mandate on the requester to either move to IPv6 or pay a hefty premium for repurchase of v4 addresses. On the other spectrum I believe that Calico and other projects that do pure L3 only networking, is a newer approach to network segregation; not just solving specific problems in the networking infra but a trend that will slowly be adopted widely in the network community. This is mainly also because of IPv6 acting as a booster to moving to a L3 pure play networking solution. This move to L3 only networking can be seen across the infra stack - from physical servers to VMs to containers.

Docker networking eco-system begun with libnetwork and after that has propagated with multitude of solutions by both community and networking vendors. Unlike Flannel (CoreOS) and VXLAN which leverage tunneling and L2 protocols, Calico is a distributed L3 networking solution that works both with Virtual Machines and Continue reading

Is the next big thing VR, AI and Robotics? Or is it already here. Recap of AT&T Shape 2016

Attend any technology conference today (2016) and I bet you there is going to be a track for IoT (Internet of Things), VR (Virtual Reality), AI (Artificial Intelligence) and other buzz words like deep learning, machine learning, big data, robotics et all. Almost all industries across the board either already have something or in the process of inventing something that inches us closer to SkyNet and science fiction. Academia which is always a few years ahead of the industries also heavily invests in these topics. VR, AI, Robotics and Machine Learning are few of the top research topics of 2016 listed by IEEE.

I was lucky to attend one such conference - AT&T Shape this year (2016) held at AT&T Park in SF on the 15th-16th of July. AT&T Shape is about showcasing future technology - a preview into what's possible & what's coming in the next few years. This year most of the exhibits & demos were VR, Robotics and AI themed. And speaking of machine learning/AI - look at the video above. It's a video generated by Google using the photos & videos I captured during the event. Google automatically has figured out the Continue reading

Enable nested virtualization on supported hardware. (Fixing WARNING KVM acceleration not available, using ‘qemu’ issue)

Source: http://samadhisoft.com/wp-content/uploads/2009/05/nested-boxes.jpg

Whats fun without pushing things to the limit, making them do things that they weren't designed for and creating something cool. If the end result isn't cool that's ok too. My take here is that giving something a try not only keeps you occupied but also contributes to gaining knowledge. Insights acquired in this process of pushing boundaries is definitely worth all the effort. Nested Virtualization once was a cool thing and is still is for a lot of people out there. If you are on a modern x86 (Intel) architecture processor backed computer, it most probably will support nested virtualization.

For beginners out there, nested virtualization is nothing but the process of allowing a virtual medium such as a virtual machine or a container to be able to not only act as physical hardware but also further create another abstraction within itself. Think about it as a smaller container/box within a larger box. Although you can perceive the smaller box as the only box when seen from within, it isn't so. Now put a smaller box within it - that's nested virtualization. Each level of nesting does make it tighter Continue reading

Gotchas to configuring Calico with Docker

Setting up Calico with Docker has been documented with step-by-step commands on the Calico github page and I will not be repeating them here again. I am going to use this as a scratch pad for the gotchas that I learnt from following that article on github. I hope it helps some of you.

Docker networking has seen a lot of improvements both in the native libnetwork library as well as other projects & solutions. Docker container networking especially across two physical hosts is an interesting problem with various solutions out there with their own pros and cons. You could go with flannel, a L2 overlay, VXLAN overlay to facilitate multi-host container networking or choose a pure L3-only solution like Calico. Speaking of the various choices for container networking I came across this article that compares the network glue : underlay/ overlay solutions using different parameters. Feel free to check that out if you want to get an idea into what each of these (VXLAN, flannel & Calico) offer but then of course making your own comparisons and benchmarks will not only provide with the differences/features you are looking for but also Continue reading

Running docker containers with native L2 networking & DHCP

A container normally comes up on the docker0 bridge with an IP from the static docker pool. This can be configured by editing docker options and restarting docker engine.

License: CC from Docker Blog / Dave Tucker

There are cases where a container needs to come up on a pre-existing bridge or a user defined bridge. This can be achieved by giving the --net option during docker run or configuring docker daemon options pre to docker engine boot. Creating an image with dhclient installed in it & programmed to start automatically is one way of doing l2-dhcp from the physical network infrastructure. There are other ways of doing this - macvlan, ipvlan or manually plugging in a veth pair between the docker container namespace and the host network stack. I will describe the former method here. macvlan and veth pair addition methods are described here [Cr1].

Step 1: Build a docker image with the required set of tools. An example is given below. This installs dhclient, netutils and iputils along with running sshd. User root’s password is set to centos using the start.sh script. Dockerfile & script source attributed to maxamillion (github link)

Dockerfile

 FROM centos:latest  
 MAINTAINER  Continue reading

Kernel Programming 101 – Creating your own Linux Kernel Module

Have you ever wanted to write your own kernel module? This multi-part blog post will talk about creating your first kernel module, using the proc file system, gathering and updating statistics and will cover topics such as your kernel as a menu items to menuconfig, setting defaults in the kernel config, registering network callbacks for specific types of packets taking Cisco CDP as an example and some tips-tricks in writing and debugging kernel code. A lot of my own learning has been through blogs and experiments. One source that is really great and does summarize a lot of what my blog will talk about and more is TLDP (The Linux Documentation Project).

Today programming involving systems is about providing the flexibility and pace for software development by providing user space APIs that interact with the linux kernel through system calls. This approach also provides the abstraction needed to carve out the complexity in direct kernel programming. So before we dive into kernel programming-101, lets answer this question - Why Kernel Programming?

When there's need to perform operations without cpu cycles wastage and/or reduce user space copy overheads, kernel programming suits the bill. Taking networking and linux networking stack as Continue reading

ContainerWorld2016 conference in review – Part II (Container Orchestration)

Picking up from Part I of ContainerWorld2016 conference in review, this post describes the various orchestration mechanisms available to manage containers. There are more than just the tree mechanisms out there to setup & manage containers but these; Kubernetes, Mesos and Docker Swarm are gaining traction and popularity. The Open Container Night Meetup (SVDevOps meetup group) although wasn't part of the conference tracks and hosted after day 1 sessions at the same venue was very informative. As part of this meetup's lightening talks, Adrian Otto provided a good in-depth comparison of the various container orchestration tools along with the roadmap for Openstack Magnum project and Carina by Rackspace.

When talking about containers and orchestration, as Adrian put it, one can think of two methods or paradigms of interacting with any system - imperative where you have complete control on how you want the system to work along with configuration knobs accessible to you at every step and declarative where you describe the outcome and the system automates everything for you making it simple and easy but giving you less flexibility and configurability. The degree of configuration control and ease of use are two important factors that Continue reading

Setting up bonding on Ubuntu 14.04

This one was a twister to me. Linux bonding is fairly simple and there are numerous blogs out there explaining the steps to get two interfaces on a linux machine bonded. There's tons of information and step-by-step guides describing what needs to be done on an Ubuntu based system for bond configuration. I was surprised when I failed to get it working after multiple attempts. I finally ended up mixing steps from more than two different blog sources to get the setup configured correctly and more importantly persist it over reboots.

Before diving deep with specifics and yet another step-by-step guide to setting up bonds on an Ubuntu 12.02 or 14.04 machine, lets see what linux bonds are and why bonding in the first place. If you were a kid in the 90's or before you'd know what a NIC is. These days with PC's going almost extinct in the home and personal computing space, it is quite hard for users to be acquainted with network cards housing one or more NICs (Network Interface Controller). This era is all about wireless - you still do have a wireless card in your laptop or notebooks or tablets but that's Continue reading

ContainerWorld2016 conference in review – Part I

One would think that attending a multi-day conference is sedentary, just meeting new people and lending your ear to speakers and watching presentations. On the contrary it isn't so. Its extremely tiring and by the end of the day you are completely drained out. In this sense, ContainerWorld2016 that took place last week (17th, 18th Feb) proved no different than other conferences and I ended quite exhausted at the end of the conference. Although exhausting, it was informative and it turned out to be more of a vendor neutral & community oriented conference compared to many others. Talk subjects varied from customer production deployment of containers to pain points of adopting the cloud native model. This multi-part post tries to summarize take-aways and interesting discussions that took place over the two days.

The conference was well received with participation from multiple vendors and customers such as RedHat, Canonical, Docker, Google, Yelp, CapitalOne, Paypal, eBay, Netflix, Veritas and Nordstrom to name a few. Right from day one keynote all the way to closing remarks on day two, customers and vendors alike reasoned the various advantages Continue reading

Want to sell your brand to a developer? Laptop stickers could be yet another way!

A sign of pride and fun for some but chaos for others; laptop stickers as I recently figured out is yet another way of marketing and branding. I'm sure a lot of people have wondered why a company gives away goodies in conferences and exhibitions. It's for direct+indirect marketing. When you see people walking around wearing a company printed T-Shirt or a cap, that company reaches people it never spoke to directly. It can be thought about as "flooding" in networking terms.

If its known that stickers are a marketing technique, why would someone want to mutilate their laptop with them?

Good question indeed. I have seen people fret at the sight of my laptop and come at me with the expression of - "Oh my god! What has he done to the poor laptop".. The picture you see above is my work laptop. I like stickers and I put them on my laptop because I agree with a product or the company that makes a product and would want to share this with others (i.m.o. it also makes it look cool).

Laptop stickers though a smaller marketing channel, are nevertheless effective. Continue reading

New to Openstack and worried about networking pre-setup? Try VXLAN or GRE first!

As an avid developer/coder I tend to try out various languages. I do have my favorites but when I embark on a new tool or language the first experience definitely leaves behind a mark. Failing at getting something to work in the very first try is fine as long as you don't end up hating the technology. I guess I've iterated this over and over again on other posts but networking is complex and hard. Getting it right does require some (not really, it's actually a lot :-) ) amount of experimentation. With technologies changing every so often it's imperative to at least give them a try if not jump the train and adopt them.

Openstack being an opensource project as massive as the linux kernel it does have a ton of features. The neutron component that facilitates networking is quite a handful for beginners and it gets even messier when you will need to configure neutron to work with the network architecture that is already present in your datacenter. Getting a production ready neutron setup with say VLAN backed networks is a handful. After working with Neutron for some time now, i've realized that it isn't as bad Continue reading

Picasa is finally going lights off! What’s next..

Google Inc. yesterday, finally announced axing of the Picasa Desktop app (no more updates to it) and "read only" access to Picasa Web Albums starting 1st May 2016. Although quite brutal, this hasn't been unexpected. I am however one of the many people who loves Picasa Web Albums and even Google+ photos.

A Lifescape Inc. technology after being acquired by Google was released as Picasa in 2002. As many other Google services Picasa has finally become yet another service to be axed out to promote its newer counterpart in photo sharing & organizing - Google Photos. Picasa brought in online photo backup and sharing space a new perspective. Many others have followed suite (Flickr, 500px..). The picasa desktop application was very handy in quickly exporting and importing pictures to and from the web. Organizing pictures was a breeze. Picasa came as both a paid and a free service.

Photo sharing functionality has been moved to different apps/services within Google Inc. ever since. I remember a similar move couple of years ago when I received a mail stating that picasa web albums were being migrated to Google+, Google's social networking platform. Google+ has its own story and Continue reading

Looking forward to ContainerWorld2016 next week!

Container technology has been there for years but is now picking steam thanks to Docker. Container packaging, image registry hosting for containers has directly affected both use and adoption of this technology. "Cloud Native" applications are being designed and written with the direct intent of running them on containers. Today I see containers in the same "boat" as virtualization years ago. Virtualization started slowly in the late 90's and has sprawled massively moving into every datacenter out there. It brought about a massive change in the way hardware resources were consumed making it highly cost effective. Virtualization was technology only used by linux pro's till it was made easy by companies to consume. The story with containers seem to be the same.

A linux container is a virtualization environment (VE) created using kernel support. It provides process isolation and resource isolation/allocation through cgroups (control groups). Docker is a tool that wraps programs to run as isolated processes in its own filesystems. There are various posts out there explaining the differences between docker and a linux container. One big advantage that containers give is the flexibility to move between various hosts quickly and easily.We've now moved Continue reading

A glance into host routes: Tenant networking & routing using Neutron (Openstack)

Software Defined Networks (SDN) and solutions have been making a lot of noise for a few years now. Rather slowly the networking industry has begun to notice this and affect change. Not only this but SDN has also become a pioneer - a big brother - a guide persona to other complementary technologies. We now have Software Defined - Storage, Data Center, Infrastructure and so on. It's Software Defined "everything" and Software Defined "anything". Software is slowly invading the big hardware only players and the sole reason being ease of customization and lower both; capex and opex. Networking in particular is very volatile and extremely configurable.

The neutron project of openstack is also fairly customizable bringing complexity with it. I recently ran into a requirement of having isolated networks talk to each other as well as some specific networks though isolated having access to the outside world (be it outside the cloud or the company WAN). This is what's giving rise to this particular blog post. I will lay out the premise of the discussion and then explain the solution. For networking experts out there, this might seem fairly obvious. I would suggest you stop right here and jump over Continue reading

1 2 Next »

Openstack

Amazon AWS

AWS side configuration