Fast and simple troubleshooting with GUI-based Dynamic Packet Capture

With the Calico 3.10 release, Dynamic Packet Capture is available in Dynamic Service Graph.

This means users who require self-service, live troubleshooting for microservices and Kubernetes workloads can capture and evaluate traffic packets on endpoints without writing a single line of code or using any 3rd-party troubleshooting tools. Users don’t need to learn about or have knowledge of kubectl or YAML to troubleshoot their microservices and Kubernetes cluster. Calico helps enforce organizational security policies by only allowing users to access their assigned namespaces and endpoints for troubleshooting.

About Dynamic Packet Capture

In most situations when you need to do a packet capture, the problem doesn’t last long and usually happens randomly. But once you narrow down the issue to a particular time or activity, you will need to set the right action plan to tackle the problem. Packet capture is now much easier, simpler, and faster than before.

Dynamic Packet Capture facilitates fast troubleshooting and easy debugging of microservice connectivity issues and performance hotspots in Kubernetes clusters. It is a Kubernetes-native custom resource that runs as part of user code against specific workloads in the cluster, without the need to execute any programs inside the cluster. Dynamic Packet Capture Continue reading

Label standard and best practices for Kubernetes security

In this blog post, I will be talking about label standard and best practices for Kubernetes security. This is a common area where I see organizations struggle to define the set of labels required to meet their security requirements. My advice is to always start with a hierarchical security design that is capable of achieving your enterprise security and compliance requirements, then define your label standard in alignment with your design. This is not meant to be a comprehensive guide for all your label requirements, but rather a framework that guides you through developing your own label standard to meet your specific security requirements.

Kubernetes labels for network policies

Labels are key/value pairs that are attached to Kubernetes objects to identify attributes that are intuitive for users and that are required for specific purposes, such as inventory reporting or the enforcement of an intent.

Label classification

Kubernetes network policies represent the intent of enforcing security controls to pods using labels to match intended endpoints. Label prefixes can be used to identify label classification. The following short-list is a high-level classification of endpoints required for developing a Kubernetes network policies design:

• Multi-tenancy
• Application microsegmentation
• External endpoints
• Host endpoints

Label scope

Labels Continue reading

Calico is celebrating 5 years

October marks the five-year anniversary of Calico Open Source, the most widely adopted solution for container networking and security. Calico Open Source was born out of Project Calico, an open-source project with an active development and user community, and has grown to power 1.5M+ nodes daily across 166 countries.

When Calico was introduced 5 years ago, the world—and technology—was much different from what it is today. The march toward distributed applications and microservices had just begun. Today, open-source projects like Project Calico are enabling the large-scale adoption of a modern architecture that is ultimately responsible for the wholesale transition to digital transformations that we are witnessing.

As part of our celebration, we’ve compiled a few comments from people who have worked on the project over the years.

“Calico works well out of the box. It scales well, rarely has bugs, and is feature rich. Tigera does a good job supporting its customers also.” —Network engineer

“[Calico is] the industry standard [for] networking for Kubernetes.” —Platform engineer

“The support for a lot of K8s distributions (either on-prem or cloud managed) is great with Calico.” —Platform architect

“[Calico helped us learn] about network segmentation in cloud-native environments.” Continue reading

Workload access control: Securely connecting containers and Kubernetes with the outside world

Containers have changed how applications are developed and deployed, with Kubernetes ascending as the de facto means of orchestrating containers, speeding development, and increasing scalability. Modern application workloads with microservices and containers eventually need to communicate with other applications or services that reside on public or private clouds outside the Kubernetes cluster. However, securely controlling granular access between these environments continues to be a challenge. Without proper security controls, containers and Kubernetes become an ideal target for attackers. At this point in the Kubernetes journey, the security team will insist that workloads meet security and compliance requirements before they are allowed to connect to outside resources.

As shown in the table below, Calico Enterprise and Calico Cloud offer multiple solutions that address different access control scenarios to limit workload access between Kubernetes clusters and APIs, databases, and applications outside the cluster. Although each solution addresses a specific requirement, they are not necessarily mutually exclusive.

Calico Cloud: What’s new in October

Calico Cloud is an industry-first security and observability SaaS platform for Kubernetes, containers, and cloud. Since its launch, we have seen customers use Calico Cloud to address a range of security and observability problems for regulatory and compliance requirements in a matter of days and weeks. In addition, they only paid for the services used, instead of an upfront investment commitment, thus aligning their budgets with their business needs.

New in October

We are excited to announce recent Calico Cloud enhancements. Highlights include:

• Managing your security and observability shouldn’t require you to manage a separate credential, authentication, and authorization access workflow. With Calico Cloud, you can bring your identity provider to manage user access to your security and observability platform. Simple sign in now requires login with the same credentials aligned to organizational roles. User management is simplified by leveraging in-house knowledge of identity management for many popular platforms. Calico Cloud supports Microsoft Azure Active Directory, Google IDP, and Open ID.
• Pre-built use case workflows are available in Calico Cloud for workload access control and enterprise security and controls. After signup:
  • Users can start monitoring and observing their application and microservices communication to external resources within minutes.
  • Users can prevent Continue reading

Kubespray 2.17 released with Calico eBPF and WireGuard support

Congratulations to the Kubespray team on the release of 2.17! This release brings support for two of the newer features in Calico: support for the eBPF data plane, and also for WireGuard encryption.

Let’s dive into configuring Kubespray to enable these new features.

If you’re interested in getting started with Kubespray and Calico, you can refer to Using Calico with Kubespray, which covers some of the settings you might want to use, as well as how to enable Calico in several of the quick start guides.

To configure Calico options when using Kubespray to deploy a cluster, you’ll need to configure some variables. If you’re using the examples in the Kubespray repository, those files are under inventory/…/group_vars/k8s_cluster/, with the Calico options residing in k8s-net-calico.yml.

eBPF data plane

Calico offers several different data planes, ensuring that end users can choose the technology that’s right for their particular use case. eBPF is a relatively new set of facilities in the Linux kernel that lets developers write code to modify its functionality at runtime in a way that is safe and efficient.

Calico’s eBPF data plane offers increased efficiency, as well as functionality like providing source IP preservation Continue reading

Rethinking observability for Kubernetes

Observability is a staple of high-performing software and DevOps teams. Research shows that a comprehensive observability solution, along with a number of other technical practices, positively contributes to continuous delivery and service uptime.

Observability is sometimes confused with monitoring, but there is a clear difference between the two; it’s important to understand the distinction. Observability refers to a technical solution that enables teams to actively debug a system. It is based on exploring activities, properties, and patterns that are not defined in advance. Monitoring, in contrast, is a technical solution that enables teams to watch and understand the state of their systems and is based on gathering pre-defined sets of metrics or logs.

What makes Kubernetes observability different?

Conventional observability and monitoring tools were designed for monolithic systems, observing the health and behavior of a single application instance. Complex distributed microservices architectures, like Kubernetes, are constantly changing, with hundreds and even thousands of pods being created and destroyed within minutes. Because this environment is so dynamic, pre-defined metrics and logs aren’t effective for troubleshooting issues. Conventional observability approaches, which work well in traditional, monolithic environments, are inadequate for Kubernetes. So an observability solution that is purpose-built for a distributed microservices Continue reading

Calico on EKS Anywhere

Amazon EKS Anywhere is an official Kubernetes distribution from AWS. It’s a new deployment option for Amazon EKS that allows the creation and operation of on-premises Kubernetes clusters on your existing infrastructure.

Since its general availability release, we’ve been working hard to ensure support for Calico on EKS Anywhere, and are happy to announce that users can now choose to use Calico for container networking and security. This gives organizations already using or planning to adopt EKS Anywhere the flexibility to choose the best container networking solution for their needs. Organizations currently using Calico can add EKS Anywhere clusters and use the same Calico solution for networking and security across on-premises and cloud platforms.

Let’s take a look at how you can get started with Calico on EKS Anywhere.

Notes:

• This procedure is disruptive; it cannot be used to migrate a cluster with applications “in-service”
• Be aware that, at present, Amazon is unlikely to be able to provide support for a cluster built in this way

Install EKS Anywhere

Install EKS Anywhere as normal on vSphere, by following this documentation.

Install the Cilium CLI and uninstall Cilium

Removing Cilium from a cluster requires using the Cilium CLI, so Continue reading

Lightning-fast Kubernetes networking with Calico & VPP

Public cloud infrastructures and microservices are pushing the limits of resources and service delivery beyond what was imaginable until very recently. In order to keep up with the demand, network infrastructures and network technologies had to evolve as well. Software-defined networking (SDN) is the pinnacle of advancement in cloud networking; by using SDN, developers can now deliver an optimized, flexible networking experience that can adapt to the growing demands of their clients.

This article will discuss how Tigera’s new Vector Packet Processing (VPP) data plane fits into this landscape and share some benchmark details about its performance. Then it will demonstrate how to run a VPP-equipped cluster using AWS public cloud and secure it with Internet Protocol Security (IPsec).

Introduction to Vector Packet Processing

Project Calico is an open-source networking and security solution. Although it focuses on securing Kubernetes networking, Calico can also be used with OpenStack and other workloads. Calico uses a modular data plane that allows a flexible approach to networking, providing a solution for both current and future networking needs.

VPP is an easily extensible, kernel-independent, highly optimised, and blazing-fast open-source data plane project that operates between layer 2 and layer 4 of the OSI Continue reading

Why securing internet-facing applications is challenging in a Kubernetes environment

Internet-facing applications are some of the most targeted workloads by threat actors. Securing this type of application is a must in order to protect your network, but this task is more complex in Kubernetes than in traditional environments, and it poses some challenges. Not only are threats magnified in a Kubernetes environment, but internet-facing applications in Kubernetes are also more vulnerable than their counterparts in traditional environments. Let’s take a look at the reasons behind these challenges, and the steps you should take to protect your Kubernetes workloads.

Threats are magnified in a Kubernetes environment

One of the fundamental challenges in a Kubernetes environment is that there is no finite set of methods that exist in terms of how workloads can be attacked. This means there are a multitude of ways an internet-facing application could be compromised, and a multitude of ways that such an attack could propagate within the environment.

Kubernetes is designed in such a way that allows anything inside a cluster to communicate with anything else inside the cluster by default, essentially giving an attacker who manages to gain a foothold unlimited access and a large attack surface. Because of this design, any time you have Continue reading

The importance of Calico’s pluggable data plane

This post will highlight and explain the importance of a pluggable data plane. But in order to do so, we first need an analogy. It’s time to talk about a brick garden wall!

Imagine you have been asked to repair a brick garden wall, because one brick has cracked through in the summer sun. You have the equipment you need, so the size of the job will depend to a great extent on how easily the brick can be removed from the wall without interfering with all the ones around it. Good luck.

Now that we have that wonderful imagery in mind, let’s look at how to go about designing walls — and how they can be maintained.

About software coupling: Fixing walls and designing great walls

“Coupling” is the term used to describe the interdependence between pieces of software. Closely coupled systems are interdependent and difficult to separate; loosely coupled systems are more like building blocks designed to work together, but they come apart cleanly. So, since the bricks in our garden wall are closely coupled (in this case, by cement), attempting to remove just one creates difficult challenges.

We can think of software as being built in “walls,” Continue reading

What’s new in Calico Enterprise 3.9: Live troubleshooting and resource-efficient application-level observability

We are excited to announce Calico Enterprise 3.9, which provides faster and simpler live troubleshooting using Dynamic Packet Capture for organizations while meeting regulatory and compliance requirements to access the underlying data. The release makes application-level observability resource-efficient, less security intrusive, and easier to manage. It also includes pod-to-pod encryption with Microsoft AKS and AWS EKS with AWS CNI.

Live troubleshooting

Enterprises that want to carry out live troubleshooting in their production environments face the following challenges when doing packet capture at an organizational scale:

• Difficult to limit access to packet capture by organizational roles
• Takes hours to days to setting up packet capture instead of making part of the code
• Extremely difficult to capture the right amount of data to lessen storage and compute cost
• Spend days and weeks to correlate the data collected from different Kubernetes components such as namespaces, workloads, pods, microservices

With Dynamic Packet Capture, organizations can enable DevOps, SREs, service owners to collect the data that they need when they need it. They can filter the data based on protocol and port to fine-tune their capture for faster debugging and subsequent analysis for shorter time-to-resolution. With just-in-time data collection and built-in smart correlation, Continue reading

Calico integration with WireGuard using kOps

It has been a while since I have been excited to write about encrypted tunnels. It might be the sheer pain of troubleshooting old technologies, or countless hours of falling down the rabbit hole of a project’s source code, that always motivated me to pursue a better alternative (without much luck). However, I believe luck is finally on my side.

In this blog post we will explore using open-source WireGuard, a new technology that offers encrypted tunnels with remarkable performance and an effortless implementation, to establish secure encrypted tunnels between workloads in K8s clusters.

Introduction: WireGuard

With the release of open-source Calico 3.14 back in June of 2020, Tigera announced a tech preview of its WireGuard integration, which allows node-to-node traffic to be encrypted using WireGuard.

Other encryption methods (e.g. TLS) were available to encrypt workloads’ traffic at higher TCP/IP layers (in this case, the Application Layer). However, WireGuard targets traffic at a lower layer (the Transport Layer), which makes it effective for a wider range of applications, and also reduces complexity for the user.

WireGuard is an open-source project that implements virtual private network (VPN) techniques to establish secure point-to-point connections leveraging Linux Continue reading

eBPF: When (and when not) to use it

Extended Berkeley Packet Filter (eBPF) is a relatively new feature for Linux kernels that has many DevOps, SREs, and engineers excited. But is it a one-stop shop solution for all of your Linux kernel needs? Let’s take a look at what eBPF does well, and how it stacks up against standard Linux iptables.

What is eBPF?

eBPF is a feature available in Linux kernels that allows you to run a virtual machine inside the kernel. This virtual machine allows you to safely load programs into the kernel, in order to customize its operation. Why is this important?

In the past, making changes to the kernel was difficult: there were APIs you could call to get data, but you couldn’t influence what was inside the kernel or execute code. Instead, you had to submit a patch to the Linux community and wait for it to be approved. With eBPF, you can load a program into the kernel and instruct the kernel to execute your program if, for example, a certain packet is seen or another event occurs.

With eBPF, the kernel and its behavior become highly customizable, instead of being fixed. This can be extremely beneficial, when used Continue reading

kOps adds support for Calico’s eBPF data plane

Kubernetes operations (kOps) is one of the official Kubernetes (K8s) projects. The kOps project allows for rapid deployment of production-grade K8s clusters in multiple cloud platforms. By leveraging yaml manifests, kOps delivers a familiar experience to users who have worked with kubectl. Similar to K8s clusters in popular cloud platforms, kOps helps set up self-managed clusters to easily deliver high availability. Given its ease of use, it is a very popular choice when users want to deploy self-hosted Kubernetes clusters.

With the recent release of kOps (v1.19), support for the Calico eBPF data plane was added to the utility. In addition to the above-mentioned features, the latest kOps update offers an effortless way to autodeploy K8s clusters utilizing Project Calico for networking and the Calico eBPF data plane. Calico eBPF data plane implementation replaces kube-proxy and delivers equivalent functionality; it also leverages the most optimal datapath for traffic. These changes deliver a network performance boost and source IP preservation to your cluster.

In this blog post, we will showcase the steps required to deploy a cluster that utilizes these newly available features.

What is eBPF?

eBPF is a virtual machine embedded within the Linux kernel. Continue reading

Using Calico with Kubespray

In the Kubernetes ecosystem there are a variety of ways for you to provision your cluster, and which one you choose generally depends on how well it integrates with your existing knowledge or your organization’s established tools.

Kubespray is a tool built using Ansible playbooks, inventories, and variable files—and also includes supplemental tooling such as Terraform examples for provisioning infrastructure. If you’re already using Ansible for configuration management, Kubespray might be a good fit, and there’s even documentation for integrating with your existing Ansible repository.

There are other reasons Kubespray might be a good solution: maybe you want to use the same tooling to deploy clusters on both bare metal and in the cloud, or you might have a niche use case where you have to support different Linux distributions. Or perhaps you want to take advantage of the project’s composability, which allows you to select which components you’d like to use for a variety of services, such as your container runtime or ingress controller, or—particularly relevant to this blog post—your CNI.

In this post, we’ll go over enabling Calico when following the Quick Start tutorial or using Vagrant to deploy Kubernetes locally, as well as how to configure your Continue reading

Kubernetes observability challenges in cloud-native architecture

Kubernetes is the de-facto platform for orchestrating containerized workloads and microservices, which are the building blocks of cloud-native applications. Kubernetes workloads are highly dynamic, ephemeral, and are deployed on a distributed and agile infrastructure. Although the benefits of cloud-native applications managed by Kubernetes are plenty, Kubernetes presents a new set of observability challenges in cloud-native applications.

Let’s consider some observability challenges:

• Data silos – Traditional monitoring tools specialize in collecting metrics at the application and infrastructure level. Given the highly dynamic, distributed, and ephemeral nature of cloud-native applications, this style of metrics collection creates data in silos that need to be stitched together in the context of a service in order to enable DevOps and SREs to debug service issues (e.g. slow response time, downtime, etc.). Further, if DevOps or service owners add new metrics for observation, data silos can cause broken cross-references and data misinterpretation, leading to data misalignment, slower communication, and incorrect analysis.
• Data volume and granular components – Kubernetes deployments have granular components such as pods, containers, and microservices that are running on top of distributed and ephemeral infrastructure. An incredibly high volume of granular data is generated at each layer as alerts, logs, and Continue reading

Kubernetes security issues: An examination of major attacks

In a never-ending game of cat and mouse, threat actors are exploiting, controlling and maintaining persistent access in compromised cloud infrastructure. While cloud practitioners are armed with best-in-class knowledge, support, and security practices, it is statistically impossible to have a common security posture for all cloud instances worldwide. Attackers know this, and use it to their advantage. By applying evolved tactics, techniques and procedures (TTPs), attackers are exploiting edge cases. As a result, organizations like Capital One, Jenkins, Docker and many others have experienced high-profile breaches.

TTPs are defined as “patterns of activities or methods associated with a specific threat actor or group of threat actors.” TTPs describe how threat actors (the bad guys) orchestrate, execute, and manage their operations attacks. Analysis of TTPs can benefit security operations by providing a description of how threat actors performed their attacks. Kubernetes is not immune to TTPs. Let’s examine some recent cases within the Kubernetes ecosystem.

Case #1: Misconfigured Docker

If you’re an attacker looking for misconfigured Docker instances to exploit, it’s as easy as probing open ports 2375, 2376, 2377, 4243, and 4244 on the internet. Vulnerable instances can also be located using search engines like Continue reading

What’s New in Calico v3.20

We’re excited to announce Calico v3.20! Thank you to everyone who contributed to this release! For detailed release notes, please go here. Below are some highlights from the release.

Service-based egress rules

Calico NetworkPolicy and GlobalNetworkPolicy now support egress rules that match on Kubernetes service names. Service matches in egress rules can be used to allow or deny access to in-cluster services, as well as services typically not backed by pods (for example, the Kubernetes API). Address and port information is learned from the individual endpoints within the service, making it easier to keep your network policy in sync with your workloads.

Check out the docs for more!

Golang API

In Calico v3.19, we introduced a tech-preview API server that allows management of Calico resources directly with kubectl. In v3.20, we’re building upon that with a new Golang API for Calico!

Install the API server and import the Golang API to manage Calico network policies and more, in your own applications! See the projectcalico/api repository, which includes an example, and the Go documentation page.

Configurable BGP graceful restart timer

If you’re using BGP in your cluster, the graceful restart timer is used during rolling updates to ensure Continue reading

A Sneak Peek at the “Calico Certified Operator: AWS Expert” Course

This One’s All About You

Recently, we released our new “Calico Certified Operator: AWS Expert” course. You can read more about why we created this course and how it can benefit your organization in the introductory blog post.

This blog post is different; it’s an opportunity for you, the potential learner, to get a glimpse of just a few interesting parts of the course. You won’t learn all the answers here, but you’ll learn some of the questions!

If you find that you already know the answers, then you’re all set to grab yourself a shiny badge. If not, consider taking some time to invest in your personal development and pick up some new knowledge in AWS and container networking—really current and relevant knowledge.

Let’s take a sneak peek at what the “Calico Certified Operator: AWS Expert” course has to offer.

Considering Cluster Types

There are several ways to build a Kubernetes cluster in AWS. There is no single “right” way. What are the considerations your organization should have in mind, and how can you help your organization make an informed choice?

You will review, amongst many other things, how to decide:

• Whether to pursue a managed cluster or BYOC Continue reading