What’s new in Calico Enterprise 3.9: Live troubleshooting and resource-efficient application-level observability

We are excited to announce Calico Enterprise 3.9, which provides faster and simpler live troubleshooting using Dynamic Packet Capture for organizations while meeting regulatory and compliance requirements to access the underlying data. The release makes application-level observability resource-efficient, less security intrusive, and easier to manage. It also includes pod-to-pod encryption with Microsoft AKS and AWS EKS with AWS CNI.

Live troubleshooting

Enterprises that want to carry out live troubleshooting in their production environments face the following challenges when doing packet capture at an organizational scale:

• Difficult to limit access to packet capture by organizational roles
• Takes hours to days to setting up packet capture instead of making part of the code
• Extremely difficult to capture the right amount of data to lessen storage and compute cost
• Spend days and weeks to correlate the data collected from different Kubernetes components such as namespaces, workloads, pods, microservices

With Dynamic Packet Capture, organizations can enable DevOps, SREs, service owners to collect the data that they need when they need it. They can filter the data based on protocol and port to fine-tune their capture for faster debugging and subsequent analysis for shorter time-to-resolution. With just-in-time data collection and built-in smart correlation, Continue reading

Calico integration with WireGuard using kOps

It has been a while since I have been excited to write about encrypted tunnels. It might be the sheer pain of troubleshooting old technologies, or countless hours of falling down the rabbit hole of a project’s source code, that always motivated me to pursue a better alternative (without much luck). However, I believe luck is finally on my side.

In this blog post we will explore using open-source WireGuard, a new technology that offers encrypted tunnels with remarkable performance and an effortless implementation, to establish secure encrypted tunnels between workloads in K8s clusters.

Introduction: WireGuard

With the release of open-source Calico 3.14 back in June of 2020, Tigera announced a tech preview of its WireGuard integration, which allows node-to-node traffic to be encrypted using WireGuard.

Other encryption methods (e.g. TLS) were available to encrypt workloads’ traffic at higher TCP/IP layers (in this case, the Application Layer). However, WireGuard targets traffic at a lower layer (the Transport Layer), which makes it effective for a wider range of applications, and also reduces complexity for the user.

WireGuard is an open-source project that implements virtual private network (VPN) techniques to establish secure point-to-point connections leveraging Linux Continue reading

eBPF: When (and when not) to use it

Extended Berkeley Packet Filter (eBPF) is a relatively new feature for Linux kernels that has many DevOps, SREs, and engineers excited. But is it a one-stop shop solution for all of your Linux kernel needs? Let’s take a look at what eBPF does well, and how it stacks up against standard Linux iptables.

What is eBPF?

eBPF is a feature available in Linux kernels that allows you to run a virtual machine inside the kernel. This virtual machine allows you to safely load programs into the kernel, in order to customize its operation. Why is this important?

In the past, making changes to the kernel was difficult: there were APIs you could call to get data, but you couldn’t influence what was inside the kernel or execute code. Instead, you had to submit a patch to the Linux community and wait for it to be approved. With eBPF, you can load a program into the kernel and instruct the kernel to execute your program if, for example, a certain packet is seen or another event occurs.

With eBPF, the kernel and its behavior become highly customizable, instead of being fixed. This can be extremely beneficial, when used Continue reading

kOps adds support for Calico’s eBPF data plane

Kubernetes operations (kOps) is one of the official Kubernetes (K8s) projects. The kOps project allows for rapid deployment of production-grade K8s clusters in multiple cloud platforms. By leveraging yaml manifests, kOps delivers a familiar experience to users who have worked with kubectl. Similar to K8s clusters in popular cloud platforms, kOps helps set up self-managed clusters to easily deliver high availability. Given its ease of use, it is a very popular choice when users want to deploy self-hosted Kubernetes clusters.

With the recent release of kOps (v1.19), support for the Calico eBPF data plane was added to the utility. In addition to the above-mentioned features, the latest kOps update offers an effortless way to autodeploy K8s clusters utilizing Project Calico for networking and the Calico eBPF data plane. Calico eBPF data plane implementation replaces kube-proxy and delivers equivalent functionality; it also leverages the most optimal datapath for traffic. These changes deliver a network performance boost and source IP preservation to your cluster.

In this blog post, we will showcase the steps required to deploy a cluster that utilizes these newly available features.

What is eBPF?

eBPF is a virtual machine embedded within the Linux kernel. Continue reading

Using Calico with Kubespray

In the Kubernetes ecosystem there are a variety of ways for you to provision your cluster, and which one you choose generally depends on how well it integrates with your existing knowledge or your organization’s established tools.

Kubespray is a tool built using Ansible playbooks, inventories, and variable files—and also includes supplemental tooling such as Terraform examples for provisioning infrastructure. If you’re already using Ansible for configuration management, Kubespray might be a good fit, and there’s even documentation for integrating with your existing Ansible repository.

There are other reasons Kubespray might be a good solution: maybe you want to use the same tooling to deploy clusters on both bare metal and in the cloud, or you might have a niche use case where you have to support different Linux distributions. Or perhaps you want to take advantage of the project’s composability, which allows you to select which components you’d like to use for a variety of services, such as your container runtime or ingress controller, or—particularly relevant to this blog post—your CNI.

In this post, we’ll go over enabling Calico when following the Quick Start tutorial or using Vagrant to deploy Kubernetes locally, as well as how to configure your Continue reading

Kubernetes observability challenges in cloud-native architecture

Kubernetes is the de-facto platform for orchestrating containerized workloads and microservices, which are the building blocks of cloud-native applications. Kubernetes workloads are highly dynamic, ephemeral, and are deployed on a distributed and agile infrastructure. Although the benefits of cloud-native applications managed by Kubernetes are plenty, Kubernetes presents a new set of observability challenges in cloud-native applications.

Let’s consider some observability challenges:

• Data silos – Traditional monitoring tools specialize in collecting metrics at the application and infrastructure level. Given the highly dynamic, distributed, and ephemeral nature of cloud-native applications, this style of metrics collection creates data in silos that need to be stitched together in the context of a service in order to enable DevOps and SREs to debug service issues (e.g. slow response time, downtime, etc.). Further, if DevOps or service owners add new metrics for observation, data silos can cause broken cross-references and data misinterpretation, leading to data misalignment, slower communication, and incorrect analysis.
• Data volume and granular components – Kubernetes deployments have granular components such as pods, containers, and microservices that are running on top of distributed and ephemeral infrastructure. An incredibly high volume of granular data is generated at each layer as alerts, logs, and Continue reading

Kubernetes security issues: An examination of major attacks

In a never-ending game of cat and mouse, threat actors are exploiting, controlling and maintaining persistent access in compromised cloud infrastructure. While cloud practitioners are armed with best-in-class knowledge, support, and security practices, it is statistically impossible to have a common security posture for all cloud instances worldwide. Attackers know this, and use it to their advantage. By applying evolved tactics, techniques and procedures (TTPs), attackers are exploiting edge cases. As a result, organizations like Capital One, Jenkins, Docker and many others have experienced high-profile breaches.

TTPs are defined as “patterns of activities or methods associated with a specific threat actor or group of threat actors.” TTPs describe how threat actors (the bad guys) orchestrate, execute, and manage their operations attacks. Analysis of TTPs can benefit security operations by providing a description of how threat actors performed their attacks. Kubernetes is not immune to TTPs. Let’s examine some recent cases within the Kubernetes ecosystem.

Case #1: Misconfigured Docker

If you’re an attacker looking for misconfigured Docker instances to exploit, it’s as easy as probing open ports 2375, 2376, 2377, 4243, and 4244 on the internet. Vulnerable instances can also be located using search engines like Continue reading

What’s New in Calico v3.20

We’re excited to announce Calico v3.20! Thank you to everyone who contributed to this release! For detailed release notes, please go here. Below are some highlights from the release.

Service-based egress rules

Calico NetworkPolicy and GlobalNetworkPolicy now support egress rules that match on Kubernetes service names. Service matches in egress rules can be used to allow or deny access to in-cluster services, as well as services typically not backed by pods (for example, the Kubernetes API). Address and port information is learned from the individual endpoints within the service, making it easier to keep your network policy in sync with your workloads.

Check out the docs for more!

Golang API

In Calico v3.19, we introduced a tech-preview API server that allows management of Calico resources directly with kubectl. In v3.20, we’re building upon that with a new Golang API for Calico!

Install the API server and import the Golang API to manage Calico network policies and more, in your own applications! See the projectcalico/api repository, which includes an example, and the Go documentation page.

Configurable BGP graceful restart timer

If you’re using BGP in your cluster, the graceful restart timer is used during rolling updates to ensure Continue reading

A Sneak Peek at the “Calico Certified Operator: AWS Expert” Course

This One’s All About You

Recently, we released our new “Calico Certified Operator: AWS Expert” course. You can read more about why we created this course and how it can benefit your organization in the introductory blog post.

This blog post is different; it’s an opportunity for you, the potential learner, to get a glimpse of just a few interesting parts of the course. You won’t learn all the answers here, but you’ll learn some of the questions!

If you find that you already know the answers, then you’re all set to grab yourself a shiny badge. If not, consider taking some time to invest in your personal development and pick up some new knowledge in AWS and container networking—really current and relevant knowledge.

Let’s take a sneak peek at what the “Calico Certified Operator: AWS Expert” course has to offer.

Considering Cluster Types

There are several ways to build a Kubernetes cluster in AWS. There is no single “right” way. What are the considerations your organization should have in mind, and how can you help your organization make an informed choice?

You will review, amongst many other things, how to decide:

• Whether to pursue a managed cluster or BYOC Continue reading

Do you really need a service mesh?

The challenges involved in deploying and managing microservices have led to the creation of the service mesh, a tool for adding observability, security, and traffic management capabilities at the application layer. While a service mesh is intended to help developers and SREs with a number of use cases related to service-to-service communication within Kubernetes clusters, a service mesh also adds operational complexity and introduces an additional control plane for security teams to manage.

What is a service mesh?

A service mesh is a software infrastructure layer for controlling and monitoring internal, service-to-service traffic in microservices applications.

Service mesh provides some of the middleware and some of the components that enable service-to-service communication, such as dynamic discovery. It provides capabilities around service discovery, load balancing traffic across services, security features around encryption and authentication, tracing observability, and more. The service mesh architecture leverages design patterns to enable communication between services without requiring microservices to rewrite applications.

Service mesh architecture

One of the key aspects of how a service mesh works is that it leverages a sidecar design pattern. Services communicate and handle requests via a proxy, which is dynamically injected into each pod. Envoy is one of the most popular proxies Continue reading

High-availability connectivity for Kubernetes with dual ToR

Many platform operators in large enterprises who run Kubernetes on-premises want to leverage Border Gateway Protocol (BGP) to peer with other infrastructure. Calico Enterprise uses BGP to establish connectivity between workloads without an overlay, peer with infrastructure inside and outside of the cluster, and integrate with top-of-rack (ToR) switches to provide that connectivity.

Calico ToR connectivity has existed for some time now. However, for customers with high-availability requirements, a new high availability Kubernetes capability in Calico Enterprise now supports connectivity with dual ToR switches. From an operational standpoint, a cluster that is peered to two ToR switches will still have an active link, even if one switch becomes unavailable, thus ensuring the cluster always has a network connection. Because of the two ToR switches per rack, the whole setup is often referred to as “dual ToR.”

Dual ToR peering provides a redundant path for customers with cluster applications that cannot tolerate service downtime or failure, and require a high-availability solution. Kubernetes cannot do this on its own.

More specifically, Calico:

• Enables cluster operators to connect with, and take advantage of, dual ToR switches
• Provides two active, independent planes of connectivity between cluster nodes when a dual plane cluster is Continue reading

Calico eBPF Data Plane Deep-Dive

Why Take the Lid Off if it’s Working?

Sometimes the best way to understand something is to take it apart and see how it works. This blog post will help you take the lid off your Calico eBPF data plane based Kubernetes cluster and see how the forwarding is actually happening. The bonus is, unlike home repairs, you don’t even have to try to figure out how to put it back together again!

The target audience for this post is users who are already running a cluster with the eBPF data plane, either as a proof-of-concept or in production. Therefore, we will not go through the steps to set up a cluster from scratch. If you would like to learn how to do that, the best starting point is this documentation.

In the best case and likely scenario, you will have no data plane issues in the future and this knowledge will still help you to make informed decisions about the Calico eBPF data plane and your future clusters, and how to get the best from them. Knowledge is power!

If you are unlucky enough to experience future issues, being armed with a good understanding of the underlying technologies will Continue reading

Kubernetes security policy design: 10 critical best practices

In this blog post, I will be looking at 10 best practices for Kubernetes security policy design. Application modernization is a strategic initiative that changes the way enterprises are doing business. The journey requires a significant investment in people, processes, and technology in order to achieve the desired business outcomes of accelerating the pace of innovation, optimizing cost, and improving an enterprise’s overall security posture. It is crucial to establish the right foundation from the beginning, to avoid the high cost of re-architecture. Developing a standard and scalable security design for your Kubernetes environment helps establish the framework for implementing the necessary checks, enforcement, and visibility to enable your strategic business objectives.

High-level design

Building a scalable Kubernetes security policy design requires that you adopt a fully cloud-native mindset that takes into account how your streamlined CI/CD process should enable security policy provisioning. A sound design would enable your enforcement and policy provisioning requirements in Day-N, while accommodating Day-1 requirements. The following list summarizes the fundamental requirements that a cloud-native security policy design should include:

1. Segmentation of the control, management, and data planes
2. Segmentation of deployment environments
3. Multi-tenancy controls
4. Application microsegmentation
5. Cluster hardening
6. Ingress access control
7. Egress access control
8. Continue reading

What’s new in Calico Enterprise 3.7: eBPF data plane, high availability, and more!

As our enterprise customers build out large, multi-cluster Kubernetes environments, they are encountering an entirely new set of complex security, observability, and networking challenges, requiring solutions that operate at scale and can be deployed both on-premises and across multiple clouds. New features in our latest release add to the already formidable capabilities of Calico Enterprise.

New feature: High-availability connectivity for Kubernetes with dual ToR

Many platform operators who run Kubernetes on-premises want to leverage Border Gateway Protocol (BGP) to peer with other infrastructure. Calico uses BGP to peer with infrastructure within the cluster as well as outside of the cluster, and integrates with top-of-rack (ToR) switches to provide that connectivity.

Calico ToR connectivity has existed for some time now. However, for cluster operators using BGP who need reliable, consistent connectivity to resources outside of the cluster as well as cluster nodes on different racks, Calico Enterprise dual ToR connectivity ensures high availability with active-active redundant connectivity planes between cluster nodes and ToR switches. A cluster that is peered to two ToR switches will still have an active link, even if one switch becomes unavailable, thus ensuring the cluster always has a network connection. Kubernetes cannot do this on its Continue reading

Calico Enterprise: Leverage multiple benefits from the new eBPF data plane

Calico was designed from the ground up with a pluggable data plane architecture. The Enterprise 3.6 release introduces an exciting new eBPF (extended Berkeley Packet Filter) data plane that provides multiple benefits to users.

Great performance, lower latency for load-balanced traffic

When compared with the standard Linux data plane (based on iptables), the eBPF data plane:

• Scales to higher throughput, using less CPU per GBit
• Natively supports Kubernetes services (without kube-proxy) in a way that:
  • Reduces latency
  • Preserves external client source IP addresses
  • Supports direct server return (DSR) for reduced latency and CPU usage
  • Uses less CPU than kube-proxy to keep the data plane in sync

The impact of NAT on source IP

The application of network address translation (NAT) by kube-proxy to incoming network connections to Kubernetes services (e.g. via a service node port) is a frequently encountered friction point with Kubernetes networking. NAT has the unfortunate side effect of removing the original client source IP address from incoming traffic. When this occurs, Kubernetes network policies can’t restrict incoming traffic from specific external clients. By the time the traffic reaches the pod it no longer has the original client IP address. For some applications, knowing the Continue reading

Observe & Troubleshoot Your Kubernetes Environments with Dynamic Service Graph

Kubernetes workloads are highly dynamic, ephemeral, and are deployed on a distributed and agile infrastructure. Application developers, DevOps teams, and site reliability engineers (SREs) often require better visibility of their different microservices, what their dependencies are, how they are interconnected, and which other clients and applications access them. This makes Kubernetes observability challenges unique. While Kubernetes helps to meet the needs of deploying and managing distributed applications, its observability challenges require a Kubernetes-native approach.

Traditional monitoring and observability solutions create data silos by collecting data at different levels (e.g. infrastructure, cluster, and application levels), or from a large number of ephemeral objects that generate data across a distributed environment. Traditional monitoring and observability solutions then stitch this data together to provide a near real-time snapshot view. This approach is not scalable given the high volume of granular data generated at each level, as well as Kubernetes’ distributed nature. It also starts to become expensive and budget unfriendly to run traditional monitoring solutions, as they require higher resource consumption (high-performance memory, more compute, and higher bandwidth).

In contrast, a Kubernetes-native observability solution can visualize all information with all relationship context intact and provide a high-fidelity view of the environment. This Continue reading

Observe & Troubleshoot Your Kubernetes Environments with Dynamic Service Graph

Kubernetes workloads are highly dynamic, ephemeral, and are deployed on a distributed and agile infrastructure. Application developers, DevOps teams, and site reliability engineers (SREs) often require better visibility of their different microservices, what their dependencies are, how they are interconnected, and which other clients and applications access them. This makes Kubernetes observability challenges unique. While Kubernetes helps to meet the needs of deploying and managing distributed applications, its observability challenges require a Kubernetes-native approach.

Traditional monitoring and observability solutions create data silos by collecting data at different levels (e.g. infrastructure, cluster, and application levels), or from a large number of ephemeral objects that generate data across a distributed environment. Traditional monitoring and observability solutions then stitch this data together to provide a near real-time snapshot view. This approach is not scalable given the high volume of granular data generated at each level, as well as Kubernetes’ distributed nature. It also starts to become expensive and budget unfriendly to run traditional monitoring solutions, as they require higher resource consumption (high-performance memory, more compute, and higher bandwidth).

In contrast, a Kubernetes-native observability solution can visualize all information with all relationship context intact and provide a high-fidelity view of the environment. This Continue reading

Enabling You to Get the Best from AWS: Introducing the New Calico AWS Expert Certification

Why Create a Course About Calico in AWS?

Calico is the industry standard for Kubernetes networking and security. It offers a proven platform for your workloads across a huge range of environments, including cloud, hybrid, and on-premises.

Given this incredibly wide support, why did we decide to create a course specifically about AWS?

Well, our previous online course continues to be a great success (it’s self-paced, so if you haven’t already, we would love for you to take it and become an expert in Kubernetes networking and security). The course covers how Kubernetes networking works, how to configure and manage a Calico network, and how to secure your Kubernetes cluster.

Once you know the underlying concepts, it becomes a more important consideration to identify the nuanced differences between possible implementations. These become even more relevant once you have selected a platform to move forward with.

Amazon’s cloud computing platform, AWS, has played a huge role in changing the landscape around how users consume compute resources and data. As little as ten years ago, it would have been difficult to anticipate the speed with which companies and other organizations would embrace moving their precious compute resources and data out of their Continue reading

CVE-2021-31440: Kubernetes container escape using eBPF

In a recent post by ZDI, researchers found an out-of-bounds access flaw (CVE-2021-31440) in the Linux kernel’s (5.11.15) implementation of the eBPF code verifier: an incorrect register bounds calculation occurs while checking unsigned 32-bit instructions in an eBPF program. The flaw can be leveraged to escalate privileges and execute arbitrary code in the context of the kernel.

This vulnerability allows a local privilege escalation, which means an attacker with non-root access to the system can gain higher privileges by exploiting this vulnerability. The non-root access can be a user account without sudo or group privileges, which are usually provided to the application user.

Why you should be worried

In a Kubernetes environment, containers use the host kernel to run themselves. Therefore, the execution of malicious eBPF code as an unprivileged user in the context of the kernel can result in container escape and privilege escalation to the host.

Unprivileged users inside the container need CAP_SYS_ADMIN permission already assigned to the container to run a malicious eBPF program. For Linux kernels 5.8 and above, a new permission, CAP_BPF, is added to allow users to run eBPF programs. CAP_BPF is a subset of CAP_SYS_ADMIN.

In Kubernetes, Continue reading

CVE-2021-31440: Kubernetes container escape using eBPF

In a recent post by ZDI, researchers found an out-of-bounds access flaw (CVE-2021-31440) in the Linux kernel’s (5.11.15) implementation of the eBPF code verifier: an incorrect register bounds calculation occurs while checking unsigned 32-bit instructions in an eBPF program. The flaw can be leveraged to escalate privileges and execute arbitrary code in the context of the kernel.

This vulnerability allows a local privilege escalation, which means an attacker with non-root access to the system can gain higher privileges by exploiting this vulnerability. The non-root access can be a user account without sudo or group privileges, which are usually provided to the application user.

Why you should be worried

In a Kubernetes environment, containers use the host kernel to run themselves. Therefore, the execution of malicious eBPF code as an unprivileged user in the context of the kernel can result in container escape and privilege escalation to the host.

Unprivileged users inside the container need CAP_SYS_ADMIN permission already assigned to the container to run a malicious eBPF program. For Linux kernels 5.8 and above, a new permission, CAP_BPF, is added to allow users to run eBPF programs. CAP_BPF is a subset of CAP_SYS_ADMIN.

In Kubernetes, Continue reading