Kubernetes observability challenges in cloud-native architecture
Kubernetes is the de-facto platform for orchestrating containerized workloads and microservices, which are the building blocks of cloud-native applications. Kubernetes workloads are highly dynamic, ephemeral, and are deployed on a distributed and agile infrastructure. Although the benefits of cloud-native applications managed by Kubernetes are plenty, Kubernetes presents a new set of observability challenges in cloud-native applications.
Let’s consider some observability challenges:
- Data silos – Traditional monitoring tools specialize in collecting metrics at the application and infrastructure level. Given the highly dynamic, distributed, and ephemeral nature of cloud-native applications, this style of metrics collection creates data in silos that need to be stitched together in the context of a service in order to enable DevOps and SREs to debug service issues (e.g. slow response time, downtime, etc.). Further, if DevOps or service owners add new metrics for observation, data silos can cause broken cross-references and data misinterpretation, leading to data misalignment, slower communication, and incorrect analysis.
- Data volume and granular components – Kubernetes deployments have granular components such as pods, containers, and microservices that are running on top of distributed and ephemeral infrastructure. An incredibly high volume of granular data is generated at each layer as alerts, logs, and Continue reading
Kubernetes security issues: An examination of major attacks
In a never-ending game of cat and mouse, threat actors are exploiting, controlling and maintaining persistent access in compromised cloud infrastructure. While cloud practitioners are armed with best-in-class knowledge, support, and security practices, it is statistically impossible to have a common security posture for all cloud instances worldwide. Attackers know this, and use it to their advantage. By applying evolved tactics, techniques and procedures (TTPs), attackers are exploiting edge cases. As a result, organizations like Capital One, Jenkins, Docker and many others have experienced high-profile breaches.
TTPs are defined as “patterns of activities or methods associated with a specific threat actor or group of threat actors.” TTPs describe how threat actors (the bad guys) orchestrate, execute, and manage their operations attacks. Analysis of TTPs can benefit security operations by providing a description of how threat actors performed their attacks. Kubernetes is not immune to TTPs. Let’s examine some recent cases within the Kubernetes ecosystem.
Case #1: Misconfigured Docker
If you’re an attacker looking for misconfigured Docker instances to exploit, it’s as easy as probing open ports 2375, 2376, 2377, 4243, and 4244 on the internet. Vulnerable instances can also be located using search engines like Continue reading
What’s New in Calico v3.20
We’re excited to announce Calico v3.20! Thank you to everyone who contributed to this release! For detailed release notes, please go here. Below are some highlights from the release.
Service-based egress rules
Calico NetworkPolicy and GlobalNetworkPolicy now support egress rules that match on Kubernetes service names. Service matches in egress rules can be used to allow or deny access to in-cluster services, as well as services typically not backed by pods (for example, the Kubernetes API). Address and port information is learned from the individual endpoints within the service, making it easier to keep your network policy in sync with your workloads.
Check out the docs for more!
Golang API
In Calico v3.19, we introduced a tech-preview API server that allows management of Calico resources directly with kubectl. In v3.20, we’re building upon that with a new Golang API for Calico!
Install the API server and import the Golang API to manage Calico network policies and more, in your own applications! See the projectcalico/api repository, which includes an example, and the Go documentation page.
Configurable BGP graceful restart timer
If you’re using BGP in your cluster, the graceful restart timer is used during rolling updates to ensure Continue reading
A Sneak Peek at the “Calico Certified Operator: AWS Expert” Course
This One’s All About You
Recently, we released our new “Calico Certified Operator: AWS Expert” course. You can read more about why we created this course and how it can benefit your organization in the introductory blog post.
This blog post is different; it’s an opportunity for you, the potential learner, to get a glimpse of just a few interesting parts of the course. You won’t learn all the answers here, but you’ll learn some of the questions!
If you find that you already know the answers, then you’re all set to grab yourself a shiny badge. If not, consider taking some time to invest in your personal development and pick up some new knowledge in AWS and container networking—really current and relevant knowledge.
Let’s take a sneak peek at what the “Calico Certified Operator: AWS Expert” course has to offer.
Considering Cluster Types
There are several ways to build a Kubernetes cluster in AWS. There is no single “right” way. What are the considerations your organization should have in mind, and how can you help your organization make an informed choice?
You will review, amongst many other things, how to decide:
- Whether to pursue a managed cluster or BYOC Continue reading
Do you really need a service mesh?
The challenges involved in deploying and managing microservices have led to the creation of the service mesh, a tool for adding observability, security, and traffic management capabilities at the application layer. While a service mesh is intended to help developers and SREs with a number of use cases related to service-to-service communication within Kubernetes clusters, a service mesh also adds operational complexity and introduces an additional control plane for security teams to manage.
What is a service mesh?
A service mesh is a software infrastructure layer for controlling and monitoring internal, service-to-service traffic in microservices applications.
Service mesh provides some of the middleware and some of the components that enable service-to-service communication, such as dynamic discovery. It provides capabilities around service discovery, load balancing traffic across services, security features around encryption and authentication, tracing observability, and more. The service mesh architecture leverages design patterns to enable communication between services without requiring microservices to rewrite applications.
Service mesh architecture
One of the key aspects of how a service mesh works is that it leverages a sidecar design pattern. Services communicate and handle requests via a proxy, which is dynamically injected into each pod. Envoy is one of the most popular proxies Continue reading
High-availability connectivity for Kubernetes with dual ToR
Many platform operators in large enterprises who run Kubernetes on-premises want to leverage Border Gateway Protocol (BGP) to peer with other infrastructure. Calico Enterprise uses BGP to establish connectivity between workloads without an overlay, peer with infrastructure inside and outside of the cluster, and integrate with top-of-rack (ToR) switches to provide that connectivity.
Calico ToR connectivity has existed for some time now. However, for customers with high-availability requirements, a new high availability Kubernetes capability in Calico Enterprise now supports connectivity with dual ToR switches. From an operational standpoint, a cluster that is peered to two ToR switches will still have an active link, even if one switch becomes unavailable, thus ensuring the cluster always has a network connection. Because of the two ToR switches per rack, the whole setup is often referred to as “dual ToR.”
Dual ToR peering provides a redundant path for customers with cluster applications that cannot tolerate service downtime or failure, and require a high-availability solution. Kubernetes cannot do this on its own.
More specifically, Calico:
- Enables cluster operators to connect with, and take advantage of, dual ToR switches
- Provides two active, independent planes of connectivity between cluster nodes when a dual plane cluster is Continue reading
Calico eBPF Data Plane Deep-Dive
Why Take the Lid Off if it’s Working?
Sometimes the best way to understand something is to take it apart and see how it works. This blog post will help you take the lid off your Calico eBPF data plane based Kubernetes cluster and see how the forwarding is actually happening. The bonus is, unlike home repairs, you don’t even have to try to figure out how to put it back together again!
The target audience for this post is users who are already running a cluster with the eBPF data plane, either as a proof-of-concept or in production. Therefore, we will not go through the steps to set up a cluster from scratch. If you would like to learn how to do that, the best starting point is this documentation.
In the best case and likely scenario, you will have no data plane issues in the future and this knowledge will still help you to make informed decisions about the Calico eBPF data plane and your future clusters, and how to get the best from them. Knowledge is power!
If you are unlucky enough to experience future issues, being armed with a good understanding of the underlying technologies will Continue reading
Kubernetes security policy design: 10 critical best practices
In this blog post, I will be looking at 10 best practices for Kubernetes security policy design. Application modernization is a strategic initiative that changes the way enterprises are doing business. The journey requires a significant investment in people, processes, and technology in order to achieve the desired business outcomes of accelerating the pace of innovation, optimizing cost, and improving an enterprise’s overall security posture. It is crucial to establish the right foundation from the beginning, to avoid the high cost of re-architecture. Developing a standard and scalable security design for your Kubernetes environment helps establish the framework for implementing the necessary checks, enforcement, and visibility to enable your strategic business objectives.
High-level design
Building a scalable Kubernetes security policy design requires that you adopt a fully cloud-native mindset that takes into account how your streamlined CI/CD process should enable security policy provisioning. A sound design would enable your enforcement and policy provisioning requirements in Day-N, while accommodating Day-1 requirements. The following list summarizes the fundamental requirements that a cloud-native security policy design should include:
- Segmentation of the control, management, and data planes
- Segmentation of deployment environments
- Multi-tenancy controls
- Application microsegmentation
- Cluster hardening
- Ingress access control
- Egress access control
- Continue reading
What’s new in Calico Enterprise 3.7: eBPF data plane, high availability, and more!
As our enterprise customers build out large, multi-cluster Kubernetes environments, they are encountering an entirely new set of complex security, observability, and networking challenges, requiring solutions that operate at scale and can be deployed both on-premises and across multiple clouds. New features in our latest release add to the already formidable capabilities of Calico Enterprise.
New feature: High-availability connectivity for Kubernetes with dual ToR
Many platform operators who run Kubernetes on-premises want to leverage Border Gateway Protocol (BGP) to peer with other infrastructure. Calico uses BGP to peer with infrastructure within the cluster as well as outside of the cluster, and integrates with top-of-rack (ToR) switches to provide that connectivity.
Calico ToR connectivity has existed for some time now. However, for cluster operators using BGP who need reliable, consistent connectivity to resources outside of the cluster as well as cluster nodes on different racks, Calico Enterprise dual ToR connectivity ensures high availability with active-active redundant connectivity planes between cluster nodes and ToR switches. A cluster that is peered to two ToR switches will still have an active link, even if one switch becomes unavailable, thus ensuring the cluster always has a network connection. Kubernetes cannot do this on its Continue reading
Calico Enterprise: Leverage multiple benefits from the new eBPF data plane
Calico was designed from the ground up with a pluggable data plane architecture. The Enterprise 3.6 release introduces an exciting new eBPF (extended Berkeley Packet Filter) data plane that provides multiple benefits to users.
Great performance, lower latency for load-balanced traffic
When compared with the standard Linux data plane (based on iptables), the eBPF data plane:
- Scales to higher throughput, using less CPU per GBit
- Natively supports Kubernetes services (without kube-proxy) in a way that:
- Reduces latency
- Preserves external client source IP addresses
- Supports direct server return (DSR) for reduced latency and CPU usage
- Uses less CPU than kube-proxy to keep the data plane in sync
The impact of NAT on source IP
The application of network address translation (NAT) by kube-proxy to incoming network connections to Kubernetes services (e.g. via a service node port) is a frequently encountered friction point with Kubernetes networking. NAT has the unfortunate side effect of removing the original client source IP address from incoming traffic. When this occurs, Kubernetes network policies can’t restrict incoming traffic from specific external clients. By the time the traffic reaches the pod it no longer has the original client IP address. For some applications, knowing the Continue reading
Observe & Troubleshoot Your Kubernetes Environments with Dynamic Service Graph
Kubernetes workloads are highly dynamic, ephemeral, and are deployed on a distributed and agile infrastructure. Application developers, DevOps teams, and site reliability engineers (SREs) often require better visibility of their different microservices, what their dependencies are, how they are interconnected, and which other clients and applications access them. This makes Kubernetes observability challenges unique. While Kubernetes helps to meet the needs of deploying and managing distributed applications, its observability challenges require a Kubernetes-native approach.
Traditional monitoring and observability solutions create data silos by collecting data at different levels (e.g. infrastructure, cluster, and application levels), or from a large number of ephemeral objects that generate data across a distributed environment. Traditional monitoring and observability solutions then stitch this data together to provide a near real-time snapshot view. This approach is not scalable given the high volume of granular data generated at each level, as well as Kubernetes’ distributed nature. It also starts to become expensive and budget unfriendly to run traditional monitoring solutions, as they require higher resource consumption (high-performance memory, more compute, and higher bandwidth).
In contrast, a Kubernetes-native observability solution can visualize all information with all relationship context intact and provide a high-fidelity view of the environment. This Continue reading
Observe & Troubleshoot Your Kubernetes Environments with Dynamic Service Graph
Kubernetes workloads are highly dynamic, ephemeral, and are deployed on a distributed and agile infrastructure. Application developers, DevOps teams, and site reliability engineers (SREs) often require better visibility of their different microservices, what their dependencies are, how they are interconnected, and which other clients and applications access them. This makes Kubernetes observability challenges unique. While Kubernetes helps to meet the needs of deploying and managing distributed applications, its observability challenges require a Kubernetes-native approach.
Traditional monitoring and observability solutions create data silos by collecting data at different levels (e.g. infrastructure, cluster, and application levels), or from a large number of ephemeral objects that generate data across a distributed environment. Traditional monitoring and observability solutions then stitch this data together to provide a near real-time snapshot view. This approach is not scalable given the high volume of granular data generated at each level, as well as Kubernetes’ distributed nature. It also starts to become expensive and budget unfriendly to run traditional monitoring solutions, as they require higher resource consumption (high-performance memory, more compute, and higher bandwidth).
In contrast, a Kubernetes-native observability solution can visualize all information with all relationship context intact and provide a high-fidelity view of the environment. This Continue reading
Enabling You to Get the Best from AWS: Introducing the New Calico AWS Expert Certification
Why Create a Course About Calico in AWS?
Calico is the industry standard for Kubernetes networking and security. It offers a proven platform for your workloads across a huge range of environments, including cloud, hybrid, and on-premises.
Given this incredibly wide support, why did we decide to create a course specifically about AWS?
Well, our previous online course continues to be a great success (it’s self-paced, so if you haven’t already, we would love for you to take it and become an expert in Kubernetes networking and security). The course covers how Kubernetes networking works, how to configure and manage a Calico network, and how to secure your Kubernetes cluster.
Once you know the underlying concepts, it becomes a more important consideration to identify the nuanced differences between possible implementations. These become even more relevant once you have selected a platform to move forward with.
Amazon’s cloud computing platform, AWS, has played a huge role in changing the landscape around how users consume compute resources and data. As little as ten years ago, it would have been difficult to anticipate the speed with which companies and other organizations would embrace moving their precious compute resources and data out of their Continue reading
CVE-2021-31440: Kubernetes container escape using eBPF
In a recent post by ZDI, researchers found an out-of-bounds access flaw (CVE-2021-31440) in the Linux kernel’s (5.11.15) implementation of the eBPF code verifier: an incorrect register bounds calculation occurs while checking unsigned 32-bit instructions in an eBPF program. The flaw can be leveraged to escalate privileges and execute arbitrary code in the context of the kernel.
This vulnerability allows a local privilege escalation, which means an attacker with non-root access to the system can gain higher privileges by exploiting this vulnerability. The non-root access can be a user account without sudo or group privileges, which are usually provided to the application user.
Why you should be worried
In a Kubernetes environment, containers use the host kernel to run themselves. Therefore, the execution of malicious eBPF code as an unprivileged user in the context of the kernel can result in container escape and privilege escalation to the host.
Unprivileged users inside the container need CAP_SYS_ADMIN permission already assigned to the container to run a malicious eBPF program. For Linux kernels 5.8 and above, a new permission, CAP_BPF, is added to allow users to run eBPF programs. CAP_BPF is a subset of CAP_SYS_ADMIN.
In Kubernetes, Continue reading
CVE-2021-31440: Kubernetes container escape using eBPF
In a recent post by ZDI, researchers found an out-of-bounds access flaw (CVE-2021-31440) in the Linux kernel’s (5.11.15) implementation of the eBPF code verifier: an incorrect register bounds calculation occurs while checking unsigned 32-bit instructions in an eBPF program. The flaw can be leveraged to escalate privileges and execute arbitrary code in the context of the kernel.
This vulnerability allows a local privilege escalation, which means an attacker with non-root access to the system can gain higher privileges by exploiting this vulnerability. The non-root access can be a user account without sudo or group privileges, which are usually provided to the application user.
Why you should be worried
In a Kubernetes environment, containers use the host kernel to run themselves. Therefore, the execution of malicious eBPF code as an unprivileged user in the context of the kernel can result in container escape and privilege escalation to the host.
Unprivileged users inside the container need CAP_SYS_ADMIN permission already assigned to the container to run a malicious eBPF program. For Linux kernels 5.8 and above, a new permission, CAP_BPF, is added to allow users to run eBPF programs. CAP_BPF is a subset of CAP_SYS_ADMIN.
In Kubernetes, Continue reading
Learn from industry experts at the Kubernetes Security and Observability Summit—next week!
The Kubernetes Security and Observability Summit is only 1 week away! The industry’s first and only conference solely focused on Kubernetes security and observability will be taking place online June 3, 2021.
During the Summit, DevOps, SREs, platform architects, and security teams will enjoy the chance to network with industry experts and explore trends, strategies, and technologies for securing, observing and troubleshooting cloud-native applications.
What does security and observability mean in a cloud-native context? What challenges should Kubernetes practitioners anticipate and what opportunities should they investigate? Join us to explore these types of questions and gain valuable insight you’ll be able to take back to your teams.
Speakers & sessions
Tigera’s President & CEO, Ratan Tipirneni, will kick off the Summit with an opening keynote address. Two additional keynotes from Graeme Hay of Morgan Stanley and Keith Neilson of Discover Financial Services will follow. Attendees will then have the opportunity to attend breakout sessions organized into three tracks:
- Stories from the real world
- Best practices
- Under the hood
During these sessions, experts from industry-leading companies like Amazon, Box, Citi, EY, Mirantis, Morgan Stanley, PayPal, Salesforce, and of course, Tigera, will share real-world stories, best practices, and technical concepts related to Continue reading
Learn from industry experts at the Kubernetes Security and Observability Summit—next week!
The Kubernetes Security and Observability Summit is only 1 week away! The industry’s first and only conference solely focused on Kubernetes security and observability will be taking place online June 3, 2021.
During the Summit, DevOps, SREs, platform architects, and security teams will enjoy the chance to network with industry experts and explore trends, strategies, and technologies for securing, observing and troubleshooting cloud-native applications.
What does security and observability mean in a cloud-native context? What challenges should Kubernetes practitioners anticipate and what opportunities should they investigate? Join us to explore these types of questions and gain valuable insight you’ll be able to take back to your teams.
Speakers & sessions
Tigera’s President & CEO, Ratan Tipirneni, will kick off the Summit with an opening keynote address. Two additional keynotes from Graeme Hay of Morgan Stanley and Keith Neilson of Discover Financial Services will follow. Attendees will then have the opportunity to attend breakout sessions organized into three tracks:
- Stories from the real world
- Best practices
- Under the hood
During these sessions, experts from industry-leading companies like Amazon, Box, Citi, EY, Mirantis, Morgan Stanley, PayPal, Salesforce, and of course, Tigera, will share real-world stories, best practices, and technical concepts related to Continue reading
Why you don’t want to miss the upcoming Kubernetes Security and Observability Summit
The inaugural Kubernetes Security and Observability Summit will be a free, live, online experience full of Kubernetes-related security and observability content. On June 3, 2021, industry experts will gather under one virtual roof to discuss trends, strategies, and technologies for Kubernetes security and observability, to help you understand and navigate today’s pressing issues in the world of cloud-native applications.
Why attend?
The Summit is a great opportunity to:
- Network with the industry’s best security, DevOps, and site reliability engineer (SRE) teams for cloud-native platforms
- Learn how to secure, observe, and troubleshoot Kubernetes environments
- Explore real-world Kubernetes security and observability use cases presented by experts from industry-leading companies like Amazon, Box, Citi, EY, Mirantis, Morgan Stanley, PayPal, Salesforce, and of course, Tigera
Who should attend?
SREs, platform architects, and DevOps and security teams will all find value in attending the Summit.
- DevOps teams and SREs – Learn how to include security and observability in your CI/CD to enable security, observability, and troubleshooting
- Platform architects – Learn architecture patterns and best practices to secure and troubleshoot cloud-native applications
- Security teams – Learn how to holistically secure your cloud-native applications following today’s best practices
Speakers & sessions
An opening keynote address from Continue reading
Why you don’t want to miss the upcoming Kubernetes Security and Observability Summit
The inaugural Kubernetes Security and Observability Summit will be a free, live, online experience full of Kubernetes-related security and observability content. On June 3, 2021, industry experts will gather under one virtual roof to discuss trends, strategies, and technologies for Kubernetes security and observability, to help you understand and navigate today’s pressing issues in the world of cloud-native applications.
Why attend?
The Summit is a great opportunity to:
- Network with the industry’s best security, DevOps, and site reliability engineer (SRE) teams for cloud-native platforms
- Learn how to secure, observe, and troubleshoot Kubernetes environments
- Explore real-world Kubernetes security and observability use cases presented by experts from industry-leading companies like Amazon, Box, Citi, EY, Mirantis, Morgan Stanley, PayPal, Salesforce, and of course, Tigera
Who should attend?
SREs, platform architects, and DevOps and security teams will all find value in attending the Summit.
- DevOps teams and SREs – Learn how to include security and observability in your CI/CD to enable security, observability, and troubleshooting
- Platform architects – Learn architecture patterns and best practices to secure and troubleshoot cloud-native applications
- Security teams – Learn how to holistically secure your cloud-native applications following today’s best practices
Speakers & sessions
An opening keynote address from Continue reading
Don’t miss our session at SUSECON Digital 2021
Join us at SUSECON Digital 2021, taking place virtually from May 18–20. It’s free! Tigera VP Product Management & Business Development, Amit Gupta, will be leading a session on Kubernetes networking, security and observability with Rancher and Calico. Our team will also be at the Tigera booth waiting to speak with you.
Speaking session
Don’t miss our session on Kubernetes networking, security and observability with Rancher and Calico! You can add our session to your schedule here.
Session details
Title: Kubernetes Networking, Security and Observability with Rancher and Calico
Date: Tuesday, May 18 at 6:00–6:30 PM (BST)
Rancher enables enterprises to deliver Kubernetes-as-a-Service across any infrastructure, including hybrid, multi-cloud and multi-cluster environments. Kubernetes’ networking, security, and observability for such deployments are critical in preventing an organization’s exposure to a multitude of security and compliance issues.
In this session, you’ll learn about how you can leverage open-source Calico in Rancher (built-in) to secure your Kubernetes environments. You will also learn about how Calico Cloud and Calico Enterprise, built on open-source Calico, can help you address performance hotspots, troubleshoot microservice communication, and carry out anomaly detection. Lastly, you will learn how to bootstrap and configure your Rancher cluster along with sample network Continue reading