Mitigating the Risks of Instance Metadata in AWS EKS

Compromising a pod in a Kubernetes cluster can have disastrous consequences on resources in an AWS Elastic Kubernetes Service (EKS) account if access to the Instance Metadata service is not explicitly blocked. The Instance Metadata service is an AWS API listening on a link-local IP address. Only accessible from EC2 instances, it enables the retrieval of metadata that is used to configure or manage an instance. Although you can only access instance metadata and user data from within the instance itself, the data is not protected by authentication or cryptographic methods.

A recent blog described a scenario where an attacker compromised a pod in an EKS cluster by exploiting a vulnerability in the web application it was running, thus enabling the attacker to enumerate resources in the cluster and in the associated AWS account. This scenario was simulated by running a pod and attaching to a shell inside it.

By querying the Instance Metadata service from the compromised pod, the attacker was able to access the service and retrieve temporary credentials for the identity and access management (IAM) role assigned to the EC2 instances acting as Kubernetes worker nodes. At that point, the attacker was able to pursue multiple exploits, Continue reading

Security Policy Self-Service for Developers and DevOps Teams

In today’s economy, digital assets (applications, data, and processes) determine business success. Cloud-native applications are designed to iterate rapidly, creating rapid time-to-value for businesses. Organizations that are able to rapidly build and deploy their applications have significant competitive advantage. To this end, more and more developers are creating and leading DevOps teams that not only drive application development, but also take on operational responsibilities formerly owned by platform and security teams.

What’s the Value of a Self-Service Approach?

Cloud-native applications are often designed and deployed as microservices. The development team that owns the microservice understands the behavior of the service, and is in the best position to define and manage the network security of their microservice. A self-service model enables developers to follow a simple workflow and generate network policies with minimal effort. When problems occur with. application connectivity, developers should be able to diagnose connectivity issues and resolve them quickly without having to depend on resources outside of the team.

Developers and DevOps teams can also take a leading role in managing security, which is an integral part of cloud-native applications. There are two aspects to security in the context of Kubernetes.

• Cluster security – is a uniform set Continue reading

Enforcing Enterprise Security Controls in Kubernetes using Calico Enterprise

Hybrid cloud infrastructures run critical business resources and are subject to some of the strictest network security controls. Irrespective of the industry and resource types, these controls broadly fall into three categories.

1. Segmenting environments (Dev, Staging, Prod)
2. Enforcing zones (DMZ, Trusted, etc.)
3. Compliance requirements (GDPR, PCI DSS)

Workloads (pods) running on Kubernetes are ephemeral in nature, and IP-based controls are no longer effective. The challenge is to enforce the organizational security controls on the workloads and Kubernetes nodes themselves. Customers need the following capabilities:

• Ability to implement security controls both globally and on a per-app basis: Global controls help enforce segmentation across the cluster, and work well when the workloads are classified into different environments and/or zones using labels. As long as the labels are in place, these controls will work for any new workloads.
• Generate alerts if security controls are tampered with: Anyone with valid permissions can make changes to the controls. There is a possibility that these controls can be modified without proper authorization or even with a malicious intent to bypass the security. Hence, it is important to monitor changes to the policies.
• Produce an audit log showing changes to security controls over time: This is Continue reading

Enabling Microsegmentation with Calico Enterprise

Microsegmentation is a security technique that is used to isolate workloads from one another. Microsegmentation limits the blast radius of a data breach by making network security more granular. Should a breach occur, the damage is confined to the affected segment.  Application workloads have evolved over time – starting from bare metal, to a mix of on-prem and cloud virtual machines and containers. Similarly, the pace of change has dramatically increased, both in terms of release updates and auto-scaling.

Enforcement of network security has also evolved over time, with organizations using a mix of physical/virtual firewalls and platform-specific security groups to manage network security. This creates the following challenges:

1. Management Overhead – Organizations have to maintain different products, teams and workflows to manage and operate segmentation across containers, VMs and bare metal. The diagram above shows how different platforms may require different approaches to segmentation, thereby creating a burden on the operations team.
2. Lack of Cloud-Native Performance – With hybrid cloud becoming a norm, products built for traditional workloads can neither scale nor enforce security for cloud-native deployments with minimal latency.

Calico Enterprise provides a common policy language for segmentation that works across all of your hybrid cloud and Continue reading

Announcing the Tigera – Nutanix Partnership

Today we are pleased to announce our partnership with Nutanix, creators of the industry’s most popular hyper-converged infrastructure (HCI) technology. HCI combines datacenter hardware using locally-attached storage resources with intelligent software to create flexible building blocks that replace legacy infrastructure consisting of separate servers, storage networks, and storage arrays.

Networking and securing microservices running Kubernetes and securely accessing external resources can be challenging, often requiring the use of overlay networks and NATs. At scale, this becomes extremely complex. Cloud-native enterprises seeking a consistent container networking experience across multiple cloud environments have adopted Calico, the de facto standard in open-source Kubernetes networking technologies.

Nutanix is now offering Calico as a component of Karbon, Nutanix’s enterprise Kubernetes management solution that enables turnkey provisioning, operations, and lifecycle management of Kubernetes. With this integration, Karbon users can now take advantage of simplified Kubernetes networking and production-grade network security based on Calico’s native tooling, providing scalable throughput that meets the performance demands of Karbon users.

“Karbon, now with Calico embedded, gives our customers significantly more powerful networking and network security capabilities while preserving the simplicity of provisioning and operating a Kubernetes cluster,” said Greg Muscarella, VP of Products at Nutanix. “Calico eliminates Continue reading

A Look at the New Calico eBPF Dataplane

Calico was designed from the ground up with a pluggable dataplane architecture. The Calico 3.13 release introduced an exciting new eBPF (extended Berkeley Packet Filter) dataplane targeted at those ready to adopt newer kernel versions and wanting to push the Linux kernel’s latest networking capabilities to the limit. In addition to improved throughput and latency performance compared to the standard Linux networking data plane, Calico’s eBPF data plane also includes native support for Kubernetes services without the need to run kube-proxy. One of the ways Calico’s eBPF dataplane realizes these improvements is through source IP preservation and Direct Server Return (DSR)

Kube-proxy and Source IP

The application of Network Address Translation (NAT) by kube-proxy to incoming network connections to Kubernetes services (e.g. via a service node port) is a frequently encountered friction point with Kubernetes networking. NAT has the unfortunate side effect of removing the original client source IP address from incoming traffic. When this occurs, Kubernetes network policies can’t restrict incoming traffic from specific external clients. By the time the traffic reaches the pod it no longer has the original client IP address. For some applications, knowing the source IP address is desirable or required. For example, Continue reading

Now GA: Data-in-Transit Encryption in Calico v3.15

We’re excited to announce that the latest release of Calico includes encryption for data-in-transit. Calico is the open source networking and network security solution for containers, virtual machines, and host-based workloads, offering connectivity and security for container workloads.

One of Calico’s best-known security features is its implementation of Kubernetes Network Policy, providing a way to secure container workloads by restricting traffic to and from trusted sources. This enables the traffic to be controlled, however, the traffic itself had previously remained vulnerable to interception.

A common solution to this problem is to encrypt traffic at the application layer using protocols like Transport Layer Security (TLS). Traffic can also be encrypted at a lower infrastructure level using IPsec. However, these approaches introduce an additional layer of complexity. Calico avoids that complexity by utilizing WireGuard to implement data-in-transit encryption.

WireGuard is run as a module inside the Linux kernel and provides better performance and lower power consumption than IPsec and OpenVPN tunneling protocols. The Linux version of WireGuard reached a stable production release in March and was introduced as a tech preview in the 3.14 release of Project Calico. We are pleased to announce that WireGuard encryption is now generally available with Continue reading

Kubernetes Security: Lateral Movement Detection and Defense

What is Lateral Movement?

Lateral movement refers to the techniques that a cyber-attacker uses, after gaining initial access, to move deeper into a network in search of sensitive data and other high-value assets. Lateral movement techniques are widely used in sophisticated cyber-attacks such as advanced persistent threats (APTs). An adversary uses these techniques to access other hosts from a compromised system and get access to sensitive resources, such as mail systems, shared folders, and legitimate credentials, ultimately gaining access to the identified target. Lateral movement techniques enable a threat actor to avoid detection and retain access over an extended dwell time of weeks, or even months, after the initial breach.

What are the Stages of Lateral Movement?

There are three primary stages of lateral movement: reconnaissance, credential/privilege gathering, and gaining access to other resources in the network.

How Does an Adversary Gain Unauthorized Access to a Kubernetes Cluster?

In a Kubernetes cluster, an attacker will gain initial access by compromising a pod. Once the pod is compromised, there are three main areas where the attacker can begin reconnaissance and move through the lateral movement stages to learn more about the cluster: the cloud provider metadata service, the pod networking and Continue reading

Everything You Need to Know about Kubernetes Services Networking

As a leading, open-source multi-cluster orchestration platform, Rancher lets operations teams deploy, manage and secure enterprise Kubernetes. Rancher also gives users a set of CNI options to choose from, including open-source Project Calico. Calico provides native Layer 3 routing capability for Kubernetes pods which simplifies the networking architecture, increases networking performance and provides a rich network policy model makes it easy to lock down communication so the only traffic that flows is the traffic you want to flow.

Calico utilizes Kubernetes Services, an abstraction layer which defines a logical set of pods and enables load balancing and service discovery for those pods. Services are one of the key Kubernetes primitives you need to understand to glue microservices together and expose your applications outside of the cluster. The Service resource provides an abstract way to expose an application running on a set of Pods as a network service. Sounds simple, but what’s the difference between a Cluster IP, Node Port, and Load Balancer service? And how do all these abstractions translate to real under the covers networking behavior?

Here is a short 7-minute video that explains all this and more!

In the video you’ll learn:

• How pods consume services within the Continue reading

Calico Egress Gateway: Universal Firewall Integration for Kubernetes

New applications and workloads are constantly being added to Kubernetes clusters. Those same apps need to securely communicate with resources outside the cluster behind a firewall or other control point. Firewalls require a consistent IP, but routable IPs are a limited resource that can be quickly depleted if applied to every service.

With the Calico Egress Gateway, a new feature in Calico Enterprise 3.0, existing firewalls and control points can now be used to securely manage access to infrastructure and services outside of the cluster. In addition, IT teams are now able to identify an application/workload in a Kubernetes namespace via the source IP.

As organizations progress on their Kubernetes journey from pilot to production, they begin migrating existing applications into the cluster, which has merged with the greater IT environment. For the platform teams involved, this creates challenges because these apps will need to communicate to services outside of the cluster.

1. The Platform team will need to enable Kubernetes connectivity to infrastructure and services behind a firewall (or other third-party enforcement point such as a proxy, a DLP solution, or a monitoring solution like a SIEM.)
2. The firewall will require a consistent IP to enable Continue reading

Calico Enterprise 3.0 with Calico Multi-Cluster Management

As our enterprise customers build out large, multi-cluster Kubernetes environments, they are encountering an entirely new set of security challenges, requiring solutions that operate at scale and can be deployed both on-premises and across multiple clouds.

Today we are thrilled to announce the release of Calico Enterprise 3.0 and the availability of Calico Multi-Cluster Management, a game-changing solution that provides centralized management for network security across every Kubernetes cluster in your organization.

Calico Multi-Cluster Management

Calico Multi-Cluster Management provides a centralized management plane and single point of control for multi-cluster and multi-cloud environments. Calico Enterprise’s centralized control simplifies and speeds routine maintenance, leaving more time for your platform team to address other important tasks.

For example, instead of logging in to 50 clusters one-at-a-time to make a policy change, with a single log-in to Calico Enterprise you can apply policy changes consistently across all 50 clusters. You can also automatically apply existing network security controls to new clusters as they are added.

Calico Multi-Cluster Management includes centralized log management, troubleshooting with Flow Visualizer, and cluster-wide IDS (intrusion detection). It also provides compliance reporting, and alerts on non-compliance and indicators of compromise. Alerts are sent to SIEMs, including Splunk and Continue reading

May Customer Newsletter

Welcome to the May 2020 edition of Calicomm! – our monthly newsletter for customers and partners. In the April edition, we discussed audit logs. This edition covers egress access control, which is an important aspect of micro-segmentation.

What problems are we solving?

Consider an enterprise datacenter deployment with hundreds of nodes and thousands of pods. These systems are running business applications with different levels of security requirements. A first-order security and compliance requirement in such a scenario is to ensure that a pod or host is only allowed to talk to authorized destinations. Now consider the real life scenario where there’s a churn rate (pods/hosts being added/removed) of hundreds of pods/minute. The challenge is to continue enforcing the microsegmentation in near real time despite a high churn rate.

An efficient mechanism for micro-segmentation has a direct impact on productivity. Ideally, you do not want to wait days for an access policy to be granted through a ticketing process, nor do you want to wait precious minutes for a policy change to take effect.

Micro-segmentation has two broad categories, East-West (E-W) and North-South (N-S). The following are typical use cases of egress access control within the N-S category:

• Configure default-deny Continue reading

Calico Enterprise 3.0 – Global Network Security Center for Kubernetes

As our enterprise customers build out large, multi-cluster Kubernetes environments, they are encountering an entirely new set of security challenges, requiring solutions that operate at scale and can be deployed both on-premises and across multiple clouds.

Today we are thrilled to announce the release of Calico Enterprise 3.0 and the availability of our Global Network Security Center, a game-changing solution that provides a central management plane for network security across every Kubernetes cluster in your organization.

Global Network Security Center for Kubernetes

The Calico Enterprise Global Network Security Center for Kubernetes is a centralized management plane and single point of control for multi-cluster and multi-cloud environments. Calico Enterprise’s centralized control simplifies and speeds routine maintenance, leaving more time for your platform team to address other important tasks.

For example, instead of logging in to 50 clusters one-at-a-time to make a policy change, with a single log-in to Calico Enterprise you can apply policy changes consistently across all 50 clusters. You can also automatically apply existing network security controls to new clusters as they are added.

Calico Enterprise also includes centralized log management, troubleshooting with Flow Visualizer, and cluster-wide IDS (intrusion detection). GNSC provides compliance reporting, and alerts on non-compliance Continue reading

Deploying to Kubernetes: The GitOps Way

Kubernetes adoption comes with a lot of challenges. One of them is consistently deploying applications to the platform. GitOps is a strategy which solves this problem and solves it at scale. In this blog, we will share how to leverage TravisCI and ArgoCD to design a highly scalable production-ready CI/CD workflow. 

Deployment Workflow

GitOps follows one simple principle “Git is the Source of Truth”. The entire pipeline can be divided into two broad categories. (1) Continuous Integration, where we enable our developers to develop new features, test the code and merge it into a master. (2) Continuous Delivery, where we release new versions of the code for our customers.

Repo Structure

The application and the Kubernetes manifests/helm chart both reside in a git repository. The application source code’s git repo consists of various branches. Following the same principle, we also keep the helm charts for our microservices in a git repo itself. For the sake of this blog we will assume that each source code repository will have at least three (3) branches.

1. Dev Branch: This gets deployed to the Dev Kubernetes Environment
2. Staging Branch: This gets deployed to the Staging Kubernetes Environment
3. Master Branch Continue reading

Designing On-Prem Kubernetes Networks for High Availability

Designing and maintaining networks is hard.  When deploying Kubernetes in your on-prem data center, you will need to answer a basic question: Should it be an overlay network on top of an existing network, or should it be part of an existing network? The Networking options table provides guidelines to choose the right type of networking based on various factors. If you decide to use native peering (pre-dominant option for on-prem), you will have to configure the network to ensure availability in the event of network outage (ex. Cable disconnected, TOR switch failure etc.). We cover a typical L3 highly-available network design in this post.

A cluster spans multiple racks. In an L3 deployment, these racks have different CIDR ranges. So the nodes in different racks should be able to talk to each other. Referring to the diagram below, that traffic goes through the network fabric. If you want to build out such a lab for your own learning, here is the example.

If you have a leaf-spine fabric with a single TOR (top-of-rack, or leaf switch), then that TOR becomes a point of failure for the entire rack. If all the master nodes are on the same Continue reading

Introducing the Calico eBPF Dataplane

eBPF is a hot topic right now; most of the infrastructure-focused conferences and events have included talks on eBPF over the past year, which is creating a lot of interest in the technology.

You might be wondering what eBPF is. eBPF stands for “extended Berkeley Packet Filter” which is a feature in modern Linux kernels that allows you to write mini-programs that are attached to low-level hooks in the Linux kernel, that execute based on certain events (e.g. filtering network traffic). While Calico is primarily focused on networking and security use cases, eBPF is a broad technology that applies to many other use cases as well.

We’ve always been tracking eBPF and it’s potential to enhance Calico, however, most users have not been ready for it. Improving on Calico’s already excellent dataplane using eBPF requires the latest Linux kernels, that are not always available to our enterprise customers that require a vendor-supported Linux distribution to run in production. Nevertheless, we decided to add an eBPF dataplane to support those users that are able to use the latest Linux kernels, as well as provide a future-proofed path for those who will wait until their vendor-supported Linux distributions will support the Continue reading

Why use Typha in your Calico Kubernetes Deployments?

Calico is an open source networking and network security solution for containers, virtual machines, and native host-based workloads. Calico supports a broad range of platforms including Kubernetes, OpenShift, Docker EE, OpenStack, and bare metal. In this blog, we will focus on Kubernetes pod networking and network security using Calico.

Calico uses etcd as the back-end datastore. When you run Calico on Kubernetes, you can use the same etcd datastore through the Kubernetes API server. This is called a Kubernetes backed datastore (KDD) in Calico. The following diagram shows a block-level architecture of Calico.

Calico-node runs as a Daemonset, and has a fair amount of interaction with the Kubernetes API server. It’s easy for you to profile that by simply enabling audit logs for calico-node. For example, in my kubeadm cluster, I used the following audit configuration

April Customer Newsletter

Welcome to the April 2020 edition of the Tigera Calicommunication newsletter! In the March edition, we discussed context-aware flow logs. This edition covers the next component of logging, the audit logs.

Using Calico Enterprise Audit Logs to Improve Visibility, Security, and Compliance

Watch this short video to see how you can benefit from using Calico Enterprise Audit Logs.

What problems are we solving?

Kubernetes is an API-driven platform. Every action happens through an API call into the kube API server. Consequently, recording and monitoring API activity is very important. While most deployments end up sending these logs to a remote destination for compliance purposes, these logs are often not easily accessible when needed. Moreover, different roles (platform, network, security) have different requirements, and many may not even have access to the logs. Some use cases relevant to log analysis are as follows.

• A policy change resulted in a sudden outage of a service. How do you find out which policies have changed in the last 24 hours? [network, security]
• You are maintaining a critical namespace and want to monitor every pod that comes up in that namespace. Can you get an alert if a pod is created in that Continue reading

How Fortinet and Tigera Protect Kubernetes in the Enterprise

What Problems are We Solving?

Container use continues to grow, and Kubernetes is the most widely adopted container orchestration system, managing nearly half of all container deployments.¹ Successful integration of container services within the enterprise depends heavily on access to external resources such as databases, cloud services, third-party application programming interfaces (APIs), and other applications. All this egress activity must be controlled for security and compliance reasons. In a recent container adoption survey, 61% of correspondents, a super-majority, listed data security as their top challenge.²

Kubernetes Requires a Different Approach to Access Control

Traditional IP-based access control doesn’t work in Kubernetes, where workloads are ephemeral, typically stateless, and use short-term IP addresses. While the Calico Enterprise security management interface provides customized control within the Kubernetes environment, using Calico Enterprise security in isolation from existing enterprise network security leaves organizations with disparate policy-enforcement regimes.

Disparate Network Security Systems Introduce Unwanted Complexity

Maintaining two separate network security systems hinders visibility into routing and connectivity within and between Kubernetes clusters. This complicates the process of troubleshooting issues that span Kubernetes and external environments. Because enterprise monitoring tools lack Kubernetes context, the impact of security policy changes are hard to predict, and Continue reading

How to Efficiently Detect Domain Generation Algorithms (DGA) in Kubernetes with Calico Enterprise

Introduction

2020 is predicted to be an exciting year with more organizations adopting Kubernetes than ever before. As critical workloads with sensitive data migrate to the cloud, we can expect to encounter various Advanced Persistent Threats (APT) targeting that environment.

Domain Generation Algorithm (DGA) – What is It?

DGA is a technique that fuels malware attacks. DGA by itself can’t harm you. But it’s a proven technique that enables modern malware to evade security products and counter-measures. Attackers use DGA so they can quickly switch the command-and-control (also called C2 or C&C) servers that they’re using for malware attacks. Security software vendors act quickly to block and take down malicious domains hard-coded in malware. So, attackers used DGA specifically to counter these actions. Now DGA has become one of the top phone-home mechanisms for malware authors to reach C2 servers. This poses a significant threat to cloud security.

Mitre defines DGA as “The use of algorithms in malware to periodically generate a large number of domain names which function as rendezvous points for malware command and control servers”. Let’s examine this definition more closely. DGA at its core generates domains by concatenating pseudo-random strings and a TLD (e.g. .com, . Continue reading