TeamTNT: Latest TTPs targeting Kubernetes

In April 2020, MalwareHunterTeam found a number of suspicious files in an open directory and posted about them in a series of tweets. Trend Micro later confirmed that these files were part of the first cryptojacking malware by TeamTNT, a cybercrime group that specializes in attacking the cloud—typically using a malicious Docker image—and has proven itself to be both resourceful and creative.

Since this first attack, TeamTNT has continuously evolved its tactics and added capabilities to expand and capture more available cloud attack surfaces. They started with targeting exposed Docker instances and quickly added support for different C2 mechanisms, encryption, DDoS, evasion, persistence and more. Now, their latest variant is targeting the most popular container orchestrator, Kubernetes. Let’s take a closer look.

Evolving Tactics, Techniques and Procedures (TTPs)

TeamTNT’s initial attack targeted an exposed, unprotected Docker API on the internet in order to run an Alpine Linux container. Once the container started running on the unprotected Docker API, a series of scripts were downloaded to facilitate the installation of a Monero cryptominer (to carry out scanning and cleaning activities). A notable script used in the attack was <clean.sh>, which removed a bit of technically advanced Kinsing malware. Kinsing is Continue reading

Honeypods: Applying a Traditional Blue Team Technique to Kubernetes

The use of honeypots in an IT network is a well-known technique to detect bad actors within your network and gain insight into what they are doing. By exposing simulated or intentionally vulnerable applications in your network and monitoring for access, they act as a canary to notify the blue team of the intrusion and stall the attacker’s progress from reaching actual sensitive applications and data. Once the blue team is aware of the situation, the attack can be traced back to the initial vector. The attack can then be contained and removed from the network.

Applying this technique into a Kubernetes environment works exceedingly well because of the declarative nature of applying manifests to deploy workloads. Whether the cluster is standalone or part of a complex pipeline, workload communications are defined by the application’s code. Any communication that’s not defined can be deemed suspicious at minimum and indicate that the source resource may have been compromised. By introducing fake workloads and services around production workloads, when a workload is compromised, the attacker cannot differentiate between other real and fake workloads. The asymmetric knowledge between the attacker and the cluster operator makes it easy to detect lateral movements from compromised Continue reading

Tigera and Microsoft Extend the Power of Calico for Windows to AKS

Tigera, in collaboration with Microsoft, is thrilled to announce the public preview of Calico for Windows on Azure Kubernetes Service (AKS). While Calico has been available for self-managed Kubernetes workloads on Azure since 2018, many organizations are migrating their .NET and Windows workloads to the managed Kubernetes environment offered by AKS. Now the leading open-source Kubernetes network policy and security solution for Kubernetes enables Windows users to fulfill their policy and compliance requirements on Azure Kubernetes Service.

With the availability of Calico for Windows on AKS in public preview, enterprises can leverage the power and simplicity of Calico to enable a single solution that provides uniform Kubernetes network policy and security for their clusters across AKS, other clouds and on-premises, as well as across their choice of Windows, Linux, and mixed-node environments.

Project Calico is the most widely adopted open-source solution for Kubernetes networking and security, used on more than 1 million nodes across 166 countries. However, thousands of our users want to be sure that choosing Calico is the right decision for many years to come. Calico is the only solution that offers a pluggable data plane supporting Windows, standard Linux and eBPF, thus future-proofing their decision Continue reading

Industry-First Pay-as-you-go SaaS Platform for Kubernetes Security and Observability

We are excited to introduce Calico Cloud, a pay-as-you-go SaaS platform for Kubernetes security and observability. With Calico Cloud, users only pay for services consumed and are billed monthly, getting immediate value without upfront investment.

Introduction

Calico Cloud gives DevOps, DevSecOps, and Site Reliability Engineering (SRE) teams a single pane of glass across multi-cluster and multi-cloud Kubernetes environments to deploy a standard set of egress access controls, enforce security policies, ensure compliance, get end-to-end visibility, and troubleshoot applications. Calico Cloud is Kubernetes-native and provides native extensions to enable security and observability as code for easy and consistent enforcement across Kubernetes distributions, multi-cloud and hybrid environments. It scales automatically with the managed clusters according to the user requirements to ensure uninterrupted real-time visibility at any scale.

Security and Observability Challenges

• North-South Controls: Often microservices need to communicate with services or API endpoints running outside the Kubernetes cluster. Implementing access control from Kubernetes pods to external endpoints is hard. Most traditional or cloud provider’s firewalls do not understand the Kubernetes context which forces the ops team to allow traffic from the entire cluster or a set of worker nodes.
• East-West Controls: Even after effective perimeter-based north-south controls, the organizations face challenges to Continue reading

Kubernetes Observability Challenges: The Need for an AI-Driven Solution

Kubernetes provides abstraction and simplicity with a declarative model to program complex deployments. However, this abstraction and simplicity create complexity when debugging microservices in this abstract layer. The following four vectors make it challenging to troubleshoot microservices.

1. The first vector is the Kubernetes microservices architecture, where tens to hundreds of microservices communicate. Debugging such a componentized application is challenging and requires specialized tools.
2. The second vector is the distributed infrastructure spread across heterogeneous on-premises and cloud environments.
3. The third vector of complexity is the dynamic nature of Kubernetes infrastructure. The platform spins up required resources and provides an ephemeral infrastructure environment to scale the application based on demand.
4. Lastly, in such a distributed environment, Kubernetes deployments need fine-grained security and an observability model with defense-in-depth to keep them secure. While modern security controls effectively protect your workloads, they can have unintended consequences by preventing applications from running smoothly and creating an additional layer of complexity when debugging applications.

Today, DevOps and SRE teams must stitch together an enormous amount of data from multiple, disparate systems that monitor infrastructure and services layers in order to troubleshoot Kubernetes microservices issues. Not only is it overwhelming to stitch this data, but troubleshooting using Continue reading

Calico Enterprise: An Overview

As we enter a new year, it’s an appropriate time to reflect on our achievements at Tigera and how much Calico Enterprise has evolved over the past year as the industry’s leading Security and Observability solution for Kubernetes Networking and Microservices. Our experience working with enterprise-class early adopters has helped us to identify the most critical requirements for them to operationalize their Kubernetes deployments and successfully make the challenging transition from pilot to production. These learnings have helped us to shape today’s Calico Enterprise, which is visually represented in this solutions architecture diagram. Let’s dig into this feature-rich layer cake of functionality, from bottom to top!

Calico Enterprise Solutions Architecture

Calico Enterprise is “Kube-native”

But first, there are some important things to keep in mind as we explore. Calico Enterprise is a Kubernetes-native solution – Kube-native – in which everything we do is an extension of Kubernetes primitives. We leverage the full power of Kubernetes by integrating with the Kubernetes API server and creating our own aggregated API server. We use an operator model to access and control custom resources to perform specific functions, like RBAC for example, natively in Kubernetes. Being Kubernetes-native means that as Kubernetes evolves, Calico Enterprise Continue reading

Calico & Calico Enterprise: Now Available as AWS Quick Starts

As an AWS Advanced Technology Partner with AWS Containers Competency, Tigera is thrilled to announce that Calico and Calico Enterprise are both now available as AWS Quick Starts. If you’re unfamiliar with the concept, an AWS Quick Start is a ready-to-use accelerator that fast-tracks deployments of key cloud workloads for AWS customers. Described as “gold-standard deployments in the AWS Cloud”, Quick Starts are designed to reduce hundreds of manual procedures into an automated, workflow-based reference deployment.

With Calico network policy enforcement, you can implement network segmentation and tenant isolation, which is especially useful when you want to create separate environments for development, staging, and production. Calico Enterprise builds on top of open source Calico to provide additional higher-level features and capabilities, and integrates with your existing AWS tools including security groups, Amazon CloudWatch, and AWS Security Hub so you can leverage existing processes and workflows in your EKS or Kubernetes infrastructure.

Everything you need to take advantage of Calico and Calico Enterprise in these Quick Starts is installed and configured in your Amazon Elastic Kubernetes (Amazon EKS) cluster, enabling you to take advantage of a rich set of Kubernetes security, observability, and networking features that Tigera provides in these Continue reading

Calico in 2020: The World’s Most Popular Kubernetes CNI

Calico and Kubernetes go hand-in-hand. Kubernetes is the de facto standard for deploying and managing container-based applications at scale, both on-premises and in the cloud. Calico continues to be the most popular open-source networking and network security solution for Kubernetes. Despite the cataclysmic events that occurred in 2020, the Calico community, supported by the team at Tigera, remained focused and achieved several major successes. We are excited to share these highlights.

Calico By The Numbers…

Since the beginning of 2020, we have experienced a 50% increase in the number of Calico Users. As of this writing, it is estimated that Calico is running on…

• 1,000,000 nodes daily
• 250,000 clusters daily
• In 166 countries around the world

That’s an 85% year-to-year increase in the number of clusters running Calico.

Calico: Clear Winner Among All Tested CNIs

Alex Ducastel published an independent benchmark comparison of Kubernetes CNIs in August which showed that among all of the CNI’s tested, Calico was the clear winner, excelling in nearly every category and delivering superlative results which are summarized in the chart below. In fact, Calico is the CNI of choice in the primary use cases presented by the author in the report’s summary.

The exceptional Continue reading

New Vulnerability Exposes Kubernetes to Man-in-the-Middle Attacks: Here’s How to Mitigate

What is CVE-2020-8554?

A few weeks ago a solution engineer discovered a critical flaw in Kubernetes architecture and design, and announced that a “security issue was discovered with Kubernetes affecting multi-tenant clusters. If a potential attacker can already create or edit services and pods, then they may be able to intercept traffic from other pods (or nodes) in the cluster.” If a hostile user can create a ClusterIP service and set the spec.externalIP field, they can intercept traffic to that IP. In addition, if a user can patch the status of a LoadBalancer service, which is a privileged operation, they can also intercept traffic by exploiting the vulnerability.

Who is Affected?

All Kubernetes versions including the latest release v1.20 are vulnerable to this attack, with the most significant impact being to multi-tenant clusters. Multi-tenant clusters that grant tenants the ability to create and update services and pods are most vulnerable. Since this is a major design flaw with no fix in sight, it becomes imperative to understand and mitigate this CVE.

Technical Overview

The man-in-the-middle (MITM) attack starts with step 1 (shown in the diagram, below). A workload sends a connection request to legitimate IP 4.4. Continue reading

Automated, Simplified DNS Troubleshooting for Kubernetes: Only in Calico Enterprise

The Domain Name System (DNS) is a naming system for computers, services, or other resources connected to the Internet or a private network. DNS translates domain names to the numerical IP addresses needed for locating and identifying computer services and devices. For decades It’s been an essential component of the Internet. It’s an essential part of Kubernetes as well, and is used to determine how workloads connect to Kubernetes services as well as resources outside the cluster.

DNS also happens to be a common source of outages and issues in Kubernetes clusters. When applications are not working as expected, the root cause is often DNS-related. However, debugging and troubleshooting DNS issues in Kubernetes environments is not a trivial task given the limited amount of information Kubernetes provides for DNS queries.

Lacking the necessary visibility into the cluster to correlate a DNS query or reply with a specific workload, for example, you are left in the dark. Without Kubernetes context, you are unable to capture even the most fundamental information needed for troubleshooting, such as the type of DNS query (or reply) or the source of the query.

Figure: The DNS Dashboard from Tigera helps Kubernetes teams more quickly confirm or Continue reading

Tigera to Support Amazon EKS-Distro

Today, we are excited to announce our commitment to support Calico and Calico Enterprise for the Amazon EKS-Distro, a Kubernetes distribution based on and used by Amazon EKS. EKS-D enables you to create reliable and secure Kubernetes clusters using the same versions of Kubernetes and its dependencies deployed by Amazon EKS.
We view EKS-D as further confirmation of the central role that Kubernetes plays in today’s IT infrastructure. We are excited to work with Amazon on this initiative to enable EKS-D users with the same robust enterprise networking and network security functionality that you rely on today to secure your EKS cluster deployments.

Tigera’s commitment to supporting EKS-D highlights our fundamental design principle of “choice”. Our customers can choose to use Calico and Calico Enterprise with their preferred Kubernetes distribution and use the same solution to operate seamlessly across different Kubernetes distributions, including multi-cloud multi-cluster and hybrid environments. Calico Enterprise, for example, allows you to manage multiple Kubernetes clusters to define, apply, and enforce consistent networking and security policy across all your clusters from a single master cluster. Adding EKS-D clusters, which use the same underlying versions of Kubernetes deployed by Amazon EKS, as an option for our customers Continue reading

Calico Delivers “Wow Effect” with 6x Faster Encryption than Any Other Solution… Confirms Leadership in Latest Independent CNI Benchmark Tests

Benchmark tests measure a repeatable set of quantifiable results that serve as a point of reference against which products and services can be compared. Since 2018, Alexis Ducastel, a Kubernetes CKA/CKAD and the founder of InfraBuilder, has been running independent benchmark tests of Kubernetes network plugins (CNI) over a 10Gbit/s network.

The latest benchmark in this periodic series of tests was published in September, and was based on CNI versions that were up-to-date as of August 2020. Only CNIs that can be set up with a single yaml file were tested and compared, and included the following:

• Antrea v.0.9.1
• Calico v3.16
• Canal v3.16 (Flannel network + Calico Network Policies)
• Cilium 1.8.2
• Flannel 0.12.0
• Kube-router latest (2020–08–25)
• WeaveNet 2.7.0

We are thrilled to report that among all of the CNI’s tested, Calico was the clear winner, excelling in nearly every category and delivering superlative results which are summarized in the chart below. In fact, Calico is the CNI of choice in the primary use cases presented by the author in the report’s summary.

The exceptional performance of Calico encryption was described as having the “real wow effect” among all of Continue reading

Introducing Fast, Automated Packet Capture for Kubernetes

If you’re an SRE or on a DevOps team working with Kubernetes and containers, you’ve undoubtedly encountered network connectivity issues with your microservices and workloads. Something is broken and you’re under pressure to fix it, quickly. And so you begin the tedious, manual process of identifying the issue using the observability tools at your disposal…namely metrics and logs. However, there are instances where you may need to go beyond these tools to confirm a potential bug with applications running in your cluster.

Packet capture is a valuable technique for debugging microservices and application interaction in day-to-day operations and incident response. But generating pcap files to diagnose connectivity issues in Kubernetes clusters can be a frustrating exercise in a dynamic environment where hundreds, possibly thousands of pods are continually being created and destroyed.

First, you would need to identify on which node your workload is running, match your workload against its host-based interface, and then (with root access to the node) use tcpdump to generate a file for packet analysis. Then you would need to transfer the pcap files to your laptop and view them in Wireshark. If this doesn’t initially generate the information you need to identify and resolve the Continue reading

Introducing Data-in-Transit Encryption for Calico Enterprise

We’re excited to announce that Calico Enterprise, the leading solution for Kubernetes networking, security and observability in hybrid and multi-cloud environments, now includes encryption for data-in-transit.

Calico Enterprise is known for its rich set of network security implementations to protect container workloads by restricting traffic to and from trusted sources. These include, but are not limited to, implementing existing enterprise security controls in Kubernetes, managing egress access using DNS policy, extending firewalls to Kubernetes, and intrusion detection and threat defense. As the Kubernetes footprint expands, however, we’ve seen demand for an even greater in-depth approach to protecting sensitive data that falls under regulatory compliance mandates.

Not all threats originate from outside an organization. According to Gartner, nearly 75% of breaches happen due to insider behavior, from people within the organization such as employees, former employees, contractors or business associates, who have inside information concerning the organization’s security practices, data and computer systems. This level of exposure is unacceptable for organizations that have strict data protection and regulatory compliance requirements. No matter where a threat originates, encrypted data is unreadable to anyone except the legitimate keyholder, thus protecting the data should a breach occur.

Several regulatory standards Continue reading

Solving Microservices Connectivity Issues with Network Logs

The network is foundational to distributed application environments. A distributed application has multiple microservices, each running in a set of pods often located on different nodes. Problem areas in a distributed application can be in network layer connectivity (think network flow logs), or application resources unavailability (think metrics), or component unavailability (think tracing). Network layer connectivity can be impacted by various factors such as routing configuration, IP pool configuration, network policies, etc. When service A cannot talk to service B over the network, or an external application cannot connect to service A, network logs become an essential source of historical data needed for troubleshooting connectivity issues. Just like in a traditional network, network logs enable cluster administrators to monitor the Kubernetes microservices network.

Network Logs Can Address Multiple Use Cases

Network logs can be used to serve the unique requirements of different teams (DevOps, SecOps, Platform, Network). The value of Kubernetes network logs resides in the information collected, such as detailed context about endpoints (e.g., pods, labels, namespaces) and the network policies deployed in configuring the connection. Within the IT estate, DevOps, SecOps, Network and Platform teams can use network logs to address use cases that Continue reading

Extend Your Fortinet FortiManager to Kubernetes

Companies are leveraging the power of Kubernetes to accelerate the delivery of resilient and scalable applications to meet the pace of business. These applications are highly dynamic, making it operationally challenging to securely connect to databases or other resources protected behind firewalls.

Visibility into Kubernetes Infrastructure is Essential

Lack of visibility has compliance implications. Like any on-premises or cloud-based networked services, Kubernetes production containers must address both organizational and regulatory security requirements. If compliance teams can’t trace the history of incidents across the entire infrastructure, they can’t adequately satisfy their audit requirements. To enable the successful transition of Kubernetes pilot projects to enterprise-wide application rollouts, companies must be able to extend their existing enterprise security architecture into the Kubernetes environment.

In response, Fortinet and Tigera jointly developed a suite of Calico Enterprise solutions for the Fortinet Security Fabric that deliver both north-south and east-west visibility and help ensure consistent control, security, and compliance. Key among these integrations is the FortiManager Calico Kubernetes Controller, which enables Kubernetes cluster management from the FortiManager centralized management platform in the Fortinet Fabric Management Center.

View and Control the Kubernetes Environment with FortiManager

The FortiManager Calico Kubernetes Controller translates FortiManager policies into granular Kubernetes network Continue reading

Kubernetes Q3-2020: Threats, Exploits and TTPs

Kubernetes has become the world’s most popular container orchestration system and is taking the enterprise ecosystem by storm. At this disruptive moment it’s useful to look back and review the security threats that have evolved in this dynamic landscape. Identifying these threats and exploits and being a proactive learner may save you a lot of time and effort…as well as help you retain your reputation in the long run. In this blog we’ll look at some critical security issues faced by the Kubernetes ecosystem in the recent past, and examine the top tactics, techniques and procedures (TTPs) used by attackers.

Major Vulnerabilities

Everyday, new Kubernetes ecosystem Common Vulnerabilities and Exposures (CVEs) are published. Let’s take a closer look at some of the cloud shakers…

CVE-2020-14386: Using privilege escalation vulnerability to escape the pod
A flaw was found in the Linux kernel before 5.9-rc4. Memory corruption can be exploited to gain root privileges from unprivileged processes.

We received notification that some instances in our cloud infrastructure are vulnerable to this CVE. When we took a closer look, it appeared to be a typical privilege escalation vulnerability using AF sockets on hosts. Unprivileged users with CAP_NET_RAW permissions can send packets Continue reading

Tigera Announces Open-Source Calico for Windows and Collaboration with Microsoft

Tigera is pleased to announce that we have open-sourced Calico for Windows and made it immediately available for all to use for free. With the launch of open-source Calico for Windows, the vast ecosystem of Windows users now has unprecedented access to Kubernetes via the industry’s de-facto standard for Kubernetes networking and network security.

We have been collaborating with Microsoft and our joint customers over the past few years to bring Project Calico to the Windows platform, and have seen increasing demand for Windows nodes ever since the release of Kubernetes 1.14.  Most enterprises have a Windows footprint, and Windows workloads are increasingly being modernized and migrated to containers and orchestrated with Kubernetes. Enterprise users want to deploy a single solution for network security that works across both Linux and Windows workloads. Open-sourcing Calico for Windows provides those users with the best and only solution available, and for free.

“We are seeing an influx in interest in Windows Kubernetes workloads, as well as interest in securing those workloads. Calico has been a key means of deploying network security policies across both Windows and Linux platforms, however, their Windows support has been commercially licensed by Tigera until today,“ said Continue reading

Announcing eBPF Mode GA

A few days ago, our team released Calico v3.16. As part of that release, we have marked the eBPF dataplane as “GA”, signalling that it is now stable and ready for wider use by the community. In this blog post I want to take you through the process of moving from tech-preview to GA. If you’re not already familiar with eBPF and the benefits of the Calico eBPF dataplane, or if you want to see throughput and latency graphs compared to the standard Linux dataplane, I recommend that you read our introductory blog post. To recap, when compared with the standard Linux dataplane (based on iptables), the eBPF dataplane:

• Scales to higher throughput, using less CPU per GBit
• Natively supports Kubernetes services (without kube-proxy) in a way that:
• Reduces latency
• Preserves external client source IP addresses
• Supports DSR (Direct Server Return) for reduced latency (and CPU usage)
• Uses less CPU than kube-proxy to keep the dataplane in sync

For the tech preview release, our focus was on covering a broad set of features and proving out the performance of the new dataplane. However to meet the bar for GA, we had to:

• Address our list of known missing features
• Continue reading

Kubernetes Pod Networking on AWS: Getting There from Here

Thinking about running Kubernetes on AWS? To optimize your chances of success, you’ll need to have a solid understanding of Kubernetes pod networking. As applications grow to span multiple containers deployed across multiple clusters, operating them becomes more complex. Containers are grouped into pods, and those pods can be networked and scaled to meet your specific needs.

Kubernetes provides an open source API to manage this complexity, but one size doesn’t fit all. So you’ll want to get a handle on the different methods available to support your project. Then when you’re ready to move forward, you’ll have a much clearer idea of what will work best for you. If this sounds challenging, not to worry. Our short video explains Kubernetes pod networking on AWS and can answer many of the questions you may have. We’ve also included some great examples to help guide you.

Want to learn more about Calico Enterprise? Check out these resources.

————————————————-

Free Online Training
Access Live and On-Demand Kubernetes Training

Calico Enterprise – Free Trial
Network Security, Monitoring, and Troubleshooting
for Microservices Running on Kubernetes

The post Kubernetes Pod Networking on AWS: Getting There from Here appeared first on Tigera.