Why use Typha in your Calico Kubernetes Deployments?

Calico is an open source networking and network security solution for containers, virtual machines, and native host-based workloads. Calico supports a broad range of platforms including Kubernetes, OpenShift, Docker EE, OpenStack, and bare metal. In this blog, we will focus on Kubernetes pod networking and network security using Calico.

Calico uses etcd as the back-end datastore. When you run Calico on Kubernetes, you can use the same etcd datastore through the Kubernetes API server. This is called a Kubernetes backed datastore (KDD) in Calico. The following diagram shows a block-level architecture of Calico.

Calico-node runs as a Daemonset, and has a fair amount of interaction with the Kubernetes API server. It’s easy for you to profile that by simply enabling audit logs for calico-node. For example, in my kubeadm cluster, I used the following audit configuration

April Customer Newsletter

Welcome to the April 2020 edition of the Tigera Calicommunication newsletter! In the March edition, we discussed context-aware flow logs. This edition covers the next component of logging, the audit logs.

Using Calico Enterprise Audit Logs to Improve Visibility, Security, and Compliance

Watch this short video to see how you can benefit from using Calico Enterprise Audit Logs.

What problems are we solving?

Kubernetes is an API-driven platform. Every action happens through an API call into the kube API server. Consequently, recording and monitoring API activity is very important. While most deployments end up sending these logs to a remote destination for compliance purposes, these logs are often not easily accessible when needed. Moreover, different roles (platform, network, security) have different requirements, and many may not even have access to the logs. Some use cases relevant to log analysis are as follows.

• A policy change resulted in a sudden outage of a service. How do you find out which policies have changed in the last 24 hours? [network, security]
• You are maintaining a critical namespace and want to monitor every pod that comes up in that namespace. Can you get an alert if a pod is created in that Continue reading

How Fortinet and Tigera Protect Kubernetes in the Enterprise

What Problems are We Solving?

Container use continues to grow, and Kubernetes is the most widely adopted container orchestration system, managing nearly half of all container deployments.¹ Successful integration of container services within the enterprise depends heavily on access to external resources such as databases, cloud services, third-party application programming interfaces (APIs), and other applications. All this egress activity must be controlled for security and compliance reasons. In a recent container adoption survey, 61% of correspondents, a super-majority, listed data security as their top challenge.²

Kubernetes Requires a Different Approach to Access Control

Traditional IP-based access control doesn’t work in Kubernetes, where workloads are ephemeral, typically stateless, and use short-term IP addresses. While the Calico Enterprise security management interface provides customized control within the Kubernetes environment, using Calico Enterprise security in isolation from existing enterprise network security leaves organizations with disparate policy-enforcement regimes.

Disparate Network Security Systems Introduce Unwanted Complexity

Maintaining two separate network security systems hinders visibility into routing and connectivity within and between Kubernetes clusters. This complicates the process of troubleshooting issues that span Kubernetes and external environments. Because enterprise monitoring tools lack Kubernetes context, the impact of security policy changes are hard to predict, and Continue reading

How to Efficiently Detect Domain Generation Algorithms (DGA) in Kubernetes with Calico Enterprise

Introduction

2020 is predicted to be an exciting year with more organizations adopting Kubernetes than ever before. As critical workloads with sensitive data migrate to the cloud, we can expect to encounter various Advanced Persistent Threats (APT) targeting that environment.

Domain Generation Algorithm (DGA) – What is It?

DGA is a technique that fuels malware attacks. DGA by itself can’t harm you. But it’s a proven technique that enables modern malware to evade security products and counter-measures. Attackers use DGA so they can quickly switch the command-and-control (also called C2 or C&C) servers that they’re using for malware attacks. Security software vendors act quickly to block and take down malicious domains hard-coded in malware. So, attackers used DGA specifically to counter these actions. Now DGA has become one of the top phone-home mechanisms for malware authors to reach C2 servers. This poses a significant threat to cloud security.

Mitre defines DGA as “The use of algorithms in malware to periodically generate a large number of domain names which function as rendezvous points for malware command and control servers”. Let’s examine this definition more closely. DGA at its core generates domains by concatenating pseudo-random strings and a TLD (e.g. .com, . Continue reading

Now Available: Calico for Windows on Red Hat OpenShift Container Platform

Approximately one year ago, Kubernetes 1.14 made support of Windows containers running on Microsoft Windows Server nodes generally available. This was a declaration that Windows node support was stable, well-tested, and ready for adoption, meaning the vast ecosystem of Windows-based applications could be deployed on the platform.

Collaborating with Microsoft, Tigera leveraged the new Windows platform capabilities to create Calico for Windows, the industry’s first cross-platform Kubernetes solution to manage networking and network policy for Kubernetes deployments on Windows and Linux.

We are excited to announce that Calico for Windows now supports the latest Windows Dev Preview on the Red Hat OpenShift Container Platform (OCP). Built on Red Hat Enterprise Linux and Kubernetes, OCP v4 provides developers and IT organizations with a hybrid and multi-cloud application platform for deploying both new and existing applications on scalable resources, with minimal configuration and management overhead. OCP enables organizations to meet security, privacy, compliance, and governance requirements.

Calico for Windows is the only Kubernetes networking solution for teams using Windows on OpenShift. The combination of Calico for Windows and Red Hat OpenShift Container Platform represents a major leap forward in productivity for organizations that are deploying Windows on Kubernetes. DevOps teams Continue reading

Extend Fortinet FortiGate to Kubernetes with Calico Enterprise 2.7

We are excited to announce the general availability of Calico Enterprise 2.7. With this release, Fortinet’s 400,000 customers can use FortiGate to enforce network security policies into and out of the Kubernetes cluster as well as traffic between pods within the cluster.

• Kubernetes workloads populate the Fortigate GUI
• The network team can then create and enforce policies in Fortigate and have them enforced as Calico Policy
• Saves time and money and lets the network team retain the firewall responsibility (which also frees up time for ITOps)

We have also added many new exciting capabilities that help platform engineers blow through barriers blocking their path to production, and advanced cybersecurity capabilities for those already running production workloads.

• Manage Network Security Across Multiple Kubernetes Clusters
• Enforce a Common Set of Security Controls Across Multiple Clusters
• Detect and Alert on Unauthorized Changes and Other Attack Vectors
• Self-Service Troubleshooting for End Users
• Detect and Prevent Malicious Data Exfiltration

Manage Network Security Across Multiple Kubernetes Clusters

As the adoption of Kubernetes continues to accelerate, our customers are seeing the number of clusters in their environments rapidly multiplying. This has created a management challenge for IT Ops teams who are constantly pushed to find ways Continue reading

Supercharging Workload Security in Your K8s Cluster

Introduction

2019 was a big year for Kubernetes adoption, and 2020 is sure to exceed that pace. Already, we have seen a large number of organizations migrating their workloads to Kubernetes (k8s) both in public and private clouds as they embrace a hybrid cloud strategy. With so much at stake, what are you currently using for network security inside your k8s cluster?

Quick Retro

Let’s take a step back to a time when you were deploying applications to VMs in AWS, GCP or Azure (in the case of public clouds) or vSphere, etc. in private clouds. One of the most important tasks before provisioning infrastructure and deploying applications was to chalk out firewall considerations. These requirements were fulfilled using security group rules in the case of AWS or firewall rules in GCP. We all understand their importance. But doing the same involving Kubernetes was extremely challenging. Today, we can solve those problems for you with just a few clicks.

Present Scenario – What If?

Most recently with the increase in k8s adoption we have seen operations and platform teams hustling to implement a plethora of monitoring tools, logging backends and CI/CD tools. While all of this is Continue reading

Decentralized Calico Network Security Policy Deployment for GitOps – Part 2

In part 1 of the GitOps blog series, we discussed the value of using GitOps for Calico policies, and how to roll out such a framework. In this second part of the series, we will expand the scope to include decentralized deployment and GitOps.

We see different personas among our customers deploying three types of controls:

1. Cluster hardening policies are enforced and controlled by the platform admin
2. Organizational security controls are enforced and controlled by the security admin
3. Each application team may have their own unique requirements. This is controlled by the DevOps admin for the specific application.

This is different from the traditional firewall world, where the security admin is responsible for managing security policies, and the change management window could be several weeks in duration. Adopting that model in Kubernetes is simply counter to the very principles of enabling the developers. So how can we make policy creation and enforcement simple, yet adhere to organizational processes? The answer lies in simple tooling, GitOps and governance.

Policies have business logic that must be implemented in YAML. The business logic (allow access for service A to service B, open port 443 inbound on service B, permit access to slack webhook Continue reading

Decentralized Calico Network Security Policy Deployment for GitOps – Part 2

In part 1 of the GitOps blog series, we discussed the value of using GitOps for Calico policies, and how to roll out such a framework. In this second part of the series, we will expand the scope to include decentralized deployment and GitOps.

We see different personas among our customers deploying three types of controls:

1. Cluster hardening policies are enforced and controlled by the platform admin
2. Organizational security controls are enforced and controlled by the security admin
3. Each application team may have their own unique requirements. This is controlled by the DevOps admin for the specific application.

This is different from the traditional firewall world, where the security admin is responsible for managing security policies, and the change management window could be several weeks in duration. Adopting that model in Kubernetes is simply counter to the very principles of enabling the developers. So how can we make policy creation and enforcement simple, yet adhere to organizational processes? The answer lies in simple tooling, GitOps and governance.

Policies have business logic that must be implemented in YAML. The business logic (allow access for service A to service B, open port 443 inbound on service B, permit access to slack webhook Continue reading

Enforcing Network Security Policies with GitOps – Part 1

“How do I enable GitOps for my network security policies?” This is a common question we hear from security teams. Getting started with Kubernetes is relatively simple, but moving production workloads to Kubernetes requires alignment from all stakeholders – developers, platform engineering, network engineering, and security.

Most security teams already have a high-level security blueprint for their data centers. The challenge is in implementing that in the context of a Kubernetes cluster and workload security. Network policy is a key element of Kubernetes security. Network policy is expressed as a YAML configuration and works very well with GitOps.

We will do a three-part blog series covering GitOps for network security policies. In part one (this part), we cover the overview and getting started with a working example tutorial. In part two, we will extend the tutorial to cover an enterprise-wide decentralized security architecture. In the final part, we will delve into policy assurance with examples.

Note that all policies in Calico Enterprise (network security policy, RBAC, threat detection, logging configuration, etc.) are enforced as YAML configuration files, and can be enforced via a GitOps practice.

By adopting GitOps, security teams benefit in the following ways:

• Take your policies Continue reading

Five Ways to Quickly Uncover Malicious Activity and Protect Your Kubernetes Workloads

Organizations are rapidly moving more and more mission-critical applications to Kubernetes (K8s) and the cloud to reduce costs, achieve faster deployment times, and improve operational efficiencies, but are struggling to achieve a strong security posture because of their inability to apply conventional security practices in the cloud environment. Commitment to cloud security grows, but security safeguards are not keeping up with the increased use of the various cloud platforms. Regardless of the cloud provider or service model, individual organizations are ultimately responsible for the security of their data.

According to a 2019 Ponemon Institute Global Cloud Data Security Study, 70 percent of respondents find it more complex to manage privacy and data protection regulations in a cloud environment than on-premises. Meanwhile, the percent of corporate data stored in the cloud environment has grown from an average of 30 percent in 2015 to an average of 48 percent in 2019. In the same study, 56 percent of respondents say the use of cloud resources increases compliance risk.

The downside associated with a security breach is severe for any organization, but especially so for companies in regulated environments like financial services, healthcare and telecommunications. Now there’s a new and highly effective way Continue reading

Security Policy as Code Now Fully Automated with Calico Enterprise 2.6

We are excited to announce the general availability of Calico Enterprise 2.6 (formerly known as Tigera Secure). With this release, it is now possible to fully-automate Security-Policy-as-Code within a CI-CD pipeline, including the ability to implement security as a Canary rollout, which is the most critical requirement to automating network security.

DevOps is now mainstream and practiced in nearly every major enterprise; it has transitioned from what was a competitive differentiator a few years ago to the industry standard today.

DevOps relies on automation to continuously optimize the cycle time from code to production. DevOps automation manifests itself in 2 forms.

1. Automation of the underlying infrastructure (infrastructure as code)
2. Automation of the software delivery process (Continuous Integration and Continuous Delivery (CICD) pipeline)

Security has become an integral part of the DevOps team’s responsibilities. A quick sample of DevOps jobs on LinkedIn is a quick example; nearly every DevOps job posting has “security” as a required responsibility. It’s no longer enough to automate the infrastructure, it is now necessary to implement security within the delivery pipeline and perhaps link SW CI-CD pipelines with the corresponding security policies that they should be deployed with. DevOps teams have struggled to automate this Continue reading

Tigera Joins the Fortinet Fabric-Ready Program and Partners with Fortinet to Secure Kubernetes Environments

We are proud to partner with Fortinet and join their Fabric-Ready Technology Alliance Partner program. With this partnership, Fortinet customers will be able to extend their network security architecture to their Kubernetes environments.

Our partnership was driven from interest from Fortinet’s customers to protect their Kubernetes based infrastructure. Kubernetes adoption is growing like wildfire and nearly every enterprise on the planet is at some stage of their Kubernetes journey.

The Tigera and Fortinet joint solution will support all cloud-based and on-premises Kubernetes environments. With this architecture, Tigera Secure will map security policies from FortiManager into each Kubernetes cluster in the cloud or on-premises. The joint solution will enable Fortinet customers to enforce network security policies for traffic into and out of the Kubernetes cluster (North/South traffic) as well as traffic between pods within the cluster (East/West traffic).

Tigera Secure will also integrate with threat feeds from FortiGuard to detect and block any malicious activity inside the clusters. Tigera will monitor the cluster traffic and send these events to FortiSIEM, enabling the security operations team to quickly diagnose the situation.

If you are attending Microsoft Ignite join us at our respective booths to learn more about our solution (Fortinet Booth #519 Continue reading

Enable GitOps for Kubernetes Security – Part 1

“How do I enable GitOps for my network policies?”

That is a common question we hear from security teams. Getting started with Kubernetes is relatively simple, but moving production workloads to Kubernetes requires alignment from all stakeholders – developers, platform engineering, network engineering, security.

Most security teams already have a high-level security blueprint for their data centers. The challenge is in implementing that in the context of a Kubernetes cluster and workload security. Network policy is a key element of Kubernetes security. Network policy is expressed as an YAML configuration, and works very well with GitOps.

We will do a 3 part blog series covering GitOps for network policies. In part 1 (this part), we cover the overview and getting started with a working example tutorial. In part 2, we will extend the tutorial to cover an enterprise-wide decentralized security architecture. In the final part, we will delve into policy assurance with examples. Note that all policies in Tigera Secure (network policy, RBAC, Threat detection, Logging configuration, etc.) are enforced as YAML configuration files, and can be enforced via a GitOps practice.

By adopting GitOps, security teams benefit as follows.

• Take your policies with you. Kubernetes cluster creation Continue reading

3 Layers to Defend Your Kubernetes Workloads

Researchers at Netflix and Google recently reported a vulnerability in the HTTP/2 protocol that enables adversaries to execute a DOS attack by legitimate use of the protocol. These types of attacks are very difficult to detect and mitigate because the traffic is valid HTTP/2 traffic. While HTTP/2 is a relatively new protocol it should be noted that even after several years of hardening we still see vulnerabilities for the TCP protocol like the recently reported SACK vulnerability.

Vulnerability Scanning and Patching

So how do we ensure that Kubernetes workloads are protected from these types of vulnerabilities? 

Security researchers work to identify new vulnerabilities and then help developers develop security patches. You can apply those patches to keep your software secure from the lastest known vulnerabilities.

The simple answer then is to scan workload images and patch your software and update your software to use the latest patches. However, that approach essentially means you have to wait for the next attack and then will need to repeat the cycle. While this works, it is not sufficient and quite disruptive to implement as we play into the hands of the adversaries where they are working on the next vulnerability while Continue reading

Single Sign-On for Kubernetes: Dashboard Experience

Over my last two posts (part 1 and part 2), I have investigated user authentication in Kubernetes and how to create a single sign-on experience within the Kubernetes ecosystem. So far I have explained how Open ID Connect (OIDC) works, how to get started with OIDC and how to perform a login from the command line.

The final piece of this puzzle is the Kubernetes dashboard, often used by our engineers alongside kubectl. To complete our move to SSO, we wanted to ensure that, when using the Dashboard, our engineers logged in to the same account they used for kubectl.

Since Kubernetes version 1.7.0, the dashboard has had a login page. It allows users to upload a kubeconfig file or enter a bearer token. If you have already logged into the command line, this allows you to copy the OIDC id-token from your kubeconfig file into the bearer token field and login. There are, however, a couple of problems with this:

• The login page has a skip button — If you aren’t using any authorization (RBAC) then this would permit anyone to access the dashboard with effective admin rights.
• Copy and pasting a token from a Continue reading

IBM’s journey to tens of thousands of production Kubernetes clusters

IBM Cloud has made a massive shift to Kubernetes. From an initial plan for a hosted Kubernetes public cloud offering it has snowballed to tens of thousands of production Kubernetes clusters running across more than 60 data centers around the globe, hosting 90% of the PaaS and SaaS services offered by IBM Cloud.

I spoke with Dan Berg, IBM Distinguished Engineer, to find out more about their journey, what triggered such a significant shift, and what they learned along the way.

So take me back to the beginning – how did this thing get started?

It must have been 3 years ago.  At that time, IBM had been building customer-facing services in the cloud for several years implemented as traditional VM or Cloud Foundry workloads. One of the services we offered was a container as a service (CaaS) platform that was built under the covers using Docker and OpenStack.  With this platform, we were the first in the industry to provide a true CaaS experience, but in some ways it was ahead of its time, and people weren’t ready to embrace a pure container as a service platform.

To this day we are still seeing double digit Continue reading

Istio Routing Basics

When learning a new technology like Istio, it’s always a good idea to take a look at sample apps. Istio repo has a few sample apps but they fall short in various ways. BookInfo is covered in the docs and it is a good first step. However, it is too verbose with too many services for me and the docs seem to focus on managing the BookInfo app, rather than building it from ground up. There’s a smaller helloworld sample but it’s more about autoscaling than anything else.

In this post, I’d like to get to the basics and show you what it takes to build an Istio enabled ‘HelloWorld’ app from ground up. One thing to keep in mind is that Istio only manages the traffic of your app. The app lifecycle is managed by the underlying platform, Kubernetes in this case. Therefore, you need to understand containers and Kubernetes basics and you need to know about Istio Routing primitives such as Gateway, VirtualService, DestinationRuleupfront. I’m assuming that most people know containers and Kubernetes basics at this point. I will focus on Istio Routing instead in this post.

Basic Steps

These are roughly the steps Continue reading

Prevent DNS (and other) spoofing with Calico

AquaSec’s Daniel Sagi recently authored a blog post about DNS spoofing in Kubernetes. TLDR is that if you use default networking in Kubernetes you might be vulnerable to ARP spoofing which can allow pods to spoof (impersonate) the IP addresses of other pods. Since so much traffic is dialed via domain names rather than IPs, spoofing DNS can allow you to redirect lots of traffic inside the cluster for nefarious purposes.

So this is bad, right? Fortunately, Calico already prevents ARP spoofing out of the box. Furthermore, Calico’s design prevents other classes of spoofing attacks. In this post we’ll discuss how Calico keeps you safe from IP address spoofing, and how to go above and beyond for extra security.

ARP Spoofing

ARP spoofing is an attack that allows a malicious pod or network endpoint to receive IP traffic that isn’t meant for it. Sagi’s post already describes this well, so I won’t repeat the details here. An important thing to note, however, is that ARP spoofing only works if the malicious entity and the target share the same layer 2 segment (e.g. have direct Ethernet connectivity). In Calico, the network is fully routed at layer 3, meaning that Continue reading

Tigera Secure 2.5 – Implement Kubernetes Network Security Using Your Firewall Manager

We are excited to announce the general availability of Tigera Secure 2.5. With this release, security teams can now create and enforce security controls for Kubernetes using their existing firewall manager.

Containers and Kubernetes adoption are gaining momentum in enterprise organizations. Gartner estimates that 30% of organizations are running containerized applications today, and they expect that number to grow to 75% by 2022. That’s tremendous growth considering the size and complexity of enterprise IT organizations. It’s difficult to put exact metrics on the growth in Kubernetes adoption; however, KubeCon North America attendance is a good proxy. KubeCon NA registrations grew from 1,139 in 2016 to over 8,000 in 2018 and are expected to surpass 12,000 this December, and the distribution of Corporate Registrations has increased dramatically.

Despite this growth, Kubernetes is a tiny percentage of the overall estate the security team needs to manage; sometimes less than 1% of total workloads. Security teams are stretched thin and understaffed, so it’s no surprise that they don’t have time to learn the nuances of Kubernetes and rethink their security architecture, workflow, and tools for just a handful of applications. That leads to stalled deployments and considerable friction between the application, infrastructure, Continue reading