Creative foot-shooting with Go RWMutex

Hi, I'm Filippo and today I managed to surprise myself! (And not in a good way.)

I'm developing a new module ("filter" as we call them) for RRDNS, CloudFlare's Go DNS server. It's a rewrite of the authoritative module, the one that adds the IP addresses to DNS answers.

It has a table of CloudFlare IPs that looks like this:

type IPMap struct {  
    sync.RWMutex
    M map[string][]net.IP
}

It's a global filter attribute:

type V2Filter struct {  
    name       string
    IPTable    *IPMap
    // [...]
}

Mexican Standoff CC-BY-NC-ND image by Martin SoulStealer

The table changes often, so a background goroutine periodically reloads it from our distributed key-value store, acquires the lock (f.IPTable.Lock()), updates it and releases the lock (f.IPTable.Unlock()). This happens every 5 minutes.

Everything worked in tests, including multiple and concurrent requests.

Today we deployed to an off-production test machine and everything worked. For a few minutes. Then RRDNS stopped answering queries for the beta domains served by the new code.

What. That worked on my laptop™.

Here's the IPTable consumer function. You can probably spot the bug.

func (f *V2Filter) getCFAddr(...) (result []dns.RR) {  
    f. Continue reading

IDG Contributor Network: Users fail to identify phishing attacks, study says

Computer users don't spend enough time looking for phishing indicators, says a new study based on tracking eye movement and brain activity.Users fail "at detecting phishing attacks even when they are mentally engaged in the task and subconsciously processing real sites differently from fake sites," Nitesh Saxena, one of the University of Alabama at Birmingham scientists involved in the study, said in an article on the university's website. The scientists want to find a way to track subconscious detection of phishing and get users to recognize attacks consciously.To read this article in full or to leave a comment, please click here

IDG Contributor Network: 4 considerations for minimizing (or eliminating) your mean time to innocence

Your users are complaining: some system is down or slow. You need to determine if the problem is under your control or if the fault lies with a third party, such as your ISP or a SaaS provider. The time it takes to figure that out is your MTTI: "Mean Time to Innocence."At the recent O'Reilly Velocity show in New York City, my colleague, Phil Stanhope, talked about this topic. He pointed out a few important reasons why determining MTTI is so much more complex now than it was 10+ years ago. The Internet is increasingly complex and routinely experiences outages, instabilities, and attacks. While cloud providers, CDNs, and acceleration services may claim to be "always up," that doesn't mean that they're "always reachable." In fact, they are almost certainly experiencing a constant rate of low-level failure that is largely outside IT's control and is still affecting users. Therefore, getting to MTTI is harder than ever.To read this article in full or to leave a comment, please click here

10 enterprise IT firsts

O PioneersImage by Baker County Tourism/FlickrWe think of IT as an essential corporate function today, driven by desire for profits. But computers largely emerged out of government- and university-funded research, much of it initially driven in the 1940s by the effort to win World War II -- in Britain, to break Nazi codes, and in the U.S., to produce artillery firing tables.To read this article in full or to leave a comment, please click here

Google Fiber eyed for three more cities: Oklahoma City, Jacksonville and Tampa

Google's crusade to light up U.S. cities with gigabit speed Google Fiber has expanded to potentially include Oklahoma City and Jacksonville and Tampa, Florida.In a blog posted Wednesday, Google said it invited the capital city of Oklahoma and the two Florida cities to "explore" bringing faster fiber to their communities.In September, Google also invited San Diego and Irvine, Calif., and Louisville, Ky., to explore bringing Google Fiber to their communities.+ ALSO ON NETWORK WORLD How Google Fiber is disrupting the broadband deployment model +To read this article in full or to leave a comment, please click here

Hackers infect MySQL servers with malware for DDoS attacks

Hackers are exploiting SQL injection flaws to infect MySQL database servers with a malware program that's used to launch distributed denial-of-service (DDoS) attacks.Security researchers from Symantec found MySQL servers in different countries infected with a malware program dubbed Chikdos that has variants for both Windows and Linux.This Trojan is not new and was first documented in 2013 by incident responders from the Polish Computer Emergency Response Team (CERT.PL). At that time the malware was being installed on servers after using brute-force dictionary attacks to guess SSH (Secure Shell) login credentials.However, the new attacks observed by Symantec abuse the user-defined function (UDF) capability of the MySQL database engine. UDF allows developers to extend the functionality of MySQL with compiled code.To read this article in full or to leave a comment, please click here

Stop CISA!

I've been following cybersecurity legislation for a number of years, including all the proceedings with the Cybersecurity Information Sharing Act (CISA). After much deliberation, I believe that CISA remains fundamentally flawed and needs a lot more work before it becomes the law of the land. To be clear, I understand and support the basic objective CISA seeks to promote. Real-time threat intelligence sharing and analysis could help public and private sector organizations proactively react to emerging cyber-threats, mitigating risk and/or minimizing the potential damages associated with devastating data breaches (i.e. Anthem, OMB, Sony Pictures, Target, etc.).To read this article in full or to leave a comment, please click here

How CISA encourages both cybersecurity information sharing and warrantless surveillance

Sharing cybersecurity data involving threats, probes, breaches, and information on attackers between companies and government agencies is a great idea. However, although shared data will strengthen the cybersecurity defenses, the Computer and Communications Industry Association (CCIA), backed by Amazon, Facebook, and Google, among others, and the Business Software Alliance (BSA), which is backed by Apple, Microsoft, and Oracle, are both against it.Smart companies are already doing something similar. At the RSA Security Conference three years ago (a century ago in cybersecurity time) Zion Bankcorp's data scientists explained how the bank went from reacting to law enforcement warnings of cyberthreats to becoming proactive, frequently reporting threats to law enforcement, who subsequently relayed official warnings to other organizations.To read this article in full or to leave a comment, please click here

The best new features in Windows Server 2016 (so far)

Microsoft continues to put out Technical Previews of Windows Server 2016 with as-yet-unseen features. The latest Technical Preview, TP3, introduced a number of new features, such as Docker integration and Windows Server Containers, along with improvements to features introduced in previous previews. It also brings new security capabilities such as Shielded VMs, which protect virtual machine contents in a multitenant environment.The TP2 release debuted Nano Server and a number of Hyper-V, networking, and storage features. It introduced a new Windows Server role named Host Guardian Service, which flags trusted Hyper-V hosts, and included a Windows Server Antimalware feature not found in previous previews. Forthcoming Technical Previews will bring more new features, notably Hyper-V Containers.To read this article in full or to leave a comment, please click here

Google threatens action against Symantec-issued certificates following botched investigation

Google wants Symantec to disclose all certificates issued by its SSL business going forward, after what Google considers a botched investigation into how Symantec employees issued SSL certificates for domain names that the company did not own. The browser maker also wants the security firm to publish a detailed analysis of how the incident was investigated. Through its acquisition of Verisign's authentication business unit in 2010, Symantec became one of the largest certificate authorities (CAs) in the world. Such organizations are trusted by browsers and operating systems to issue digital certificates to domain owners which are then used to encrypt online communications.To read this article in full or to leave a comment, please click here

Risky Business #387 — Hack people to death!

In this week's feature interview we're chatting with Chris Rock from Kustodian. Chris did a great presentation at Ruxcon last week about how easy it is to hack people to death!

He's found out just how easy it is to register births and deaths in the united states and Australia via online systems. He says it's a problem that could result in a virtual baby harvest for fraudsters who plan ahead. It's really fun stuff, that's this week's feature.

read more

An introduction to network namespaces

Network namespaces allow you to provide unique views of the network to different processes running on a Linux host.  If you’re coming from a traditional networking background, the closest relative to network namespaces would be VRF (Virtual Routing and Forwarding) instances.  In both cases the constructs allow us to provide a different network experience to different processes or interfaces.  For the sake of starting the conversation, let’s quickly look at an example of both VRFs and network namespaces so you get an idea of how they work.

The easiest scenario to illustrate either of these technologies is out of band management.  Take for instance this very simple network diagram…

image     
Note: I’m being purposefully vague here about the network layout and addressing.  Bear with me for a moment while I get to the point. 

As you can see, we have two users that live on the same segment (forgive me for not drawing an Ethernet segment connecting the two).  Let’s assume that the user on the left has to traverse northbound to get to resources that hang off the top network cloud.  Let’s also assume the user on the right has to Continue reading

Active Route Manager

SDN Active Route Manager has been released on GitHub, https://github.com/sflow-rt/active-routes. The software is based on the article White box Internet router PoC. Active Route Manager peers with a BGP route reflector to track prefixes and combines routing data with sFlow measurements to identify the most active prefixes. Active prefixes can be advertised via BGP to a commodity switch, which acts as a hardware route cache, accelerating the performance of a software router.
There is an interesting parallel with the Open vSwitch architecture, see Open vSwitch performance monitoring, which maintains a cache of active flows in the Linux kernel to accelerate forwarding. In the SDN routing case, active prefixes are pushed to the switch ASIC in order to bypass the slower software router.
In this example, the software is being used in passive mode, estimating the cache hit / miss rates without offloading routes. The software has been configured to manage a cache of 10,000 prefixes. The first screen shot shows the cache warming up.

The first panel shows routes being learned from the route reflector: the upper chart shows the approximately 600,000 routes being learned from the BGP route reflector, and the lower chart shows the rate at which Continue reading