Creative foot-shooting with Go RWMutex
Hi, I'm Filippo and today I managed to surprise myself! (And not in a good way.)
I'm developing a new module ("filter" as we call them) for RRDNS, CloudFlare's Go DNS server. It's a rewrite of the authoritative module, the one that adds the IP addresses to DNS answers.
It has a table of CloudFlare IPs that looks like this:
type IPMap struct {
sync.RWMutex
M map[string][]net.IP
}
It's a global filter attribute:
type V2Filter struct {
name string
IPTable *IPMap
// [...]
}
CC-BY-NC-ND image by Martin SoulStealer
The table changes often, so a background goroutine periodically reloads it from our distributed key-value store, acquires the lock (f.IPTable.Lock()), updates it and releases the lock (f.IPTable.Unlock()). This happens every 5 minutes.
Everything worked in tests, including multiple and concurrent requests.
Today we deployed to an off-production test machine and everything worked. For a few minutes. Then RRDNS stopped answering queries for the beta domains served by the new code.
What. That worked on my laptop™.
Here's the IPTable consumer function. You can probably spot the bug.
func (f *V2Filter) getCFAddr(...) (result []dns.RR) {
f. Continue reading
Texas company joins the fray in providing service assurance in virtual environments.
"Traditional BSS will not be able to handle the shift from the physical network to the virtual network."
