The revenge of the listening sockets
Back in November we wrote a blog post about one latency spike. Today I'd like to share a continuation of that story. As it turns out, the misconfigured rmem
setting wasn't the only source of added latency.
It looked like Mr Wolf hadn't finished his job.
After adjusting the previously discussed rmem
sysctl we continued monitoring our systems' latency. Among other things we measured ping
times to our edge servers. While the worst case improved and we didn't see 1000ms+ pings anymore, the line still wasn't flat. Here's a graph of ping latency between an idling internal machine and a production server. The test was done within the datacenter, the packets never went to the public internet. The Y axis of the chart shows ping
times in milliseconds, the X axis is the time of the measurement. Measurements were taken every second for over 6 hours:
As you can see most pings finished below 1ms. But out of 21,600 measurements about 20 had high latency of up to 100ms. Not ideal, is it?
System tap
The latency occurred within our datacenter and the packets weren't lost. This suggested a kernel issue again. Linux responds to ICMP pings from its soft Continue reading