The Porcupine Attack: investigating millions of junk requests
We extensively monitor our network and use multiple systems that give us visibility including external monitoring and internal alerts when things go wrong. One of the most useful systems is Grafana that allows us to quickly create arbitrary dashboards. And a heavy user of Grafana we are: at last count we had 645 different Grafana dashboards configured in our system!
grafana=> select count(1) from dashboard;
count
-------
645
(1 row)
This post is not about our Grafana systems though. It's about something we noticed a few days ago, while looking at one of those dashboards. We noticed this:
This chart shows the number of HTTP requests per second handled by our systems globally. You can clearly see multiple spikes, and this chart most definitely should not look like a porcupine! The spikes were large in scale - 500k to 1M HTTP requests per second. Something very strange was going on.
Tracing the spikes1
Our intuition indicated an attack - but our attack mitigation systems didn't confirm it. We'd seen no major HTTP attacks at those times.
It would be bad if we were under such heavy HTTP attack and our mitigation systems didn't notice it. Without more ideas, we Continue reading
Automation For All The Things! What Happens Next?
Over the last five years, there has been increasing noise about whether the growing automation and orchestration of networks (and infrastructure in general) will lead to our jobs being eliminated. Concerns about mass layoffs are understandable given what happened when large scale automation was introduced to manufacturing.
What is left after automation has taken its toll on an industry? Presumably there is work for those who create and maintain the automation systems and there will be a need for workers to do the tasks which cannot be adequately automated, but the people who don’t fit into these categories might be facing a tough future. Some workers will retrain or adapt their skills to shift themselves into one of the “needed” categories, but since the idea of automation in most industries is to reduce the need for salaried humans be more agile and respond faster to customer needs, the competition for those positions is likely to be strong.
Does Automation Mean A Bleak Future?
Up front let me say that I believe that the predictions of imminent doom are utter codswallop. In order for the jobocalypse to occur, automation has to be present across the every industry because IT infrastructure exists pretty much everywhere as a business enabler, but only in subset of industries (e.g. Continue reading