What Happens When 20 Programs Poll The Network?

Packetpushers show 198 was a great episode about Network Automation. At one point, Greg asks:

“What happens when you’ve got 20 apps polling one device?”

Well, you might hit the same problem I did:

SECURITY-SSHD-6-INFO_GENERAL : Incoming SSH session rate limit exceeded

I have some Python scripts that poll performance and configuration data from a couple of ASR9Ks, and I was getting some gaps in my data. The scripts run on different polling cycles (some hourly, some every 15 minutes, etc). It wasn’t consistent, but now and then my script would fail to collect any data.

I dug into it, and found that I was hitting the default SSH rate limit of 60 per minute, calculated as 1 per second. Because I couldn’t control the exact scheduling of when my polls ran, I inserted a short random wait timer into some of them. That helped, and I had fewer failures, but it still wasn’t quite right.

So I used the command “ssh server rate-limit 120″ to allow 2 SSH connections per second. That has helped, and now I’m not getting any failures.

But it won’t be pretty if I do have 20 different apps all trying to poll at once.

(Yes, I know, I should Continue reading

ScienceLogic Global Network Manager

ScienceLogic 7.5 includes many enhancements and new features. One I’m interested in is “Global Manager” which can be used to massively scale out the ScienceLogic architecture. Here’s some more detail on why ScienceLogic introduced this feature, and what it does.

Problem: A Single Database

I’ve talked before about the ScienceLogic architecture, and noted that the Database can be a bottleneck:

You’ll notice that all the variations only ever have one “active” database at any one time. All the processing is done on this system, with the results replicated to the other databases. You can scale out your Collectors or User Interface by adding more servers – but you can’t scale out the core database. Right now you have to scale up the database – ie. allocate more RAM/CPU/IOPS. This gets around the performance bottlenecks, but comes at a cost.

In this diagram, we can see the database is at the heart of everything. We can have HA & DR options for it, but there is only ever one active DB:

Distributed Architecture – click for larger

We can have multiple web interfaces, but they all query the same database.

Solution: More Databases!

The new Global Manager option from Continue reading

Review: ScienceLogic – One Network Management Tool To Rule Them All?

ScienceLogic has been getting the right kind of press recently – e.g. they were a winner of Best of Interop 2013 – Management & Monitoring, and Infoworld had some rather nice things to say. They’ve got some high-profile customers too, such as Fasthosts and Equinix. But what exactly is their product all about, and is it any […]

Author information

Lindsay Hill

Network Management Consultant

Lindsay (@northlandboy) is a network management consultant, with experience across networks, servers, applications and security. He is CCIE #36708, RHCE, CISSP and HP MASE. More of his own content is at lkhill.com.

TwitterGoogle+

The post Review: ScienceLogic – One Network Management Tool To Rule Them All? appeared first on Packet Pushers Podcast and was written by Lindsay Hill.