How Ably Efficiently Implemented Consistent Hashing

This is a guest post by Srushtika Neelakantam, Developer Advovate for Ably Realtime, a realtime data delivery platform.

Ably’s realtime platform is distributed across more than 14 physical data centres and 100s of nodes. In order for us to ensure both load and data are distributed evenly and consistently across all our nodes, we use consistent hashing algorithms.

In this article, we’ll understand what consistent hashing is all about and why it is an essential tool in scalable distributed system architectures. Further, we’ll look at data structures that can be used to implement this algorithm efficiently at scale. At the end, we’ll also have a look at a working example for the same.

Hashing revisited

Remember the good old naïve Hashing approach that you learnt in college? Using a hash function, we ensured that resources required by computer programs could be stored in memory in an efficient manner, ensuring that in-memory data structures are loaded evenly. We also ensured that this resource storing strategy also made information retrieval more efficient and thus made programs run faster.

The classic hashing approach used a hash function to generate a pseudo-random number, which is then divided by the size of the memory Continue reading

Network Break 189: The Big Cisco Live Roundup; LiveAction Buys Savvius

Take a Network Break! Cisco Live US 2018 took place last week, so we spend a some time covering show news, overall impressions, and a touch of tea-leaf reading.

In non-Cisco news, VMware has a new lower-cost pricing tier to encourage customers to try VMware on AWS, LiveAction acquires packet capture/network monitoring vendor Savvius for an undisclosed amount, and orchestration vendor Gluware can now upgrade OSs for seven different vendors.

Metaswitch joins the OpenSwitch project, ONAP announces the Beijing release of its network automation package, and Comcast has deactivated its “congestion management system” (aka throttling).

Speaking of Comcast, the ISP has made a $65 billion bid for 21st Century Fox. In other provider news, AT&T gets the greenlight to merge with Time Warner. And last but not least, Cisco has joined an investment round in the startup Avi Networks, which makes software load balancers and service meshes.

Get links to all these stories after our sponsor messages.

Sponsor: ThousandEyes

ThousandEyes gives you performance visibility from every user to every app over any network, both internal and external, so you can smoothly migrate to the cloud, transform your WAN, troubleshoot faster and deliver exceptional user experiences. Sign up for a free Continue reading

Conference Impostor Syndrome

In IT we’ve all heard of Impostor Syndrome by now. The feeling that you’re not just a lucky person that has no real skills or is skating by on the seat of their pants is a very real thing. I’ve felt it an many of my friends and fellow members of the community have felt it too. It’s easy to deal with when you have time to think or work on your own. However, when you take your show on the road it can creep up before you know it.

Conferences are a great place to meet people and learn about new ideas. It’s also a place where your ideas will be challenged and put on display. It’s not to difficult to imagine meeting a person for the first time at a place like Cisco Live or VMworld and not feeling little awe-inspired. After all, this could be a person whose works you’ve read for a long time. It could be a person you look up to or someone you would like to have mentor you.

For those in the position of being thrust into the limelight, it can be extremely difficult to push aside those feelings of Impostor Syndrome Continue reading

Net Neutrality Round Table

Debates regarding net neutrality regulation in the United States have been carried out for over a decade. Rulemakings by the FCC have been passed numerous times, won and lost in court, and been repealed, resulting in years of political back and forth. Now, net neutrality is being argued for and against on Capitol Hill and its regulatory future is unclear.

To address this political limbo, the Internet Society convened experts from the technical community, public interest groups, and academia to discuss how we can create a permanent solution for net neutrality that protect the interests of Internet users while fostering an environment that encourages investment and innovation. During this half-day event, participants began a conversation to define net neutrality, what conduct it should cover, how compliance could be assured, and how to balance consumer and private sector interests.

The discussion was moderated by Larry Stickling, Executive Director of the Collaborative Governance Project at the Internet Society, and included a balanced group of politically left- and right-leaning public interest groups, private sector organizations, and academics. The event was under Chatham House Rule and did not allow tweeting during the meeting in order to encourage participants to freely and respectfully voice their Continue reading

IDG Contributor Network: 3 providers fixing the middle mile problems of internet-based SD-WANs

A new global backbone provider emerged from stealth last week, giving organizations even more choice in how they build their Internet-based SD-WANs.  Mode introduced what it calls a “software-defined core” (SD-CORE) network that offers IT “affordable private-network reliability and quality of service” across the globe.The company joins Aryaka and Cato Networks as one of the few independent backbone providers helping enterprises solve the variability problems of the Internet core. Middle-mile performance forms the biggest challenge for delivering stable, global, low-latency connections.To read this article in full, please click here

IDG Contributor Network: 3 providers fixing the middle mile problems of internet-based SD-WANs

A new global backbone provider emerged from stealth last week, giving organizations even more choice in how they build their Internet-based SD-WANs.  Mode introduced what it calls a “software-defined core” (SD-CORE) network that offers IT “affordable private network reliability and quality of service” across the globe.The company joins Aryaka and Cato Networks as one of the few independent backbone providers helping enterprises solve the variability problems of the Internet core.  "Internet testing results: Why fixing the internet middle mile is essential for SD-WAN performance," middle-mile performance, forms the biggest challenge for delivering stable, global, low-latency connections.To read this article in full, please click here

The Week in Internet News: X-Ray I

AI to get X-ray vision: Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory are getting close to creating AI that can see through walls, Geek.com reported. The research team is using AI to analyze radio signals bouncing off human bodies. The result is a neural network-generated stick figure that moves like the targeted person does.

Dr. AI will see you now: Perhaps more useful that looking through walls, some AI technologies are now being used to identify tuberculosis, pneumonia, upper respiratory infection, and bronchitis based on how a cough sounds, said AdWeek. Several companies are exploring other ways to use AI in healthcare settings.

Encryption wars, part 207: Apple has moved to close a security hole that law enforcement agencies used to defeat encryption on iPhones, according to many news reports, including one in the New York Times. The Apple move set off a new round of debate about encrypted devices and law enforcement access, the Washington Post noted.

It appears that at least one company that builds iPhone cracking tools already has a workaround, however, Motherboard reported.

Meanwhile, an FBI official suggested that each encrypted device that law enforcement agencies cannot crack represents a victim without justice, BusinessInsider. Continue reading

Cisco makes SD-WAN integration a top priority

Software and programmable intelligent networks were hot topics at Cisco Live last week, and one of the key components of that discussion was the direction of the company’s SD-WAN strategy.Central to that dialog is how Cisco plans to use and integrate the SD-WAN technology it acquired last year when it bought Viptela for $610 million. For the moment, Cisco says Viptela has brought with it interest to the tune of about 800 new customers in recent months.To read this article in full, please click here

Microsoft adds resiliency, redundancy, security to Windows Server 2019

With Windows Server 2019, Microsoft is adding resiliency and redundancy enhancements to the Shielded Virtual Machines security controls it introduced with Windows Server 2016.Shielded VMs originally provided a way to protect virtual machine assets by isolating them from the hypervisor infrastructure and could also help prove to auditors that systems were adequately isolated and controlled. Now Shielded VM enhancements in Window Server 2019 provide real-time failback configurations and host- and policy-based security improvements.[ Don’t miss customer reviews of top remote access tools and see the most powerful IoT companies . | Get daily insights by signing up for Network World newsletters. ] Host key attestation Under Windows Server 2016, key authentication was based on trusted platform module (TPM) cryptoprocessors and Microsoft Active Directory authentication. Both of these are great solutions but were limited when it comes to extensibility and redundancy.To read this article in full, please click here(Insider Story)

Cisco makes SD-WAN integration a top priority

Software and programmable intelligent networks were hot topics at Cisco Live last week, and one of the key components of that discussion was the direction of the company’s SD-WAN strategy.Central to that dialog is how Cisco plans to use and integrate the SD-WAN technology it acquired last year when it bought Viptela for $610 million. For the moment, Cisco says Viptela has brought with it interest to the tune of about 800 new customers in recent months.To read this article in full, please click here

Cisco makes SD-WAN integration a top priority

Software and programmable intelligent networks were hot topics at Cisco Live last week, and one of the key components of that discussion was the direction of the company’s SD-WAN strategy.Central to that dialog is how Cisco plans to use and integrate the SD-WAN technology it acquired last year when it bought Viptela for $610 million.   For the moment Cisco says Viptela has brought with it interest to the tune of about 800 new customers in recent months.To read this article in full, please click here

Vertical Integration Musings

One of my readers asked me a question that came up in his business strategy class:

Why did routers and switches end up being vertically integrated (the same person makes the hardware and the software)? Why didn't they go down the same horizontal path as compute (with Intel making chips, OEMs making systems and Microsoft providing the OS)? Why did this resemble the pre-Intel model of IBM, DEC, Sun…?

Simple answer: because nobody was interested in disaggregating them.

Read more ...

DeepTest: automated testing of deep-neural-network-driven autonomous cars

DeepTest: automated testing of deep-neural-network-driven autonomous cars Tian et al., ICSE’18

How do you test a DNN? We’ve seen plenty of examples of adversarial attacks in previous editions of The Morning Paper, but you couldn’t really say that generating adversarial images is enough to give you confidence in the overall behaviour of a model under all operating conditions. Adversarial images approach things from a ‘think like an attacker’ mindset. We want to ‘think like a tester.’ For example, the work on DeepXplore which uses model ensembles to find differences in outputs that suggest bugs. The importance of testing DNNs is especially obvious when it comes to applications such as autonomous driving. Several of the ideas from DeepXplore are used in DeepTest, which looks specifically at testing of autonomous driving system. I think you could apply the DeepTest techniques to test other kinds of DNNs as well.

…despite the tremendous progress, just like traditional software, DNN-based software, including the ones used for autonomous driving, often demonstrate incorrect/unexpected corner-case behaviours that lead to dangerous consequences like a fatal collision.

DeepTest is a system designed to aid in the testing of autonomous driving models. When used to test three of Continue reading

SEE 7: Connectivity, Routing Security & IoT

The 7th RIPE South-East Europe (SEE 7) meeting is being held on 18-19 June 2018 in Timisoara, Romania, and is focusing on several of the subjects of interest to the Internet Society. It’s also being chaired by our colleague Jan Žorž, whilst I’ll be talking about IoT Security and the OTA IoT Trust Framework.

In Monday, there are talks on BGP monitoring from Paolo Lucente (pmacct), and from Krzysztof Grzegorz Szarkowicz (Juniper Networks) on improvements to routing protocols to suit the centralised data centre-based architectures that are becoming more prevalent on the Internet, and which are the subject of an Internet Draft. Zoran Perovic (SOX) will also talk about paradigm shifts in the implementation of Internet Exchange Points.

On Tuesday, there will be a discussion led by Goran Slavic (SOX) on implementing MANRS in an IXP, which is very relevant to the current MANRS initiative which is increasingly being adopted by IXPs. Our colleague Jan will then be presenting about RIPE-690 which provides recommendations for IPv6 address prefix assignments for end-users. Preceding this, will be an update on IPv6 adoption in the SEE region from Massimiliano Stucchi (RIPE NCC).

Some other highlights are the talk on Quad9DNS by Nishal Goburdhan (PCH) that’s supporting Continue reading

Notes on “The President is Missing”

Former president Bill Clinton has contributed to a cyberthriller "The President is Missing", the plot of which is that the president stops a cybervirus from destroying the country. This is scary, because people in Washington D.C. are going to read this book, believe the hacking portrayed has some basis in reality, and base policy on it. This "news analysis" piece in the New York Times is a good example, coming up with policy recommendations based on fictional cliches rather than a reality of what hackers do.


The cybervirus in the book is some all powerful thing, able to infect everything everywhere without being detected. This is fantasy no more real than magic and faeries. Sure, magical faeries is a popular basis for fiction, but in this case, it's lazy fantasy, a cliche. In fiction, viruses are rarely portrayed as anything other than all powerful.

But in the real world, viruses have important limitations. If you knew anything about computer viruses, rather than being impressed by what they can do, you'd be disappointed by what they can't.

Go look at your home router. See the blinky lights. The light flashes every time a packet of data goes across the network. Continue reading

Don’t Rely on Long Support Lifecycles

I hate long support lifecycles for hardware and software. Yes, you should be able to buy a new iPhone or switch and use it for 3+ years. But some people want 10+yrs of support, and wail and moan when vendors end support. This is wrong. It drives up costs & complexity, and makes your systems less robust, not more. It’s a false sense of security. Plan to buy smaller & cheaper, and upgrade frequently.

Why Vendors Don’t Like Them

Vendors don’t want to do long support lifecycles. They will do them, because people pay for it, but there comes a point where they put a line in the sand. “Sorry, that system is now EoL.”

Why?

  • Costs: Testing software and hardware combinations is hard work. Add many years of released hardware & software combinations, and it gets much harder. More racks of gear & more permutations == more costs.

  • Complexity: It’s hard enough to test against a small set. But now you have to deal with obscure systems acquired from a third party 7 years ago? Complexity == time and money.

  • Motivation: Hands up who wants to work on legacy systems? Exactly. It’s hard to motivate engineers to support Continue reading

1 … 1,523 1,524 1,525 1,526 1,527 … 3,797