Engineering dependability and fault tolerance in a distributed system
This is a guest post by Paddy Byers, Co-founder and CTO at Ably, a realtime data delivery platform. You can view the original article on Ably's blog.
Users need to know that they can depend on the service that is provided to them. In practice, because from time to time individual elements will inevitably fail, this means you have to be able to continue in spite of those failures.
In this article, we discuss the concepts of dependability and fault tolerance in detail and explain how the Ably platform is designed with fault tolerant approaches to uphold its dependability guarantees.
As a basis for that discussion, first some definitions:
Dependability
The degree to which a product or service can be relied upon. Availability and Reliability are forms of dependability.
Availability
The degree to which a product or service is available for use when required. This often boils down to provisioning sufficient redundancy of resources with statistically independent failures.
Reliability
The degree to which the product or service conforms to its specification when in use. This means a system that is not merely available but is also engineered with extensive redundant measures to continue to work as its Continue reading
Sponsored Post: 3T, Bridgecrew, Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Discover the MongoDB data masking tool in Studio 3T Enterprise. Enable data compliance and bolster security with powerful field-level data obfuscation. Try for free!
- Bridgecrew is the cloud security platform for developers. By leveraging automation and delivering security-as-code, Bridgecrew empowers teams to find, fix, and prevent misconfigurations in deployed cloud resources and in infrastructure as code. Get started for free!
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. Continue reading
Benchmark (YCSB) numbers for Redis, MongoDB, Couchbase2, Yugabyte and BangDB
This is guest post by Sachin Sinha who is passionate about data, analytics and machine learning at scale. Author & founder of BangDB.
This article is to simply report the YCSB bench test results in detail for five NoSQL databases namely Redis, MongoDB, Couchbase, Yugabyte and BangDB and compare the result side by side. I have used latest versions for each NoSQL DB and have followed the recommendations to run all the databases in optimized conditions. I have also used the default six test scenarios as defined by the YCSB framework. I have restricted it to 10M records for each test. However, user can run the bench for as many numbers as they practically find suitable.
About YCSB
Following configurations were used for the evaluation purpose.
- Redis Server: 5.07, x86/64
- MongoDB server: 4.4.2, x86_64
- YugabyteDB:2.5.0, x86_64
- Couchbase2: 7.0 Beta, x86_64
- BangDB server: 2.0.0, x86_64
- Number of records: 10M
- RAM: 32GB, Cores: 16
- YCSB workloads: see github.com/brianfrankcooper/YCSB/wiki/Core-Workloads
Each of these workload test runs in two steps, 1. Load and 2. Run. Load stage is to load the data and then run stage we run the test. I have run each test with Continue reading
Sponsored Post: 3T, Bridgecrew, Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Discover the MongoDB data masking tool in Studio 3T Enterprise. Enable data compliance and bolster security with powerful field-level data obfuscation. Try for free!
- Bridgecrew is the cloud security platform for developers. By leveraging automation and delivering security-as-code, Bridgecrew empowers teams to find, fix, and prevent misconfigurations in deployed cloud resources and in infrastructure as code. Get started for free!
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. Continue reading
Stuff The Internet Says On Scalability For February 1st, 2021
Hey, it's HighScalability time once again!
Amazon converts expenses into revenue by transforming needs into products. Take a look at a fulfillment center and you can see the need for Outpost, machine learning, IoT, etc, all dogfooded. Willy Wonka would be proud.
Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen.
Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 238 mostly 5 star reviews. Here's a review that has not been shorted by a hedge fund:
Number Stuff:
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...Sponsored Post: 3T, Bridgecrew, Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Discover the MongoDB data masking tool in Studio 3T Enterprise. Enable data compliance and bolster security with powerful field-level data obfuscation. Try for free!
- Bridgecrew is the cloud security platform for developers. By leveraging automation and delivering security-as-code, Bridgecrew empowers teams to find, fix, and prevent misconfigurations in deployed cloud resources and in infrastructure as code. Get started for free!
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. Continue reading
Sponsored Post: Bridgecrew, Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Bridgecrew is the cloud security platform for developers. By leveraging automation and delivering security-as-code, Bridgecrew empowers teams to find, fix, and prevent misconfigurations in deployed cloud resources and in infrastructure as code. Get started for free!
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!
- ipdata is a Continue reading
The Read Aloud Cloud: An Interview With Forrest Brazeal On His New Book
It's Christmas time and you've been wracking your brain trying to find the perfect gift that will teach your loved ones about the cloud in a simple and entertaining way. What to do?
Fortunately for you, Santa has a new elf this year—Forrest Brazeal—who is part AWS Serverless Hero, part skilled cartoonist, and part cloud guru.
Yes, it's a cartoon book about the cloud!
No, I didn't think it could be done either, but Forrest pulled it off with a twinkle in his eyes and little round belly that shook when he laughed, like a bowl full of jelly.
One of the many five-star reviews:
The Read Aloud Cloud is a delightful book. It's 165 pages of hand-drawn cartoons, entertaining verse, and hard-won wisdom.
It truly is a load of fun to flip through. I read it to my kids (8 & 6), and they love it. I'll share it with my parents so they can finally understand what I do. I learned a ton about the history of computing and of course all the ways that we as humans stumble through making our computers do what we want.
Sponsored Post: Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!
- ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!
- Developers care Continue reading
Stuff The Internet Says On Scalability For December 19th, 2020
Hey, it's HighScalability time once again!
Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen.
Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 212 mostly 5 star reviews. Here's a load-balanced and fault-tolerant review:
Number Stuff:
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...Sponsored Post: Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!
- ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!
- Developers care Continue reading
Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!
- ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!
- Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading
Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!
- ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!
- Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading
Stuff The Internet Says On Scalability For November 6th, 2020
Hey, no power outages this week, so it's finally HighScalability time!
Stunning: Tycho Crater Region with Colours by Alain Paillou
Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen.
Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 189 mostly 5 star reviews. Here's a 100% lactose-free review:
Number Stuff:
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...ShiftLeft on Refactoring a Live SaaS Environment
This is guest a post by Preetam Jinka, Senior Infrastructure Engineer at ShiftLeft. Originally published here.
ShiftLeft NextGen Static Analysis (NG SAST) is a software-as-a-service static analysis solution that allows developers to scan every pull request for security issues. Earlier this year we released Secrets, Security Insights, and a v4 API. Secrets and Security Insights are two new types of results we extract from code analysis, and the V4 API is a brand new RESTful JSON API with an OpenAPI/Swagger specification that you can use to access all of your results. Read more about these features in our announcement post.
NG SAST was initially designed only for vulnerabilities. In order to implement Secrets and Security Insights, we either had to retrofit these new result types into our existing implementation or significantly refactor our back-end to support their unique characteristics. Even though it would take longer and be more difficult to implement, we decided to do the latter. We rewrote almost all of the storage used for storing code analysis results while maintaining backwards compatibility and without any outages. The analogy is that it’s like changing the engine on an airplane in flight without the passengers noticing.
We could’ve saved Continue reading
Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!
- ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!
- Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading
Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!
- ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!
- Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading
Snakes in a Facebook Datacenter
Fault tolerance is the property that enables a system to continue operating properly in the event of the failure of some of its components. You might think Facebook solved all of its fault tolerance problems long ago, but when a serpent enters the Edenic datacenter realm, even Facebook must consult the Tree of Knowledge.
In this case, it's not good or evil we'll learn about, but Workload Placement, which is a method of optimally placing work over a set of failure domains.
Here's my gloss of the Fault Tolerance through Optimal Workload Placement talk:
How did they send money in the olden days of telegraphs?
It's not all that different really, especially that part where you can lose all your bitcoin. Here's an excerpt from East of Eden by John Steinbeck:
Say, Carlton, how do you go about telegraphing money?”
“Well, you bring me a hundred and two dollars and sixty cents and I send a wire telling the Valdosta operator to pay Adam one hundred dollars. You owe me sixty cents too.”
“I’ll pay—say, how do I know it’s Adam? What’s to stop anybody from collecting it?”
The operator permitted himself a smile of worldliness.
“Way we go about it, you give me a question couldn’t nobody else know the answer. So I send both the question and the answer. Operator asks this fella the question, and if he can’t answer he don’t get the money.””
“Say, that’s pretty cute. I better think up a good one.”
“You better get the hundred dollars while Old Breen still got the window open.”
Charles was delighted with the game. He came back with the money in his hand.
“I got the question,” he said.
“I hope it ain’t your mother’s middle name. Lot of people don’t remember.”
“No, nothing Continue reading
Stuff The Internet Says On Scalability For Sep 18th, 2020
Hey, it's HighScalability time!
I can't wait for the duel. Just don't shoot into the air.
Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff won't happen.
Know someone who could benefit from becoming one with the cloud? Of course you do. I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 167 mostly 5 star reviews. Here's a 100% lectin-free review: