Archive

Category Archives for "High Scalability"

Free Google Book: Building Secure and Reliable Systems

 

Google added another book into their excellent SRE series: Building Secure and Reliable Systems. It's free to download, so don't be shy. 

It's not short: 557 pages and 21 chapters! So what's it about? In short it's about "reliability through the lens of security."

In long, Ana Oprea, one of the authors, gave a good overview. anaoprea:

There are multiple questions about what this book is about, who it's for and what might be relevant for me. We recommend going through the Preface to get answers to these questions. Copy/pasting a few paragraphs: "In this book we talk generally about systems, which is a conceptual way of thinking about the groups of components that cooperate to perform some function.

We wanted to write a book that focuses on integrating security and reliability directly into the software and system lifecycle, both to highlight technologies and practices that protect systems and keep them reliable, and to illustrate how those practices interact with each other.

We’d like to explicitly acknowledge that some of the strategies this book recommends require infrastructure support that simply may not exist where you’re currently working.

Because security and reliability are everyone’s responsibility, we’re Continue reading

One Team at Uber is Moving from Microservices to Macroservices

 

There may be an undiscovered tribe deep in some jungle somewhere that hasn’t made up their mind on microservices, but I doubt it. People love microservices or love to hate microservices. There’s not much in between.

So it means something when even a team at a company like Uber announces a change away from microservices to something else. What? Macroservices. But we’ll get to that. Think what you want about Uber the company, but from a software perspective Uber has been a good citizen.

Gergely Orosz, an Engineering Manager on the Payments Experience Platform at Uber, in a tweet signaled a change in architectural direction:

Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the Continue reading

Stuff The Internet Says On Scalability For March 27th, 2020

Hey, it's HighScalability time!

 

Awesome explanation of how to build a PID controller to fly a rocket! (BPS.space via Orbital Index)

 

Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the intelligent world.

Know someone who wants to understand the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 103 mostly 5 star reviews. Here's a recent authentic unfaked review:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Post: Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • At Scrapinghub you will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also Continue reading

Post: Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber. 

  • Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!

  • PA File Sight - Actively protect servers from ransomware, audit file access to see who is deleting files, reading files or moving files, Continue reading

Stuff The Internet Says On Scalability For March 13th, 2020

p>Hey, it's HighScalability time!

 

The top 10,000 most spoken words in English represented by a point in hundreds of dimensions where the distance and direction between points encodes the relationship between words. (roadmaps)

 

Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the intelligent world.

Know someone who wants to understand the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 100 (!!!) mostly 5 star reviews. Here's a recent authentic unfaked review:

 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Post: Essilen Research, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • Are you looking to improve how you hire technical talent? Essilen Research's free video series can help. Fix common issues in screening, interviewing, closing and on-boarding. Learn how world-class tech companies crush the hiring game!

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber. 

  • Stateful JavaScript Apps. Effortlessly add state Continue reading

Stuff The Internet Says On Scalability For February 28th, 2020

Wake up! It's HighScalability time:

 

The Smithsonian has million of pieces of delicious open access content. I ate up this 3D representation of the 1903 Wright Flyer

 

Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff won't happen. That's how important you are to the fate of the world.

Need to understand the cloud? Know someone who does? I wrote Explain the Cloud Like I'm 10 just for you...and them. On Amazon it has 98 mostly 5 star reviews. Here's a recent authentic unfaked review:

 

Number Stuff: 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Socratic vs. Euclidean Forms of API Documentation

 

I was emailing a service about their documentation and while their doc was good, about one particularly tricky concept they told me that once you use it for a while, that’s when you’ll understand it.

In other words: you’ll only understand it after you understand it.

I didn’t like that response. I want documentation that takes me from an unproductive newbie to a somewhat functioning journeyperson. Not an expert, but I want to get stuff done as soon as possible. And for that you need to understand the mental model behind the API. Otherwise, how do you know how to make anything happen?

I realize it’s hard to make good documentation. I spent a lot of time writing Explain the Cloud Like I'm 10 just to communicate the mental model behind the cloud. It’s not easy.

Then I read that something that showed me there are two different styles of documentation: Euclidean and Socratic:

Euclidean - state your axioms and let users derive the rest. Easiest for the API provider, but hardest on the user. This is the most common form of documentation. You see it all the time. Each entry point in the API is sort of Continue reading

Moore’s Law is not Ending Soon and the Reason May Surprise You

Schrodingers cat

 

 

Jim Keller recently gave a fascinating and far ranging interview on the AI Podcast. You can find it at Moore's Law, Microprocessors, Abstractions, and First Principles.

One of the many topics of discussion was the often predicted death of Moore's Law. In case you've never heard of Jim Keller before, from this intro you can immediately understand why he may have special insight on the topic:

Jim Keller is a legendary microprocessor engineer, having worked at AMD, Apple, Tesla, and now Intel. He's known for his work on the AMD K7, K8, K12 and Zen microarchitectures, Apple A4, A5 processors, and co-author of the specifications for the x86-64 instruction set and HyperTransport interconnect.

Before we can understand why Moore's Law is not ending soon, we need to understand the idea of a diminishing return curve (this is a gloss of the talk, any errors or omissions are mine, but I tried to get the feel of it):

A project first goes up and then shows diminishing returns over time. To get to the next level you need to start a new project. The initial starting point of that new project will be lower than the return of the Continue reading

Post: Essilen Research, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • Are you looking to improve how you hire technical talent? Essilen Research's free video series can help. Fix common issues in screening, interviewing, closing and on-boarding. Learn how world-class tech companies crush the hiring game!

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber. 

  • Stateful JavaScript Apps. Effortlessly add state Continue reading

Important Health Checks for your MySQL Master-Slave Servers

In a MySQL master-slave high availability (HA) setup, it is important to continuously monitor the health of the master and slave servers so you can detect potential issues and take corrective actions. In this blog post, we explain some basic health checks you can do on your MySQL master and slave nodes to ensure your setup is healthy. The monitoring program or script must alert the high availability framework in case any of the health checks fails, enabling the high availability framework to take corrective actions in order to ensure service availability.

MySQL Master Server Health Checks

Stuff The Internet Says On Scalability For February 14th, 2020

Wake up! It's HighScalability time:

 

Visualize the huge scale of Deep Time by identifying key reference points along the way.

 

Do you like this sort of Stuff? Without your support on Patreon Stuff won't happen. I also wrote Explain the Cloud Like I'm 10 for everyone needing to understand the cloud (who doesn't?). On Amazon it has 93 mostly 5 star reviews (152 on Goodreads). Please be a real cloud hero and recommend it.

 

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Post: Essilen Research, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • Are you looking to improve how you hire technical talent? Essilen Research's free video series can help. Fix common issues in screening, interviewing, closing and on-boarding. Learn how world-class tech companies crush the hiring game!

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber. 

  • Stateful JavaScript Apps. Effortlessly add state Continue reading

Stuff The Internet Says On Scalability For February 7th, 2020

 Wake up! It's HighScalability time:

 .

A network visualization of an information operation on Twitter which originated in Russia (Watch six decade-long disinformation operations unfold in six minutes)

 

Do you like this sort of Stuff? Your support on Patreon is appreciated more than you can know. I also wrote Explain the Cloud Like I'm 10 for everyone who needs to understand the cloud (which is everyone). On Amazon it has 90 mostly 5 star reviews (152 on Goodreads). Please recommend it. You'll be a real cloud hero.

 

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber. 

  • Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!

  • PA File Sight - Actively protect servers from ransomware, audit file access to see who is deleting files, reading files or moving files, Continue reading

Stuff The Internet Says On Scalability For January 24th, 2020

 Wake up! It's HighScalability time:

 

Instead of turning every car into rolling sensor studded supercomputers, roads could be festooned with stationary edge command and control pods for offloading compute, sensing and managing traffic. Cars become mostly remote controlled pleasure palaces. Solves compute, latency, and interop.

 

Do you like this sort of Stuff? Your support on Patreon is appreciated more than you can know. I also wrote Explain the Cloud Like I'm 10 for everyone who needs to understand the cloud (which is everyone). On Amazon it has 87 mostly 5 star reviews (149 on Goodreads). Please recommend it. You'll be a real cloud hero.

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

Who's Hiring? 

  • Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber. 

  • Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!

  • PA File Sight - Actively protect servers from ransomware, audit file access to see who is deleting files, reading files or moving files, Continue reading

Follower Clusters – 3 Major Use Cases for Syncing SQL & NoSQL Deployments

Follower Clusters – 3 Major Use Cases for Syncing SQL & NoSQL Deployments

Follower clusters are a ScaleGrid feature that allows you to keep two independent database systems (of the same type) in sync. Unlike cloning or replication, this allows you to maintain an active, point-in-time copy of your production data. This extra cluster, known as a follower cluster, can be leveraged for multiple use cases, including for analyzing, optimizing and testing your application performance for MongoDB, MySQL and PostgreSQL. In this blog post, we will cover the top three scenarios to leverage follower clusters for your application.

How Do Follower Clusters Differ From Replication?

Unlike a static clone, this data imports on a set schedule so your follower cluster is always in sync with your production cluster. Here are a few critical ways in which it differs from replication:

1 3 4 5 6 7 30