Highscalability is Up For Sale

Hi everyone,

First, I’d like to sincerely thank everyone who has supported Highscalability over the years.

We’ve grown together through some interesting times. Massive changes have occurred in how systems are built, and my goal has always been to help people learn how to build them better. I’d like to think that goal was accomplished.

Obviously, for that last several years, things have been quiet around here. This has always been a one programmer show, and for everything, there is a season. The season for me is changing. That means http://highscalability.com is for sale.

My preference is for someone to take HS over and keep it going in some form or another. If you are a content farm, please don’t reply. I’ve never once taken payment for a post and I don’t plan on changing that now.

You get the site, the domain, and the twitter account. If you are not a scam artist and have a reasonable offer, please contact me at [email protected].

Thanks!

The Swedbank Outage shows that Change Controls don’t work

This week I’ve been reading through the recent judgment from the Swedish FSA on the Swedbank outage. If you’re unfamiliar with this story, Swedbank had a major outage in April 2022 that was caused by an unapproved change to their IT systems. It temporarily left nearly a million customers with incorrect balances, many of whom were unable to meet payments.

After investigation, the regulator found that Swedbank had not followed their change management process and issued a SEK850M (~85M USD) fine. That’s a lot of money to you and me, but probably didn’t impact their bottom line very much. Either way I’m sure the whole episode will have been a big wake up call for the people at the bank whose job it is to ensure adequate risk and change controls. So, what went wrong and how could it have been avoided?

How did the Swedbank incident happen?

The judgment doesn’t describe the technical details behind the incident, but it does provide glimpses into how they assessed what went wrong:

“The deficiencies that were present in Swedbank’s internal control made it possible to make changes to one of the bank’s most central IT systems without following the process Continue reading

Lessons Learned Running Presto at Meta Scale

Presto is a free, open source SQL query engine. We’ve been using it at Meta for the past ten years, and learned a lot while doing so. Running anything at scale - tools, processes, services - takes problem solving to overcome unexpected challenges. Here are four things we learned while scaling up Presto to Meta scale, and some advice if you’re interested in running your own queries at scale.

Scaling Presto rapidly to meet growing demands: What challenges did we face?

Deploying new Presto releases

Gossip Protocol Explained

You can subscribe to the system design newsletter to excel in system design interviews and software architecture. The original article was published on systemdesign.one website.

What Is Gossip Protocol?

The typical problems in a distributed system are the following [1], [11]:

maintaining the system state (liveness of nodes)
communication between nodes

The potential solutions to these problems are as follows [1]:

centralized state management service
peer-to-peer state management service

Consistent hashing algorithm

This is a guest article by NK. You can view the original article Consistent hashing explained on systemdesign.one website.

How does consistent hashing work?

At a high level, consistent hashing performs the following operations:

The output of the hash function is placed on a virtual ring structure (known as the hash ring)
The hashed IP addresses of the nodes are used to assign a position for the nodes on the hash ring
The key of a data object is hashed using the same hash function to find the position of the key on the hash ring
The hash ring is traversed in the clockwise direction starting from the position of the key until a node is found
The data object is stored or retrieved from the node that was found

Terminology

What is Cloud Computing? According to ChatGPT.

DALL-E picture generated from the answer.

I was curious, given all the ChatGPT love, what it would make of some of our favorite topics. I was both impressed and unimpressed.

Impressed. ChatGPT instantly generated an response that might make a good answer in a Miss Universe contest. You know, the kind of answer delivered with a radiant smile calculated to offend no one yet still sounds comprehensive and deep.

Unimpressed. The answers are a mile wide and an inch deep. There's not a lot of insight. It's more of a zeitgeist oracle. Nothing wrong with that, but I don't think we'll be unemployed soon. It's as many have predicted about AI/ML. These are tools to augment us, flawed humans, not replace us...yet.

Thank you for agreeing to be interviewed today.

You're welcome! I'm happy to answer any questions you have. Please go ahead and ask your first question.

Explain cloud computing to me at a professional level?

Cloud computing is a model of computing that delivers computing services over the internet, including storage, data processing, and networking. It allows users to access and use shared computing resources, such as servers, storage, and applications, on demand and without the need Continue reading

Stuff The Internet Says On Scalability For December 2nd, 2022

Never fear, HighScalability is here!

1958: An engineer wiring an early IBM computer 2021: An engineer wiring an early IBM quantum computer. @enclanglement

My Stuff:

I'm proud to announce a completely updated and expanded version of Explain the Cloud Like I'm 10! This version adds 2x more coverage, with special coverage of AWS, Azure, GCP, and K8s. It has 482 mostly 5 star reviews on Amazon. Here's a 100% organic, globally sourced review:
Love this Stuff? I need your support on Patreon to keep this stuff going.

Number Stuff:

The Cloud is Not a Railroad – An Argument Against the Vertical Separation of Cloud Providers

There's a move to regulate cloud providers by vertically separating the services they offer.

Like railroads of yore, who were not allowed to provide freight services on top of their base services, cloud providers would not be allowed to provide services on top of their base platform services.

Vertical separation would be new to the cloud industry. Is it a good idea? Would it actually solve any problems? My answers are no and no, but probably not for the reasons you think. Let's dive in.

Here are a few useful resources for exploring this argument:

Maintaining monopolies with the cloud by Cory Doctorow. https://pluralistic.net/2022/09/28/other-peoples-computers.
A group advocating for fair licensing. https://www.fairsoftwarelicensing.com/our-principles.
Cloud Infrastructure Services -- An analysis of potentially anti-competitive practices by Professor Frédéric Jenny. https://www.fairsoftwarestudy.com/.

Now Cory is about 100x times smarter than I am, but this analogy is wrong:

That's why the trustbusters were so big on "structural separation": the principle that a business can own a platform or use the platform, but not both.

So this remedy is also wrong:

Rather than trying to construct rules that kept the referee honest even when they played on one of the teams, they Continue reading

Announcing: A Completely New and Updated Version of Explain the Cloud Like I’m 10

I've been working hard on a completely updated version of Explain the Cloud Like I'm 10 and I'm happy to announce it's ready!

You can buy version 2.0 at: https://www.amazon.com/Explain-Cloud-Like-Im-10-ebook/dp/B0765C4SNR.

If you've already bought, Explain the Cloud Like I'm 10, then you should just be able to update to the new version for free.

What's new? This version has about 2x the material. I go in-depth on:

The major cloud providers and all the different platforms for deploying workloads.
How to decide between cloud providers.
How to switch to the cloud.
How to decide if you should switch to the cloud.
Cloud economics and how to save money in the cloud.
How you can learn more and take your cloud adventure to the next step.
And lots more. So much more.

In version 1 of the book, I didn't cover cloud providers in any detail. Almost all the new content is platform related, so that's fixed. And I say platform rather than cloud provider on purpose.

The biggest change in the book is it is now oriented around the "cloud model" as the ultimate goal for software development, deployment, and use, not just the Continue reading

Who's Hiring?

Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble humans, then come check us out!

Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Continue reading

Stuff The Internet Says On Scalability For July 11th, 2022

Never fear, HighScalability is here!

Every cell a universe. Most detailed image of a human cell to date. @microscopicture

Other images considered: one byte of RAM in 1946; visual guide on troubleshooting Kubernetes; Cloudflare using lava lamps to generate cryptographic keys; 5MB of data looked like in 1966

My Stuff:

Love this Stuff? I need your support on Patreon to help keep this stuff going.

Know anyone who needs to go from zero to cloud? My book teaches them all they need to know about the cloud. Explain the Cloud Like I'm 10. It has 440 mostly 5 star reviews on Amazon. Here's a 100% sincere, heartfelt review:
Do you like Zone 2, Zone 5, interval, REHIT, HIIT, and bodyweight workouts? I made an app for that. Max Workout. I’m not just the programmer, I’m a client. I use it 4 times a week and if you want to retrain your mitochondria, get fit, get healthy, and live longer, I think you’ll like it too.

Number Stuff:

Who's Hiring?

Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble humans, then come check us out!

Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Continue reading

Who's Hiring?

Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble humans, then come check us out!

Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Continue reading

Who's Hiring?

Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Stuff The Internet Says On Scalability For January 28th, 2022

Never fear, HighScalability is here!

Think your software diagram is complex? This is a single cell modeled using X-ray, nuclear magnetic resonance (NMR), and cryo-electron microscopy datasets. Gael McGill

My Stuff:

Love this Stuff? I need your support on Patreon to keep this stuff going.
Know anyone who needs to fix their cloud-obliviousness? My book teaches them all they need to know about the cloud. Explain the Cloud Like I'm 10. It has 364 mostly 5 star reviews on Amazon. Here's a 100% antibody free review:
Do you like Zone 2, Zone 5, interval, reHIT, or HIIT workouts? I made an app for that. Max reHIT Workout. I’m not just the programmer, I’m a client. I use it 4 times a week and if you want to retrain your mitochondria, get fit, get healthy, and live longer, I think you’ll like it too.

Number Stuff:

Designing Uber

This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypotheses on Amazon product pages using reinforcement learning techniques. Currently, he is in the Alexa Shopping organization where he is developing machine-learning-based solutions to send personalized reorder hints to customers for improving their experience.

Requirements

In Scope

Designing Tinder

Problem Statement

Design a location-based social search application similar to Tinder which if often used as a dating service. It allows users to use a swiping motion to like (swipe right) or dislike (swipe left) other users, and allows users to chat if both parties like each other(a “match”).

Gathering Requirements

Designing Instagram

Problem Statement

Design a photo-sharing platform similar to Instagram where users can upload their photos and share it with their followers. Subsequently, the users will be able to view personalized feeds containing posts from all the other users that they follow.

Gathering Requirements

In Scope

The application should be able to support the following requirements.

Users should be able to upload photos and view the photos they have uploaded.
Users should be able to follow other users.
Users can view feeds containing posts from the users they follow.
Users should be able to like and comment the posts.

Out of Scope

Sending and receiving messages from other Continue reading

Designing WhatsApp

Problem Statement

Design an instant messenger platform such as WhatsApp or Signal which users can utilize tosend messages to each other. An essential aspect of the application is that the chat messageswon’t be permanently stored in the application.

FUN FACT: Some of the chat messengers such as FB Messenger stores the chat messages unless the users explicitly delete it. However, instant messengers such as WhatsApp don’t save the messages permanently on their server.

Gathering Requirements

Designing Netflix

Problem Statement

Design a video streaming platform similar to Netflix where content creators can upload their video content and viewers are able to play video on different devices. We should also be able to store user statistics of the videos such as number of views, video watched duration, and so forth.

Gathering Requirements

1 2 3 … 30 Next »

Archive

How did the Swedbank incident happen?

Scaling Presto rapidly to meet growing demands: What challenges did we face?

Deploying new Presto releases

You can subscribe to the system design newsletter to excel in system design interviews and software architecture. The original article was published on systemdesign.one website.

What Is Gossip Protocol?

How does consistent hashing work?

Terminology

Thank you for agreeing to be interviewed today.

Explain cloud computing to me at a professional level?

My Stuff:

Number Stuff:

Who's Hiring?

My Stuff:

Number Stuff:

Who's Hiring?

Who's Hiring?

Who's Hiring?

My Stuff:

Number Stuff:

Requirements

In Scope

Problem Statement

Gathering Requirements

Problem Statement

Gathering Requirements

In Scope

Out of Scope

Problem Statement

Gathering Requirements

Problem Statement

Gathering Requirements