Archive

Category Archives for "High Scalability"

Sponsored Post: G-Core Labs, Close, Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble humans, then come check us out!

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Continue reading

Stuff The Internet Says On Scalability For July 11th, 2022

Never fear, HighScalability is here!

My Stuff:

  • Love this Stuff? I need your support on Patreon to help keep this stuff going.

  • Know anyone who needs to go from zero to cloud? My book teaches them all they need to know about the cloud. Explain the Cloud Like I'm 10. It has 440 mostly 5 star reviews on Amazon. Here's a 100% sincere, heartfelt review:

  • Do you like Zone 2, Zone 5, interval, REHIT, HIIT, and bodyweight workouts? I made an app for that. Max Workout. I’m not just the programmer, I’m a client. I use it 4 times a week and if you want to retrain your mitochondria, get fit, get healthy, and live longer, I think you’ll like it too.

Number Stuff:

Sponsored Post: G-Core Labs, Close, Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble humans, then come check us out!

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Continue reading

Sponsored Post: Close, Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble humans, then come check us out!

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Continue reading

Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Stuff The Internet Says On Scalability For January 28th, 2022

Never fear, HighScalability is here!


Think your software diagram is complex? This is a single cell modeled using X-ray, nuclear magnetic resonance (NMR), and cryo-electron microscopy datasets. Gael McGill

My Stuff:

  • Love this Stuff? I need your support on Patreon to keep this stuff going.
  • Know anyone who needs to fix their cloud-obliviousness? My book teaches them all they need to know about the cloud. Explain the Cloud Like I'm 10. It has 364 mostly 5 star reviews on Amazon. Here's a 100% antibody free review:
  • Do you like Zone 2, Zone 5, interval, reHIT, or HIIT workouts? I made an app for that. Max reHIT Workout. I’m not just the programmer, I’m a client. I use it 4 times a week and if you want to retrain your mitochondria, get fit, get healthy, and live longer, I think you’ll like it too.

Number Stuff:

Designing Uber

 

This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypotheses on Amazon product pages using reinforcement learning techniques. Currently, he is in the Alexa Shopping organization where he is developing machine-learning-based solutions to send personalized reorder hints to customers for improving their experience.

Requirements

In Scope

Designing Tinder

This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypotheses on Amazon product pages using reinforcement learning techniques. Currently, he is in the Alexa Shopping organization where he is developing machine-learning-based solutions to send personalized reorder hints to customers for improving their experience.

Problem Statement

Design a location-based social search application similar to Tinder which if often used as a dating service. It allows users to use a swiping motion to like (swipe right) or dislike (swipe left) other users, and allows users to chat if both parties like each other(a “match”).

Gathering Requirements

Designing Instagram

 

This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypotheses on Amazon product pages using reinforcement learning techniques. Currently, he is in the Alexa Shopping organization where he is developing machine-learning-based solutions to send personalized reorder hints to customers for improving their experience.

Problem Statement

Design a photo-sharing platform similar to Instagram where users can upload their photos and share it with their followers. Subsequently, the users will be able to view personalized feeds containing posts from all the other users that they follow.

Gathering Requirements

In Scope

The application should be able to support the following requirements.

  • Users should be able to upload photos and view the photos they have uploaded.
  • Users should be able to follow other users.
  • Users can view feeds containing posts from the users they follow.
  • Users should be able to like and comment the posts.

Out of Scope

Designing WhatsApp

 

This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypotheses on Amazon product pages using reinforcement learning techniques. Currently, he is in the Alexa Shopping organization where he is developing machine-learning-based solutions to send personalized reorder hints to customers for improving their experience.

Problem Statement

Design an instant messenger platform such as WhatsApp or Signal which users can utilize tosend messages to each other. An essential aspect of the application is that the chat messageswon’t be permanently stored in the application.

FUN FACT: Some of the chat messengers such as FB Messenger stores the chat messages unless the users explicitly delete it. However, instant messengers such as WhatsApp don’t save the messages permanently on their server.

Gathering Requirements

Designing Netflix

 

This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning Lead/Sr. Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. Ankit has been working on applying machine learning to solve ambiguous business problems and improve customer experience. For instance, he created a platform for experimenting with different hypotheses on Amazon product pages using reinforcement learning techniques. Currently, he is in the Alexa Shopping organization where he is developing machine-learning-based solutions to send personalized reorder hints to customers for improving their experience.

Problem Statement

Design a video streaming platform similar to Netflix where content creators can upload their video content and viewers are able to play video on different devices. We should also be able to store user statistics of the videos such as number of views, video watched duration, and so forth.

Gathering Requirements

Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Architecture of Max reHIT Workout

 

Today I have the pleasure of announcing my new app—Max reHIT Workout—on Product Hunt. Max reHIT Workout is an exercise app that guides you through interval workouts.

I won’t pitch the app here. I'll just say I’m proud of how it turned out and if you want an optimal algorithm for exercising, you might like it.

I know I haven’t been writing much lately. That's because there’s been very little evolution in software system architecture. It’s pretty much same thing, different day. In many ways that’s good, but it’s not interesting to write about.

This article, while definitely self serving, targets the choice of using a native iOS environment versus a cloud environment for an app. It’s a choice every developer must make. How do you make that choice? What are the implications? What choice would I make next time?

Out with the Old

Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Scaling indexing and search – Algolia New Search Architecture Part 2

What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the second article in a series. Here's Part 1.

Search engines need to support fast scaling for both Read and Write operations. Rapid scaling is essential in most use cases. For example, adding a vendor in a marketplace generates a spike of indexing operations (Write), and a marketing campaign generates a spike of queries (Read). In most use cases, both Read and Write operations scale but not at the exact same moment. The architecture needs to handle efficiently all these situations as the scaling of Read and Write operations varies over time in most use cases.

Until now, search engines were scaling with Read and Write operations colocated on the same VMs. This scaling method brings drawbacks, such asWrite operations unnecessarily hurting the Read performance and using a significant amount of duplicated CPU at indexing. This article explains those drawbacks and introduces a new way to scale more quickly and efficiently by splitting Read and Write operations.

1. Anatomy of an index

Stuff The Internet Says On Scalability For October 4th, 2021

Hey, HighScalability is here again!

The circulatory system of the internet. @tylermorganwall

Love this Stuff? I need your support on Patreon to keep this stuff going.

Sorry for the long gap in posting, but I’ve been building a new app. I’m looking for testers for my new iOS fitness app: Max reHIT Workout. It guides you through proven ​​reduced-exertion high-intensity interval workouts. If that interests you, please give it a try through TestFlight. I’d appreciate any feedback and suggestions for improvement. Thanks!

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Sponsored Post: Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and LXC, you should definitely check us out and apply here!

  • JavaScript Developer: Kinsta’s Development Team aims to create truly effective solutions for our customers and our internal teams. We believe the only way to maintain and scale Continue reading

Evolution of search engines architecture – Algolia New Search Architecture Part 1

 

What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the first article in a series.


Search engines, and more generally, information retrieval systems, play a central role in almost all of today’s technical stacks. Information retrieval started in the beginning of computer science. Research accelerated in the early 90s with the introduction of the Text REtrieval Conference (TREC). After more than 30 years of evolution since TREC, search engines continue to grow and evolve, leading to new challenges.

In this article, we look at some key milestones in the evolution of search engine architecture. We also describe the challenges those architectures face today. As you’ll see, we grouped the engines into four architecture categories. This is a simplification, as there are in reality a lot of different engines with various mix of architectures. We did this to focus our attention on the most important characteristics of those architectures.

1. The Inverted Index —  the early days of search

1 2 3 29