Archive

Category Archives for "High Scalability"

Scaling indexing and search – Algolia New Search Architecture Part 2

What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the second article in a series. Here's Part 1.

Search engines need to support fast scaling for both Read and Write operations. Rapid scaling is essential in most use cases. For example, adding a vendor in a marketplace generates a spike of indexing operations (Write), and a marketing campaign generates a spike of queries (Read). In most use cases, both Read and Write operations scale but not at the exact same moment. The architecture needs to handle efficiently all these situations as the scaling of Read and Write operations varies over time in most use cases.

Until now, search engines were scaling with Read and Write operations colocated on the same VMs. This scaling method brings drawbacks, such asWrite operations unnecessarily hurting the Read performance and using a significant amount of duplicated CPU at indexing. This article explains those drawbacks and introduces a new way to scale more quickly and efficiently by splitting Read and Write operations.

1. Anatomy of an index

Stuff The Internet Says On Scalability For October 4th, 2021

Hey, HighScalability is here again!

The circulatory system of the internet. @tylermorganwall

Love this Stuff? I need your support on Patreon to keep this stuff going.

Sorry for the long gap in posting, but I’ve been building a new app. I’m looking for testers for my new iOS fitness app: Max reHIT Workout. It guides you through proven ​​reduced-exertion high-intensity interval workouts. If that interests you, please give it a try through TestFlight. I’d appreciate any feedback and suggestions for improvement. Thanks!

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up.

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and Continue reading

Sponsored Post: Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and LXC, you should definitely check us out and apply here!

  • JavaScript Developer: Kinsta’s Development Team aims to create truly effective solutions for our customers and our internal teams. We believe the only way to maintain and scale Continue reading

Evolution of search engines architecture – Algolia New Search Architecture Part 1

 

What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the first article in a series.


Search engines, and more generally, information retrieval systems, play a central role in almost all of today’s technical stacks. Information retrieval started in the beginning of computer science. Research accelerated in the early 90s with the introduction of the Text REtrieval Conference (TREC). After more than 30 years of evolution since TREC, search engines continue to grow and evolve, leading to new challenges.

In this article, we look at some key milestones in the evolution of search engine architecture. We also describe the challenges those architectures face today. As you’ll see, we grouped the engines into four architecture categories. This is a simplification, as there are in reality a lot of different engines with various mix of architectures. We did this to focus our attention on the most important characteristics of those architectures.

1. The Inverted Index —  the early days of search

Sponsored Post: Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and LXC, you should definitely check us out and apply here!

  • JavaScript Developer: Kinsta’s Development Team aims to create truly effective solutions for our customers and our internal teams. We believe the only way to maintain and scale Continue reading

Sponsored Post: Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and LXC, you should definitely check us out and apply here!

  • JavaScript Developer: Kinsta’s Development Team aims to create truly effective solutions for our customers and our internal teams. We believe the only way to maintain and scale Continue reading

Stuff The Internet Says On Scalability For June 25th, 2021

Hey, it's HighScalability time!


 Only listen if you want a quantum earworm for the rest of the day.

 

Not your style? This is completely different. No, it’s even more different than that.

 

Love this Stuff? I need your support on Patreon to help keep this stuff going.

 

Do employees at your company need to know about the cloud? My book will teach them all they need to know. Explain the Cloud Like I'm 10. On Amazon it has 307 mostly 5 star reviews. Here's a 100% keto paleo low carb carnivore review:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and LXC, you should definitely check us out and apply here!

  • JavaScript Developer: Kinsta’s Development Team aims to create truly effective solutions for our customers and our internal teams. We believe the only way to maintain and scale Continue reading

Sponsored Post: Failover Conf!, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Fun and Interesting Events

  • Since we’re all feeling the fatigue of staring at screens all year, we’re switching things up at this year’s Failover Conf! This one-day conference will be featuring LIVE fireside chats and keynotes where you’ll be able to get your questions answered in real time. You’ll also hear from a variety of industry experts in our two panel discussions. And when you need a break, take a load off and watch some cartoons or listen to music in the breakout rooms where you can jump in and chat with other attendees or just sit back and relax. Join your peers for this virtual experience on April 27 from 9am - 3:30pm PDT. Register now!

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have Continue reading

Stuff The Internet Says On Scalability For April 30th, 2021

Hey, HighScalability is back!

This channel is the perfect blend of programming, hardware, engineering, and crazy. After watching you’ll feel inadequate, but in an entertained sort of way.

 

Love this Stuff? I need your support on Patreon to keep this stuff going.

 

Do employees at your company need to know about the cloud? My book will teach them all they need to know. Explain the Cloud Like I'm 10. On Amazon it has 285 mostly 5 star reviews. Here's a 100% bipartisan review:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: Failover Conf!, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

Fun and Interesting Events

  • Since we’re all feeling the fatigue of staring at screens all year, we’re switching things up at this year’s Failover Conf! This one-day conference will be featuring LIVE fireside chats and keynotes where you’ll be able to get your questions answered in real time. You’ll also hear from a variety of industry experts in our two panel discussions. And when you need a break, take a load off and watch some cartoons or listen to music in the breakout rooms where you can jump in and chat with other attendees or just sit back and relax. Join your peers for this virtual experience on April 27 from 9am - 3:30pm PDT. Register now!

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have Continue reading

Sponsored Post: Kinsta, 3T, Bridgecrew, Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta’s DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here.

  • SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and LXC, you should definitely check us out and apply here!

  • JavaScript Developer: Kinsta’s Development Team aims to create truly effective solutions for our customers and our internal teams. We believe the only way to maintain and scale Continue reading

Stuff The Internet Says On Scalability For March 17th, 2021

Hey, HighScalability is here again!

 

Reverse engineering an ancient analog computer is a detective story worth reading. A Model of the Cosmos in the ancient Greek Antikythera Mechanism.

 

Do you love this Stuff? Without your encouragement on Patreon this Stuff won't stuffin’ stuff. 

 

Know someone who needs to know the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 262 mostly 5 star reviews. Here's a review that is not on the block chain:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: 3T, Bridgecrew, Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Discover the MongoDB data masking tool in Studio 3T Enterprise. Enable data compliance and bolster security with powerful field-level data obfuscation. Try for free!

  • Bridgecrew is the cloud security platform for developers. By leveraging automation and delivering security-as-code, Bridgecrew empowers teams to find, fix, and prevent misconfigurations in deployed cloud resources and in infrastructure as code. Get started for free!

  • IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. Continue reading

Engineering dependability and fault tolerance in a distributed system

 

This is a guest post by Paddy Byers, Co-founder and CTO at Ably, a realtime data delivery platform. You can view the original article on Ably's blog.

Users need to know that they can depend on the service that is provided to them. In practice, because from time to time individual elements will inevitably fail, this means you have to be able to continue in spite of those failures.

In this article, we discuss the concepts of dependability and fault tolerance in detail and explain how the Ably platform is designed with fault tolerant approaches to uphold its dependability guarantees.

As a basis for that discussion, first some definitions:

Dependability
The degree to which a product or service can be relied upon. Availability and Reliability are forms of dependability.

Availability
The degree to which a product or service is available for use when required. This often boils down to provisioning sufficient redundancy of resources with statistically independent failures.

Reliability
The degree to which the product or service conforms to its specification when in use. This means a system that is not merely available but is also engineered with extensive redundant measures to continue to work as its Continue reading

Sponsored Post: 3T, Bridgecrew, Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Discover the MongoDB data masking tool in Studio 3T Enterprise. Enable data compliance and bolster security with powerful field-level data obfuscation. Try for free!

  • Bridgecrew is the cloud security platform for developers. By leveraging automation and delivering security-as-code, Bridgecrew empowers teams to find, fix, and prevent misconfigurations in deployed cloud resources and in infrastructure as code. Get started for free!

  • IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. Continue reading

Benchmark (YCSB) numbers for Redis, MongoDB, Couchbase2, Yugabyte and BangDB

This is guest post by Sachin Sinha who is passionate about data, analytics and machine learning at scale. Author & founder of BangDB.

This article is to simply report the YCSB bench test results in detail for five NoSQL databases namely Redis, MongoDB, Couchbase, Yugabyte and BangDB and compare the result side by side. I have used latest versions for each NoSQL DB and have followed the recommendations to run all the databases in optimized conditions. I have also used the default six test scenarios as defined by the YCSB framework. I have restricted it to 10M records for each test. However, user can run the bench for as many numbers as they practically find suitable.

About YCSB

Following configurations were used for the evaluation purpose.

Each of these workload test runs in two steps, 1. Load and 2. Run. Load stage is to load the data and then run stage we run the test. I have run each test with Continue reading

1 2 3 29