Archive

Category Archives for "High Scalability"

Stuff The Internet Says On Scalability For December 19th, 2020

Hey, it's HighScalability time once again!

 

NASA

 

Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen. 

 

Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 212 mostly 5 star reviews. Here's a load-balanced and fault-tolerant review:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: Toptal, IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • Looking to rapidly hire Top Software Developers? Get Started with Toptal. Toptal will match you with top-quality, pre-screened freelance software developers that meet your project requirements. All in under 48 hours. Get started right away with a no risk trial.

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care Continue reading

Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading

Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading

Stuff The Internet Says On Scalability For November 6th, 2020

Hey, no power outages this week, so it's finally HighScalability time!

 

Stunning: Tycho Crater Region with Colours by Alain Paillou

 

Do you like this sort of Stuff? Without your support on Patreon this Stuff won't happen. 

 

Know someone who could benefit from becoming one with the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 189 mostly 5 star reviews. Here's a 100% lactose-free review:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

ShiftLeft on Refactoring a Live SaaS Environment

 

This is guest a post by Preetam Jinka, Senior Infrastructure Engineer at ShiftLeft. Originally published here.

ShiftLeft NextGen Static Analysis (NG SAST) is a software-as-a-service static analysis solution that allows developers to scan every pull request for security issues. Earlier this year we released Secrets, Security Insights, and a v4 API. Secrets and Security Insights are two new types of results we extract from code analysis, and the V4 API is a brand new RESTful JSON API with an OpenAPI/Swagger specification that you can use to access all of your results. Read more about these features in our announcement post.

NG SAST was initially designed only for vulnerabilities. In order to implement Secrets and Security Insights, we either had to retrofit these new result types into our existing implementation or significantly refactor our back-end to support their unique characteristics. Even though it would take longer and be more difficult to implement, we decided to do the latter. We rewrote almost all of the storage used for storing code analysis results while maintaining backwards compatibility and without any outages. The analogy is that it’s like changing the engine on an airplane in flight without the passengers noticing.

We could’ve saved Continue reading

Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading

Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • IP2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading

Snakes in a Facebook Datacenter

 

What do you do when you find a snake in your datacenter? You might say this. (NSFW)

 

Fault tolerance is the property that enables a system to continue operating properly in the event of the failure of some of its components. You might think Facebook solved all of its fault tolerance problems long ago, but when a serpent enters the Edenic datacenter realm, even Facebook must consult the Tree of Knowledge.

In this case, it's not good or evil we'll learn about, but Workload Placement, which is a method of optimally placing work over a set of failure domains. 

Here's my gloss of the Fault Tolerance through Optimal Workload Placement talk:

How did they send money in the olden days of telegraphs?

 

It's not all that different really, especially that part where you can lose all your bitcoin. Here's an excerpt from East of Eden by John Steinbeck:

Say, Carlton, how do you go about telegraphing money?”

“Well, you bring me a hundred and two dollars and sixty cents and I send a wire telling the Valdosta operator to pay Adam one hundred dollars. You owe me sixty cents too.”

“I’ll pay—say, how do I know it’s Adam? What’s to stop anybody from collecting it?”

The operator permitted himself a smile of worldliness.

“Way we go about it, you give me a question couldn’t nobody else know the answer. So I send both the question and the answer. Operator asks this fella the question, and if he can’t answer he don’t get the money.””

“Say, that’s pretty cute. I better think up a good one.”
“You better get the hundred dollars while Old Breen still got the window open.”
Charles was delighted with the game. He came back with the money in his hand.
“I got the question,” he said.
“I hope it ain’t your mother’s middle name. Lot of people don’t remember.”
“No, nothing Continue reading

Stuff The Internet Says On Scalability For Sep 18th, 2020

Hey, it's HighScalability time!

 

I can't wait for the duel. Just don't shoot into the air.

Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff won't happen. 

Know someone who could benefit from becoming one with the cloud? Of course you do. I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 167 mostly 5 star reviews. Here's a 100% lectin-free review:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: IP2Location, Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • P2Location is IP address geolocation service provider since 2002. The geolocation database or API detects location, proxy and other >20 parameters. The technology has been cited in more than 700 research papers and trusted by many Fortune 500 companies. Try it today!

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how Continue reading

Moving HPC to the Cloud: A Guide for 2020

 

This is a guest post by Limor Maayan-Wainstein, a senior technical writer with 10 years of experience writing about cybersecurity, big data, cloud computing, web development, and more.

High performance computing (HPC) enables you to solve complex problems which cannot be solved by regular computing. Traditionally, HPC solutions provided mainly supercomputers. Today, HPC is typically a mix of resources, including supercomputing and virtualized and bare metal servers, platforms for management, sharing and integration capabilities, and more. When coupled with the cloud, HPC is made more affordable, accessible, efficient and shareable.

What Is HPC?

Sponsored Post: Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how engineering teams are using products like StackHawk and Snyk to add security bug testing to their CI pipelines.

  • Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that Continue reading

MySQL on Azure Performance Benchmark – ScaleGrid vs. Azure Database

Microsoft Azure is one of the most popular cloud providers in the world, and a natural fit for database hosting on applications leveraging Microsoft across their infrastructure. MySQL is the number one open source database that’s commonly hosted through Azure instances. While Microsoft offers their own Azure Database product, there are other alternatives available that may be able to help you improve your MySQL performance. In this blog post, we compare Azure Database for MySQL vs. ScaleGrid MySQL on Azure so you can see which provider offers the best throughput and latency performance. We measure latency in ms 95th percentile latency.

Sponsored Post: ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • ipdata is a reliable IP Address Geolocation API that allows you to lookup the approximate location of any IP Address, detect proxies and identify a company from an IP Address. Trusted by 10,000+ developers. Try it now!

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how engineering teams are using products like StackHawk and Snyk to add security bug testing to their CI pipelines.

  • Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that Continue reading

Stuff The Internet Says On Scalability For July 31st, 2020

Hey, it's HighScalability time!


 

Serverless is really complex. Or is it? @paulbiggar sparked a thoughtful Twitter thread.

Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff won't happen. 

Usually I pitch my book Explain the Cloud Like I'm 10 here, but if you're an author with books on Amazon I'd like to ask you to give Best Sellers Rank a try. I started it to send data to me about my own books, but in the best bootstrapping tradition I extended it to work for the entire planet.

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Sponsored Post: StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

Who's Hiring? 

  • InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Cool Products and Services

  • Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how engineering teams are using products like StackHawk and Snyk to add security bug testing to their CI pipelines.

  • Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years.

  • Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try Continue reading

Stuff The Internet Says On Scalability For June 26th, 2020

Hey, it's HighScalability time!


 

Line noise? Perl? Still uncertain? It's how you program a quantum computer. Silq.

Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the smart and thoughtful world.

Know someone who could benefit from understanding the cloud? Of course you do. I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 115 mostly 5 star reviews. Here's a 100% gluten-free review:

Number Stuff:

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

How TripleLift Built an Adtech Data Pipeline Processing Billions of Events Per Day

 

This is a guest post by Eunice Do, Data Engineer at TripleLift, a technology company leading the next generation of programmatic advertising.

What is the name of your system and where can we find out more about it?

The system is the data pipeline at TripleLift. TripleLift is an adtech company, and like most companies in this industry, we deal with high volumes of data on a daily basis. Over the last 3 years, the TripleLift data pipeline scaled from processing millions of events per day to processing billions. This processing can be summed up as the continuous aggregation and delivery of reporting data to users in a cost efficient manner. In this article, we'll mostly be focusing on the current state of this multi-billion event pipeline.

To follow the 5 year journey leading up to the current state, check out this talk on the evolution of the TripleLift pipeline by our VP of Engineering.

Why did you decide to build this system?