Todd Hoff
Author Archives: Todd Hoff
- Home → Author's archive:
Todd Hoff
Sponsored Post: StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how engineering teams are using products like StackHawk and Snyk to add security bug testing to their CI pipelines.
- Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years.
- Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try Continue reading
Stuff The Internet Says On Scalability For June 26th, 2020
Hey, it's HighScalability time!
Line noise? Perl? Still uncertain? It's how you program a quantum computer. Silq.
Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the smart and thoughtful world.
Know someone who could benefit from understanding the cloud? Of course you do. I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 115 mostly 5 star reviews. Here's a 100% gluten-free review:
Number Stuff:
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...How TripleLift Built an Adtech Data Pipeline Processing Billions of Events Per Day
This is a guest post by Eunice Do, Data Engineer at TripleLift, a technology company leading the next generation of programmatic advertising.
What is the name of your system and where can we find out more about it?
The system is the data pipeline at TripleLift. TripleLift is an adtech company, and like most companies in this industry, we deal with high volumes of data on a daily basis. Over the last 3 years, the TripleLift data pipeline scaled from processing millions of events per day to processing billions. This processing can be summed up as the continuous aggregation and delivery of reporting data to users in a cost efficient manner. In this article, we'll mostly be focusing on the current state of this multi-billion event pipeline.
To follow the 5 year journey leading up to the current state, check out this talk on the evolution of the TripleLift pipeline by our VP of Engineering.
Why did you decide to build this system?
Sponsored Post: StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna
Who’s Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Developers care about shipping secure applications. Application security products and processes, however, have not kept up with advances in software development. There are a new breed of tools hitting the market that enable developers to take the lead on AppSec. Learn how engineering teams are using products like StackHawk and Snyk to add security bug testing to their CI pipelines.
- Learn the stuff they don’t teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you’d be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years.
- Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try Continue reading
Sponsored Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts Continue reading
A Short on How Zoom Works
Zoom scaled from 20 million to 300 million users virtually over night. What's incredible is from the outside they've shown little in the way of apparent growing pains, though on the inside it's a good bet a lot of craziness is going on.
Sure, Zoom has made some design decisions that made sense as a small spunky startup that don't make a lot of sense as a defacto standard, but that's to be expected. It's not a sign of bad architecture as many have suggested. It's just realistically how products evolve, especially when they must uplift over weeks, days, and even hours.
Sudden success invites scrutiny, so everyone wants to know how Zoom works. The problem is we don't know much, but we do have a few information sources:
Stuff The Internet Says On Scalability For May 14th, 2020
Hey, it's HighScalability time!
LOL. Who knew a birthday service could lead to an existential crisis?
Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the smart and thoughtful world.
Know someone who wants to understand the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 111 mostly 5 star reviews. Here's a recent 100% organic non-GMO review:
Number Stuff:
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...Sponsored Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts Continue reading
Part 2: The DynamoDB Book: An Interview With Alex DeBrie On His New Book
This is the second part of my interview with Alex DeBrie on his new insta-classic: The DynamoDB Book.
To read the first part of the interview please mosey on over to The DynamoDB Book: An Interview With Alex DeBrie On His New Book. Go ahead. Take your time. It's worth it.
Quick Fire Round. What's your quick reaction to:
Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Learn to balance architecture trade-offs and design scalable enterprise-level software. Check out Educative.io's bestselling new 4-course learning track: Scalability and Continue reading
The DynamoDB Book: An Interview With Alex DeBrie On His New Book
You know nothing about DynamoDB. At least that’s what I realized the first time I heard Rick Houlihan give his now infamous talk at AWS re:Invent 2018 on Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB.
In that talk Rick revealed for the first time the inner arcana of single-table design. Minds were blown. Weaknesses were revealed. Futures were changed.
As a mere novice in the ways of DynamoDB I realized there were many levels of understanding needed before one could become a true AWS Data Hero. For that we need a guide.
Our guide on the Hero’s Journey that is mastering DynamoDB is a wise young wizard named Alex DeBrie. Alex wrote what you might consider to be the Gnostic Gospels of DynamoDB: The DynamoDB Book.
You will know something after reading this book
But it's more than just a book. You can’t buy it on Amazon. Instead, Alex uses Gumroad to offer packages at three different price points along with a team option. Each level provides additional content:
- Basic package ($79). A 450 page book and six cheatsheets.
- Plus package ($129). 60-page Analytics supplement guide. 60-page Operations supplement guide. Five deployable code implementations.
- Premium Continue reading
Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Learn to balance architecture trade-offs and design scalable enterprise-level software. Check out Educative.io's bestselling new 4-course learning track: Scalability and Continue reading
Free Google Book: Building Secure and Reliable Systems
Google added another book into their excellent SRE series: Building Secure and Reliable Systems. It's free to download, so don't be shy.
It's not short: 557 pages and 21 chapters! So what's it about? In short it's about "reliability through the lens of security."
In long, Ana Oprea, one of the authors, gave a good overview. anaoprea:
There are multiple questions about what this book is about, who it's for and what might be relevant for me. We recommend going through the Preface to get answers to these questions. Copy/pasting a few paragraphs: "In this book we talk generally about systems, which is a conceptual way of thinking about the groups of components that cooperate to perform some function.
We wanted to write a book that focuses on integrating security and reliability directly into the software and system lifecycle, both to highlight technologies and practices that protect systems and keep them reliable, and to illustrate how those practices interact with each other.
We’d like to explicitly acknowledge that some of the strategies this book recommends require infrastructure support that simply may not exist where you’re currently working.
Because security and reliability are everyone’s responsibility, we’re Continue reading
One Team at Uber is Moving from Microservices to Macroservices
There may be an undiscovered tribe deep in some jungle somewhere that hasn’t made up their mind on microservices, but I doubt it. People love microservices or love to hate microservices. There’s not much in between.
So it means something when even a team at a company like Uber announces a change away from microservices to something else. What? Macroservices. But we’ll get to that. Think what you want about Uber the company, but from a software perspective Uber has been a good citizen.
Gergely Orosz, an Engineering Manager on the Payments Experience Platform at Uber, in a tweet signaled a change in architectural direction:
Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- InterviewCamp.io has hours of system design content. They also do live system design discussions every week. They break down interview prep into fundamental building blocks. Try out their platform.
- Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the Continue reading
Stuff The Internet Says On Scalability For March 27th, 2020
Hey, it's HighScalability time!
Awesome explanation of how to build a PID controller to fly a rocket! (BPS.space via Orbital Index)
Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the intelligent world.
Know someone who wants to understand the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 103 mostly 5 star reviews. Here's a recent authentic unfaked review:
Number Stuff:
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...Post: Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- At Scrapinghub you will be designing and implementing distributed systems: large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. - this is going to be a challenging journey for any backend engineer! Please apply here.
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also Continue reading
Post: Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber.
- Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with FaunaDB. Generous free tier. Try now!
- PA File Sight - Actively protect servers from ransomware, audit file access to see who is deleting files, reading files or moving files, Continue reading
Stuff The Internet Says On Scalability For March 13th, 2020
p>Hey, it's HighScalability time!
The top 10,000 most spoken words in English represented by a point in hundreds of dimensions where the distance and direction between points encodes the relationship between words. (roadmaps)
Do you like this sort of Stuff? Without your support on Patreon this kind of Stuff can't happen. You are that important to the fate of the intelligent world.
Know someone who wants to understand the cloud? I wrote Explain the Cloud Like I'm 10 just for them. On Amazon it has 100 (!!!) mostly 5 star reviews. Here's a recent authentic unfaked review:
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...
Post: Essilen Research, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream
Who's Hiring?
- Are you looking to improve how you hire technical talent? Essilen Research's free video series can help. Fix common issues in screening, interviewing, closing and on-boarding. Learn how world-class tech companies crush the hiring game!
- Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. Apply here.
- Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.
- Need excellent people? Advertise your job here!
Cool Products and Services
- Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. For the first time ever, you can now sign up for a subscription to get unlimited access to every course on the platform at a discounted price through the holiday period only. You'll also get to lock in this price as long as you're a subsciber.
- Stateful JavaScript Apps. Effortlessly add state Continue reading