Todd Hoff

Author Archives: Todd Hoff

Stuff The Internet Says On Scalability For April 19th, 2019

Wake up! It's HighScalability time:

 

Spirit? Smoke? Lightning? Nope. It's a gorgeous LIDAR image showing 1500 years of Willamette River movement (@Blacky_Himself)

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. I wrote Explain the Cloud Like I'm 10 for people who need to understand the cloud. And who doesn't these days? On Amazon it has 44 mostly 5 star reviews (102 on Goodreads). They'll learn a lot and hold you in awe.

 

  • 536: IRS tax return submissions per second; 400,000: drone planted trees in a day; 200 million: smart speaker observers installed by year end; 54 million: GoT pirated in first 24 hours; 123,052: kg of crashed human spacecraft on the surface of the moon; 610 pounds: 128 kilobytes of  IBM S/360 core memory; 33%: per account month over month Lambda function growth; $100,000: Netflix bug bounty payout; $1 million: Shopify bug bount payout; ~$2300: cost to transfer 23TB from S3 to Backblaze B2 in 7 hours; 14%: Netflix users share passwords; 88%: believe P != NP; 30-90: minutes saved by StackOverflow per week; 83%: US teens have Continue reading

Sponsored Post: PerfOps, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr

Who's Hiring? 


  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Join Etleap, an Amazon Redshift ETL tool to learn the latest trends in designing a modern analytics infrastructure. Learn what has changed in the analytics landscape and how to avoid the major pitfalls which can hinder your organization from growth. Watch a demo and learn how Etleap can save you on engineering hours and decrease your time to value for your Amazon Redshift analytics projects. Register for the webinar today.

  • Advertise your event here!

Cool Products and Services

  • PerfOps is a data platform that digests real-time performance data for CDN and DNS providers as measured by real users worldwide. Leverage this data across your monitoring efforts and integrate with PerfOps’ other tools such as Alerts, Health Monitors and FlexBalancer – a smart approach to load balancing. FlexBalancer makes it easy to manage traffic between multiple CDN providers, API’s, Databases or any custom endpoint helping you achieve better performance, ensure the availability Continue reading

Sponsored Post: PerfOps, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr

Who's Hiring? 


  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Join Etleap, an Amazon Redshift ETL tool to learn the latest trends in designing a modern analytics infrastructure. Learn what has changed in the analytics landscape and how to avoid the major pitfalls which can hinder your organization from growth. Watch a demo and learn how Etleap can save you on engineering hours and decrease your time to value for your Amazon Redshift analytics projects. Register for the webinar today.

  • Advertise your event here!

Cool Products and Services

  • PerfOps is a data platform that digests real-time performance data for CDN and DNS providers as measured by real users worldwide. Leverage this data across your monitoring efforts and integrate with PerfOps’ other tools such as Alerts, Health Monitors and FlexBalancer – a smart approach to load balancing. FlexBalancer makes it easy to manage traffic between multiple CDN providers, API’s, Databases or any custom endpoint helping you achieve better performance, ensure the availability Continue reading

Stuff The Internet Says On Scalability For April 5th, 2019

Wake up! It's HighScalability time:

 

How unhappy do you have to be as a customer to take so much joy in end-of-lifing a product?

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. I wrote Explain the Cloud Like I'm 10 for people who need to understand the cloud. And who doesn't these days? On Amazon it has 44 mostly 5 star reviews (100 on Goodreads). They'll learn a lot and love you for the hookup.

 

  • $40 million: Fortnite World Cup prize money; 89%: of people who like Go say they like Go; 170 million: paid iCloud accounts; 533: days bacteria lived on the outside of ISS; 95%: BTC volume is fake; 51: LTE vulnerabilities found by fuzzing; 13,000: CRISPR edits in a single cell; 5G: 762Mbps down and a 19ms ping; 17,000: awesome Historic Blues & Folk Recordings; 3,236: Amazon broadband LEO satellite network; 5.1 million: emails sent during 10 day spam campaign; 

  • Quoteable Quotes:

Stuff The Internet Says On Scalability For March 29th, 2019

Wake up! It's HighScalability time:

 

Uber's microservice Graph. Thousands of microservices. Crazy like a fox? Or just crazy? (@msuriar)

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. I wrote Explain the Cloud Like I'm 10 for people who need to understand the cloud. And who doesn't these days? On Amazon it has 42 mostly 5 star reviews (100 on Goodreads). They'll learn a lot and love you for the hookup.

 

  • 1.5 billion: monthly What's App users; 80 billion: docker downloads in 6 years; 1 billion: players on the App Store. 300,000 games; 13.5 billion: Voyager 1 miles from earth; 11 years: Teeny-Tiny Bluetooth Transmitter; 500 million: Airbnb guests; 12.5 million bits: information learned by average adult; 7.7%: Amazon's share of US retail sales; 100 million: Stack Overflow monthly visitors; $156B: Consumer spending in apps across iOS and Google Play by 2023; 

  • Quotable Quotes:
    • John C. Lilly: When I say we may be our programs, nothing more, nothing less, I mean the substrate, the basic substratum under all else of our metaprograms is our programs. All we are as Continue reading

Stuff The Internet Says On Scalability For March 22nd, 2019

Wake up! It's HighScalability time:

 

Van Gogh? Nope. A satellite image of phytoplankton populations or algae blooms in the Baltic Sea. (NASA)

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. It has 41 mostly 5 star reviews. They'll learn a lot and love you even more.

 

  • .5 billion: weekly visits to Apple App store; $500m: new US exascale computer; $1.7 billion: EU's newest fine on Google; flat: 2018 global box office; up: digital rentals, sales, and subscriptions to streaming services; 43%: Dutch say let AI run the country; 2 min: time from committing an AWS key to Github to first attack; 40: bugs in blockchain platforms found in 30 days; 100%: Norwegian government 2025 EV target; 

  • Quotable Quotes:
    • @raganwald: You know what would really help restore confidence? Have members of the executive leadership team, and Boeing’s board of directors, fly on 737 Max’s, every day, for a month. Let them dogfood the software patch. I am 💯 serious.
    • skamille: I worry that the cloud Continue reading

Sponsored Post: Software Buyers Council, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr

Who's Hiring? 


  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Join Etleap, an Amazon Redshift ETL tool to learn the latest trends in designing a modern analytics infrastructure. Learn what has changed in the analytics landscape and how to avoid the major pitfalls which can hinder your organization from growth. Watch a demo and learn how Etleap can save you on engineering hours and decrease your time to value for your Amazon Redshift analytics projects. Register for the webinar today.

  • Advertise your event here!

Cool Products and Services

  • Shape the future of software in your industry. The Software Buyers Council is a panel of engineers and managers who want to share expert knowledge, contribute to improvement of software, and help startups in their industry. Receive occasional invitations to chat with for 30 minutes about your area of expertise and software usage. No obligations, no marketing emails or sales calls. Upcoming topics include infrastructure and application monitoring, AI/ML platforms, and more. Learn Continue reading

Stuff The Internet Says On Scalability For March 15th, 2019

Wake up! It's HighScalability time:

 

The web is 30! Some say it's not the web we wanted. But if we got that web, would it have ever grown so big? Worse usually is better.

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. It has 40 mostly 5 star reviews. They'll learn a lot and love you even more.

 

  • 300%: AWS IoT growth per year; 74%: mobile games user spending in the App store; 31.4 trillion: new record for calculating digits of pi (121 days); 112Gbps: Intel's SerDes; 100M: image and video dataset; 1.5 trillion suns: weight of the Milky Way; 300+: backdoored apps on GitHub; 10%: hacked self-driving cars needed to bring traffic to a halt; $3 million: Marriott data breach cost after insurance; 

  • Quoteable Quotes:
    • @kelseyhightower: Platform in a box solutions that are attempting to turn Kubernetes into a PaaS are missing the "as a service" part. It's more like PaaR: Platform as a Responsibility. Your responsibility to purchase, staff, patch, scale, and Continue reading

Stuff The Internet Says On Scalability For March 8th, 2019

Wake up! It's HighScalability time:

 

A highly simplified diagram of serverless. (@jbesw)

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. It has 40 mostly 5 star reviews. They'll learn a lot and love you even more.

 

  • 5%: France's new digital tax revolution; $15 trillion: AI contribution to global GDP by 2030; 70%: better response time using HTTP keep-alive in lambda; 115 million: Akamai found bots (per day) compromising user accounts by credential stuffing; 83%: of all internet traffic is API calls, not HTML; $1 million: first millionaire bug-bounty hacker is 19 years old; 15%: mooch their Netflix account; 5%: Microsoft's app store take; $15: Tensorflow at the edge; 30%: first quarter drop in DRAM prices; $2 billion: IBM's microkernel folly; ~1TWh: lithium-ion batteries production per year by 2030; 25%: Tesla supercharger time improvement by a software update; 

  • Quoteable Quotes:
    • Jeff Bezos: I've witnessed this incredible thing happen on the internet over the last two decades. I started Amazon in my Continue reading

Sponsored Post: Software Buyers Council, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr

Who's Hiring? 


  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Join Etleap, an Amazon Redshift ETL tool to learn the latest trends in designing a modern analytics infrastructure. Learn what has changed in the analytics landscape and how to avoid the major pitfalls which can hinder your organization from growth. Watch a demo and learn how Etleap can save you on engineering hours and decrease your time to value for your Amazon Redshift analytics projects. Register for the webinar today.

  • Advertise your event here!

Cool Products and Services

  • Shape the future of software in your industry. The Software Buyers Council is a panel of engineers and managers who want to share expert knowledge, contribute to improvement of software, and help startups in their industry. Receive occasional invitations to chat with for 30 minutes about your area of expertise and software usage. No obligations, no marketing emails or sales calls. Upcoming topics include infrastructure and application monitoring, AI/ML platforms, and more. Learn Continue reading

How is software developed at Amazon?

 

How is software developed at Amazon? Get a couple of prime pizzas delivered and watch this excellent interview with Ken Exner, GM of AWS Developer Tools. It's notable Ken is from the tools group, because progress in an industry is almost always made possible by the development of better tools.

The key themes from the talk: decomposition, automation, and organize around the customer.

The key idea:

Scaling is by mitosis. Teams split apart into smaller teams that completely own a service. EC2 started as one two pizza team. 

This quote nicely embodies all three of the themes and is the key reason AWS keeps on winning the public cloud. Bottom up, Amazon adaptively grows their entire organization in response to customer inputs. 

And here's a short gloss of the talk...

Stuff The Internet Says On Scalability For March 1st, 2019

Wake up! It's HighScalability time:

 

10 years of AWS architecture increasing simplicity or increasing complexity? (Michael Wittig)

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. It has 39 mostly 5 star reviews. They'll learn a lot and love you forever.

 

  • 1.3 billion: npm package downloads per day; 20: honeybee communication signals used to coordinate thousands of workers; 71: average global life expectancy; 120K: max inflight SQS messages; 80%: shared code between iOS, Android, the web; 1 TB: microSD card; 20%: increase in value wind energy using ML; 64%: respondents cite optimizing cloud spend as the topvinitiative; 250: drones augmenting small military units; 35,880: record robots shipped to North American companies; 50K: aerial photos of the UK; 119%: increase in demand for AI talent; 18TB: MAMR hard drive; $20 million: Pinterest paid more than expected for AWS; 100,000: MySQL connections; 19%: all requests come from Bots, APIs, and search engine crawlers; 

  • Quotable Quotes:

Give Meaning to 100 Billion Events a Day — The Shift to Redshift

In part one, we described our Analytics data ingestion pipeline, with BigQuery sitting as our data warehouse. However, having our analytics events in BigQuery is not enough. Most importantly, data needs to be served to our end-users.

TL;DR — Teads Analytics big picture

In this article, we will detail:

  • Why we chose Redshift to store our data marts,
  • How it fits into our serving layer,
  • Key learnings and optimization tips to make the most out of it,
  • Orchestration workflows,
  • How our data visualization apps (Chartio, web apps) benefit from this data.

Data is in BigQuery, now what?

Design Of A Modern Cache—Part Deux

This is a guest post by Benjamin Manes, who did engineery things for Google and is now doing engineery things as CTO of Vector.

The previous article described the caching algorithms used by Caffeine, in particular the eviction and concurrency models. Since then we’ve made improvements to the eviction algorithm and explored a new approach towards expiration.

Eviction Policy

Window TinyLFU (W-TinyLFU) splits the policy into three parts: an admission window, a frequency filter, and the main region. By using a compact popularity sketch, the historic frequencies are cheap to retain and lookup. This allows for quickly discarding new arrivals that are unlikely to be used again, guarding the main region from cache pollution. The admission window provides a small region for recency bursts to avoid consecutive misses when an item is building up its popularity.

 

 

This structure works surprisingly well for many important workloads like database, search, and analytics. These cases are frequency-biased where a small admission window is desirable to filter aggressively...

Stuff The Internet Says On Scalability For February 22nd, 2019

Wake up! It's HighScalability time:

 

Isn't inetd a better comp? (link)

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. It has 39 mostly 5 star reviews. They'll learn a lot and love you forever.

 

  • 2%: of sales spent by consumer packaged goods companies on R&D (14% for tech); 272 million: metric tons of plastic are produced each year around the globe; 100+ fps: Google's Edge TPU; 6,000: bugs per million lines of code; 2.2 GB/sec: SIMD JSON parser; 20-30%: fall in DRAM prices; 8x: Russian hackers faster than North Korean hackers; 50%: EV car sales in China by 2025;

  • Quoteable Quotes:
    • @davygreenberg: If I do a job in 30 minutes it’s because I spent 10 years learning how to do that in 30 minutes. You owe me for the years, not the minutes.
    • @PaulDJohnston: Lambda done badly is still better than Kubernetes done well
    • Ross Mcilroy: we now believe that speculative vulnerabilities on today's hardware defeat all language-enforced confidentiality with no known Continue reading

Sponsored Post: Software Buyers Council, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr

Who's Hiring? 


  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Join Etleap, an Amazon Redshift ETL tool to learn the latest trends in designing a modern analytics infrastructure. Learn what has changed in the analytics landscape and how to avoid the major pitfalls which can hinder your organization from growth. Watch a demo and learn how Etleap can save you on engineering hours and decrease your time to value for your Amazon Redshift analytics projects. Register for the webinar today.

  • Advertise your event here!

Cool Products and Services

  • Shape the future of software in your industry. The Software Buyers Council is a panel of engineers and managers who want to share expert knowledge, contribute to improvement of software, and help startups in their industry. Receive occasional invitations to chat with for 30 minutes about your area of expertise and software usage. No obligations, no marketing emails or sales calls. Upcoming topics include infrastructure and application monitoring, AI/ML platforms, and more. Learn Continue reading

Stuff The Internet Says On Scalability For February 15th, 2019

Wake up! It's HighScalability time:

 

Opportunity crossed over the rainbow bridge after 15 years of loyal service. "Our beloved Opportunity remains silent." 

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. It has 39 mostly 5 star reviews. They'll learn a lot and love you forever.

 

  • 200 million: per day YouTube videos recommended on home page; $9.3 billion: 27% increase in AI funding; 70%: Microsoft security bugs are memory safety issues; 11: new version of Perl; 24%: serverless users are new to cloud computing; 1 million: SpaceX satellite uplinks; $500K: ticket to mars; $13 billion: Google's new datacenter construction; 59%: increase in Tesla Autosteer accidents; $.30: reddit per user revenue; 38%: Airbnb bugs preventable by using types; 60K: data breaches reported since GDPR; 350: theoretical max rock stone skips;

  • Quoteable Quotes:
    • @gchaslot: Brian's hyper-engagement slowly biases YouTube: 1/ People who spend their lives on YT affect recommendations more 2/ So the content they watch gets more views 3/ Continue reading

Stuff The Internet Says On Scalability For February 8th, 2019

Wake up! It's HighScalability time:

 

Change is always changing. What will the next 5 years look like?

 

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. It has 35 mostly 5 star reviews. They'll learn a lot and love you forever.

 

  • 16,000: Chrome bugs found with ClusterFuzz;  $2,000,000: for Apple iOS remote jailbreak; $1 million: think twice when profiting from a bug; 0: clicks to over the air explotation of Marvell Avastar Wi Fi; $300: cost for a bounty hunter to track your phone's location; 321M: Twitter MAUs; 3: years of falling smartphone shipments; 50%: new development uses microservices; 8 inches: big difference in cell phone radiation; ...
  • Quoteable Quotes:
    • @pczarkowski: As I keep telling people, if you have a kubernetes strategy you've already failed. Kubernetes should be an implementation detail at the tactical level to deal with the strategic imperative of solving the problems that are halting the flow of money.
    • EFF: EU countries that do not have zero rating practices enjoyed a double digit drop Continue reading
1 2 3 21