Stuff The Internet Says On Scalability For April 20th, 2018

Hey, it's HighScalability time:

 

Freeman Dyson dissects Geoffrey West's book on universal scaling laws, Scale. (Image: Steve Jurvetson)

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics. 

• 5x: BPF over iptables; 51.21%: SSL certificates now issued by Let's Encrypt; 15,000x: acceleration from a genomics co-processor on long read assembly; 100 Million: Amazon Prime members; 20 minutes: time it takes a robot to assemble an Ikea chair; 1.7 Tbps: DDoS Attack; 200 Gb/sec: future network fabric speeds; $7: average YouTube earnings per 1000 views; 800 million: viruses cascading onto every square meter of the planet each day; <10m: error in  Uber's GPS enhancement; $45 million: total value of Bitcoin ransomware extortion; 


• Quotable Quotes:
  • @sachinrekhi: Excited to read the latest Amazon shareholder letter. Amazing the scale they are operating at: 100M prime members, $20B AWS business, >50% of products sold from third-party sellers...Bezos Continue reading

Google: A Collection of Best Practices for Production Services

Fail Sanely

Sanitize and validate configuration inputs, and respond to implausible inputs by both continuing to operate in the previous state and alerting to the receipt of bad input. Bad input often falls into one of these categories:

Incorrect data

Validate both syntax and, if possible, semantics. Watch for empty data and partial or truncated data (e.g., alert if the configuration is N% smaller than the previous version).

Delayed data

This may invalidate current data due to timeouts. Alert well before the data is expected to expire.

Fail in a way that preserves function, possibly at the expense of being overly permissive or overly simplistic. We’ve found that it’s generally safer for systems to continue functioning with their previous configuration and await a human’s approval before using the new, perhaps invalid, data.

Examples

In 2005, Google’s global DNS load- and latency-balancing system received an empty DNS entry file as a result of file permissions. It accepted this empty file and served NXDOMAIN for Continue reading

Stuff The Internet Says On Scalability For April 13th, 2018

Hey, it's HighScalability time:

 

Bathroom tile? Grandma's needlepoint? Nope. It's a diagram of the dark web. Looks surprisingly like a tumor.

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics. 

• $23 billion: Amazon spend on R&D in 2017; $0.04: cost to unhash your email address; $35: build your own LIDAR; 66%: links to popular sites on Twitter come from bots; 60.73%: companies report JavaScript as primary language; 11,000+: object dataset provide real objects with associated depth information; 150 years: age of the idea of privacy; 30%~ AV1's better video compression; 100s of years: rare-earth materials found underneath Japanese waters; 67%: better image compression using Generative Adversarial Networks; 1000 bit/sec: data exfiltrated from air-gapped computers through power lines using conducted emissions; 


• Quotable Quotes:
  • @Susan_Hennessey: Less than two months ago, Apple announced its decision to move mainland Chinese iCloud data to state-run servers.
  • @PaulTassi Continue reading

Sponsored Post: InMemory.Net, Educative, Triplebyte, Exoscale, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

• Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• 5 Signs You’ve Outgrown DynamoDB. Companies often select a database that seems to be the best choice at first glance, as well as the path of least resistance, and then are subsequently surprised by cost overruns and technology limitations that quickly hinder productivity and put the business at risk. This seems to be the case with many enterprises that chose Amazon Web Service’s (AWS) DynamoDB. In this white paper we’ll cover elements of costing as well as the results of benchmark-based testing. Read 5 Signs You’ve Outgrown DynamoDB to determine if your organization has outgrown this technology.


• Advertise your event here!

Cool Products and Services

• InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC Continue reading

Give Meaning to 100 billion Events a Day – The Analytics Pipeline at Teads

This is a guest post by Alban Perillat-Merceroz, Software Engineer at Teads.tv.

In this article, we describe how we orchestrate Kafka, Dataflow and BigQuery together to ingest and transform a large stream of events. When adding scale and latency constraints, reconciling and reordering them becomes a challenge, here is how we tackle it.

Teads for Publisher, one of the webapps powered by Analytics

 

In digital advertising, day-to-day operations generate a lot of events we need to track in order to transparently report campaign’s performances. These events come from:

• Users’ interactions with the ads, sent by the browser. These events are called tracking events and can be standard (start, complete, pause, resume, etc.) or custom events coming from interactive creatives built with Teads Studio. We receive about 10 billion tracking events a day.
• Events coming from our back-ends, regarding ad auctions’ details for the most part (real-time bidding processes). We generate more than 60 billion of these events daily, before sampling, and should double this number in 2018.

In the article we focus on tracking events as they are on the most critical path of our business.

Simplified overview of our technical context with the two main Continue reading

Stuff The Internet Says On Scalability For April 6th, 2018

Hey, it's HighScalability time:

 

Programmable biology - engineered cells execute programmable multicellular full-adder logics. (Programmable full-adder computations)

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics. 

• $1: AI turning MacBook into a touchscreen; $2000/month: BMW goes subscription; 20MPH: 15′ Tall, 8000 Pound Mech Suit; 1,511,484 terawatt hours: energy use if bitcoin becomes world currency; $1 billion: Fin7 hacking group; 1.5 million: ethereum TPS, sort of; 235x: AWK faster than Hadoop cluster; 37%: websites use a vulnerable Javascript library; $0.01: S3, 1 Gig, 1 AZ; 


• Quotable Quotes:
  • Huang’s Law~ GPU technology advances 5x per year because the whole stack can be optimized. 
  • caseysoftware: Metcalfe lives here in Austin and is involved in the local startup community in a variety of ways.  One time I asked him how he came up with the law and he said something close Continue reading

Do you have too many microservices? – Five Design Attributes that can Help

This is a guest Post by Jake Lumetta, Founder and CEO, ButterCMS, an API-first CMS. For more content like this, follow @ButterCMS on Twitter and subscribe to our blog.

Are your microservices too small or too tightly coupled? Are you confident in your decision-making about service boundaries? In interviews with dozens of experienced CTOs, they offered design attributes that they consider when creating a set of microservices. This article distills that wisdom into five key principles to help you better design microservices.

The importance of microservice boundaries

The design attributes discussed below matter because reaping the benefits of microservices requires designing thoughtful microservice boundaries.

One of the major challenges when it comes to creating a new system with a microservice architecture. It came about when I mentioned that one of the core benefits of developing new systems with microservices is that the architecture allows developers to build and modify individual components independently — but problems can arise when it comes to minimizing the number of callbacks between each API. The solution according to McFadden, is to apply the appropriate service boundaries.

But in contrast to the sometimes difficult-to-grasp and abstract concept of domain driven design (DDD) —  a framework for Continue reading

How ipdata serves 25M API calls from 10 infinitely scalable global endpoints for $150 a month

This is a guest post by Jonathan Kosgei, founder of ipdata, an IP Geolocation API. 

I woke up on Black Friday last year to a barrage of emails from users reporting 503 errors from the ipdata API.

Our users typically call our API on each page request on their websites to geolocate their users and localize their content. So this particular failure was directly impacting our users’ websites on the biggest sales day of the year. 

I only lost one user that day but I came close to losing many more.

This sequence of events and their inexplicable nature — cpu, mem and i/o were nowhere near capacity. As well as concerns on how well (if at all) we would scale, given our outage, were a big wake up call to rethink our existing infrastructure.

Our Tech stack at the time

Stuff The Internet Says On Scalability For March 30th, 2018

Hey, it's HighScalability time:

 

Objective painting is not good painting unless it is good in the abstract sense. A hill or tree cannot make a good painting just because it is a hill or tree. It is lines and colors put together so that they may say something.” – Georgia O’Keeffe

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

 

• 6,000: new viri spotted by AI; 300,000: Uber requests per second; 10TB & 600 years: new next-gen optical disk; 32,000: sites running Coinhive’s JavaScript miner code; $1 billion: Uber loss per quarter; 3.5%: global NAND flash output lost to power outage; 100TB: new SSD; 48TB: RAM on one server; 200 million: Telegram monthly active users; 2,000: days Curiosity Rover on Mars; 225: emerging trends; 4,425: SpaceX satellites approved; 


• Quotable Quotes:
  • @msuriar: Uber's worst outage ever: - Master log Continue reading

Sponsored Post: Educative, Clover, Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

• Clover is looking for seasoned software engineers to help us solve the most complicated problem in the world: healthcare. We're using sophisticated data analytics, custom software, and machine learning to coordinate care and build a clearer model of our member's health and risk factors. We are on a mission to help seniors and low-income members live healthier while keeping costs down. This is an opportunity for those who want to be at the intersection of health and technology and thrive in a collaborative environment as well as the freedom of self-direction. If you're interested, please directly apply here!


• Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.


• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation Continue reading

Stuff The Internet Says On Scalability For March 16th, 2018

Hey, it's HighScalability time:

 

Hermetic symbolism was an early kind of programming. Symbols explode into layers of other symbols, like a programming language, only the instruction set is the mind.

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

• ~30: AWS services used by iRobot; 450,000: Shopify S3 operations per second; $240: yearly value of your data; ~day: time to load a terabyte from Postgres into BigQuery; 5 million: viewers for top Amazon Prime shows; 130,000: Airbusians move from Microsoft Office to Google Suite; trillion: rows per second processed by MemSQL; 38 million: Apple Music paid members; 4 million: Microsoft git commits for a Windows release; 


• Quotable Quotes:
  • Stephen Hawking: Although I cannot move and I have to speak through a computer, in my mind I am free.
  • Roger Penrose: Despite [Stephen Hawking] terrible physical circumstance, he almost always remained positive about life. He Continue reading

Sponsored Post: Clover, Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

• Clover is looking for seasoned software engineers to help us solve the most complicated problem in the world: healthcare. We're using sophisticated data analytics, custom software, and machine learning to coordinate care and build a clearer model of our member's health and risk factors. We are on a mission to help seniors and low-income members live healthier while keeping costs down. This is an opportunity for those who want to be at the intersection of health and technology and thrive in a collaborative environment as well as the freedom of self-direction. If you're interested, please directly apply here!


• Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.


• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation Continue reading

Stuff The Internet Says On Scalability For March 9th, 2018

Hey, it's HighScalability time:

• 72 bits: Google's new quantum computer; 50,000: sites infected with cryptocurrency mining malware; $40 billion: purchases via talking tubes by 2022; $12,000: value of 1 million YouTube views a month; $15 billion: Netflix 2018 revenue; 


• Quotable Quotes:
  • @ValaAfshar: Jeff Bezos, CEO @amazon: I very frequently get the question: "what's going to change in he next 10 years?" I almost never get the question: "what's not going to change in the next 10 years?" I submit to you that the second question is actually the more important of the two.
  • @svscarpino: Sharing and consuming fake news is highly concentrated. Continue reading

Stuff The Internet Says On Scalability For March 2nd, 2018

Hey, it's HighScalability time: 

 

Algorithms described like IKEA instructions. Can anyone assemble these? (Algorithms and data structures)

 

If you like this sort of Stuff then please support me on Patreon. And please consider recommending my new book—Explain the Cloud Like I'm 10—to whole entire world. 

 

• $75 million: Dropbox saved moving out of S3; 159 million: Spotify monthly active users; 80 million: more records added to Have I Been Pwned; 9%: universe expanding faster than predicted; $2,222,279: Warren Buffett won his long bet against hedge fund mangers; 60,000: Mayan houses found in Guatemala using LiDAR; $14.2 billion: PaaS revenue; ~180 million: years until first sun after the big whatever it was; $1,599: cost of stolen Extended Validation (EV) certificate; 8,000X: query speedup using GPU database; 2.4 million: Google requests to be forgotten; 6 minutes: time to IoT device attack on the internet; 103 million: tweets sent about the Olympics; 320,000: increase in Chloe Kim's twitter followers; 150 kg: acorns stored by woodpeckers in a telecom antenna; 0.14ms: Fsync performance on Intel PC-3700; Q: earliest known article on Wikipedia; 800Gbps+: memcached reflection/amplification attacks; Continue reading

Sponsored Post: Clover, Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

• Clover is looking for seasoned software engineers to help us solve the most complicated problem in the world: healthcare. We're using sophisticated data analytics, custom software, and machine learning to coordinate care and build a clearer model of our member's health and risk factors. We are on a mission to help seniors and low-income members live healthier while keeping costs down. This is an opportunity for those who want to be at the intersection of health and technology and thrive in a collaborative environment as well as the freedom of self-direction. If you're interested, please directly apply here!


• Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.


• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation Continue reading

Stuff The Internet Says On Scalability For February 23rd, 2018

Hey, it's HighScalability time: 

 

What does a bubble look like before it bursts? The image shows a brief period of stability before succumbing to molecular forces that pinch the film together and cause the bubble to burst.  (Mr Li Shen - Imperial College London)

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

 

• 20 million: daily DuckDuckGo searches, 55% growth; 38.3 billion: WeChat messages sent per day; 500 million: database of pwned passwords to check against; 38%: China's consumption of world IC production; 92%: Fortune 500 traffic is from bots; $2: Blue Pill: A 72MHz 32-Bit Computer; 5,000: Martian days of operation for the Opportunity Rover; 30.72 terabytes: Samsung SSD; 185%: Golden State Warriors drive up ticket prices; $1.3 billion: loss that happens when Kylie Jenner tweets about your new UI; 


• Quotable Quotes:
  • @slava_pestov: A senior programmer is merely a junior Continue reading

How fast can a bird search a tree?

I was wondering if you could help me figure something out: what is the algorithmic complexity of a bird searching a tree for food?

Over the years I've had the pleasure of watching a lot of cute little birds feed in our oak trees. I've noticed they have a search pattern.

A bird will hop from branch to branch looking for insects. They don't hop on a branch and explore every square inch of it, so it's not an exhaustive search. They'll take a couple hops, peck at a branch a few times, and hop to a nearby branch. Birds also search the underside of branches, so the whole surface area of a tree is game. 

I've often marveled in wonder at how efficient this whole process is. They scour huge trees in no time. Then they'll move on to the next tree and repeat the process until they fly away to a completely different area.

My dog when searching for a ball seems to follow a similar Lévy flight sort of pattern. Search a local area by bouncing around for bit and then take a bee-line for a completely different area and repeat the process. 

Often Continue reading

Stuff The Internet Says On Scalability For February 16th, 2018

Hey, it's HighScalability time: 

• 63.2%: Americans with one and only one cable provider; $1.5 billion: spend on chip startups last year; $7.5 billion: Uber sales; $4.5 billion: Uber loss; 180 TFLOPS: computation accessible via the TensorFlow programming model from a Google Cloud VM; 10x: computational capabilities of the human brain than previously thought; 10 million: went live on Facebook sharing 47% more Live videos than the previous year; 1.7 million: HQ players during Superbowl; 1/400: power to perform public-key encryption; 8 bit: custom CPU build from scratch; $8,500: daily take from mining Monero with your botnet; 10,000: datasets shared on Kaggle; 41%: NVIDIA revenue growth; 14x: real world 4G LTE vs. 5G Continue reading

Sponsored Post: Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

• Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.


• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume. 


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• 5 Signs You’ve Outgrown DynamoDB. Companies often select a database that seems to Continue reading

HelloFresh: Navigating the rough seas of environment scaling with Aho

In the past couple of years, our engineering team at HelloFresh has experienced a huge growth spurt. Within 2 years our team grew from 35 engineers to well over 150. One of the biggest challenges we faced was how to enable over 20 teams to test their code independently to other teams in a stable environment.

HelloTech Team Growth

I’m going to tell you about how we solved scaling our staging and local environments using phoenix environments.

The Problem

When we were smaller we had only 2 environments, what we called stagingand production. This setup was great for simple applications and small teams. Once we moved to a microservice architecture with 90+ services it quickly became apparent that it was not a scalable solution.

With more teams blocking staging for their testing, the environment quickly became unusable. The solution seemed simple, create an environment for each team called team staging.

Team Staging: Not the solution

We hoped that by creating a staging environment (which is a subset of all services from the main staging) for each team, it would enable them to test whatever they wanted in isolation. Or so we thought; in reality, it was much more complicated than that.

The Continue reading