Stuff The Internet Says On Scalability For February 9th, 2018

Hey, it's HighScalability time: 

• 7.2 terabytes: data used during Super Bowl;  $220 million: projected podcast revenue this year; $127 billion: total addressable value of drone-powered solutions in all applicable industries; 100 billion billion billion: living microbial cells underlying all the worlds oceans, 200x biomass of humans; 110 billion: total market for memory; 1,000: drones North Korea may have, possibly with chemical or biological weapons, ready to attack South Korea; $123 billion: US apparel market; 100 million: iOS devices sold in Q4; $3: earnings in 24 hours from conscripting 5,000 Android devices into a mining botnet; 46%: cloud market growth in Q4; 104%: YoY Alibaba cloud growth; 


• Quotable Quotes:
  • John Perry Barlow: The Internet is the Continue reading

Stuff The Internet Says On Scalability For February 2nd, 2018

Hey, it's HighScalability time: 

• 2 billion: Siri requests per week; 1 trillion: semiconductor unit shipments, 9.1% compound annual growth rate over a 40 year span; 150 million: IPv4 addresses recirculated over 7 years; $100 Billion:  value lost in cryptocurrency markets in 24 hours; $1: microcontroller; 16 Gb/s: GDDR6 SGRAM; 1/3: IPv4 addresses registered to US entities; eight minutes and thirty-five seconds: breath-hold record; $32.32 billion: Google revenue, up 24%; 1.3 billion: active Apple devices, up 30% in 2 years; 500 petabytes: Backblaze; 6th or 7th: HTTP version of hypertex on the internet; $2.1 billion: Amazon's Q4 2017 operating income, up 69%; 45%: jump in Amazon cloud revenue; 9%: global smartphone market drop; 3 billion: photos uploaded to Google on New Year's eve; Continue reading

Sponsored Post: Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, Zohocorp

Who's Hiring? 

• Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.


• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume. 


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• Webinar: January 23, 11am GMT & again at 11am Pacific / 2pm Eastern. Continue reading

Stuff The Internet Says On Scalability For January 26th, 2018

Hey, it's HighScalability time: 

• $172 billion: amount Hackers stole in 2017; $100B: Netflix's value; 400G: ethernet; 2005: resurrection of a NASA satellite; 800: kilograms of oil equivalent needed per capita per year;


• Quotable Quotes:
  • Edgar D. Mitchell: You develop an instant global consciousness, a people orientation, an intense dissatisfaction with the state of the world, and a compulsion to do something about it. From out there on the moon, international politics look so petty. You want to grab a politician by the scruff of the neck and drag him a quarter of a million miles out and say, ‘Look at that, you son of a bitch.”
  • Laura Gilmore: It wasn’t by accident that I ended up Continue reading

How Apple Fumbled the Voice First Future

Xerox fumbled the future when it invented and then ignored the personal computer. With the Macintosh, Apple created the personal computer the Xerox Alto might have been.

Apple is also fumbling the future—the Voice First future. Voice First simply means our primary mode of interacting with computers in the future will be with our voice. When Apple bought Siri it had a solid 5 year lead in voice control. Now Amazon’s Alexa and Google’s Assistant have not only caught Siri, but they’ve surpassed her.

The story of how Apple is fumbling the Voice First future is passionately told by Brian Roemmele in a great interview with Rene Ritchie in his Vector podcast Why Siri needs to be a platform.

Brian covers a lot of ground in the interview, but there are a few main themes: Voice First is the Future; Apple Fumbled Voice First; Engineering First Cultures Suck at Product; Apple Needs to Lose the iPhone Tax and Build Siri as a Platform.

In each section I paraphrase quotes Brian made in the interview to explain the theme. I think you'll find it fun and provocative. Brian is an interesting guy.

Voice First is the Future

Stuff The Internet Says On Scalability For January 19th, 2017

Hey, it's HighScalability time:

• $268,895,000,000: Apple's cash and investments; 60%: growth in Amazon's ad revenue; ~80%: movie tickets sold in China are sold through mobile apps; £27,000: King Edward's yearly income; 3: new Google undersea cables; 7,500: Google edge caching nodes; 50,000x: microprocessor performance compared to a 1978 mini-computer at 0.25% of the cost; $15bn: spending on hosting services; 0.2 cycles per byte: ridiculously fast base64 encoding and decoding; $165B+: 2018 games software/hardware spending; 328 feet: air purification tower in China; 42 million: proteins molecules in a yeast cell;


• Quotable Quotes:
  • Richard Jones: For now, what we can say is that the age of exponential growth of computer power is over. It gave us an extraordinary 40 years, but in our world all exponentials come to an end, and we’re now firmly Continue reading

Sponsored Post: Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, Zohocorp

Who's Hiring? 

• Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.


• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume. 


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• Webinar: January 23, 11am GMT & again at 11am Pacific / 2pm Eastern. Continue reading

Monolith or Microservices: Which should you start with?

This is a guest Post by Jake Lumetta, Founder and CEO, ButterCMS, an API-first CMS. For more content like this, follow @ButterCMS on Twitter and subscribe to our blog.

Conventional wisdom for startups counsels starting with a monolith, but are there situations where you should start with microservices instead? Interviews with dozens of CTOs illuminated the key considerations when deciding whether to start with a monolith or microservices.

Challenging Conventional Wisdom

My good friend Darby Frey recently kicked off a greenfield project after assuming his new role as Sr. Platform Engineering Lead of Gamut. Despite starting out with monolith at his previous company Belly, he discovered that — in the right circumstances — starting with a monolith isn’t always the best way to go.

“As one does, I allowed much of my thinking in my early days [at my new company] to be influenced by my previous company,” Darby told me.

At Belly, Darby and his team broke down their monolith into a fairly large microservices architecture. They managed to get it to a good place, but only after months of trials and tribulations migrating to microservices.

With this experience fresh in his mind, he approached his new project Continue reading

Stuff The Internet Says On Scalability For January 12th, 2017

Hey, it's HighScalability time: 

• 28.5 billion: PornHub visitors; 3 billion: computer chips have Spectre security hole; 75.8%: people incorrectly think private browsing is actually private; 380,000: streams needed to make minimum wage on Spotify; 30Gbps: throughput for servers in Azure using new network interface cards packing field-programmable gate arrays; 8 quadrillion calculations per second: new NOAA supercomputer; $2 billion: market cap for parody cryptocurrency dogecoin; $1 trillion: IoT spending by 2035; 100,000: IoT sensors monitor canal in China; 1,204: definitions for emo; 23 million: digits in largest prime number; 2.8%: decline in PC shipments; 


• Quoteable Quotes:
  • @Lee_Holmes: We owe a debt of gratitude to the unsung heroes of Spectre and Meltdown: the thousands of engineers that spent their Continue reading

How Technology is Changing Music to fit Inside Streaming’s New Box

Wikimedia Commons

Technology shapes and forms. A Roman war chariot could still ride to battle on a modern road. Songs are typically about 3 minutes long because a 78rpm record held about three minutes of sound per side. So it shouldn't be a surprise streaming—a new technology for distributing music—pounds songs, business models, and production methods into new shapes fitting a new medium.

How would artists respond to near zero digital production costs, zero marginal distribution costs, and streaming's pay per play business model? Uniquely, as described in great detail in the podcast WS More or Less: Why Albums are Getting Longer.

Michael Jackson's Thriller album had nine tracks and runs at just over 42 minutes. Chris Brown released Heartbreak on a Full Moon, an album with 45 tracks and runs well over 2 hours. Albums are getting longer.

Why? You get what you measure.

1500 plays or streams of a track from a single album counts as a record sale. So the longer the album the fewer people have to listen to the whole thing to increase record sales. Record sales are how chart rankings are determined. Doing better in the charts gets you more exposure, which Continue reading

Stuff The Internet Says On Scalability For January 5th, 2017

Hey, it's HighScalability time: 

• 15: datacenters not built because of Google's TPU; 5 billlion: items shipped by Amazon Prime; 600: free online courses; 1.6 million: React downloads per week; 140 milliseconds: time Elon Musk's massive backup battery took to respond to crisis at power plant; 16: world spanning Riot Games clusters; < $100 a kilowatt-hour: Lithium-ion battery packs by 2025; 23%: performance OS penalty fixing Intel bug; 200k: pending Bitcoin transactions; 123 million: household data leak from marketing analytics company Alteryx; .67: hashes per day mining Bitcoin with pencil and paper; $21 million: projected cost of redundant power for Atlanta Airport; 62: nuclear test films; 10x: more galaxies in the universe; 55%: DuckDuckGo growth; 49%: increase Continue reading

Explain the Cloud Like I’m 10

 

Explain the Cloud Like I'm 10

 

“Todd, can you explain ‘The Cloud’? What is it?” I was asked this question at lunch by Joe, a fellow tour member on a recent trip Linda (my wife) and I took to France.

It was not a question I was expecting on vacation, but with many years of experience as a programmer, a lot of it spent in cloud computing, it’s a question I should have been able to knock out of the park.

Except I didn’t. My answer stank. I hemmed and hawed. I stuttered and sputtered. I could tell that nothing I said was making any sense. I gave a horrible answer, and it has haunted me ever since.

While talking, I noticed a lot of other people at the table were interested in my answer as well. It seemed a lot of smart people were confused about the cloud.

When I got back home I did a lot of research. I was trying to redeem myself by finding the perfect book to recommend. I couldn't find one! So I decided to write Explain the Cloud Like I'm 10. It's the answer I wish I'd given Joe in Continue reading

Stuff The Internet Says On Scalability For December 15th, 2017

Hey, wake up, it's HighScalability time:

 

Merry Christmas and Happy New Year everyone! I'll be off until the new year. Here's hoping all your gifts were selected using machine learning.

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate your recommending my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll like it. Now with twice the brightness and new chapters on Netflix and Cloud Computing.

 

• 157 terabytes: per second raw data output of the Square Kilometre Array; $11 million: made by a 6 year old on YouTube; 14TB: helium hard drive; 1: year education raises IQ 1-5 points; 10: seconds mining time to pay for wifi; 110 TFLOPS: Nvidia Launches $3,000 Titan V; 400: lines of JavaScript injected by Comcast; 20 million: requests per second processed by Netflix to personalize artwork; 270: configuration parameters in postgresql.conf; hundreds: eyes in scallop from a unique mirroring system; $72 billion: record DRAM revenue; 20: rockets landed by SpaceX; 


• Quotable Quotes:
  • Bill Walton: Mirai was originally developed to Continue reading

Netflix: What Happens When You Press Play?

This article is a chapter from my new book Explain the Cloud Like I'm 10. The first release was written specifically for cloud newbies. I've made some updates and added a few chapters—Netflix: What Happens When You Press Play? and What is Cloud Computing?—that level it up to a couple ticks past beginner. I think even fairly experienced people might get something out of it.

So if you are looking for a good introduction to the cloud or know someone who is, please take a look. I think you'll like it. I'm pretty proud of how it turned out. 

I pulled this chapter together from dozens of sources that were at times somewhat contradictory. Facts on the ground change over time and depend who is telling the story and what audience they're addressing. I tried to create as coherent a narrative as I could. If there are any errors I'd be more than happy to fix them. Keep in mind this article is not a technical deep dive. It's a big picture type article. For example, I don't mention the word microservice even once :-)

Netflix seems so simple. Press play and video magically Continue reading

Stuff The Internet Says On Scalability For December 8th, 2017

Hey, it's HighScalability time: 

AWS Geek creates spectacular visual summaries.

 

If you like this sort of Stuff then please support me on Patreon. And please recommend my new book—Explain the Cloud Like I'm 10—to those looking to understand the cloud. I think they'll like it.

• 127 terabytes: per year growth in blockchain if bitcoin wins; 4: hours from tabula rasa to chess god; 1.4 billion: Slack jobs per day; 400: hyperscale data centers worldwide by 2018; 9.8X: Machine Learning Engineer job growth; 14%: Ethereum transactions are for Cryptokitties; 80: seconds per hash on 55 year old IBM 1401 mainframe; $110 billion: app stores spending in 2018; 25: years since first text message; 4,000: AWS code pushes per day; two elephants: of space dust hits earth every day; 


• Quotable Quotes:
  • @DavidBrin: Now that's what I call engineering! [Voyager 1] Thrusters that haven't been used in 37 years - still reliable!
  • drkoalaman: So despite not supporting other cryptos the majority of my time on the DNM's I think its officially time to step away from bitcoin, at least for the time being. Went to do a direct deal today with Continue reading

Sponsored Post: Symbiont, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, Zohocorp

Who's Hiring? 

• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume. 


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• On-demand Webinar. Fast & Frictionless - The Decision Engine for Seamless Digital Business. In this session, guest speakers Michele Goetz, Principal Analyst at Forrester Research and Matthias Baumhof, VP Worldwide Engineering at ThreatMetrix, discuss: How risk-based authentication leveraging digital identities is key to empowering customer transactions; How real-time customer trust decisions can reduce fraud and improve customer satisfaction; How a high performance Hybrid Memory Continue reading

The Eternal Cost Savings of Netflix’s Internal Spot Market

Netflix used their internal spot market to save 92% on video encoding costs. The story of how is told by Dave Hahn in his now annual A Day in the Life of a Netflix Engineer. Netflix first talked about their spot market in a pair of articles published in 2015: Creating Your Own EC2 Spot Market Part 1 and Part 2.

The idea is simple:

• Netflix runs out of three AWS regions and uses hundreds of thousands of EC2 instances; many are underutilized at various parts in the day.

• Video encoding is 70% of Netflix’s computing needs, running on 300,000 CPUs in over 1000 different autoscaling groups.

• So why not create a spot market to process video encoding?

As background, Dave explained the video encoding process:

Stuff The Internet Says On Scalability For December 1st, 2017

Hey, it's HighScalability time: 

  Isn't this all of software? @thomasfuchs: Here we see a group of JavaScript engineers implementing a method that adds two numbers

 

If you like this sort of Stuff then please support me on Patreon. And there's my new book, Explain the Cloud Like I'm 10, for complete cloud newbies. 

• 82%: chance a file on GitHub is a duplicate; 11: new AWS regions; 42%: AWS yearly growth; 1,100: new AWS services in 2017; 300%: year of year growth in Lambda; 00000000: code to launch a Minuteman missile; 100 megawatts in 100 days: biggest battery in the world; 40: months in prison for VW engineer; 3,000 cores: Raspberry Pi cluster; 11: lost cities found by building a database from 4,000-year-old clay tablets; 1.25 million: Riot Games builds per year; 41.78: miles walked at reinvent; 


• Quotable Quotes:
  • @gigastacey: This FCC is going to destroy net neutrality, strangle competition in media, let wireline providers off the hook for replacing copper with fiber or an equivalent to copper AND kill broadband access for the poor. This is an unprecedented attack on consumers.
  • Continue reading

Sponsored Post: Symbiont, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, Zohocorp

Who's Hiring? 

• Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume. 


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• On-demand Webinar. Fast & Frictionless - The Decision Engine for Seamless Digital Business. In this session, guest speakers Michele Goetz, Principal Analyst at Forrester Research and Matthias Baumhof, VP Worldwide Engineering at ThreatMetrix, discuss: How risk-based authentication leveraging digital identities is key to empowering customer transactions; How real-time customer trust decisions can reduce fraud and improve customer satisfaction; How a high performance Hybrid Memory Continue reading

Stuff The Internet Says On Scalability For November 17th, 2017

Hey, it's HighScalability time: 

The BOSS Great Wall. The largest structure yet found in the universe. Contains 830 galaxies. A billion light years across. 10,000 times the mass of the Milky Way.

 

If you like this sort of Stuff then please support me on Patreon. And there's my new book, Explain the Cloud Like I'm 10, for complete cloud newbies. 

• $25 billion: Alibaba's Singles' Day sales; 6+ million: Slack daily active users; 4ms: boot time for a unikernel based VM; 1 billion: out of date Android devices; 10-20%: increase in RAM prices; 8 million: lines of code in F-35; $3 million: lost by Isaac Newton in the stock market; 30: it's RAID's birthday!; thousands: bugs fixed with Pentagon hackathon; 6+ terabytes: earth satellite data downloaded per day; 


• Quotable Quotes:
  • Berners-Lee: When I invented the web, I didn’t have to ask Vint Cerf [the ‘father of the internet’] for permission to use the internet
  • Germaine de Stael: Ridicule dries up the imagination.
  • Alex Hudson: A lot of technical write-ups focus on scaling, performance and large-scale systems. It’s definitely interesting to see what problems Netflix have, and how Continue reading