Stuff The Internet Says On Scalability For September 22nd, 2017

Hey, it's HighScalability time: 

 

Ever feel like howling at the universe? (Greg Rakozy)

 

If you like this sort of Stuff then please support me on Patreon.

• 10 billion: API calls made every second in Google datacenters; $767,758,000,000: collected by Apple on iPhones sold to the end of June; 20: watts of power consumed by human brain, autonomous vehicles peak at 3000 watts; 59%: drop in leads using AMP; 27%: success rate of AIs guessing passwords; 2.8 kilometers: distance devices running on almost zero power can xmit using backscatter; 96: age at which Lotfi Zadeh, inventor of Fuzzy Logic, passed away; 35%: store time series data in a RDBMS; $1.1 billion: Google's spend on self-driving tech;  $5.1 billion: Slack valuation; 15%: bugs reduced by strong typing; ~1 ft: new smartphone GPS accuracy; 


• Quotable Quotes:
  • Napoleon: [Sir Hudson Lowe] was a man wanting in education and judgment. He was a stupid man, he knew nothing at all of the world, and like all men who knew nothing of the world, he was suspicious and jealous.
  • Rich Werner: Data center operations, to me, is 362 Continue reading

Evolution of data structures in Yandex.Metrica

Yandex.Metrica is the world's second largest web analytics system. Metrica takes in a stream of data representing events that took place on sites or on apps. Our task is to process this data and present it in an analyzable form.

Processing the data in itself is not a problem. The real difficulty lies in trying to determine what form the processed results should be saved in so that they are easy to work with. During the development process, we had to completely change our approach to data storage organization several times. We started with MyISAM tables, then used LSM-trees and eventually came up with column-oriented database, ClickHouse. In this article I'll explain what led us to settle on this last option.

Yandex.Metrica was launched in 2008 and has now been running for more than nine years. Every time we changed our approach to data storage in the past it was because a particular solution proved inefficient: either there was insufficient performance reserve, or the solution was unreliable, or it used too many computational resources, or it just did not allow us to implement what we needed to.

The old Yandex.Metrica for websites has more than 40 "fixed" Continue reading

Stuff The Internet Says On Scalability For September 15th, 2017

Hey, it's HighScalability time: 

 

Earth received Cassini’s final signal at 7:55am ET. Let's bid a fond farewell. After a 13-year tour of duty, job well done!

 

If you like this sort of Stuff then please support me on Patreon.

• 12.9 million: DynamoDB requests per second on Prime Day; 4 billion: transistors on Apple's A11 Bionic chip; 4x: extreme weather events since 1970; 51: qubit device; 50%: Messenger.com converted to Reason; 56.6 million: US cord cutters; 5000: bikes abandoned at Burning Man; 500 million: yearly visitors to Apple stores; 30 min: time to send one HD color image from Mars to Earth; 


• Quoteable Quotes:
  • @randyshoup: Interesting idea of a *Negative* MTTR by @adrianco: notice something is going to fail and proactively fix it before it breaks!
  • @rob_pike: "The Equifax executives who let my data be stolen will probably suffer fewer consequences than I will for an overdue library book." @nytimes
  • @avantgame: on weaponized social media: "We’re in an information war with Russia. It’s time we started acting like it."
  • Jamie Dimon: It's [Bitcoin] worse than tulip bulbs. It won't end well. Someone is going Continue reading

Have you noticed there’s a lot more collaboration going on these days? Why?

Thanks to zero marginal cost digital production methods, we're seeing content markets—for the first time—develop in conditions free from supply and price constraints.

In the process we've learned something: consumers have an unquenchable thirst for new content; content creators are willing to oblige with an equally prodigious stream of new content; platforms that best control access to the customer are the biggest winners; the reward for content creators varies drastically by medium and platform.

For consumers, life is now a streaming fixed priced buffet of unending variety and diversion.

For producers, the changes have been terrifying. Old modes have crumbled, leaving everyone scrambling to figure out what, if anything, comes next.

To adapt, content creators are learning to exploit capture loops, bundling, and collaboration to extract money from a digital economy that has collectively decided it rarely wants to pay artists directly for their content anymore.

The most highly evolved form of digital content platform strategies can be found in the book market. Why? Because Amazon.

Kindle Unlimited is the Clear Platform Winner

Sponsored Post: Close.io, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Close.io is a ~25 person fully remote team that is profitable and building a product our customers love! We’re hiring Senior Backend Developers to join our team. Our backend tech stack currently includes Python (Flask, Gunicorn, TaskTiger), Elasticsearch, MongoDB, Postgres, and Redis running in Docker/Kubernetes on AWS. Learn more and apply here!


• Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.


• Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Continue reading

Stuff The Internet Says On Scalability For September 8th, 2017

Hey, it's HighScalability time: 

 

May you live in interesting times. China games swarming drone attacks. Portable EMP anyone? (Tech in Asia)

 

If you like this sort of Stuff then please support me on Patreon.

• 100GB: entire corpus of articles written at the NY Times; 80GB: data for one human genome; 3%: Linux desktop market share; 3.5M: fake Wells Fargo accounts; $18,000: world’s most expensive vacuum; 2000: Netflix recommender taste groups; 27%: year-over year-growth rate of Python on SO; 4M: Time Warner hacked; 143M: Equifax hacked; $800M: ICO funding in Q2; $257M: Filecoin ICO; 


• Quotable Quotes:
  • Brendan Gregg: jobs are also migrating from both Solaris and Linux to cloud jobs instead, specifically AWS. The market for OS and kernel development roles is actually shrinking a little. The OS is becoming a forgotten cog in a much larger cloud-based system. The job growth is in distributed systems, cloud SRE, data science, cloud network engineering, traffic and chaos engineering, container scheduling, and other new roles. 
  • @DrQz: The Performance Paradox: The better u do ur job, the more invisible u become. https://goo.gl/1aTRvw  ? ?
  • @kennwhite: $100,000+ spent Continue reading

Stuff The Internet Says On Scalability For September 1st, 2017

Hey, it's HighScalability time: 

Obviously, cloud native is simplicity itself. (Cloud Native Landscape Project)

• $10: price for IPv4 address (and falling); 10-15%: better IPv6 network performance; 711M: Record Onliner Spambot Dump; 41.4-tesla: strongest resistive magnet; 85%: cell towers offline due to Hurricane Harvey; 1M: Facebook accounts turned off every single day; 700K: Lyft drivers; $160B: Crypto Market Cap; $1.5M: bounty for iPhone jail break; 2.93M: pirated views of Mayweather-McGregor fight; 465k: people need to update pacemaker firmware; 70,065,920: views of Taylor Swift in 2 days; 2 trillion: kafka messages per day at LinkedIn; $35 billion: saved when planes fly themselves; 990: bird species in North America; 108B: number of people who have ever lived; 500M: DuckDuckGo anonymous searches in one month; 


• Quotable Quotes:
  • @asymco: The iPhone is not only the best selling product of all time but it may be the most used. 60 trillion minutes a year.
  • @Nick_Craver: Stack Overflow Questions (last 30 days): 1,280,911,699 Hits 26.92 ms Avg Render Time 11,040,996,940 SQL Queries Continue reading

Sponsored Post: Close.io, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Close.io is a ~25 person fully remote team that is profitable and building a product our customers love! We’re hiring Senior Backend Developers to join our team. Our backend tech stack currently includes Python (Flask, Gunicorn, TaskTiger), Elasticsearch, MongoDB, Postgres, and Redis running in Docker/Kubernetes on AWS. Learn more and apply here!


• Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.


• Grokking the System Design Interview is an interactive course that explains the Large Scale Distributed System Design problems in detail. System Continue reading

Stuff The Internet Says On Scalability For August 25th, 2017

Hey, it's HighScalability time: 

 

View of the total solar eclipse from a hill top near Madras Oregon, August 21, 2017. As totality approaches, dragons gorge on sun flesh; darkness cleaves the day; a chill chases away the heat; all becomes still. Contact made! Diamonds glitter; beads sparkle; shadow band snakes slither across pale dust; moon shadow races across the valley, devouring all in wonder. Inside a circle of standing stones, obsidian knives slash and stab. Sacrifices offered, dragons take flight. In awe we behold the returning of the light.

• ~5: ethereum transactions per second; 29+M: Snapchat news viewers; 100K: largest Mastodon instance; 2x: Alibaba's cloud base growth; 1B: trees planted by a province in Pakistan; 90.07%: automated decoding of honey bee waggle dances; $86.4B: Worldwide Information Security Spending; 1200: db migrations from Mysql to Postgres; $7B: Netflix content spend (most not original); 13%: increased productivity by making vacation mandatory; 75%: US teens use iPhones; 30,000x: energy use for Bitcoin transaction compared to Visa; ~1 trillion: observations processed for Gaia mission; 50% Continue reading

Stuff The Internet Says On Scalability For August 18th, 2017

Sorry about missing last week, but my birthday won out over working: 



 

Ouch! @john_overholt: My actual life is now a science exhibit about the primitive conditions of the past.

If you like this sort of Stuff then please support me on Patreon.

 

• 1PB: SSD in 1U chassis; 90%: savings using EC2 Spot for containers; 16: forms of inertia; $2.1B: Alibaba’s profit; 22.6B: app downloads in Q2; 25%: Google generated internet traffic; 20 by 20 micrometers: quantum random number generators; 16: lectures on Convolutional Neural Networks for Visual Recognition; 25,000: digitized gramophone records; 280%: increase in IoT attacks; 6.5%: world's GDP goes to subsidizing fossil fuel; 832 TB: ZFS on Linux;  $250,000: weekly take from breaking slot machines; 30: galatic message exchanges using artificial megastructures in 100,000 years; 


• Quotable Quotes:
  • @chris__martin: ALIENS: we bring you a gift of reliable computing technol--
    HUMANS: oh no we have that already but JS is easier to hire for
  • @rakyll: "You woman, you like intern." I interned on F-16's flight computer. Even my internship was 100x more legit than any job you will have.
  • @CodeWisdom Continue reading

Sponsored Post: Domino Data Lab, Etleap, Aerospike, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• Grokking the System Design Interview is an interactive course that explains the Large Scale Distributed System Design problems in detail. System design questions have become a standard part of the software engineering hiring process. This course is created by hiring managers who’ve been working at Google, Facebook, Microsoft, and Amazon, and is a complete guide to master the System Design Interviews.


• Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)


• What engineering and IT leaders need to know about data science. As data science becomes more mature within an organization, you may be pulled into leading, enabling, and collaborating with data science teams. While Continue reading

Why Morningstar Moved to the Cloud: 97% Cost Reduction

Enterprises won't move to the cloud. If they do, it's tantamount to admitting your IT group sucks. That has been the common wisdom. Morningstar, an investment research provider, is moving to the cloud and they're about as enterprisey as it gets. And they don't strike me as incompetent, it just seems they don't want to worry about all the low level IT stuff anymore. 

Mitch Shue, Morningstar's CTO, gave a short talk at AWS Summit Series 2017 on their move to AWS. It's not full of nitty gritty technical details. That's not the interesting part. The talk is more about their motivations, the process they used to make the move, and some of the results they've experienced. While that's more interesting, we've heard a lot of it before.

What I found most interesting was the idea of Morningstar as a canary test. If Morningstar succeeds, the damn might bust and we'll see a lot more adoption of the cloud by stodgy mainstream enterprises. It's a copy cat world. That sort of precedent gives other CTOs the cover they need to make the same decision.

The most important idea in the whole talk: the cost savings of moving to Continue reading

Stuff The Internet Says On Scalability For August 4th, 2017

Hey, it's HighScalability time:

Hands down the best ever 25,000 year old selfie from Pech Merle cave in southern France. (The Ice Age)

• 35%: US traffic is now IPV6; 10^161: decision points in no-limit Texas hold’em; 4.5 billion: Facebook translations per day; 90%: savings by moving to Lambda; 330TB: IBM's tiny tape cartridge, enough to store 330 million books; $108.9 billion: game revenues in 2017; 85%: of all research papers are on Sci-Hub; 1270x: iPhone 5 vs Apollo guidance computer; 16 zettabytes: 2017 growth in digital universe; 


• Quotable Quotes:
  • Andrew Roberts: [On Napoleon] No aspect of his command was too small to escape notice.
  • Jason Calacanis: The world has trillions of dollars sitting in bonds, cash, stocks, and real estate, which is all really “dead money.” It sits there and grows slowly and safely, taking no risk and not changing the world at all. Wouldn’t it be more interesting if we put that money to work on crazy experiments like the next Tesla, Google, Uber, Cafe X, or SpaceX?
  • @icecrime Continue reading

The Next Scalability Hurdle: Massively Multiplayer Mobile AR

Many moons ago, in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud, I said we still had scaling challenges ahead, that we've not yet begun to scale, that we still don't know how to scale at a planetary level.

That was 7 years ago. Now Facebook has 2 billion monthly users. There's no reason to think they can't scale an unimpressive 3.5x to handle the rest of the planet. WhatsApp is at one billion daily users. YouTube is at 1.5 billion monthly users.

So it appears we do know how to service a whole planet full of people (and bots). At least a select few companies with vast resources know how. We are still no closer to your average developer being able to field a planet scale service. The winner take all nature of the Internet seems to fend off decentralization like it's a plague. Maybe efforts like Filecoin will change the tide. 

There's another area we have scaling challenges: Massively Multiplayer Mobile AR (Augmented Reality). While AR has threatened to be the future for quite some time, it now looks like the future may be just around the virtual Continue reading

Sponsored Post: Apple, Domino Data Lab, Etleap, Aerospike, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Apple is looking for passionate VoIP engineer with a strong technical background to transform our Voice platform to SIP. It will be an amazing journey with highly skilled, fast paced, and exciting team members. Lead and implement the engineering of Voice technologies in Apple’s Contact Center environment. The Contact Center Voice team provides the real time communication platform for customers’ interaction with Apple’s support and retail organizations. You will lead the global Voice, SIP, and network cross-functional engineers to develop world class customer experience. More details are available here.


• Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)


• What engineering and IT leaders need Continue reading

Stuff The Internet Says On Scalability For July 28th, 2017s

Hey, it's HighScalability time:

• 1.8x: faster code on iPad MacBook Pro; 1 billion: WhatsApp daily active users; 100 milliamps: heart stopping current; $25m: surprisingly low take from ransomware; 2,700x: improvement in throughput with TCP BBR; 620: Uber locations; $35.5 billion: Facebook's cash hoard; 2 billion: Facebook monthly active users; #1: Apple is the world's most profitable [legal] company; 500,000x: return on destroying an arms depot with a drone; 


• Quotable Quotes:
  • Alasdair Allan: Jeff Bezos’ statement that “there’s not that much interesting about CubeSats” may well turn out to be the twenty first century’s “nobody needs more than 640kb.”
  • @hardmaru: Decoding the Enigma with RNNs. They trained a LSTM with 3000 hidden units to decode ciphertext with 96%+ accuracy. 
  • @tj_waldorf: Morningstar achieved 97% cost reduction by moving to AWS. #AWSSummit Chicago
  • Ed Sperling: Moore’s Law is alive and well, but it is no longer the only approach. And depending on Continue reading

7 Interesting Parallels Between the Invention of Tiny Satellites and Cloud Computing

CubeSats are revolutionizing space exploration because they are small, modular, and inexpensive to build and launch. On an episode of embedded.fm, Professor Jordi Puig-Suari gives a fascinating interview on the invention of the CubeSat. 195: A BUNCH OF SPUTNIKS.

What struck in the interview is how the process of how the CubeSat was invented parallels how the cloud developed. They followed a very similar path driven by many of the same forces and ideas. 

Just what is a CubeSat? It's a "type of miniaturized satellite for space research that is made up of multiples of 10×10×10 cm cubic units. CubeSats have a mass of no more than 1.33 kilograms per unit, and often use commercial off-the-shelf (COTS) components for their electronics and structure."

Parallel #1:  University as Startup Incubator

Stuff The Internet Says On Scalability For July 21st, 2017

Hey, it's HighScalability time:

• 222x: Bitcoin less efficient than a physical system of metal coins and paper/fabric/plastic; #1: Python use amongst Spectrum readers; 3x: time spent in apps that don't make us happy; 1 million: DigitalOcean users; 11.6 million: barrels of oil a day saved via tech and BigData; 200,000: cores on Cray super computer;$200B: games software/hardware revenue by 2021; $3K: for 50 Teraflops AMD Vega Deep Learning Box; 24.4 Gigawatts: China New Solar In First Half Of 2017; 


• Quotable Quotes:
  • sidlls: I think instead there is a category error being made: that CS is an appropriate degree (on its own) to become a software engineer. It's like suggesting a BS in Physics qualifies somebody to work as an engineer building a satellite.
  • Elon Musk: AI is a fundamental existential risk for human civilization, and I don’t think people fully appreciate that
  • Mike Elgan: Continue reading

You’ll Never Believe the Big Hairy Audacious Startup John Jacob Astor Created in 1808

Think your startup has a Big Hairy Audacious Goal? Along with President Thomas Jefferson, John Jacob Astor  conceived (in 1808), and implemented (in 1810) a plan to funnel the entire tradable wealth of the westernmost sector of the North American continent north of Mexico through his own hands. Early accounts described it as “the largest commercial enterprise the world has ever known.”

Think your startup raised a lot of money? Astor put up $400,000 ($7,614,486 in today's dollars) of his own money, with more committed after the first prototype succeeded.

Think competition is new? John Jacob Astor dealt with rivals in one of three ways: he tried to buy them out; if that didn’t work, he tried to partner with them; if he failed to join them, he tried to crush them.

Think your startup requires commitment? Joining Astor required pledging five years of one’s life to a start-up venture bound for the unknownn.

Think your startup works hard? Voyageur's paddled twelve to fifteen hours per day, with short breaks while afloat for a pipe of tobacco. During that single day each voyageur would make more than thirty thousand paddle strokes. On the upper Great Continue reading

Stuff The Internet Says On Scalability For July 14th, 2017

Hey, it's HighScalability time:

 

• 2 trillion: web requests served daily by Akamai; 9 billion: farthest star ever seen in light-years; 10^31: bacteriophages on earth; 7: peers needed to repair ransomware damage; $30,000: threshold of when to leave AWS; $300K-$400K: beginning cost of running Azure Stack on HPE ProLiant; 3.5M: files in the Microsoft's git repository; 300M: Google's internal image data training set size; 7.2 Mbps: global average connection speed; 85 million: Amazon Prime members; 35%: Germany generated its electricity from renewables;


• Quotable Quotes:
  • Jessica Flack: I believe that science sits at the intersection of these three things — the data, the discussions and the math. It is that triangulation — that’s what science is. And true understanding, if there is such a thing, comes only when we can do the translation between these three ways of representing the world.
  • gonchs: “If your whole business relies on us [Medium], you might Continue reading