Stuff The Internet Says On Scalability For October 14th, 2016
Hey, it's HighScalability time:
A pattern from the collective unconscious of the universe. Scott Kelly's brilliant Year in Space Photos.
- $1.5 million: new iOS hack bug bounty; 120 Terabits per second: Google and Facebook's submarine cable between Los Angeles with Hong Kong; 142,000: IT jobs lost last month; $17 billion: cost of recall to Samsung; $4.1 Billion: IRS detected identity theft tax fraud; 1956: first mention of P vs NP by Kurt Gödel to John von Neumann; 1 million HTTP requests per second: DDoS attacks coming from IoT cameras; 90 petaflops: capacity of volunteer computing; 500 msec: time it takes the brain to integrate all sensory data into consciousness;
- Quotable Quotes:
- @GreatDismal: Silicon Valley fantasy that our universe is a simulation is actually the fantasy that our universe is a *sucessful startup*
- @gblache: Being POTUS must be like inheriting a 240 year old code base and being asked to fix it in 4 years while half your team tries to sandbag you.
- chrissnell: I'm a huge believer in colocation/on-prem in the post-Kubernetes era. I manage technical Continue reading
Lessons Learned from Scaling Uber to 2000 Engineers, 1000 Services, and 8000 Git repositories
For a visual of the growth Uber is experiencing take a look at the first few seconds of the above video. It will start in the right place. It's from an amazing talk given by Matt Ranney, Chief Systems Architect at Uber and Co-founder of Voxer: What I Wish I Had Known Before Scaling Uber to 1000 Services (slides).
It shows a ceaseless, rhythmic, undulating traffic grid of growth occurring in a few Chinese cities. This same pattern of explosive growth is happening in cities all over the world. In fact, Uber is now in 40 cities and 70 countries. They have over 6000 employees, 2000 of whom are engineers. Only a year and half a go there were just 200 engineers. Those engineers have produced over 1000 microservices which are stored in over 8000 git repositories.
That's crazy 10x growth in a crazy short period of time. Who has experienced that? Not many. And as you might expect that sort of unique, compressed, fast paced, high stakes experience has to teach you something new, something deeper than you understood before.
Matt is not new to this game. He was co-founder of Voxer, which experienced its Continue reading
Sponsored Post: ScaleArc, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp
Who's Hiring?
-
IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.
Fun and Informative Events
- Learn how Nielsen Marketing Cloud (NMC) leverages online machine learning and predictive personalization to drive its success in a live webinar on Tuesday, September 20 at 11 am PT / 2 pm ET. Hear from Nielsen’s Kevin Lyons, Senior VP of Data Science and Digital Technology, and Brent Keator, VP of Infrastructure, as well as from Brian Bulkowski, CTO and Co-Founder at Aerospike, as they describe the front-edge architecture and technical choices – including the Aerospike NoSQL database – that have led to NMC’s success. RSVP: https://goo.gl/xDQcu4
Cool Products and Services
- ScaleArc's database load balancing software empowers you to “upgrade your Continue reading
Stuff The Internet Says On Scalability For October 7th, 2016
Hey, it's HighScalability time:
- 70 billion: facts in Google's knowledge graph; 80 million: monthly visitors to walmart.com; 50%: lower cost for sending a container from Shanghai to Europe; 6 billion: Docker Hub pulls per 6 weeks; 5x: impact reduction using new airbag helmet; 400: node Cassandra + Spark Cluster in Azure; 66%: loss of installs when apps > 100MB; 223GB: Udacity open sources self-driving car data;
- Quotable Quotes:
- rfrey: The success of many companies, and probably all of the unicorns, has nothing to do with technology. The tech is necessary, of course, but so are desks and an accounting department. Internalizing that has been difficult for me as an engineer.
- @mza: 72 new features/services released last month on #AWS. 706 so far this year (up 42.9% YoY).
- Marc Andreessen: To me the problem is clear: The problem is insufficient technological adoption, innovation, and disruption in these high-escalating price sectors of the economy. My thesis is that we're not in a tech bubble — Continue reading
Stuff The Internet Says On Scalability For September 30th, 2016
Hey, it's HighScalability time:
- 18: Google can now drink and drive in Washington DC.; $10 billion: cost of a Vision Quest to Mars; 620 Gbps: DDoS attack on KrebsOnSecurity; 1 Tbps: DDoS attack on OVH; $200,000: cost of a typical cyber incident; 8 million: video training dataset labeled with 4800 labels; 180: Amazon warehouses in the US; 10: bits of info per photon; 16: GPUs in new AI killer P2 instance type;
- Quotable Quotes:
- @markmccaughrean: 1,000,000 people to Mars in 100 yrs. 10 people/launch? That's 3 a day, every day, for a century. 1% failure rate? One explosion every month
- @jeremiahg: Any sufficiently advanced exploit is indistinguishable from a 400lb hacker.
- BrianKrebs: I suggested to Mr. Wright perhaps a better comparison was that ne’er-do-wells now have a virtually limitless supply of Stormtrooper clones that can be conscripted into an attack at a moment’s notice.
- Sonia: Academia’s not-so-subtle distain for applied research does more than damage a few promising careers; it Continue reading
How Uber Manages a Million Writes Per Second Using Mesos and Cassandra Across Multiple Datacenters
If you are Uber and you need to store the location data that is sent out every 30 seconds by both driver and rider apps, what do you do? That’s a lot of real-time data that needs to be used in real-time.
Uber’s solution is comprehensive. They built their own system that runs Cassandra on top of Mesos. It’s all explained in a good talk by Abhishek Verma, Software Engineer at Uber: Cassandra on Mesos Across Multiple Datacenters at Uber (slides).
Is this something you should do too? That’s an interesting thought that comes to mind when listening to Abhishek’s talk.
Developers have a lot of difficult choices to make these days. Should we go all in on the cloud? Which one? Isn’t it too expensive? Do we worry about lock-in? Or should we try to have it both ways and craft brew a hybrid architecture? Or should we just do it all ourselves for fear of being cloud shamed by our board for not reaching 50 percent gross margins?
Uber decided to build their own. Or rather they decided to weld together their own system by fusing together two very capable open source components. What was Continue reading
Sponsored Post: ScaleArc, Spotify, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp
Who's Hiring?
-
Spotify is looking for individuals passionate in infrastructure to join our Site Reliability Engineering organization. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. We are looking for an SRE Engineering Manager in NYC and SREs in Boston and NYC.
-
IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.
Fun Continue reading
Stuff The Internet Says On Scalability For September 23rd, 2016
Hey, it's HighScalability time:
- October 2017: ICANN changes the DNSSEC root keys; $2.91M: cost of running Let's Encrypt; 20%: Amazon convenience tax; 100%: increase in spam; 6.2 km: Quantum teleportation across a metropolitan fibre network; March 18, 1982: birth of containers; 6 months: how long a lightening bolt can power a 60 watt bulb; trillions: EV cache hits per day @ Netflix; 5x: Spark is faster than MapReduce; billions: HTTP, Git and SSH connections served per day at GitHub; 28: # of websites in North Korea;
- Quotable Quotes:
- @vgcerf: It is time to admit after 18 years that the multistakeholder model of Internet operation works. #yestoIANA
- @EricLathrop: Netflix found a 5x performance variation between AWS instances at the same price! They benchmark to avoid overpaying. @indirect #Strangeloop
- @swardley: Perfectly reasonable @NigelBarron. Larry's statements are ludicrous, play is to milk existing customers whilst hoping to find a new future.
- @BethanyMacri: Etsy is very anti-SOA. Monolith forever!
- janfoeh: I've said it before Continue reading
Stuff The Internet Says On Scalability For September 16th, 2016
Hey, it's HighScalability time:
- 60%: time spent cleaning dirty dirty BigData; 10 million: that's a lot of Raspberry Pi; 365: days living in a Mars simulation; 100M: monthly League of Legends players; 1.75 billion: copyright takedowns by Google; 3.5 petabytes: data Evernote has to move to Google cloud; 11%: YoY growth in time spent on mobile apps; 4 hours: time between Lambda coldstarts;
- Quotable Quotes:
- Camille Fournier: humans struggle to tangibly understand domains that are theoretically separate when they are presented as colocated by the source code.
- @songcarver: The better example: iPhone 7 is showing 115% of 2016 Macbook single core performance, 88% of multi-core.
- ex3ndr: We (actor.im) also moved from google cloud to our servers + k8s. Shared persistent storage is a huge pain. We eventually stopped to try to do this, will try again when PetSets will be in Beta and will be able to update it's images.
- @mcclure111: "Well maybe you should get your Continue reading
If Traffic is an Iterated Prisoner’s Dilemma Game Can Smart Cars Evolve Co-operative Behavior?
Can small tribes of cooperating smart cars improve overall traffic even if they are not in the majority? Sure, if every car was a self-driving car maybe traffic jams could dissolve like blood clots on anticoagulants, but what about that messy in-between period? It will be some time before smart cars rule the road. Until then can smart cars make traffic better?
Adoption is hard. This is a general problem in tech. You want people to join your social network yet people won't join until enough people have already joined. What you really want is that virtuous circle to develop, where as more people adopt a technology it causes even more people to adopt it. So startups spend their VC money fast and furiously in hopes of acquiring new customers betting the lifetime value of a customer will be worth the investment. VC money is the dead corpse that feeds the rest of the ecosystem.
Traffic is already an example of a vicious cycle. Horrendous traffic jams are now the norm and "good" traffic windows are just tall tales texted to children. And it keeps on getting worse and not in a worse is better sort of way. Yet the incentives are still not Continue reading
Sponsored Post: ScaleArc, Spotify, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp
Who's Hiring?
-
Spotify is looking for individuals passionate in infrastructure to join our Site Reliability Engineering organization. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. We are looking for an SRE Engineering Manager in NYC and SREs in Boston and NYC.
-
IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.
Fun Continue reading
The Dollar Shave Club Architecture Unilever Bought for $1 Billion
This is a guest post by Jason Bosco, the Dollar Shave Club’s Director of Engineering, Core Platform & Infrastructure, on the infrastructure of its ecommerce technology.
With more than 3 million members, Dollar Shave Club will do over $200 million in revenue this year. Although most are familiar with the company’s marketing, this immense growth in just a few years since launch is largely due to its team of 45 engineers.
Dollar Shave Club engineering by the numbers:
Core Stats
Stuff The Internet Says On Scalability For September 9th, 2016
Hey, it's HighScalability time:
- 15%: Facebook's reduction in latency using HTTP2's server push; 1.9x: nanotube transistors outperform silicon; 200: projectors used to film a "hologram"; 50%: of people fall for phishing attacks (it's OK to click); 5x: increased engagement using Google's Progressive Web Apps; 115,000+: Cassandra nodes at Apple; $500 million: Pokémon Go; $150M: Delta's cost for datacenter outage;
- Quotable Quotes:
- Dan Lyons: I wanted to write a book about what it’s like to be 50 and trying to reinvent yourself – that struggle. There are all these books and inspirational speakers talking about being a lifelong learner and it’s so great to reinvent yourself, the brand of you. And I wanted to say, you know, it’s not like that. It’s actually really painful.
- Engineers & Coffee~ In modern application development everything is a stream now versus historically everything was a transaction. Make a request and the you're done. It's easier to write analytics on top of streams versus using Hive. It's cool Continue reading
Code Generation: The Inner Sanctum of Database Performance
This is guest post by Drew Paroski, architect and engineering manager at MemSQL. Previously he worked at Facebook and developed HHVM, the popular real-time PHP compiler used across the company’s web scale application.
Achieving maximum software efficiency through native code generation can bring superior scaling and performance to any database. And making code generation a first-class citizen of the database, from the beginning, enables a rich set of speed improvements that provide benefits throughout the software architecture and end-user experience.
If you decide to build a code generation system you need to clearly understand the costs and benefits, which we detail in this article. If you are willing to go all the way in the name of performance, we also detail an approach to save you time leveraging existing compiler tools and frameworks such as LLVM in a proven and robust way.
Code Generation Basics
Stuff The Internet Says On Scalability For September 2nd, 2016
Hey, it's HighScalability time:
- 4,000 GB: projected bandwidth used per autonomous vehicle per day; 100K: photos of US national parks; 14 terabytes: code on Github in 1 billion files held in 400K repositories; 25: age of Linux; $5 billion: cost of labor for building Linux; $3800: total maintenance + repairs after 100K miles and 5 years of Tesla ownership; 2%: reduction in Arizona's economy by deporting all illegal immigrants; 15.49TB: available research data; 6%: book readers who are "digital only";
- Quotable Quotes
- @jennyschuessler: "Destroy the printing press, I beg you, or these evil men will triumph": Venice, 1473
- @Carnage4Life: Biggest surprise in this "Uber for laundry" app shutting down is that there are still 3 funded startups in the space
- @tlipcon: "backpressure" is right up there with "naming things" on the top 10 list of hardest parts of programming
- cmcluck: Please consider K8s [kubernetes] a legitimate attempt to find a better way to build both internal Google systems and the next wave of cloud products in the open with the community. We Continue reading
My Test Tube Filled with DNA is Better than Your Mesos Cluster
We’ve seen computation using slime mold, soap film, water droplets, there’s even a 10,000 Domino Computer. Now DNA can do math In a test tube. Using addition, subtraction, multiplication, and division.
It’s not fast. Calculations can take hours. The upside: they are tiny and can work in wet environments. Think of running calculations in your bloodstream or in cells, like a programmable firewall, to monitor and alert on targeted health metrics and then trigger a localized response. Or if you are writing science fiction perhaps the ocean could become one giant computer?
The applications already sound like science fiction:
Prior devices for control of chemical reaction networks and DNA doctor applications have been limited to finite-state control, and analog DNA circuits will allow much more sophisticated analog signal processing and control. DNA robotics have allowed devices to operate autonomously (e.g., to walk on a nanostructure) but also have been limited to finite-state control.
Analog DNA circuits can allow molecular robots to include real-time analog control circuits to provide much more sophisticated control than offered by purely digital control. Many artificial intelligence systems (e.g., neural networks and probabilistic inference) that dynamically learn Continue reading
The cat-and-mouse story of implementing anti-spam for Mail.Ru Group’s email service and what Tarantool has to do with this
Hey guys!
In this article, I’d like to tell you a story of implementing the anti-spam system for Mail.Ru Group’s email service and share our experience of using the Tarantool database within this project: what tasks Tarantool serves, what limitations and integration issues we faced, what pitfalls we fell into and how we finally arrived to a revelation.
Let me start with a short backtrace. We started introducing anti-spam for the email service roughly ten years ago. Our first filtering solution was Kaspersky Anti-Spam together with RBL (Real-time blackhole list — a realtime list of IP addresses that have something to do with spam mailouts). This allowed us to decrease the flow of spam messages, but due to the system’s inertia, we couldn’t suppress spam mailouts quickly enough (i.e. in the real time). The other requirement that wasn’t met was speed: users should have received verified email messages with a minimal delay, but the integrated solution was not fast enough to catch up with the spammers. Spam senders are very fast at changing their behavior model and the outlook of their spam content when they find out that spam messages are not delivered. So, we couldn’t put up Continue reading
Sponsored Post: Spotify, Aerospike, Exoscale, Host Color, Scalyr, Gusto, LaunchDarkly, VividCortex, MemSQL, InMemory.Net, Zohocorp
Who's Hiring?
-
Spotify is looking for individuals passionate in infrastructure to join our Site Reliability Engineering organization. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. We are looking for an SRE Engineering Manager in NYC and SREs in Boston and NYC.
-
IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.
Fun Continue reading
Stuff The Internet Says On Scalability For August 26th, 2016
Hey, it's HighScalability time:
- 33.0%: of all retail goods sold online in the US are sold on Amazon; 110.9 million: monthly Amazon unique visitors; 21 cents: cost of 30K batch derived page views on Lambda; 4th: grade level of Buzzfeed articles; $1 trillion: home value threatened by rising sea levels; $1.2B: Uber lost $1.2B on $2.1B in revenue in H1 2016; 1.58 trillion: miles Americans drove through June;
- Quotable Quotes:
- @bendystraw: My best technical skill isn't coding, it's a willingness to ask questions, in front of everyone, about what I don't understand
- @vmg: "ls is the IDE of producing lists of filenames"
- @nicklockwood: The hardest problem in computer science is fighting the urge to solve a different, more interesting problem than the one at hand.
- @RexRizzo: Wired: "Machine learning will TAKE OVER THE WORLD!" Amazon: "We see you bought a wallet. Would you like to buy ANOTHER WALLET?"
- @viktorklang: "The very Continue reading
The Always On Architecture – Moving Beyond Legacy Disaster Recovery
Failover does not cut it anymore. You need an ALWAYS ON architecture with multiple data centers.-- Martin Van Ryswyk, VP of Engineering at DataStax
Failover, switching to a redundant or standby system when a component fails, has a long and checkered history as a way of dealing with failure. The reason is your failover mechanism becomes a single point of failure that often fails just when it's needed most. Having worked on a few telecom systems that used a failover strategy I know exactly how stressful failover events can be and how stupid you feel when your failover fails. If you have a double or triple fault in your system failover is exactly the time when it will happen.
For a long time the only real trick we had for achieving fault tolerance was to have a hot, warm, or cold standby (disk, interface, card, server, router, generator, datacenter, etc.) and failover to it when there's a problem. This old style of Disaster Recovery planning is no longer adequate or necessary.
Now, thanks to cloud infrastructures, at least at a software system level, we have an alternative: an always on architecture. Google calls this a Continue reading