The QuickBooks Platform

This is a guest post by Siddharth Ram – Chief Architect, Small Business. [email protected].

The QuickBooks ecosystem is the largest small business SaaS product. The QuickBooks Platform supports bookkeeping, payroll and payment solutions for small businesses, their customers and accountants worldwide. Since QuickBooks is also a compliance & tax filing platform, consistency in reporting is extremely important.. Financial reporting requires flexibility in queries – a given report may have dozens of different dimensions that can be tweaked. Collaboration requires multiple edits by employees, Accountants and Business owners at the same time, leading to potential conflicts. All this leads to solving interesting scaling problems at Intuit.

Solving for scalability requires thinking on multiple time horizons and axes. Scaling is not just about scaling software – it is also about people scalability, process scalability and culture scalability. All these axes are actively worked on at Intuit. Our goal with employees is to create an atmosphere that allows them to do the best work of their lives.

Background

Gone Fishin’

Well, not exactly Fishin', but I'll be on a month long vacation starting today. I won't be posting (much) new content, so we'll all have a break. Disappointing, I know. Please use this time for quiet contemplation and other inappropriate activities. See you on down the road...

Datanet: a New CRDT Database that Let’s You Do Bad Bad Things to Distributed Data

We've had databases targeting consistency. These are your typical RDBMSs. We've had databases targeting availability. These are your typical NoSQL databases.

If you're using your CAP decoder ring you know what's next...what databases do we have that target making concurrency a first class feature? That promise to thrive and continue to function when network partitions occur?

No many, but we have a brand new concurrency oriented database: Datanet - a P2P replication system that utilizes CRDT algorithms to allow multiple concurrent actors to modify data and then automatically & sensibly resolve modification conflicts.

Datanet is the creation of Russell Sullivan. Russell spent over three years hidden away in his mad scientist layer researching, thinking, coding, refining, and testing Datanet. You may remember Russell. He has been involved with several articles on HighScalability and he wrote AlchemyDB, a NoSQL database, which was acquired by Aerospike.

So Russell has a feel for what's next. When he built AlchemyDB he was way ahead of the pack and now he thinks practical, programmer friendly CRDTs are what's next. Why?

Concurrency and data locality. To quote Russell:

Datanet lets you ship data to the spot where the action is happening. When the action happens it Continue reading

Stuff The Internet Says On Scalability For October 14th, 2016

Hey, it's HighScalability time:

 

A pattern from the collective unconscious of the universe. Scott Kelly's brilliant Year in Space Photos.

• $1.5 million: new iOS hack bug bounty; 120 Terabits per second: Google and Facebook's submarine cable between Los Angeles with Hong Kong; 142,000: IT jobs lost last month;  $17 billion: cost of recall to Samsung; $4.1 Billion: IRS detected identity theft tax fraud; 1956: first mention of P vs NP by Kurt Gödel to John von Neumann; 1 million HTTP requests per second: DDoS attacks coming from IoT cameras; 90 petaflops: capacity of volunteer computing; 500 msec: time it takes the brain to integrate all sensory data into consciousness;


• Quotable Quotes:
  • @GreatDismal: Silicon Valley fantasy that our universe is a simulation is actually the fantasy that our universe is a *sucessful startup*
  • @gblache: Being POTUS must be like inheriting a 240 year old code base and being asked to fix it in 4 years while half your team tries to sandbag you.
  • chrissnell: I'm a huge believer in colocation/on-prem in the post-Kubernetes era. I manage technical Continue reading

Lessons Learned from Scaling Uber to 2000 Engineers, 1000 Services, and 8000 Git repositories

For a visual of the growth Uber is experiencing take a look at the first few seconds of the above video. It will start in the right place. It's from an amazing talk given by Matt Ranney, Chief Systems Architect at Uber and Co-founder of Voxer: What I Wish I Had Known Before Scaling Uber to 1000 Services (slides).

It shows a ceaseless, rhythmic, undulating traffic grid of growth occurring in a few Chinese cities. This same pattern of explosive growth is happening in cities all over the world. In fact, Uber is now in 40 cities and 70 countries. They have over 6000 employees, 2000 of whom are engineers. Only a year and half a go there were just 200 engineers. Those engineers have produced over 1000 microservices which are stored in over 8000 git repositories.

That's crazy 10x growth in a crazy short period of time. Who has experienced that? Not many. And as you might expect that sort of unique, compressed, fast paced, high stakes experience has to teach you something new, something deeper than you understood before.

Matt is not new to this game. He was co-founder of Voxer, which experienced its Continue reading

Sponsored Post: ScaleArc, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?

Fun and Informative Events

• Learn how Nielsen Marketing Cloud (NMC) leverages online machine learning and predictive personalization to drive its success in a live webinar on Tuesday, September 20 at 11 am PT / 2 pm ET. Hear from Nielsen’s Kevin Lyons, Senior VP of Data Science and Digital Technology, and Brent Keator, VP of Infrastructure, as well as from Brian Bulkowski, CTO and Co-Founder at Aerospike, as they describe the front-edge architecture and technical choices – including the Aerospike NoSQL database – that have led to NMC’s success. RSVP: https://goo.gl/xDQcu4

Cool Products and Services

• ScaleArc's database load balancing software empowers you to “upgrade your Continue reading

Stuff The Internet Says On Scalability For October 7th, 2016

Hey, it's HighScalability time:

 

The worlds oldest analog computer, from 87 BC, the otherworldly Antikythera mechanism.

• 70 billion: facts in Google's knowledge graph; 80 million: monthly visitors to walmart.com; 50%: lower cost for sending a container from Shanghai to Europe; 6 billion: Docker Hub pulls per 6 weeks; 5x: impact reduction using new airbag helmet; 400: node Cassandra + Spark Cluster in Azure; 66%: loss of installs when apps > 100MB; 223GB: Udacity open sources self-driving car data; 


• Quotable Quotes:
  • rfrey: The success of many companies, and probably all of the unicorns, has nothing to do with technology. The tech is necessary, of course, but so are desks and an accounting department. Internalizing that has been difficult for me as an engineer.
  • @mza: 72 new features/services released last month on #AWS. 706 so far this year (up 42.9% YoY).
  • Marc Andreessen: To me the problem is clear: The problem is insufficient technological adoption, innovation, and disruption in these high-escalating price sectors of the economy. My thesis is that we're not in a tech bubble — Continue reading

Stuff The Internet Says On Scalability For September 30th, 2016

Hey, it's HighScalability time:

 

Everything is a network. Map showing the global genetic interaction network of a cell. 

• 18: Google can now drink and drive in Washington DC.; $10 billion: cost of a Vision Quest to Mars; 620 Gbps: DDoS attack on KrebsOnSecurity; 1 Tbps: DDoS attack on OVH; $200,000: cost of a typical cyber incident; 8 million: video training dataset labeled with 4800 labels; 180: Amazon warehouses in the US; 10: bits of info per photon; 16: GPUs in new AI killer P2 instance type;


• Quotable Quotes:
  • @markmccaughrean: 1,000,000 people to Mars in 100 yrs. 10 people/launch? That's 3 a day, every day, for a century. 1% failure rate? One explosion every month
  • @jeremiahg: Any sufficiently advanced exploit is indistinguishable from a 400lb hacker.
  • BrianKrebs: I suggested to Mr. Wright perhaps a better comparison was that ne’er-do-wells now have a virtually limitless supply of Stormtrooper clones that can be conscripted into an attack at a moment’s notice.
  • Sonia: Academia’s not-so-subtle distain for applied research does more than damage a few promising careers; it Continue reading

How Uber Manages a Million Writes Per Second Using Mesos and Cassandra Across Multiple Datacenters

If you are Uber and you need to store the location data that is sent out every 30 seconds by both driver and rider apps, what do you do? That’s a lot of real-time data that needs to be used in real-time.

Uber’s solution is comprehensive. They built their own system that runs Cassandra on top of Mesos. It’s all explained in a good talk by Abhishek Verma, Software Engineer at Uber: Cassandra on Mesos Across Multiple Datacenters at Uber (slides).

Is this something you should do too? That’s an interesting thought that comes to mind when listening to Abhishek’s talk.

Developers have a lot of difficult choices to make these days. Should we go all in on the cloud? Which one? Isn’t it too expensive? Do we worry about lock-in? Or should we try to have it both ways and craft brew a hybrid architecture? Or should we just do it all ourselves for fear of being cloud shamed by our board for not reaching 50 percent gross margins?

Uber decided to build their own. Or rather they decided to weld together their own system by fusing together two very capable open source components. What was Continue reading

Sponsored Post: ScaleArc, Spotify, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?

Fun Continue reading

Stuff The Internet Says On Scalability For September 23rd, 2016

Hey, it's HighScalability time:

 

Will Minority Report for developers really help us program better? (Primitive)

• October 2017: ICANN changes the DNSSEC root keys; $2.91M: cost of running Let's Encrypt; 20%: Amazon convenience tax; 100%: increase in spam; 6.2 km: Quantum teleportation across a metropolitan fibre network; March 18, 1982: birth of containers; 6 months: how long a lightening bolt can power a 60 watt bulb; trillions: EV cache hits per day @ Netflix; 5x: Spark is faster than MapReduce; billions: HTTP, Git and SSH connections served per day at GitHub; 28: # of websites in North Korea; 


• Quotable Quotes:
  • @vgcerf: It is time to admit after 18 years that the multistakeholder model of Internet operation works. #yestoIANA
  • @EricLathrop: Netflix found a 5x performance variation between AWS instances at the same price! They benchmark to avoid overpaying. @indirect #Strangeloop
  • @swardley: Perfectly reasonable @NigelBarron. Larry's statements are ludicrous, play is to milk existing customers whilst hoping to find a new future.
  • @BethanyMacri: Etsy is very anti-SOA. Monolith forever!
  • janfoeh: I've said it before Continue reading

Stuff The Internet Says On Scalability For September 16th, 2016

Hey, it's HighScalability time:

 

The struggle for life that kills. Stunning video of bacteria mutating to defeat antibiotics. 

• 60%: time spent cleaning dirty dirty BigData; 10 million: that's a lot of Raspberry Pi; 365: days living in a Mars simulation; 100M: monthly League of Legends players; 1.75 billion: copyright takedowns by Google; 3.5 petabytes: data Evernote has to move to Google cloud; 11%: YoY growth in time spent on mobile apps; 4 hours: time between Lambda coldstarts; 


• Quotable Quotes:
  • Camille Fournier: humans struggle to tangibly understand domains that are theoretically separate when they are presented as colocated by the source code.
  • @songcarver: The better example: iPhone 7 is showing 115% of 2016 Macbook single core performance, 88% of multi-core.
  • ex3ndr: We (actor.im) also moved from google cloud to our servers + k8s. Shared persistent storage is a huge pain. We eventually stopped to try to do this, will try again when PetSets will be in Beta and will be able to update it's images.
  • @mcclure111: "Well maybe you should get your Continue reading

If Traffic is an Iterated Prisoner’s Dilemma Game Can Smart Cars Evolve Co-operative Behavior?

Can small tribes of cooperating smart cars improve overall traffic even if they are not in the majority? Sure, if every car was a self-driving car maybe traffic jams could dissolve like blood clots on anticoagulants, but what about that messy in-between period? It will be some time before smart cars rule the road. Until then can smart cars make traffic better?

Adoption is hard. This is a general problem in tech. You want people to join your social network yet people won't join until enough people have already joined. What you really want is that virtuous circle to develop, where as more people adopt a technology it causes even more people to adopt it. So startups spend their VC money fast and furiously in hopes of acquiring new customers betting the lifetime value of a customer will be worth the investment. VC money is the dead corpse that feeds the rest of the ecosystem.

Traffic is already an example of a vicious cycle. Horrendous traffic jams are now the norm and "good" traffic windows are just tall tales texted to children. And it keeps on getting worse and not in a worse is better sort of way. Yet the incentives are still not Continue reading

Sponsored Post: ScaleArc, Spotify, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?

Fun Continue reading

The Dollar Shave Club Architecture Unilever Bought for $1 Billion

This is a guest post by Jason Bosco, the Dollar Shave Club’s Director of Engineering, Core Platform & Infrastructure, on the infrastructure of its ecommerce technology.

With more than 3 million members, Dollar Shave Club will do over $200 million in revenue this year. Although most are familiar with the company’s marketing, this immense growth in just a few years since launch is largely due to its team of 45 engineers.

Dollar Shave Club engineering by the numbers:

Core Stats

Stuff The Internet Says On Scalability For September 9th, 2016

Hey, it's HighScalability time:

 

An alternate universe where Zeppelins rule the sky. 1929. (@AeroDork)

• 15%: Facebook's reduction in latency using HTTP2's server push; 1.9x: nanotube transistors outperform silicon; 200: projectors used to film a "hologram"; 50%: of people fall for phishing attacks (it's OK to click); 5x: increased engagement using Google's Progressive Web Apps; 115,000+: Cassandra nodes at Apple; $500 million: Pokémon Go; $150M: Delta's cost for datacenter outage; 


• Quotable Quotes: 
  • Dan Lyons: I wanted to write a book about what it’s like to be 50 and trying to reinvent yourself – that struggle. There are all these books and inspirational speakers talking about being a lifelong learner and it’s so great to reinvent yourself, the brand of you. And I wanted to say, you know, it’s not like that. It’s actually really painful.
  • Engineers & Coffee~ In modern application development everything is a stream now versus historically everything was a transaction. Make a request and the you're done. It's easier to write analytics on top of streams versus using Hive. It's cool Continue reading

Code Generation: The Inner Sanctum of Database Performance

This is guest post by Drew Paroski, architect and engineering manager at MemSQL. Previously he worked at Facebook and developed HHVM, the popular real-time PHP compiler used across the company’s web scale application.

Achieving maximum software efficiency through native code generation can bring superior scaling and performance to any database. And making code generation a first-class citizen of the database, from the beginning, enables a rich set of speed improvements that provide benefits throughout the software architecture and end-user experience.

If you decide to build a code generation system you need to clearly understand the costs and benefits, which we detail in this article. If you are willing to go all the way in the name of performance, we also detail an approach to save you time leveraging existing compiler tools and frameworks such as LLVM in a proven and robust way.

Code Generation Basics

Stuff The Internet Says On Scalability For September 2nd, 2016

Hey, it's HighScalability time:

 

Spectacular iconic drawing of Aurora Borealis as observed in 1872. (Drawings vs. NASA Images)

• 4,000 GB: projected bandwidth used per autonomous vehicle per day; 100K: photos of US national parks; 14 terabytes: code on Github in 1 billion files held in 400K repositories; 25: age of Linux; $5 billion: cost of labor for building Linux; $3800: total maintenance + repairs after 100K miles and 5 years of Tesla ownership; 2%: reduction in Arizona's economy by deporting all illegal immigrants; 15.49TB: available research data; 6%: book readers who are "digital only";


• Quotable Quotes
  • @jennyschuessler: "Destroy the printing press, I beg you, or these evil men will triumph": Venice, 1473
  • @Carnage4Life: Biggest surprise in this "Uber for laundry" app shutting down is that there are still 3 funded startups in the space
  • @tlipcon: "backpressure" is right up there with "naming things" on the top 10 list of hardest parts of programming
  • cmcluck: Please consider K8s [kubernetes] a legitimate attempt to find a better way to build both internal Google systems and the next wave of cloud products in the open with the community. We Continue reading

My Test Tube Filled with DNA is Better than Your Mesos Cluster

We’ve seen computation using slime mold, soap film, water droplets, there’s even a 10,000 Domino Computer. Now DNA can do math In a test tube. Using addition, subtraction, multiplication, and division.

It’s not fast. Calculations can take hours. The upside: they are tiny and can work in wet environments. Think of running calculations in your bloodstream or in cells, like a programmable firewall, to monitor and alert on targeted health metrics and then trigger a localized response. Or if you are writing  science fiction perhaps the ocean could become one giant computer?

The applications already sound like science fiction:

Prior devices for control of chemical reaction networks and DNA doctor applications have been limited to finite-state control, and analog DNA circuits will allow much more sophisticated analog signal processing and control. DNA robotics have allowed devices to operate autonomously (e.g., to walk on a nanostructure) but also have been limited to finite-state control.

Analog DNA circuits can allow molecular robots to include real-time analog control circuits to provide much more sophisticated control than offered by purely digital control. Many artificial intelligence systems (e.g., neural networks and probabilistic inference) that dynamically learn Continue reading

Sponsored Post: Spotify, Aerospike, Exoscale, Host Color, Scalyr, Gusto, LaunchDarkly, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?

Fun Continue reading