Sponsored Post: Domino Data Lab, Etleap, Aerospike, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• Grokking the System Design Interview is an interactive course that explains the Large Scale Distributed System Design problems in detail. System design questions have become a standard part of the software engineering hiring process. This course is created by hiring managers who’ve been working at Google, Facebook, Microsoft, and Amazon, and is a complete guide to master the System Design Interviews.


• Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)


• What engineering and IT leaders need to know about data science. As data science becomes more mature within an organization, you may be pulled into leading, enabling, and collaborating with data science teams. While Continue reading

Why Morningstar Moved to the Cloud: 97% Cost Reduction

Enterprises won't move to the cloud. If they do, it's tantamount to admitting your IT group sucks. That has been the common wisdom. Morningstar, an investment research provider, is moving to the cloud and they're about as enterprisey as it gets. And they don't strike me as incompetent, it just seems they don't want to worry about all the low level IT stuff anymore. 

Mitch Shue, Morningstar's CTO, gave a short talk at AWS Summit Series 2017 on their move to AWS. It's not full of nitty gritty technical details. That's not the interesting part. The talk is more about their motivations, the process they used to make the move, and some of the results they've experienced. While that's more interesting, we've heard a lot of it before.

What I found most interesting was the idea of Morningstar as a canary test. If Morningstar succeeds, the damn might bust and we'll see a lot more adoption of the cloud by stodgy mainstream enterprises. It's a copy cat world. That sort of precedent gives other CTOs the cover they need to make the same decision.

The most important idea in the whole talk: the cost savings of moving to Continue reading

Stuff The Internet Says On Scalability For August 4th, 2017

Hey, it's HighScalability time:

Hands down the best ever 25,000 year old selfie from Pech Merle cave in southern France. (The Ice Age)

• 35%: US traffic is now IPV6; 10^161: decision points in no-limit Texas hold’em; 4.5 billion: Facebook translations per day; 90%: savings by moving to Lambda; 330TB: IBM's tiny tape cartridge, enough to store 330 million books; $108.9 billion: game revenues in 2017; 85%: of all research papers are on Sci-Hub; 1270x: iPhone 5 vs Apollo guidance computer; 16 zettabytes: 2017 growth in digital universe; 


• Quotable Quotes:
  • Andrew Roberts: [On Napoleon] No aspect of his command was too small to escape notice.
  • Jason Calacanis: The world has trillions of dollars sitting in bonds, cash, stocks, and real estate, which is all really “dead money.” It sits there and grows slowly and safely, taking no risk and not changing the world at all. Wouldn’t it be more interesting if we put that money to work on crazy experiments like the next Tesla, Google, Uber, Cafe X, or SpaceX?
  • @icecrime Continue reading

The Next Scalability Hurdle: Massively Multiplayer Mobile AR

Many moons ago, in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud, I said we still had scaling challenges ahead, that we've not yet begun to scale, that we still don't know how to scale at a planetary level.

That was 7 years ago. Now Facebook has 2 billion monthly users. There's no reason to think they can't scale an unimpressive 3.5x to handle the rest of the planet. WhatsApp is at one billion daily users. YouTube is at 1.5 billion monthly users.

So it appears we do know how to service a whole planet full of people (and bots). At least a select few companies with vast resources know how. We are still no closer to your average developer being able to field a planet scale service. The winner take all nature of the Internet seems to fend off decentralization like it's a plague. Maybe efforts like Filecoin will change the tide. 

There's another area we have scaling challenges: Massively Multiplayer Mobile AR (Augmented Reality). While AR has threatened to be the future for quite some time, it now looks like the future may be just around the virtual Continue reading

Sponsored Post: Apple, Domino Data Lab, Etleap, Aerospike, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Apple is looking for passionate VoIP engineer with a strong technical background to transform our Voice platform to SIP. It will be an amazing journey with highly skilled, fast paced, and exciting team members. Lead and implement the engineering of Voice technologies in Apple’s Contact Center environment. The Contact Center Voice team provides the real time communication platform for customers’ interaction with Apple’s support and retail organizations. You will lead the global Voice, SIP, and network cross-functional engineers to develop world class customer experience. More details are available here.


• Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)


• What engineering and IT leaders need Continue reading

Stuff The Internet Says On Scalability For July 28th, 2017s

Hey, it's HighScalability time:

• 1.8x: faster code on iPad MacBook Pro; 1 billion: WhatsApp daily active users; 100 milliamps: heart stopping current; $25m: surprisingly low take from ransomware; 2,700x: improvement in throughput with TCP BBR; 620: Uber locations; $35.5 billion: Facebook's cash hoard; 2 billion: Facebook monthly active users; #1: Apple is the world's most profitable [legal] company; 500,000x: return on destroying an arms depot with a drone; 


• Quotable Quotes:
  • Alasdair Allan: Jeff Bezos’ statement that “there’s not that much interesting about CubeSats” may well turn out to be the twenty first century’s “nobody needs more than 640kb.”
  • @hardmaru: Decoding the Enigma with RNNs. They trained a LSTM with 3000 hidden units to decode ciphertext with 96%+ accuracy. 
  • @tj_waldorf: Morningstar achieved 97% cost reduction by moving to AWS. #AWSSummit Chicago
  • Ed Sperling: Moore’s Law is alive and well, but it is no longer the only approach. And depending on Continue reading

7 Interesting Parallels Between the Invention of Tiny Satellites and Cloud Computing

CubeSats are revolutionizing space exploration because they are small, modular, and inexpensive to build and launch. On an episode of embedded.fm, Professor Jordi Puig-Suari gives a fascinating interview on the invention of the CubeSat. 195: A BUNCH OF SPUTNIKS.

What struck in the interview is how the process of how the CubeSat was invented parallels how the cloud developed. They followed a very similar path driven by many of the same forces and ideas. 

Just what is a CubeSat? It's a "type of miniaturized satellite for space research that is made up of multiples of 10×10×10 cm cubic units. CubeSats have a mass of no more than 1.33 kilograms per unit, and often use commercial off-the-shelf (COTS) components for their electronics and structure."

Parallel #1:  University as Startup Incubator

Stuff The Internet Says On Scalability For July 21st, 2017

Hey, it's HighScalability time:

• 222x: Bitcoin less efficient than a physical system of metal coins and paper/fabric/plastic; #1: Python use amongst Spectrum readers; 3x: time spent in apps that don't make us happy; 1 million: DigitalOcean users; 11.6 million: barrels of oil a day saved via tech and BigData; 200,000: cores on Cray super computer;$200B: games software/hardware revenue by 2021; $3K: for 50 Teraflops AMD Vega Deep Learning Box; 24.4 Gigawatts: China New Solar In First Half Of 2017; 


• Quotable Quotes:
  • sidlls: I think instead there is a category error being made: that CS is an appropriate degree (on its own) to become a software engineer. It's like suggesting a BS in Physics qualifies somebody to work as an engineer building a satellite.
  • Elon Musk: AI is a fundamental existential risk for human civilization, and I don’t think people fully appreciate that
  • Mike Elgan: Continue reading

You’ll Never Believe the Big Hairy Audacious Startup John Jacob Astor Created in 1808

Think your startup has a Big Hairy Audacious Goal? Along with President Thomas Jefferson, John Jacob Astor  conceived (in 1808), and implemented (in 1810) a plan to funnel the entire tradable wealth of the westernmost sector of the North American continent north of Mexico through his own hands. Early accounts described it as “the largest commercial enterprise the world has ever known.”

Think your startup raised a lot of money? Astor put up $400,000 ($7,614,486 in today's dollars) of his own money, with more committed after the first prototype succeeded.

Think competition is new? John Jacob Astor dealt with rivals in one of three ways: he tried to buy them out; if that didn’t work, he tried to partner with them; if he failed to join them, he tried to crush them.

Think your startup requires commitment? Joining Astor required pledging five years of one’s life to a start-up venture bound for the unknownn.

Think your startup works hard? Voyageur's paddled twelve to fifteen hours per day, with short breaks while afloat for a pipe of tobacco. During that single day each voyageur would make more than thirty thousand paddle strokes. On the upper Great Continue reading

Stuff The Internet Says On Scalability For July 14th, 2017

Hey, it's HighScalability time:

 

• 2 trillion: web requests served daily by Akamai; 9 billion: farthest star ever seen in light-years; 10^31: bacteriophages on earth; 7: peers needed to repair ransomware damage; $30,000: threshold of when to leave AWS; $300K-$400K: beginning cost of running Azure Stack on HPE ProLiant; 3.5M: files in the Microsoft's git repository; 300M: Google's internal image data training set size; 7.2 Mbps: global average connection speed; 85 million: Amazon Prime members; 35%: Germany generated its electricity from renewables;


• Quotable Quotes:
  • Jessica Flack: I believe that science sits at the intersection of these three things — the data, the discussions and the math. It is that triangulation — that’s what science is. And true understanding, if there is such a thing, comes only when we can do the translation between these three ways of representing the world.
  • gonchs: “If your whole business relies on us [Medium], you might Continue reading

Sponsored Post: Apple, Domino Data Lab, Etleap, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Apple is looking for passionate VoIP engineer with a strong technical background to transform our Voice platform to SIP. It will be an amazing journey with highly skilled, fast paced, and exciting team members. Lead and implement the engineering of Voice technologies in Apple’s Contact Center environment. The Contact Center Voice team provides the real time communication platform for customers’ interaction with Apple’s support and retail organizations. You will lead the global Voice, SIP, and network cross-functional engineers to develop world class customer experience. More details are available here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA to learn about key differences between Hadoop, NoSQL and RDBMS. Topics include primary use cases, selection criteria, when a hybrid approach will best fit your needs and best practices for managing, securing and integrating data across platforms. Brian Bulkowski, CTO and Co-founder of Aerospike, presented along with speakers from Cask Data and Splice Machine. View now.


• Advertise your event here!

Cool Products and Services

• What engineering and IT leaders need to know about data science. As data science Continue reading

Stuff The Internet Says On Scalability For July 7th, 2017

Hey, it's HighScalability time:

 

• Vladimir Putin: We must take into account the plans and directions of development of the armed forces of other countries… Our responses must be based on intellectual superiority, they will be asymmetric, and less expensive.
• @swardley: What most fail to realise is that the Chinese corporate corpus has devoured western business thinking and gone beyond it.
• @discostu105: I am a 10X developer. Everything I do takes ten times as long as I thought.
• DINKDINK: You grossly underestimate the hashing capacity of the bitcoin network. The hashing capacity, at time of posting, is approximately 5,000,000,000 Gigahashes/second[1]. Spot measurement of the hashing capacity of an EC2 instance is 0.4 Gigahashes/second[2]. You would need 12 BILLION EC2 instances to 51% attack the bitcoin network.[3] Using EC2 to attack the network is impractical and inefficient.
• danielsamuels && 19eightyfour~ Machiavelli's Guide to PaaS: Keep your friends close, and Continue reading

Gone Fishin’

Well, not exactly Fishin', but I'll be on a month long vacation starting today. I won't be posting (much) new content, so we'll all have a break. Disappointing, I know. Please use this time for quiet contemplation and other inappropriate activities. Au revoir!

Stuff The Internet Says On Scalability For May 26th, 2017

Hey, it's HighScalability time:

 

• 42%: increase US wireless traffic since 2015; 44: age of Ethernet; I $18.5m: low cost of Target data breach; 25 million: record set from Library of Congress; 98%: WannaCry infections on Windows 7; 100 terabytes: daily Pinterest logging; 2020: when Microsoft will have DNA storage in the cloud; 220 μm: size of microbots; 2 billion: lines of code in Google repository; 40%+: esports industry growth; 


• Quotable Quotes:
  • @Werner: There is no compression algorithm for experience.
  • @colinmckerrache: We just crossed over 2m EVs on the road. So yeah, second million took just under 18 months. Next million in about 10 months.
  • @swardley: When discussing China, stop thinking cheap labour, communism & copying ... to understand changes, start thinking World's largest VC.
  • @JOTB17: "Cars generate more than 4Tb of data a day, humans are becoming irrelevant in data collection" ? @saleiva #JOTB17
  • Wojciech Kudla: that's why blacklisting workqueues from critical Continue reading

Sponsored Post: Etleap, Pier 1, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Pier 1 Imports is looking for an amazing Sr. Website Engineer to join our growing team!  Our customer continues to evolve the way she prefers to shop, speak to, and engage with us at Pier 1 Imports.  Driving us to innovate more ways to surprise and delight her expectations as a Premier Home and Decor retailer.  We are looking for a candidate to be another key member of a driven agile team. This person will inform and apply modern technical expertise to website site performance, development and design techniques for Pier.com. To apply please email [email protected]. More details are available here.


• Etleap is looking for Senior Data Engineers to build the next-generation ETL solution. Data analytics teams need solid infrastructure and great ETL tools to be successful. It shouldn't take a CS degree to use big data effectively, and abstracting away the difficult parts is our mission. We use Java extensively, and distributed systems experience is a big plus! See full job description and apply here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA Continue reading

Stuff The Internet Says On Scalability For May 19th, 2017

Hey, it's HighScalability time:

 

• 2 billion: Android users; 1,000: cloud TPUs freely available to researchers; 11.5 petaflops: in Google's machine learning pod; 86 billion: neurons in the human brain, not 100 billion; 1,300: Amazon's new warehouses across Europe; $1 trillion: China self-investment; 1/7th: California's portion of US GDP; more: repetition in songs; 99.999%: Spanner availability, strong consistency, good latency; 6: successful SpaceX launch in 4 months; 160TB: RAM in HPE computer; 40,000+ workers: private offices > open offices


• Quotable Quotes:
  • Tim Bray: with­out ex­cep­tion, I ob­served that they [Per­son­al com­put­er­s, Unix, C, the In­ter­net and We­b, Java, REST, mo­bile, pub­lic cloud] were ini­tial­ly load­ed in the back door by geek­s, with­out ask­ing per­mis­sion, be­cause they got shit done and helped peo­ple with their job­s. That’s not hap­pen­ing with blockchain. Not in the slight­est. Which is why I don’t be­lieve in it.
  • @swardley: Amazon continues to take industry after industry not because those companies lack engineering talent but executive talent.
  • Continue reading

Is Serverless the New Visual Basic?

With Serverless hiring less experienced developers can work out better than hiring experienced cloud developers. That's an interesting point I haven't heard before and it was made by Paul Johnston, CTO of movivo, in The ServerlessCast #6 - Event-Driven Design Thinking.

The thought process goes something like this...

An experienced cloud developer will probably think procedurally, in terms of transactional systems, frameworks, and big fat containers that do lots of work. 

That's not how a Serverless developer needs to think. A Serverless developer needs to think in terms of small functions that do one thing linked together by events; and they need to grok asynchronous and distributed thinking.

So the idea is you don't need typical developer skills. Paul finds people with sysadmin skills have the right stuff. Someone with a sysadmin background is more likely than a framework developer to understand the distributed thinking that goes with building an entire system of events.

Paul also makes the point that once a system has built experienced developers will get bored because Serverless systems don't require the same amount of maintenance.

For example, they had good success hiring a person with two years of vo-tech on-the-job training because they didn't have Continue reading

Stuff The Internet Says On Scalability For May 12th, 2017

Hey, it's HighScalability time:

 

• 1 million: cord cutters in Q1; 500 billion: FINRA validations of stock trades every day on Lambda; 100k: messages sent per hour at Airbnb; 21.1 billion: transistors in GV100 GPU; 11,500: crashes to train a drone; 84,469: Backblaze hard drives; 8,000: questions per day asked on StackOverflow; 


• Quotable Quotes:
  • Jonathan Taplin: Google Is as Close to a Natural Monopoly as the Bell System Was in 1956
  • Tom Goldenberg: more companies on the site [StackShare.io] use JavaScript on the back-end (6,000) than Python (4,100) or Java (3,900).
  • Andrew Shafer: The dark ages of of the relational database and the Java middleware stack paused everything for a decade. 
  • @Taytus: "We are early stage investors. Call me when you hit 1 million monthly active users"
  • @chrisjrn: "At this point I was drunk on Perl" @bradfitz #tweetsincontext #oscon
  • Bryan Cantrill: AWS is underwriting a war on big box retail. 
  • Continue reading

Sponsored Post: Etleap, Pier 1, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Pier 1 Imports is looking for an amazing Sr. Website Engineer to join our growing team!  Our customer continues to evolve the way she prefers to shop, speak to, and engage with us at Pier 1 Imports.  Driving us to innovate more ways to surprise and delight her expectations as a Premier Home and Decor retailer.  We are looking for a candidate to be another key member of a driven agile team. This person will inform and apply modern technical expertise to website site performance, development and design techniques for Pier.com. To apply please email [email protected]. More details are available here.


• Etleap is looking for Senior Data Engineers to build the next-generation ETL solution. Data analytics teams need solid infrastructure and great ETL tools to be successful. It shouldn't take a CS degree to use big data effectively, and abstracting away the difficult parts is our mission. We use Java extensively, and distributed systems experience is a big plus! See full job description and apply here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA Continue reading

Privacy: Bartering Data for Services

Data is the new currency. A phrase we’ve heard frequently in the wake of the story of Unroll.me selling user data to Uber.

Two keys to that story:

• Users didn’t realize their data was being sold.
• Free services can be considered a sophisticated form of phishing attack.

In both cases prevention requires user awareness. How do we get user awareness? Force meaningful disclosure. How do we force meaningful disclosure? Here’s an odd thought: use the tax system.

If data is the new currency then why isn’t exchanging data for use of a service a barter transaction? If a doctor exchanges medical services for chickens, for example, that is a taxable event at fair market value. It's a barter arrangement. A free service that sells user data is similarly bartering the service for data, otherwise said service would not be offered. 

How would it work?

• Service providers send out 1099-Bs to users for the fair market value of the service. Fair market value could be determined using a similar for pay service or as a percentage of the income generated from the data being sold.

• The IRS treats barter transactions as income received. Users would need to pay income Continue reading