Sponsored Post: Apple, Domino Data Lab, Etleap, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Apple is looking for passionate VoIP engineer with a strong technical background to transform our Voice platform to SIP. It will be an amazing journey with highly skilled, fast paced, and exciting team members. Lead and implement the engineering of Voice technologies in Apple’s Contact Center environment. The Contact Center Voice team provides the real time communication platform for customers’ interaction with Apple’s support and retail organizations. You will lead the global Voice, SIP, and network cross-functional engineers to develop world class customer experience. More details are available here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA to learn about key differences between Hadoop, NoSQL and RDBMS. Topics include primary use cases, selection criteria, when a hybrid approach will best fit your needs and best practices for managing, securing and integrating data across platforms. Brian Bulkowski, CTO and Co-founder of Aerospike, presented along with speakers from Cask Data and Splice Machine. View now.


• Advertise your event here!

Cool Products and Services

• What engineering and IT leaders need to know about data science. As data science Continue reading

Stuff The Internet Says On Scalability For July 7th, 2017

Hey, it's HighScalability time:

 

• Vladimir Putin: We must take into account the plans and directions of development of the armed forces of other countries… Our responses must be based on intellectual superiority, they will be asymmetric, and less expensive.
• @swardley: What most fail to realise is that the Chinese corporate corpus has devoured western business thinking and gone beyond it.
• @discostu105: I am a 10X developer. Everything I do takes ten times as long as I thought.
• DINKDINK: You grossly underestimate the hashing capacity of the bitcoin network. The hashing capacity, at time of posting, is approximately 5,000,000,000 Gigahashes/second[1]. Spot measurement of the hashing capacity of an EC2 instance is 0.4 Gigahashes/second[2]. You would need 12 BILLION EC2 instances to 51% attack the bitcoin network.[3] Using EC2 to attack the network is impractical and inefficient.
• danielsamuels && 19eightyfour~ Machiavelli's Guide to PaaS: Keep your friends close, and Continue reading

What is NASA Doing with Big Data? Check this Out

 

Within the time you read the above sentence, NASA could have collected 1.73 gigabytes of data from around 100 missions which are active currently. NASA doesn’t stop doing this and the rate of collection is growing in an exponential manner. So, managing this kind of data is an uphill task for them. But the data which NASA collects is highly precious and its significance is immense in NASA’s science and research. NASA is trying extremely hard to make this data as approachable and accessible as possible for their daily tasks, various predictions in the universe, and for the human well-being through its innovations and creativity.

In version 2.0 of their “Open Government Plan” in the year 2012, NASA discussed, but did not go deeply into the work they have been doing regarding “Big Data” and they believed that they have much more to explore in this field.

We all know what big data is and what its uses are. So, I don’t think there is any need to mention what really big data is and let’s move on with other topic.

NASA’s Big Data Challenge

Gone Fishin’

Well, not exactly Fishin', but I'll be on a month long vacation starting today. I won't be posting (much) new content, so we'll all have a break. Disappointing, I know. Please use this time for quiet contemplation and other inappropriate activities. Au revoir!

Stuff The Internet Says On Scalability For May 26th, 2017

Hey, it's HighScalability time:

 

• 42%: increase US wireless traffic since 2015; 44: age of Ethernet; I $18.5m: low cost of Target data breach; 25 million: record set from Library of Congress; 98%: WannaCry infections on Windows 7; 100 terabytes: daily Pinterest logging; 2020: when Microsoft will have DNA storage in the cloud; 220 μm: size of microbots; 2 billion: lines of code in Google repository; 40%+: esports industry growth; 


• Quotable Quotes:
  • @Werner: There is no compression algorithm for experience.
  • @colinmckerrache: We just crossed over 2m EVs on the road. So yeah, second million took just under 18 months. Next million in about 10 months.
  • @swardley: When discussing China, stop thinking cheap labour, communism & copying ... to understand changes, start thinking World's largest VC.
  • @JOTB17: "Cars generate more than 4Tb of data a day, humans are becoming irrelevant in data collection" ? @saleiva #JOTB17
  • Wojciech Kudla: that's why blacklisting workqueues from critical Continue reading

Sponsored Post: Etleap, Pier 1, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Pier 1 Imports is looking for an amazing Sr. Website Engineer to join our growing team!  Our customer continues to evolve the way she prefers to shop, speak to, and engage with us at Pier 1 Imports.  Driving us to innovate more ways to surprise and delight her expectations as a Premier Home and Decor retailer.  We are looking for a candidate to be another key member of a driven agile team. This person will inform and apply modern technical expertise to website site performance, development and design techniques for Pier.com. To apply please email [email protected]. More details are available here.


• Etleap is looking for Senior Data Engineers to build the next-generation ETL solution. Data analytics teams need solid infrastructure and great ETL tools to be successful. It shouldn't take a CS degree to use big data effectively, and abstracting away the difficult parts is our mission. We use Java extensively, and distributed systems experience is a big plus! See full job description and apply here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA Continue reading

Stuff The Internet Says On Scalability For May 19th, 2017

Hey, it's HighScalability time:

 

• 2 billion: Android users; 1,000: cloud TPUs freely available to researchers; 11.5 petaflops: in Google's machine learning pod; 86 billion: neurons in the human brain, not 100 billion; 1,300: Amazon's new warehouses across Europe; $1 trillion: China self-investment; 1/7th: California's portion of US GDP; more: repetition in songs; 99.999%: Spanner availability, strong consistency, good latency; 6: successful SpaceX launch in 4 months; 160TB: RAM in HPE computer; 40,000+ workers: private offices > open offices


• Quotable Quotes:
  • Tim Bray: with­out ex­cep­tion, I ob­served that they [Per­son­al com­put­er­s, Unix, C, the In­ter­net and We­b, Java, REST, mo­bile, pub­lic cloud] were ini­tial­ly load­ed in the back door by geek­s, with­out ask­ing per­mis­sion, be­cause they got shit done and helped peo­ple with their job­s. That’s not hap­pen­ing with blockchain. Not in the slight­est. Which is why I don’t be­lieve in it.
  • @swardley: Amazon continues to take industry after industry not because those companies lack engineering talent but executive talent.
  • Continue reading

Is Serverless the New Visual Basic?

With Serverless hiring less experienced developers can work out better than hiring experienced cloud developers. That's an interesting point I haven't heard before and it was made by Paul Johnston, CTO of movivo, in The ServerlessCast #6 - Event-Driven Design Thinking.

The thought process goes something like this...

An experienced cloud developer will probably think procedurally, in terms of transactional systems, frameworks, and big fat containers that do lots of work. 

That's not how a Serverless developer needs to think. A Serverless developer needs to think in terms of small functions that do one thing linked together by events; and they need to grok asynchronous and distributed thinking.

So the idea is you don't need typical developer skills. Paul finds people with sysadmin skills have the right stuff. Someone with a sysadmin background is more likely than a framework developer to understand the distributed thinking that goes with building an entire system of events.

Paul also makes the point that once a system has built experienced developers will get bored because Serverless systems don't require the same amount of maintenance.

For example, they had good success hiring a person with two years of vo-tech on-the-job training because they didn't have Continue reading

Stuff The Internet Says On Scalability For May 12th, 2017

Hey, it's HighScalability time:

 

• 1 million: cord cutters in Q1; 500 billion: FINRA validations of stock trades every day on Lambda; 100k: messages sent per hour at Airbnb; 21.1 billion: transistors in GV100 GPU; 11,500: crashes to train a drone; 84,469: Backblaze hard drives; 8,000: questions per day asked on StackOverflow; 


• Quotable Quotes:
  • Jonathan Taplin: Google Is as Close to a Natural Monopoly as the Bell System Was in 1956
  • Tom Goldenberg: more companies on the site [StackShare.io] use JavaScript on the back-end (6,000) than Python (4,100) or Java (3,900).
  • Andrew Shafer: The dark ages of of the relational database and the Java middleware stack paused everything for a decade. 
  • @Taytus: "We are early stage investors. Call me when you hit 1 million monthly active users"
  • @chrisjrn: "At this point I was drunk on Perl" @bradfitz #tweetsincontext #oscon
  • Bryan Cantrill: AWS is underwriting a war on big box retail. 
  • Continue reading

Sponsored Post: Etleap, Pier 1, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Pier 1 Imports is looking for an amazing Sr. Website Engineer to join our growing team!  Our customer continues to evolve the way she prefers to shop, speak to, and engage with us at Pier 1 Imports.  Driving us to innovate more ways to surprise and delight her expectations as a Premier Home and Decor retailer.  We are looking for a candidate to be another key member of a driven agile team. This person will inform and apply modern technical expertise to website site performance, development and design techniques for Pier.com. To apply please email [email protected]. More details are available here.


• Etleap is looking for Senior Data Engineers to build the next-generation ETL solution. Data analytics teams need solid infrastructure and great ETL tools to be successful. It shouldn't take a CS degree to use big data effectively, and abstracting away the difficult parts is our mission. We use Java extensively, and distributed systems experience is a big plus! See full job description and apply here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA Continue reading

Privacy: Bartering Data for Services

Data is the new currency. A phrase we’ve heard frequently in the wake of the story of Unroll.me selling user data to Uber.

Two keys to that story:

• Users didn’t realize their data was being sold.
• Free services can be considered a sophisticated form of phishing attack.

In both cases prevention requires user awareness. How do we get user awareness? Force meaningful disclosure. How do we force meaningful disclosure? Here’s an odd thought: use the tax system.

If data is the new currency then why isn’t exchanging data for use of a service a barter transaction? If a doctor exchanges medical services for chickens, for example, that is a taxable event at fair market value. It's a barter arrangement. A free service that sells user data is similarly bartering the service for data, otherwise said service would not be offered. 

How would it work?

• Service providers send out 1099-Bs to users for the fair market value of the service. Fair market value could be determined using a similar for pay service or as a percentage of the income generated from the data being sold.

• The IRS treats barter transactions as income received. Users would need to pay income Continue reading

Stuff The Internet Says On Scalability For May 5th, 2017

Hey, it's HighScalability time:

 

• 25ms: SpaceX satellite latency; 17 million: tax returns received by IRS during week ending April 21; 1.94 billion: Facebook users; 1.2 billion: Lambda requests by Expedia / month; ~$91.5K: Capital One's yearly Serverless TCO; 1.2 billion: Facebook Messenger users; 215 petabytes: storage per gram of DNA; 1/2: households in US are Amazon Prime members; 50.8%: households in US that are mobile phone only; 80 billion: street view images; 3 million: open sourced Instacart orders; $175: RaaS (ransomware-as-a-service); 350,000+: Amazon employees; 


• QuotableQuotes:
  • Paul Barnum: You can have a second computer when you've shown you know how to use the first one
  • @chrisalbon: 2007: “You are the product.”  2017: “You are the training data.”
  • shitloadofbooks: As an Ops guy, I preach Ansible + systemd all day everyday, but so many of our Devs (and Ops) have drunk the containerization Kool-aid.
  • roland-s: Like you, Continue reading

Homegrown master-master replication for a NoSQL database

Many of you may have already heard about the high performance of the Tarantool DBMS, about its rich toolset and certain features. Say, it has a really cool on-disk storage engine called Vinyl, and it knows how to work with JSON documents. However, most articles out there tend to overlook one crucial thing: usually, Tarantool is regarded simply as storage, whereas its killer feature is the possibility of writing code inside it, which makes working with your data extremely effective. If you’d like to know how igorcoding and I built a system almost entirely inside Tarantool, read on.

If you’ve ever used the Mail.Ru email service, you probably know that it allows collecting emails from other accounts. If the OAuth protocol is supported, we don’t need to ask a user for third-party service credentials to do that — we can use OAuth tokens instead. Besides, Mail.Ru Group has lots of projects that require authorization via third-party services and need users’ OAuth tokens to work with certain applications. That’s why we decided to build a service for storing and updating tokens.

I guess everybody knows what an OAuth token looks like. To refresh your memory, it’s a structure consisting of 3–4 fields:

The AdStage Migration from Heroku to AWS

This is a guest repost by G Gordon Worley III, Head of Site Reliability Engineering at AdStage.

When I joined AdStage in the Fall of 2013 we were already running on Heroku. It was the obvious choice: super easy to get started with, less expensive than full-sized virtual servers, and flexible enough to grow with our business. And grow we did. Heroku let us focus exclusively on building a compelling product without the distraction of managing infrastructure, so by late 2015 we were running thousands of dynos (containers) simultaneously to keep up with our customers.

We needed all those dynos because, on the backend, we look a lot like Segment, and like them many of our costs scale linearly with the number of users. At $25/dyno/month, our growth projections put us breaking $1 million in annual infrastructure expenses by mid-2016 when factored in with other technical costs, and that made up such a large proportion of COGS that it would take years to reach profitability. The situation was, to be frank, unsustainable. The engineering team met to discuss our options, and some quick calculations showed us we were paying more than $10,000 a month for the convenience of Continue reading

Stuff The Internet Says On Scalability For April 28th, 2017

Hey, it's HighScalability time:

 

• 220,000-Core: largest Google Compute Engine job; 100 million: Netflix subscribers; 1.3M: Sling TV subscribers; 200: Downloadable Modern Art Books; 25%: Americans Won't Subscribe To Traditional Cable; 84%: image payload savings using smart CDN; 10^5: number of world-wide cloud data centers needed; 63%: more Facebook clicks using personality targeting; 2.5 million: red blood cells created per second; 


• Quotable Quotes:
  • Silicon Valley~ The only reason Gilfoyle and I stayed up 48 f*cking straight hours was to decrease server load, not keep it the same. 
  • Robert Graham: In other words, if the entire Mirai botnet of 2.5 million IoT devices was furiously mining bitcoin, it's total earnings would be $0.25 (25 cents) per day.
  • @BoingBoing: John Deere just told US Copyright office that only corporations can own Continue reading

Sponsored Post: Etleap, Pier 1, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Pier 1 Imports is looking for an amazing Sr. Website Engineer to join our growing team!  Our customer continues to evolve the way she prefers to shop, speak to, and engage with us at Pier 1 Imports.  Driving us to innovate more ways to surprise and delight her expectations as a Premier Home and Decor retailer.  We are looking for a candidate to be another key member of a driven agile team. This person will inform and apply modern technical expertise to website site performance, development and design techniques for Pier.com. To apply please email [email protected]. More details are available here.


• Etleap is looking for Senior Data Engineers to build the next-generation ETL solution. Data analytics teams need solid infrastructure and great ETL tools to be successful. It shouldn't take a CS degree to use big data effectively, and abstracting away the difficult parts is our mission. We use Java extensively, and distributed systems experience is a big plus! See full job description and apply here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA Continue reading

Stuff The Internet Says On Scalability For April 21st, 2017

Hey, it's HighScalability time:

• year 1899: “Nobody has to use the Internet”; 12MPH: Speed news of Lincoln's assassination traveled the US; $200 million: Lyft tips; 500: data structures and algorithms interview questions; %0.00244140625: Odds of 13 straight male Dr. Who regens; 100: gigafactories could power the world; 100K: bots on Messenger; 1 million: containers Netflix lanched in one week; 5.2 trillion: 2014 US revenue; 52,129: iterations to converge on NFL schedule; 36 Gbps: Facebook's network in the sky; 


• Quotable Quotes:
  • @mipsytipsy: "That doesn't sound hard. I could build that in a weekend."
  • @Noahpinion: The Elon Musk Future is the good future. The Peter Thiel Future is the bad future. But honestly you'll probably get the Jeff Bezos Future.
  • @BenedictEvans: In 2007 Google, Apple, Facebook & Amazon had maybe 50k staff between them. Today it's more like 400k.
  • @AWSonAir: @Expedia inserting 70,000 rows per second of hotel data with Amazon Aurora.
  • @swardley: STOP! If you're thinking of moving Continue reading

Stuff The Internet Says On Scalability For April 14th, 2017

Hey, it's HighScalability time:

 

• 10^15: synapses activated per second in human brain (2/3rds fail); $4.5B: Amazon spend on video (Netflix $6 billion); 22,000: AWS database migrations served; ~15%: Dropbox reduced CPU usage using Brotli; $3.5 trillion: IT spending in 2017; 10%: reduction in QoQ hard drive shipments; 33.3%: Nginx share of webserver market; 37.2 trillion: human cells in a Cell Atlas; 6.2 miles: journey to the center of the earth; 200: lines of code for blockchain; 95%: Wikipedia pages end up at philosophy; 1.2 billion: Messenger monthly users; 


• Quotable Quotes:
  • Jeff Bezos: Day 2 is stasis. Followed by irrelevance. Followed by excruciating, painful decline. Followed by death. And that is why it is always Day 1.
  • Bob Schmidt: If debugging is the process of removing errors from a design, then designing must be the process of putting errors into a design!
  • @swardley Continue reading

Sponsored Post: Pier 1, Aerospike, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

• Pier 1 Imports is looking for an amazing Sr. Website Engineer to join our growing team!  Our customer continues to evolve the way she prefers to shop, speak to, and engage with us at Pier 1 Imports.  Driving us to innovate more ways to surprise and delight her expectations as a Premier Home and Decor retailer.  We are looking for a candidate to be another key member of a driven agile team. This person will inform and apply modern technical expertise to website site performance, development and design techniques for Pier.com. To apply please email [email protected]. More details are available here.


• Etleap is looking for Senior Data Engineers to build the next-generation ETL solution. Data analytics teams need solid infrastructure and great ETL tools to be successful. It shouldn't take a CS degree to use big data effectively, and abstracting away the difficult parts is our mission. We use Java extensively, and distributed systems experience is a big plus! See full job description and apply here.


• Advertise your job here! 

Fun and Informative Events

• DBTA Roundtable Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Thursday April 20, 2017 | 11:00 AM Pacific Time. Continue reading

Five things we’ve learned about monitoring containers and their orchestrators

This is a guest post by Apurva Davé, who is part of the product team at Sysdig.

Having worked with hundreds of customers on building a monitoring stack for their containerized environments, we’ve learned a thing or two about what works and what doesn’t. The outcomes might surprise you - including the observation that instrumentation is just as important as the application when it comes to monitoring.

In this post, I wanted to cover some details around what it takes to build a scale-out, highly reliable monitoring system to work across tens of thousands of containers. I’ll share a bit about what our infrastructure looks like, the design choices we made, and tradeoffs. The five areas I’ll cover:

• Instrumenting the system

• Relating your data to your applications, hosts, and containers.

• Leveraging orchestrators

• Deciding what to data to store

• How to enable troubleshooting in containerized environments

For context, Sysdig is the container monitoring company. We’re based on the open source Linux troubleshooting project by the same name. The open source project allows you to see every single system call down to process, arguments, payload, and connection on a single host. The commercial offering turns all this data into thousands of Continue reading