The Story of Batching to Streaming Analytics at Optimizely

Our mission at Optimizely is to help decision makers turn data into action. This requires us to move data with speed and reliability. We track billions of user events, such as page views, clicks and custom events, on a daily basis. To provide our customers with immediate access to key business insights about their users has always been our top most priority. Because of this, we are constantly innovating on our data ingestion pipeline.

In this article we will introduce how we transformed our data ingestion pipeline from batching to streaming to provide our customers with real-time session metrics.

Motivations 

Unification. Previously, we maintained two data stores for different use cases - HBase is used for computing Experimentation metrics, whereas Druid is used for calculating Personalization results. These two systems were developed with distinctive requirements in mind:

Experimentation

Personalization

Instant event ingestion

Delayed event ingestion ok

Query latency in seconds

Query latency in subseconds

Visitor level metrics

Session level metrics

As our business requirements evolve, however, things quickly became difficult to scale. Maintaining a Druid + HBase Lambda architecture (see below) to satisfy these business needs became a technical burden for the engineering team. We need a solution that Continue reading

Cybersecurity fabric vs. a security platform: Fabric wins

The shift to digital has introduced several new technologies into businesses. Internet of Things (IoT), mobility, cloud and the like allow companies to become highly agile and move with speed.However, the increased agility businesses are realizing has come with a price, which is that the complexity of IT has never been higher. There are many implications to increased complexity, but the biggest is that securing the business has become more difficult.INSIDER: 5 ways to prepare for Internet of Things security threats Securing organizations used to be straight forward: Put up a big, expensive firewall at the sole ingress/egress point, and all was good. Today there are dozens or even hundreds of entry points created from an increase in the use of cloud services, mobile workers and consumer devices. Security must now be applied at the perimeter, but also in the data center, campus, cloud, branch offices and anywhere else the business might have assets or people.To read this article in full or to leave a comment, please click here

DataStax buys DataScale, plans to launch managed cloud service

Distributed database software vendor DataStax announced today that it had completed the acquisition of DataScale, a specialist in cloud-based management services for data infrastructure, paving the way for a fully managed version of DataStax's offering in early 2017."There is a huge need for an always-on, distributed database," says Martin Van Ryswyk, executive vice president of Engineering, DataStax. "Whether you're in retail, transportation, the hotel business — everyone has to have an online presence. That has driven this need for this type of database we've built.""What happens in these technology shifts, when something gets really important and really hot at a certain time, you start to get a skills shortage," he adds.To read this article in full or to leave a comment, please click here

SWIFT has not seen its last ‘bank robbery’

A former CSO of the World Bank Treasury calls the SWIFT system outdated and open to malware attacks. Those vulnerabilities could lead to manipulation of financial transactions.SWIFT is the interbank financial messaging system for sending international money transfer instructions. The Society for Worldwide Interbank Financial Telecommunications, which the industry refers to as the SWIFT co-op maintains this system.CSO looks at the SWIFT co-op’s denial of the real issue, the cost of attacks, informed expert insights into these security flaws, how hackers are using and abusing these to their profit, and what the co-op should do to seal its messaging system to mitigate further falsifications.To read this article in full or to leave a comment, please click here

SWIFT has not seen its last ‘bank robbery’

A former CSO of the World Bank Treasury calls the SWIFT system outdated and open to malware attacks. Those vulnerabilities could lead to manipulation of financial transactions.SWIFT is the interbank financial messaging system for sending international money transfer instructions. The Society for Worldwide Interbank Financial Telecommunications, which the industry refers to as the SWIFT co-op maintains this system.CSO looks at the SWIFT co-op’s denial of the real issue, the cost of attacks, informed expert insights into these security flaws, how hackers are using and abusing these to their profit, and what the co-op should do to seal its messaging system to mitigate further falsifications.To read this article in full or to leave a comment, please click here

Beyond brainstorming: 4 cool action items hatched from Cisco ideation session

Matt Cutler, whose startup Cisco bought three years ago, found himself moving from an outfit with eight people to one with tens of thousands. He looked forward to accessing resources beyond his reach at Collaborate.com, but was wary of being able "to get stuff done" within the larger organization.One way he’s been able to get stuff done is through Cisco’s little-known Founders Forum for the entrepreneurial leaders of companies it has acquired. Passionate about the sort of modern workstyles and design thinking (and "a lot of sticky notes") he incorporated at his startup, Cutler recently volunteered to lead a break-out ideation session in an effort to extract great ideas from his peers in the Forum. That of course could be easier said than done when dealing with a large group of intelligent, highly opinionated people.To read this article in full or to leave a comment, please click here

Meet Cisco’s Founders Forum members

Cisco Talent ShowCisco has acquired 193 companies over its history, and in an effort to retain as much talent from its buyouts as possible, the company has created a Founders Forum consisting of 46 entrepreneurs who started and/or led businesses Cisco has purchased. These creative minds meet quarterly to share ideas. Here is a sampling of the members, who are listed alphabetically by acquired company name.To read this article in full or to leave a comment, please click here

Cisco Founders Forum: One creative way to keep & energize talent

The tech industry brims with examples of bright entrepreneurs who have struck it big by selling their startups and then hightailing it out of those larger companies once contractually eligible so that they can pursue their next venture.So what the heck is Matt Cutler still doing at Cisco three years after selling his mobile collaboration startup to the networking giant? Well, among other things, he’s teaching a bunch of his peers who have stayed at Cisco after having their own companies acquired -- as well as any Cisco lifers who will listen -- a thing or two about how to keep cranking out new ideas. Cisco Matt Cutler, Lead Evangelist for Cisco Cloud Collaboration Technologies, has big ideas on ideation.To read this article in full or to leave a comment, please click here

Cisco Founders Forum: One creative way to keep & energize talent

The tech industry brims with examples of bright entrepreneurs who have struck it big by selling their startups and then hightailing it out of those larger companies once contractually eligible so that they can pursue their next venture.So what the heck is Matt Cutler still doing at Cisco three years after selling his mobile collaboration startup to the networking giant? Well, among other things, he’s teaching a bunch of his peers who have stayed at Cisco after having their own companies acquired -- as well as any Cisco lifers who will listen -- a thing or two about how to keep cranking out new ideas. Cisco Matt Cutler, Lead Evangelist for Cisco Cloud Collaboration Technologies, has big ideas on ideation.To read this article in full or to leave a comment, please click here

IDG Contributor Network: Gartner predicts: SD-WANs to replace routers, but which SD-WAN is the question

That SD-WANs will replace routing was not the most important message from last week’s Gartner webinar with Gartner vice president and distinguished analyst Joe Skorupa.No, the biggest message came in some startling statistics. Half the market revenue is held by just two startups, which begs the question: With 30-plus vendors in the SD-WAN space, are you sure the SD-WAN vendor you’re considering has the cash for the long haul?No more routing Back in September, we wrote about the argument for replacing routing with SD-WANs. It’s a message we've been thinking about for some time, listening to the frustrations of many of our enterprise customers. It’s also a trend that Skorupa's market data supports—and for good reason.To read this article in full or to leave a comment, please click here

IDG Contributor Network: Gartner predicts: SD-WANs to replace routers, but which SD-WAN is the question

That SD-WANs will replace routing was not the most important message from last week’s Gartner webinar with Gartner vice president and distinguished analyst Joe Skorupa.No, the biggest message came in some startling statistics. Half the market revenue is held by just two startups, which begs the question: With 30-plus vendors in the SD-WAN space, are you sure the SD-WAN vendor you’re considering has the cash for the long haul?No more routing Back in September, we wrote about the argument for replacing routing with SD-WANs. It’s a message we've been thinking about for some time, listening to the frustrations of many of our enterprise customers. It’s also a trend that Skorupa's market data supports—and for good reason.To read this article in full or to leave a comment, please click here

Microsoft and Google bury the hatchet in one small way

There's no love lost between Google and Microsoft. The two companies have been fiercely competitive with one another in the public cloud, productivity and operating system markets, at times leaving users in the lurch. There's one small glimmer of hope in the relationship between the two companies: Google has joined the .NET Foundation, to help drive forward the programming language framework Microsoft originated. Google will be a part of the technical steering group for the foundation, which helps guide the future of the platform and consults on changes to the .NET roadmap and project release schedule. The foundation oversees projects including .NET Core and the Roslyn .NET compiler.To read this article in full or to leave a comment, please click here

Microsoft doubles down on Linux love, joins foundation

After a long campaign against open source and Linux, Microsoft has for the past few been pushing its love of the popular operating system. On Wednesday, the company made that even more official by joining the Linux Foundation, an organization that shepherds development of the operating system's kernel and provides funding for open source projects.Microsoft also launched the public beta of SQL Server on Linux, the much-anticipated port of the relational database software that was first announced in March. Linux developers can also start working with a beta of Azure App Service, which is designed to take away the work of managing infrastructure for cloud-based apps.To read this article in full or to leave a comment, please click here

Psimulator2 forked, updated

Roland Kuebert forked the psimulator2 network simulator project from the original, seemingly discontinued source and made the new version available at https://github.com/rkuebert/psimulator.

Roland posted this announcement in the comments under my psimulator2 blog post. So that his announcement receives a bit more visibility, I am re-posting his comment verbatim below:

Hi all,

Just a heads up, I forked the project from the original, seemingly discontinued source and it is available at https://github.com/rkuebert/psimulator .

I have fixed the issue preventing the use of Java 8, but I have yet to look into making a release on GitHub. You can, however, clone the repository and use gradle to build jar files – I recommend using gradle shadowJar to create jar files which can be run without specifying any further dependencies.

For the frontend, use java -jar java -jar frontend/build/libs/psimulator-frontend-master-*.jar (replace the asterisk with the exact name, the star represents the git commit you used to checkout).

For the backend, use java -jar backend/build/libs/psimulator-backend-master-*-all.jar (replace the asterisk with the exact name, the star represents the git commit you used to checkout).

Cheers
Roland

Review: Spark lights up machine learning

As I wrote in March of this year, the Databricks service is an excellent product for data scientists. It has a full assortment of ingestion, feature selection, model building, and evaluation functions, plus great integration with data sources and excellent scalability. The Databricks service provides a superset of Spark as a cloud service. Databricks the company was founded by the original developer of Spark, Matei Zaharia, and others from U.C. Berkeley’s AMPLab. Meanwhile, Databricks continues to be a major contributor to the Apache Spark project.To read this article in full or to leave a comment, please click here(Insider Story)

Is critical infrastructure the next DDoS target?

The massive Distributed Denial of Service (DDoS) attack last month on Dyn, the New Hampshire-based Domain Name System (DNS) provider, was mostly an inconvenience.While it took down a portion of the internet for several hours, disrupted dozens of major websites and made national news, nobody died. Nobody even got hurt, other than financially.But the attack, enabled by a botnet of millions of Internet of Things (IoT) devices, inevitably led to speculation on what damage a DDoS of that scale or worse could do to even a portion of the nation’s critical infrastructure (CI).To read this article in full or to leave a comment, please click here

Is critical infrastructure the next DDoS target?

The massive Distributed Denial of Service (DDoS) attack last month on Dyn, the New Hampshire-based Domain Name System (DNS) provider, was mostly an inconvenience.While it took down a portion of the internet for several hours, disrupted dozens of major websites and made national news, nobody died. Nobody even got hurt, other than financially.But the attack, enabled by a botnet of millions of Internet of Things (IoT) devices, inevitably led to speculation on what damage a DDoS of that scale or worse could do to even a portion of the nation’s critical infrastructure (CI).To read this article in full or to leave a comment, please click here