ScootR: scaling R dataframes on dataflow systems

ScootR: scaling R dataframes on dataflow systems Kunft et al., SoCC’18

The language of big data is Java ( / Scala). The languages of data science are Python and R. So what do you do when you want to run your data science analysis over large amounts of data?

…programming languages with rich support for data manipulation and statistics, such as R and Python, have become increasingly popular… [but]… they are typically designed for single machine and in-memory usage…. In contrast, parallel dataflow systems, such as Apache Flink and Apache Spark, are able to handle large amounts of data. However, data scientists are often unfamiliar with the systems’ native language and programming abstraction, which is crucial to achieve good performance.

A tempting solution is to embed Python / R support within the dataflow engine. There are two basic approaches to this today:

  1. Keep the guest language components in a separate process and use IPC (inter-process communication) to exchange input and output data between the dataflow engine and the guest language process. This approach can support the full power of the guest language, but pays a heavy price in IPC and serialisation costs.
  2. Use source-to-source (STS) translation to translate guest Continue reading

Some notes about HTTP/3

HTTP/3 is going to be standardized. As an old protocol guy, I thought I'd write up some comments.

Google (pbuh) has both the most popular web browser (Chrome) and the two most popular websites (#1 Google.com #2 Youtube.com). Therefore, they are in control of future web protocol development. Their first upgrade they called SPDY (pronounced "speedy"), which was eventually standardized as the second version of HTTP, or HTTP/2. Their second upgrade they called QUIC (pronounced "quick"), which is being standardized as HTTP/3.


SPDY (HTTP/2) is already supported by the major web browser (Chrome, Firefox, Edge, Safari) and major web servers (Apache, Nginx, IIS, CloudFlare). Many of the most popular websites support it (even non-Google ones), though you are unlikely to ever see it on the wire (sniffing with Wireshark or tcpdump), because it's always encrypted with SSL. While the standard allows for HTTP/2 to run raw over TCP, all the implementations only use it over SSL.

There is a good lesson here about standards. Outside the Internet, standards are often de jure, run by government, driven by getting all major stakeholders in a room and hashing it out, then using rules to force people to adopt it. Continue reading

OpenMP Reaches Into The Parallel Universe Of GPUs

OpenMP is probably the most popular tool in the world to parallelize applications running on processors, but ironically it is not a product, but rather a specification that those who make compilers and middleware use to implement their own ways of parallelizing code to run on multicore processors and now, GPU accelerators. …

OpenMP Reaches Into The Parallel Universe Of GPUs was written by Timothy Prickett Morgan at .

JT65 with SDR

JT65 is a slow protocol for propagation reports. In short it takes 60 seconds to send 13 characters. Then you wait 60 seconds for a reply, and repeat.

The 60 seconds are actually 1 second silence, 46.872 second of signal, then another 12.128 seconds of silence, allowing for clock drifts and for a human to choose the reply.

The mode is this slow in order to add a lot of redundancy and to make it easier for the receiver to dig out a signal way below the noise floor. It was originally meant for making contacts by bouncing signals off the moon, which has a path loss of ~250dB. Someone even managed a JT65 moonbounce on 10 Watts using JT65 with gear you and I could buy/build!. That’s the power of a low energy light bulb!

I wanted to do propagation experiments with SDR, with low power in various frequency bands, but couldn’t find a GNURadio module for JT65. So I made one.

The JT65 specification is very well written, except for the parts it says “the code is the specification”. Which would normally be fine, but the code is in Fortran and Fortran is terrible.

JT65A is Continue reading

Cloudflare’s Response to a Privacy Framework

Cloudflare's Response to a Privacy Framework
Photo by Dayne Topkin / Unsplash
Cloudflare's Response to a Privacy Framework

Why We Weighed In on US Privacy Efforts

Cloudflare’s mission is to help build a better internet, and privacy has to be at the heart of that effort. That’s why we submitted comments last week on the National Telecommunications and Information Administration (NTIA)’s request for comment on its proposed approach to advance consumer privacy.

We think it is important for Internet infrastructure companies like us to be a part of the conversation about the future of internet privacy. We want to advocate for an internet that remains accessible to all, while becoming more secure and protective of privacy.

What is NTIA and what is it trying to do?

In 2018, we’ve seen high profile data breaches and data misuse, Europe’s sweeping data protection law – the General Data Protection Regulation (GDPR) – come into effect, and California pass its own comprehensive Consumer Privacy Act (CCPA). All of this has captured the attention of Washington, D.C. lawmakers and regulators.

On September 25, 2018, NTIA began a process to solicit feedback from stakeholders on a proposed approach to consumer data privacy.  NTIA is the Executive Branch agency in the Department of Commerce that is principally Continue reading

Turning The CPU-GPU Hybrid System On Its Head

Sales of various kinds of high performance computing – not just technical simulation and modeling applications, but also cryptocurrency mining, massively multiplayer gaming, video rendering, visualization, machine learning, and data analytics – run on little boom-bust cycles that make it difficult for all suppliers to this market to make projections when they look ahead. …

Turning The CPU-GPU Hybrid System On Its Head was written by Timothy Prickett Morgan at .

IBM reinforces enterprise multicloud growth with automation tools, ServiceNow expansion

IBM is looking to make it easier for customers to move to multicloud environments by adding automation tools to its cloud services, and the company is extending its relationship with cloud migration specialists ServiceNow.The driving idea behind both moves is to help customer simplify what can be a daunting task – moving new and legacy applications to multicloud environments be they based on IBM's own cloud service or others such as Amazon Web Services, Google Cloud, Microsoft Azure and Red Hat.As a backdrop to the new services, IBM last month said it would buy open-source software pioneer Red Hat in a $34 billion stock acquisition.  For IBM the deal could mean many things. It makes it a bigger open source and enterprise software player for example, but mostly it gets Big Blue into the lucrative hybrid-cloud party targeting its towering competitors Google, Amazon and Microsoft among others. Gartner says that market will be worth $240 billion by next year.To read this article in full, please click here

Weekly Show 416: From Reactive To Proactive NetOps With Cisco NAE (Sponsored)

Today's sponsored Weekly Show dives into Cisco's Network Assurance Engine. This software creates a real-time and continuously updated model of the network that IT can use to assess the impact of changes, get deep visibility into network state, and move network ops from reactive to proactive. We hear from an IT engineer inside Cisco who uses NAE in production.

The post Weekly Show 416: From Reactive To Proactive NetOps With Cisco NAE (Sponsored) appeared first on Packet Pushers.

Gotcha pricing from the cloud pushes workloads back on premises

A new survey by cloud software vendor Nutanix finds that most firms are embracing the hybrid model, but few have actually achieved it. And many are shifting their workloads back on premises because of cloud costs.This was Nutanix’s first global Enterprise Cloud Index, so it doesn’t have historical data by which to measure, but its initial findings match what we’ve known for a while. Read also: How to make hybrid cloud work The hybrid cloud, a mix of on-premises and public cloud computing, working in tandem is the preferred method for most firms; 91 percent to be exact. But only 19 percent of firms surveyed said they have that model today. One reason is that app vendors make it hard to operate in hybrid mode, said Wendy Pfeiffer, CIO at Nutanix.To read this article in full, please click here

1 … 1,387 1,388 1,389 1,390 1,391 … 3,832