The data calculator: data structure design and cost synthesis from first principles and learned cost models

The Data Calculator: data structure design and cost synthesis from first principles and learned cost models Idreos et al., SIGMOD’18

This paper preceded the work on data continuums that we looked at last time, and takes a more general look at interactive and semi-automated design of data structures. A data structure here is defined as a combination of (1) a data layout describing how the data is stored, and (2) algorithms that describe how its basic functionality is achieved over the specific data layout. For data structures with just two different types of nodes (e.g., leaf-nodes and non-leaf nodes in a tree), the authors estimate there are more than 10^{32} possible valid data structure designs! Dozens of new data structures are published each year, but we’re still only scratching the surface.

Our intuition as that most designs (and even inventions) are about combining a small set of fundamental concepts in different ways or tunings… Our vision is to build the periodic table of data structures so we can express their massive design space. We take the first step in the paper, presenting a set of first principles that can synthesize an order of magnitude more data structure designs Continue reading

BGP in 2018 – Part2: BGP Churn

The scalability of BGP as the Internet’s routing protocol is not just dependant on the number of prefixes carried in the routing table. Dynamic routing updates are also part of this story. If the update rate of BGP is growing faster than we can deploy processing capability to match then the routing system will lose data, and at that point the routing system will head into turgid instability. This second part of the report of BGP across 2018 will look at the profile of BGP updates across 2018 to assess whether the stability of the routing system, as measured by the level of BGP update activity, is changing.

Operations guide

One of the most common requests we, as consultants, get from our customers is for an operations guide as the final deliverable for any data center build out. There are a few goals for such a guide:

  • Allow the customer to better transfer knowledge as their teams grow and change.
  • Provide an “as built” guide that explains step-by-step how to deploy and manage the infrastructure.
  • Tie together the operational workflow for all the new components that are leveraged in the modern open-networking paradigm.

Since Scott and I have been working on many operations guides, we thought it would be great to document our process so that customers can write their own operations guides.

The operations guide for web scale networking goes beyond just documenting configuration backups, user account access and change requests though. Web scale networking integrates proven software development processes and as such, the operations guide needs to account for these workflows.

As Built

The starting point of all operations guides is the initial build. Most of the cabling architecture, traffic flows and features, along with decision making and architectural choices, are captured within the High level Design and Low Level Design document. The operations guide on the other Continue reading

Federate oVirt engine authentication to OpenID Connect infrastructure

In this post I will introduce how to integrate OIDC with oVirt engine using Keycloak and LDAP user federation.

Prerequisites: I assume you have already setup the 389ds directory server, but the solution is very similar for any other LDAP provider. As OIDC is not integrated into oVirt directly, we use Apache to do the OIDC authentication for us. The mod_auth_openidc module nicely covers all needed functionality.

Overview

Integrate with external OpenID Connect Identity Provider (IDP) to provide Single Sign-On (SSO) across products that use the IDP for authenticating users. We currently have oVirt SSO for providing unified authentication across Administrator and VM portals. The oVirt engine SSO also provides tokens for REST API clients and supports bearer authentication to reuse tokens to access oVirt engine RESTAPI. With external IDP integration the internal oVirt SSO is disabled and browser users will be redirected to the external IDP for authentication. After successful authentication users can access both Admin and VM portals as they normally do. REST API clients don't have to change, they can still obtain a token from engine SSO and use the token for bearer authentication to access oVirt engine RESTAPI. Engine SSO acts as a proxy obtaining the Continue reading

Enjoy a slice of QUIC, and Rust!

Enjoy a slice of QUIC, and Rust!

During last year’s Birthday Week we announced early support for QUIC, the next generation encrypted-by-default network transport protocol designed to secure and accelerate web traffic on the Internet.

We are not quite ready to make this feature available to every Cloudflare customer yet, but while you wait we thought you might enjoy a slice of quiche, our own open-source implementation of the QUIC protocol written in Rust.

Enjoy a slice of QUIC, and Rust!

Quiche will allow us to keep on top of changes to the QUIC protocol as the standardization process progresses and experiment with new features more easily. Let’s have a quick look at it together.

Simple and genuine ingredients

The main design principle that guided quiche’s initial development was exposing most of the QUIC complexity to applications through a minimal and intuitive API, but without making too many assumptions about the application itself, in order to allow us to reuse the same library in different contexts.

For example, while we think Rust is great, most of the stack that deals with HTTP requests on Cloudflare’s edge network is still written in good ol’ C, which means that our QUIC implementation would need to be integrated into that.

The quiche API can process Continue reading

Community Networks: In Tanzania, Helping to Close the Connectivity Gap

Community established networks, also referred to as “community networks” (CNs), have existed for many years and provide a sustainable solution to address the connectivity gaps that exist in urban, remote, and rural areas around the world. While the global statistics estimate that about half of the world population has access to the Internet, the connectivity gap is wide between the developed and developing countries.

In Tanzania, there are 41.8 million voice telephone subscriptions and only 23 million Internet users. A study by Research ICT Africa reported that when Internet access is compared between rural and urban areas, 86% of rural dwellers remain unconnected to the Internet compared to 44.6% in urban areas. Similarly, in Tanzania, fewer women have access to and use of the Internet than men.

In order to address the connectivity challenges in Tanzania, the Internet Society Tanzania Chapter in partnership with the University of Dodoma, supported by Beyond the Net Funding Programme, has built a pilot project using TV white space as a community network solution. The deployed network has connected four educational institutions in rural Tanzania and at the same time provided Internet access to community members around the schools.

In order Continue reading

Review: 4 open-source network management tools improve usability, performance

Network management tools have come a long way from the early command-line products with arcane, text-based configuration files that kept everyone except the resident (typically Linux) guru in the dark. Today’s management tools, replete with desktop or web-based GUIs, easy installs and configuration wizards, are far more accessible. With each iteration vendors find ways to make these tools more powerful and easier to use. For this review, we evaluated newer versions of three established open-source network management products – OpenNMS, Zenoss Core and NetXMS – as well as a relative newcomer, Sensu Core. All four products are free and open source.To read this article in full, please click here

Review: 4 open-source network management tools improve usability, performance

Network management tools have come a long way from the early command-line products with arcane, text-based configuration files that kept everyone except the resident (typically Linux) guru in the dark. Today’s management tools, replete with desktop or web-based GUIs, easy installs and configuration wizards, are far more accessible. With each iteration vendors find ways to make these tools more powerful and easier to use. For this review, we evaluated newer versions of three established open-source network management products – OpenNMS, Zenoss Core and NetXMS – as well as a relative newcomer, Sensu Core. All four products are free and open source.To read this article in full, please click here

1 … 1,340 1,341 1,342 1,343 1,344 … 3,841