Egnyte Architecture: Lessons learned in building and scaling a multi petabyte content platform

 

This is a guest post by Kalpesh Patel, an Engineer, who for  Egnyte from home. Egnyte is a Secure Content Platform built specifically for businesses. He and his colleagues spend their productive hours scaling large distributed file systems. You can reach him at @kpatelwork.

Introduction

Your Laptop has a filesystem used by hundreds of processes. There are a couple of downsides though in case you are looking to use it to support tens of thousands of users working on hundreds of millions of files simultaneously containing petabytes of data. It is limited by the disk space; it can’t expand storage elastically; it chokes if you run few I/O intensive processes or try collaborating  with 100 other users. Let’s take this problem and transform it to a cloud-native file system used by millions of paid users spread across the globe and you get an idea of our roller coaster ride of scaling this system to meet monthly growth and SLA requirements while providing stringent consistency and durability characteristics we all have come to expect from our laptops. 

Egnyte is a secure Content Collaboration and Data Governance platform, founded in 2007 when Google drive wasn't born and AWS S3 Continue reading

Network Break 262: Extreme Announces Fabric Automation And New Switches; Google Rolls Out Smarter Cloud Networking

Today's Network Break episode discusses new data center software and switches from Extreme, new networking cloud tools from Google, Microsoft's support for DoH, an internal re-organization at Cisco and more tech news.

The post Network Break 262: Extreme Announces Fabric Automation And New Switches; Google Rolls Out Smarter Cloud Networking appeared first on Packet Pushers.

The Week in Internet News: Balloon-Based Internet Comes to the Amazon

Internet from the skies: Loon, Google’s sister company, is teaming up with Internet provider Telefonica to provide Internet access to remote areas of the Amazon rainforest in Peru, TechCrunch reports. Loon, the high-altitude balloon company, plans to have the service available in 2020. The area of Peru targeted by the service has about 200,000 residents.

Internet from the highway: Meanwhile, Osceola County Schools in Florida has equipped an unused bus with computer equipment in an effort to bring Internet access to homeless students living in motels, WSBTV.com reports. The school district, south of Orlando, has about 500 students living in motels, some with limited Internet access.

Investigating encryption: A top official at the U.S. Department of Justice has hinted that end-to-end encryption services could be part of a sweeping investigation into some big tech companies, the New York Times reports. The DOJ and law enforcement agencies from other countries have been pushing large tech companies like Facebook to drop their end-to-end encryption services, to the chagrin of many security experts.

Iran shuts it down: The Iranian government shut down Internet access for citizens for several days in response to protests about huge hikes in fuel prices, CNN.com reports. Continue reading

What’s New In Red Hat Ansible Tower 3.6

blog_whats-new-Tower-3.6

As part of the release of Red Hat Ansible Automation Platform, we’re happy to announce the release of Red Hat Ansible Tower 3.6. Ansible Tower is the scalable execution framework of the Ansible Automation Platform, providing an API around automation that you can use to scale automation across your enterprise and integrate automation into your tools and processes.

 

Workflow Pause and Approvals

Not all automation processes can proceed entirely without human input. In Ansible Tower 3.6, we’ve added pause and approval to Ansible Tower workflows to help enable more flexible automation. At any step in a workflow you can pause and wait for an approval from an administrator, or any other you delegate approval permissions to. Need to verify that your deployment was fully successful before updating the external DNS entries? Need to ensure that your developers won’t spin up 300 extra cloud servers when provisioning new dev environments? Now you can do that, integrated directly in Ansible Tower workflows.

 

Customizable Notifications

Notifications were introduced in Ansible Tower 3.0, allowing the status of any job to be reported out via email, Slack, IRC, and more. In Ansible Tower 3.6, we’ve made the content Continue reading

Best Analysis Finalist – Cisco IT Blog Awards for 2019

This blog was selected as a finalist in Cisco BLOG Awards in the Best Analysis category, the category for resources that provide insightful discussions and help for networking architects around the world. Fancy right? Do you agree? Go and vote, it’s the second one on the list ? https://www.ciscofeedback.vovici.com/se/705E3ECD18791A68

The post Best Analysis Finalist – Cisco IT Blog Awards for 2019 appeared first on How Does Internet Work.

Upcoming Workshops: NSX, ACI, VXLAN, EVPN, DCI and More

I’m running two workshops in Zurich in the next 10 days:

I published the slide deck for the NSX versus ACI workshop a few days ago (and you can already download it if you have a paid ipSpace.net subscription) and it’s full of new goodness like ACI vPod, multi-pod ACI, multi-site ACI, ACI-on-AWS, and multi-site NSX-V and NSX-T.

Mergeable replicated data types – Part I

Mergeable replicated data types Kaki et al., OOPSLA’19

This paper was published at OOPSLA, but perhaps it’s amongst the distributed systems community that I expect there to be the greatest interest. Mergeable Replicated Data Types (MRDTs) are in the same spirit as CRDTs but with the very interesting property that they compose. Furthermore, a principled approach for deriving MRDTs is provided, allowing a wide range of data-types to be lifted to their mergeable, replicated counterparts with very little effort on the part of the programmer. The paper sets the discussion in the context of geo-replicated distributed systems, but of course the same mechanisms could be equally useful in the context of local-first applications.

There’s a lot of material in this paper’s 29 pages, and to make sure I have the space to properly understand the ideas (the process of summary writing is a critical part of that for me), I’m going to spread my coverage over two posts.

We’ll begin by looking at the context and the high-level idea behind MRDTs, then we’ll investigate how merge specifications can be derived and composed, and how merge implementations can be derived too. Finally we’ll touch on the implementation of MRDTs in Continue reading

8 ways to prepare your data center for AI’s power draw

As artificial intelligence takes off in enterprise settings, so will data center power usage. AI is many things, but power efficient is not one of them.For data centers running typical enterprise applications, the average power consumption for a rack is around 7 kW. Yet it’s common for AI applications to use more than 30 kW per rack, according to data center organization AFCOM. That’s because AI requires much higher processor utilization, and the processors – especially GPUs – are power hungry. Nvidia GPUs, for example, may run several orders of magnitude faster than a CPU, but they also consume twice as much power per chip. Complicating the issue is that many data centers are already power constrained.To read this article in full, please click here

8 ways to prepare your data center for AI’s power draw

As artificial intelligence takes off in enterprise settings, so will data center power usage. AI is many things, but power efficient is not one of them.For data centers running typical enterprise applications, the average power consumption for a rack is around 7 kW. Yet it’s common for AI applications to use more than 30 kW per rack, according to data center organization AFCOM. That’s because AI requires much higher processor utilization, and the processors – especially GPUs – are power hungry. Nvidia GPUs, for example, may run several orders of magnitude faster than a CPU, but they also consume twice as much power per chip. Complicating the issue is that many data centers are already power constrained.To read this article in full, please click here

The many faces of awk

If you only use awk when you need to select specific fields from lines of text, you might be missing out on a lot of other services that the command can provide. In this post, we'll look at this simple use along with many other things that awk can do for you with enough examples to show you that the command is a lot more flexible than you might have imagined.Plucking out columns of data The easiest and most commonly used service that awk provides is selecting specific fields from files or from data that is piped to it. With the default of using white space as a field separator, this is very simple:To read this article in full, please click here

Worth Reading: Early History of Usenet

Steve Bellovin wrote a great series of articles describing the early history of Usenet. The most interesting part in the “security and authentication” part was probably this gem:

That left us with no good choices. The infrastructure for a cryptographic solution was lacking. The uux command rendered illusory any attempts at security via the Usenet programs themselves. We chose to do nothing. That is, we did not implement fake security that would give people the illusion of protection but not the reality.

A lot of other early implementers chose the same route, resulting in SMTP, BGP… which wouldn’t be a problem if someone kept track of that and implemented security a few years later. Unfortunately we considered those problems solved and moved on to chase other squirrels. We’re still paying the interest on that technical debt.

CCDE Study Guide

CCDE Study Guide – Are you looking for a book that will teach you all the topics on advanced technical networking? If so, I would be very pleased to recommend CCDE Study Guide written by Marwan Al-Shawi to you.

As one of the professionals who contributed immensely to this book, I must admit that Marwan wrote this book in collaboration with a number of savvy designers. IT experts who contributed to this wonderful book include Russ White, Andre Laurent, Denise Fishbourne, Ivan Papeljnak, and Orhan Ergun. In fact, all the IT concepts in this book are enlightening! The book has many drawings, which will assist learners to understand network design.

Today, I spoke with one of my old friend, an expert in CCDE, who read Marwan’s book, and his comment was this: “The book contains pictures that explain a thousand words.”

The most important topics of the networking design, especially for the CCDE exam, are layer 3 technologies such as IGP, BGP, MPLS, Inter-AS MPLS, and IPv6 and VPNs. These topics are extensively covered in this book.

These topics are very important because CCDE exam is a layer 3 infrastructure exam and because these technologies provide an Continue reading

Go Notes: Structs

A Structure or struct for short, is a type defined by the user that stores a collection of fields. go // create a struct type stuffAndThings struct // Instantiate a struct st := stuffAndThings // Use the dot (.) operator to access struct fields st.stuff st.things

Signals, Go & Immutable Infrastructure

From the days of old, setting fire to a large torch would signal to a neighbouring town something was going on. On the Great Wall in China, reports of signals reaching some 470 miles can be read on Wikipedia! Back to the future and modern day times, signals are transmitted and received as part of every application we touch. Signals underpin a system’s communications, irrelevant of what that system is. Software gives off many signals of a wide variety in normal operations and through signal correlation, we can yield useful events. Signals can also be used to achieve an outcome in a remote system as well as direct application API calls.

Being a fan of systems that have a natural synergy to them, I also look for ways to tie application functionality into natural system interactions.

For this post, I want to talk about the separation of concerns between an application’s functionality via it’s primary operational interface, likely an API of some sort, versus the application’s operational configuration, which allows it start on the correct TCP/IP port and consume the correct credential information.

Why not just get the application to refresh its configuration through the operational interface? The best way Continue reading

Asynchronous Zsh prompt with Git status

Zsh ships vcs_info, a function fetching information about the VCS state for the current directory and populating a variable that can be used in a shell prompt. It supports several VCS, including Git and SVN. Here is an example of configuration:

autoload -Uz vcs_info
zstyle ':vcs_info:*' enable git

() {
    local formats="${PRCH[branch]} %b%c%u"
    local actionformats="${formats}%{${fg[default]}%} ${PRCH[sep]} %{${fg[green]}%}%a"
    zstyle ':vcs_info:*:*' formats           $formats
    zstyle ':vcs_info:*:*' actionformats     $actionformats
    zstyle ':vcs_info:*:*' stagedstr         "%{${fg[green]}%}${PRCH[circle]}"
    zstyle ':vcs_info:*:*' unstagedstr       "%{${fg[yellow]}%}${PRCH[circle]}"
    zstyle ':vcs_info:*:*' check-for-changes true
}

add-zsh-hook precmd vcs_info

You can use ${vcs_info_msg_0_} in your prompt to display the current branch, the presence of staged and unstaged changes, as well as the ongoing action.1 Have a look at the documentation for more details.

Prompt with Git-related information, including branch name and presence of tracked and untracked changes
Example of prompt including information from the vcs_info function.

On large repositories, some information are expensive to fetch. While vcs_info queries Git, interactions with Zsh are stuck. A possible solution is to execute vcs_info asynchronously with zsh-async.

Continue reading

Operators Dish on Edge Computing Strategies at MEF 2019

Edge computing is a dispersion or distribution of the cloud and latency is the driving force of...

Read More »

© SDxCentral, LLC. Use of this feed is limited to personal, non-commercial use and is governed by SDxCentral's Terms of Use (https://www.sdxcentral.com/legal/terms-of-service/). Publishing this feed for public or commercial use and/or misrepresentation by a third party is prohibited.

TCP MD5

TCP_MD5 (RFC 2385) is something that doesn’t come up often. There’s a couple of reasons for that, good and bad.

I used it with tlssh, but back then (2010) it was not practical due to the limitations in the API on Linux and OpenBSD.

This is an updated post, written after I discovered TCP_MD5SIG_EXT.

What it is

In short it’s a TCP option that adds an MD5-based signature to every TCP packet. It signs the source and destination IP addresses, ports, and the payload. That way the data is both authenticated and integrity protected.

When an endpoint enables TCP MD5, all unsigned packets (including SYN packets) are silently dropped. For a signed connection it’s not even possible for an eavesdropper to reset the connection, since the RST would need to be signed.

Because it’s on a TCP level instead of part of the protocol on top of TCP, it’s the only thing that can protect a TCP connection against RST attacks.

It’s used by the BGP protocol to set a password on the connection, instead of sending the password in the handshake. If the password doesn’t match the TCP connection doesn’t even establish.

But outside of BGP it’s essentially Continue reading

1 … 1,157 1,158 1,159 1,160 1,161 … 3,912