Helios: hyperscale indexing for the cloud & edge – part 1

Helios: hyperscale indexing for the cloud & edge, Potharaju et al., PVLDB’20

On the surface this is a paper about fast data ingestion from high-volume streams, with indexing to support efficient querying. As a production system within Microsoft capturing around a quadrillion events and indexing 16 trillion search keys per day it would be interesting in its own right, but there’s a lot more to it than that. Helios also serves as a reference architecture for how Microsoft envisions its next generation of distributed big-data processing systems being built. These two narratives of reference architecture and ingestion/indexing system are interwoven throughout the paper. I’m going to tackle the paper in two parts, focusing today on the reference architecture, and in the next post on the details of Helios itself. What follows is a discussion of where big data systems might be heading, heavily inspired by the remarks in this paper, but with several of my own thoughts mixed in. If there’s something you disagree with, blame me first!

Why do we need a new reference architecture?

Cloud-native systems represent by far the largest, most distributed, computing systems in our history. And the established cloud-native architectural principles behind them Continue reading

Why Biden: Principle over Party

There exist many #NeverTrump Republicans who agree that while Trump would best achieve their Party's policies, that he must nonetheless be opposed on Principle. The Principle at question isn't about character flaws, such as being a liar, a misogynist, or a racist. The Principle isn't about political policies, such as how to hand the coronavirus pandemic, or the policies Democrats want. Instead, the Principle is that he's a populist autocrat who is eroding our liberal institutions ("liberal" as in the classic sense).

Countries don't fail when there's a leftward shift in government policies. Many prosperous, peaceful European countries are to the left of Biden. What makes prosperous countries fail is when civic institutions break down, when a party or dear leader starts ruling by decree, such as in the European countries of Russia or Hungary.

Our system of government is like football. While the teams (parties) compete vigorously against each other, they largely respect the rules of the game, both written and unwritten traditions. They respect each other -- while doing their best to win (according to the rules), they nonetheless shake hands at the end of the match, and agree that their opponents are legitimate.

The rules of the Continue reading

ACI Fabric Access Policies Part 4: Leaf Interface Profile, Leaf Switch Policy Group, and Leaf Switch Profile,


Leaf Interface Profile

 

This section explains how to create an object Interface Profile whose basic purpose is to attach the set of physical interfaces into this object. Phase 6 in Figure 1-40 illustrates the APIC Management Information Model (MIM) from the Interface Profile perspective. We are adding an object L101__102_IPR under the class AccPortP (Leaf Interface Profile). The name of the object includes Leaf switch identifiers (Leaf-101 and Leaf-102) in which I am going to use this Interface Profile. This object has a Child object Eth1_1-5 (class InfraHPorts) that defines the internet block and which has a relationship with the object Port_Std_ESXi-Host_IPG. By doing this we state that ethernet interfaces 1/1-5 are LLDP enabled 10Gbps ports which can use VLAN Identifiers from 300-399. Note that in this phase we haven’t yet specified in which switches we are using this Interface Profile.

 The RN rules used with related objects:

 Objects created under the class InfraAccportP (Leaf Interface Profile):Prefix1-{name}, where the Prefix1 is “accportprof”. This gives us RN “accportprof-L101_L102_IPR”.

 Objects created under the class InfraHPortS (Access Port Selector): Prefix1-{name}-Prefix2-{type}, where the Prefix1 is “hports” and the Prefix2 is “typ”. This gives us RN “hports-Eth1_1-5_typ-range”.

Objects created under the class InfraPortBlk (Access Port Block): Prefix1-{name}, where the Prefix1 is “portblk” and where the name is Property (autogenerated). This gives us the RN “portblk-Block2”.



Figure 1-39: APIC MIM Reference: Interface Profile.

Continue reading

ACI Fabric Access Policies Part 3: AAEP, Interface Policy and Interface Policy Group

 

Attachable Access Entity Profile - AAEP


This section explains how to create an object Attachable Access Entity Profile (AAEP) that is used for attaching a Domain into Port Group. Phase 3 in Figure 1-20 illustrates the APIC Management Information Model (MIM) from the AAEP perspective. Class AttEntityP is a Child class for infra, and they both belong to packages Infra. I have already added the object attentp-AEP_PHY into the figure.The format of the RN for this object is Prefix1-{name}, where the Prefix1 is attentp. This gives us the RN attentp-PHY-AEP.



Figure 1-20: APIC MIM Reference: Attachment Access Entity Profile.

Continue reading

Worth Reading: The Shared Irresponsibility Model in the Cloud

A long while ago I wrote a blog post along the lines of “it’s ridiculous to allow developers to deploy directly to a public cloud while burdening them with all sorts of crazy barriers when deploying to an on-premises infrastructure,” effectively arguing for self-service approach to on-premises deployments.

Not surprisingly, the reality is grimmer than I expected (I’m appalled at how optimistic my predictions are even though I always come across as a die-hard grumpy pessimist), as explained in The Shared Irresponsibility Model in the Cloud by Dan Hubbard.

For more technical details, watch cloud-focused ipSpace.net webinars, in particular the Cloud Security one.

Worth Reading: The Shared Irresponsibility Model in the Cloud

A long while ago I wrote a blog post along the lines of “it’s ridiculous to allow developers to deploy directly to a public cloud while burdening them with all sorts of crazy barriers when deploying to an on-premises infrastructure,” effectively arguing for self-service approach to on-premises deployments.

Not surprisingly, the reality is grimmer than I expected (I’m appalled at how optimistic my predictions are even though I always come across as a die-hard grumpy pessimist), as explained in The Shared Irresponsibility Model in the Cloud by Dan Hubbard.

For more technical details, watch cloud-focused ipSpace.net webinars, in particular the Cloud Security one.

Optus Android MMS No Workie

I migrated away from an iphone to an Android phone recently and MMS was not working. I was unable to send or receive MMS and it was pretty annoying as I was not getting funny memes from all of my non-existant friends. It looks like this is a pretty common issue going back years and...

Progress on Image classification and Home-assistant and Dyson integration

Contunuing from the previous post, I was determined to learn some sort of image classification in my free time, I went with Tensorflow/keras as the language or ml language as they have a lot of tutorials around it.

Below youtube series from tensor-flow team will help you get started if you are interested in this

The other part that was a sort of problem for a long time is integration of Dyson Fan with any sort of automation platform like Alexa or python api, Dyson does have an app but apparently they have some issues in UK/Ireland integration, after some research I found that home-assistant supports integration and after a lot of documentation and trials, it looks like home-assistant has now support for it.

Here is the below file that you need to use in the configuration.yaml snippet. 

dyson:
username: yourdysonemailaccount
password: yourpassword
language: GB
devices:
- device_id: xxx-xx-xxxxxx
device_ip: a.b.c.d

This has now control for any sort of automation

-Rakesh

Technology Short Take 132

Welcome to Technology Short Take #132! My list of links and articles from around the web seems to be a bit heavy on security-related topics this time. Still, there’s a decent collection of networking, cloud computing, and virtualization articles as well as a smattering of other topics for you to peruse. I hope you find something useful!

Networking

  • I think a fair number of folks may not be aware that the Nginx ingress controller for Kubernetes—both the community version and the Nginx-maintained open source version—do suffer from timeouts and errors resulting from changes in the back-end application’s list of endpoints (think pods being added or removed). This performance testing post lays out all the details. In particular, see the section titled “Timeout and Error Results for the Dynamic Deployment.”
  • Ivan Pepelnjak attempts to answer the question, “How much do I need to know about Linux networking?”
  • Speaking of Linux networking…Marek Majkowski of Cloudflare digs deep into conntrack, used for stateful firewalling functionality.

Servers/Hardware

  • Normally I talk about server hardware and such here, but with so much moving to public cloud providers, let’s expand that focus a little bit: in this post, Jeramiah Dooley provides his perspective Continue reading

Weekend Reads 102320

One of the most common ways such access is monetized these days is through ransomware, which holds a victim’s data and/or computers hostage unless and until an extortion payment is made. But in most cases, there is a yawning gap of days, weeks or months between the initial intrusion and the deployment of ransomware within a victim organization.

There is a lot of discussion about the Expedited Policy Development Process (EPDP) Phase 2 report on evaluating a System for Standardized Access/Disclosure (SSAD) to non-public gTLD registration data after the decisions taken by the GNSO Council on September 24th.

Nvidia’s announcement a month ago of a plan to acquire Arm Holdings for $40 billion was followed last week by reports of another towering deal in semiconductors: Advanced Micro Device’s prospective purchase of Xilinx for about $30 billion.

For a long time, datacenter compute has been the very picture of stability – Intel-based servers running enterprise workloads in central facilities. The workloads are changing fast and the datacenter is dissolving, and this is all having a ripple effect throughout the infrastructure, from the servers and storage appliances down to the components, most notability the silicon that is powering the systems.

Booter services Continue reading

Best Networking ideas For Small Groups

Whether for small or large groups, networking is a challenging task. It takes a lot of effort to give the best impression of yourself while trying to socialize no matter what the purpose of socializing or networking is. This can become more challenging when you are living in a place where there are not many resources available for networking. Networking can be done for different purposes. So, if you are looking for networking ideas for small groups, you need to find out the purpose of networking for the group.

Check out a few networking ideas for small groups below.

A Mobile Event App

If you have an event coming up, you can create a mobile event app to organize your event and connect with the attendees of the event. This is a great way to stay in touch and coordinate with your small group of attendees who are going to look after various tasks of the event. These mobile apps are one of the easiest ways for staying in touch with the team. You even get the option to rate your favorite session and provide the admin with feedback once the event ends.

Thorough Social Media

If the purpose of Continue reading

Do You Do What You’ve Always Done?

When I was an intern at IBM twenty something years ago, my job was deploying new laptops to people. The job was easy enough. Transfer their few hundred megabytes of data to the new machine and ensure their email was all setup correctly. There was a checklist that needed to be followed in order to ensure that it was done correctly.

When I arrived for my internship, one of my friends was there finishing his. He was supposed to train me in how to do the job before he went back to school. He helped me through the first day of deploying laptops following the procedure. The next day he handed me a different sheet with some of the same information but in a different order. He said, “I realized we had too many reboots in the process and this way cuts about twenty minutes off the deployment time.” I’m all about saving time so I jumped at the chance.

Everything went smashingly for the next month or so. My friend was back at school and I used his modified procedure to be as productive as possible. One day, my mentor wanted to shadow my deployment day to see Continue reading

1 … 872 873 874 875 876 … 3,849