Who Pays The Price of Redundancy?
No doubt by now you’ve seen the big fire that took out a portion of the OVHcloud data center earlier this week. These kinds of things are difficult to deal with on a good day. This is why data centers have reductant power feeds, fire suppression systems, and the ability to get back up to full capacity. Modern data centers are getting very good at ensuring they can stay up through most events that could impact an on-premises private data center.
One of the issues I saw that was ancillary to the OVHcloud outage was the small group of people that were frustrated that their systems went down when the fire knocked out the racks where their instances lived. More than a couple of comments mentioned that clouds should not go down like this or asked about credit for time spent being offline or some form of complaints about unavailability. By and large, most of those complaining were running non-critical systems or were using the cheapest possible instances for their hosts.
Aside from the myopia that “cloud shouldn’t go down”, how do we deal with this idea that cloud redundancy doesn’t always translate to single instance availability? I think we Continue reading
Tom’s Virtual Corner at Cisco Live Global 2021 – Anniversary
We made it through the year that was March 2020. Here were are on the other side trying to find out whatever this normal is supposed to look like. We’re not out of the woods yet but we do know that things aren’t going to be back to the way they were any time soon. That includes the events that we enjoyed traveling to and hanging out at.
Cisco Live has made the decision to go virtual again this year. One can’t blame them to be honest. Travel uncertainty and the potential liability of having a huge event just didn’t make sense. If you thought the old Conference Crud was bad you really don’t want this new-and-improved version! Cisco has also decided that one global event makes more sense than several events scattered across the calendar. That means that Cisco Live Europe and Cisco Live US are now global and happening at the end of March instead of January or June.
With the announcement that everything will be virtual again this year it also means that the social aspect of the event is going to be virtual as well. As much as we would have liked to hang out at Continue reading
Building Snowflakes On Purpose
We all know that building snowflake networks is bad, right? If it’s not a repeatable process it’s going to end up being a problem down the road. If we can’t refer back to documentation to shows why we did something we’re going to end up causing issues and reducing reliability. But what happens when a snowflake process is required to fix a bigger problem? It’s a fun story that highlights where process can break down sometimes.
Reloaded
I’ve mentioned before that I spent about six months doing telephone tech support for Gateway computers. This was back in 2003 so Windows XP was the hottest operating system out there. The nature of support means that you’re going to be spending more time working on older things. In my case this was Windows 95 and 98. Windows 98 was a pain but it was easy to work on.
One of the most common processes we had for Windows 98 was a system reload. It was the last line of defense to fix massive issues or remove viruses. It was something that was second nature to any of the technicians on the help desk:
- Boot from the Gateway tools CD and use GWSCAN Continue reading
Tech Field Day Changed My Life
It’s amazing to me that it’s been ten years since I attended by first Tech Field Day event. I remember being excited to be invited to Tech Field Day 5 and then having to rush out of town a day early to beat a blizzard to be able to attend. Given that we just went through another blizzard here I thought the timing was appropriate.
How did attending an industry event change my life? How could something with only a dozen people over a couple of days change the way I looked at my career? I know I’ve mentioned parts of this to people in the past but I feel like it’s important to talk about how each piece of the puzzle built on the rest to get me to where I am today.
Voices Carry
The first thing Tech Field Day did to change my life was to show me that I mattered. I grew up in a very small town and spent most of my formative school years being bored. The Internet didn’t exist in a usable form for me. I devoured information wherever I could find it. And I languished as I realized that I needed more Continue reading
Solutions In Search of a Problem
During a few recent chats with my friends in the industry, I’ve heard a common refrain coming up about technologies or products being offered for sale. Typically these are advanced ideas given form that are then positioned as products for sale in the market. Overwhelmingly the feedback comes down to one phrase:
This is a solution in search of a problem.
We’ve probably said this a number of times about a protocol or a piece of hardware. Something that seems to be built to solve a problem we don’t have and couldn’t conceive of. But why does this seem to happen? And what can we do to fix this kind of mentality?
Forward Looking Failures
If I told you today that I was creating software that would revolutionize the way your autonomous car delivers music to the occupants on their VR headsets you’d probably think I was crazy, right? Every one of the technologies I mentioned in the statement is a future thing that we expect may be big down the road. We love the idea of autonomous vehicles and VR headsets and such.
Now, let’s change the statement. I’m working on a new algorithm for HD-DVD players to produce Continue reading
Friction Finders
Do you have a door that sticks in your house? If it’s made out of wood the odds are good that you do. The kind that doesn’t shut properly or sticks out just a touch too far and doesn’t glide open like it used to. I’ve dealt with these kinds of things for years and Youtube is full of useful tricks to fix them. But all those videos start with the same tip: you have to find the place where the door is rubbing before you can fix it.
Enterprise IT is no different. We have to find the source of friction before we can hope to repair it. Whether it’s friction between people and hardware, users and software, or teams going at each other we have to know what’s causing the commotion before we can repair it. Just like with the sticking door, adding more force without understand the friction points isn’t a long-term solution.
Sticky Wickets
Friction comes from a variety of sources. People don’t understand how to use a device or a program. Perhaps it’s a struggle to understand who is supposed to be in charge of a change control or a provisioning process. It could even Continue reading
The Double-Edged Grindstone
Are you doing okay out there? I hope that you’re well and not running yourself thin with all the craziness still going on. Sometimes it seems like we can’t catch a break and that work and everything keep us going all the time. In fact, that specific feeling and the resulting drive around it is what I wanted to talk about today.
People have drive. We want to be better. We want to learn and grow and change. Whether it’s getting a faster time running a 5K or learning new skills to help our career along. Humans can do amazing things given the right motivation and resource availability. I know because I taught myself a semester of macroeconomics in a Waffle House the night before the final exam. Sure, I was groggy and crashed for a 10-hour nap after the final but I did pass!
It’s that kind of ability to push ourselves past our limits that both defines us and threatens to destroy us. I’m a huge fan of reading and fiction. Growing up I latched on to the Battletech novels, especially those written by Michael A. Stackpole. In his book Lost Destiny there is a great discussion about Continue reading
Planning For The Worst Case You Can’t Think Of
Remember that Slack outage earlier this month? The one that happened when we all got back from vacation and tried to jump on to share cat memes and emojis? We all chalked it up to gremlins and went on going through our pile of email until it came back up. The post-mortem came out yesterday and there were two things that were interesting to me. Both of them have implications on reliability planning and how we handle the worst-case scenarios we come up with.
It’s Out of Our Hands
The first thing that came up in the report was that the specific cause for the outage came from an AWS Transit Gateway not being able to scale fast enough to handle the demand spike that came when we all went back to work on the morning of January 4th. What, the cloud can’t scale?
The cloud is practically limitless when it comes to resources. We can create instances with massive CPU resources or storage allocations or even networking pipelines. However, we can’t create them instantly. No matter how much we need it takes time to do the basic provisioning to get it up and running. It’s the old story of Continue reading
Managing Leaders, Or Why Pat Gelsinger Is Awesome
In case you missed it, Intel CEO Bob Swan is stepping down from his role effective February 15 and will be replaced by current VMware CEO Pat Gelsinger. Gelsinger was the former CTO at Intel for a number of years before leaving to run EMC and VMware. His return is a bright spot in an otherwise dismal past few months for the chip giant.
Why is Gelsinger’s return such a cause for celebration? The analysts that have been interviewed say that Intel has been in need of a technical leader for a while now. Swan came from the office of the CFO to run Intel on an interim basis after the resignation of Brian Krzanich. The past year has been a rough one for Intel, with delays in their new smaller chip manufacturing process and competition heating up from long-time rival AMD but also from new threats like ARM being potentially sold to NVIDIA. It’s a challenging course for any company captain to sail. However, I think one key thing makes is nigh impossible for Swan.
Management Mentality
Swan is a manager. That’s not meant as a slight inasmuch as an accurate label. Managers are people that have things and Continue reading
Building Backdoors and Fixing Malfeasance
You might have seen the recent news this week that there is an exploitable backdoor in Zyxel hardware that has been discovered and is being exploited. The backdoor admin account with the clever name ‘zyfwp’ is not something that has been present in the devices forever. The account was put in during firmware version 4.60, which was released in Q4 2020.
Zyxel is rushing to patch the devices and remove the backdoor account. Users are being advised to disable remote administration until the accounts can be deactivated and proven to be removed. However, the bigger question in my mind relates to the addition of the user account in the first place. Why would you knowingly install a backdoor?
Hello, Joshua
Backdoors are nothing new in the computer world. I’d argue the most famous backdoor account in the history of computer hacking belongs to Joshua, the dormant login for the War Operations Programmed Response (WOPR) computer system in the 1983 movie Wargames. Joshua was an old login for the creator to access the system outside of the military chain of command. When the developer was removed from the project the account was forgotten about until a kid discovered it and Continue reading
Winning in 2021
I’d jump in here and say something about 2020 being a crazy year but we all know it’s nothing we haven’t heard before. I’d also say that we’re going to look back at my big plans for the year however we also know that those got scrapped right after the end of February. I like looking back at a couple of things and then looking forward to what the next year will accomplish. Why? Because retrospectives are boring and putting your planning out there for the world to see is a much more interesting use of your time. The journey you’re taking changes greatly when you change your thinking about the destination.
2020 Good or Bad
2020 wasn’t all bad. I finally justified getting a new office chair! All kidding aside, 2020 was a year that challenged everyone greatly when it came to mental health, professional output, and even personal capability. My biggest focus for 2020 was to start putting blog posts out earlier in the week and focus on continuous improvement. I’d say the first was another miss due to the hectic workload, as a lot of my posts still came out on Fridays.
The second point was a Continue reading
Making Time For Yourself
I was a recent pop-in guest on the Network Collective Holiday Show with my friends Jordan Martin and Tony Efantis. One of the questions they had been asking their guests was about the big lessons we’ve learned this year. As I thought back on the roller coaster ride that was 2020, I realized that one of the biggest lessons that I’ve learned is that I need to make time for the important things for myself.
Mark It Down
I know it sounds like a given, but we all need to make time for ourselves. I realized that when my usual schedule of running myself in overdrive and jumping from one event or travel opportunity to the next evaporated back in March. I found myself sitting at home and working toward some uncertain future. I never thought that there were going to be huge problems but I also didn’t know how things would end up turning out.
As the days grew into weeks and eventually into months, I quickly figured out that the normal I once knew was going to stay gone for quite a while. In place of that was a situation that I needed to adjust to. And that Continue reading
Setting Boundaries Before You’re Swamped
We’re at the tail end of 2020 and things are hopeful for 2021. People are looking at the way IT has pulled together to enable working from anywhere and moving resources to the cloud and enabling users to get their jobs done. It’s a testament to the resilience of a group of sanitation workers behind the scenes whose job it is to clean up after management and sales and do the jobs no one else wants to do.
The cynic in me is worried about what the future is going to hold now that we’ve managed to transform the way we work. I couldn’t quite put my finger on it until I was checking out this Reddit thread from last week. The top rant had an interesting perspective on the way that 2021 is going to go for workers and I couldn’t agree more. My dread has a name, and it’s Overwork.
Harder, Not Smarter
If anything, 2020 proved that we can do amazing things with the right motivation. The superhero mentality of IT paid off handsomely as we stood up remote access servers and found ways to get access to resources for people that couldn’t come into the office Continue reading
How Long Should You Practice
A reporter once asked boxing legend Muhammad Ali how many sit-ups he did each day. I’m sure the reporter wasn’t expecting Ali’s answer. Ali replied with:
I don’t know. I don’t start counting them until it hurts. Those are the only ones that count. That’s what makes you a champion.”
Ali knew that counting things is just a numbers game. Five hundred poor sit-ups don’t count as much a fifty done the right way. With any practice that you do the only things that count are the things that teach your something or that push you to be better.
Don’t Practice Until It’s Right
People used to ask me how long I would spend at night studying for the CCIE lab. I told them I usually spent between five and seven hours depending on what I was studying. Sometimes those people would say things like “I’m not talking about setup time. I’m talking about actual lab work.” I always countered by making them explain why the setup isn’t part of the “real” work. That’s usually when they went quiet.
It’s far too easy to fall into the trap of overlooking things that you think are unimportant. A task Continue reading
Seeking Knowledge and Willful Ignorance
I had a great time recording a fun episode of Seeking Truth in Networking, an awesome podcast with my friends Derick Winkworth and Brandon Heller. We talked a lot about a variety of different topics, but the one I want to spend a few more minutes on here came in the first five minutes. Brandon asked me what question I liked to be asked and I mentioned that love to be asked about learning. My explanation included the following line:
I feel like the gap between people that don’t understand something and the willfully ignorant is that ability to take a step out and say “I don’t know the answer to this but I’m going to find out.”
I’ve always said that true learners are the ones that don’t accept the unknown. They want to find the answer. They want to be able to understand something as completely as they can. Those that I consider to be willfully ignorant choose not to do that.
Note that there is a difference between incidentally ignorant and willfully ignorant. People who are incidentally ignorant are unaware they don’t know something. They haven’t had the opportunity to learn or change their thought Continue reading
What’s Your Work From Home DR Plan?
It’s almost December and the signs are pointing to a continuation of the current state of working from home for a lot of people out there. Whether it’s a surge in cases that is causing businesses to close again or a change in the way your company looks at offices and remote work, you’re likely going to ring in the new year at your home keyboard in your pajamas with a cup of something steaming next to your desk.
We have all spent a lot of time and money investing in better conditions for ourselves at home. Perhaps it was a fancy new mesh chair or a more ergonomic keyboard. It could have been a bigger monitor with a resolution increase or a better webcam for the dozen or so Zoom meetings that have replaced the water cooler. There may even be more equipment in store, such as a better home wireless setup or even a corporate SD-WAN solution to help with network latency. However, have you considered what might happen if it all goes wrong and you need to be online?
In and Outage
Outages happen more often than we realize. That’s never been more evident than the situation Continue reading
A Different Viewpoint of Lock-In
First things first: Go watch this great video on lock-in from Ethan Banks (@ECBanks). We’ll reference it.
Welcome back. Still carrying that pitchfork around screaming about how you want to avoid vendor lock-in? Ready to build the most perfect automation system in history that does multi-cloud, multi-vendor, multi-protocol networking in a seamless manner with full documentation? Nice. How hard was is to build that unicorn farm?
I get it. No one wants to be beholden to a specific vendor. No one likes being forced into buying things. Everyone hates the life of the engineer forced to work on something they don’t like or had to use because someone needed a new boat. Or do they?
Ford and Chevys and Dodge, Oh My!
What kind of car do you drive? Odds are good you’re either ready to get a new one or you’re proud of what you’re driving. I find that the more flashy a car is the more likely people are to talk about how amazing it is. And when there are two dominant manufacturers in a market for cars, you tend to see people dividing into camps to sing the praises of their favorite brands. Ford people love their Continue reading
Looking For a Mentor? Don’t Forget This Important Step!
With the insanity of the pandemic and the knowledge drain that we’re seeing across IT in general, there’s never been a more important time than right now to help out those that are getting started on this rise. The calls for mentors across the community is heartwarming. I’ve been excited personally to see many recognizable names and faces in the Security, Networking, and Wireless communities reaching out to let people know they are available to mentor others or connect them with potential mentors. It’s a way to give back and provide servant leadership to those that need it.
If you’re someone that’s reading this blog right now and looking for a mentor you’re in luck. There are dozens of people out there that are willing to help you out. The kindness of the community is without bounds and there are those that know what it was like to wander through the wilderness for a while before getting on the right track. They are the ones that will be of the most help to you. However, before you slide into someone’s DMs looking for help, you need to keep a few things in mind.
Make Me One With Everything
The single Continue reading
Securing Your Work From Home
Wanna make your security team’s blood run cold? Remind them that all that time and effort they put in to securing the enterprise from attackers and data exfiltration is currently sitting unused while we all work from home. You might have even heard them screaming at the sky just now.
Enterprise security isn’t easy, nor should it be. We constantly have to be on the offensive to find new attack vectors and hunt down threats and exploits. We have spent years and careers building defense-in-depth to an artform not unlike making buttery croissants. It’s all great when that apparatus is protecting our enterprise data center and cloud presence like a Scottish castle repelling invaders. Right now we’re in the wilderness with nothing but a tired sentry to protect us from the marauders.
During Security Field Day 4, I led a discussion panel with the delegates about the challenges of working from home securely. Here’s a link to our discussion that I wanted to spend some time elaborating on:
Home Is Where the Exploits Are
BYOD was a huge watershed moment for the enterprise because we realized for the first time that we had to learn to secure other people’s Continue reading
Learning To Listen For Learning
Can you hear me? Are you listening to me? Those two statements are used frequently to see if someone is paying attention to what you’re saying. Their connotation is very different though. One asks a question about whether you can tell if there are words coming out of someone’s mouth. Is the language something you can process? The other question is all about understanding.
Taking Turns Speaking
“Seek first to understand,then to be understood.” – Stephen Covey
Listening is hard. Like super hard. How often do you find yourself on a conference call with your mind wandering to other things you need to take care of? How many times have we seen someone shopping online for shoes or camping gear instead of taking notes on the call they should be paying attention to? They answer is more often than we should.
Attention spans are hard for everyone, whether you’re affected by attention disorders or have normal brain chemistry. Our minds hate being bored. They’re always looking for a way to escape to something more exciting and stimulating. You know you can feel it when there’s a topic that seriously interests you and pulls you in versus the same old Continue reading