STELLA: report from the SNAFU-catchers workshop on coping with complexity

STELLA: report from the SNAFU-catchers workshop on coping with complexity, Woods 2017, Coping with Complexity workshop

“Coping with complexity” is about as good a three-word summary of the systems and software challenges facing us over the next decade as I can imagine. Today’s choice is a report from a 2017 workshop convened with that title, and recommended to me by John Allspaw – thank you John!

Workshop context

The workshop brought together about 20 experts from a variety of different companies to share and analyse the details of operational incidents (and their postmortems) that had taken place at their respective organisations. Six themes emerged from those discussions that sit at the intersection of resilience engineering and IT. These are all very much concerned with the interactions between humans and complex software systems, along the lines we examined in Ironies of Automation and Ten challenges for making automation a ‘team player’ in joint human-agent activity.

There’s a great quote on the very front page of the report that is worth the price of admission on its own:

Woods’ Theorem: As the complexity of a system increases, the accuracy of any single agent’s own model of that system decreases rapidly.

Remember Continue reading

Windows Server vulnerability disclosed by NSA; don’t wait to patch

Microsoft’s monthly Patch Tuesday included a hefty haul of fixes: 49 total, and one of them is more than just critical. For enterprises running Windows Server 2016 and Server 2019, it's vital you implement the patch ASAP.The National Security Agency (NSA) disclosed the Windows vulnerability on Tuesday, the same day the fix was issued. That means the NSA found the flaw likely months ago but held off on public notification until Microsoft could come up with a fix. It would be irresponsible for the NSA, or anyone else, to announce a vulnerability and not give the software maker time to patch it.The vulnerability was spotted in "crypt32.dll," a Windows module that has been in both desktop and server versions since NT 4.0 more than 20 years ago. Microsoft describes the library as handling certificate and cryptographic messaging functions in the CryptoAPI.To read this article in full, please click here

Windows Server vulnerability disclosed by NSA; don’t wait to patch

Microsoft’s monthly Patch Tuesday included a hefty haul of fixes: 49 total, and one of them is more than just critical. For enterprises running Windows Server 2016 and Server 2019, it's vital you implement the patch ASAP.The National Security Agency (NSA) disclosed the Windows vulnerability on Tuesday, the same day the fix was issued. That means the NSA found the flaw likely months ago but held off on public notification until Microsoft could come up with a fix. It would be irresponsible for the NSA, or anyone else, to announce a vulnerability and not give the software maker time to patch it.The vulnerability was spotted in "crypt32.dll," a Windows module that has been in both desktop and server versions since NT 4.0 more than 20 years ago. Microsoft describes the library as handling certificate and cryptographic messaging functions in the CryptoAPI.To read this article in full, please click here

Windows Server vulnerability disclosed by NSA; Don’t wait to patch

Microsoft’s monthly Patch Tuesday included a hefty haul of fixes: 49 total, and one of them is more than just critical. For enterprises running Windows Server 2016 and Server 2019, it's vital you implement the patch ASAP.The National Security Agency (NSA) disclosed the Windows vulnerability on Tuesday, the same day the fix was issued. That means the NSA found the flaw likely months ago but held off on public notification until Microsoft could come up with a fix. It would be irresponsible for the NSA, or anyone else, to announce a vulnerability and not give the software maker time to patch it.The vulnerability was spotted in "crypt32.dll," a Windows module that has been in both desktop and server versions since NT 4.0 more than 20 years ago. Microsoft describes the library as handling certificate and cryptographic messaging functions in the CryptoAPI.To read this article in full, please click here

Windows Server vulnerability disclosed by NSA; Don’t wait to patch

Microsoft’s monthly Patch Tuesday included a hefty haul of fixes: 49 total, and one of them is more than just critical. For enterprises running Windows Server 2016 and Server 2019, it's vital you implement the patch ASAP.The National Security Agency (NSA) disclosed the Windows vulnerability on Tuesday, the same day the fix was issued. That means the NSA found the flaw likely months ago but held off on public notification until Microsoft could come up with a fix. It would be irresponsible for the NSA, or anyone else, to announce a vulnerability and not give the software maker time to patch it.The vulnerability was spotted in "crypt32.dll," a Windows module that has been in both desktop and server versions since NT 4.0 more than 20 years ago. Microsoft describes the library as handling certificate and cryptographic messaging functions in the CryptoAPI.To read this article in full, please click here

Loodse Steers Telco 5G Plans Toward Kubernetes

MacOS Catalina = Windows Vista

Remember the Windows version that was so security-focused that it broke everything, and needed a gazillion changes/updates/upgrades to get back to where you had a working computer? I think it was Vista, but maybe my memory is failing me. Anyway, Apple got its Vista moment with macOS Catalina.

I was stupid enough to upgrade just before New Year, and I’m still struggling with aftereffects and skeletons falling out of every cupboard I look at. I appreciate Apple trying to make their operating system ever more secure, but breaking stuff every time I upgrade it is borderline ridiculous.

Weekly Top Posts: 2020-01-19

1. Equinix CEO Talks Edge: Friend or Foe?
2. Money Moves: December 2019
3. LogicMonitor Gobbles Up AIOps Provider Unomaly
4. Google Absorbs AppSheet to Automate Code
5. Vodafone UK Activates 5G Multi-Operator RAN

QFX Upgrades – Check Host Version

I came across a situation where a software upgrade failed for some members in a Juniper QFX Virtual Chassis. There is a known issue with upgrades with a certain configuration + version combination, but I thought it didn’t apply to me. Turns out that the key was the host OS version, not the Junos VM version. Your host and guest versions can be out of sync with Juniper QFX 5K devices, and this can lead to confusing behavior, especially in a virtual chassis where host OS versions might vary.

Upgrade Failures - post-install error

When upgrading an old Juniper QFX5100, you might see these messages when running the upgrade:

1
2
Error: jinstall-vjunos fails post-install
Error: jinstall-vjunos-14.1X53-D34-domestic-signed fails post-install

In my case, I saw it for some nodes in a Virtual Chassis. Some worked, some failed. KB31923 says that this error is due to this configuration:

1
2
3
# show system internet-options
tcp-drop-synfin-set;
no-tcp-reset drop-tcp-with-syn-only;

Easy enough to change:

1
2
3
4
5
6
7
8
9
10
# delete system internet-options
{master:0}[edit]
root# show |compare
[edit system]
  internet-options {
      tcp-drop-synfin-set;
     no-tcp-reset drop-tcp-with-syn-only;
 
{master:0}[edit]
root# commit

Continue reading

QFX Upgrades – Check Host Version

I came across a situation where a software upgrade failed for some members in a Juniper QFX Virtual Chassis. There is a known issue with upgrades with a certain configuration + version combination, but I thought it didn’t apply to me. Turns out that the key was the host OS version, not the Junos VM version. Your host and guest versions can be out of sync with Juniper QFX 5K devices, and this can lead to confusing behavior, especially in a virtual chassis where host OS versions might vary.

Upgrade Failures - post-install error

When upgrading an old Juniper QFX5100, you might see these messages when running the upgrade:

1
2
Error: jinstall-vjunos fails post-install
Error: jinstall-vjunos-14.1X53-D34-domestic-signed fails post-install

In my case, I saw it for some nodes in a Virtual Chassis. Some worked, some failed. KB31923 says that this error is due to this configuration:

1
2
3
# show system internet-options
tcp-drop-synfin-set;
no-tcp-reset drop-tcp-with-syn-only;

Easy enough to change:

1
2
3
4
5
6
7
8
9
10
# delete system internet-options
{master:0}[edit]
root# show |compare
[edit system]
  internet-options {
      tcp-drop-synfin-set;
     no-tcp-reset drop-tcp-with-syn-only;
 
{master:0}[edit]
root# commit

Continue reading

Worth Reading: Seven Deadly Sins of Predicting the Future of AI

The next time the sales system engineer working for your beloved $vendor drops by with a glitzy unicorn-based slide deck full of AI/ML goodies, read this article to get a slightly better understanding of where we are... from the perspective of someone who has actual experience doing that stuff.

Akraino Edge Stack Gains NFV, Mixed Reality Blueprints

Daily Roundup: DigitalOcean Drowns Jobs

Heavy Networking 498: Creating A Single Source Of Truth For Network Automation

For network automation you need a single source of truth that’s programmatically accessible, reflects intended state, and enables others to stand up infrastructure correctly without you getting in the middle of every provisioning request. Tim Schreyack joins us today to discuss network automation approaches using Ansible and Python, and of course, a single source of truth.

The post Heavy Networking 498: Creating A Single Source Of Truth For Network Automation appeared first on Packet Pushers.

Pacific Report to the Dynamic Coalition of Small Island Developing States in the Internet Economy

2019 was an active year for Pacific involvement in the Internet economy. What we have demonstrated is that originating from small island developing states (SIDS) in the Pacific does not restrict one’s opportunity to become a leader within large international organizations like ICANN, which manages and allocates domain names and IP addresses globally.

I was very honored that my colleagues from the ICANN At-Large Advisory Committee (ALAC) elected me to be their Chair for 2019, and again for the upcoming year. It has enabled me to use my organizational management skills which I did by distance learning from Rarotonga through Massey University in New Zealand.

My Cook Islands colleague, Pua Hunter, was also elected at the recent ICANN meeting as regional co-chair for the Government Advisory Committee (GAC). She is already the chair of GAC’s Underserved Regions Committee. Such leadership roles have also been achieved by others from SIDS in other Internet-related organizations, which goes to show that being from small islands does not mean that we will go unnoticed if we are prepared to be active in our commitment to improving our regions.

The Pacific Islands Chapter of the Internet Society (PICISOC) received a boost at the elections last Continue reading

IBM Secures $1.1B Contract With Banco Sabadell

Verizon Expands 4G LTE Small Cell Navy Base Footprint

Capturing Logs in Docker Desktop

Docker Desktop runs a Virtual Machine to host Docker containers. Each component within the VM (including the Docker engine itself) runs as a separate isolated container. This extra layer of isolation introduces an interesting new problem: how do we capture all the logs so we can include them in Docker Desktop diagnostic reports? If we do nothing then the logs will be written separately into each individual container which obviously isn’t very useful!

The Docker Desktop VM boots from an ISO which is built using LinuxKit from a list of Docker images together with a list of capabilities and bind mounts. For a minimal example of a LinuxKit VM definition, see https://github.com/linuxkit/linuxkit/blob/master/examples/minimal.yml — more examples and documentation are available in the LinuxKit repository. The LinuxKit VM in Docker Desktop boots in two phases: in the first phase, the init process executes a series of one-shot “on-boot” actions sequentially using runc to isolate them in containers. These actions typically format disks, enable swap, configure sysctl settings and network interfaces. The second phase contains “services” which are started concurrently and run forever as containerd tasks.

The following diagram shows a simplified high-level view of the boot process:

By default Continue reading

Oracle Fixes 334 Bugs in Record-Breaking Critical Patch Update

DigitalOcean Slashes Jobs as Part of Restructure