Streaming and longer context lengths for LLMs on Workers AI
Workers AI is our serverless GPU-powered inference platform running on top of Cloudflare’s global network. It provides a growing catalog of off-the-shelf models that run seamlessly with Workers and enable developers to build powerful and scalable AI applications in minutes. We’ve already seen developers doing amazing things with Workers AI, and we can’t wait to see what they do as we continue to expand the platform. To that end, today we’re excited to announce some of our most-requested new features: streaming responses for all Large Language Models (LLMs) on Workers AI, larger context and sequence windows, and a full-precision Llama-2 model variant.
If you’ve used ChatGPT before, then you’re familiar with the benefits of response streaming, where responses flow in token by token. LLMs work internally by generating responses sequentially using a process of repeated inference — the full output of a LLM model is essentially a sequence of hundreds or thousands of individual prediction tasks. For this reason, while it only takes a few milliseconds to generate a single token, generating the full response takes longer, on the order of seconds. The good news is we can start displaying the response as soon as the first tokens are generated, Continue reading
Is Anyone Using netlab on Windows?
Tomas wants to start netlab with PowerShell, but it doesn’t work for him, and I don’t know anyone running netlab directly on Windows (I know people running it in a Ubuntu VM on Windows, but that’s a different story).
In theory, netlab (and Ansible) should work fine with Windows Subsystem for Linux. In practice, there’s often a gap between theory and practice – if you run netlab on Windows (probably using VirtualBox with Vagrant), I’d love to hear from you. Please leave a comment, email me, add a comment to Tomas’ GitHub issue, or fix the documentation and submit a PR. Thank you!
Is Anyone Using netlab on Windows?
Tomas wants to start netlab with PowerShell, but it doesn’t work for him, and I don’t know anyone running netlab directly on Windows (I know people running it in a Ubuntu VM on Windows, but that’s a different story).
In theory, netlab (and Ansible) should work fine with Windows Subsystem for Linux. In practice, there’s often a gap between theory and practice – if you run netlab on Windows (probably using VirtualBox with Vagrant), I’d love to hear from you. Please leave a comment, email me, add a comment to Tomas’ GitHub issue, or fix the documentation and submit a PR. Thank you!
NB455: Extreme Announces ZTNA Offering; Palo Alto Networks Spends Big On A Browser Startup
Extreme Networks is jumping into Zero Trust Network Access, Palo Alto Networks is reportedly spending more than half a billion dollars to acquire a corporate browser startup, and Forrester predicts as much as 20% of VMware's customers may jump ship after the Broadcom acquisition completes. We cover these stories and more in today's Network Break podcast.
The post NB455: Extreme Announces ZTNA Offering; Palo Alto Networks Spends Big On A Browser Startup appeared first on Packet Pushers.
NB455: Extreme Announces ZTNA Offering; Palo Alto Networks Spends Big On A Browser Startup
Extreme Networks is jumping into Zero Trust Network Access, Palo Alto Networks is reportedly spending more than half a billion dollars to acquire a corporate browser startup, and Forrester predicts as much as 20% of VMware’s customers may jump ship after the Broadcom acquisition completes. Arista touts a strong third quarter, while F5 forecasts a... Read more »Tech Bytes: Why AI Workloads Require Optimized Ethernet Fabrics (Sponsored)
Network engineers have a good grasp on how to build data center networks to support all kinds of apps, from traditional three-tier designs to applications built around containers and microservices. But what about building a network fabric to support AI? Today on the Tech Bytes podcast, sponsored by Nokia, we talk about the special requirements to build a data center fabric for AI use cases such as training and inference.
The post Tech Bytes: Why AI Workloads Require Optimized Ethernet Fabrics (Sponsored) appeared first on Packet Pushers.
Tech Bytes: Why AI Workloads Require Optimized Ethernet Fabrics (Sponsored)
Network engineers have a good grasp on how to build data center networks to support all kinds of apps, from traditional three-tier designs to applications built around containers and microservices. But what about building a network fabric to support AI? Today on the Tech Bytes podcast, sponsored by Nokia, we talk about the special requirements... Read more »SC23 SCinet traffic
Finally, check out the SC23 Dropped packet visibility demonstration to learn about one of newest developments in sFlow monitoring and see a live demonstration.
How Certified SASE Solutions Build Trust and Resilience
Implementing a certified SASE solution improves interoperability and enables an enterprise to take advantage of the latest SASE features and functionality.LAN Data Link Layer Addressing
Last week, we discussed Fibre Channel addressing. This time, we’ll focus on data link layer technologies used in multi-access networks: Ethernet, Token Ring, FDDI, and other local area- or Wi-Fi technologies.
The first local area networks (LANs) ran on a physical multi-access medium. The first one (original Ethernet) started as a thick coaxial cable1 that you had to drill into to connect a transceiver to the cable core.
Later versions of Ethernet used thinner cables with connectors that you put together to build whole network segments out of pieces of cable. However, even in that case, we were dealing with a single multi-access physical network – disconnecting a cable would bring down the whole network.
LAN Data Link Layer Addressing
Last week, we discussed Fibre Channel addressing. This time, we’ll focus on data link layer technologies used in multi-access networks: Ethernet, Token Ring, FDDI, and other local area- or Wi-Fi technologies.
The first local area networks (LANs) ran on a physical multi-access medium. The first one (original Ethernet) started as a thick coaxial cable1 that you had to drill into to connect a transceiver to the cable core.
Later versions of Ethernet used thinner cables with connectors that you put together to build whole network segments out of pieces of cable. However, even in that case, we were dealing with a single multi-access physical network – disconnecting a cable would bring down the whole network.