Workers AI: serverless GPU-powered inference on Cloudflare’s global network

If you're anywhere near the developer community, it's almost impossible to avoid the impact that AI’s recent advancements have had on the ecosystem. Whether you're using AI in your workflow to improve productivity, or you’re shipping AI based features to your users, it’s everywhere. The focus on AI improvements are extraordinary, and we’re super excited about the opportunities that lay ahead, but it's not enough.

Not too long ago, if you wanted to leverage the power of AI, you needed to know the ins and outs of machine learning, and be able to manage the infrastructure to power it.

As a developer platform with over one million active developers, we believe there is so much potential yet to be unlocked, so we’re changing the way AI is delivered to developers. Many of the current solutions, while powerful, are based on closed, proprietary models and don't address privacy needs that developers and users demand. Alternatively, the open source scene is exploding with powerful models, but they’re simply not accessible enough to every developer. Imagine being able to run a model, from your code, wherever it’s hosted, and never needing to find GPUs or deal with setting up the infrastructure to support Continue reading

Writing poems using LLama 2 on Workers AI

Matthew and Michelle, co-founders of Cloudflare, published their annual founders’ letter today. The letter ends with a poem written by an AI running using Workers AI on Cloudflare’s global network.

Here’s the code that wrote the poem. It uses Workers AI and the Meta Llama 2 model with 7B parameters and 8-bit integers. Just 14 lines of code running on the Cloudflare global network, and you’ve got your very own AI to chat with.

import { Ai } from "@cloudflare/ai";

export default {
    async fetch(request: Request, env: Env): Promise<Response> {
        const body = await request.json();
        const ai = new Ai(env.AI);
        const response = await ai.run("@cf/meta/llama-2-7b-chat-int8", body);
        return new Response(JSON.stringify(response));
    },
};

export interface Env {
    AI: any;
}

That was deployed on Workers AI and all I had to do was ask for poems. Here’s my terminal output (with just the domain name changed).

% curl -X POST https://example.com/ -d '{"prompt":"Write a poem \
that talks about the connectivity cloud"}' | jq -r .response

Cloud computing provides a connectivity that's unmatched,
A bridge that spans the globe with ease and grace.
It brings us closer, no matter where we are,
And makes the world a Continue reading

Announcing AI Gateway: making AI applications more observable, reliable, and scalable

Today, we’re excited to announce our beta of AI Gateway – the portal to making your AI applications more observable, reliable, and scalable.

AI Gateway sits between your application and the AI APIs that your application makes requests to (like OpenAI) – so that we can cache responses, limit and retry requests, and provide analytics to help you monitor and track usage. AI Gateway handles the things that nearly all AI applications need, saving you engineering time, so you can focus on what you're building.

Connecting your app to AI Gateway

It only takes one line of code for developers to get started with Cloudflare’s AI Gateway. All you need to do is replace the URL in your API calls with your unique AI Gateway endpoint. For example, with OpenAI you would define your baseURL as "https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/openai" instead of "https://api.openai.com/v1" – and that’s it. You can keep your tokens in your code environment, and we’ll log the request through AI Gateway before letting it pass through to the final API with your token.

// configuring AI gateway with the dedicated OpenAI endpoint

const openai = new OpenAI({
  apiKey: env.OPENAI_API_KEY,
  baseURL: "https://gateway.ai. Continue reading

Partnering with Hugging Face to make deploying AI easier and more affordable than ever 🤗

Today, we’re excited to announce that we are partnering with Hugging Face to make AI models more accessible and affordable than ever before to developers.

There are three things we look forward to making available to developers over the coming months:

1. We’re excited to bring serverless GPU models to Hugging Face — no more wrangling infrastructure or paying for unused capacity. Just pick your model, and go;
2. Bringing popular Hugging Face optimized models to Cloudflare’s model catalog;
3. Introduce Cloudflare integrations as a part of Hugging Face’s Inference solutions.

Hosting over 500,000 models and serving over one million model downloads a day, Hugging Face is the go-to place for developers to add AI to their applications.

Meanwhile, over the past six years at Cloudflare, our goal has been to make it as easy as possible for developers to bring their ideas and applications to life on our developer platform.

As AI has become a critical part of every application, this partnership has felt like a natural match to put tools in the hands of developers to make deploying AI easy and affordable.

“Hugging Face and Cloudflare both share a deep focus on making the latest AI innovations as accessible and affordable Continue reading

Vectorize: a vector database for shipping AI-powered applications to production, fast

Vectorize is our brand-new vector database offering, designed to let you build full-stack, AI-powered applications entirely on Cloudflare’s global network: and you can start building with it right away. Vectorize is in open beta, and is available to any developer using Cloudflare Workers.

You can use Vectorize with Workers AI to power semantic search, classification, recommendation and anomaly detection use-cases directly with Workers, improve the accuracy and context of answers from LLMs (Large Language Models), and/or bring-your-own embeddings from popular platforms, including OpenAI and Cohere.

Visit Vectorize’s developer documentation to get started, or read on if you want to better understand what vector databases do and how Vectorize is different.

Why do I need a vector database?

Machine learning models can’t remember anything: only what they were trained on.

Vector databases are designed to solve this, by capturing how an ML model represents data — including structured and unstructured text, images and audio — and storing it in a way that allows you to compare against future inputs. This allows us to leverage the power of existing machine-learning models and LLMs (Large Language Models) for content they haven’t been trained on: which, given the tremendous cost of training models, turns Continue reading

What AI companies are building with Cloudflare

What AI applications can you build with Cloudflare? Instead of us telling you we reached out to a small handful of the numerous AI companies using Cloudflare to learn a bit about what they’re building and how Cloudflare is helping them on their journey.

We heard common themes from these companies about the challenges they face in bringing new products to market in the ever-changing world of AI ranging from training and deploying models, the ethical and moral judgements of AI, gaining the trust of users, and the regulatory landscape.  One area that is not a challenge is trusting their AI application infrastructure to Cloudflare.

Azule.ai

Azule, based in Calgary, Canada, was founded to apply the power of AI to streamline and improve ecommerce customer service. It’s an exciting moment that, for the first time ever, we can now dynamically generate, deploy, and test code to meet specific user needs or integrations. This kind of flexibility is crucial to create a tool like Azule that is designed to meet this demand, offering a platform that can handle complex requirements and provide flexible integration options with other tools.

The AI space is evolving quickly and that applies to the Continue reading

Cloudflare’s 2023 Annual Founders’ Letter

Cloudflare is officially a teenager. We launched on September 27, 2010. Today we celebrate our thirteenth birthday. As is our tradition, we use the week of our birthday to launch products that we think of as our gift back to the Internet. More on some of the incredible announcements in a second, but we wanted to start by talking about something more fundamental: our identity.

Like many kids, it took us a while to fully understand who we are. We chafed at being put in boxes. People would describe Cloudflare as a security company, and we'd say, "That's not all we do." They'd say we were a network, and we'd object that we were so much more. Worst of all, they'd sometimes call us a "CDN," and we'd remind them that caching is a part of any sensibly designed system, but it shouldn't be a feature unto itself. Thank you very much.

And so, yesterday, the day before our thirteenth birthday, we announced to the world finally what we realized we are: a connectivity cloud.

The connectivity cloud

What does that mean? "Connectivity" means we measure ourselves by connecting people and things together. Our job isn't to be the Continue reading

The best place on Region: Earth for inference

Today, Cloudflare’s Workers platform is the place over a million developers come to build sophisticated full-stack applications that previously wouldn’t have been possible.

Of course, Workers didn’t start out that way. It started, on a day like today, as a Birthday Week announcement. It may not have had all the bells and whistles that exist today, but if you got to try Workers when it launched, it conjured this feeling: “this is different, and it’s going to change things”. All of a sudden, going from nothing to a fully scalable, global application took seconds, not hours, days, weeks or even months. It was the beginning of a different way to build applications.

If you’ve played with generative AI over the past few months, you may have had a similar feeling. Surveying a few friends and colleagues, our “aha” moments were all a bit different, but the overarching sentiment across the industry at this moment is unanimous — this is different, and it’s going to change things.

Today, we’re excited to make a series of announcements that we believe will make a similar impact as Workers did in the future of computing. Without burying the lede any further, here they Continue reading

BGP Labs: Use BFD to Speed Up Convergence

In the next BGP labs exercise, you can practice tweaking BGP timers and using BFD to speed up BGP convergence.

BGP Labs: Use BFD to Speed Up Convergence

In the next BGP labs exercise you can practice tweaking BGP timers and using BFD to speed up BGP convergence.

I would strongly recommend using netlab to run BGP labs, but if you insist you can use any system you like including physical hardware.

NFA v23.09 is here, featuring custom SSL/TLS certificates, MAC Address dictionaries, Mattermost notification channels support, and more.

The post NFA v23.09 is here, featuring custom SSL/TLS certificates, MAC Address dictionaries, Mattermost notification channels support, and more. appeared first on Noction.

Meta Platforms Is Determined To Make Ethernet Work For AI

We said it from the beginning: There is no way that Meta Platforms, the originator of the Open Compute Project, wanted to buy a complete supercomputer system from Nvidia in order to advance its AI research and move newer large language models and recommendation engines into production. …

The post Meta Platforms Is Determined To Make Ethernet Work For AI first appeared on The Next Platform.

Meta Platforms Is Determined To Make Ethernet Work For AI was written by Timothy Prickett Morgan at The Next Platform.

Oracle Cloud delivers Nvidia-powered generative AI service

Oracle last week announced limited availability of Oracle Cloud Infrastructure (OCI) Generative AI, which aims to enable business users to integrate AI into their line-of-business applications.OCI Generative AI is a managed service built on Oracle's cloud infrastructure in collaboration with Cohere, an AI platform for enterprise. The managed service and Cohere models will work in conjunction with AI Vector Search, a feature of Oracle Database 23c that provides retrieval augmented generation (RAG), an AI technique that combines pretrained LLMs and proprietary business data to deliver responses with higher accuracy. OCI Generative AI will also form the basis for generative AI capabilities embedded across Oracle’s suite of SaaS applications, including Oracle Fusion Cloud Applications Suite, Oracle NetSuite, and industry applications such as Oracle Cerner.To read this article in full, please click here

IBM brings AI to Z series mainframes, z/OS, Cloud Paks

New AI toolkits, machine learning (ML) frameworks and AI-based private cloud tools are on their way to IBM Z-series mainframe users, as the company looks to preserve its share of the fast-growing AI marketplace.The company announced today that its newest offerings, meant to help organizations get to work on the latest and greatest in AI frameworks, will be available for IBM Z, LinuxOne, z/OS and Cloud Pak architectures. The first new feature, AI Toolkit for IBM Z and LinuxONE, is designed to help enable mission-critical business applications on open-source frameworks like IBM’s Z Accelerated line with TensorFlow, SnapML and more. A Python AI Toolkit for the underlying z/OS also provides an open-source Python library for AI/ML workloads, designed to conform with the company’s Security and Privacy by Design guidelines.To read this article in full, please click here

IBM brings AI to Z series mainframes, z/OS, Cloud Paks

New AI toolkits, machine learning (ML) frameworks and AI-based private cloud tools are on their way to IBM Z-series mainframe users, as the company looks to preserve its share of the fast-growing AI marketplace.The company announced today that its newest offerings, meant to help organizations get to work on the latest and greatest in AI frameworks, will be available for IBM Z, LinuxOne, z/OS and Cloud Pak architectures. The first new feature, AI Toolkit for IBM Z and LinuxONE, is designed to help enable mission-critical business applications on open-source frameworks like IBM’s Z Accelerated line with TensorFlow, SnapML and more. A Python AI Toolkit for the underlying z/OS also provides an open-source Python library for AI/ML workloads, designed to conform with the company’s Security and Privacy by Design guidelines.To read this article in full, please click here

BrandPost: What do music and SSE have in common?

By: Nav Chander, Head of Service Provider SD-WAN/SASE Product Marketing.This is part 3 of a 3-part blog series on SD-WAN, Secure Service Edge (SSE), and multi-cloud networking (MCN), where we will highlight how these 3 technology areas are analogous to 3 different musical instruments that can be played separately but combined, are better together. The first two blogs focused on MCN and SD-WAN, and this blog focuses on SSE technology. Feel free to check out Part 1 and Part 2.To read this article in full, please click here

Julia Still Not Grown Up Enough to Ride Exascale Train

We’ve been watching Julia, an HPC-oriented programming language designed for technical and scientific computing for a number of years to see it can make inroads into supercomputing. …

The post Julia Still Not Grown Up Enough to Ride Exascale Train first appeared on The Next Platform.

Julia Still Not Grown Up Enough to Ride Exascale Train was written by Nicole Hemsoth Prickett at The Next Platform.

Tailscale, Mullvad partner to deliver secure, private web browsing

Security networking startup Tailscale has partnered with Mullvad to make Mullvad’s global network of VPN services available to Tailscale customers looking to maintain user privacy while browsing the internet.Tailscale provides software that creates a peer-to-peer mesh VPN, which allows for customers to create direct connections between devices. Tailscale enables encrypted point-to-point connections using the opensource WireGuard protocol, which means only devices on a private network can communicate with each other. WireGuard is an open-source project that is designed for quick reconnections, which improves reliability. WireGuard is also supported by consumer-grade VPN provider Mullvad.To read this article in full, please click here

Tailscale, Mullvad partner to deliver secure, private web browsing

Security networking startup Tailscale has partnered with Mullvad to make Mullvad’s global network of VPN services available to Tailscale customers looking to maintain user privacy while browsing the internet.Tailscale provides software that creates a peer-to-peer mesh VPN, which allows for customers to create direct connections between devices. Tailscale enables encrypted point-to-point connections using the opensource WireGuard protocol, which means only devices on a private network can communicate with each other. WireGuard is an open-source project that is designed for quick reconnections, which improves reliability. WireGuard is also supported by consumer-grade VPN provider Mullvad.To read this article in full, please click here

IBM service uses DNS to deliver multicloud connectivity

IBM is rolling out a new DNS-based service that will let customers securely control connectivity between distributed multicloud environments.NS1 Connect is one of the first fruits from IBM’s acquisition of DNS specialist NS1 earlier this year. The service is aimed at helping organizations set up the best connection between clouds and end users to deliver applications optimized for performance, cost, security and availability.Core to the NS1 Connect package, which will be available Oct. 17, is traffic-steering technology that intelligently distributes DNS traffic across the network. DNS is often described as the Internet’s phone book, working in the background to match the names of web sites that users type into a search box with the corresponding IP address.To read this article in full, please click here