CNCF Releases

OpenTelemetry is returning to FOSDEM!

After this year’s resounding success of OTel Unplugged at FOSDEM 2026</

Batching in the OpenTelemetry Collector is a process that ensures requests are the right size for the exporter, large enough to improve compression and amortize round-trip latency, but not so large as to exceed limits. Batching combines a number of concerns that are critical to performance, reliability, and cost in running Collectors.

Since OpenTelemetry began in 2019, its Collector has included a batch processor component, responsible for combining and sp

An empty dashboard can mean that nothing happened. It can also mean that the system stopped reporting what happened. On a dashboard, they look the same. One needs no action. The other means you are missing the information you need to act.

Take a workflow where events enter a service, pass through a processing stage, and reach storage. An operator then uses the stored result. Every service can remain available while one data transfer silently drops events. Error rates stay low, yet the

In August 2016, the first public commit appeared in the kubevirt/kubevirt repo. To celebrate this 10 year milestone for the KubeVirt project, we interviewed four of the project maintainers who were there in those early days.

Q1. Tell me about how the project started.

Roman Mohr: Itamar, our Director back then at Red Hat, had the vision. So he put us on the project and we started hacking. I vividly remember the discussions about whether we should use Python or Golang, and you will find traces of Python in the very first commits :)

Also, while quite familiar with containers, I knew nothing about k8s. One of my first efforts was fixing the Vagrant setup (yes, Vagrant!) turnup flow in the Kubernetes repo, just to finally see k8s for the first time in reality.

Fabian Deutsch: Back then, a few months past the public release of Kubernetes by Google, a group of Red Hatters from Red Hat Virtualization got together in order to discuss what this actually means for virtualization: What are the commonalities, what makes them different.

This led to two paths that we could take: Adding VM support to Kubernetes or running VMs on Kubernetes.

In order to make a call, we kicked off an internal hackathon to try out both options, leading to an internal (temporary) fork of Kubernetes with VM support. This was way before CRI-O — which made container runtimes pluggable — and the other approach was one building on Third Party Resources, an early extension to Kube that was eventually replaced by CRDs.

After a few iterations we settled on CRDs, also because this would allow us to write the whole application (that KubeVirt is) with container principles. Roman was one of those who really kickstarted this whole effort.

Stu Gott: By the time I got on board, the core codebase basically already existed. The architecture was a bit different back in the inception of the project. I recall there was some sort of experiment using Cockpit early on.

We always used containers from the very beginning, but in KubeVirt’s first iteration, we used the host system’s libvirt daemon. This came with its own challenges that we need not dig into too deeply — this is a celebration, not a doctoral thesis. The premise that we were running VMs inside containers was still true, by the way; we originally had a shim where we replaced the qemu command with a shell script that fired off a container to run the VM inside. In modern KubeVirt, of course, we’ve reversed that, and now run a libvirt daemon inside the virt-launcher container, which handles the VM running inside that container.

Vladik Romanovsky: I wasn’t there for the first days. I joined in 2017, when there was already a skeleton: a VM as a Kubernetes object, QEMU running outside of the pod.

What I remember as the start of the project is the next sentence. OpenStack and oVirt/RHV could do networking, storage, scheduling, and live migration because they owned the node. We didn’t. A VM in KubeVirt had to be a pod. That meant using the scheduler, CNI, and volumes Kubernetes already had — and giving up virt features when they required us to stop being a Kubernetes citizen.

There was competition (virtlet?) early on that used Kubernetes objects and then did the real work off to the side. We decided the pod is the VM: VMM inside the pod, same decoupling as any other workload. That is the decision that aged well. CSI, CNI, later ImageVolume, DRA, and the rest could attach to VMs the same way they attach to pods (almost :) ).

Q2. What was your “Wow, this idea might have some real legs” moment?

Fabian: Well, I’m biased, but very early on — like a few months in — I think we saw that this can work out. We managed to achieve live migration, we could update seamlessly — all with bugs, and it took a few years to make this rock solid, but the overall concept seemed to work out.

Then I was amazed to hear what early adopters we had, and how fast they were able to adopt KubeVirt and make it useful for them. Some of them have not disclosed their adoption until this day, but that was like my “holy cow, that’s cool” moment.

By today, by the way, I think that the thing that is holding KubeVirt most back is the whole situation around it: managing bare metal. All the features (disaster recovery, backup — incremental backup! — SDN, deployment of kube, …). All of this is not handled by KubeVirt, intentionally, but took a long while to get rolling, and many of those are these days the differentiating factors between the different Kubernetes and KubeVirt distributions. Every vendor (and adopter) is making their own decisions on this.

Stu: When KubeVirt first started, the concept of operators in Kubernetes wasn’t really an established pattern that the project had adopted yet. Between putting libvirt into the container and the introduction of virt-operator into the code, we suddenly found ourselves with a project where an admin could basically install the operator and literally everything else was just handled. No host-level changes needed to be made. Install virt-operator, get VMs. Profit.

Vladik: Networking.

CNI could already put an interface in the pod. That wasn’t enough. The guest still had no identity on that network, and we refused to solve it by picking a CNI vendor. We wanted the same deal a normal application gets: whatever CNI the cluster runs, the workload still works.

We wrote a small DHCP server inside virt-launcher, took the address and routes off the pod interface, and handed that identity to the guest. Petr Horáček was central to that first DHCP and to making the idea hold. When the VM came up on the pod network — same CNI, no private networking stack — the architecture stopped being a demo. A VM really was a pod that happened to run QEMU.

Q3. When you started, where did you think KubeVirt was going? Did you have an inkling to where it is 10 years later?

Roman: At that point, I lacked vision and was simply happy to learn a lot of new stuff about k8s and fit pieces together.

Vladik: I didn’t, and that was never the question for me.

I was sold on running VMs alongside pods on the same platform — including as pieces of one application — and I wanted to make that real. It wasn’t easy. What I hoped for was narrower than a ten-year forecast: hit the technical objectives, serve cloud-native apps with modern use cases, and still be a stable platform for traditional datacenter workloads. I also wanted the project not to be a hostile place, and I tried to live up to that myself.

I did not forecast the later scale, other vendors, or people leaving classic hypervisors. I thought VMs belonged in Kubernetes as first-class citizens. That was the whole bet.

Stu: Implausibly, I kind of expected us to be sort of where we are today. No, really. When I took my first software engineering job, my manager at the time asserted that it takes 10 years for a project to become mature. When KubeVirt first started, we were in more of a move-fast-and-break-things mode. About a year in, we announced the project to the public. From there we gradually moved from a skunkworks project to something with more rigor and a maintenance mindset.

The one thing I did misestimate is how many features would be in active development. With the VEP process having exploded onto the scene, it really highlights how many different initiatives are converging on the project all at once.

Fabian: I’m not sure if I ever thought about where KubeVirt would be in 10 years. Come on, 10 years, that’s a long time. And to some extent I’m surprised that we are not further along; on the other hand, virtualization is also a challenging space to be in. Virtualization is still — and is expected to continue — running some of the most critical workloads in the world, be it in finance, governments, or factories, often for operation and efficiency reasons, and gaining the trust to run those is taking a while. In that sense, KubeVirt is growing at a fair pace.

Q4. What was a difficult technical decision the project had to make early on, and would you make the same decision today with hindsight?

Roman: Two aspects are very present in my mind. First: should we build KubeVirt around the idea of a container runtime implementation, or should we run VMs inside a Pod? I was initially favouring the CRI approach, but Fabian convinced me to take the other route: running VMs inside containers. This was a key decision that set the project up for success.

Second: We had a workgroup to directly use QEMU and not use libvirt in between. I was a strong proponent of that approach. While I appreciate all the support we received from the libvirt team over the years, I still think using libvirt was the architecturally wrong call, causing us more problems than benefits. I would have insisted more on that one.

Fabian: To me there are two as well. First: VMs inside of Kubernetes or on top of Kubernetes. It’s difficult to say if the on-top-of approach is the superior one, but to me it is exactly giving us what we wanted to achieve back then: develop KubeVirt just like any cloud native application would be developed, and benefit from the aspects that cloud native applications have. Note also that Hyper.sh, for example, took the different approach and provided a VM runtime for a Pod API, and some features were much more difficult to achieve that way than with a native VM API.

Second: KubeVirt is a VM API and runtime. KubeVirt really only solves this one problem — specifying and running VMs. Almost everything else was expected to be provided by Kubernetes and the CNCF ecosystem, and this took looong to grow. Because it took so long, we also made some compromises (i.e., on incremental backup) in order to expedite some features.

Vladik: Yes: stay Kubernetes-native even when that meant giving features up. The alternative was a hidden virt control plane that only looked like Kubernetes. Faster in 2017. Wrong now.

We still paid a tax. Traditional virt expects to hotplug a NIC, a disk, memory, CPU on a running guest. A Kubernetes pod did not work that way. For a long time the honest implementation was a coordinated live migration wearing a hotplug costume, and the APIs were split — some declarative, some subresources. We were solving a VM problem with a mechanism pods did not have. Live migration itself started as “can we even move this thing.” It is now how we upgrade clusters, hotplug, and rebalance. We built containerDisk before ImageVolume, DataVolume before volume populators, AAQ because ResourceQuota still does not understand a VM. Waiting on Kubernetes was the cost. Growing with it was the payoff.

A second, slower decision sat on top of that: treat virt-launcher as an untrusted, unprivileged pod, same as any default workload. The guest is third-party code. If something breaks out of QEMU, it should land in a tight box. We spent years stripping capabilities and privileges off virt-launcher to make that true. I would do both again.

Stu: When KubeVirt first started, we used the name “VirtualMachine” to reference the ThirdPartyResource that was running the actual VM. We then realized we needed an object to track the state of what a VM was supposed to look like when there wasn’t a VM actually running. We called this an OfflineVirtualMachine. These names were confusing and kind of misleading. We eventually changed the naming convention to what we have today: VirtualMachine and VirtualMachineInstance. A radical API change of that impactful a nature is never fun, but it was a necessary transition. I’m glad we decided to embrace that change, as it turned out well.

Q5. What’s something about the project’s culture you’d want to preserve for the next 10 years?

Stu: I like the open collaboration we enjoy as a community. Anybody can contribute, and anybody can provide a code review. You don’t have to be a member or maintainer of the project to contribute.

Roman: I appreciate the transformations the project went through. I loved the anarchic early days when I could just think of a use case and implement a new feature end-to-end. I also appreciate the work that many long-term contributors are putting in nowadays to wrestle with the bureaucracy that comes with growing projects.

I would especially want to preserve our CI culture, which we built up, and which can test so much, and so thoroughly, on every PR that gets merged.

Fabian: We have so many friendly and skilled people — I hope to see this throughout the next 10 years as well.

Vladik: The Razor: if something is useful for pods, do not implement it only for VMs.

We did step over it. Hotplug is the obvious case. I still think the Razor is right overall. Those exceptions should stay rare, and they should shrink when Kubernetes grows the primitive (in-place pod resize is the current example).

I won’t pretend people stay because they like each other. Plenty of early people left. What I felt in the early days was personal. What I would actually point at as culture is this: people motivated enough by the idea to keep making KubeVirt more of a Kubernetes project, not a private hypervisor with YAML on top — and to keep the community welcoming rather than hostile. That is how the project grew from “can we boot QEMU in a pod” into storage depth, network binding plugins, DRA, a general plugin mechanism, even multiple hypervisors, without throwing the original shape away.

Q6. Who is someone whose early contributions maybe never got the recognition that they deserve?

Vladik: Petr Horáček — first DHCP, and the idea that the guest can inherit the pod’s network identity without a vendor CNI.

Marc Sluiter — early, and not on the usual founder list.

There were and still are a lot of people in that category. Those two are the names I would not let fall off a ten-year post.

Stu: The earliest members of the KubeVirt project were actually Fabian, Roman, myself, and Adam Young. Adam left early enough in the project’s lifecycle that he doesn’t often get counted as an original community member. Subsequently, we also enjoyed the company of David Vossel and Vladik Romanovsky, who both have had a profound impact on the project.

Roman: There were too many small but important improvements — every single one mattered to bring the project to where it is now.

While it was not an active contribution, Itamar, our director back then, just decided to put some people on the experiment and let us run with practically no constraints. As a result, we brought a quite nice thing to the OSS world.

Fabian: Oh, there are many. Red Hat itself should be named. Even during tough times, Red Hat stayed committed to KubeVirt, and in my opinion took a big share in making KubeVirt what it is today.

Q7. KubeVirt made the decision to join the CNCF as a sandbox project in 2019. What was the impetus behind that, and would you recommend it to other projects?

Stu: In my view, the motivation to pursue joining the CNCF was to emphasize the fact that KubeVirt is truly a community project, not owned by a single company. This turned out to be an extremely good decision, in that such a stance on vendor neutrality has encouraged a welcoming ecosystem where other vendors and companies feel willing to contribute. Yes, the CNCF incubation and graduation process is… rigorous… but the payoff in community support is well worth it. I’d highly recommend it to any project with the opportunity to join.

Roman: Being part of the CNCF ecosystem is an important indicator of product readiness. Joining the CNCF was an important message to external contributors and adopters, signaling our readiness as we worked through the graduation process.

Fabian: That was also a small “wow” moment. To me, the CNCF is — and was — managing this ecosystem of projects very well. They are welcoming, supportive, and really try to have an eye on the needs of the projects. So, tipping my hat and kudos to them.

Seeing how the CNCF supports projects was amazing, and we needed this at that time. It also allowed us to join KubeCons more easily and get in touch with adopters and developers. And lastly, this is also driven by adopters — they see graduation as a critical step in the maturing of the project itself. While it is not so much of a technical nature, it is still a reflection of the technical maturity and adoption, which in turn is a manifestation, in some way, of trust of others.

Vladik: I was not driving that proposal. From where I sat, the useful part showed up later: the project stopped reading as a Red Hat side project. Contributors and users showed up who would have had a harder time if the home had stayed a vendor repo.

I also thought VMs belong in Kubernetes as first-class citizens. A foundation next to Kubernetes was the right kind of home for that claim.

I would still tell a project that wants to be infrastructure — not a product feature — to go to a foundation when the design is stable enough that neutrality matters more than speed. Sandbox first was the right shape for us.

Bonus question: Is there anything else you’d like to add about KubeVirt’s origins, early development, or the 10 years journey since?

Stu: A fun piece of trivia: in its original inception, KubeVirt used Docker as the primary container engine. In modern KubeVirt, we now default to podman.

Fabian: Well, at some point the trademark on Kubernetes was more enforced. And for a couple of weeks — when it was unclear who could keep the “Kube” prefix — we were racing to figure out new or alternative names for KubeVirt. It could have led to the same project now being named Amodea, McLean, Kerosene, Virtnetes, Veliform, TriVirt, Virtrinity, Littoral, and others to run your VMs. I’m quite happy that we could stick to KubeVirt, which is simple and spot-on: Kubernetes Virtualization.

Vladik: The interesting ten years were not the first boot of QEMU in a pod. They were the years of looking like we were behind traditional virt — no full control of the node — while we waited for Kubernetes to grow the primitives we had faked. ImageVolume showing up after containerDisk is the pattern I care about. Same architecture, less private machinery.

Dear Falco Community, we are thrilled to announce Falco Operator 0.5.0!

If you have ever wondered "is my new rule actually loaded on every node?", this release is for you. Version 0.5.0 rethinks how rules, plugins and configuration travel from your cluster's API to the Falco pods that need them, and gives you a clear answer, node by node.

Around that new core, we hardened almost every step of the artifact lifecycle and prepare

Earlier this year, ControlPlane, a cloud native security consultancy, decided to build a threat model of open source Linkerd on their own volition. They tried to anticipate every way an attacker or a bad configuration could undermine it, and shared the results with our team as creators of Linkerd, before publishing this for anyone to read. In this blog, I will summarize the findings and share actions you can take right now

Just in time for Kubecon US 2026 we’re excited to announce the release candidate for Argo Rollouts 1.11. This release has 156 commits from 49 contributors. It adds rollout duration tracking, shared secrets for analysis, new authentication options for Prometheus, and better Datadog and New Relic support. It also fixes several reliability issues with aborts and traffic routing. If you use the Web metric provider with plain-text responses, please read the possible breaking change below before you upgrade.

You can read the full CHANGELOG for the complete list of changes, and download the RC from the v1.11.0-rc1 release page.

Rollout duration tracking

How long does a full rollout take in your organization? And how much of this time do you spend waiting for someone to click “promote”? Up until now, you needed custom tooling to answer these questions. New in 1.11, every Rollout records timing information for the current update in its status:

status:
  duration:
    rolloutStartedAt: "2026-11-04T10:00:00Z"
    finishedAt: "2026-11-04T10:42:00Z"
    totalManualPauseDurationSeconds: 1800
    completionStatus: Promoted

The controller also exposes 3 new Prometheus histograms:

• rollout_duration_seconds measures the total time of a rollout from start to finish, including all pauses.
• rollout_progression_duration_seconds measures the active progression time without manual pauses. This is the actual time your automation spent deploying.
• rollout_manual_pause_duration_seconds measures the time a rollout waited for a person to act. This includes pause: {} steps, blue/green rollouts without auto-promotion, and inconclusive analysis.

Each histogram has a status label that shows how the rollout ended: Promoted, FastPromoted, Aborted, Superseded, RolledBack, or FastRolledBack. You can use these metrics to build dashboards for deployment lead time, track how often rollouts abort, and find canaries that stay paused for hours. The controller metrics documentation has all the details.

This work also fixes several edge cases in blue/green rollback detection and scale-down delays.

Thanks to Alexandre Gaudreault (Intuit) and Zach Aller (Intuit) for this work.

Shared secrets for analysis

Most metric providers need a credential, like an API token for Datadog or New Relic. Until now, an AnalysisRun could only read secrets from its own namespace, so platform teams had to copy the same token into every application namespace, resulting in duplication and difficult maintenance.

Cluster administrators can now set up one dedicated namespace for shared analysis secrets:

--analysis-shared-secret-namespace=analysis-secrets
--analysis-shared-secret-mode=LabelFilter   # default, or AllSecrets

An analysis argument then opts in to the shared namespace:

args:
- name: api-token
  valueFrom:
    secretKeyRef:
      name: token-secret
      key: apiToken
      sharedSecret: true

For security reasons this feature is off by default. When you turn it on, the default LabelFilter mode only lets AnalysisRuns read secrets with the argo-rollouts.argoproj.io/shared-secret: “true” label. This means an AnalysisRun can only read secrets that someone created for this purpose. It works like the way Argo CD restricts tokenRef references. You can find more details in the analysis documentation.

Thanks to Mujib for this work.

Possible breaking change: Web metrics now evaluate plain-text responses

In earlier versions of Argo Rollouts, the Web metric provider reported every empty or non-JSON response as Successful. It never checked your successCondition or failureCondition. This meant a bad rollout could continue even when your health endpoint didn’t return what you expected.

Argo Rollouts now evaluates your conditions for every response. For plain-text bodies, the raw response goes into result:

metrics:
- name: health-check
  successCondition: result == "I am OK"
  provider:
    web:
      url: "http://my-server.com/health"

An empty body becomes an empty string. If any of your AnalysisTemplates query endpoints that return plain text or an empty body, check their conditions before you upgrade. Analysis that passed before might fail now.

Thanks to something-ai-hash for this fix.

Upgrading from 1.10.0 or earlier: The dashboard listens on localhost only

The dashboard now listens on 127.0.0.1 by default instead of 0.0.0.0. This change applies to both kubectl argo rollouts dashboard and the argo-rollouts-dashboard Deployment. It fixes a security issue where anyone on your network could reach the dashboard.

This isn’t new in 1.11. We already released it in 1.8.5, 1.9.2, and 1.10.1 to fix CVE-2026–82277. If you’re upgrading from 1.10.0 or earlier and you expose the dashboard through a Kubernetes Service, add — address 0.0.0.0 to the dashboard command to keep it reachable. You can find more details in the dashboard documentation.

Thanks to Jonathan West (Red Hat) for this fix.

Prometheus: Basic authentication and Google Managed Prometheus

The Prometheus metric provider gets 2 new authentication options in this release. They join the existing support for custom headers, SigV4, and OAuth2.

You can now use basic authentication, which is the usual setup for hosted Prometheus instances like Grafana Cloud:

provider:
  prometheus:
    address: http://prometheus.example.com:9090
    authentication:
      basicAuth:
        username: "{{ args.basicAuthUsername }}"
        password: "{{ args.basicAuthSecret }}"

If you run on Google Cloud, you can also query Google Managed Prometheus with Application Default Credentials, for example through Workload Identity on GKE:

provider:
  prometheus:
    address: https://monitoring.googleapis.com/v1/projects/$PROJECT_ID/location/global/prometheus
    authentication:
      google: {} # scope defaults to monitoring.read

Thanks to Yahia Berchi (Tiime) for basic authentication, and to Matías Barra, Lior Friedman (Yahoo), and Noy Segal (Yahoo) for Google authentication.

Aborts with dynamic stable scale no longer get stuck

If you use dynamicStableScale: true with an explicit abortScaleDownDelaySeconds, aborting a canary could get stuck forever. The controller never scaled down the canary, traffic never moved back to the stable version, and the stable ReplicaSet never returned to full size. In 1.11, aborts complete as expected. Traffic returns to stable first, and then the controller scales down the canary.

Thanks to Zach Aller (Intuit) for this fix.

Datadog: Grouped queries

In earlier versions, Datadog queries with a by {tag} clause crashed the metric provider. They now return one value for each group, and you can evaluate them with standard expressions:

metrics:
- name: per-endpoint-error-rate
  successCondition: max(result) < 0.05 # fail if ANY endpoint exceeds the threshold
  provider:
    datadog:
      apiVersion: v2
      query: "sum:trace.http.request.errors{service:my-service} by {resource_name}.as_count()"

This helps you catch a regression in a single endpoint that a global average would hide. The measurement also includes a metadata.groups field, so you can see which endpoint caused the failure.

Thanks to Chris Greeno (Fresha) for this work.

New Relic: Account ID for each metric

The New Relic metric provider has a new, optional accountId field that overrides the account in the profile secret. You can now use one set of credentials to query different New Relic accounts from different rollouts:

provider:
  newRelic:
    query: |
      FROM Transaction SELECT percentage(count(*), WHERE httpResponseCode != 500) as successRate where appName = '{{ args.application-name }}'
    accountId: "1234567" # optional, overrides the account-id from the profile secret

Datadog already supports multiple accounts in a similar way.

Thanks to malekkhaled for this work.

Notification context

Rollout notifications now support user-defined context, just like Argo CD. You can define shared values once in the notifications ConfigMap and use them in any template:

data:
  context: |
    region: east
    environmentName: staging
  template.rollout-completed: |
    message: "Rollout {{ .rollout.metadata.name }} completed in {{ .context.environmentName }} in the {{ .context.region }} region."context: |

Thanks to Kalen Wessel (Zapier) for this work.

Other improvements

• Istio: Experiments with multiple destinations, like a baseline and a canary, no longer leave a stale destination in the VirtualService after they finish. Thanks to Kyle Zhu (Coinbase).
• Dashboard: Actions like promote and abort no longer hang. Thanks to sungyoon kim.
• Dashboard: An empty list of rollouts no longer breaks the page. Thanks to Dave Dennis (Stablecore).
• Dashboard: The namespace autocomplete now filters by what you type. Thanks to Weimin Wang (Turo).
• CLI: kubectl argo rollouts status — watch no longer panics. Thanks to Jade (Pyler).
• Traffic routing: Argo Rollouts now rejects maxTrafficWeight values below 1, and very large values no longer overflow. Thanks to Sergey Sannikov.
• APISIX: We fixed the permissions for managed routes and updated the getting started guide. Thanks to Yilia Lin (API7.ai).

A better experience for contributors

Not everything in a release is a new feature. A big part of this release makes Argo Rollouts easier and faster to work on for everyone who contributes:

• We fixed several flaky unit tests, and the tests no longer call external services like the GitHub API or AWS during a run. The battle against flaky tests is always ongoing…
• CI now always runs the unit and dashboard tests. The e2e tests report one combined result that works well with branch protection, and they now run against Kubernetes 1.37. The Dependabot configuration was updated to match Argo CD.
• Easier local setup: A fresh make codegen no longer fails because the dist/ folder is missing, protoc now works on Apple Silicon Macs, and the contributor setup instructions are correct.
• Refactoring: Canary and blue/green reconciliation now run on a shared stage pipeline instead of separate, hand-written chains. Nothing changes for end users, but future fixes to rollout reconciliation will be smaller and easier to review. As we promised in 1.10, more improvements are coming in this area.

Thanks to Kostis Kapelonis (Octopus Deploy), Peter Jiang (Intuit), Niko Kurtti, potterhe, and jxj (DaoCloud) for this work.

Bug fixes

This release also includes several smaller bug fixes and documentation updates for analysis, traffic routing, the dashboard, and the controller. You can find the complete list in the full CHANGELOG.

Summary

Thank you to everyone who contributed code, documentation, bug reports, and reviews to this release. Please test the release candidate and tell us about any problems in the CNCF Slack or open an issue. If you’re building a plugin for Argo Rollouts (metric, traffic, or step), we’d also love to hear about it!

Photo by Muhammad Raihan Hadianto on Unsplash

Argo Rollouts 1.11 release candidate was originally published in Argo Project on Medium, where people are continuing the conversation by highlighting and responding to this story.

Meet wasmCloud at KubeCon + CloudNativeCon NA 2026 in Salt Lake City: find maintainers in the Project Pavilion, join a hands-on sandboxing workshop, and see WASI WebGPU at Cloud Native AI + Inference Day.

Headlamp AI Assistant, an open source Headlamp plugin maintained under Kubernetes SIG UI, adds a conversational interface to Headlamp. It can inspect cluster state and request changes. That access requires clear trust boundaries, predictable cluster targeting, provider choice, accessibility, and tests that do not depend on a live model.

Add organization-specific knowledge without rebuilding the plugin​

The 0.3.0-alpha release introduced skills, which are versioned sources of an organization's instructions and reference information from GitHub repositories. The assistant can inspect a skill, match it to a request by keywords or embeddings, and add the relevant content. Users can pin a source to a branch, tag, or commit SHA and verify it with a SHA-256 integrity hash.

The loader rejects path traversal and symbolic links, restricts supported protocols, and limits archive size and file count. Skill content becomes part of the model's instruction context. Review that content, pin production sources to an immutable revision, and use the integrity field for sources outside your control.

The same release expanded Model Context Protocol (MCP) support. Desktop users can configure MCP servers, discover their tools, inspect input schemas, and enable tools individually. Servers that use HEADLAMP_CURRENT_CLUSTER restart when the active cluster changes. HolmesGPT integration also became configurable and now checks service health before switching into agent mode, preserving a pending prompt when the service is unavailable.

Skills add organization-specific knowledge without changing the plugin. MCP servers add tools.

Auto-detect available models​

In the desktop application, Auto Detect finds model access that is already available on your computer. It currently discovers:

• GitHub Copilot through GitHub command-line authentication
• Local Llama and other models exposed through Ollama
• Azure OpenAI and Azure AI Foundry chat deployments across enabled Azure subscriptions

For detected Azure deployments, the plugin checks access before saving them. It reports missing role-based access control permissions and private-network connectivity problems separately, which helps explain why a discovered Azure model might not be usable. The Azure detector retrieves account keys when needed but does not persist them.

Manual configuration remains available for other hosted, local, and OpenAI-compatible providers. Support for more automatic discovery sources is planned.

Provider choice does not remove the need for a data review. Depending on the tools and provider that you enable, prompts can include Kubernetes resource data and tool results. Before using the plugin, review the model provider's data-handling terms, the enabled tools, and the access granted to the Headlamp session.

Diagnose problems as they happen​

Without proactive diagnosis, an investigation starts after you open the assistant and ask for help. Depending on the agent and tools involved, gathering context and returning an answer can take several minutes.

After you sign in to Headlamp and open the assistant with proactive diagnosis enabled, the plugin checks recent warning and error events and starts investigating automatically. You do not need to enter a prompt first. While the panel remains open, the plugin periodically checks for more events and can prepare summaries of affected workloads and suggested follow-up questions.

The current preview does not run while the assistant panel is closed. Proactive diagnosis remains an opt-in alpha preview and is off by default. Feedback from early users is guiding further improvements to when investigations start, how results are prioritized, and how progress is presented.

The longer-term goal is to save time during troubleshooting by starting an investigation earlier. When a user needs to diagnose an issue, the agent should already have examined the issue and prepared useful context instead of beginning from scratch.

Improve accessibility and internationalization​

The shared UI work also added keyboard controls for resizing and dialogs, accessible names and status announcements, stronger dialog semantics, and theme-aware code rendering. Automated component tests include axe checks. These checks do not show complete Web Content Accessibility Guidelines (WCAG) conformance. A separate fix made the top-bar icon readable against light, dark, and custom navigation themes.

Internationalization work connected the shared UI to Headlamp's translation instance and made production strings extractable. The plugin now has locale catalog support for English and 18 additional locales. The UI translates display labels without changing the canonical prompt sent to a model.

Improve quality through shared packages, tests, and documentation​

Contributors separated the assistant into four parts:

• ai-common contains provider, conversation, prompt, skill, tool, MCP, and security behavior.
• ai-ui contains reusable React components, settings, accessibility behavior, and rendering.
• The Headlamp plugin code connects the shared packages to Headlamp, Kubernetes resources, plugin settings, and the desktop application.
• The standalone headlamp-ai command-line interface uses the same core without the Headlamp UI.

This structure makes the shared behavior testable without loading Headlamp and gives each package a defined public interface. It also centralizes secret redaction, tool approval, cancellation, and provider configuration.

Resource updates now use Kubernetes merge semantics instead of merging live objects in the browser. The tool layer also redacts common credentials and keeps added tool authority configurable. Contributors use the CLI to test providers, skills, and Kubernetes tools outside the Headlamp UI. Fixture-backed chat models, scripted agents, mock tools, and offline conversations make tests reproducible and avoid spending model tokens during routine development.

Automated coverage now includes package and plugin unit tests, axe accessibility checks, and end-to-end scenarios that use KWOK and Playwright. The main README documents provider, HolmesGPT, and MCP configuration. A separate end-to-end guide documents how to build the plugin, create the test cluster, and run the Playwright scenarios.

Make multi-cluster context explicit​

From the Home view, the assistant now receives the available clusters and can target one by name. It selects a default only when one cluster is configured and includes the target cluster in write confirmations. You can discuss resources in multiple configured clusters from one conversation without navigating into each cluster first.

Try it and contribute​

The AI Assistant remains alpha software, but it is already useful for asking questions about cluster state, gathering context, and starting guided investigations in Headlamp. Over the past six months, shared packages, broader automated and end-to-end tests, clearer documentation, and more predictable provider and cluster handling have improved its quality.

Model output can still be wrong. Review the cluster, namespace, resource, and patch before approving a write. Start with read-only tools and a non-production cluster while evaluating the plugin.

Installation and configuration instructions are in the AI Assistant README. To report a problem or propose an improvement, use the Headlamp plugins repository's issue tracker. Contributions to provider support, evaluation fixtures, accessibility, documentation, and Kubernetes-specific safety behavior are welcome.

There are many reasons for building your own OpenTelemetry Collector distribution. The existing distributions are designed to be broadly useful, which means they include many more components than what an individual deployment uses. Building your own gives you the ability to select only the components you care about, add your own custom components, and control exactly what ends up in the binary you run in production.

One of the reasons I prefer to build my own distribution is to limit t

Imagine launching an AI/ML distributed training job across 64 GPUs. Two frequent problems happen:

1. Idle Resource Trap: The cluster finds capacity for 63 GPUs and sits idle waiting for the 64th to join, wasting massive resources and increasing the cost. Progress is zero.
2. Dependency Deadlock: Suppose your training pipeline requires one initializer (to download datasets) and 64 GPU workers (to train). Kubernetes allocates all 64 GPUs to the workers within your quota, so the initializer cannot be scheduled. The workers cannot start without the data, and the initializer cannot be scheduled without capacity: a deadlock.

Standard Kubernetes struggles with AI/ML distributed training (data/model parallelism) that needs the jobs/tasks to be done via groups/gangs of pods.

To solve these problems, the Kubernetes community introduced the Workload API, which brings gang scheduling, topology-aware scheduling, workload-aware preemption, DRA integration, and more through the Workload, PodGroup, and CompositePodGroup objects.

In this blog, we walk you through why TrainJob adopts Workload-Aware Scheduling (WAS), why it matters for AI/ML distributed training, and the architectural tradeoffs and future work involved. TrainJob implements WAS at three levels, and we explain each level and the reasoning behind it.

You’ll also learn how it simplifies the workflow for both data scientists and platform engineers to avoid boilerplate code.

Why is WAS Important for TrainJob?

By default, kube-scheduler uses a greedy approach, trying to schedule pods one at a time instead of planning for the whole group. It has no native understanding of PyTorch, TrainJobs, or the needs of distributed ML training, where all pods must be scheduled together and may have specific placement requirements, such as running on the same rack.

To bridge this gap, Kubernetes SIG-Scheduling started a Workload-Aware Scheduling (WAS) working group to establish WAS building blocks: Workload, PodGroup, and CompositePodGroups. However, these building blocks require actual workload owners like the TrainJob controller to create such resources.

The TrainJob controller creates Workload objects to guide kube-scheduler on how to admit the pods as a unified group, enforcing all-or-nothing gang scheduling so expensive GPUs are never wasted, alongside topology-aware scheduling for low communication latency.

Best of all, Trainer APIs hide these complex, low-level scheduling resources from data scientists: practitioners simply specify numNodes in their TrainJob while platform administrators pre-configure ClusterTrainingRuntime with the desired scheduling configuration.

Hierarchical Scheduling

To understand the three levels, we must look at how Kubernetes scheduling aligns with the physical reality of an AI data center.

In distributed training, GPU workers communicate constantly to synchronize gradients. If communicating GPUs are placed across distant servers or cloud zones, network latency explodes and training throughput degrades significantly.

Kubeflow Trainer v2.4 introduces a 3-tier hierarchy to map software jobs to physical hardware topology:

TrainJob Multi-Tier Scheduling Topology across Zone, Block, and Rack domains

Level 1 - TrainJob Level (Zone)

• Kubernetes Object: PodGroup in v2.4 (CompositePodGroup planned)
• Physical Domain: Cloud Zone (topology.kubernetes.io/zone)
• Why it’s needed: Treats the entire training job as a single unit. It ensures all workers are located within the same cloud zone, avoiding cross-zone latency and bandwidth charges. Ideal for uniform workloads (e.g., PyTorch DDP).
• User Story: Distributed PyTorch Training with Gang Scheduling:

  As a platform engineer, I want to run a distributed PyTorch TrainJob with 100 nodes that must all be scheduled together. If only 99 workers can be scheduled, no pods should start.

1. ClusterTrainingRuntime and TrainJob Definitions

apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
  name: torch-distributed
spec:
  mlPolicy:
    numNodes: 1
    torch: {}
  scheduling: {}  # Enable workload-aware scheduling!
  template:
    spec:
      replicatedJobs:
        - name: node
          template:
            metadata:
              labels:
                trainer.kubeflow.org/trainjob-ancestor-step: trainer
            spec:
              template:
                spec:
                  containers:
                    - name: node
                      image: pytorch/pytorch:2.15.0-cuda13.4-cudnn9-runtime
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: my-job
spec:
  runtimeRef:
    name: torch-distributed
  trainer:
    image: docker.io/torch-run
    numNodes: 100
    resourcesPerNode:
      requests:
        nvidia.com/gpu: 4

2. Workload Resource Created by TrainJob Controller

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
  name: my-job-<hash>
  ownerReferences:
    - apiVersion: trainer.kubeflow.org/v1alpha1
      kind: TrainJob
      name: my-job
      controller: true
spec:
  controllerRef:
    apiGroup: trainer.kubeflow.org
    kind: TrainJob
    name: my-job
  podGroupTemplates:
    - name: my-job
      schedulingPolicy:
        gang:
          minCount: 100 # Computed from trainJob.spec.trainer.numNodes

3. Automatically Generated PodGroup by the TrainJob controller

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: <workload-name>-<hash>
  ownerReferences:
    - apiVersion: trainer.kubeflow.org/v1alpha1
      kind: TrainJob
      name: my-job
      controller: true
spec:
  workloadRef:
    workloadName: <workload-name>
    templateName: my-job
  schedulingPolicy:
    gang:
      minCount: 100

4. Resulting Pod Spec Association

spec:
  schedulingGroup:
    podGroupName: <workload-name>-<hash>

Level 2 - ReplicatedJob Level (Network Block)

• Kubernetes Object: PodGroup in v2.4 (CompositePodGroup planned)
• Physical Domain: Network Block / Spine Switch
• Why it’s needed: Prevents dependency deadlocks in multi-role pipelines. It allows platform engineers to give data initializers a basic policy (so they can run lazily on cheap nodes to download datasets), while allocating a coordinated gang policy for GPU workers once data is ready.
• User Story:

apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
  name: deepspeed-distributed
spec:
  mlPolicy:
    numNodes: 1
    mpi:
      numProcPerNode: 1
  scheduling:
    replicatedJobs:
      # The dataset initializer doesn't need GPUs, so it's scheduled individually.
      - targetReplicatedJobs: ["dataset-initializer"]
        schedulingPolicy:
          basic: {}
      # MPI launcher and workers are scheduled as one gang within the same zone.
      - targetReplicatedJobs: ["launcher", "node"]
        schedulingPolicy:
          gang: {}
        schedulingConstraints:
          topology:
            - key: topology.kubernetes.io/zone
  template:
    spec:
      replicatedJobs:
        - name: dataset-initializer
          ...
        - name: node
          dependsOn:
            - name: dataset-initializer
              status: Complete
          ...
        - name: launcher
          dependsOn:
            - name: node
              status: Ready

Level 3 - Job / Per-Replica Level (Rack-Level)

• Kubernetes Object: PodGroup
• Physical Domain: Rack / Top-of-Rack Switch (topology.kubernetes.io/rack)
• Why it’s needed: Maximum network throughput and failure isolation. Workers under the same Top-of-Rack (ToR) switch communicate at fast, non-blocking speeds. Each Job can also get its own DRA claims and failure boundary, the building block for Multi-Slice architectures (like TPU slices). In v2.4, the targeted ReplicatedJob can have at most 1 replica, since per-replica PodGroups require Kubernetes v1.38.

Given the above requirements, the TrainJob topology can be represented using the following WAS building blocks for each Level.

Kubeflow Trainer v2.4 creates PodGroups at the level you choose

In v2.4, Trainer creates only PodGroups, and each ReplicatedJob is scheduled at exactly one level. Platform admins choose the level in the runtime’s spec.scheduling.

Introducing the Three Levels in TrainJob Topology

Each PodGroup is independent, so today the hierarchy has a single layer of PodGroups with no parent above them. That is already enough to solve both problems from the introduction:

• Idle Resource Trap: A gang PodGroup admits all of its workers or none, so GPUs are never held while the job waits for the last worker.
• Dependency Deadlock: At Level 2, the initializers get a basic PodGroup and start right away. The GPU workers wait in their own gang until all of them can be scheduled.

Adopting CompositePodGroup - ROADMAP

CompositePodGroup is alpha in Kubernetes 1.37. Once Trainer adopts it, Levels 1 and 2 become CPGs, Level 3 stays a PodGroup, and the levels can be combined instead of choosing just one, as the diagram shows. This enables:

• Gang of Gangs: A TrainJob-level CPG admits several Job-level gangs together or not at all. An example is multiple TPU slices, each with its own DRA claims.

• Nested Topologies: Each level keeps its own placement constraint. For example, the whole TrainJob stays in one zone, each ReplicatedJob in one network block, and each Job in one rack.

• Mixed Policies in One Tree: The initializers keep a basic policy, while a parent gang keeps the GPU worker gangs together.

In short: today you choose between one big gang and several independent ones. With CPG, Trainer can express “all of these gangs, together, close to each other”.

To learn more about Workload-Aware Scheduling in Trainer, check out the docs and follow the progress in KEP-3015.

Other Feature Highlights in Trainer v2.4

Beyond Workload-Aware Scheduling, Kubeflow Trainer v2.4 includes:

• Per-TrainJob schedulerName overrides (#4265)
• Job replicas for multi-slice TPU training (#4250)
• Secured metrics endpoint with TLS and RBAC-based auth (#3912)
• Prometheus monitoring overlay (#4102)
• PyTorch 2.15 and CUDA 13.4 in the default runtimes (#4369)

Dynamic Resource Allocation (DRA)

Trainer 2.4 adds support for attaching accelerators and other specialised hardware to TrainJobs using Kubernetes Dynamic Resource Allocation (DRA) API (GA in kubernetes 1.34).

• DRA provides a more flexible way to request and attach hardware to TrainJobs, complementing traditional extended resource requests such as nvidia.com/gpu: 1. Unlike extended resources, DRA supports more advanced device selection, sharing, and allocation policies, giving users greater control over how hardware is assigned to training workloads.

• To learn more about using DRA in Kubeflow Trainer, check out the Dynamic Resource Allocation (DRA) docs.

OptimizationJob Alpha Integrations

• Declarative HPO: Enables the OptimizationJob -> TrainJob hierarchy, leveraging TrainJob’s native progress tracking to eliminate the legacy Katib DB dependency.
• Getting Started: Enable the OptimizationJob and TrainJobStatus alpha feature gates (#4331) and follow the OptimizationJob user guide.
• CEL-Native Validation: Implements server-side CEL validation for SearchSpace and TrialConfig constraints, ensuring high-quality hyperparameter search configurations at the API layer.
• OptimizationJob KEP-3562

Release Notes: Kubeflow Trainer v2.4.0 Changelog

Breaking Changes & Upgrade Notices (v2.4)

1. ReplicatedJob Name Length at 20 Characters (#4391)
   TrainingRuntimes and ClusterTrainingRuntimes with ReplicatedJob names longer than 20 characters are rejected on creation and on updates that modify spec.template.

2. CertManagement Configuration Fields Renamed (#3386)
   Renames the CertManagement configuration fields from webhook-specific names to generic names, since the TLS certificate is used by the metrics server and the status server in addition to webhooks.

3. TrainJob image synchronization for MPI Runtimes (#4301)
   For MPI runtimes, Trainjob spec.trainer.image now configures the image for both the worker nodes and the launcher pod. Previously it only configured the launcher pod.

4. Helm manifests retain CRDs on uninstall (#4122)
   Helm charts now keep the Custom Resource Definitions (CRDs) on helm uninstall (crds.keep: true by default), since deleting a CRD deletes every TrainJob, TrainingRuntime, ClusterTrainingRuntime, and OptimizationJob in the cluster. Set crds.keep=false to remove the CRDs with the release.

5. Enabling Data Cache in Helm Chart no longer automatically installs LWS subchart (#4120)
   When data cache is enabled, the Helm chart no longer automatically installs the LeaderWorkerSet (LWS) dependency. Users who rely on the chart to install LWS must explicitly set dataCache.lws.install=true. This allows users with an existing LWS installation to enable data caching without installing LWS again.

6. MPI runtimes emit PRTE env vars for Open MPI 5 (#4029)
   Suspended TrainJobs that use the MPI plugin (e.g., DeepSpeed, MLX) and were created before the upgrade fail to resume after upgrading to v2.4, because the new PRTE_MCA_* env vars can’t be applied to the existing immutable JobSet. Wait for MPI-based TrainJobs to complete before upgrading, or recreate them after the upgrade.

References

1. Proposal for Integration of WAS with TrainJob Controller

2. Gang Scheduling using Workload Object

3. Workload API

4. How to Use Workload API in TrainJob Controller

Join the Community

The Kubeflow Trainer is built by and for the community. We welcome contributions, feedback, and participation from everyone! We want to thank the community for their contributions to this release. We invite you to:

Contribute:

• Read the Contributing Guide.
• Browse the good first issues
• Explore the GitHub Repository

Connect with the Community:

• Join #kubeflow-trainer on CNCF Slack
• Attend our biweekly Kubeflow Trainer meetings

Learn More:

• View the GitHub Release
• Explore the Kubeflow Trainer docs

KubeCon + CloudNativeCon North America 2026 is taking place next week and we are happy to be heading to Salt Lake City again! Here is where you can find us.

Maintainer Summit​

On Sunday, November 8, we will attend the Maintainer Summit to connect with other maintainers before the main event kicks off. This pre-event gathering is honestly one of our favorite parts of KubeCon.

Project Pavilion​

We will be at the Headlamp kiosk (P-5B) in the Project Pavilion on Tuesday, November 10: 3:00 PM - 7:30 PM MST. Whether you are an active contributor, a user, or just curious about Kubernetes UIs, come say hi!

See You There​

Looking forward to another great KubeCon!

Today we are proud to announce the release of etcd-operator v0.3.0. This release represents a substantial rewrite of the operator code in order to move from StatefulSet to a custom controller that can handle more failure conditions.

Versions v0.1.0 and v0.2.0 were built with a CRD backed by the StatefulSet workload. The team found that certain limitations and assumptions in Statefu

OpenTelemetry JavaScript has released SDK 3.0 (stable v3.0.0 packages and experimental v0.300.0 packages) and API

Strimzi 1.3.0 and 1.3.1 have been released with various new features and improvements. As with previous releases, we have created a video that introduces the main changes. You can watch the video here or on our YouTube channel and check the release notes for 1.3.0 and 1.3.1.

To download the release go to Keycloak downloads.

Upgrading

Before upgrading refer to the migration guide for a complete list of changes.

All resolved issues

Security fixes

• #53061 [CVE-2026-92358] Cross-browser fallback proof survives successful ac

KubeCon + CloudNativeCon North America 2026 kicks off next week in Salt Lake City, November 9-12, and the OpenFeature community will be there. Whether you want to catch a talk, chat with maintainers at our kiosk, or join us for an evening out, here's everything you need to know.

Sessions at KubeCon​

OpenFeature maintainers, contributors, and adopters are presenting four sessions this year:

How We Are Enabling Feature Flagging At Scale​

CVS Health's DevEx team needed feature flagging for a large engineering organization and found that the options were either expensive vendor platforms or limited home-grown tools. Juan shares how they built a standardized, open source flagging platform on OpenFeature, how they collaborated with the project along the way, and their plans to bring that work to the open source community.

Hope Is Not a Rollout Strategy: Balancing (AI) Quality, Cost, and Risk Through Experimentation​

Shipping AI features responsibly is an experimentation problem. Todd covers the three axes of quality, cost, and risk, and walks through a reference loop using OpenFeature, flagd, and OpenTelemetry that you can build next sprint.

OpenFeature Update from the Maintainers​

This year the project reached a major milestone: platforms like Cloudflare, Google Cloud, Dynatrace, and Datadog now build their feature flag products directly on OpenFeature. We'll dig into what that means for adopters and the ecosystem, then cover the rest of what's new across the SDKs, flagd, and OFREP. Bring your questions.

OpenFeature as a Control Plane for OpenTelemetry​

What if telemetry verbosity could adapt automatically? Simon and Juliano combine OpenTelemetry and OpenFeature to control instrumentation at runtime, switching between lightweight traces and deep diagnostic telemetry without redeploying, and show how OpenTelemetry semantic conventions can act as a standard control surface.

Visit Our Kiosk​

Stop by and chat with OpenFeature maintainers at our kiosk in the Project Pavilion (Solutions Showcase, Level 1, Hall 5, Cosmic Canyon Zone).

Kiosk Number: P-15A

Schedule:

• Tuesday, November 10: 15:00 - 19:00 MST
• Wednesday, November 11: 13:55 - 17:00 MST
• Thursday, November 12: 12:25 - 14:00 MST

We'll be there to answer questions, demo the latest features, and hear how you're using OpenFeature in your organization.

Meet the Community​

A number of OpenFeature maintainers and contributors will be in Salt Lake City throughout the week, and we're organizing informal meetups and getting together for the evening parties. If you'd like to join or help coordinate, hop into the #openfeature-kubecon channel on the CNCF Slack. It's a great opportunity to connect face-to-face with the people building the project.

What's New in OpenFeature​

A lot has happened since KubeCon EU 2026. Our mid-2026 update covers it in depth, but here are some of the highlights:

• Platform-native feature flags: Datadog, Vercel, Cloudflare, Google Cloud, Mixpanel, and Octopus Deploy all shipped feature flag services built on OpenFeature.
• New vendors in the ecosystem: ConfigDirector, Feat, Featureflip, Featurevisor, Pendo, Rollgate, Superposition, and Switchbox shipped their first OpenFeature providers.
• Specification v0.9.0: the largest specification release in over two years.
• Isolated API instances: SDKs such as .NET and Ruby now support isolated API instances alongside the global singleton.
• C++ SDK: the new C++ SDK continues to mature with regular releases.
• OFREP server-sent events: the OFREP ADRs adding SSE support and updated caching behavior have been accepted.

We're looking forward to seeing you in Salt Lake City. If you haven't registered yet, there's still time:

Register for KubeCon + CloudNativeCon North America 2026

Follow us on Bluesky, LinkedIn, and join the CNCF Slack to connect with the community. See you in Salt Lake City!

🎉 Score turned 2 as a CNCF Sandbox project this past July! 🎉
On July, 2024, Score joined the CNCF and last year, we celebrated our first anniversary. Time still flies, and this second year was packed with great moments, with the same takeaway: Score only goes as far as its community takes it. Thank you to every contributor, user, speaker and maintainer who helped along the way. 💙
Our mission hasn’t changed:

wasmCloud 2.12 adds the WorkloadDaemonSet CRD and a /scale subresource on WorkloadReplicaSet, lets a host plugin serve secrets to other plugins, sizes the instance pool from the node, adds an S3 blobstore backend and NATS client authentication, and signs release tags with Sigstore. It includes breaking changes.

Published on November 4, 2026

We are excited to announce the release of KServe v0.21. This release focuses on autoscaling flexibility, operational hardening, and deeper llm-d integration, with key highlights including:

• Direct KEDA autoscaling for LLMInferenceService, with true scale-to-zero
• Kernel Cache initial code donation with image signature verification
• Standalone vLLM tokenizer deployment replacing the UDS tokenizer sidecar
• Canary traffic splitting on RawDeployment HTTPRoutes
• Rollout strategy controls for GPU-constrained rolling updates
• LocalModelCache support for LoRA adapters
• oci+fetch:// KServe-side OCI image pull and ModelScope (modelscope://) storage support
• TLS hardening: cluster TLS profile flags, hot reload of rotated certificates, and TLS for transformers
• Python 3.13 support and Go 1.26 upgrade

🚀 LLMInferenceService Enhancements​

📈 Direct KEDA Autoscaling and Scale-to-Zero​

Until now, KEDA-based scaling on LLMInferenceService was only available through the Workload Variant Autoscaler (WVA) path. KServe v0.21 adds a direct KEDA path via the new spec.scaling.keda field (#5839), letting users define their own KEDA triggers (CPU, memory, custom Prometheus queries, etc.) without WVA. The controller creates the ScaledObject from the user-defined triggers. spec.scaling.keda and spec.scaling.wva are mutually exclusive, and decode/prefill workloads must use the same scaling mode.

spec:
  scaling:
    minReplicas: 1
    maxReplicas: 4
    keda:
      idleReplicaCount: 0
      triggers:
        - type: prometheus
          metadata:
            serverAddress: http://prometheus.monitoring:9090
            query: sum(vllm:num_requests_waiting{namespace="my-ns"})
            threshold: "5"

In addition, idleReplicaCount: 0 is now accepted for both the WVA-mediated and direct KEDA paths (#5996), enabling true KEDA scale-to-zero. triggerAuthName can now also be used without authModes (#6004).

🔤 Standalone vLLM Tokenizer Deployment​

The UDS tokenizer sidecar inside the scheduler pod has been replaced by a standalone tokenizer Deployment and Service running vLLM's /v1/completions/render endpoint over HTTP (#5712). This follows the llm-d router v0.9.0+ architecture and enables independent scaling, lifecycle management, and health checking of the tokenizer.

The monolithic precise-prefix-cache-scorer plugin is decomposed into the new 3-plugin pipeline (token-producer + precise-prefix-cache-producer + prefix-cache-scorer). Existing configurations are migrated automatically with a deprecation warning event, and a new TokenizerReady condition reports the tokenizer status.

🔄 Rollout Strategy for Rolling Updates​

A new rolloutStrategy field (#5916) lets users configure maxUnavailable and maxSurge for both single-node Deployments and multi-node LeaderWorkerSets, including prefill workloads. This enables rolling updates on GPU-constrained clusters without spare hardware, or faster rollouts with additional surge capacity:

spec:
  rolloutStrategy:
    maxSurge: 0
    maxUnavailable: 1

💾 LocalModelCache for LoRA Adapters​

LoRA adapter URIs can now be served from LocalModelCache and LocalModelNamespaceCache (#5690). Cached adapters are mounted from PVCs instead of being downloaded at pod startup, independently of whether the base model is cached. Cache deletion is blocked while an LLMInferenceService references a cached adapter.

🧭 Scheduler (EPP) Configuration as Presets​

The default EndpointPicker configuration is now shipped as LLMInferenceServiceConfig presets instead of hardcoded strings (#5921), updated to the latest llm-d router plugins. Two presets mirror the llm-d guides: an optimized baseline profile and a prefill/decode disaggregation profile. Rendered config templates are now also validated through the admission chain before any child resources are created (#5918).

🔗 Working KV Transfer for Prefill/Decode​

Prefill/decode disaggregated deployments now get a NixlConnector injected automatically (kv_producer on prefill, kv_consumer on decode) when no other --kv-transfer-config is set (#6027). Previously, decode silently recomputed the prompt KV produced by prefill. User-supplied configuration still takes precedence.

📋 Additional LLMInferenceService Improvements​

• Workloads automatically labeled with the InferencePool selector when an InferencePool ref is set (#5624)
• Migration to new llm-d metrics flags and the metrics-data-source plugin (#5877, #5930)
• EndpointPickerConfig apiVersion migration (#5774)
• LoRA adapter validation and disambiguation of colliding mount paths (#6092, #6085)
• Traffic splitting group fixes for stopped and terminating members and pool matching (#5847, #6149, #6156)
• Agentic tool calling guide and samples (#5906)

🧠 Kernel Cache​

KServe v0.21 includes the initial code donation of Kernel Cache (#5590), starting with the MCV CLI tool and API for building kernel cache container images, extracting caches, and running preflight checks. Pre-compiled GPU kernel caches can significantly reduce model server startup times by avoiding repeated kernel compilation.

To ensure that only trusted kernel caches are used, this release also introduces a security verification framework (#6099), cert-mode signature verification (#6103), and a signing contract (#6118).

See the Kernel Cache documentation to get started.

🔧 InferenceService and Platform Improvements​

🐦 Canary Traffic Splitting on RawDeployment HTTPRoutes​

Building on the canary rollout support introduced in v0.20, HTTPRoutes in RawDeployment mode now carry weighted backend refs for canaries (#5912). The stable predictor receives the remaining weight after all canary percentages, and HTTPRoute names and hostnames remain stable through promotion — only backend refs change:

# HTTPRoute: my-model-predictor (20% canary)
backendRefs:
  - name: my-model-predictor
    weight: 80
  - name: my-model-v2-predictor
    weight: 20

Canary traffic is now gated on readiness to prevent transient 503s (#5984), canary weights are restricted to predictor rules (#5944), and orphaned autoscaler and OpenTelemetry resources are cleaned up on promotion (#5899).

🔍 InferenceService Tracing API​

InferenceService now has an optional top-level spec.tracing field (#6108), aligned with the LLMInferenceService tracing API. Setting tracing: {} applies defaults (otlp exporter to http://otel-collector:4317, parentbased_traceidratio sampler at 0.05), with admission validation for sampler names and exporter endpoints. This release establishes the API contract; workload injection will follow in a future release.

📦 Storage Enhancements​

• oci+fetch:// (#5739) — Step 3 of the OCI storage harmonization roadmap (#4083). An init container pulls the OCI image with oras-py and extracts the /models/ subtree, without depending on container runtime features. This works on older Kubernetes versions and restricted environments, and supports private registries through imagePullSecrets.

• ModelScope support (#5330) — download models directly from ModelScope using the modelscope:// URI scheme, mirroring the existing Hugging Face integration:

  storageUri: "modelscope://qwen/Qwen2-0.5B-Instruct"
  

• IBM Power (ppc64le) support for the storage initializer (#5235)

• AWS SDK migrated from v1 to v2 in the agent (#5438)

🎯 DRA Support for ServingRuntime​

ServingRuntime and ClusterServingRuntime now support pod-level resourceClaims for Dynamic Resource Allocation (#5828). Runtime defaults are merged with InferenceService predictor overrides, so DRA-backed InferenceServices can be scheduled correctly.

🧩 Independent CRD Installation​

CRD management has been restructured so CRDs can be installed independently from the controllers with proper cert-manager webhook annotations (#5843). All existing installation methods continue to work.

Additional Enhancements​

• Python 3.13 support (#5897)
• AutoGluon server injects a synthetic item ID for two-column time series inputs (#6009)
• Headless InferenceService status URL now reports the correct port (#5968)
• KEDA ScaledObject is skipped for components without autoscaling configured (#6029)
• Storage resources templated in the inferenceservice-config ConfigMap (#5697)
• JWE decryption supports both full JWK and raw key formats for confidential model serving (#6034)

🔒 Security Improvements​

TLS Hardening​

• Cluster TLS security profile integration (#5791) — new --tls-min-version and --tls-cipher-suites flags for webhook and metrics servers across all KServe controllers, defaulting to hardened TLS 1.2. The --enable-http2 flag has been removed from the llmisvc controller.
• Rotated TLS certificates are reloaded by the model server without a restart (#6106).
• TLS for transformers via the KSERVE_TLS_CERT_FILE and KSERVE_TLS_KEY_FILE environment variables, enabling end-to-end TLS (#6045), plus CA bundle injection for httpx (#5932).

Hardening and CVE Fixes​

• Controller manager ClusterRoles hardened by removing unused permissions (#5785)
• AutoGluon artifact loading secured and path traversal prevented (#5803, #5802)
• Agent crash prevented when model spec unmarshalling fails (#5700)
• Pillow — CVE-2026-40192, CVE-2026-54058, CVE-2026-59200 (#5959, #5998, #6010)
• cryptography — CVE-2026-69247 (#5957, #5999)
• black — CVE-2026-32274 (#5953)
• go-jose — GHSA-78h2-9frx-2jm8 (#5985)
• golang.org/x/crypto — CVE-2026-46595 (#5993)
• aiohttp — zip bomb DoS (#5971)
• kin-openapi — nil-pointer panic in request validation (#5992)
• pyasn1 — CVE-2026-59885 (#6011)
• urllib3 — CVE-2026-21441 (#6023)
• opentelemetry-go — CVE-2026-39883 (#6019)
• setuptools — CVE-2025-47273 (#6055)

📦 Dependency Upgrades​

• llm-d images upgraded to the latest v0.10 release (#6026)
• Envoy AI Gateway upgraded to v1.1.0 (#6122)
• KEDA upgraded to v2.20.2 (#6153)
• WVA bumped to v0.8.0 (#5722)
• Go upgraded to 1.26 (#6127)
• pyarrow now allows versions newer than 20 (#6125)

🏗️ Infrastructure and CI​

• Makefile split into fragment files and kustomize image overrides refactored (#6058, #6060)
• Updated install scripts for older versions and a new install script (#6061)
• Kustomize validation workflow for manifests (#5878)
• CLI flag drift detection when image tags are bumped (#6035)
• Shell script linting with shellcheck (#6038)
• Native ARM image builds (#6070)
• Doc samples verified against the real admission chain (#6039)
• Per-test namespace isolation and expanded canary, KEDA, and weighted InferencePool e2e coverage (#5832, #5849, #6018, #5886)
• Timeouts on all GitHub Actions jobs (#5933)

🔍 Release Notes​

For the complete list of all merged pull requests, bug fixes, and known issues, visit the GitHub release pages:

• v0.21.0
• v0.21.0-rc1
• v0.21.0-rc0

🙏 Acknowledgments​

We extend our gratitude to all 39 contributors who made this release possible, including 15 first-time contributors. Your efforts continue to drive the advancement of KServe as a leading platform for serving machine learning models.

• New Contributors: @aneeshkp, @ugiordan, @unichronic, @iasthc, @harshhh817, @mgonzalezo, @0-Zaid-0, @anxkhn, @noah4477, @mateenali66, @xrwang8, @Wojciech-Rebisz, @ThatsMrTalbot, @NoOverflow, @Sunidhi-Gaonkar1 — welcome and thank you!
• Core Contributors: The KServe maintainers and regular contributors
• Community: Everyone who reported issues, provided feedback, and tested features

🤝 Join the Community​

We invite you to explore the new features in KServe v0.21 and contribute to the ongoing development of the project:

• Visit our Website or GitHub
• Join the Slack (#kserve)
• Attend our community meeting by subscribing to the KServe calendar.
• View our community github repository to learn how to make contributions. We are excited to work with you to make KServe better and promote its adoption!

Happy serving!

The KServe team is committed to making machine learning model serving simple, scalable, and standardized. Thank you for being part of our community!

Agents write more of the code now, but systems still have to fit together properly. Protobuf contracts and Buf checks keep APIs aligned and standardized.

If you follow the Apache Kafka ecosystem, you may have already heard the news: Cruise Control, the load-balancing engine originally developed at LinkedIn, has been donated to the Linux Foundation and will continue its development under a new, community-driven governance model.

This is good news for Strimzi, and in this short post we want to explain why.

Cruise Control and Strimzi: a long story

Strimzi has relied on Cruise Control for cluster rebalancing since the very early days. Native Cruise Control support landed in Strimzi 0.18.0, bringing the KafkaRebalance custom resource and a fully operator-managed Cruise Control deployment to life, enabling cluster rebalancing by balancing partition replicas and leaders across brokers using goals and constraints.

Since then, the integration has grown steadily:

• Intra-broker (JBOD) rebalancing: moving data between disks on the same broker when you add or remove volumes from a JBOD storage configuration.
• Auto-rebalancing on scaling: as of Strimzi 0.44.0, Strimzi can automatically trigger a Cruise Control rebalance whenever the cluster is scaled up or down, keeping things balanced without any manual intervention.
• Auto-rebalancing on imbalance: leveraging Cruise Control’s anomaly detection, this upcoming feature will allow Strimzi to keep your Kafka clusters continuously balanced without requiring any human action at all.

Why the Linux Foundation move matters

Until recently, Cruise Control development had been driven almost exclusively by a small number of contributors, with limited bandwidth and uncertain long-term direction. Moving the project under the Linux Foundation establishes a transparent governance structure, broadens the contributor base, and gives the project a sustainable home.

The new Technical Steering Committee (TSC) includes contributors from several organizations in the Kafka ecosystem, and notably some Strimzi contributors are among them: Kyle Liberti (Strimzi contributor and Cruise Control co-chair), Mickael Maison (Strimzi Metrics Reporter component owner) and Paolo Patierno (Strimzi maintainer). This means the Strimzi community has a direct voice in the future direction of Cruise Control, and can help shape the features that matter most for Kubernetes-based Kafka deployments.

What changes for Strimzi users

Cruise Control 3.1.0 is the first release driven by the new community under the Linux Foundation, and it already delivers meaningful improvements:

• support for Apache Kafka 4.3.1
• a major Jetty upgrade
• a set of security fixes addressing known vulnerabilities in Netty, Log4j, commons-io, and Jackson

Beyond the fixes, two structural changes are worth noting. The license has changed from BSD 2-Clause to Apache License 2.0, and the release artifacts are now published to Maven Central rather than LinkedIn’s JFrog Artifactory, under the new coordinates:

io.cruise-control:cruise-control
io.cruise-control:cruise-control-core
io.cruise-control:cruise-control-metrics-reporter

Strimzi is already tracking these changes: the upcoming Strimzi 1.3.1 and 1.4.0 releases will be the first to ship with Cruise Control 3.1.0 and its new artifact coordinates.

If you are running Strimzi today, you do not need to take any action right now. When upgrading to a Strimzi version using the Cruise Control 3.1.0 or higher the Strimzi operator should handle the update smoothly, but keep an eye on the Strimzi release notes for those versions for any actions you need to take.

Looking ahead

The donation of Cruise Control to the Linux Foundation is a positive signal for the entire Kafka-on-Kubernetes ecosystem. A well-governed, community-driven Cruise Control means more confidence in the dependency, more contributors addressing edge cases, and a better foundation for the advanced automation features Strimzi is building on top of it.

We are looking forward to collaborating with the broader Cruise Control community and to continuing to deliver a seamless, automated cluster management experience for Strimzi users.

As part of an ongoing series, the Developer Experience SIG interviews organizations about their real-world OpenTelemetry deployments to share practical lessons with the broader community. This post features Keycloak, a CNCF project, and looks at the journey from the perspective of project maintainers who add native OpenTelemetry support to the software they ship.

If you have ever needed an Iden

Knative v1.24 Release Announcement