CNCF Releases

YAML has been the standard way to write Kubernetes manifests for years. Every example, tutorial, and configuration file you come across is written in it. The problem isn't that YAML is a bad format. It's that YAML gives you a lot of choices, and not all of them are equally good for writing Kubernetes manifests. Some features make files harder to read, some are easy to misuse and others can lead to surprising behavior.

The interesting part is that Kubernetes doesn't actually need most o

Protobuf has ten integer types. Use int64. In Go, even the slow cases decode a thousand integers in a few microseconds, so the choice almost never matters.

OpenTelemetry metrics are designed to be safe to use in production. One part of that safety is the cardinality limit in the metrics SDK. The limit protects your process from unbounded memory growth when a metric receives too many unique attribute combinations.

That protection is useful, but it has a consequence many users do not expect: when a metric stream overflows, the total value remains correct, while queries that filter or group by attributes can undercount. This

Upgrading

Before upgrading refer to the migration guide for a complete list of changes.

All resolved issues

Security fixes

• #234 Normalize URIs in PathConfigMatcher to ensure mutated URIs share the same cache key as their canonical form client

Bugs

To download the release go to Keycloak downloads.

Upgrading

Before upgrading refer to the migration guide for a complete list of changes.

All resolved issues

Security fixes

• #49429 [CVE-2026-9793] JWE request object bypasses requestObjectSignatureAl

The Kubernetes SIG Network community is thrilled to share the release of Gateway API v1.6.0, which was released on June 30th of this year!

Gateway API has become the standard for modern, role-oriented, and expressive service networking in Kubernetes. In previous releases, Gateway API established a production-grade foundation for HTTP and TLS lay

We are excited to introduce drift ignore rules for Flux Kustomizations, a long-requested capability that lets you tell Flux to leave specific fields alone during drift detection and correction, while continuing to reconcile everything else.

One of th

wasmCloud 2.6 ships trigger services, host component plugins (WebAssembly components that extend the host), warm-instance pooling, consumer groups by default for NATS messaging, and per-component DNS allowlists.

Guest post by David Farah and Sven Jonscher, creators of Cotal .
We build Cotal, the open standard for AI agents to work together in one shared space. Our first multi-agent prototypes taught us where the real difficulty lives: in the wiring between agents. Who can talk to whom? What happens to a message when the recipient is busy? Who is even online right now?
Most frameworks settle those questions for you by imposing a single shape, usually a controller handing out work to workers that never ta

As we get closer to the release date for Kubernetes v1.37, the project develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. This blog outlines some of the planned changes for the Kubernetes v1.37 release that the release team feels you should be aware of for the continued maintenance of your Kubernetes environment and keeping up to date with the latest changes. The information below reflects the current status of the v1.37 r

Knative v1.23 Release Announcement

v1.9.0

Released on: Thu Jul 30 12:39:35 2026 +0000

• [PR #18567][kubevirt-bot] Bug fix: Expose VFIO cdev devices (/dev/vfio/devices/vfioN) to virt-launcher containers, fixing IOMMUFD-based GPU passthrough on ARM64 systems with SMMUv3.
• [PR #18532][kubevirt-bot] virt-launcher pods get cleaned up along with migration objects
• [PR #18539][kubevirt-bot] BugFix: hotplug volume detach deadlock when VMIs exchange RWO PVCs between attachment pods.
• [PR #18507][kubevirt-bot] Fixed missing validation for network interface bindings: interfaces with no binding or multiple bindings are now rejected at admission instead of failing silently at runtime.
• [PR #18466][kubevirt-bot] Network binding plugin sidecars can now infer their registration name using the NETWORK_BINDING_PLUGIN_NAME environment variable.
• [PR #18438][iholder101] Bug fix: VMs with NFS-backed storage are no longer incorrectly restarted during temporary NFS unavailability (e.g., NFS server failover).
• [PR #18446][kubevirt-bot] vep-10: move GPUsWithDRA and HostDevicesWithDRA to beta
• [PR #18445][kubevirt-bot] Add E2E tests support for DRA feature.
• [PR #18433][kubevirt-bot] Network conformance tests can now be run with a custom network binding plugin via the --primary-network-binding-plugin test flag.
• [PR #18412][kubevirt-bot] Set explicit SMMUv3 address capability defaults for Grace I/O Virtualization.
• [PR #18404][kubevirt-bot] The kubevirt_vmi_guest_os_panic_total metric is now emitted
• [PR #18405][kubevirt-bot] BugFix: Importer pod rejected by ValidatingAdmissionPolicy ‘kubevirt-plugin-sidecar-subpath-policy’
• [PR #18331][kubevirt-bot] Live migration data stream can be compressed now by using .spec.experimental.compression field in MigrationPolicy applied to a set of VMs. Currently only “zstd” algorithm is supported.
• [PR #18356][kubevirt-bot] Support inferFromVolume for VolumeSnapshot-backed DataVolumes, DataVolumeTemplates, and DataSources
• [PR #18343][kubevirt-bot] Bug fix: PreferredPanicDeviceModel now creates a panic device when none exist on the VMI, instead of silently ignoring the preference.
• [PR #18299][lyarwood] Bug fix: Increase IOMMUFD socket accept timeout from 60 seconds to 15 minutes to prevent silent fallback to legacy VFIO on environments with slow container image pulls.
• [PR #18095][fra2404] Add PortRanges support for masquerade interfaces (Alpha, guarded by PortRangesSpec feature gate)
• [PR #18011][fanzhangio] When the alpha GraceIOVirtualization feature gate is enabled, KubeVirt now configures NVIDIA Grace GPU passthrough with SMMUv3/IOMMUFD, ACPI Generic Initiator NUMA topology, NUMA distance mapping, and automatic PCI 64-bit hole sizing.
• [PR #18048][alancaldelas] WorkloadEncryptionSEV graduated from Alpha -> Beta
• [PR #18216][awels] BugFix: Fixes bug in cancelling of decentralized live migration not removing finalizer from migration resource
• [PR #18004][avlitman] Add VMNonRecoverableOSPanic alert
• [PR #18065][0xFelix] The Template feature gate has been graduated to Beta and is now enabled by default. Virt-template components are deployed automatically unless explicitly disabled.
• [PR #18116][brianmcarey] Build KubeVirt with go 1.26.4
• [PR #18161][machadovilaca] Add kubevirt_vmi_gpu_info metric to correlate GPU UUIDs with VMIs
• [PR #17837][keinsword] Nebius is listed as an adopter of KubeVirt since 2023.
• [PR #17846][ksimon1] virt-api: add X-Content-Type-Options: nosniff security header
• [PR #18101][dsanatar] Remove ephemeral hotplug volume metric and alert
• [PR #18082][alromeros] Expose FS freeze status early in VirtualMachineBackup
• [PR #18252][kubevirt-bot] Updated virt-template to v0.2.2
• [PR #18198][orenc1] VEP-160: Graduate OptOutRoleAggregation feature gate to Beta
• [PR #18112][xpivarc] SRIOV vGPU now supports display
• [PR #17199][lyarwood] Add CrossArchitectureVirtualization Alpha feature gate enabling cross-architecture VM execution. When enabled, VMs can run on nodes with a different architecture (e.g. ARM64 guest on AMD64 host) via QEMU TCG software emulation. The scheduler prefers native-arch nodes and falls back to cross-arch emulation when unavailable. The feature gate is designed to accommodate future hardware-accelerated backends (e.g. SAE) under the same gate.
• [PR #18102][iholder101] Plugin sidecars can now mutate the libvirt domain XML via gRPC domain hooks, enabling arbitrary domain customization through sidecar containers injected alongside virt-launcher pods.
• [PR #18103][orelmisan] Fix infinite migration target pod loop caused by stale VMI MigrationState when the target pod fails after preparation.
• [PR #18141][machadovilaca] Skip PrometheusRule generation when VMStatsCollector is enabled
• [PR #17818][fossedihelm] MigrationPriorityQueue feature gate has been promoted to GA.
• [PR #18228][kubevirt-bot] Updated common-instancetypes bundles to v1.7.0
• [PR #17540][orenc1] Add RBAC role aggregation upgrade e2e test
• [PR #18220][0xFelix] Updated virt-template to v0.2.1
• [PR #17770][jschintag] IBM Secure Execution is now generally available
• [PR #18188][machadovilaca] Add VMStatsHandler with HTTP endpoint for VM stats collection
• [PR #18032][0xFelix] Add VirtualMachineTemplate as a source kind for OCI export via VMExport.
• [PR #18001][iholder101] Add node lifecycle hooks for the plugin framework, enabling plugins to run callbacks at VM start/stop and migration phases (alpha, behind Plugins feature gate).
• [PR #17998][fanzhangio] VMIs requesting NVIDIA Grace GPU PCI passthrough resources are now validated by admission. Such VMIs require the GraceIOVirtualization, PCINUMAAwareTopology, and IOMMUFD feature gates, arm64 architecture, the virt machine type, and dedicated CPU placement; ambiguous NVIDIA wildcard PCI selectors are rejected when GraceIOVirtualization is enabled.
• [PR #17886][fanzhangio] PCI NUMA-aware placement (PCINUMAAwareTopology) now fails with an error for VMIs using numa.guestMappingPassthrough when a host device cannot be placed on a NUMA-aligned PCIe topology. Non-passthrough VMIs retain the existing fallback behavior.
• [PR #17544][Aseeef] Adds a robust migration stall detection mechanism that triggers post-copy or stop-and-copy opportunistically at local minimas reducing both total migration duration and downtime.
• [PR #18105][keinsword] BugFix: fixed hotplug mount resolution when multiple attachment pods exist for volumes from the same underlying device, and prevented virt-handler from reusing one hotplug volume’s source pod UID for later volumes in the same sync loop.
• [PR #17664][ksimon1] feat: Add annotation-based GuestAgentPing probe pausing
• [PR #18129][iholder101] Fixed missing RBAC rules for plugin.kubevirt.io in the kubevirt.io:view ClusterRole, restoring cluster-reader access to Plugin resources.
• [PR #17910][jcanocan] virtctl: updated virt-template subcomands to v1beta1
• [PR #18143][akalenyu] BugFix: Storage Live Migration Filesystem-to-Block fails with libvirt error Code 84 pre-creation of storage target for incremental storage migration of disk is not supported
• [PR #18090][Acedus] VirtualMachineBackups now consist of only 3 conditions: Progressing, Complete, and Failed, with matching reasons to replace the now removed Initializing, ExportInitiated, ExportReady, Aborting and Done.
• [PR #17861][mhenriks] VEP 250: Add serviceAccountName to VirtualMachineInstance Spec
• [PR #17960][dsanatar] Fix the virtctl image-upload command token expiry issue by refreshing the token.
• [PR #17976][ethan-gallant] virt-operator: customizeComponents.Patches now apply to the install-strategy job (match it with resourceType “Job” and resourceName “virt-operator-install-strategy”). The job’s generated name changed from “kubevirt--job" to "virt-operator-install-strategy-".
• [PR #17505][alromeros] Enable live migration for VMs with SCSI persistent reservations
• [PR #17956][fossedihelm] Add IOMMUFD device plugin to virt-handler behind the IOMMUFD Alpha feature gate. When enabled, virt-controller requests devices.kubevirt.io/iommufd for every launcher pod. Nodes without /dev/iommu (kernel <6.2) will report unhealthy devices, making pods unschedulable there. This feature also emits domain-level and uses virDomainFDAssociate, which require a libvirt/QEMU stack that supports fdgroup-based IOMMUFD, currently libvirt >= 12.2. Enable this gate only on clusters where target nodes and virt-launcher images provide compatible kernel, libvirt, and QEMU support.
• [PR #17993][Acedus] BugFix: Live migration with CBT and RWO backend storage now correctly retains checkpoints post-migration
• [PR #17623][MarSik] The RebootPolicy feature was graduated to Beta status.
• [PR #17422][jean-edouard] Libvirt bumped to v11.10
• [PR #17891][vladikr] Add foundation types for Grace IO Virtualization: GraceIOVirtualization feature gate, IOMMU domain XML schema, HostDevice IOMMU/ACPI extensions, and NUMACell pointer change for zero-memory cells.
• [PR #17897][iholder101] Added CEL-based domain hook evaluation for the structured plugin system (alpha, behind Plugins feature gate).
• [PR #17551][jschintag] Allow configuring launch security via preference.
• [PR #17923][alaypatel07] vep-10: add dra packages to lint configuration and fix linter errors
• [PR #16990][lyarwood] Migrated port-forwarding and remote exec from hardcoded SPDY to WebSocket-primary with SPDY fallback, fixing intermittent stream creation failures against Kubernetes 1.31+ clusters where the PortForwardWebsockets feature gate is enabled.
• [PR #18076][jcanocan] Updated virt-template to v0.2.0
• [PR #18052][vidit-bhat] Fixed a build failure in pkg/hypervisor on GOOS/GOARCH combinations outside linux/{amd64,arm64,s390x}, which previously prevented downstream packagers from cross-building virtctl on architectures such as riscv64, ppc64le, and 386. On those architectures, common.SchedSetScheduler now returns the new sentinel error common.ErrUnsupportedRTScheduling.
• [PR #17996][ksimon1] virt-operator: fix DeploymentInProgress after toggling optional feature gates
• [PR #17984][Acedus] Incremental backups now distinguish discarded blocks from data changes by merging base:allocation and qemu:dirty-bitmap contexts, reducing transfer size by skipping dirty extents that read as zero.
• [PR #17845][orelmisan] Network binding plugins can now read the interface MTU from the network-info downward API annotation.
• [PR #17809][michalskrivanek] cgroup v1 support is deprecated now, with removal planned for the next release
• [PR #17797][alaypatel07] vep-10: migrate away from k8sv1.PodResourceClaim to kubevirts own type
• [PR #17790][iholder101] Introduce the Plugin CRD (plugin.kubevirt.io/v1alpha1) behind the Plugins feature gate (Alpha), enabling declarative VM extension via domain hooks, node hooks, and admission references (VEP-190).
• [PR #17947][Barakmor1] Use –expand-cpu-features and –supported-cpu-features in node-labeller for
• [PR #17944][0xFelix] Add OCI artifact export support behind the OCIExport feature gate. Use virtctl vmexport download --format=oci to export a VM as an OCI image layout TAR.
• [PR #17989][UdayYendva] Bump github.com/moby/spdystream from v0.5.0 to v0.5.1 to address CVE-2026-35469 (GHSA-pc3f-x583-g7j2).
• [PR #16599][dasionov] Promote VideoConfig FG to General Availability
• [PR #17661][oshoval] Add alpha support for DRA SR-IOV networks via spec.networks[].resourceClaim behind NetworkDevicesWithDRA, with webhook validation and virt-launcher hostdev generation from DRA metadata.
• [PR #16932][dsanatar] Graduate HotplugVolumes and DeclarativeHotplugVolumes to Beta
• [PR #17883][yaroslavborbat] Fixed a target-side hotplug mount leak that could remain after failed or canceled live migration.
• [PR #17836][iholder101] Add kubevirt_vmi_guest_os_panic_total Prometheus counter metric to track guest OS panic events per VMI, with labels for panic type and hyper-v bugcheck code
• [PR #17959][mhenriks] Fix symlink traversal in VMExport dir handler
• [PR #16674][akalenyu] Persistent Reservation GA
• [PR #17550][machadovilaca] Add GetVMStats unified gRPC RPC for monitoring data collection
• [PR #17757][Acedus] Fix: Handle disks with qcow2 overlay in migration related code
• [PR #17825][akalenyu] BugFix: endless cycle of attachment pod deletion/creation
• [PR #17699][dsanatar] only populate vmexport with a vm manifest if source is not pvc
• [PR #17798][SamAlber] Fixed a gRPC connection leak in virt-handler’s GetLauncherClient that caused unbounded memory growth, socket accumulation, and goroutine leaks when multiple controllers raced to create connections for the same VMI.
• [PR #17644][kubevirt-bot] Updated virt-template to v0.1.8
• [PR #17272][ksimon1] Feat: Record K8s event on VMI when GuestAgentPing probe fails
• [PR #17536][nirdothan] Fixed VMI status reporting the pod’s IPv6 address instead of the guest’s when using bridge binding on a network with IPv6 IPAM.
• [PR #17405][iholder101] Enable all Beta feature gates by default. Users can opt out of individual Beta features by adding them to spec.configuration.developerConfiguration.disabledFeatureGates in the KubeVirt CR. A feature gate report with all non-GA feature gates will be added to each release’s artifacts.
• [PR #17805][lyarwood] Fixed multi-device VFIO passthrough VMs failing to start with “cannot limit locked memory” by scaling virt-handler’s memlock rlimit to account for per-device memory locking, matching libvirt’s calculation introduced in v8.7.0.
• [PR #17496][fanzhangio] The –additional-launcher-annotations-sync and –additional-launcher-labels-sync flags now support prefix wildcards via a trailing ‘’ suffix (e.g. ‘vendor.io/’), allowing all matching labels/annotations to be propagated from VM template to VMI and virt-launcher pod without enumerating each key individually.
• [PR #17029][mhenriks] Fix PCI address stability across upgrades with v3 hotplug port topology
• [PR #17475][samt-ai] Fix race condition in VM force restart where the pod could remain stuck in Terminating state for the full terminationGracePeriodSeconds instead of terminating promptly. The VMI’s terminationGracePeriodSeconds is now patched before triggering the restart, ensuring the short grace period is always honored.
• [PR #17031][suPer8Hu] N/A
• [PR #17548][Barakmor1] Add container-level SecurityContext to virt-exportproxy and virt-synchronization-controller deployments
• [PR #17755][dasionov] fix: cross-namespace live migration now works on IPv6 clusters
• [PR #17490][oshoval] Fixed virt-controller DRA claim rendering for GPU/HostDevice resources by preserving per-device claim/request tuples (including shared claim names with different requests).
• [PR #17527][mhenriks] Fix VM with PCI hostdev failing to restart after hotplug block volume
• [PR #17573][lyarwood] Bug fix: virt-operator error messages no longer dump entire resource structs via %+v, preventing the KubeVirt CR from exceeding the etcd 3MB object size limit when resource creation fails
• [PR #17691][vishnuchalla] NA
• [PR #17049][frenzyfriday] Graduates LiveUpdateNADRef feature gate
• [PR #16697][dippydocus] Fixed DHCP failure after live migration followed by guest reboot when using bridge binding without a specified MAC address.
• [PR #17599][Sreeja1725] Add release 1.8 perf-scale benchmarks data
• [PR #17576][ema-aka-young] Bugfix: Added enum validation for the targetReadinessPolicy field for restore resources.
• [PR #17571][0xFelix] Fixed virt-api truncating deep subresources (vnc/screenshot, sev/*, evacuate/cancel) when constructing SubjectAccessReviews, causing authorization checks against incorrect subresource names.
• [PR #17015][Ronilerr] Adding missing metrics, recording rules and alerts for virt components
• [PR #17130][sradco] Fix VirtualMachineStuckOnNode and VMCannotBeEvicted alerts failing during live migration due to duplicate kubevirt_vmi_info series
• [PR #17488][alaypatel07] change /var/lib/kubelet mount from Bidirectional to HostToContainer
• [PR #17028][alaypatel07] VEP-10: bug fixes for DRA Devices to align kubevirt implementation to KEP-5304
• [PR #17557][dshchedr] Fixed GuestPanicked event details for non-root virt-launcher
• [PR #17235][tiraboschi] Fix: GuestAgentPing liveness/readiness probes no longer cause Kubernetes to restart the virt-launcher pod when the guest agent is temporarily unreachable for a non-fault reason; suppression covers live migration (both pre-copy target and post-copy source) and any intentional or transient VM pause such as user pause, snapshot, save, or dump.
• [PR #17497][sbiradar10] Remediate CVE-2026-33186 by bumping grpc to 1.79.3
• [PR #17295][avlitman] new metric kubevirt_vmi_sync_total added in order to track number of times a controller has synced a VMI.
• [PR #17512][0xFelix] Remove vnc/screenshot from kubevirt.io:edit
• [PR #16882][alromeros] Snapshot: Add PartialSnapshot indication for excluded volumes
• [PR #17398][dasionov] Bug-fix: virt-handler now detects when domain-notify.sock is deleted and automatically restarts the notify server.
• [PR #16890][xpivarc] KubeVirt’s PCI device plugin now supports passing of pre-setup VF (vGPU)
• [PR #17425][shubham-pampattiwar] fix: VirtualMachineBackup printer columns (Type, CheckpointName) now display correctly in kubectl output
• [PR #17407][dsanatar] preserve annotation for restore pvc
• [PR #17297][Acedus] fix: correctly handle source resolution for disks with a qcow2 overlay, preventing incorrect disk expansion and wrong cache/IO mode detection.
• [PR #16742][keinsword] Improve Unmount() cleanup by processing all entries and preserving failed paths for retry.
• [PR #16853][dankenigsberg] VMs with backend storage volume use and report the volume name as persistent-state-for-this-vm rather than trying to embed the vm name in the volume name.
• [PR #17315][frenzyfriday] Fixes bug in Live NAD Ref Update feature where a VM with no interfaces/networks is unable to start when LiveNADRefUpdate FG is enabled.
• [PR #17102][Barakmor1] Bug fix: sync-controller healthz server and virt-exportserver now respect TLSConfiguration from the KubeVirt CR.
• [PR #16730][alromeros] GA VMExport feature gate
• [PR #17139][dsanatar] fix hotplug volume status being stuck in Detaching phase
• [PR #16514][varunrsekar] The PanicDevices feature has graduated to GA and no longer requires the associated feature gate to be enabled.
• [PR #17251][iholder101] Fixed VM startup failure under software emulation when /dev/kvm is absent, caused by cgroup device rules not accounting for the emulation case.
• [PR #17042][awels] Fixed migration not reporting succeeded when doing compute migration after decentralized live migration
• [PR #16842][ksimon1] OpenApi V3 paths for subresources.kubevirt.io is present
• [PR #17065][avlitman] multiple recording rules are deprecated in favor of new names, in order to comply with the recording rules naming conventions. kubevirt_vm_created_total recording rule and kubevirt_vm_created_by_pod_total metric are deprecated completely
• [PR #17285][dhiller] Maintenance: revert bazel server network change monitoring
• [PR #17071][samt-ai] Make –vnc-path optional for –vnc-type, allowing VNC viewer binary lookup from $PATH automatically
• [PR #17109][dasionov] bug-fix: restart virt-handler’s domain-notify server on unexpected exit.
• [PR #16885][akalenyu] Maintenance: make the bazel server container reload on network changes
• [PR #16675][csomani1] vGPU (mdev) now supports live migration (limited to one device)
• [PR #17106][iholder101] Allow multifd (parallel migration threads) with post-copy migration
• [PR #15298][Acedus] bugfix: Online snapshots now correctly include live-changes applied to a VM.
• [PR #16737][keinsword] Fix virt-handler hotplug mount target record growth by preventing duplicate TargetFile entries.
• [PR #17041][orelmisan] Fixed an infinite VMI status update loop between virt-controller and virt-handler that occurred when the VMI spec listed the primary network interface after a secondary one.
• [PR #16783][dasionov] Fixed SMBIOS system information not being visible inside ARM64 guest VMs
• [PR #16556][alaypatel07] VEP-10: Update DRA devices implementation to read from metadata file instead of VMI status
• [PR #16930][Acedus] Introduce incremental backup pull mode support
• [PR #16927][harshitgupta1337] Add MSHV backend for multi-hypervisor support interfaces.
• [PR #16817][ShellyKa13] Handle migration with CBT and backup checkpoints
• [PR #16632][mresvanis] Add PCIe NUMA-aware topology placement for GPU and host devices behind the PCINUMAAwareTopology feature gate (Alpha). When enabled, devices are automatically placed on PCIe expander buses matching their NUMA affinity for improved performance.
• [PR #16884][EdDev] ‘virtctl expose’ creates now services with an ownerReference pointing to the exposed resource.
• [PR #16412][frenzyfriday] Allows the user to update the NAD reference (networkName) of a network on a running VM through Live Migration.
• [PR #16746][Barakmor1] Expose Memory Overhead on VMI Status behind VmiMemoryOverheadReport feature gate
• [PR #16350][orenc1] Add a new config option to opt-out RBAC aggregation
• [PR #16806][machadovilaca] Use defined deployment number of replicas as base to fire low count alerts
• [PR #16604][dsanatar] Graduate ExpandDisk Feature Gate
• [PR #15958][Aseeef] Added support for attestation on the Intel TDX Confidential Computing Platform
• [PR #16877][ShellyKa13] Handle migration during backup according to migration priority
• [PR #16871][awels] BugFix: VMs requiring enlightenment are now able to be live migrated after a decentralized live migration
• [PR #16865][machadovilaca] Subtract non-schedulable nodes from kubevirt_allocatable_nodes
• [PR #16846][Aneesh-Hegde] fix VMExport failure with long PVC names
• [PR #16399][Acedus] fix: Prevent stale VMI backup status update when reusing backup names
• [PR #16786][alromeros] Allow disabling Velero hooks in virt-launcher via Annotation

Kubernetes has long been the default platform for distributed workloads, and writing your own controller for it is now a matter of a few hours. The common path — Golang, using kubebuilder on top of controller-runtime — gives you a project scaffold, types, and a reconciler. For typ

Author:

• Wenbo Qi(Gaius), Dragonfly Maintainer

Dragonfly speeds up file and container image distribution with P2P technology, but a full installation deploys several components and their dependencies. Besides the Scheduler, Seed Client and Client that actually move the data, you need a Manager for dynamic configuration, with MySQL and Redis behind it. That is a reasonable footprint for a platform team running Dragonfly across a fleet of clusters. It is a lot to ask of someone who has one cluster and whose only problem is that image pulls are overloading the registry.

Dragonfly now supports a lightweight deployment that removes the Manager, and with it MySQL and Redis. The Scheduler is the only coordination component left. You install three components with one Helm command. This post explains how it works and walks through trying it on a kind cluster.

What the Manager was doing for you​

The Manager is Dragonfly's control plane. It hosts the web console, exposes an Open API for integrations such as registry-triggered preheating, manages relationships across multiple P2P clusters, and distributes dynamic configuration to Schedulers and Clients. It stores its data in MySQL and uses Redis for caching and for distributing asynchronous jobs such as preheating.

None of those resources are wasted at fleet scale. But if you operate a single cluster and only want the P2P distribution, the piece you actually depend on is the last one, dynamic configuration. And the dynamic configuration for one cluster is, in practice, a page of YAML with scheduling limits, blocklists, and a way for Clients to find Schedulers. Running a database to serve a page of YAML is a hard sell, so we decoupled the Scheduler and Client from the Manager. If no Manager address is configured, they run without one.

How it works​

The lightweight deployment replaces the Manager with two mechanisms that Kubernetes already gives you, a ConfigMap and a headless Service.

Dynamic configuration from a ConfigMap. When manager.addr is not set, the Scheduler and Client load dynamic configuration from a local dynconfig.yaml file (/etc/dragonfly/dynconfig.yaml by default), which the Helm chart mounts as a ConfigMap. If the file does not exist, it is generated with default values on startup, so you do not need to write one by hand to get going. The Scheduler's dynconfig.yaml carries cluster-level scheduling parameters:

# seedPeerClusterConfig is the seed peer cluster configuration.
seedPeerClusterConfig:
  # loadLimit is the seed peer concurrent upload limit.
  loadLimit: 2000

# schedulerClusterConfig is the scheduler cluster configuration.
schedulerClusterConfig:
  # candidateParentLimit is the candidate parent limit for scheduling.
  candidateParentLimit: 3
  # filterParentLimit is the filter parent limit for scheduling.
  filterParentLimit: 15

# schedulerClusterClientConfig is the client configuration.
schedulerClusterClientConfig:
  # loadLimit is the peer concurrent upload limit.
  loadLimit: 200

The Scheduler and Client re-read the file periodically (every minute by default, controlled by refreshInterval), so an edit to the ConfigMap reaches running Pods within one refresh interval without restarts. The same knobs are exposed as Helm values (scheduler.dynconfig, seedClient.dynconfig and client.dynconfig), which keeps the whole configuration declarative and friendly to GitOps.

Scheduler discovery through a headless Service. In a Manager-based deployment, Clients ask the Manager for the list of available Schedulers. In the lightweight deployment, the Client's dynconfig.yaml points at the Scheduler's headless Service instead:

scheduler:
  # addr is the address of the scheduler headless service with port,
  # resolved via DNS to discover all scheduler addresses.
  addr: dragonfly-scheduler.dragonfly-system.svc.cluster.local:8002

The Client's dfdaemon process resolves that name via DNS to get the full set of Scheduler Pod IPs, health-checks each one, and filters out anything unhealthy. Scale the Scheduler StatefulSet up or down and Clients pick up the change through DNS, just as they would have from the Manager.

DNS is not the only option. If your Schedulers sit at fixed addresses, for example outside the cluster, you can list them directly with scheduler.addrs, and a non-empty list takes precedence over DNS resolution:

scheduler:
  # addrs is a static list of scheduler addresses with port. When non-empty,
  # it takes precedence over addr.
  addrs:
    - 192.168.1.10:8002
    - 192.168.1.11:8002

Three components land in the cluster. The Scheduler and Seed Client each run as a StatefulSet, with the Seed Client acting as the root peer of the P2P network that fetches from the origin, and the Client runs as a DaemonSet on every node, where dfinit rewrites the containerd configuration so that image pulls flow through the local Client. There is no database to back up and no migration to run on upgrade. The only state anywhere in the system is the piece cache on disk, which can always be rebuilt from the origin.

Choosing a deployment model​

Dragonfly supports three deployment models, and the available features depend on which components are deployed. The Helm chart disables the Manager, MySQL and Redis by default, so lightweight is the standard starting point.

Lightweight deployment. Deploy only the Scheduler, Seed Client and Client. This is the recommended model for most scenarios that only need the P2P distribution capabilities, such as small Kubernetes clusters, edge environments, or CI systems. Task distribution and preheating via dfctl both work here.

Lightweight deployment with Redis. Deploy Redis in addition to the lightweight deployment and configure the Scheduler's database.redis.addrs to use it. The persistent task and persistent cache task features store their metadata in Redis, so they become available in this model. With the Helm chart, set redis.enable: true to deploy Redis, or point externalRedis.addrs at a Redis you already run.

Deployment with Manager. Deploy the Manager, with MySQL and Redis behind it, in addition to the other three components. The Manager provides the control plane, which covers the web console, Open API, preheating jobs and multi-cluster management.

The feature matrix from the deployment models documentation summarizes the differences:

Trying it on a kind cluster​

Create a multi-node kind cluster:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --config kind-config.yaml

Pull the Dragonfly images and load them into the cluster:

docker pull dragonflyoss/scheduler:latest
docker pull dragonflyoss/client:latest
docker pull dragonflyoss/dfinit:latest

kind load docker-image dragonflyoss/scheduler:latest
kind load docker-image dragonflyoss/client:latest
kind load docker-image dragonflyoss/dfinit:latest

The entire Helm values file fits on one screen. Note that there is no manager, mysql or redis section, because all three are disabled by default.

scheduler:
  image:
    repository: dragonflyoss/scheduler
    tag: latest
  metrics:
    enable: true

seedClient:
  image:
    repository: dragonflyoss/client
    tag: latest
  metrics:
    enable: true

client:
  image:
    repository: dragonflyoss/client
    tag: latest
  metrics:
    enable: true
  dfinit:
    enable: true
    image:
      repository: dragonflyoss/dfinit
      tag: latest
    config:
      containerRuntime:
        containerd:
          configPath: /etc/containerd/config.toml
          proxyAllRegistries: true

Install:

helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
helm install --wait --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly -f charts-config.yaml

Four Pods come up, a Client on each worker node plus one Scheduler and one Seed Client.

$ kubectl get po -n dragonfly-system
NAME                      READY   STATUS    RESTARTS   AGE
dragonfly-client-dhqfc    1/1     Running   0          3m
dragonfly-client-h58x6    1/1     Running   0          3m
dragonfly-scheduler-0     1/1     Running   0          3m
dragonfly-seed-client-0   1/1     Running   0          3m

Pull an image on a worker node and it goes through Dragonfly:

docker exec -i kind-worker /usr/local/bin/crictl pull alpine:3.19

To confirm that Dragonfly served the pull, find the task in the local Client's logs:

# Find the client pod on kind-worker.
export POD_NAME=$(kubectl get pods --namespace dragonfly-system -l "app=dragonfly,release=dragonfly,component=client" -o=jsonpath='{.items[?(@.spec.nodeName=="kind-worker")].metadata.name}' | head -n 1)

# Find the task id for the alpine image.
export TASK_ID=$(kubectl -n dragonfly-system exec ${POD_NAME} -- sh -c "grep -hoP 'library/alpine.*task_id=\"\K[^\"]+' /var/log/dragonfly/dfdaemon/* | head -n 1")

# Check that the download succeeded through Dragonfly.
kubectl -n dragonfly-system exec -it ${POD_NAME} -- sh -c "grep ${TASK_ID} /var/log/dragonfly/dfdaemon/* | grep 'download task succeeded'"

Preheating still works​

Preheating, which warms the P2P cache ahead of a rollout, sounds like a control-plane feature, but it does not actually require the Manager. The dfctl CLI calls the Scheduler's gRPC interface directly and triggers Seed Clients or Clients to download the content:

dfctl task preheat oci://docker.io/library/alpine:3.19 \
  --scheduler-endpoint http://dragonfly-scheduler.dragonfly-system.svc.cluster.local:8002 \
  --scope all_seed_peers

It handles both images (oci:// URLs, with --username/--password for private registries) and plain files (https:// URLs), and --scope controls whether the preheat goes to a single Seed Client, all Seed Clients (the default), or every Client in the cluster. Since dfctl ships in the Client image, running a preheat inside Kubernetes is a kubectl exec away. See the preheat guide for details.

Injecting Dragonfly into application Pods​

Registry mirroring through containerd covers image pulls, but applications download other things too, such as models, datasets and build artifacts. For those, Dragonfly provides dragonfly-injector, a Kubernetes mutating admission webhook that injects the Dragonfly Client binaries (dfget, dfcache, dfstore, dfdaemon) and the dfdaemon socket mount into application Pods, so a Pod can download files through the P2P network without rebuilding its container image. It works with the lightweight deployment and does not require the Manager.

The injector needs cert-manager to manage TLS certificates for the webhook server. Once cert-manager is installed, enable the injector by adding an injector section to the same values file:

injector:
  enable: true
  replicas: 2
  image:
    registry: docker.io
    repository: dragonflyoss/injector
    tag: latest
  initContainerImage:
    registry: docker.io
    repository: dragonflyoss/client
    tag: latest
  certManager:
    enable: true

Opting a Pod in is a matter of annotations:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  annotations:
    dragonfly.io/inject: 'true'
    dragonfly.io/init-container-image: 'dragonflyoss/client:latest'
spec:
  containers:
    - name: app
      image: debian:stable-slim
      command: ['/bin/sh', '-c', 'sleep 3600']

On admission, the webhook adds an init container that copies the CLI tools into a shared volume and mounts the Unix socket of the dfdaemon already running on the node, so the application can call dfget and friends directly. The full walkthrough is in Using Dragonfly with webhook injection.

When you still want the Manager​

The Manager remains the right choice when you are operating Dragonfly as a platform rather than as a cluster add-on, with multiple P2P clusters managed from one place, a web console for the on-call team, and preheating driven through the Open API by an external system such as a registry. Enabling it is a Helm values change (set manager.enable, mysql.enable and redis.enable to true), and the deployment with Manager guide covers the rest. Starting lightweight does not stop you from adding the Manager later.

For everything else, three components and a ConfigMap are enough for P2P image and file distribution. The full walkthrough lives in the lightweight deployment documentation.

Links​

• Dragonfly Website: https://d7y.io/
• Dragonfly Repository: https://github.com/dragonflyoss/dragonfly
• Dragonfly Client Repository: https://github.com/dragonflyoss/client
• Dragonfly Injector Repository: https://github.com/dragonflyoss/dragonfly-injector
• Dragonfly Charts Repository: https://github.com/dragonflyoss/helm-charts
• Lightweight Deployment Documentation: https://d7y.io/docs/next/getting-started/quick-start/kubernetes/lightweight-deployment/
• Slack Channel: #dragonfly on CNCF Slack

If you’ve been running the Demo for some time, you may have seen a couple of structural changes lately, and you may even have gotten mad about things not working as expected. We feel your pain and we totally understand it. Unfortunately there was no better way to change things without breaking the existing flow.

Some of what you knew is gone. New services were added. Attribute names have changed. If you had dashboards with custom metrics, everything is broken now.

GitOps helps us declare our desired workloads, but how do we deal with and manage secrets? Additionally, as our fleet grows, we also blend artifacts and configuration from many different sources. How do we trust what we are running?

OpenBao is an open sourc

Introduction

KubeCon + CloudNativeCon India 2026 in Mumbai brought together an incredible crowd of developers, maintainers, and cloud-native enthusiasts. The energy throughout the event was inspiring, especially seeing how central AI, machine learning, and MLOps have become within the Kubernetes landscape. (Above image shows part of our volunteer team at the Kubeflow booth, ready to connect with the community.)

This year, Kubeflow had a strong presence at the CNCF Project Pavilion, serving as a central hub for anyone looking to scale AI workloads on cloud-native infrastructure. The conference highlighted the growing adoption of the ecosystem, with a noticeable increase in enterprise users and developers looking to standardize their AI platforms.

This post captures the key takeaways from the booth, the most common architecture questions from the community, and the overall presence of the Kubeflow ecosystem at the event. We hope this gives you a glimpse into the week and encourages you to get involved at future gatherings!

Featured Talks

• “Hey AI, Train Llama”: Making Kubeflow Agent-Native with MCP Speakers: Akash Jaiswal (Oracle), Abhijeet Dhumal (Red Hat) Sched Link (Open-Source Summit): View on Sched

• When Kubeflow Fights Cilium: Debugging 60% Idle GPUs in Kubernetes Speakers: Ramkumar Nagaraj, Bingi Narasimha Karthik (Adobe) Sched Link: View on Sched Blog Post: Read on CNCF Blog

Kubeflow Booth Highlights

From the moment the exhibition doors opened, the Kubeflow booth was buzzing with activity. We had conversations with a wide mix of attendees, from students looking for their first open-source project to senior platform engineers running large-scale training workloads in production.

Here are some of the main takeaways from our time at the booth:

• From Experimentation to Production: Many engineers shared how their organizations are moving away from isolated scripts and adopting Kubeflow to standardize their machine learning workflows. We heard stories about teams scaling production pipelines, managing complex deployments, and cutting down model training times.

• Growing Interest in GenAI and LLMs: A big portion of our discussions centered around large language models and RAG pipelines. Visitors wanted to know how Kubeflow handles distributed training, model serving, and efficient resource allocation for GPU workloads.

• Onboarding New Contributors: We met many developers eager to contribute to the project. We walked them through the repository structure, showed them how to find good first issues on GitHub, and invited them to join our weekly community calls. Several GSoC mentors and active contributors at the booth also shared their firsthand experiences and guidance on getting started.

Popular Questions from the Audience

How does end-to-end Kubeflow work?

Kubeflow is the foundation for AI Platforms on Kubernetes. It provides a composable, modular, and scalable community distribution that covers every stage of the AI lifecycle. We explained how teams can deploy subprojects independently or together, using Kubeflow Notebooks for development, Kubeflow Trainer for scalable LLM fine-tuning, Kubeflow Pipelines (KFP) for automated workflows, and Kubeflow Hub for managing model metadata.

How is Kubeflow different from MLflow?

They work best together. Kubeflow integrates directly with MLflow. While MLflow focuses on tracking experiments and registering models, Kubeflow acts as your cloud-native platform foundation, handling the underlying Kubernetes infrastructure, multi-tenant security, and GPU compute resources at scale.

How can Kubeflow help optimize costs when training LLMs on GPUs?

With generative AI costs top of mind, visitors wanted to know how to get more out of their hardware. We shared how Kubernetes-native scheduling, resource quotas, and multi-tenancy allow teams to safely share centralized GPU pools. Combined with scale-to-zero serving and spot instance support, organizations can avoid wasting budget on idle compute.

How does Kubeflow handle pipeline resilience and step failures?

When training jobs run for hours or days, hardware issues are inevitable. We discussed how Kubeflow Pipelines uses isolated, containerized steps with built-in retry policies and task caching. If a single pod crashes or hits a network issue, the pipeline can automatically recover and resume without losing progress from previously completed steps.

Our Experience

Volunteering at the Kubeflow booth this year was a lot of fun. Beyond the packed presentation rooms and technical sessions, what stood out was the energy of the people who stopped by. Whether we were troubleshooting a deployment setup for a platform engineer or breaking down MLOps basics for a student, every conversation reminded us why open-source collaboration is so rewarding.

We also had a great time behind the scenes, swapping stories about debugging disasters over coffee breaks while running the booth shifts. Nothing beats getting the community together in person. We hope this motivates more of you to volunteer at future events. You will not want to miss the next one! 😊

A big thank you to everyone who volunteered their time and energy to make the booth a success: Akash Jaiswal, Yash Pal, Abhijeet Dhumal, Aniket Patil, Rohit Kumar, Khushi Agrawal, Danish Siddiqui, Digvijay Yeware, Kapil Nema, Krishna Gupta, Kunal Dugar, Yash Agarwal, Milind Dethe, Sachin Jha, and Valentina Rodriguez Sosa for her ongoing guidance and support. You all made this event special!

Want to Help?

The Kubeflow community holds open meetings and is always looking for more volunteers, developers, and users to help shape the future of machine learning on Kubernetes. If you are interested in getting involved, check out the resources below. We would love to build with you!

• Visit the Kubeflow website or GitHub repositories.
• Join the Kubeflow Slack channels.
• Subscribe to the kubeflow-discuss mailing list.
• To volunteer for future events, join the #kubeflow-outreach channel on the CNCF Slack.
• Attend our weekly community meeting to see what we are currently working on.

Feel free to share your thoughts, questions, or KubeCon memories in the comments below. See you at the next event!

Here we are, at the tail end of July. Belugas are migrating up north for the season, green turtles are moving to the islands for nesting, and spawned sardines are moving up the east coast of Africa. Kubewarden doesn’t want to miss this season!
Migration to new single Helm chart admission-controller-6.0.0 Since its inception, the Kubewarden Admission Controller has shipped as 3 separate Helm charts: kubewarden-crds, kubewarden-controller, and kubewarden-defaults.
This was done for flexibili

As part of GSoC 2026, we’ve been extending SparkClient (KEP-107) — the Kubeflow SDK’s Python interface for running Spark on Kubernetes — with support for batch job submission and lifecycle management. Previously, SparkClient covered interactive workloads well via connect(), but running a batch job (the “submit a script, walk away, come back to results” kind of work that powers ETL pipelines and scheduled data prep) meant working with the SparkApplication CRD directly.

This post walks through how the new submit_job() API and its accompanying lifecycle APIs work, what’s happening on the cluster when you call them, and where the current implementation draws its boundaries.

Why this matters in practice

Running Spark on Kubernetes usually assumes a platform or infra team is around to stand up and maintain the cluster — the Spark Operator, the SparkApplication CRDs, the driver/executor resource tuning. That’s a fair assumption at a large org with a dedicated platform team. It’s a much bigger ask for a smaller team, or for the data engineer or ML engineer who just needs a job to run and doesn’t have that infra support behind them.

submit_job() is built for exactly that gap — FileJob for teams running existing ETL scripts, FuncJob for those who’d rather hand over a Python function directly — with the same predictable lifecycle underneath either way. The practical payoff: running a Spark step no longer requires users to author and manage SparkApplication resources directly—the SDK handles that for them.

Quick start

If you’d rather run something than read about it first, the examples/spark directory in the kubeflow/sdk repo has runnable scripts covering everything in this post:

• batch_job_lifecycle.py — submits spark_job.py as a FileJob, then walks through waiting, checking status, pulling logs, and cleanup
• batch_func_job_lifecycle.py — the same lifecycle, with a FuncJob instead
• batch_job_options.py — the same lifecycle with options applied, then verified on the created resource
• batch_failed_job.py — what a failure looks like end to end
• spark_job.py — the minimal PySpark script the lifecycle examples actually submit and run

Clone the repo and run any of them directly against a cluster with the Spark Operator installed. The rest of this post walks through what those scripts are actually doing under the hood.

Two ways to submit a job

submit_job() accepts either a FileJob or a FuncJob, and builds a SparkApplication spec from whichever one you pass in.

A FileJob points at an existing script:

from kubeflow.spark import FileJob

job = FileJob(
    file_source="https://raw.githubusercontent.com/kubeflow/sdk/main/examples/spark/spark_job.py",
)

file_source can reference both remote URIs (such as s3a://, gs://, hdfs://, or https://) and local:// paths. The SDK passes the URI through to the Spark runtime, which is responsible for resolving and accessing it. For a local:// path, the SDK does not package or upload anything on your behalf — the file must already be available inside the driver pod, for example through a volume mounted with spark.kubernetes.driver.volumes.* entries in spark_conf. Batch jobs currently run on a fixed apache/spark image and don’t accept PodTemplateOverride (Spark Connect only), so baking the script into a custom image or mounting a PVC through a pod template isn’t an option yet. FileJob describes where the file lives, not how it gets there.

A FuncJob takes a Python function instead of a file:

from kubeflow.spark import FuncJob

def transform():
    # your Spark logic here
    ...
job = FuncJob(func=transform)

The function has to be a plain, top-level function defined in an importable .py module — the SDK reads its source directly via inspect.getsource() to serialize it, which means lambdas, decorated functions, async functions, and anything defined interactively (a REPL or notebook cell) aren’t supported.

Since only the function source is serialized, the function should be self-contained. Any required imports should be placed inside the function body so they’re available when the generated script is executed. Parameters are passed separately via FuncJob(func=..., func_args={"samples": 10}), and their values must be JSON-like primitives — strings, numbers, booleans, None, and lists or dicts of those.

For example:

def transform():
    from pyspark.sql import SparkSession

<span class="n">spark</span> <span class="o">=</span> <span class="n">SparkSession</span><span class="p">.</span><span class="n">builder</span><span class="p">.</span><span class="n">getOrCreate</span><span class="p">()</span>

<span class="c1"># your Spark logic here
    ...

If you’re prototyping in a notebook, the function still needs to live in a module you import from, not be defined inline in a cell.

For a FuncJob, the SDK serializes the function into a generated script, writes it to a shared emptyDir volume through an init container, and points the driver at the generated file instead of a script you had to author yourself. This is the one case where the SDK does the packaging step for you.

Once a job is submitted, both types converge on the same shape — a name, a namespace, a status, a driver pod, some executors — so every lifecycle call downstream treats a FileJob and a FuncJob identically.

What happens when you call submit_job()

submit_job() doesn’t run the job — it hands off a request and returns as soon as the request is accepted:

1. SparkClient validates and builds a spec. Your FileJob or FuncJob gets translated into a SparkApplication custom resource.
2. The Kubernetes API stores it. At this point nothing is running yet — the resource exists, but nobody’s acted on it.
3. The Spark Operator notices. It watches for SparkApplication resources in the namespaces it was configured for and reacts as soon as a new one appears.
4. A driver pod starts. For a FuncJob, an init container writes the serialized function to a volume first; for a FileJob, the driver points straight at the referenced script.
5. The driver fans out to executors, however many were requested.

submit_job() returns once step 2 completes — it doesn’t block on job completion. That’s what the lifecycle APIs are for.

Configuring a job with spark_conf and options

Beyond the job definition itself, submit_job() takes spark_conf for Spark-level tuning and options for Kubernetes-level configuration.

spark_conf maps straight to Spark configuration properties:

client.submit_job(
    job=FileJob(
        file_source="https://raw.githubusercontent.com/kubeflow/sdk/main/examples/spark/spark_job.py",
    ),
    spark_conf={
        "spark.sql.adaptive.enabled": "true",
        "spark.sql.shuffle.partitions": "200",
    },
)

options covers the Kubernetes-native pieces that don’t belong in Spark config — labels, annotations, node placement, and job naming — as a list of typed option objects:

import uuid

from kubeflow.spark import (
    Annotations,
    FileJob,
    Labels,
    Name,
    NodeSelector,
    SparkClient,
    Toleration,
)
client.submit_job(
    job=FileJob(
        file_source="https://raw.githubusercontent.com/kubeflow/sdk/main/examples/spark/spark_job.py",
    ),
    options=[
        Name(f"batch-job-options-{uuid.uuid4().hex[:8]}"),
        Labels({"app": "spark", "team": "ml"}),
        Annotations({"owner": "kubeflow", "environment": "dev"}),
        NodeSelector({"kubernetes.io/os": "linux"}),
        Toleration(
            key="dedicated",
            operator="Equal",
            value="spark",
            effect="NoSchedule",
        ),
    ],
)

Both are optional — the earlier examples in this post work fine without either — but together they cover the two axes teams usually need to customize: how Spark runs the job, and how Kubernetes schedules and labels it. Worth knowing what you get when you leave them out: one executor, 1 core and 512Mi for both driver and executor, and the spark-operator-spark service account. Driver and executor resources aren’t configurable through submit_job() today, unlike connect(), which accepts both Driver and Executor objects.

Lifecycle APIs

One design goal for these lifecycle APIs was consistency across Kubeflow SDKs. Rather than introducing a new management model, SparkClient follows the same lifecycle pattern as TrainerClient. If you’ve previously managed distributed training workloads with TrainerClient, the APIs for submitting, monitoring, and managing Spark batch jobs should feel immediately familiar.

This consistency makes it easier to move between different stages of an ML workflow. A team might use SparkClient to prepare and transform large-scale datasets with Apache Spark, then use TrainerClient to launch distributed PyTorch training on the processed data. Although the underlying workloads are different, both clients expose a similar lifecycle interface.

Submission is the first of six checkpoints; the remaining five manage the job regardless of how it started:

Submit a batch job

from kubeflow.spark import FileJob, SparkClient, SparkJobStatus

client = SparkClient()
job_name = client.submit_job(
    FileJob(
        file_source="https://raw.githubusercontent.com/kubeflow/sdk/main/examples/spark/spark_job.py",
    ),
    num_executors=3,
    resources_per_executor={"cpu": "2", "memory": "4Gi"},
)

Output:

spark-job-e09b45b5

Wait for a job to complete

client.wait_for_job_status(
    job_name,
    status={SparkJobStatus.COMPLETED, SparkJobStatus.FAILED},
    timeout=600,
    polling_interval=2,
)

print(job.status)

Output:

COMPLETED

Get job details

job = client.get_job(job_name)
print(job.status, job.num_executors, job.driver_pod_name)

Output:

COMPLETED 3 spark-job-e09b45b5-driver

List running jobs

for job in client.list_jobs(status={SparkJobStatus.RUNNING}):
    print(job.name, job.status)

Output:

spark-job-a13d8c72 RUNNING
spark-job-b91f7a2e RUNNING

Stream job logs

for line in client.get_job_logs(job_name, follow=True):
    print(line)

Output:

25/08/08 17:54:02 INFO SparkContext: Running Spark version 4.x.x
25/08/08 17:54:05 INFO SparkContext: Starting job...
25/08/08 17:54:12 INFO DAGScheduler: Job 0 finished successfully
25/08/08 17:54:18 INFO SparkContext: Spark application completed successfully

Delete a job

client.delete_job(job_name)

Output:

Job deleted successfully.

wait_for_job_status() polls the SparkApplication’s status on a timer and returns the resolved SparkJob once it reaches one of the target statuses. SDK-level status is a simplified four-state model (CREATED, RUNNING, COMPLETED, FAILED) mapped from the underlying SparkApplication states, so callers don’t need to interpret raw CRD conditions. Two details worth knowing: if the job reaches FAILED and FAILED isn’t in your target set, the call raises RuntimeError right away instead of waiting out the timeout — which is why the example above passes both COMPLETED and FAILED — and any operator state the SDK doesn’t recognize is mapped to FAILED conservatively.

Log retrieval

get_job_logs() reads from the driver pod only, via the Kubernetes API, with follow=True supported for streaming as new lines are written.

Executor-level log access isn’t wired in yet — the driver is where Spark surfaces stage failures, exceptions, and final job status, so it covers the common debugging path for this first release. Executor logs are a reasonable follow-up, not a blocker for shipping batch support. One caveat either way: logs are only readable while the driver pod exists, so get_job_logs() raises RuntimeError if the driver hasn’t started yet or has already been cleaned up — pull them before delete_job().

What’s next

Batch submission and the lifecycle APIs have unit and end-to-end tests for both success and failure paths. The next phase of this project focuses on observability — pulling metrics from the Spark REST API (stage progress, executor stats, job duration), structured event tracking, and eventually a Prometheus-compatible export path.

None of this replaces the Spark Operator or reinvents Spark on Kubernetes — SparkClient stays a thin, Kubernetes-native layer on top of it. The goal is narrower and more practical: let the people who actually need to run Spark jobs — data engineers scheduling ETL, ML engineers prepping training data — do it from Python, without first becoming experts in SparkApplication YAML.

Want to Help?

The Kubeflow community is always looking for more contributors, testers, and users interested in Spark on Kubernetes and the broader SDK ecosystem. If you’d like to get involved:

• Explore the Kubeflow SDK documentation to learn more about SparkClient, TrainerClient, and the rest of the SDK.
• Learn about the ML Experience and Kubeflow SDK APIs, which outlines the vision for consistent Python APIs across Kubeflow projects.
• Visit the Kubeflow website or GitHub repositories.
• Join the Kubeflow Slack channels.
• Subscribe to the kubeflow-discuss mailing list.
• Attend the biweekly Kubeflow Spark on Kubernetes call hosted by the Data working group.

Feel free to share questions or feedback in the comments below — always happy to talk SparkClient.

KubeCon + CloudNativeCon Japan takes place July 28-30, 2026, in Yokohama, Japan. Register today to join us! Here are all the OpenTelemetry-related sessions, up to date at the time of writing. Session details can change, so c

OpenTelemetry as system dependency

Setting up OpenTelemetry for your applications and systems depends on where those apps and systems run. Some are very automated, especially Kubernetes, thanks to the OpenTelemetry Operator, or AWS Lambda, with the OpenTelemetry Lambda layers.

But the countless Java, .NET, Node.js, and Python app

If you write Java, Python, Node.js, or .NET, you have been able to add OpenTelemetry to an application without editing its code for years: attach an agent at startup and telemetry starts flowing. Go has been the exception. A Go program compiles to a single static binary with no runtime to hook into at startup, so Go developers have had to instrument by hand or reach for an out-of-process eBPF agent.

SIG-etcd has released patch updates across all three supported release branches. These releases fix two security vulnerabilities, several minor security issues, and address several reliability issues in the server, client library, and TLS stack. Users on v3.5, v3.6, and v3.7 should update at the next scheduled maintenance window.

Obtain the updates here:

• v3.7.1

In a previous blog post, we announced that Istio will retire the gcr.io/istio-release container registry in late 2026 and switch to registry.istio.io/release as the new home for Istio images. The original design was that registry.istio.io/release would be a Cloudflare worker that proxied requests to any OCI-compliant registry, allowing us to switch registries without any interruption to Istio users. Currentl