Cloud Native Architecture – Latest architectures

Architectures: A Private Cloud "Kubernetes Service" deployable anywhere.

Fri, 03 Jul 2026 00:00:00 +0000

Relevant projects

ArgoCD

Using since: 2024
Current version: v3.3.6

ArgoCD is used here as our main infrastructure engine. We’ve configured it so it can manage Day 1 and Day 2 operations seamlessly: Cluster APIs primitives on the management cluster to manage control plane operations, and helm applications to manage tooling and configuration of workload clusters. This infrastructure ArgoCD is centralized and dedicated to the platform team.

Cluster API

Using since: 2024
Current version: v1.10.5

Cluster API manages K8s control planes & machines at scale leaning on the ad hoc infrastucture providers (CAPO for OpenStack. CABPT and CACPPT for Talos), and serves as autoscaling & autohealing provider.

External-dns

Using since: 2021
Current version: v2.0.1

External-dns automates Designate DNS records management for workload Clusters.

External Secrets Operator

Using since: 2022
Current version: v1.3.2

External Secrets Operator is the glue between Hashicorp Vault and workload clusters, allowing for secure and centralized secrets management.

Harbor

Using since: 2021
Current version: v2.14

Harbor is the centralized registry: storing and distributing every image used on any container based infrastructure at SNCF.

Kubernetes

Using since: 2018
Current version: v1.35

SNCF’s sole container orchestrator. Used across all hosting zones, including critical infrastructure like high-speed trains.

OpenStack

Using since: 2024
Current version: Yoga

OpenStack provides Machines, DNS, Storage and Network to the platform, it is fully automated by CAPO and therefore abstracted from App Teams.

Renovate

Using since: 2024
Current version: v43

Renovate helps us automate OS and dependency patch management.

Talos Linux

Using since: 2024
Current version: v1.13.2

Talos Linux provides an immutable OS and Kubernetes distribution combo.

TLDR; Synopsis

This reference architecture describes SNCF’s second generation On-Premise Kubernetes Platform, built upon a bare-metal, open source OpenStack Infrastructure to provide at scale Container orchestration and scheduling. It has revolutionized hosting options for the organization’s applications, successfully providing a public cloud-managed Kubernetes service equivalent to internal teams. This platform allows them to build a sovereign strategy aligned with application requirements.

In particular, this architecture aims to show:

How legacy, non-tech organizations can address cloud migration, sovereignty and infrastructure automation in real life.
How to design a Managed Kubernetes Service using a 100 % open-source with zero vendor lock-in approach.
How the proposed architecture is designed for end-to-end automation for all its components.

Organization

SNCF is the publicly owned French rail operator, in charge of every layer of the rail transportation system except actually building trains. It encompasses thousands of different trades, resulting in needing thousands of different software applications operated mostly by one IT department.

The company decided in the late 2010s to move massively to the public cloud, giving birth to a containers platform team leaning on public cloud-managed Kubernetes services to create a hosting solution for container compatible applications. In 2023, that massive public cloud move was mainly wrapped up, the Kubernetes platform entering a growth sustainability phase, with at scale management problematics enforcing GitOps adoption. These stabilization efforts soon highlighted the need to offer the same level of service for container compatible applications not eligible for public cloud migrations, in order to prevent the organization to risk vendor lock-in with a fragmented two-tier system.

This was addressed by a joined initiative between a private cloud provider built on premises and an end-to-end automated kubernetes platform build on top of it.

Teams

Private Cloud Infrastructure is in charge of OpenStack deployment and configuration, managing hardware and network across SNCF’s private hosting zones owned by the SNCF. They offer virtualization, storage, network, LB on which our Kubernetes platform is built.
Cloud Native Integration builds, deploys, and maintains Kubernetes platforms on various hosting zones. We manage infrastructure, tooling and common services integration Apps Teams to deploy their software the easiest and best way possible.
Apps Teams develop, build, and ship applications anywhere in SNCF’s numerous hosting zones.

Architecture

The architecture described below, is our OnPremise implementation of our Kubernetes deployment strategy.

Goals

A unified and streamlined way to deploy and maintain Kubernetes across all our landing zones (private and public clouds).
A centralized end-to-end cluster and tooling lifecycle management.
Security and compliance policies enforcement across all clusters.
A large number of clusters manageable by a small infrastructure team.
Deploying new clusters in minutes.
A maintainable architecture.
100 % open-source.

Can you expand on why you are using those projects/services?

CNCF projects and OpenSource are at the heart of our architecture:

ClusterAPI - Clusters and Machines Management (adopted in 2025): ClusterAPI uses CAPO for lifecycle management of our Kubernetes clusters’ control plane. It also provides through CACPPT machine configuration, enabling node autoscaling and autohealing thanks to cluster autoscaler.
Talos - OS/Kubernetes Combo (Kubernetes: adopted in 2018, Talos: adopted in 2025): We use Kubernetes on every SNCF hosting zone. Talos was chosen as “Kubernetes Operating System” because it is an streamlined way to deploy production-grade clusters with immutability and security in mind. We manage Talos nodes through the Cluster API’s Talos provider (CACPPT).
Openstack - Infrastructure provider (adopted in 2025): OpenStack allow us to consume the bare-metal infrastructure in a cloud native way. It gives us a similar resource abstraction for infrastructure as public clouds. We consume OpenStack through the Cluster API’s Openstack provider (CAPO).
ArgoCD - Clusters and tooling source of truth (adopted in 2023): We are using ArgoCD on all our clusters on private and public clouds. It allows us to manage the lifecycle of clusters infrastructure components.
Kyverno - Policy enforcer and resource mutation (adopted in 2022): Admission policies allows us to enforce compliance with the organization’s policy (e.g., images coming from our private registry, specifying requests/limits). We also leverage resource mutation capabilities to simplify workload best practices for apps teams without having to develop a dedicated controller or operator (e.g., automaticaly configure pod disruption budgets or topology spread constraints).
Cilium - CNI (adopted in 2023): We use Cilium as CNI on all our infrastructures. We especially use it for its network policies capabilities. We may use it for cluster mesh in the future.
Harbor - Private registry (adopted in 2020): Harbor is our private registry used to store OCI artifacts (like images, charts, etc.).
Renovate - Automatic Patch Management (adopted in 2025): Renovate is used to automate patch management for Talos, Kubernetes, and infrastructure components.

What has worked well?

The combination of OpenStack + Kubernetes with Talos + ClusterAPI + ArgoCD allows us to easily build clusters on our on-premise datacenters, and enables cluster provisioning for users in minutes. They provide cloud-native capabilities on-prem, like: resilience, auto-scaling, on-demand load balancing and storage, easy upgrades, etc.

What needs improvement?

The main challenge lies with Cluster API Talos providers, which do not always keep pace with CAPI core releases. This prevents us from upgrading CAPI core to the latest version. We plan to invest engineering time in upstream contributions to the project.

What sort of “glue” have you had to develop to enable usage of your architecture ?

Capix: a controller linking cluster creation by ClusterAPI with the deployment of Kubernetes manifests via ArgoCD. It basically shares the newly generated kubeconfig from ClusterAPI with ArgoCD to deploy day2 manifests on the cluster.
GoIDC: is an OIDC authentication CLI for our on-premises clusters. It allows users to authenticate to their clusters using their corporate credentials.
get-onprem-cluster: is our CLI for users to easily download their kubeconfigs.

Has your architecture evolved? What lessons have you learned from previous iterations?

First iteration: This version started with Terraform, OpenStack, and Talos Linux, but we identified gaps in Day 2 operations efficiency.

Second iteration: We added GitOps using ArgoCD, leveraging the work already done for our other Kubernetes stacks. This improved management of CNI, CSI, security, and monitoring features.

Third iteration: We noticed one missing piece: autoscaling, which was not supported by the standard cluster-autoscaler on OpenStack. As a result, we migrated to Cluster API, which is fully compatible.

The goal was to enable autoscaling, but Cluster API provides even more features and simplifies our workflow. By adding Helm templating, the solution now offers:

Cluster provisioning in just a few dozen minutes.
Native autoscaling capabilities.
Easier node swapping and replacement.
Full cluster upgrades (Kubernetes and/or OS versions) in about 15 minutes.

However, it required significant custom integration work and we encountered several pain points when using the cluster-api-control-plane-provider-talos and cluster-api-bootstrap-provider-talos.

Indeed, we had to invest in these two projects because keeping them up to date with the Cluster API core depends heavily on community efforts.

What’s next for your architecture? What are you looking to do next?

We currently offer single-AZ clusters, as the underlying OpenStack platform operates within one availability zone. We’re working with the OpenStack platform team on a multi-AZ OpenStack deployment to enable multi-AZ cluster support.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: From Afterthought to Practice: Flipkart’s Multi-Tenant Chaos Engineering Platform on LitmusChaos

Mon, 08 Jun 2026 00:00:00 +0000

Relevant CNCF projects

LitmusChaos

Using since: 2023

Central chaos engineering platform. Provides Kubernetes-native chaos orchestration, a workflow controller, a probe framework, a drag-and-drop UI, and resilience scoring. Flipkart deploys a single centralized control plane (operator + workflow controller) and federates tenant access through subscribers — forming the foundation of the multi-tenant chaos platform.

Kubernetes

Using since: 2021

Runtime substrate for hundreds of Flipkart microservices and for the LitmusChaos control plane itself. Custom Resource Definitions (ChaosEngine, ChaosExperiment, ChaosResult) model chaos as Kubernetes objects. DaemonSets are used to provide a node-local, long-lived chaos injection surface for high-availability experiment execution.

Argo Workflows

Using since: 2023

Underlying workflow engine used by Litmus to model multi-step chaos experiments — sequencing pre-checks, fault injection, probes, and recovery validation as a DAG. Flipkart’s Script Runner fault plugs into this DAG to enable dynamic target selection and context chaining between steps.

Prometheus

Using since: 2021

Metrics backbone for both production workloads and chaos experiments. Chaos drills are validated against Prometheus-driven SLO and alert thresholds (CPU saturation, 5xx error rates, latency, queue depth). Litmus’s Prometheus probes assert hypotheses during fault injection.

Helm

Using since: 2022

Used to package and roll out the centralized Litmus control plane, tenant subscribers, the DaemonSet-based injection layer, and Flipkart’s custom fault images consistently across staging and production clusters.

containerd

Using since: 2021

Container runtime on Flipkart’s Kubernetes nodes. The DaemonSet injection model relies on stable runtime primitives to execute parallel chaos sessions safely on a single node.

Other Projects

Project	Using Since	Notes
Grafana	2022	Dashboarding layer for chaos experiment telemetry. Flipkart reuses Litmus’s out-of-the-box Grafana exporters to surface experiment run history, resilience scores, and infrastructure-level signals to both Kubernetes and VM-team consumers — giving both audiences the same observability surface.
MongoDB	2023	Backing store for the Litmus control plane. Flipkart’s probe-uniqueness contribution required reworking a global MongoDB index into a project-scoped one to make probes truly multi-tenant.
Linux stress tooling on Debian	2023	Internal stress packages (CPU hog, memory hog, network hog, disk-IO) targeted at the Debian host OS of Flipkart’s VM fleet, invoked by the Hybrid VM Chaos extension when the workload doesn’t live in Kubernetes.
Internal Flipkart Cloud Platform APIs	2024	Lifecycle APIs (start, stop, restart) for Flipkart’s VM fleet. The Hybrid VM Chaos extension calls these APIs to deliver VM-power experiments in parity with the Kubernetes-native experience.
Internal Chaos Hub	2024	Private, Flipkart-internal catalog of pre-curated chaos experiment templates. Built so that tenants can publish, version, and consume reusable experiments across teams — the chaos-engineering analogue of an internal package registry.

Synopsis

Flipkart is one of India’s leading digital commerce companies, running hundreds of tightly coupled microservices across Kubernetes and VM workloads that must withstand the traffic surges of Big Billion Days and other festive sales. The Central Reliability Engineering team built a centralized chaos engineering platform on LitmusChaos to convert chaos engineering from an afterthought into a continuous practice.

On top of upstream Litmus, Flipkart engineered four high-leverage customizations: (1) a hybrid multi-tenancy model that combines the operational simplicity of a cluster-wide install with the isolation of a namespace-wide install; (2) a DaemonSet-based high-availability layer for chaos injection that runs one persistent injector per node and supports parallel sessions; (3) a first-class “Script Runner” fault that allows dynamic target selection and context chaining across steps in an experiment; and (4) a hybrid VM chaos extension that reuses the Litmus experience for non-Kubernetes workloads via Flipkart’s internal cloud APIs and Debian-host stress tooling.

The result is operational readiness validated through live chaos drills ahead of Big Billion Days, a measurable shift in engineering culture from “prevention” to “practice,” and a steady stream of upstream contributions back to the LitmusChaos project.

Organization

Flipkart is one of India’s leading digital commerce companies, serving hundreds of millions of customers across an ecosystem that includes flagship apparel, electronics, grocery, and fintech offerings. Its flagship event, Big Billion Days, is the largest e-commerce sales event in India and drives massive concurrent traffic across a tightly integrated microservices estate. Other high-scale moments — the Independence Day sale and Diwali sale among them — generate similar surges throughout the year.

Flipkart’s engineering organization runs an extensive Kubernetes-as-a-Service platform alongside a large VM fleet, with internal central platform teams providing service mesh, networking, and database offerings to hundreds of downstream application teams. Reliability at the scale of an Indian retail tentpole event is therefore not a product-team concern — it is a horizontal platform discipline.

Teams

Multiple teams collaborate to build, operate, and consume the chaos platform:

Central Reliability Engineering — owns the chaos platform end-to-end: the centralized LitmusChaos control plane, tenant onboarding, the DaemonSet injection layer, the Script Runner fault, the hybrid VM chaos extension, the internal Chaos Hub, and upstream contributions back to Litmus.
Kubernetes-as-a-Service team — first tenant of the chaos platform; runs chaos drills against the Kubernetes control plane primitives consumed by every Flipkart workload.
Network Services and Service Mesh team — uses chaos to validate Layer-4/7 failure behavior, retry semantics, and circuit-breaker configurations across the mesh.
Central Database team — leverages the Script Runner fault to verify leader-election behavior in leader-follower database topologies during chaos drills.
Application teams — onboarded progressively after the central platform teams; consume curated experiments from the internal Chaos Hub.
VM Infrastructure teams — use the hybrid VM chaos extension to obtain the same observability surface (Grafana, run history, resilience score) as Kubernetes-native consumers, despite running outside Kubernetes.

Architecture overview & Goals

Goals

Make chaos engineering a continuous practice, not an afterthought. Move from runbooks-on-paper to regularly rehearsed failure scenarios that feed back into incident response.
Operate one chaos platform for the whole company, safely. Provide a centralized control plane that hundreds of tenants can use without each tenant carrying operational burden — and without any tenant being able to touch another tenant’s workloads.
Eliminate self-inflicted experiment failures. Remove brittleness from the chaos infrastructure itself — specifically, helper-pod scheduling failures that were masking real reliability signal.
Express realistic, real-world failure scenarios. Support dynamic target selection and context chaining inside experiments so a single chaos run can express “find the leader, kill it, watch a new leader get elected” as one declarative workflow.
Treat VM workloads as first-class citizens. Reuse the Litmus experience for workloads that don’t run on Kubernetes — same UI, same probes, same observability — instead of building a parallel tool.
Validate operational readiness for Big Billion Days. Convert chaos drills into a measurable, repeatable pre-event sign-off rather than a manual exercise.
Contribute upstream. Push fixes and architectural patterns back into LitmusChaos so the broader CNCF community benefits and Flipkart avoids long-term fork maintenance.

Architecture Overview

At a high level, the architecture consists of a centralised control plane that is setup on a Kubernetes namespace or cluster that helps in designing and coordinating the chaos experiments. The chaos experiments are designed, implemented and certified and finally placed in a ChaosHub, which then are used by various teams in the organization to run them on the target Kubernetes clusters. Chaos experiment results are pushed back into the control plane as resilience scores, which can be used to create resilience testing trends and resilience posture reports for various teams’ consumption.

The platform is organized into four logical layers:

Control Plane Layer. A single, centralized LitmusChaos installation — the operator, workflow controller, MongoDB backing store, and Grafana dashboards — lives in a dedicated central namespace and is operated by Central Reliability Engineering. There is one control plane per cluster footprint (staging, production), not one per tenant.

Tenant Subscriber Layer or a chaos agent. For each tenant team, a dedicated subscriber is deployed in the central namespace. The subscriber is scoped at install time to the set of tenant-owned namespaces it is allowed to target. The subscriber is the only Litmus identity that can dispatch chaos into a tenant’s namespaces — giving namespace-grade isolation without forcing each tenant to operate its own Litmus install.

Injection Layer. Instead of spawning ephemeral helper pods on-demand per experiment, Flipkart runs a DaemonSet (one replica per node, using the litmus-go image) as a long-lived, highly privileged injection surface. Experiment pods now offload target information to the node-local DaemonSet pod, which executes the chaos in parallel shell sessions. This supports concurrent chaos injection on the same node and eliminates a class of failures where the previous on-demand helper pod couldn’t be scheduled in time.

Extension Layer. Two domain-specific extensions plug into the platform: the Script Runner fault (a user-defined container running a user-defined script, whose stdout feeds the target list of any subsequent fault in the experiment), and the Hybrid VM Chaos extension (which uses Flipkart’s internal cloud-platform APIs for VM power operations and internal Debian stress packages for resource-level chaos). An internal Chaos Hub layers on top, letting tenants publish reusable experiment templates.

Tenants interact with the platform almost entirely through the Litmus UI. They author or pick a template from the internal Chaos Hub, attach probes (including SSH-based probes for the VM surface), and run the experiment against namespaces or VMs they own. Resilience scores, run history, and Grafana dashboards present a single observability surface regardless of whether the target is a Kubernetes workload or a VM.

Key Design Principles

Centralized control plane, federated trust. One operator and one workflow controller for the whole organization; isolation is enforced at the subscriber boundary, not by running N copies of Litmus.
Persistent, node-local injection. Trade ephemeral safety for scheduling determinism by accepting a long-lived privileged DaemonSet — a deliberate trade-off made only where the operational win (no scheduling failures during a chaos drill) outweighs the cost (privileged residency).
Context chaining over static targeting. Chaos experiments are workflows, not single faults; the output of one step (“who is the leader?”) feeds the inputs of the next (“kill that pod”).
One experience, two surfaces. Kubernetes and VM chaos must look, feel, and report the same to the user. The UI, probes, and observability layer are surface-agnostic; the differences are pushed down into the extension layer.
Practice, then expand. Start with central platform teams that already run shared infrastructure; only then expand to application teams. Staging is the primary chaos surface; production chaos is selective and gated.
Upstream where possible. Local patches create long-term maintenance burden. Bug fixes and reusable patterns go back to LitmusChaos.

Can you expand on why you are using those projects/services?

Why LitmusChaos

Before standardizing on Litmus, the team ran a detailed proof-of-concept across Chaos Monkey, Chaos Mesh, and LitmusChaos. Litmus won on four criteria that mattered for Flipkart’s context: it is Kubernetes-native end-to-end (aligning with the broader K8s-first direction), it ships a rich drag-and-drop UI that drives adoption beyond SREs, it is deeply extensible (custom faults were a non-negotiable requirement), and its resilience-probe model gave the team a path toward a measurable resiliency score per service.

From single-tenant to hybrid multi-tenancy

The two install modes Litmus offers out of the box are cluster-wide (low operational burden, but no isolation) and namespace-wide (strong isolation, but every tenant has to operate its own Litmus install). At Flipkart’s scale — hundreds of teams across central platform and application areas — neither extreme works. The hybrid model takes the best of both: one centralized control plane (so no team has to run Litmus) plus a per-tenant subscriber pinned to that tenant’s namespaces (so isolation is enforced at the dispatch boundary). Litmus maintainer Karthik Satchitanand described this pattern as “a very nice model… a good balance” and has suggested it be documented upstream as a preferred mode of operation.

Why a DaemonSet instead of helper pods

Litmus’s default execution model spins up an on-demand helper pod per experiment. At scale, in busy staging clusters with active chaos drills, those helper pods can fail to schedule — producing a chaos infrastructure failure that masquerades as the chaos itself. The DaemonSet inverts this: one long-lived pod per node, with the chaos image already present, executes parallel chaos sessions invoked by the experiment pods. This makes injection deterministic at the cost of a persistent privileged surface — a trade-off Flipkart accepts in staging, where platform availability for chaos drills matters most.

Why a Script Runner fault

Real-world failure scenarios are rarely “delete pod X.” They are “find the current leader of a leader-follower system, delete it, and verify re-election.” Litmus’s built-in faults could not express dynamic target selection or pass state forward from one step to the next. The Script Runner is a first-class fault type that executes a user-defined container running a user-defined script; its stdout becomes the target list for any subsequent fault in the experiment, not just the immediate next one. The database team’s leader-election verification was the canonical use case that motivated this design.

Why a Hybrid VM Chaos extension

A meaningful slice of Flipkart’s estate runs on VMs, not Kubernetes. Building a parallel chaos tool for that surface would have fragmented the user experience and split the observability story. Instead, the team rebuilt the existing VM Power experiment against Flipkart’s internal cloud platform APIs (start/stop/restart) and added resource-level stress chaos (CPU hog, memory hog, network hog, disk-level) using internal Debian stress packages. Because the experiments are still driven by Litmus, VM teams get the same Grafana exporters, the same experiment run history, the same resilience score tab, and the same probe model (with custom SSH-based probes for pre- and post-experiment infrastructure checks) that Kubernetes users get.

What has worked well?

Hybrid multi-tenancy. The centralized-control-plane plus per-tenant-subscriber pattern absorbed onboarding for many tenants with effectively zero per-tenant infra burden, while still preventing cross-tenant blast radius. Recognized by upstream Litmus maintainers as a candidate preferred-mode pattern.
DaemonSet injection. Eliminated experiment failures caused by helper-pod scheduling issues in staging. Chaos drills now fail because the system under test failed — not because the chaos infrastructure couldn’t schedule a pod.
Script Runner. Unlocked previously inexpressible scenarios such as leader-election verification. Context chaining turned chaos experiments into proper workflows rather than single-shot fault injections.
Unified VM and Kubernetes experience. Reusing Grafana exporters, run history, and the resilience score tab gave VM teams a first-class user experience and avoided fragmenting the chaos tooling landscape inside Flipkart.
Pre-event readiness validation. Chaos drills validated operational readiness ahead of the recent Big Billion Days sale, confirmed that CPU thresholds trigger the right alerts, that network failures surface correctly as 5xx spikes, and that weak links in deployment specs and alert configurations could be fixed before they became real incidents.
Cultural shift. Engineering teams moved from “prevention” (try not to fail) to “practice” (rehearse failure), and from “panic” to “procedure” during incident response. Chaos scenarios now form the foundation of incident runbooks.
Internal Chaos Hub. Tenant-published experiment templates accelerate reuse across teams — the chaos engineering analogue of an internal package registry.

What has not worked well?

Privileged DaemonSet residency. Persistent, highly privileged pods on every node are an accepted trade-off in staging but constrain how aggressively the same pattern can be deployed in production. Reducing the blast radius of the injection layer remains an open problem.
Multi-tenant gaps in upstream Litmus. Several places in upstream Litmus assumed single-tenant scope — most visibly, probe-name uniqueness enforced globally instead of project-scoped (a MongoDB index issue). These needed upstream fixes before the multi-tenant model could behave correctly.
Terminology drift across the project. The renaming of “experiment” to “fault” in parts of Litmus left behind frontend/backend mismatches — e.g., experiment pod-block visibility — that required cleanup contributions.
Heterogeneous VM hosts. The stress chaos extension is currently scoped to Debian hosts; broadening to additional host OSes will mean factoring out the OS-specific layer.
Production chaos is still selective. Most chaos still runs in staging. Building enough confidence (and enough guardrails) to run the full chaos surface in production is an ongoing journey.
Custom image registry support. A long-standing (year-old) issue prevented the custom image registry setting from flowing through to the generated workflow YAML — a blocker for air-gapped and internal-registry deployments. Required a Flipkart upstream fix to resolve.

What sort of “glue” have you had to develop?

Per-tenant subscriber onboarding pipeline. Tooling and a defined onboarding flow so that adding a new tenant means deploying a scoped subscriber, registering its namespaces, and attaching it to the central control plane — without any tenant operating its own Litmus install.
DaemonSet injection runtime. Replaced the on-demand helper-pod model with a long-lived, node-local DaemonSet (using the litmus-go image) that accepts target information from experiment pods and executes parallel chaos sessions in independent shells.
Script Runner fault. New first-class fault type backed by a user-defined container image running a user-defined script, with stdout captured as structured context that any subsequent fault in the experiment can consume — not just the immediate next step.
Hybrid VM Chaos extension. A reimplementation of VM Power on top of Flipkart’s internal cloud APIs, plus a stress-chaos suite (CPU, memory, network, disk) built on internal Debian stress packages, plus custom SSH-based probes that mirror the Kubernetes probe experience for VM workloads.
Internal Chaos Hub publishing pipeline. A workflow that lets tenants publish reusable experiment templates to a private, Flipkart-internal catalog and pull them down into their own experiments.
Upstream fixes packaged as patches and PRs. Probe-name uniqueness (MongoDB index), probe-name reuse-after-deletion (filtration), experiment-update duplicate-name validation, UI consistency (experiment-completion icon, GCP/Azure fault logos), custom image registry, and pod-block visibility — each shipped back to the LitmusChaos project.

How did the Architecture Evolve**

Journey

Phase 1 — Reactive runbooks. Failure modes existed only on paper. Runbooks were written but not practiced. Chaos engineering was an afterthought.
Phase 2 — Tool selection. Detailed PoC across Chaos Monkey, Chaos Mesh, and LitmusChaos. Litmus selected on Kubernetes-native design, drag-and-drop UI, extensibility, and resilience-probe model.
Phase 3 — Centralization. Stood up a single centralized Litmus control plane and built the per-tenant subscriber model to hit multi-tenancy without forcing tenants to run their own installs.
Phase 4 — Hardening the injection layer. Replaced ephemeral helper pods with the DaemonSet HA model after observing scheduling-induced experiment failures in staging.
Phase 5 — Expressing real scenarios. Added the Script Runner fault to unlock dynamic target selection and context chaining (e.g., leader-election verification for the database team).
Phase 6 — Beyond Kubernetes. Built the Hybrid VM Chaos extension so VM teams get the same Litmus experience and observability surface as Kubernetes teams.
Phase 7 — Internalize and contribute back. Stood up the internal Chaos Hub for cross-team reuse and began the upstream contribution cadence back to LitmusChaos.
Phase 8 — Pre-event validation. Used the platform to validate operational readiness ahead of Big Billion Days; chaos drills now feed incident runbooks directly.

Key Lessons

Hybrid is the right answer for multi-tenancy at scale. Neither cluster-wide nor namespace-wide install models fit a hundreds-of-teams organization on their own. A centralized control plane plus per-tenant subscribers is the productive middle.
Chaos infrastructure must be more reliable than the systems it tests. Every false negative — a chaos experiment that failed for infrastructure reasons — erodes trust in the practice. The DaemonSet HA model exists because that trust was real and worth defending.
Failure modes are workflows, not single faults. Dynamic target selection and context chaining (Script Runner) were not nice-to-have — they were the difference between toy chaos and useful chaos.
Treat non-Kubernetes workloads as first-class. Pushing VM chaos through the same Litmus experience and observability surface kept the cultural and tooling story unified.
Start with central platform teams. Teams that already run shared infrastructure (Kubernetes-as-a-Service, service mesh, central databases) had the runbooks and the motivation to be the first chaos consumers. Application teams followed.
Documentation and discipline. Clear hypotheses before the experiment, clear results after; this is what turns a one-off chaos run into reusable organizational knowledge.
Don’t roll chaos out everywhere at once. Pilot, learn, then expand. The internal Chaos Hub exists because the expansion phase needed reusable artifacts, not from-scratch experiments per team.
Recovery time matters as much as prevention. You can’t prevent every failure, but you can systematically improve how quickly you recover from it — and chaos drills are the most honest measurement of that.
Contribute upstream. Local patches become long-term debt; upstream contributions become permanent improvements for everyone, including Flipkart’s future self.

What’s next for your architecture?

Production chaos at scale. Expand selective production chaos runs into a more regular, more comprehensive practice — with new guardrails for the injection layer to reduce the privileged surface area.
Broader host OS support for VM chaos. Factor out the OS-specific stress layer so the hybrid VM chaos extension is not tied to Debian.
Deeper integration with incident runbooks. Close the loop so that every documented incident generates (or updates) a corresponding chaos scenario.
Resiliency score as a service-level metric. Promote the Litmus resilience score from “informational” to a tracked, dashboarded, team-level reliability KPI.
Continued upstream contribution. Push the hybrid multi-tenancy pattern, the DaemonSet HA injection model, and the Script Runner fault toward generalization in upstream LitmusChaos for the broader CNCF community.
Chaos in CI/CD. Move from out-of-band chaos drills to chaos gates that run as part of the deployment pipeline for high-tier services.
Observability and AIOps. Use the rich telemetry produced by repeated chaos runs to drive AI-assisted blast-radius prediction and pre-event readiness scoring for events like Big Billion Days.

Community Contributions

Contribution	Details
Probe scoping fix	Made probe-name uniqueness project-scoped instead of globally scoped (MongoDB index fix), enabling true multi-tenant use of probes in LitmusChaos.
Probe name reuse	Filtration fix that allows probe names to be reused after deletion — removed a stale-record blocker for teams iterating on probe definitions.
Experiment update bug	Fixed a duplicate-name validation that was incorrectly firing on tag/description-only edits when updating an experiment.
UI consistency fixes	Corrected the experiment-completion icon; fixed fault-card logos for GCP and Azure faults that were rendering the generic Litmus logo.
Custom image registry	Fixed long-standing (year-old) issue so the custom image registry setting correctly reflects in the generated workflow YAML — essential for air-gapped and internal-registry deployments.
Pod-block visibility	Fixed experiment pod-block visibility on the portal; resolved terminology drift between frontend and backend after the “experiment → fault” rename.
Architecture pattern	Hybrid multi-tenant architecture (centralized operator + per-tenant subscriber) recognized by Litmus maintainers as a candidate preferred mode of operation to be documented upstream.
Conference contribution	Case study presented at KubeCon India keynote (2026), sharing the multi-tenant chaos platform pattern with the broader CNCF community.

Voices from the team

“We moved from prevention to practice; instead of just trying to prevent issues, we actively practice handling them. And for incident response, we’ve gone from panic to procedure.”

— Khushi Tiwari, Software Developer, Central Reliability team

“Implementing Litmus chaos at the kind of scale we operate at, and enabling the platform teams that directly impact hundreds of development teams — that’s been a very rewarding journey.”

— Aditya, Software Developer, Central Reliability team

“The scenarios we test become the foundation for our incident runbooks, which makes the actual incident response much better.”

— Hashith, Site Reliability Engineer

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: ZEISS Vision Care - Order Fulfillment

Tue, 17 Mar 2026 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2020
Current version: 1.32.3

Hosts > 200 microservices supporting order fulfillment processes on managed Kubernetes. Provides the core compute platform for containerized services.

Dapr

Using since: 2020
Current version: 1.17.0

Provides common building blocks like service invocation, pub/sub, and state management across microservices; vendor-neutral abstractions enable portability across cloud providers.

KEDA

Using since: 2021/2022
Current version: 2.19.0

Event-driven scaling. KEDA acts as an event-driven scaler; examples of triggers include message-broker queue depth and resource utilization.

OpenTelemetry

Using since: 2020
Current version: Collector Contrib 0.145.0

Provides consistent instrumentation and exports telemetry to multiple targets.

Helm

Using since: 2020

Package management and templating for Kubernetes deployments.

Organization

ZEISS Vision Care produces spectacle lenses, instruments for refraction, and glasses adjustment equipment. Often, lenses are manufactured to an individual consumer’s prescription, effectively a batch size of one, which makes order fulfillment a multi-step coordination process.

Our Business Enablement & IT team develops, designs, and operates the order fulfillment platform that underpins this process.

Synopsis

We are reworking the order fulfillment process using a greenfield approach to modernize core systems. Our goal is a reliable, scalable platform that can evolve with business needs while remaining cost-efficient. The platform supports several key capabilities:

Order routing: decide where the order should be produced.
Order document generation: generate the necessary manufacturing and shipping documents for the order.
Logistics routing: decide how to ship the order to the customer.
Additional auxiliary domains as required.

Architecture overview & Goals

Goals

The architecture is designed for future-proofing and scale. By leveraging event-driven scaling with KEDA and decoupled microservices via Dapr, the system is built to seamlessly absorb high-volume, global order loads as new processes are continuously migrated to the new platform.

Key Requirements:

Modern cloud-based infrastructure
High reliability and scalability
Maintainable and extensible
Cost-effective operations

Architecture Overview

Data & Storage Strategy

The platform uses Azure-managed data services as backing stores:

MSSQL: Transactional order data and relational schemas
Cosmos DB: Scaled state management and cross-region replication
Azure Blob Storage: Order documents and manufacturing files
Redis: Caching operations to improve latency and throughput

Dapr’s state abstraction layer is used for Cosmos DB, Azure Blob Storage, and Redis, decoupling microservices from those storage backends and enabling migrations without application-level changes. For Azure Blob Storage, we also use Dapr’s blob binding to interact with data, in addition to the state abstraction layer.

Messaging & Asynchronous Communication

Azure Service Bus serves as the central message broker for asynchronous processes. Services publish and subscribe to topics via Dapr’s pub/sub building block, enabling loose coupling between applications. This event-driven approach provides resilience and allows each service to scale independently based on demand.

Network & Service Discovery

Services use Dapr’s service invocation building block instead of hardcoded endpoints, enabling resilience and the ability to replace or upgrade service implementations without changing callers. External traffic is routed to services via Ingress NGINX.

CI/CD & Deployment

Azure DevOps Pipelines automate building, testing, and deploying applications. We use a single, centralized Helm chart and store deployment configuration in Git as the single source of truth; the pipeline generates environment- and service-specific values.yaml files and deploys each service as its own Helm release to Kubernetes.

Can you expand on why you are using those projects/services?

We rely heavily on CNCF projects and open-source tooling to form the backbone of our platform:

Kubernetes (AKS) (Using since 2020): Hosts > 200 microservices supporting order fulfillment workflows. It provides the core compute platform for containerized services, offering a flexible model that supports multiple frameworks, programming languages, and dynamic scaling.
Dapr (Using since 2020): Provides common building blocks like service invocation, pub/sub, and state management across microservices. Dapr enables vendor-neutral capabilities, addressing service discovery and maintaining portability.
KEDA (Using since 2021/2022): Event-driven scaling based on Azure Service Bus queue depth and CPU/memory utilization. This allows the system to scale aggressively to match real-time demand while running economically during quieter periods.
Helm: Package management and templating. Charts are stored in Git, enabling reproducible deployments across environments since the project’s inception.
Ingress NGINX (currently in use, pending replacement – see Future Outlook): External traffic routing and load balancing.
OpenTelemetry Collector (Using since 2020): Provides consistent instrumentation and exports telemetry to multiple targets. It enables distributed tracing and metrics collection across microservices, ensuring observability and performance monitoring.

What has worked well?

Kubernetes, Helm, and KEDA have proven reliable and are widely used.

Scaling to Zero and Back: We initially attempted to scale services to 0 replicas with KEDA to minimize costs, but this introduced operational challenges: delayed data flows during testing when pods needed to start up, and frequent scale churn during traffic pauses between messages. We learned that setting a minimum replica count for baseline load while scaling out for peaks was much more cost-effective and reliable for our specific workload patterns.

The Advantage of Abstraction: A major architectural risk early on was building the entire system on Dapr (starting at pre-1.0 release 0.7.0). Early adoption carried organizational risk and we initially faced instability with actors under heavy load. Over time, those issues were fully resolved. The decision proved highly beneficial: Dapr’s maturity and vendor-neutral approach validated the initial decision, giving the platform extreme portability, flexibility, and saving custom boilerplate.

Platform Evolution: Dapr is consistently evolving, allowing us to streamline our platform by replacing specific SDKs with built-in functionalities over time. For instance, we are transitioning from Azure App Configuration to Dapr’s Configuration Building Block. In hindsight, we would have standardized on CNCF native components sooner (e.g., migrating earlier to OpenTelemetry Collector and a CNCF ingress solution) to avoid replacing vendor-specific SDKs mid-project.

Data Ownership and Boundaries: A critical lesson was the strict enforcement of data ownership. In a microservices architecture with over 200 services, we learned that clearly defining which service owns which data is non-negotiable. We found that any direct data access between services, bypassing their dedicated APIs, inevitably leads to tight coupling and maintenance challenges. Adhering to Domain-Driven Design (DDD) principles, where each service exposes its data only through a well-defined API, was essential for maintaining a scalable and evolvable system. This prevented a “distributed monolith” and ensured long-term architectural integrity.

What sort of “glue” have you had to develop?

Since we had a greenfield start, we were not constrained by legacy endpoints or migration paths. However, managing over 200 microservices required strict boundaries and standardized communication patterns. We heavily utilized Domain-Driven Design (DDD) principles—specifically Bounded Contexts to ensure each microservice owns its data and domain logic.

To enforce these boundaries and standardize the “glue” between services, we developed:

Standardized Helm Templates: A unified set of Helm charts that abstract away the complexity of Kubernetes manifests and Dapr sidecar configuration. Developers provide application-specific KEDA ScaledObject definitions and Dapr component definitions.
Common Libraries: While Dapr abstracts away many infrastructure concerns, we built thin, language-specific wrappers around the Dapr SDKs to enforce internal logging and error-handling standards.

Impact & Results

By adopting this cloud-native stack, we have built a highly scalable and future-ready order fulfillment system. A key driver of this success has been our extensive use of Dapr, which not only significantly reduced boilerplate code but also provided us with a high degree of vendor neutrality. This abstraction layer means we are not tightly coupled to specific cloud providers’ SDKs, enabling an architecture that is portable, flexible, and robust enough for long-term growth.

What’s next for your architecture?

We are constantly investigating how to run our services as efficiently and economically as possible. A near-term priority is replacing our existing Ingress NGINX setup. Because the Kubernetes project has officially announced the retirement of the ingress-nginx project, we are actively evaluating alternatives—including NGINX Gateway Fabric, Envoy Gateway, and Traefik—while striving to preserve our routing behavior, TLS automation, and operational stability. Transitioning to the Gateway API is a key step in future-proofing our external traffic routing.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: End-to-End Cloud Native Telco Platform Automation at Swisscom

Mon, 16 Mar 2026 00:00:00 +0000

Relevant Projects

CNCF Projects

Kubernetes

Using since: 2021

The Kubernetes API is the database backend and control plane of the entire automation platform. It acts as the runtime for all CNFs, operators, and platform services. Custom Resource Definitions extend the API to cover telco-specific concerns like IPAM, DNS, and network function configuration.

Flux

Using since: 2022

Flux is the GitOps engine for continuous reconciliation. It monitors Git repositories and synchronizes all desired state — CNF deployment manifests, Custom Resources, DNS endpoints, certificate requests, IP claims, and test definitions — into Kubernetes clusters.

cert-manager

Using since: 2023

Automated certificate lifecycle management integrated with Swisscom’s internal PKI. Certificate requests are expressed as Kubernetes CRs, reconciled by Flux, and managed by cert-manager. Private keys never leave the cluster.

Headlamp

Using since: 2025

Kubernetes dashboard for the management cluster, providing cluster visibility, RBAC-based access control, a CRD documentation browser, and extensible plugin system. Swisscom is listed as an official Headlamp adopter.

SDC (Schema Driven Configuration)

Using since: 2024

Used as the Config Sync Operator to push assembled configurations to CNFs. SDC enables vendor-agnostic, declarative configuration management using YANG schemas and NETCONF/gNMI protocols. Swisscom adopted SDC as its strategic configuration management solution and actively contributes features including config blame, drift detection, validation, testing compatibility with CNFs, and NETCONF Actions support. Swisscom is listed as an official SDC adopter.

CoreDNS

Using since: 2021

In-cluster DNS service discovery for Kubernetes services. Also used with conditional forwarding to route queries for private 5G zones (e.g., 3gppnetwork.org) to the authoritative PowerDNS servers.

ExternalDNS

Using since: 2023

Kubernetes-native automation of DNS records in PowerDNS using Custom Resources and annotations.

MetalLB

Using since: 2022

Load balancer for bare-metal Kubernetes clusters. MetalLB IP address pools are managed via KRM, with IP addresses dynamically allocated from NetBox via the NetBox Operator.

Kubebuilder / controller-runtime

Using since: 2022

Scaffolding framework and libraries for building custom Kubernetes operators. Used to build all domain-specific operators for CNF configuration abstraction, IPAM integration, config synchronization, and DNS automation.

Other Projects

PowerDNS

Using since: 2023

Authoritative DNS server supporting automation of advanced resource records (NAPTR, SRV) required for 5G/SIP via ExternalDNS.

NetBox Operator

Using since: 2024

Kubernetes operator for IPAM integration, open-sourced by Swisscom. Brings IPAM into the Kubernetes API with a claim model inspired by PersistentVolumeClaims — dynamically allocating IP prefixes and addresses from NetBox, managing their lifecycle through Kubernetes garbage collection, and supporting sticky IPs for disaster recovery.

NetBox

Using since: 2023

IP Address Management (IPAM) and network infrastructure modeling. Used as the IPAM backend for dynamic IP allocation across all CNFs and platform services.

TL;DR or Synopsis

Swisscom has built a cloud native telco platform for the end-to-end automation of its 5G standalone core network and cross-domain resource orchestration. The architecture replaces traditional imperative network management (Jenkins pipelines, Ansible playbooks) with a fully declarative, Kubernetes-native automation model driven by GitOps and the Kubernetes Resource Model (KRM).

A mobile core development environment contains approximately 2,000 pods with over 5,000 interdependent configuration parameters across Cloud-Native Network Functions (CNFs) such as UPF, SMF, AMF, UDM, UDR, BSF, NRF, NSSF, and AUSF. Engineers express high-level intents as Kubernetes Custom Resources; custom operators dynamically assemble full configurations at runtime — fetching IP addresses from IPAM, secrets from Vault, certificates from PKI, and infrastructure details from the cluster.

While the 5G core is the primary domain, the orchestration framework extends across multiple network domains and infrastructure services, applying the same intent-based automation patterns consistently.

Organisation

Swisscom is the leading Telecommunications/ISP and ICT company and offers mobile, Internet and TV products, as well as comprehensive IT and digital services to private and business customers. Swisscom’s expertise in cloud native technologies is well-established, as evidenced by its status as a former Gold member and Management Board member of the Cloud Foundry Foundation, along with its certification for Cloud Foundry. Additionally, Swisscom demonstrates a strong commitment to the Open-Source community, having been a CNCF Silver Member for several years and serving as a Kubernetes Certified Service Provider (KCSP) partner. Our skilled employees have delivered numerous talks and presentations at prestigious events such as KubeCon, Cloud Native Zürich, Swiss Cloud Native Day, KCD Suisse Romande, ContainerDays.

The company has embarked on a strategic transformation from a traditional telecom operator (“Telco”) to a technology company (“TechCo”) with 5G as a central driver. Swisscom operates an extensive 5G Non-Standalone (NSA) network covering 99% of the Swiss population. The cloud native platform described here powers the 5G Standalone (SA) core.

Teams

Multiple teams collaborate on this platform:

Cloud Native Resource Orchestration — creates a robust framework for orchestrating cloud-native resources such as CNFs, IPAM, Networks, DNS, Kubernetes Clusters, and more. Designs and operates GitOps pipelines, builds Kubernetes operators, and develops the management cluster UI/UX.
Mobile Cloud Native Engineering — designs, implements, and operates the cloud native 5G core platform, including GitOps pipelines, Kubernetes operators, and network function lifecycle management.
DNS Engineering — builds and operates the highly reliable cloud native DNS service underpinning the 5G core and other infrastructures.
Network Engineering — provides IPAM and Network-as-a-Service.
Platform & Developer Experience — manages Kubernetes clusters and builds developer tooling.

Architecture overview & Goals

Goals

Full GitOps for the 5G Core — Extend GitOps beyond CNF deployment to include network function configuration, certificate management, DNS record provisioning, IP address management, and testing — achieving continuous reconciliation across all layers.
Declarative, Intent-Based Configuration — Replace static, low-level configuration manifests with abstract, intent-driven Custom Resources. Engineers specify what they want using a high level intent (e.g., “this CNF needs an IP address from a subnet in network zone A”) rather than how to achieve it, with Kubernetes operators dynamically assembling configurations at runtime.
Automated CD — High level of automation for telco deployment rollouts. This includes rethinking Change Processes as well as building solid CI/CD/CT pipelines to ensure a highly reliable network.
In-Band with Kubernetes — Bring all automation in-band with the Kubernetes API, eliminating out-of-band tools like Jenkins pipelines and Ansible playbooks. This ensures that the Kubernetes orchestrator has full visibility and control over all resources, enabling self-healing and reconciliation.
Cloud Native DNS Service — Operate a highly reliable, geo-redundant, on-premises DNS service for the 5G core using open-source technologies (CoreDNS, PowerDNS, ExternalDNS), fully automated via GitOps and Kubernetes Custom Resources.
Contribute to the Ecosystem and Shape the Industry Discussion — Open-source key components built or contributed to during this journey (NetBox Operator, SDC, demo code) to enable other organizations to adopt similar patterns. Contribute to Meetups and Conferences in order to achieve broader success of Cloud Native adoption in the Telco Community.

Architecture overview

The platform is organized in three layers:

The Intent Layer stores high-level desired state in Git — engineers define what they want using concise Custom Resources (e.g., a DNN configuration with hostname, region, and SNSSAI).

The Automation Layer runs on Kubernetes and continuously reconciles. Flux pulls intents from Git. Custom operators dynamically assemble full configurations by fetching IP addresses from NetBox (via the NetBox Operator) and creating connectivity in a Network-as-a-Service platform. It then pushes the KRM formatted Runtime Configuration to a Git repository for the Runtime layer to consume.

The Runtime Layer hosts the 5G core CNFs and supporting services. Flux pulls intents from Git. SDC pushes the assembled configuration to CNFs via NETCONF/gNMI. MetalLB, cert-manager, External Secrets Operator, ExternalDNS are used to configure the Workload cluster.

Key Design Principles

GitOps + KRM: Git stores high-level intents; Kubernetes manages dynamic, low-level configuration assembly. This is a shared source of truth across Git and Kubernetes.
Continuous Reconciliation: Every aspect of the system is continuously reconciled against the desired state — following the four OpenGitOps principles.
Abstraction over Complexity: Engineers work with simple, high-level intents; operators handle the complex assembly.

Can you expand on why you are using those projects/services?

From GitOps to KRM

Standard GitOps tools (Flux, Argo CD) combined with Helm/Kustomize have limitations for complex telco use cases: they cannot use live Kubernetes resources as inputs during rendering, cannot invoke custom business logic during template processing, and cannot dynamically assemble configurations from multiple sources. By extending GitOps with the Kubernetes Resource Model (KRM) — Custom Resource Definitions, Custom Resources, and custom Operators — the platform achieves dynamic configuration assembly at runtime.

Configuration Abstraction & Dynamic Assembly

A typical 5G core configuration includes IP addresses, VLAN IDs, DNS records, NF variables, secret references, and certificate references. Previously, engineers had to know all values upfront and embed them statically. The new intent-based model inverts this. The following is an example of a DNN configuration in pseudo-yaml-code.

On the intent layer, only a very stripped-down KRM manifest exists:

apiVersion: telco.swisscom.com/v1
kind: Dnn
spec:
 hostname: "gprs.swisscom.com"
 ipv4: true
 region: ch-east
 type: MobileInternet
 snssai: [1, 10]

This KRM manifest is stored in git and synced to the Management cluster using Flux. From this, the Operators in the automation layer create intermediate resources, in this case an IP Address via NetBox:

apiVersion: netbox.dev/v1
kind: IpAddressClaim
spec:
 parentPrefixSelector:
 region: "ch-east" # from Dnn.spec.region
 family: "IPv4" # from Dnn.spec.ipv4
status:
 ipAddress: 1.2.3.4

The Automation Layer creates the following low level resources for the Runtime Layer:

apiVersion: config.sdcio.dev/v1alpha1
kind: Config
metadata:
 labels:
 config.sdcio.dev/targetName: ch-east  # from Dnn.spec.region
 config.sdcio.dev/targetNamespace: default
spec:
 priority: 10
 config:
 - path: /
 value:
 dnn:
 - name: "gprs.swisscom.com" # from Dnn.spec.hostname
 ip: 1.2.3.4 # from IpAddressClaim.status.ipAddress
 snssai: [1, 10] # from Dnn.spec.snssai
 type: MobileInternet  # from Dnn.spec.type
---
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
spec:
 endpoints:
 - dnsName: "gprs.swisscom.com" # from Dnn.spec.hostname
 recordType: A
 targets:
 - 1.2.3.4 # from IpAddressClaim.status.ipAddress

SDC will now sync the configuration to the 5G CNF and ExternalDNS will create the DNS records in the authoritative PowerDNS backend.

What has worked well?

Intent-based configuration dramatically reduced the complexity engineers face. Instead of managing thousands of interdependent parameters, they work with concise Custom Resources.
Full GitOps reconciliation across all layers (deployment, configuration, DNS, certificates, IPAM), configuration drift is detected and reverted to ensure consistency.
Custom Kubernetes Operators (built with Kubebuilder/controller-runtime) proved to be the right pattern for telco domain-specific concerns, providing full reconciliation support and native KRM integration.
The claim model for IPAM (NetBox Operator) elegantly solved dynamic IP allocation by following established Kubernetes patterns (PVC analogy).
Bringing all automation in-band with Kubernetes gave the orchestrator full visibility and control, enabling self-healing and eliminating the brittleness of out-of-band tools.
DNS resilience engineering — dedicated hackathons, chaos testing, and disaster recovery playbooks significantly improved DNS service reliability.
Cross-team collaboration on a shared platform and KRM patterns accelerated adoption across multiple network domains.

What has not worked well?

NETCONF as a configuration protocol introduces complexity — it requires SDC as an intermediary and prevents fully Kubernetes-native configuration. Ideally, CNF vendors would support native K8s APIs using CRs/CRDs or Secrets/ConfigMaps.
Tooling gap for KRM-based configuration assembly — no mature, community-standard Kubernetes-native tool exists for dynamic configuration hydration. Swisscom had to build custom operators to fill this gap.
GitOps+KRM auditability trade-off — with dynamically assembled configurations, not all state is visible in Git history. The team continues to explore automated intermediary Git layers.
Cumbersome vendor configuration manifests — large, monolithic configuration files from CNF vendors (with ~5,000 interdependent parameters) required significant effort to decompose into intent-based abstractions.
Telco’s imperative model and Kubernetes’ declarative approach do not align well - SDC follows the declarative paradigm, where users define the desired state and SDC determines the actions to achieve it. In contrast, NETCONF/gNMI use an imperative model that requires explicit ordered steps (“do A, then B, then C”). Translating declarative goals into imperative sequences is complex when user‑defined ordering matters, such as for firewall rules where evaluation order affects behaviour. Example: Issue “Support user sorted lists”

What sort of “glue” have you had to develop?

Custom Kubernetes Operators — domain-specific operators for CNF configuration abstraction, config synchronization, IPAM integration, and DNS automation, all scaffolded with Kubebuilder and controller-runtime.
Configuration hydration logic — the CNF Config Operator dynamically assembles full configurations from multiple sources (NetBox, Vault, cluster environment) based on high-level intents.
SDC contributions — significant development work on SDC including early testing of CNF compatibility, configuration validation, monitoring, config blame, drift detection, and NETCONF Actions support.
ExternalDNS NAPTR support — contributed PR #4212 to enable NAPTR record support for SIP phone calls.
Resilient DNS architecture — created a reference architecture and open sourced on GitHub.
Headlamp plugins — Custom Resource plugin.

How did the Architecture Evolve

Journey

The architecture evolved significantly from traditional imperative automation to the current declarative, KRM-based model:

Phase 1 — Ansible + Jenkins: Initial automation used Ansible playbooks triggered by Jenkins pipelines. Configuration was fire-and-forget with no continuous reconciliation.
Phase 2 — GitOps for deployment: Introduced Flux for CNF deployment, but configuration remained out-of-band via Ansible/NETCONF.
Phase 3 — Full GitOps + KRM: Extended GitOps to cover configuration, DNS, IPAM, certificates, and testing. Built custom operators and adopted SDC for config synchronization. Achieved continuous reconciliation across all layers.

Key lessons

De facto GitOps for operators is not true GitOps — creating Helm releases in Git while configuring NFs via NETCONF out-of-band breaks the GitOps model. Bringing configuration into the Kubernetes API was essential.
Bringing everything in-band with Kubernetes enables self-healing, reconciliation, and eliminates the brittleness of out-of-band tools.
Git as the only source of truth is insufficient — the shared source of truth model (Git for intents, Kubernetes for dynamic state) was a deliberate and necessary evolution.
Abstraction is critical — engineers cannot effectively manage 5,000+ parameters directly. Intent-based CRs with dynamic assembly significantly reduced cognitive load and errors.
Custom Kubernetes Operators are the right pattern for domain-specific concerns that existing tools cannot address.
Contribute upstream — local patches create long-term maintenance burden. Swisscom prioritizes upstream contributions (ExternalDNS, SDC, NetBox Operator) for sustainability.

What’s next for your architecture?

Mature SDC Integration — Continue expanding SDC for full lifecycle management with continuous reconciliation via gNMI and NETCONF, including completion of NETCONF Actions support.
Eliminate NETCONF Dependency — Work with CNF vendors to move toward fully Kubernetes-native configuration APIs, reducing reliance on legacy telco protocols.
Advanced Dynamic Configuration Assembly — Develop more sophisticated Kubernetes operators for multi-source configuration hydration, enabling even more complex intent-based workflows across multiple network domains.
Multi-Cluster & Edge Expansion — Scale the architecture to additional edge locations and Kubernetes clusters while maintaining consistent GitOps-driven automation.
Community Tooling for KRM — Contribute toward a mature, Kubernetes-native tool for dynamic configuration assembly that the wider cloud native community can adopt, addressing the current gap in tooling identified during the project.
Resilience & Reliability — Ongoing improvements to cross-cluster redundancy, disaster recovery playbooks, chaos testing framework, and enhanced monitoring/alerting.
Observability & AIOps — Integrate AI-driven operations capabilities leveraging the rich telemetry data from the platform’s monitoring stack.
Cross-Domain Expansion — Extend the orchestration framework to additional network domains and infrastructure services beyond the 5G core, applying the same intent-based automation patterns consistently.

Community Contributions

Contribution	Details
NetBox Operator	Open-sourced under https://github.com/netbox-community/netbox-operator
SDC Contributions	Active contributor to the SDC project (on its path to CNCF incubation)
KRM Demo Code	https://github.com/swisscom/containerdays-2024-krm
Conference Talks	KubeCon EU, ContainerDays, Open Source Summit EU
CNCF/LFN Whitepaper	Co-authored Accelerating Cloud Native in Telco

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: A Cloud Native Scientific Computing Platform for CERN NextGen AI

Mon, 09 Mar 2026 00:00:00 +0000

Relevant Projects

Argo

ArgoCD is used to manage deployments of all services across multiple clusters and environments. Argo Workflows is used to manage multiple day-2 cluster operations.

Bootc

Bootc provides transactional, in-place operating system images and updates using OCI/Docker container images. Bootc is used to build the minimal base images for our cluster nodes.

ContainerSSH

ContainerSSH offers a SSH frontend to containers/pods running on Kubernetes clusters. Used to expose SSH as a way to access existing sessions in the cluster, with multiple authentication mechanisms offered (Kerberos, OIDC/OAuth2).

Longhorn

Longhorn offers cloud native distributed block storage for Kubernetes. Used to offer in-cluster shared storage to users, with individual and team getting dedicated volumes with read-write-multi access and automated backups.

Kubernetes

Kubernetes provides the required workload scheduling and orchestration for the diverse workloads running in our scientific platform.

Kubeflow

Kubeflow offers tools to manage the complete MLOps lifecycle. Used for profile management and quotas for users and teams, instantiation of notebook servers, pipelines for common and reusable tasks, hyper-parameter tuning with Katib and managing inference endpoints.

Kserve

Kserve standardizes hosting of inference endpoints. Used to encapsulate multiple runtime flavors, such as ONNX, Triton and several others, and offering a declarative way to define inference servers.

Kyverno

Kyverno offers policy as code with support for YAML and CEL based policies. Used as a key component for policy enforcement and mutating workloads according to those policies, adding required settings to expose storage systems, set resources based on GPUs, etc.

Kueue

Kueue is a kubernetes-native system offering advanced scheduling capabilities and quota management. Used to provide job queues and quotas, gang scheduling, fair sharing, among other capabilities.

Prometheus

Prometheus gathers the metrics and insights from all components in the cluster. Used for system and service metrics as well as providing individual workload performance insights on cpu, memory, power and other areas.

stargz-snapshotter

stargz-snapshotter provides lazy pulling of container images. Used to handle efficient job start and execution with image sizes of over 100GB in some cases.

Some additional projects outside the CNCF are essential to this deployment.

MLFlow, a project under the Linux Foundation used for model management and versioning
Nvidia GPU operator to setup and manage drivers and configurations for Nvidia GPUs
AMD GPU operator to setup and manage drivers and configurations for AMD GPUs

TLDR; Synopsis

This reference architecture describes a deployment supporting multiple teams in CERN’s new flagship “Next Generation Triggers” project, looking at innovative computing technologies for data acquisition and processing for the High-Luminosity Large Hadron Collider and beyond.

The cluster and platform target:

Multiple scientific use cases, covering traditional numerical computing as well as machine learning workloads across the different CERN experiments. Scientific computing and in particular high performance computing (HPC) have relied on cloud native tooling for parts of their workloads for several years, but relied on tools like SLURM for advanced scheduling capabilities. This is our first production deployment to offer the full stack based only on cloud native infrastructure
Access to both accelerators (in particular GPUs) as well as specialized nodes such as high CPU core count and high CPU clock frequency. In an earlier stage of experimentation also FPGAs are being integrated targeting fast inference in the CERN experiment online filters
In a shared pool of resources allow interactive access (including SSH, VSCode, Notebooks and just kubectl), traditional batch, MPI and training workloads and inference
Integration with the existing infrastructure at CERN for CI / CD systems (on-premises GitLab and GitHub), identity, and efficient access to multiple storage systems for both user and physics data

A pure cloud native based infrastructure can now successfully serve scientific computing workloads, with advanced scheduling features such as co-scheduling, fair sharing, among others.

Use Cases & Requirements

A set of requirements and use cases was initially defined when designing the architecture. The figure below shows how a shared pool of resources, mostly on-premises but integrating public cloud and supercomputing centers, should be accessed from different services.

Below we highlight specific requirements in terms of hardware and user facing functionality.

Hardware

Support for an heterogeneous set of resources: multiple CPU types, GPUs from multiple vendors, FPGAs and specialized accelerators, all in a shared pool of resources
Integration with multiple network interconnects targeting low latency, including at least Infiniband and RDMA over Converged Ethernet (RoCEv2)
A hybrid deployment integrating external resources, both from public cloud providers and supercomputing centers

User Facing

Curated environments based on container images and maintained by the platform team for the most common user setups, covering ML workloads but also traditional scientific computing. Particularly important has been ensuring these environments are compatible with the existing ways of working, with session environments setup with backwards compatibility for existing physicist tools and scripts
Easily customizable environments, either via dedicated environments maintained by user teams or the ability to install additional packages at runtime. This means users have sudo capabilities inside their sessions
Interactive access to sessions with the ability to choose the amount of GPUs at creation, and a corresponding CPU and memory allocation depending on the type of GPU selected. Once created, access available to the session via notebooks, local vscode instances and most importantly SSH for compatibility with the existing ways of working
Batch access to resources, with support for advanced scheduling capabilities such as queues, quotas, co-scheduling, fair sharing. In addition to the high priority user submissions of training or MPI jobs, the system should be able to backfill unused resources with lower priority workloads to ensure high usage efficiency
Support for the complete machine learning lifecycle, including data preparation, training, hyper-parameter tuning and model inference. In particular, support, efficient integration and automation using common training and tuning frameworks
Model management and versioning, integrated with the rest of the platform with collection and storage of training metadata and logging

Architecture

The diagram below shows how the different projects and tools match the requirements.

Areas of particular interest where effort was required include compute, scheduling, networking, storage and observability.

Compute

Proper isolation and reproducibility is essential for reliable performance and results, removing the effect of noisy neighbors and the latency between CPU and GPU. GPU nodes follow a NUMA-aware dual-socket layout, designed to preserve locality between CPU, memory, and accelerator resources. Each node has two CPU sockets, exposed as two NUMA nodes.

Depending on the node type, GPUs are distributed evenly across these NUMA domains: either 8 GPUs per node, with 4 GPUs attached to each NUMA node, or 4 GPUs per node, with 2 GPUs attached to each NUMA node.

Some relevant configurations to ensure the desired reproducibility and isolation.

CPU and memory resource allocations (requests and limits) scale with the number of GPUs requested by a session: pods receive resources in proportion to the selected GPU count while remaining aligned with the corresponding NUMA locality. This minimizes cross-socket communication, reduces latency between CPU and GPU, and improves the consistency of performance-sensitive workloads

Control CPU Management Policies on the Node, as documented here with the following settings on the kubelet.

cpu-manager-policy=static
cpu-manager-policy-options=full-pcpus-only=true
memory-manager-policy=Static
topology-manager-policy=restricted

Reserved systems resources for kubelet and other add-ons.

system-reserved=cpu=2,memory=1000Mi
reserved-memory=0:memory=1000Mi

Efficient access and distribution of container images, to accelerate the start of sessions based on both curated and custom environments each being multiple 10s of GBs in size. We provide this with a custom daemonset pre-pulling all curated images in advance when published, as well as the ability to do image streaming with the stargz-snapshotter.

Capability to burst out to external resources, in particular public cloud providers and HPC resources.

Scheduling

Bin packing in the scheduling profile instead of the default workload spread across nodes, with strategy MostAllocated ensuring better availability for workloads requiring full nodes.

Advanced scheduling features for queues supporting different resource types and QoS, workload co-scheduling, quotas and fair sharing to optimize overall resource utilization.
Kueue is the main component being used to achieve the advanced scheduling functionality we need.

Networking

Low latency networking such as Infiniband and RDMA over Converged Ethernet (RoCEv2) supporting both traditional CPU and GPU MPI workloads. Currently done by enabling hostNetwork and exposing the corresponding PCI devices for these specific use cases. Driver and lifecycle management of IB/RoCEv2 networking resources is controlled using the Nvidia network operator.

Storage

Users get different storage tiers which fit different usages.

Node local

Very high IOPS but limited space, typically on the low TBs available to all workloads on that node. Not useful for multi-node jobs requiring a shared filesystem. Used also for GPU Direct Storage (GDS) with local NVMEs.

Cluster local

Shared filesystem across all nodes in the cluster, deployed using Longhorn. Limiting the number of network hops as much as possible ensures reasonable IOPS and scales out well in space available with the number of nodes in the cluster (typically 10s of TBs per node). Connection through a single switch for higher performance, as much as possible. Volumes stored in this filesystem are backed up to S3 storage relying on the internal Longhorn backup functionality, with incremental points daily for a week and monthly.

Central

Shared filesystem outside the cluster with much higher storage space available. Managed using CEPH with different IOPS available, up to 2000 guaranteed with bursting to higher values.

Observability

The stack provides visibility into hardware performance, resource efficiency, and environmental impact.

Telemetry Collection

Leveraging a multi-layered collection strategy integrated with the kube-prometheus-stack.

Accelerators: NVIDIA dcgm-exporter and AMD device-metrics-exporter provide deep-field GPU telemetry (utilization, memory, power, temperature, and frequency).

Power & Sustainability: IPMI and Kepler capture hardware-level power metrics. Kepler utilizes RAPL to attribute energy consumption to individual workloads.

System Metrics: Standardized node and container metrics are ingested via Prometheus for a unified view of the cluster.

Visualization and Analysis

Data is exposed via Grafana through three specialized dashboard tiers.

Cluster Overview: Tracks aggregate utilization (CPU, GPU, RAM, Network, Thermals) and node-level health. It highlights idle resources and historical trends to guide capacity planning.

User/Workload Analytics: Provides namespace-filtered views for individual developers to monitor their specific deployments. This view balances resource efficiency (allocated vs. actual usage) with performance profiling (GPU/CPU/RAM saturation) and power consumption, allowing users to independently debug bottlenecks and optimize job performance.

Sustainability Tracking: A dedicated dashboard for CO2-equivalent emissions, offering transparency into the carbon footprint at both the cluster and individual workload levels.

Alerting and Optimization

Alertmanager is configured to trigger notifications for idle resources. By monitoring the delta between allocated requests and actual utilization, the system identifies “zombie” workloads or over-provisioned namespaces, allowing for potential automated or manual resource reclamation to reduce costs and energy waste.

What works particularly well

Workload isolation which is a key aspect when considering needs for reliable benchmarking results. Recent versions of Kubernetes have all the required capabilities to ensure NUMA affinity between CPUs and GPUs, resource pinning to individual workloads and reservation for system services and add-ons.

GPU setup, configuration and monitoring with well supported and up to date operators for both Nvidia and AMD GPUs and automation for metric collection on utilization, power, memory, etc. This includes the initial node configuration required with loading drivers and exposing them to the workloads, as well as day-2 operations such as driver upgrades with integration with the default methods for cordoning and draining nodes.

Kyverno for validation and mutation of cluster resources, allowing a policy based mutation of the resource capabilities based on labels available to users. This ranges from attaching volumes for access to external storage, setting environment variables such as home directories or authentication, automation of resources for cpu and memory and many others. Validation policies also include ensuring users do not attempt invalid NUMA allocations of CPUs and GPUs. Kyverno was chosen after the initial choice of the OPA Gatekeeper had limitations when modifying fields outside the matching location.

What needs improvement

GPU failure detection and integration with the scheduler, either by cordoning nodes or blocking access to faulty GPUs. Depending on the type of fault, the device plugins (for both Nvidia and AMD) may stop exposing faulty devices, but this is not reliable in all cases. Options such as Nvidia Sentinel are being evaluated.

GPU partitioning currently at node level, limiting the ability to have in the same node devices being exposed fully and others being partitioned using MIG. This is currently not supported by the GPU operators, but should be available in the future with the DRA drivers.

Scheduling workloads across multiple clusters, while possible, does not allow seamless access to logs or launching interactive sessions as done for single clusters - `kubectl log` and `kubectl exec` type of request. This is ongoing work in Kueue but currently limits the workloads submitted outside the main cluster to batch-like workloads.

Limited support for checkpoint and restore in several types of workloads, in particular the non machine-learning workloads. This limits the ability to push overall usage of the cluster further up by suspending / preempting idle sessions without losing any work. Efforts such as criu and the checkpoint-restore working group promise to greatly advance the capabilities of the cloud native ecosystem in this area in an workload agnostic way.

Low latency networking with InfiniBand or RoCEv2 in our setup is currently not namespaced and exposed to the users through hostNetwork. In case user workloads are not trusted, other options that provide better network-level isolation should be explored, including SR-IOV and via efforts such as dranet.

What sort of “glue” have you had to develop?

A key goal of our architecture was ensure the complete functionality is available via cloud native APIs, easing the integration with all other tools in the ecosystem. The glue pieces below target ease of use.

Access via SSH was one of the main requests from our users, allowing backwards compatibility with years of custom scripts, continuous integration and several other “ways of working” that require this type of access. We invested internally in developing the required capabilities in the containerssh project, with management of multiple sessions, multiple authenticated methods (OAuth2, Kerberos, X509), among others.

Large number of mutating policies, allowing us to give a better experience to users that do not want to use kubectl or write yaml. Relying on metadata labels in the different resources hides the complexity of setting up volume mounts, environment variables, etc. Our current policies include setting tolerations to assign workloads to specific node flavors, additional environment configurations for MPI workloads, injecting user metadata to access storage systems and interact with internal services, mounting multiple storage systems at CERN or enabling RDMA and GPU Direct Storage.

What’s next for your architecture?

Interactive session management via notebooks, relying on the Kubeflow Notebooks UI. As of today, users require a minimal yaml and usage of the kubectl client to create, list and delete their interactive sessions, even if access is then available via ssh, notebooks, vscode, etc. An upcoming improvement is to offer a UI based interface to manage sessions, likely relying on the Kubeflow Notebook UI but applying to any type of workload.

DRA and automated partitioning in the cluster, as currently we still rely on the Nvidia and AMD operators to manage GPU resources for this particular setup and need to manually set the desired MIG configuration for each node/pool of nodes. This will allow us to have heterogeneous configurations in the same node (with both partitioned and non-partitioned devices) as well as, in the future when the DRA drivers get this functionality, automatic partitioning of devices based on the current workloads.

Bursting to HPC resources, as existing supercomputers and upcoming AI factories have a large number of available GPUs. The main requirement is to integrate with SLURM as an API to manage these remote resources, but in a way that is seamless to users of the service. Projects such as interLink promise to hide the SLURM backends behind the Kubernetes APIs in our platform.

Key Takeaways / Lessons

Adapt to existing ways of working: the success of the platform depends on acceptance by users, who often will not have the time to change their ways of working. Anticipate where effort is needed to meet users where they’re at, building the required glue on top of your cloud native infrastructure.

Iterative and quick development: when exposing a new platform to users with so many stack changes from previous deployments, the ability to iterate very quickly taking into account user feedback is essential. This likely means planning for an intense period after first exposing the services, with the risk of loosing users from the start otherwise.

Upstream first: this is only way to ensure long term sustainability of a platform, exposing requirements and working together with the rest of the community. Local, temporary patches, when required, should be done in parallel with the upstream contributions.

Cloud native is ready for scientific computing and AI/ML: if there were doubts, this experience cleared them up. Cloud native enables the next generation of scientific computing and AI/ML platforms, with all the advanced requirements from high performance computing together with the integration with all modern tools that talk cloud native.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: A modern and sovereign Private Cloud «Kubernetes Service» for Swiss-based enterprises.

Wed, 04 Mar 2026 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2024
Current version: 1.32.8 (CNIP)
Current version: 1.31.x - 1.34.x (SKP)

Kubernetes enables high availability, scalability, and performance for infrastructure, offering a centralized and policy-driven platform to manage network and service data supporting Managed Kubernetes for our cloud customers.

KubeVirt

Using since: 2024
Current version: v1.5.0 (CNIP)

Cluster resources are constructed using KubeVirt for virtual machine abstraction of Control Plane and Worker instances.

Kube-OVN

Using since: 2024
Current version: v1.13.14 (CNIP)

Kube-OVN is utilized as network stack of the infrastructure cluster to enable intra-cluster/east-west network communication of user clusters. It enables a policy-driven security model as well as customer network isolation using VPCs.

MetalLB

Using since: 2024
Current version: v0.15.3

MetalLB is an integral component of the infrastructure deployment process, offering automated access to the framework that provisions individual user cluster resources on bare metal Kubernetes environments.

Container Storage Interface (CSI)

Using since: 2024
Current version: v25.06.3 (trident-csi)
Current version: v0.4.5 (kubevirt-csi)

Kubevirt-CSI is the standard storage interface for persistent volumes in user clusters. Trident-CSI manages NetApp storage requests and can also be used directly to integrate with Swisscom’s File Service Kubernetes, which provides iSCSI and NFS shared storage across all Availability Zones.

Kyverno

Using since: 2024
Current version: v1.13.4 (CNIP)

Kyverno serves as the default policy engine for infrastructure and user clusters, providing robust security constraints. In addition to Kyverno, also Chainsaw (a Kyverno sub-project) is used for automated, declarative e2e testing.

ArgoCD

Using since: 2024
Current version: v3.2.0 (CNIP)

ArgoCD allows us to deliver comprehensive infrastructure using a fully automated GitOps methodology.

Helm

Using since: 2024
Current version: v3.5.1 (CNIP)

Helm automates the creation, packaging, configuration, and deployment of Kubernetes applications by creating reusable charts.

CloudNativePG

Using since: 2024
Current version: v1.27.0 (CNIP)

CloudNativePG (CNPG) manages PostgreSQL databases in cloud-native environments. It handles the full lifecycle of highly available PostgreSQL clusters (primary/standby with native streaming replication), including declarative deployment, scaling, backups, self-healing, failover and monitoring.

Describe your organisation

Swisscom is the leading ICT company in Switzerland and offers mobile, Internet and TV, as well as comprehensive IT and digital services to private and business customers. Swisscom’s expertise in cloud native technologies is well-established, as evidenced by its status as a former Gold member and Management Board member of the Cloud Foundry Foundation, along with its certification for Cloud Foundry. Additionally, Swisscom demonstrates a strong commitment to the Open-Source community, having been a CNCF Silver Member for several years and serving as a Kubernetes Certified Service Provider (KCSP) partner. Our skilled employees have delivered numerous speeches and presentations at prestigious events such as KubeCon, Cloud Native Zürich, Swiss Cloud Native Day, KCD Suisse Romande, ContainerDays, among others.

Our next generation Private Cloud Container as a Service offering «Kubernetes Service» for the B2B market addresses customer’s need for scalable and highly available Kubernetes workload as a flexible and secure IT foundation. It is part of our Swiss-based Enterprise Service Cloud (ESC) market channel as a sovereign, Private Cloud Kubernetes offering for effortless provisioning and usage of our customer’s container workloads.

Describe your entity and/or team

The development and delivery of the new «Kubernetes Service» is done at within Swisscom’s IT-Clouds Value Stream and shared across two teams:

Pathfinders: responsible for the Cloud Native Infrastructure Platform (CNIP). CNIP handles the creation, delivery, and lifecycle management of the KubeVirt-based virtual machines (VMs). These VMs function as nodes for both the Control Plane and Workers. The VMs are ephemeral and can be re-created immediately in case of any failure. They are solely used to enable container-based workloads and do not act as standalone VMs.
Guardians: responsible for the Swisscom Kubernetes Platform (SKP), which runs on top of CNIP. It consists of the installation of Kubermatic Kubernetes Platform (KKP) for the customer tenant (environment) and the setup and support of the highly available Control Plane for any customer (user) cluster.

The layered approach allows Swisscom to manage technological aspects distinctly by segregating the cloud native infrastructure (managed by the Pathfinders team) from the Kubernetes platform (managed by the Guardians team). This strategy ensures considerable flexibility, permitting each layer to be combined or integrated with other technologies in the future.

Brief overview of your architecture and any potential goals you are trying to achieve with it?

Summary

Kubernetes Service is the successor to our current container offering, representing a significant shift towards a more cloud-native approach using advanced Open-Source technology. Currently bound to a vendor-specific implementation, Swisscom has opted to employ open-source tools for the development of cloud native products for customer use. This strategy aims to minimize dependencies and mitigate the risk of vendor lock-ins.

By adopting this architecture, Swisscom can uphold quality within the cloud native domain while maintaining a competitive pricing model due to reduced reliance on external licensing and subscription models. Furthermore, having the ability to develop, maintain, and operate all components internally enhances our decision-making processes and strengthens our roadmap capabilities.

Another important point is that our customers’ data will always remain within Switzerland and under Swiss law. Since we fully own the platform and do not rely on any external vendors, we can confidently guarantee true data sovereignty, hosted entirely on our premises without relying on vague marketing claims. Additionally, because Swisscom is not subject to the US Cloud Act or similar foreign regulations, no non-Swiss legislation can access the data.

Brief overview of architecture

A simplified high-level diagram describes Kubernetes Service, including multi-tenancy and security aspects:

As illustrated in the figure, two separate and independent user tenants, BLUE and RED, are established on shared resources (depicted in yellow), managed by the Kubernetes Infrastructure Cluster. The foundation for all virtual abstractions is the Consolidated Infrastructure (COI) in Swisscom’s data centers.

Each customer-specific environment comprises a management zone (MGMT Zone) and a workload zone. These zones address shared responsibilities, where Swisscom provides the Control Plane for each customer’s environment (illustrated in blue and orange in the next figure).

Customers have the flexibility to deploy workloads within the workload zone independently of the management resources as required. Furthermore, each customer is able to maintain multiple environments. This provides an alternative method for segregating workloads at the tenant level instead of the Kubernetes cluster level, thereby ensuring comprehensive isolation from the outset.

Goals and objectives

One of the primary objectives of the product refresh is to offer more desired features to customers. Compared to the current offering, enhancements include:

Upstream Kubernetes versions with faster updates
Node Autoscaling
Integrated Backup functionalities
Native Kubernetes Load Balancer
Modern customer self-service portal
Additional Kubernetes add-ons available via Application Catalog

Moreover, additional options are directly available to our customers:

Choose from different Container Network Interfaces (CNI)
Access persistent storage through kubevirt-csi

With KubeVirt providing abstraction, KVM is employed as the hypervisor on bare-metal servers. From the customer’s perspective (Customer X), the administrator of their user cluster manages all selections and abstractions shown in the figure below, enabling customers to make independent decisions, e.g. choosing a default CNI from the available options (Cilium, Canal, None).

In addition to technical improvements, we aimed to minimise reliance on external vendors and build a truly sovereign cloud solution that can compete with Public Cloud offerings, free from outside service dependencies. Our goal is for customers to run their Kubernetes workloads in our sovereign ESC Cloud, providing a comprehensive alternative to US hyperscalers - in terms of functionality and, most importantly, data privacy.

Can you expand on why you are using those projects/services?

Cloud-Native Implementation: Utilized CNCF projects and technologies to deploy a comprehensive stack consistent with a microservices-based architecture, resulting in enhanced scalability and operational agility.
Kubernetes for Orchestration: Adopted Kubernetes to manage containerized workloads, enabling automated deployment, scaling, and resilience on management as well as user cluster level.
Kube-OVN as network layer: With Kube-OVN as CNI on the infrastructure clusters and it’s VPC functionality, it allows customer environments to be fully segregated on a shared platform, providing maximum flexibility and strong security enforcement. It enables the teams to use familiar cloud-native development, operations, and debugging tools and skills.
KubeVirt for VM abstraction: A high-quality, Kubernetes-native virtual machine abstraction facilitates the deployment of container-based resources on a centralized cloud-native infrastructure platform, while maintaining flexibility for future use of VM resources.
Open-Source & Cost Efficiency: CNCF components deliver vendor-neutral, cost-effective solutions that are fundamental to container orchestration and observability. These open-source tools form the foundation of our sovereign cloud initiative, empowering internal teams to design customized architectures independently of third-party vendors. Utilizing CNCF technologies allows us to maintain flexibility, scalability, and comprehensive control over our cloud infrastructure, supporting our strategic objectives of autonomy and innovation.
Declarative & Configuration-Driven Approach: CNCF tools align with the low-code/no-code principle by enabling declarative configuration management.

What has worked well?

The implementation has eventually lead to the product launch of Kubernetes Service in August 2025, with some strong outcomes:

Layered Architecture for Enhanced Robustness: The integration of Cloud Native Infrastructure Platform (CNIP) and Swisscom Kubernetes Platform (SKP) forms the foundation of the new Kubernetes Service, enabling flexible handling as separate platform layers for streamlined future operations.
Vendor-Agnostic Production Platform: By eliminating proprietary technology, a resilient and adaptable foundation has been established to host managed Kubernetes clusters within Swisscom’s Private Cloud, ensuring a high degree of flexibility and scalability, as well as privacy.
Modern Cloud-Native Foundation: The implementation of Kubernetes to deliver managed Kubernetes clusters to end customers enables a unified cloud-native stack across all layers of responsibility, promoting consistency and efficiency.
Best Practice Design: Collaborating with Kubermatic, a modern Kubernetes platform was designed, incorporating the latest technologies such as KubeVirt and Kube-OVN, to ensure an enterprise-ready solution for end customers.
Operational Excellence: Equipping teams with essential cloud-native and Kubernetes expertise enhances the attractiveness of Swisscom’s tech stack to potential candidates and reinforces the company’s commitment to the Open-Source community.
Successful Internal Adoption: The Kubernetes Service was successfully launched as Swisscom’s internal Container platform, achieving significant traction with over 60% of workloads migrated within the first 9 months of operation.

What has not worked well?

While the architecture delivered significant improvements, several challenges emerged during implementation:

Enterprise-Readiness of Cloud Native Technologies: Despite successful scaling in test and internal production environments, many advanced cloud-native technologies faced difficulties when deployed in enterprise-grade settings for end customers (e.g., B2B market). This highlighted the need for further refinement and testing in real-world scenarios.

For KubeVirt, for instance, there are only limited real-world examples, best practices or reference designs available to draw upon for large-scale, production-grade business deployments. Switching fully to Kube-OVN as the main network layer also demands extra effort and is less straightforward than traditional network solutions with established production lifecycles.
Limited Support and Professional Services: The availability of professional support, particularly 24/7, for open-source and cloud-native technologies is limited. This poses challenges for enterprises seeking to adopt these technologies and provide services with guaranteed service levels (SLAs).

A possible solution is for more companies to provide professional support and make these services more transparent. Furthermore, the CNCF could introduce a “Certified Supporter” verification system to strengthen trust in firms that offer professional support.
Knowledge Gaps and Skills Requirements: Adopting new technologies demands specialized knowledge and expertise. In-house engineers required additional training and support to effectively maintain and troubleshoot products built on these technologies.
Customer Acceptance and Migration Challenges: Introducing a new platform based on modern technologies, without a proven track record in enterprise-grade deployments, required significant effort to educate customers, facilitate migration from legacy stacks, and promote the benefits of a sovereign cloud solution. This process demanded substantial resources and support to ensure a smooth transition.

What sort of “glue” have you had to develop to enable usage of your architecture ?

The reference architecture provides a strong foundation, making it practical and easy to use. The below elements were designed to simplify adoption, improve usability, and ensure seamless interaction across layers acting as the “glue”:

Unified Abstraction APIs: Developed APIs (Open Service Broker spec) that hide complexity and provide a consistent interface for orchestration and other consuming Operational Support Systems (OSS).
Advanced Routing Functionality: In order to integrate the customer environments into the Swisscom Core Network (MPLS), we developed and implemented our own concept of edge routers using BGP on FRRouting pods. This custom solution supports NAT, Fail-over (VRRP) as well as north-south firewalling (traffic from/to customer environments). These router pods are managed by an operator and configured with custom resource definitions.
Policy Integration Layer: Built operators to dynamically apply and manage Kyverno policies across different stages without requiring deep technical intervention.
Firewall Management: Implemented operators and API endpoints to allow customers to manage firewall rules on the SDN layer of the KubeVirt infrastructure, via Kube-OVN network policies.
Workflow Orchestration Logic: Developed and implemented the entire platform orchestration logic and automated pipelines from bottom-up.
Commandline tooling: Various commandline tools for human operators to manage and control the entire platform and all parts of it with ease.
Testing & Fine-tuning: With limited experience in large-scale bare-metal Kubernetes deployments, we had to do a lot of testing, validation and fine-tuning. We had to make sure, that the platform scales properly with more workloads being migrated every day.

Has your architecture evolved? What lessons have you learned from previous iterations?

Our architecture and product have undergone significant evolution through iterative development, driven by customer feedback and emerging requirements.

Iterative Development Approach: We began by establishing foundational layers and meeting the needs of our internal Swisscom customers. Subsequent iterations introduced advanced features for end customers, incorporating feedback from both internal and external stakeholders.
Continuous Improvement and Feedback Loop: Each iteration allowed us to gather valuable insights and add new functionalities, refining our product and enhancing customer satisfaction.
Steep Learning Curve and Expertise Development: As we ramped up the product, our teams faced a significant learning curve, developing essential expertise and professionalizing DevOps processes to ensure seamless operation.
Strategic Partnerships and Support: Our collaboration with Kubermatic enabled us to leverage professional support for key components, including KubeVirt and Kube-OVN, ultimately maturing our production platform and solidifying its readiness for enterprise-grade deployments.

Through this iterative process, we’ve gained valuable lessons and refined our architecture to better meet the needs of our customers, while developing the expertise and partnerships necessary to drive continued success.

Outcome

By embracing open-source and cloud native technologies, Swisscom successfully created a sovereign cloud solution, modernizing its container offering while reducing vendor lock-in and providing advanced features to customers. The new «Kubernetes Service» demonstrates the power of cloud native architectures in creating flexible, scalable, and cost-effective solutions for enterprise-grade services, all while ensuring true data sovereignty and regulatory compliance. This approach positions Swisscom as a leader in sovereign cloud solutions, offering Swiss (and European) customers a trusted alternative to global hyperscalers.

What’s next for your architecture? What are you looking to do next?

Building on the success of our proven reference architecture, which now supports both internal and external customer workloads in production, we’re focused on expanding and enhancing our offerings:

Hybrid Cloud Expansion and Multi-Cloud Flexibility: We’re working to enable seamless public cloud deployments, complementing our existing Swiss-based data centers and strengthening hybrid cloud use cases.
Edge Cloud Support: With cloud sovereignty in mind, we are developing a «Kubernetes Service On-Prem» extension that will deliver the Private Cloud product on a Cloud Edge Stack at customer premises, enabling an autonomous instance of our Kubernetes Service. This is currently in development with an interested customer.
GPU-Enabled Workloads and Emerging Technologies: Next, we’ll be integrating GPU support and exploring other emerging technologies to unlock new possibilities for compute-intensive applications.
Customer-Driven Features and Enhancements: We’re committed to delivering additional features and functionalities requested by our customers, further enriching our platform and services.
Simplified Onboarding and Resource Optimization: To improve efficiency and resource utilization, we’ll be introducing a shared cluster concept, allowing for more flexible and efficient use of our bare-metal infrastructure.
Exploring New Use Cases - VM Workloads: We’re also investigating the possibility of hosting classical VM workloads on our Cloud Native Infrastructure Platform (CNIP), expanding the platform’s use cases beyond container-based workloads and further increasing its versatility.

By pursuing these initiatives, we aim to continue delivering value to our customers, drive innovation, and grow our architecture and services to meet evolving needs.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: Enabling Allianz Direct's scaling through Platform Engineering

Mon, 28 Oct 2024 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2021
Current version: 1.29.6

Kubernetes has been the foundation for our Internal Developer Platform and almost all of Adobe’s containerized workloads run on Kubernetes clusters.

Helm

Using since: 2021
Current version: v3.12.3

Helm is our package manager and helps us abstract out some of the complexity by including them as dependencies and only exposing the values.yaml file.

Argo

Using since: 2021

We are using all four projects under Argo:

Argo CD: v2.9.22

Platform Engineering @ Allianz Direct

Allianz Direct has been working on building Platform Engineering for the past several years. We have created a cloud-native future-proof platform which offers an organisation-wide-devops(orgOps) experience on top of public cloud. Business, Software Engineers, Security, Compliance, Architecture, Support and Cost Efficiency are few of our stakeholders.

Platform Engineering Team Topology - internal

The PE team is internally organised in DDOs - Distributed DevOps, where each ddo owns a Domain. This setup follows the reverse Conway manoeuver where the reference architecture structure dictates the teams setup. The PE team consist of 13 members. All of them internal employees.

The PE team is being supported by auxiliary teams: Developer Service Hub (DSH): entry level engineers handling the low complexity tasks such as: access management, 1st level technical support, SRE onboarding, ticket sorting and filtering Cloud Auditing Hub (CAH): entry level engineers with a focus on compliance and audit Production Management Hub (PMH): entry level engineers with a focus on incident and problem management Technical Tribe Leads (TTL): Technical leads on the Tribe (department) level that became our evangelist for best practices and continuous innovation partners.

Platform Engineering Team Topology - external

Externally, the team has decided to scale by building on the concept of FDOs - Federated DevOps, where each squad (developing team) has built internal devops expertise focusing a minimum amount of time on Operations concerns. A FDO community has flourished and enabled successfully an extremely aggressive organisation scale up The PE serves an approx. 500 engineers and 1500 employees.

Platform Engineering Architecture

The PE team has built a Cloud Native Platform that serves the core B2C business of a fully digital insurance player acting on several european markets.

It serves also all the newly created B2B business initiatives. The highly aggressive scale-up of the both B2B and B2B businesses forced the PE team to redesign the architecture towards a multi-tenant architecture with minimum operational effort. On top, the Platform has become a Organisation Group Standard and an example for Global initiatives. This forced the PE team to choose a light architecture under a second Platform based on Fargate ECS, reducing the operational effort and talent demand to minimum.

PE team delivered several cloud native best practices:

GitOps (ORGops to be more precise)
Stateless clusters orchestrated by a management cluster
Fargate ECS multitenancy
DevSecOps
SRE

Special projects that served us well

AWS obsession with customers served us well. The technical support and continuous relation with AWS architects helped us maximise our effectiveness and evolve our architecture as well as complementing our stack with SaaS services such as EKS, Redis, MSK.
AWS managed Kubernetes gave us the flexibility we needed to meet our users needs with as little overhead as possible. Most of our application choices come directly from our users, however our infrastructure choices were made to balance our compliance needs, usability and supportability.
ArgoCD has become instrumental for us and the core of our internal innovation. We are currently using it for spinning up and managing our multitenant implementation whether that is on namespace level or Fargate ECS level.
PAGT(Prometheus, Alert manager, Grafana and Thanos) stack is the core of our Observability. It is complemented by OTEL and ELK Stack and enabled us to build a state of the art SRE journey for all our Stakeholders

Things that worked well for us

EKS and now EKS + Fargate ECS works great for us. We can fully owns our core domains and stay hardened with minimum effort. Complementing our Stack with AWS services especially on Persistence, Data Streaming and Storage increase our innovation speed. ArgoCD with helm, Atlantis with terraform and Crossplane are working great for us, allowing us to follow the GitOps way, a great way of working for heavily regulated industries.

Stakeholders value streams

Business

Cost savings of X%
Business Delivery expedited by X%
Security incidents reduced by X%
Reliability incidents reduced by X%
Organizational Performance increased by X%

End-Customers

User Experience improved by X%
Fast Feedback loop increased by X%
Reliability increased by X%

Developers

DevX improved by X%
Fast Software Delivery improved by X%
Productivity increased by X%
High Quality increased by X%
Innovation increased by X%

Security

Cloud native Security costs reduced by X%

Compliance

Cloud compliance costs reduced by X%

Legal

Cloud native legal cost reduced by X%

QA

Automation increased by X%

FinOps/Sustainability

Cloud Costs reduced by X%

Some things did not work as expected

Service Mesh did not work well for us. The Operational costs and lack of integration on the CNI level made us revert the journey. Sidecar, eBPF Service Mesh seems to be the way forward for us.

Architecture evolution

The Architecture was designed as a single tenant but due to its success it needed to evolve to support multitenancy on AWS account level, next on namespace level and now, due to the Global impact, on the Fargate ECS level. Inevitably we ended with three Platforms(Architectures) that need to be simplify towards a maximum two that will evolve rapidly. Migration work is inevitable.

The Journey

It took us 4 years to move towards the current Platform Engineering setup. There is more to come. The full Journey as it happened till today 28.10.2024 can be seen in the following diagram:

Architectures: Scaling Adobe’s Service Delivery Foundation with a Cell-based Architecture

Fri, 11 Oct 2024 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2019
Current version: 1.29.6

Kubernetes has been the foundation for our Internal Developer Platform and almost all of Adobe’s containerized workloads run on Kubernetes clusters.

Helm

Using since: 2019
Current version: v3.12.3

Helm is our package manager and helps us abstract out some of the complexity by including them as dependencies and only exposing the values.yaml file.

Argo

Using since: 2022

We are using all four projects under Argo:

Argo CD: v2.9.22
Argo Workflow: v3.4.6
Argo Events: v1.9.0
Argo Rollouts v1.6.0

Backstage

Using since: 2023
Current version: TBD

Backstage is the backbone for our unified internal developer portal.

Developer Platforms @ Adobe

Adobe has been working on building an Internal Developer Platform (IDP) for the past several years. We have a core infrastructure layer which offers a compute layer on top of cloud providers such as AWS and Azure. On top of the infrastructure layer, we provide an end-to-end developer experience which helps developers in different phases of SDLC.

Security, Compliance, Support and Cost Efficiency are cross-cutting concerns for both infrastructure and developer experience.

Developer Experience / Flex @ Adobe

Adobe offers a rich developer experience to its developers to enable them to write better software faster. This includes a seamless experience for all phases of their application lifecycle: Concept → Code → Cloud → Customer. Flex is the codename for Adobe’s GitOps-based Service Delivery foundation.

Concept → Code: Adobe offers standardized bootstrappable golden templates via the unified developer portal (based on Backstage), which provides a guided path that generates boilerplate code (with best practices built-in) for the app and infrastructure.
- The promise is that you should be able to run and test the app locally right after code generation.
- The generated infrastructure includes, among other things, the manifests for the service’s CI/CD pipeline.
- Toolchain - The developer portal is based on the industry-standard open-source tool Backstage.
Code → Cloud: Adobe offers customizable and flexible GitOps-based CI/CD pipelines (based on Argo projects) to developers.
- The developers start with a paved path but Flex allows users to customize their CI/CD pipeline if needed, including adding new steps or changing the pipeline DAG.
- The CI/CD pipeline does just-in-time provisioning during deployments i.e. it provisions necessary resources to make deployments successful. For example, DNS endpoints, K8s namespaces, Argo CD apps etc.
- Advanced deployment strategies like Canary, Blue-Green etc. are supported out of the box.
- Toolchain - We use industry-standard open-source Argo projects: Argo CD, Argo Workflows, Argo Events, and Argo Rollouts.
Cloud → Customer: Adobe offers a “Single pane of glass” developer experience (called Flexperience) in the developer portal for managing the application.
- Toolchain - The experience is provided as a Backstage plug-in in the developer portal.
Control Plane – The control plane for the service delivery workflow is based on AWS EKS clusters.

Flex has had rapid organic growth in the last 2 years since its launch. Here are some numbers that represent the scale that we are handling (as of Oct 2024):

> 360 remote K8s clusters
> 22K Argo CD apps
> 30K deployments per month
> 1K services in production

And these numbers are increasing by the day.

In this article, we will cover Flex’s initial architecture, its challenges, and the need for a re-architecture. Then, we will deep-dive into our new Flex-in-a-box architecture, including the architecture requirements and an introduction to cell-based architecture. We’ll conclude by discussing the benefits, challenges, use cases, and future ideas.

Initial Architecture

Here is a simplified high-level diagram describing how Flex acts as the foundation between the source (Github), destinations, and the developer surfaces.

As you can see in the diagram, the CI/CD foundation acts as the glue between the source and the destinations. The developer surfaces, powered by the foundation, offer paved roads and seamless workflows for various use cases.

Here is the initial architecture that we started with.

We have a Hub-and-Spoke architecture with the Hub in the middle (aka Flex Hub k8s cluster) connected to the various remote K8s clusters, which act as the spokes. As you can see, we have Git as the source of truth on the left, and K8s clusters as the remote clusters (destinations) on the right.

We use Argo Workflows for creating the CI/CD pipeline, Argo CD for providing GitOps capabilities, and Argo Events for catching events and triggering workflows. The CI/CD pipeline runs on the Hub cluster in a per-service namespace. Argo Rollouts is installed on remote K8s clusters and provides advanced deployment capabilities with Argo CD. The service runtime runs on namespaces provisioner on these remote K8s clusters.

Provisioner is a homegrown component that does the just-in-time provisioning during deployments. A couple of home-grown Observability components send events and information to our homegrown Flex backend, where the data is aggregated and exposed via a set of Flex APIs. The Flex developer experience (aka Flexperience) in the Developer Portal / Backstage uses the Flex APIs provided by the Flex backend to populate the service’s info.

Each service has two repos associated with it: an app repo and a deploy repo. App repo has the code for business logic, and the deploy repo has the configs and manifests for CI/CD, K8s resources etc.

Challenges with Initial Architecture

While the initial architecture provided a starting point, it had its challenges. We realized these challenges after 6 months when we crossed approximately 2.5K Argo CD applications.

No tuning - We used the Argo components with no/minimal tuning, and realized that there were several knobs and controls available on the Argo components that could be leveraged to improve the stability, performance, and scalability of the components.

Heavy load on K8s control plane - Flex Hub cluster is based on AWS EKS, and we just had one Hub cluster. We realized that Argo CD and Argo Workflows interact heavily with the EKS control plane, which includes the K8s API Server and etcd. The CI/CD experience is severely impacted when the K8s control plane is under heavy load.

Shared control plane - Both Argo CD and Argo Workflows were running on the same EKS cluster, which multiplied the overhead and load on the EKS control plane. This impact was observed in terms of degraded latency of the K8s API server, high etcd usage/churn etc., that caused a severe impact on the performance of anything interacting with the K8s control plane, including the Argo components. As a result, the user experience was getting impacted and client deployments were either taking a long time or failing.

No automation to create Flex Hub cluster - Flex Hub cluster was brought up manually, over many quarters, by different individuals. We did not have a concrete list of steps and automation to bring up a Flex Hub cluster, thereby putting us at risk if the Hub cluster goes down. It was a huge risk for the platform team and client teams because it would have taken us a long time to create a new one, and client deployments would have been stuck during that duration.

As a result of these challenges, we experienced several outages for the developer experience CI/CD workflows, including slow builds, hung workflows, and deployments taking too long, among other things.

Potential Solutions

Given the problems we faced, we realized quickly that one Flex Hub cluster would not be enough. We invested heavily in vertically scaling Flex Hub cluster by tuning and optimizing the Argo components, EKS control plane, and homegrown tooling. This helped us get out of the woods and support >10K Argo CD apps. While we will continue to scale Flex Hub cluster vertically to support more services/Argo CD apps, we knew that the runway will be limited.

We were projecting 10X growth in the next two years, and one Flex Hub cluster could not support all that scale alone. We needed more than one Flex Hub cluster.

Scale Up

There were a lot of configuration changes and optimizations we did to scale Flex Hub cluster vertically. These included:

Scale up replica count for Argo CD application controller and repo server
Ask Argo CD to exclude resources that we should not manage
Tune Argo CD self-heal and reconciliation timeouts
Adjust Argo CD repo server parallelism
Enable gzip compression for server responses
Archive old Argo Workflows by setting a short TTL
Switch from K8s APIs to Argo CD APIs to leverage informer cache

This list is not exhaustive and there were many more improvements which helped us tremendously. We have talked about some of these improvements in previous talks:

Scale Out

While the scale-up improvements were helping, we decided to re-architect. Though we had a couple of options, we decided to scale horizontally by having more than one Flex Hub cluster because it satisfied many of the architecture requirements (listed below) and gave us the biggest bang for the buck.

This was not going to be easy, because we had hundreds of production services running and we did not want to impact any of them. The rest of the article goes into the details of how we approached the re-architecture.

Flexbox concept

As a first step, we came up with the concept of a Flexbox. A Flexbox is a collection of Flex components that work together to provide the GitOps foundation for our CI/CD solution. We called the Flex Hub K8s cluster which formed the basis of our initial architecture, Flexbox 1.

Keeping future extensibility in mind, we also decoupled the concept of Flexbox from clusters. A Flexbox can be extended to support many physical and virtual Hub clusters inside it. Because the initial architecture had just one physical Flex Hub cluster, Flexbox 1 also had one Hub cluster.

New Architecture Requirements

We came up with the following key requirements for the new architecture.

Ability to predictably add scale (horizontally and vertically) as needed, for short-term and future needs.
Ability to relocate services out of Flexbox 1 and relieve pressure.
The services should not decide which Flexbox they are on.
No impact of rearchitecture on any existing services (runtime or CI/CD) on Flexbox 1.
No impact on service runtime during box-2-box relocation.
A service (app and deploy repos) should be associated with one Flexbox at a time.
Minimal impact/downtime during relocation to the service’s CI/CD pipeline, if at all.
During relocation, the service owners should be aware of the relocation and the downtime.
Ability to support more than one physical or virtual clusters inside a Flexbox.
Ability to support non-container workflows as well e.g. Serverless, Cloud Infrastructure etc.

What is Cell-Based architecture

Cell-based architecture is an architecture pattern that involves designing systems with multiple cells capable of executing the designated tasks independently. The basic idea is to have the ability to scale out by adding identical cells that handle the same kind of workload and isolate fault boundaries to limit the impact in case of failures.

Each cell is independent, does not share any state with other cells, and handles a subset of the overall workload requests. This reduces the potential impact of a failure, such as a bad software update, on an individual cell and the requests that it’s processing.

Here’s a diagram that depicts the cell-based architecture at a high level:

More details here: Diagram Source: https://newsletter.systemdesign.one/p/cell-based-architecture
Reference: https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html

Flex-in-a-Box (FiaB) architecture

With the requirements above, we developed a Flex-in-a-box (FiaB) architecture. This architecture allows us to encapsulate Flex components in boxes and easily replicate them to add scale on demand. We started by adding a new Flexbox 2. Before FiaB, all services were associated with Flexbox 1. This means that the CI/CD pipelines for these services run on Flexbox1. With FiaB and newer Flexboxes, now services can be onboarded to Flexbox 2 also, and any of the future Flexboxes.

To keep the impact of architectural changes to a minimum, we decided to have just one physical Hub cluster in Flexbox 2 too, and make it very similar to the Flex Hub cluster in Flexbox 1.

Our FiaB architecture is based on three Rs:

Recreation – Ability to (re)create and upgrade more Flexboxes with repeatability and predictability, with a GitOps-based CI/CD pipeline.
Redirection – Ability for Flex admins to decide which service maps to which Flexbox via declarative rules, and have services routed to the correct boxes based on those rules.
Relocation – Ability to relocate a service (with automation) from one Flexbox to another with zero downtime for the service runtime, zero work for the service team, and minimal downtime (< 15 mins) for the CI/CD pipeline.

Here’s a diagram that captures these three concepts. More details on these 3 Rs below.

Recreation

We created a GitOps-based pipeline for Flexbox lifecycle management. This means that there is a Flexbox config in Git which defines what a Flexbox looks like. Here is a diagram showing the workflow in action:

As can be seen from the diagram, any changes to the Flexbox config triggers a Flexbox automation Argo workflow. The workflow has a series of steps that automate the setup of a Flexbox on top of an EKS cluster.

The workflow has two key objectives:

Creates the Helm charts for the various Flexbox components - This step understands the Flexbox configs and creates the Helm charts for Flexbox components, including Argo CD, Argo Workflows, Argo Events, Provisioner and Monitoring.
- The workflow step then commits these Helm charts to Git, allowing Argo CD apps to sync these changes to the destination Flexbox.
Provisioning resources for Flexbox components, including creating Argo CD apps for each of the Flex components (Argo CD, Argo Workflows, Argo Events etc.) that need to be installed in the Flexbox. It’s an Argo of Argos architecture, where an Argo CD 0 (level zero) manages the Argo CD apps (level 1) for the various Flex components to be deployed on the destination Flexbox.

Redirection

Once we have a new Flexbox, we need the ability to map services to Flexboxes and redirect the Github events to the correct Flexbox. This function is performed by a new component called Redirector, which sits between Github and the Flexboxes (as can be seen in the new architecture diagram above).

Loosely speaking, Redirector acts as a poor man’s load balancer for Flexboxes. When the first event arrives for any service, Redirector evaluates the rules defined by Flex admins and maps the service to a Flexbox based on the rules. These rules are specified declaratively in a Github repo for Redirector by Flex admins. Redirector stores the box<>service mapping in a DB for future reference. For any subsequent GitHub events, it just picks up the mapping from the DB and routes the events to the appropriate Flexbox.

Relocation

While Flexbox 2 and other components in FiaB architecture were being built, there was constant organic adoption of Flex, Flexbox 1 was getting filled up and was under increasing pressure.

To relieve the pressure on Flexbox 1 and future use cases too, we needed a way to cleanly relocate a service from a source Flexbox to a destination Flexbox, without disrupting the service runtime. We created a new component called Relocator which does just that. Here are the four broad phases in the Relocator:

Stop the service’s CI/CD pipeline on source Flexbox - This would include pausing the Argo CD sync and stopping Redirector from sending the events for this service to the source Flexbox. This step also flags the service status (in Redirector) as “relocating” which helps us inform the developers that their service’s deployment pipeline is “under maintenance”.
Recreate the service’s CI/CD pipeline on the destination Flexbox - This includes recreating/copying the resources in the source Flexbox on the destination Flexbox.
Validate and start the service’s CI/CD on destination Flexbox - This includes starting the Argo CD sync and allowing Redirector to send the events for this service to the destination Flexbox.
Delete Flex pipeline on source Flexbox - Once everything is working fine, delete the resources on the source Flexbox to clean things up. This step also removes the “relocating” service status in Redirector.

Flex-in-a-box (FiaB) vs Cell-based architecture

When we started on FiaB, we did not use cell-based architecture as a reference. We realized later when we had conceived the new architecture that it is conceptually very similar to cell-based architecture.

In terms of comparison, a Cell in a cell-based architecture is loosely equivalent to a Flexbox in Flex-in-a-Box (FiaB).

Benefits / What worked well

The new Flex-in-a-box architecture offers various benefits:

“Stamp” setup - Ability to predictably and reliably create new Flexboxes.
Scale horizontally - Industry-proven way to scale horizontally by adding scale on demand.
Reduce existing load - Ability to relieve pressure on a Flexbox by allowing us to relocate services across Flexboxes, without any downtime.
Minimal relocation impact - Minimal impact/downtime to the service’s CI/CD pipeline for the duration of the relocation.
Abstracted Flexbox mapping - Service teams need not worry about which Flexbox they are on. Flex team decides the service<>Flexbox mapping.
Dedicated Flexboxes - Ability to provide dedicated Flexboxes to teams if need be.
Enhanced testing ability - Allow Flex team to test features/rollouts/infrastructure better using multiple boxes.
Configuration flexibility - Different Flexboxes may have different versions of Flex components installed if need be. In fact, different Flexboxes may have different components altogether. For example, one may host GitHub Action runners inside a Flexbox, rather than Argo Workflows. One Flexbox may have two clusters in it. We need to be wary of snowflakes though.
Better audit - Ability to track, monitor and audit GitHub events coming via Redirector.
Efficiency - Ability to maintain Flexboxes at optimum capacity rather than stretching one Flexbox to the limit.
Limit blast radius - If one of the boxes goes down, only services mapped to that Flexbox are affected, and can be relocated.
Better disaster recovery - Ability to recover quickly from disasters because of relocation.

Performance Tests / Benchmarking

We did several benchmarking and testing efforts to see the benefits of the various vertical and horizontal scaling improvements.

We recently presented them with the details at KubeCon EU 2024. Here’s the link:
Key Takeaways from Scaling Adobe’s CI/CD Solution to Support 50K Argo CD Apps

We also published a blog with focus on Argo Workflows scalability testing:
Argo Workflows Controller Scalability Testing on Amazon EKS

Challenges / Risks

While this architecture works well for us and provides us with a lot of flexibility, every architecture has some challenges, risks and overheads. Here are a few for FiaB architecture:

Infrastructure cost - There is some extra cost of running each Flexbox due to the additional control plane and the extra capacity buffer per cluster.
Complexity - Architecture becomes more complex because of adding a new Redirector component between GitHub and Flexbox. As a side effect, Redirector becomes a single point of failure across Flexboxes.
Thresholds/Limits - Defining and managing limits for each Flexbox is a challenge.
Support cost - The support team has to understand the role played by Redirector and Relocator when troubleshooting issues.
User experience - If a service gets relocated, some links exposed to clients may change e.g. Argo CD and Argo Workflows URLs, along with Hub cluster information.

Use cases

The Flex-in-a-box architecture opens up a world of possibilities for us in the future. Here are a few use cases:

Dedicated Flexboxes for specific orgs/teams - Some teams have already requested dedicated Flexboxes for their teams to effectively manage the noisy neighbor problem, and also enable efficient chargeback.
- This also allows us to shard the clusters better across Flexboxes.
Domain-specific Flexboxes - Specific Flexboxes could be dedicated to specific domains. For example:
1. Windows - Windows builds are a different beast than builds for Linux. Services workflows needing Windows builds may get routed to a Flexbox with specific tools/components to do a Windows build.
2. Desktop / Mobile - Builds for desktop and mobile apps may need a dedicated build farm. One or more Flexboxes can be designated/dedicated for this.
3. Cloud Infrastructure Provisioning - Cloud infrastructure provisioning requires custom operators and may need a separate Flexbox to ensure that specific custom operators do not adversely affect the performance of existing workloads. We can provide a dedicated box for hosting such workloads.
4. FaaS/Wasm - Building and testing Wasm modules could be very different than building containers. We can provide a dedicated box for hosting FaaS workflows.
Toolchain-specific Flexboxes - Flexboxes could cater to specific toolchains too. For example, if a service is using GitHub Actions (GHA) as the technology for pipeline creation and executions, GHA runners for the same could run in a specific Flexbox.

What’s Next / Future Exploration

Flex-in-a-Box improvements

While this architecture will serve us well in the future, here are some areas of future exploration:

Better sharding - As of now, all remote clusters are registered with Flexbox 1 and Flexbox 2. There is a hidden performance impact on the remote cluster for each Argo CD instance monitoring it. We can improve the sharding to have specific orgs/services on a specific Flexbox. This allows only clusters associated with those services to be registered with Argo CD on that box.
Split Argo CD and Argo Workflows across clusters - One of the challenges with current architecture is that both Argo CD and Argo Workflows are on the same cluster. This can be improved by running Argo CD on a cluster other than the one running Argo Workflows. This separate cluster could be another physical cluster in the same Flexbox, or it could be a virtual cluster (e.g. vCluster) running on the existing physical/host cluster.

Open-sourcing

Alongside the architecture improvements, Adobe is an integral part of CNOE cohort and we are working with a few companies (inside and outside of CNOE) to see how they can use what we have already built, and open-source Flex (or perhaps parts of it to start with).

Unified CI/CD

We have started a new initiative called Unified CI/CD which has the following goals:

Beyond containers - We primarily cater to container based workloads and this initiative helps us serve non-container use cases such as mobile, static website, desktop, serverless etc.
Secure by default - We aim to incorporate all non-negotiable security best practices by default in our pipelines and enforce them.
Sealed Paved Roads - We are working towards providing paved roads/paths for the most common use cases and make them “sealed”, minimizing the need (and making it hard) to deviate from the paved road while ensuring standardization, configurability, security, compliance, and other best practices.
Simplified workflows - Make the workflows simpler to ensure faster onboarding, reduced time for “zero to first commit”, faster “Concept -> Code -> Running state”.

Key Takeaways / Lessons Learnt

There were many key takeaways in this journey and several lessons learnt that can benefit the community. Here are a few:

Developer Experience as a Product - Developer Experience should be treated as a product and non-functional requirements (NFRs, such as scalability, stability, performance etc.) should be given as much importance as functionality.
Plan for future scale - Plan upfront about horizontal and vertical scaling needs, and how a combination of the two can help you achieve your scaling needs. The architecture should be able to evolve to handle future requirements and scale.
Sharding Strategy - Think of a sharding strategy for your services depending on your requirements. For Adobe, specific orgs/services on a specific Flexbox works out well.
Flexbox automation - Automate the creation of “your Flexboxes” from the beginning. This helps bring in predictability and reliability.
Performance Monitoring - Monitor key performance metrics from the beginning to figure when your infrastructure is under stress. We figured it too late and it led to clients facing issues before we could detect them.
K8s Control Plane as the bottleneck - We realized it pretty late that K8s control plane can become a bottleneck and how Argo CD and Argo Workflows can pound the K8s API server. Avoid having Argo CD and Argo Workflows on the same cluster.
Plan for Disasters / Relocation - Disasters will happen. Clusters will go down. Think of how you would deal with it. Can you relocate workloads to another cluster? Another Flexbox? What would be the Developer Experience before/during/after relocation?

Conclusion

A rich and seamless developer experience acts as a developer productivity multiplier and helps any enterprise compete better. Flex-in-a-box architecture provides Adobe with a solid foundation which we can use to scale our current developer experience foundation for future needs. This includes developer experience needs for use cases such as Windows, Serverless, Cloud Infrastructure, Desktop and Mobiles, Static Websites etc.

Discussion

End user members may participate in the discussion thread for this architecture.