Cloud Native Architecture – Platform Engineering

Architectures: ZEISS Vision Care - Order Fulfillment

Tue, 17 Mar 2026 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2020
Current version: 1.32.3

Hosts > 200 microservices supporting order fulfillment processes on managed Kubernetes. Provides the core compute platform for containerized services.

Dapr

Using since: 2020
Current version: 1.17.0

Provides common building blocks like service invocation, pub/sub, and state management across microservices; vendor-neutral abstractions enable portability across cloud providers.

KEDA

Using since: 2021/2022
Current version: 2.19.0

Event-driven scaling. KEDA acts as an event-driven scaler; examples of triggers include message-broker queue depth and resource utilization.

OpenTelemetry

Using since: 2020
Current version: Collector Contrib 0.145.0

Provides consistent instrumentation and exports telemetry to multiple targets.

Helm

Using since: 2020

Package management and templating for Kubernetes deployments.

Organization

ZEISS Vision Care produces spectacle lenses, instruments for refraction, and glasses adjustment equipment. Often, lenses are manufactured to an individual consumer’s prescription, effectively a batch size of one, which makes order fulfillment a multi-step coordination process.

Our Business Enablement & IT team develops, designs, and operates the order fulfillment platform that underpins this process.

Synopsis

We are reworking the order fulfillment process using a greenfield approach to modernize core systems. Our goal is a reliable, scalable platform that can evolve with business needs while remaining cost-efficient. The platform supports several key capabilities:

Order routing: decide where the order should be produced.
Order document generation: generate the necessary manufacturing and shipping documents for the order.
Logistics routing: decide how to ship the order to the customer.
Additional auxiliary domains as required.

Architecture overview & Goals

Goals

The architecture is designed for future-proofing and scale. By leveraging event-driven scaling with KEDA and decoupled microservices via Dapr, the system is built to seamlessly absorb high-volume, global order loads as new processes are continuously migrated to the new platform.

Key Requirements:

Modern cloud-based infrastructure
High reliability and scalability
Maintainable and extensible
Cost-effective operations

Architecture Overview

Data & Storage Strategy

The platform uses Azure-managed data services as backing stores:

MSSQL: Transactional order data and relational schemas
Cosmos DB: Scaled state management and cross-region replication
Azure Blob Storage: Order documents and manufacturing files
Redis: Caching operations to improve latency and throughput

Dapr’s state abstraction layer is used for Cosmos DB, Azure Blob Storage, and Redis, decoupling microservices from those storage backends and enabling migrations without application-level changes. For Azure Blob Storage, we also use Dapr’s blob binding to interact with data, in addition to the state abstraction layer.

Messaging & Asynchronous Communication

Azure Service Bus serves as the central message broker for asynchronous processes. Services publish and subscribe to topics via Dapr’s pub/sub building block, enabling loose coupling between applications. This event-driven approach provides resilience and allows each service to scale independently based on demand.

Network & Service Discovery

Services use Dapr’s service invocation building block instead of hardcoded endpoints, enabling resilience and the ability to replace or upgrade service implementations without changing callers. External traffic is routed to services via Ingress NGINX.

CI/CD & Deployment

Azure DevOps Pipelines automate building, testing, and deploying applications. We use a single, centralized Helm chart and store deployment configuration in Git as the single source of truth; the pipeline generates environment- and service-specific values.yaml files and deploys each service as its own Helm release to Kubernetes.

Can you expand on why you are using those projects/services?

We rely heavily on CNCF projects and open-source tooling to form the backbone of our platform:

Kubernetes (AKS) (Using since 2020): Hosts > 200 microservices supporting order fulfillment workflows. It provides the core compute platform for containerized services, offering a flexible model that supports multiple frameworks, programming languages, and dynamic scaling.
Dapr (Using since 2020): Provides common building blocks like service invocation, pub/sub, and state management across microservices. Dapr enables vendor-neutral capabilities, addressing service discovery and maintaining portability.
KEDA (Using since 2021/2022): Event-driven scaling based on Azure Service Bus queue depth and CPU/memory utilization. This allows the system to scale aggressively to match real-time demand while running economically during quieter periods.
Helm: Package management and templating. Charts are stored in Git, enabling reproducible deployments across environments since the project’s inception.
Ingress NGINX (currently in use, pending replacement – see Future Outlook): External traffic routing and load balancing.
OpenTelemetry Collector (Using since 2020): Provides consistent instrumentation and exports telemetry to multiple targets. It enables distributed tracing and metrics collection across microservices, ensuring observability and performance monitoring.

What has worked well?

Kubernetes, Helm, and KEDA have proven reliable and are widely used.

Scaling to Zero and Back: We initially attempted to scale services to 0 replicas with KEDA to minimize costs, but this introduced operational challenges: delayed data flows during testing when pods needed to start up, and frequent scale churn during traffic pauses between messages. We learned that setting a minimum replica count for baseline load while scaling out for peaks was much more cost-effective and reliable for our specific workload patterns.

The Advantage of Abstraction: A major architectural risk early on was building the entire system on Dapr (starting at pre-1.0 release 0.7.0). Early adoption carried organizational risk and we initially faced instability with actors under heavy load. Over time, those issues were fully resolved. The decision proved highly beneficial: Dapr’s maturity and vendor-neutral approach validated the initial decision, giving the platform extreme portability, flexibility, and saving custom boilerplate.

Platform Evolution: Dapr is consistently evolving, allowing us to streamline our platform by replacing specific SDKs with built-in functionalities over time. For instance, we are transitioning from Azure App Configuration to Dapr’s Configuration Building Block. In hindsight, we would have standardized on CNCF native components sooner (e.g., migrating earlier to OpenTelemetry Collector and a CNCF ingress solution) to avoid replacing vendor-specific SDKs mid-project.

Data Ownership and Boundaries: A critical lesson was the strict enforcement of data ownership. In a microservices architecture with over 200 services, we learned that clearly defining which service owns which data is non-negotiable. We found that any direct data access between services, bypassing their dedicated APIs, inevitably leads to tight coupling and maintenance challenges. Adhering to Domain-Driven Design (DDD) principles, where each service exposes its data only through a well-defined API, was essential for maintaining a scalable and evolvable system. This prevented a “distributed monolith” and ensured long-term architectural integrity.

What sort of “glue” have you had to develop?

Since we had a greenfield start, we were not constrained by legacy endpoints or migration paths. However, managing over 200 microservices required strict boundaries and standardized communication patterns. We heavily utilized Domain-Driven Design (DDD) principles—specifically Bounded Contexts to ensure each microservice owns its data and domain logic.

To enforce these boundaries and standardize the “glue” between services, we developed:

Standardized Helm Templates: A unified set of Helm charts that abstract away the complexity of Kubernetes manifests and Dapr sidecar configuration. Developers provide application-specific KEDA ScaledObject definitions and Dapr component definitions.
Common Libraries: While Dapr abstracts away many infrastructure concerns, we built thin, language-specific wrappers around the Dapr SDKs to enforce internal logging and error-handling standards.

Impact & Results

By adopting this cloud-native stack, we have built a highly scalable and future-ready order fulfillment system. A key driver of this success has been our extensive use of Dapr, which not only significantly reduced boilerplate code but also provided us with a high degree of vendor neutrality. This abstraction layer means we are not tightly coupled to specific cloud providers’ SDKs, enabling an architecture that is portable, flexible, and robust enough for long-term growth.

What’s next for your architecture?

We are constantly investigating how to run our services as efficiently and economically as possible. A near-term priority is replacing our existing Ingress NGINX setup. Because the Kubernetes project has officially announced the retirement of the ingress-nginx project, we are actively evaluating alternatives—including NGINX Gateway Fabric, Envoy Gateway, and Traefik—while striving to preserve our routing behavior, TLS automation, and operational stability. Transitioning to the Gateway API is a key step in future-proofing our external traffic routing.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: End-to-End Cloud Native Telco Platform Automation at Swisscom

Mon, 16 Mar 2026 00:00:00 +0000

Relevant Projects

CNCF Projects

Kubernetes

Using since: 2021

The Kubernetes API is the database backend and control plane of the entire automation platform. It acts as the runtime for all CNFs, operators, and platform services. Custom Resource Definitions extend the API to cover telco-specific concerns like IPAM, DNS, and network function configuration.

Flux

Using since: 2022

Flux is the GitOps engine for continuous reconciliation. It monitors Git repositories and synchronizes all desired state — CNF deployment manifests, Custom Resources, DNS endpoints, certificate requests, IP claims, and test definitions — into Kubernetes clusters.

cert-manager

Using since: 2023

Automated certificate lifecycle management integrated with Swisscom’s internal PKI. Certificate requests are expressed as Kubernetes CRs, reconciled by Flux, and managed by cert-manager. Private keys never leave the cluster.

Headlamp

Using since: 2025

Kubernetes dashboard for the management cluster, providing cluster visibility, RBAC-based access control, a CRD documentation browser, and extensible plugin system. Swisscom is listed as an official Headlamp adopter.

SDC (Schema Driven Configuration)

Using since: 2024

Used as the Config Sync Operator to push assembled configurations to CNFs. SDC enables vendor-agnostic, declarative configuration management using YANG schemas and NETCONF/gNMI protocols. Swisscom adopted SDC as its strategic configuration management solution and actively contributes features including config blame, drift detection, validation, testing compatibility with CNFs, and NETCONF Actions support. Swisscom is listed as an official SDC adopter.

CoreDNS

Using since: 2021

In-cluster DNS service discovery for Kubernetes services. Also used with conditional forwarding to route queries for private 5G zones (e.g., 3gppnetwork.org) to the authoritative PowerDNS servers.

ExternalDNS

Using since: 2023

Kubernetes-native automation of DNS records in PowerDNS using Custom Resources and annotations.

MetalLB

Using since: 2022

Load balancer for bare-metal Kubernetes clusters. MetalLB IP address pools are managed via KRM, with IP addresses dynamically allocated from NetBox via the NetBox Operator.

Kubebuilder / controller-runtime

Using since: 2022

Scaffolding framework and libraries for building custom Kubernetes operators. Used to build all domain-specific operators for CNF configuration abstraction, IPAM integration, config synchronization, and DNS automation.

Other Projects

PowerDNS

Using since: 2023

Authoritative DNS server supporting automation of advanced resource records (NAPTR, SRV) required for 5G/SIP via ExternalDNS.

NetBox Operator

Using since: 2024

Kubernetes operator for IPAM integration, open-sourced by Swisscom. Brings IPAM into the Kubernetes API with a claim model inspired by PersistentVolumeClaims — dynamically allocating IP prefixes and addresses from NetBox, managing their lifecycle through Kubernetes garbage collection, and supporting sticky IPs for disaster recovery.

NetBox

Using since: 2023

IP Address Management (IPAM) and network infrastructure modeling. Used as the IPAM backend for dynamic IP allocation across all CNFs and platform services.

TL;DR or Synopsis

Swisscom has built a cloud native telco platform for the end-to-end automation of its 5G standalone core network and cross-domain resource orchestration. The architecture replaces traditional imperative network management (Jenkins pipelines, Ansible playbooks) with a fully declarative, Kubernetes-native automation model driven by GitOps and the Kubernetes Resource Model (KRM).

A mobile core development environment contains approximately 2,000 pods with over 5,000 interdependent configuration parameters across Cloud-Native Network Functions (CNFs) such as UPF, SMF, AMF, UDM, UDR, BSF, NRF, NSSF, and AUSF. Engineers express high-level intents as Kubernetes Custom Resources; custom operators dynamically assemble full configurations at runtime — fetching IP addresses from IPAM, secrets from Vault, certificates from PKI, and infrastructure details from the cluster.

While the 5G core is the primary domain, the orchestration framework extends across multiple network domains and infrastructure services, applying the same intent-based automation patterns consistently.

Organisation

Swisscom is the leading Telecommunications/ISP and ICT company and offers mobile, Internet and TV products, as well as comprehensive IT and digital services to private and business customers. Swisscom’s expertise in cloud native technologies is well-established, as evidenced by its status as a former Gold member and Management Board member of the Cloud Foundry Foundation, along with its certification for Cloud Foundry. Additionally, Swisscom demonstrates a strong commitment to the Open-Source community, having been a CNCF Silver Member for several years and serving as a Kubernetes Certified Service Provider (KCSP) partner. Our skilled employees have delivered numerous talks and presentations at prestigious events such as KubeCon, Cloud Native Zürich, Swiss Cloud Native Day, KCD Suisse Romande, ContainerDays.

The company has embarked on a strategic transformation from a traditional telecom operator (“Telco”) to a technology company (“TechCo”) with 5G as a central driver. Swisscom operates an extensive 5G Non-Standalone (NSA) network covering 99% of the Swiss population. The cloud native platform described here powers the 5G Standalone (SA) core.

Teams

Multiple teams collaborate on this platform:

Cloud Native Resource Orchestration — creates a robust framework for orchestrating cloud-native resources such as CNFs, IPAM, Networks, DNS, Kubernetes Clusters, and more. Designs and operates GitOps pipelines, builds Kubernetes operators, and develops the management cluster UI/UX.
Mobile Cloud Native Engineering — designs, implements, and operates the cloud native 5G core platform, including GitOps pipelines, Kubernetes operators, and network function lifecycle management.
DNS Engineering — builds and operates the highly reliable cloud native DNS service underpinning the 5G core and other infrastructures.
Network Engineering — provides IPAM and Network-as-a-Service.
Platform & Developer Experience — manages Kubernetes clusters and builds developer tooling.

Architecture overview & Goals

Goals

Full GitOps for the 5G Core — Extend GitOps beyond CNF deployment to include network function configuration, certificate management, DNS record provisioning, IP address management, and testing — achieving continuous reconciliation across all layers.
Declarative, Intent-Based Configuration — Replace static, low-level configuration manifests with abstract, intent-driven Custom Resources. Engineers specify what they want using a high level intent (e.g., “this CNF needs an IP address from a subnet in network zone A”) rather than how to achieve it, with Kubernetes operators dynamically assembling configurations at runtime.
Automated CD — High level of automation for telco deployment rollouts. This includes rethinking Change Processes as well as building solid CI/CD/CT pipelines to ensure a highly reliable network.
In-Band with Kubernetes — Bring all automation in-band with the Kubernetes API, eliminating out-of-band tools like Jenkins pipelines and Ansible playbooks. This ensures that the Kubernetes orchestrator has full visibility and control over all resources, enabling self-healing and reconciliation.
Cloud Native DNS Service — Operate a highly reliable, geo-redundant, on-premises DNS service for the 5G core using open-source technologies (CoreDNS, PowerDNS, ExternalDNS), fully automated via GitOps and Kubernetes Custom Resources.
Contribute to the Ecosystem and Shape the Industry Discussion — Open-source key components built or contributed to during this journey (NetBox Operator, SDC, demo code) to enable other organizations to adopt similar patterns. Contribute to Meetups and Conferences in order to achieve broader success of Cloud Native adoption in the Telco Community.

Architecture overview

The platform is organized in three layers:

The Intent Layer stores high-level desired state in Git — engineers define what they want using concise Custom Resources (e.g., a DNN configuration with hostname, region, and SNSSAI).

The Automation Layer runs on Kubernetes and continuously reconciles. Flux pulls intents from Git. Custom operators dynamically assemble full configurations by fetching IP addresses from NetBox (via the NetBox Operator) and creating connectivity in a Network-as-a-Service platform. It then pushes the KRM formatted Runtime Configuration to a Git repository for the Runtime layer to consume.

The Runtime Layer hosts the 5G core CNFs and supporting services. Flux pulls intents from Git. SDC pushes the assembled configuration to CNFs via NETCONF/gNMI. MetalLB, cert-manager, External Secrets Operator, ExternalDNS are used to configure the Workload cluster.

Key Design Principles

GitOps + KRM: Git stores high-level intents; Kubernetes manages dynamic, low-level configuration assembly. This is a shared source of truth across Git and Kubernetes.
Continuous Reconciliation: Every aspect of the system is continuously reconciled against the desired state — following the four OpenGitOps principles.
Abstraction over Complexity: Engineers work with simple, high-level intents; operators handle the complex assembly.

Can you expand on why you are using those projects/services?

From GitOps to KRM

Standard GitOps tools (Flux, Argo CD) combined with Helm/Kustomize have limitations for complex telco use cases: they cannot use live Kubernetes resources as inputs during rendering, cannot invoke custom business logic during template processing, and cannot dynamically assemble configurations from multiple sources. By extending GitOps with the Kubernetes Resource Model (KRM) — Custom Resource Definitions, Custom Resources, and custom Operators — the platform achieves dynamic configuration assembly at runtime.

Configuration Abstraction & Dynamic Assembly

A typical 5G core configuration includes IP addresses, VLAN IDs, DNS records, NF variables, secret references, and certificate references. Previously, engineers had to know all values upfront and embed them statically. The new intent-based model inverts this. The following is an example of a DNN configuration in pseudo-yaml-code.

On the intent layer, only a very stripped-down KRM manifest exists:

apiVersion: telco.swisscom.com/v1
kind: Dnn
spec:
 hostname: "gprs.swisscom.com"
 ipv4: true
 region: ch-east
 type: MobileInternet
 snssai: [1, 10]

This KRM manifest is stored in git and synced to the Management cluster using Flux. From this, the Operators in the automation layer create intermediate resources, in this case an IP Address via NetBox:

apiVersion: netbox.dev/v1
kind: IpAddressClaim
spec:
 parentPrefixSelector:
 region: "ch-east" # from Dnn.spec.region
 family: "IPv4" # from Dnn.spec.ipv4
status:
 ipAddress: 1.2.3.4

The Automation Layer creates the following low level resources for the Runtime Layer:

apiVersion: config.sdcio.dev/v1alpha1
kind: Config
metadata:
 labels:
 config.sdcio.dev/targetName: ch-east  # from Dnn.spec.region
 config.sdcio.dev/targetNamespace: default
spec:
 priority: 10
 config:
 - path: /
 value:
 dnn:
 - name: "gprs.swisscom.com" # from Dnn.spec.hostname
 ip: 1.2.3.4 # from IpAddressClaim.status.ipAddress
 snssai: [1, 10] # from Dnn.spec.snssai
 type: MobileInternet  # from Dnn.spec.type
---
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
spec:
 endpoints:
 - dnsName: "gprs.swisscom.com" # from Dnn.spec.hostname
 recordType: A
 targets:
 - 1.2.3.4 # from IpAddressClaim.status.ipAddress

SDC will now sync the configuration to the 5G CNF and ExternalDNS will create the DNS records in the authoritative PowerDNS backend.

What has worked well?

Intent-based configuration dramatically reduced the complexity engineers face. Instead of managing thousands of interdependent parameters, they work with concise Custom Resources.
Full GitOps reconciliation across all layers (deployment, configuration, DNS, certificates, IPAM), configuration drift is detected and reverted to ensure consistency.
Custom Kubernetes Operators (built with Kubebuilder/controller-runtime) proved to be the right pattern for telco domain-specific concerns, providing full reconciliation support and native KRM integration.
The claim model for IPAM (NetBox Operator) elegantly solved dynamic IP allocation by following established Kubernetes patterns (PVC analogy).
Bringing all automation in-band with Kubernetes gave the orchestrator full visibility and control, enabling self-healing and eliminating the brittleness of out-of-band tools.
DNS resilience engineering — dedicated hackathons, chaos testing, and disaster recovery playbooks significantly improved DNS service reliability.
Cross-team collaboration on a shared platform and KRM patterns accelerated adoption across multiple network domains.

What has not worked well?

NETCONF as a configuration protocol introduces complexity — it requires SDC as an intermediary and prevents fully Kubernetes-native configuration. Ideally, CNF vendors would support native K8s APIs using CRs/CRDs or Secrets/ConfigMaps.
Tooling gap for KRM-based configuration assembly — no mature, community-standard Kubernetes-native tool exists for dynamic configuration hydration. Swisscom had to build custom operators to fill this gap.
GitOps+KRM auditability trade-off — with dynamically assembled configurations, not all state is visible in Git history. The team continues to explore automated intermediary Git layers.
Cumbersome vendor configuration manifests — large, monolithic configuration files from CNF vendors (with ~5,000 interdependent parameters) required significant effort to decompose into intent-based abstractions.
Telco’s imperative model and Kubernetes’ declarative approach do not align well - SDC follows the declarative paradigm, where users define the desired state and SDC determines the actions to achieve it. In contrast, NETCONF/gNMI use an imperative model that requires explicit ordered steps (“do A, then B, then C”). Translating declarative goals into imperative sequences is complex when user‑defined ordering matters, such as for firewall rules where evaluation order affects behaviour. Example: Issue “Support user sorted lists”

What sort of “glue” have you had to develop?

Custom Kubernetes Operators — domain-specific operators for CNF configuration abstraction, config synchronization, IPAM integration, and DNS automation, all scaffolded with Kubebuilder and controller-runtime.
Configuration hydration logic — the CNF Config Operator dynamically assembles full configurations from multiple sources (NetBox, Vault, cluster environment) based on high-level intents.
SDC contributions — significant development work on SDC including early testing of CNF compatibility, configuration validation, monitoring, config blame, drift detection, and NETCONF Actions support.
ExternalDNS NAPTR support — contributed PR #4212 to enable NAPTR record support for SIP phone calls.
Resilient DNS architecture — created a reference architecture and open sourced on GitHub.
Headlamp plugins — Custom Resource plugin.

How did the Architecture Evolve

Journey

The architecture evolved significantly from traditional imperative automation to the current declarative, KRM-based model:

Phase 1 — Ansible + Jenkins: Initial automation used Ansible playbooks triggered by Jenkins pipelines. Configuration was fire-and-forget with no continuous reconciliation.
Phase 2 — GitOps for deployment: Introduced Flux for CNF deployment, but configuration remained out-of-band via Ansible/NETCONF.
Phase 3 — Full GitOps + KRM: Extended GitOps to cover configuration, DNS, IPAM, certificates, and testing. Built custom operators and adopted SDC for config synchronization. Achieved continuous reconciliation across all layers.

Key lessons

De facto GitOps for operators is not true GitOps — creating Helm releases in Git while configuring NFs via NETCONF out-of-band breaks the GitOps model. Bringing configuration into the Kubernetes API was essential.
Bringing everything in-band with Kubernetes enables self-healing, reconciliation, and eliminates the brittleness of out-of-band tools.
Git as the only source of truth is insufficient — the shared source of truth model (Git for intents, Kubernetes for dynamic state) was a deliberate and necessary evolution.
Abstraction is critical — engineers cannot effectively manage 5,000+ parameters directly. Intent-based CRs with dynamic assembly significantly reduced cognitive load and errors.
Custom Kubernetes Operators are the right pattern for domain-specific concerns that existing tools cannot address.
Contribute upstream — local patches create long-term maintenance burden. Swisscom prioritizes upstream contributions (ExternalDNS, SDC, NetBox Operator) for sustainability.

What’s next for your architecture?

Mature SDC Integration — Continue expanding SDC for full lifecycle management with continuous reconciliation via gNMI and NETCONF, including completion of NETCONF Actions support.
Eliminate NETCONF Dependency — Work with CNF vendors to move toward fully Kubernetes-native configuration APIs, reducing reliance on legacy telco protocols.
Advanced Dynamic Configuration Assembly — Develop more sophisticated Kubernetes operators for multi-source configuration hydration, enabling even more complex intent-based workflows across multiple network domains.
Multi-Cluster & Edge Expansion — Scale the architecture to additional edge locations and Kubernetes clusters while maintaining consistent GitOps-driven automation.
Community Tooling for KRM — Contribute toward a mature, Kubernetes-native tool for dynamic configuration assembly that the wider cloud native community can adopt, addressing the current gap in tooling identified during the project.
Resilience & Reliability — Ongoing improvements to cross-cluster redundancy, disaster recovery playbooks, chaos testing framework, and enhanced monitoring/alerting.
Observability & AIOps — Integrate AI-driven operations capabilities leveraging the rich telemetry data from the platform’s monitoring stack.
Cross-Domain Expansion — Extend the orchestration framework to additional network domains and infrastructure services beyond the 5G core, applying the same intent-based automation patterns consistently.

Community Contributions

Contribution	Details
NetBox Operator	Open-sourced under https://github.com/netbox-community/netbox-operator
SDC Contributions	Active contributor to the SDC project (on its path to CNCF incubation)
KRM Demo Code	https://github.com/swisscom/containerdays-2024-krm
Conference Talks	KubeCon EU, ContainerDays, Open Source Summit EU
CNCF/LFN Whitepaper	Co-authored Accelerating Cloud Native in Telco

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: A modern and sovereign Private Cloud «Kubernetes Service» for Swiss-based enterprises.

Wed, 04 Mar 2026 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2024
Current version: 1.32.8 (CNIP)
Current version: 1.31.x - 1.34.x (SKP)

Kubernetes enables high availability, scalability, and performance for infrastructure, offering a centralized and policy-driven platform to manage network and service data supporting Managed Kubernetes for our cloud customers.

KubeVirt

Using since: 2024
Current version: v1.5.0 (CNIP)

Cluster resources are constructed using KubeVirt for virtual machine abstraction of Control Plane and Worker instances.

Kube-OVN

Using since: 2024
Current version: v1.13.14 (CNIP)

Kube-OVN is utilized as network stack of the infrastructure cluster to enable intra-cluster/east-west network communication of user clusters. It enables a policy-driven security model as well as customer network isolation using VPCs.

MetalLB

Using since: 2024
Current version: v0.15.3

MetalLB is an integral component of the infrastructure deployment process, offering automated access to the framework that provisions individual user cluster resources on bare metal Kubernetes environments.

Container Storage Interface (CSI)

Using since: 2024
Current version: v25.06.3 (trident-csi)
Current version: v0.4.5 (kubevirt-csi)

Kubevirt-CSI is the standard storage interface for persistent volumes in user clusters. Trident-CSI manages NetApp storage requests and can also be used directly to integrate with Swisscom’s File Service Kubernetes, which provides iSCSI and NFS shared storage across all Availability Zones.

Kyverno

Using since: 2024
Current version: v1.13.4 (CNIP)

Kyverno serves as the default policy engine for infrastructure and user clusters, providing robust security constraints. In addition to Kyverno, also Chainsaw (a Kyverno sub-project) is used for automated, declarative e2e testing.

ArgoCD

Using since: 2024
Current version: v3.2.0 (CNIP)

ArgoCD allows us to deliver comprehensive infrastructure using a fully automated GitOps methodology.

Helm

Using since: 2024
Current version: v3.5.1 (CNIP)

Helm automates the creation, packaging, configuration, and deployment of Kubernetes applications by creating reusable charts.

CloudNativePG

Using since: 2024
Current version: v1.27.0 (CNIP)

CloudNativePG (CNPG) manages PostgreSQL databases in cloud-native environments. It handles the full lifecycle of highly available PostgreSQL clusters (primary/standby with native streaming replication), including declarative deployment, scaling, backups, self-healing, failover and monitoring.

Describe your organisation

Swisscom is the leading ICT company in Switzerland and offers mobile, Internet and TV, as well as comprehensive IT and digital services to private and business customers. Swisscom’s expertise in cloud native technologies is well-established, as evidenced by its status as a former Gold member and Management Board member of the Cloud Foundry Foundation, along with its certification for Cloud Foundry. Additionally, Swisscom demonstrates a strong commitment to the Open-Source community, having been a CNCF Silver Member for several years and serving as a Kubernetes Certified Service Provider (KCSP) partner. Our skilled employees have delivered numerous speeches and presentations at prestigious events such as KubeCon, Cloud Native Zürich, Swiss Cloud Native Day, KCD Suisse Romande, ContainerDays, among others.

Our next generation Private Cloud Container as a Service offering «Kubernetes Service» for the B2B market addresses customer’s need for scalable and highly available Kubernetes workload as a flexible and secure IT foundation. It is part of our Swiss-based Enterprise Service Cloud (ESC) market channel as a sovereign, Private Cloud Kubernetes offering for effortless provisioning and usage of our customer’s container workloads.

Describe your entity and/or team

The development and delivery of the new «Kubernetes Service» is done at within Swisscom’s IT-Clouds Value Stream and shared across two teams:

Pathfinders: responsible for the Cloud Native Infrastructure Platform (CNIP). CNIP handles the creation, delivery, and lifecycle management of the KubeVirt-based virtual machines (VMs). These VMs function as nodes for both the Control Plane and Workers. The VMs are ephemeral and can be re-created immediately in case of any failure. They are solely used to enable container-based workloads and do not act as standalone VMs.
Guardians: responsible for the Swisscom Kubernetes Platform (SKP), which runs on top of CNIP. It consists of the installation of Kubermatic Kubernetes Platform (KKP) for the customer tenant (environment) and the setup and support of the highly available Control Plane for any customer (user) cluster.

The layered approach allows Swisscom to manage technological aspects distinctly by segregating the cloud native infrastructure (managed by the Pathfinders team) from the Kubernetes platform (managed by the Guardians team). This strategy ensures considerable flexibility, permitting each layer to be combined or integrated with other technologies in the future.

Brief overview of your architecture and any potential goals you are trying to achieve with it?

Summary

Kubernetes Service is the successor to our current container offering, representing a significant shift towards a more cloud-native approach using advanced Open-Source technology. Currently bound to a vendor-specific implementation, Swisscom has opted to employ open-source tools for the development of cloud native products for customer use. This strategy aims to minimize dependencies and mitigate the risk of vendor lock-ins.

By adopting this architecture, Swisscom can uphold quality within the cloud native domain while maintaining a competitive pricing model due to reduced reliance on external licensing and subscription models. Furthermore, having the ability to develop, maintain, and operate all components internally enhances our decision-making processes and strengthens our roadmap capabilities.

Another important point is that our customers’ data will always remain within Switzerland and under Swiss law. Since we fully own the platform and do not rely on any external vendors, we can confidently guarantee true data sovereignty, hosted entirely on our premises without relying on vague marketing claims. Additionally, because Swisscom is not subject to the US Cloud Act or similar foreign regulations, no non-Swiss legislation can access the data.

Brief overview of architecture

A simplified high-level diagram describes Kubernetes Service, including multi-tenancy and security aspects:

As illustrated in the figure, two separate and independent user tenants, BLUE and RED, are established on shared resources (depicted in yellow), managed by the Kubernetes Infrastructure Cluster. The foundation for all virtual abstractions is the Consolidated Infrastructure (COI) in Swisscom’s data centers.

Each customer-specific environment comprises a management zone (MGMT Zone) and a workload zone. These zones address shared responsibilities, where Swisscom provides the Control Plane for each customer’s environment (illustrated in blue and orange in the next figure).

Customers have the flexibility to deploy workloads within the workload zone independently of the management resources as required. Furthermore, each customer is able to maintain multiple environments. This provides an alternative method for segregating workloads at the tenant level instead of the Kubernetes cluster level, thereby ensuring comprehensive isolation from the outset.

Goals and objectives

One of the primary objectives of the product refresh is to offer more desired features to customers. Compared to the current offering, enhancements include:

Upstream Kubernetes versions with faster updates
Node Autoscaling
Integrated Backup functionalities
Native Kubernetes Load Balancer
Modern customer self-service portal
Additional Kubernetes add-ons available via Application Catalog

Moreover, additional options are directly available to our customers:

Choose from different Container Network Interfaces (CNI)
Access persistent storage through kubevirt-csi

With KubeVirt providing abstraction, KVM is employed as the hypervisor on bare-metal servers. From the customer’s perspective (Customer X), the administrator of their user cluster manages all selections and abstractions shown in the figure below, enabling customers to make independent decisions, e.g. choosing a default CNI from the available options (Cilium, Canal, None).

In addition to technical improvements, we aimed to minimise reliance on external vendors and build a truly sovereign cloud solution that can compete with Public Cloud offerings, free from outside service dependencies. Our goal is for customers to run their Kubernetes workloads in our sovereign ESC Cloud, providing a comprehensive alternative to US hyperscalers - in terms of functionality and, most importantly, data privacy.

Can you expand on why you are using those projects/services?

Cloud-Native Implementation: Utilized CNCF projects and technologies to deploy a comprehensive stack consistent with a microservices-based architecture, resulting in enhanced scalability and operational agility.
Kubernetes for Orchestration: Adopted Kubernetes to manage containerized workloads, enabling automated deployment, scaling, and resilience on management as well as user cluster level.
Kube-OVN as network layer: With Kube-OVN as CNI on the infrastructure clusters and it’s VPC functionality, it allows customer environments to be fully segregated on a shared platform, providing maximum flexibility and strong security enforcement. It enables the teams to use familiar cloud-native development, operations, and debugging tools and skills.
KubeVirt for VM abstraction: A high-quality, Kubernetes-native virtual machine abstraction facilitates the deployment of container-based resources on a centralized cloud-native infrastructure platform, while maintaining flexibility for future use of VM resources.
Open-Source & Cost Efficiency: CNCF components deliver vendor-neutral, cost-effective solutions that are fundamental to container orchestration and observability. These open-source tools form the foundation of our sovereign cloud initiative, empowering internal teams to design customized architectures independently of third-party vendors. Utilizing CNCF technologies allows us to maintain flexibility, scalability, and comprehensive control over our cloud infrastructure, supporting our strategic objectives of autonomy and innovation.
Declarative & Configuration-Driven Approach: CNCF tools align with the low-code/no-code principle by enabling declarative configuration management.

What has worked well?

The implementation has eventually lead to the product launch of Kubernetes Service in August 2025, with some strong outcomes:

Layered Architecture for Enhanced Robustness: The integration of Cloud Native Infrastructure Platform (CNIP) and Swisscom Kubernetes Platform (SKP) forms the foundation of the new Kubernetes Service, enabling flexible handling as separate platform layers for streamlined future operations.
Vendor-Agnostic Production Platform: By eliminating proprietary technology, a resilient and adaptable foundation has been established to host managed Kubernetes clusters within Swisscom’s Private Cloud, ensuring a high degree of flexibility and scalability, as well as privacy.
Modern Cloud-Native Foundation: The implementation of Kubernetes to deliver managed Kubernetes clusters to end customers enables a unified cloud-native stack across all layers of responsibility, promoting consistency and efficiency.
Best Practice Design: Collaborating with Kubermatic, a modern Kubernetes platform was designed, incorporating the latest technologies such as KubeVirt and Kube-OVN, to ensure an enterprise-ready solution for end customers.
Operational Excellence: Equipping teams with essential cloud-native and Kubernetes expertise enhances the attractiveness of Swisscom’s tech stack to potential candidates and reinforces the company’s commitment to the Open-Source community.
Successful Internal Adoption: The Kubernetes Service was successfully launched as Swisscom’s internal Container platform, achieving significant traction with over 60% of workloads migrated within the first 9 months of operation.

What has not worked well?

While the architecture delivered significant improvements, several challenges emerged during implementation:

Enterprise-Readiness of Cloud Native Technologies: Despite successful scaling in test and internal production environments, many advanced cloud-native technologies faced difficulties when deployed in enterprise-grade settings for end customers (e.g., B2B market). This highlighted the need for further refinement and testing in real-world scenarios.

For KubeVirt, for instance, there are only limited real-world examples, best practices or reference designs available to draw upon for large-scale, production-grade business deployments. Switching fully to Kube-OVN as the main network layer also demands extra effort and is less straightforward than traditional network solutions with established production lifecycles.
Limited Support and Professional Services: The availability of professional support, particularly 24/7, for open-source and cloud-native technologies is limited. This poses challenges for enterprises seeking to adopt these technologies and provide services with guaranteed service levels (SLAs).

A possible solution is for more companies to provide professional support and make these services more transparent. Furthermore, the CNCF could introduce a “Certified Supporter” verification system to strengthen trust in firms that offer professional support.
Knowledge Gaps and Skills Requirements: Adopting new technologies demands specialized knowledge and expertise. In-house engineers required additional training and support to effectively maintain and troubleshoot products built on these technologies.
Customer Acceptance and Migration Challenges: Introducing a new platform based on modern technologies, without a proven track record in enterprise-grade deployments, required significant effort to educate customers, facilitate migration from legacy stacks, and promote the benefits of a sovereign cloud solution. This process demanded substantial resources and support to ensure a smooth transition.

What sort of “glue” have you had to develop to enable usage of your architecture ?

The reference architecture provides a strong foundation, making it practical and easy to use. The below elements were designed to simplify adoption, improve usability, and ensure seamless interaction across layers acting as the “glue”:

Unified Abstraction APIs: Developed APIs (Open Service Broker spec) that hide complexity and provide a consistent interface for orchestration and other consuming Operational Support Systems (OSS).
Advanced Routing Functionality: In order to integrate the customer environments into the Swisscom Core Network (MPLS), we developed and implemented our own concept of edge routers using BGP on FRRouting pods. This custom solution supports NAT, Fail-over (VRRP) as well as north-south firewalling (traffic from/to customer environments). These router pods are managed by an operator and configured with custom resource definitions.
Policy Integration Layer: Built operators to dynamically apply and manage Kyverno policies across different stages without requiring deep technical intervention.
Firewall Management: Implemented operators and API endpoints to allow customers to manage firewall rules on the SDN layer of the KubeVirt infrastructure, via Kube-OVN network policies.
Workflow Orchestration Logic: Developed and implemented the entire platform orchestration logic and automated pipelines from bottom-up.
Commandline tooling: Various commandline tools for human operators to manage and control the entire platform and all parts of it with ease.
Testing & Fine-tuning: With limited experience in large-scale bare-metal Kubernetes deployments, we had to do a lot of testing, validation and fine-tuning. We had to make sure, that the platform scales properly with more workloads being migrated every day.

Has your architecture evolved? What lessons have you learned from previous iterations?

Our architecture and product have undergone significant evolution through iterative development, driven by customer feedback and emerging requirements.

Iterative Development Approach: We began by establishing foundational layers and meeting the needs of our internal Swisscom customers. Subsequent iterations introduced advanced features for end customers, incorporating feedback from both internal and external stakeholders.
Continuous Improvement and Feedback Loop: Each iteration allowed us to gather valuable insights and add new functionalities, refining our product and enhancing customer satisfaction.
Steep Learning Curve and Expertise Development: As we ramped up the product, our teams faced a significant learning curve, developing essential expertise and professionalizing DevOps processes to ensure seamless operation.
Strategic Partnerships and Support: Our collaboration with Kubermatic enabled us to leverage professional support for key components, including KubeVirt and Kube-OVN, ultimately maturing our production platform and solidifying its readiness for enterprise-grade deployments.

Through this iterative process, we’ve gained valuable lessons and refined our architecture to better meet the needs of our customers, while developing the expertise and partnerships necessary to drive continued success.

Outcome

By embracing open-source and cloud native technologies, Swisscom successfully created a sovereign cloud solution, modernizing its container offering while reducing vendor lock-in and providing advanced features to customers. The new «Kubernetes Service» demonstrates the power of cloud native architectures in creating flexible, scalable, and cost-effective solutions for enterprise-grade services, all while ensuring true data sovereignty and regulatory compliance. This approach positions Swisscom as a leader in sovereign cloud solutions, offering Swiss (and European) customers a trusted alternative to global hyperscalers.

What’s next for your architecture? What are you looking to do next?

Building on the success of our proven reference architecture, which now supports both internal and external customer workloads in production, we’re focused on expanding and enhancing our offerings:

Hybrid Cloud Expansion and Multi-Cloud Flexibility: We’re working to enable seamless public cloud deployments, complementing our existing Swiss-based data centers and strengthening hybrid cloud use cases.
Edge Cloud Support: With cloud sovereignty in mind, we are developing a «Kubernetes Service On-Prem» extension that will deliver the Private Cloud product on a Cloud Edge Stack at customer premises, enabling an autonomous instance of our Kubernetes Service. This is currently in development with an interested customer.
GPU-Enabled Workloads and Emerging Technologies: Next, we’ll be integrating GPU support and exploring other emerging technologies to unlock new possibilities for compute-intensive applications.
Customer-Driven Features and Enhancements: We’re committed to delivering additional features and functionalities requested by our customers, further enriching our platform and services.
Simplified Onboarding and Resource Optimization: To improve efficiency and resource utilization, we’ll be introducing a shared cluster concept, allowing for more flexible and efficient use of our bare-metal infrastructure.
Exploring New Use Cases - VM Workloads: We’re also investigating the possibility of hosting classical VM workloads on our Cloud Native Infrastructure Platform (CNIP), expanding the platform’s use cases beyond container-based workloads and further increasing its versatility.

By pursuing these initiatives, we aim to continue delivering value to our customers, drive innovation, and grow our architecture and services to meet evolving needs.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: Enabling Allianz Direct's scaling through Platform Engineering

Mon, 28 Oct 2024 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2021
Current version: 1.29.6

Kubernetes has been the foundation for our Internal Developer Platform and almost all of Adobe’s containerized workloads run on Kubernetes clusters.

Helm

Using since: 2021
Current version: v3.12.3

Helm is our package manager and helps us abstract out some of the complexity by including them as dependencies and only exposing the values.yaml file.

Argo

Using since: 2021

We are using all four projects under Argo:

Argo CD: v2.9.22

Platform Engineering @ Allianz Direct

Allianz Direct has been working on building Platform Engineering for the past several years. We have created a cloud-native future-proof platform which offers an organisation-wide-devops(orgOps) experience on top of public cloud. Business, Software Engineers, Security, Compliance, Architecture, Support and Cost Efficiency are few of our stakeholders.

Platform Engineering Team Topology - internal

The PE team is internally organised in DDOs - Distributed DevOps, where each ddo owns a Domain. This setup follows the reverse Conway manoeuver where the reference architecture structure dictates the teams setup. The PE team consist of 13 members. All of them internal employees.

The PE team is being supported by auxiliary teams: Developer Service Hub (DSH): entry level engineers handling the low complexity tasks such as: access management, 1st level technical support, SRE onboarding, ticket sorting and filtering Cloud Auditing Hub (CAH): entry level engineers with a focus on compliance and audit Production Management Hub (PMH): entry level engineers with a focus on incident and problem management Technical Tribe Leads (TTL): Technical leads on the Tribe (department) level that became our evangelist for best practices and continuous innovation partners.

Platform Engineering Team Topology - external

Externally, the team has decided to scale by building on the concept of FDOs - Federated DevOps, where each squad (developing team) has built internal devops expertise focusing a minimum amount of time on Operations concerns. A FDO community has flourished and enabled successfully an extremely aggressive organisation scale up The PE serves an approx. 500 engineers and 1500 employees.

Platform Engineering Architecture

The PE team has built a Cloud Native Platform that serves the core B2C business of a fully digital insurance player acting on several european markets.

It serves also all the newly created B2B business initiatives. The highly aggressive scale-up of the both B2B and B2B businesses forced the PE team to redesign the architecture towards a multi-tenant architecture with minimum operational effort. On top, the Platform has become a Organisation Group Standard and an example for Global initiatives. This forced the PE team to choose a light architecture under a second Platform based on Fargate ECS, reducing the operational effort and talent demand to minimum.

PE team delivered several cloud native best practices:

GitOps (ORGops to be more precise)
Stateless clusters orchestrated by a management cluster
Fargate ECS multitenancy
DevSecOps
SRE

Special projects that served us well

AWS obsession with customers served us well. The technical support and continuous relation with AWS architects helped us maximise our effectiveness and evolve our architecture as well as complementing our stack with SaaS services such as EKS, Redis, MSK.
AWS managed Kubernetes gave us the flexibility we needed to meet our users needs with as little overhead as possible. Most of our application choices come directly from our users, however our infrastructure choices were made to balance our compliance needs, usability and supportability.
ArgoCD has become instrumental for us and the core of our internal innovation. We are currently using it for spinning up and managing our multitenant implementation whether that is on namespace level or Fargate ECS level.
PAGT(Prometheus, Alert manager, Grafana and Thanos) stack is the core of our Observability. It is complemented by OTEL and ELK Stack and enabled us to build a state of the art SRE journey for all our Stakeholders

Things that worked well for us

EKS and now EKS + Fargate ECS works great for us. We can fully owns our core domains and stay hardened with minimum effort. Complementing our Stack with AWS services especially on Persistence, Data Streaming and Storage increase our innovation speed. ArgoCD with helm, Atlantis with terraform and Crossplane are working great for us, allowing us to follow the GitOps way, a great way of working for heavily regulated industries.

Stakeholders value streams

Business

Cost savings of X%
Business Delivery expedited by X%
Security incidents reduced by X%
Reliability incidents reduced by X%
Organizational Performance increased by X%

End-Customers

User Experience improved by X%
Fast Feedback loop increased by X%
Reliability increased by X%

Developers

DevX improved by X%
Fast Software Delivery improved by X%
Productivity increased by X%
High Quality increased by X%
Innovation increased by X%

Security

Cloud native Security costs reduced by X%

Compliance

Cloud compliance costs reduced by X%

Legal

Cloud native legal cost reduced by X%

QA

Automation increased by X%

FinOps/Sustainability

Cloud Costs reduced by X%

Some things did not work as expected

Service Mesh did not work well for us. The Operational costs and lack of integration on the CNI level made us revert the journey. Sidecar, eBPF Service Mesh seems to be the way forward for us.

Architecture evolution

The Architecture was designed as a single tenant but due to its success it needed to evolve to support multitenancy on AWS account level, next on namespace level and now, due to the Global impact, on the Fargate ECS level. Inevitably we ended with three Platforms(Architectures) that need to be simplify towards a maximum two that will evolve rapidly. Migration work is inevitable.

The Journey

It took us 4 years to move towards the current Platform Engineering setup. There is more to come. The full Journey as it happened till today 28.10.2024 can be seen in the following diagram: