KAFKA MIGRATION

Sonae.

Kafka migration to on-premise infrastructure

Industry: Retail Conglomerate
Category: OUTSOURCING
Stack: Kafka · MirrorMaker2

About the client

Sonae is a multinational business group headquartered in Maia, Portugal. Present in 90 countries, it operates across multiple sectors, including retail, financial services, shopping center management, software and information systems, media, and telecommunications. It is Portugal's largest private employer, with more than 50,000 employees.

Challenge

Sonae kept Kafka running on Azure HDInsight, far from the applications and databases already hosted in the on-premise datacenter. This network distance increased latency and dependency on the link, while limiting the team's control over version, tuning, capacity, and platform evolution.

Solution

Migration of Kafka to on-premise VMs, with continuous replication of topics and data via MirrorMaker2 and a phased cutover: consumers first, producers next. Provisioning and configuration via Ansible and AWX, observability with Grafana and Prometheus, and capacity management with Cruise Control.

Result

Reduced network distance between Kafka and the applications, with full platform control shifting to the engineering team, along with a significant reduction in monthly operating costs. The solution includes failover and failback between datacenters with state preservation via shared storage, and layered observability for fast diagnostics.

Stack & practices