Data Engineering Manager – Streaming Specialist

Our client is seeking a hands-on Data Engineering Manager to lead a small team in developing and maintaining advanced streaming data solutions, and also bridging on-premises and cloud environments. This role combines deep technical expertise in streaming technologies with team leadership, focusing on building robust, scalable data pipelines and real-time analytics systems.

Work closely with cross-functional teams, predominantly architecture, product and software engineering teams, to implement high-quality data solutions aimed at achieving the company’s business goals.

Key responsibilities

• Manage and mentor a small team of data engineers specialising in streaming technologies.
• Design and implement high-throughput, low-latency streaming data pipelines using Kafka.
• Develop and optimise stream processing applications using Apache Spark or Apache Flink.
• Implement real-time data transformations and analytics using KSQL.
• Design and maintain scalable data storage solutions with ClickHouse for fast analytics on streaming data.
• Bridge on-premises streaming solutions with cloud services like BigQuery and Looker.
• Oversee the implementation of reporting solutions using Superset.
• Ensure best practices in data engineering, including data quality, security and performance.
• Collaborate with stakeholders to define streaming data requirements and deliverables.
• Stay current with emerging streaming and cloud technologies and evaluate their potential application.

Requirements

• 5+ years experience in data engineering, with at least two years focusing on streaming technologies.
• At least two years in a leadership or management role.
• Strong hands-on experience with Apache Kafka, including cluster management and tuning.
• Proficiency in developing stream processing applications using either Apache Spark Structured Streaming or Apache Flink.
• Experience with KSQL for stream processing and real-time analytics.
• Solid understanding of ClickHouse for high-performance analytics on streaming data.
• Familiarity with Superset for data visualization and exploration.
• Excellent programming skills in Java, Scala, or Python, with a focus on stream processing.
• Experience leading and mentoring a team of data engineers.
• Experience with Kafka Connect for data integration and Kafka Streams for stateful processing.
• Familiarity with stream processing patterns like windowing, watermarking, and exactly-once processing.
• Knowledge of data modelling and optimization techniques for ClickHouse and BigQuery.
• Experience integrating on-premises streaming solutions with cloud-based analytics platforms.
• Proficiency in developing and managing data pipelines that span on-premises and cloud environments.
• Experience with cloud services, particularly BigQuery and Looker.
• Experience with real-time machine learning model serving in streaming environments.
• Experience with monitoring and observability tools for streaming systems across on-premises and cloud environments.

Benefits

• Bi-Monthly company-wide social and team-building activities.
• Lunch vouchers.
• Health premium.
• Training and development opportunities
• Hybrid and remote work arrangements.
• Flexible working hours.