Lead Data Engineer

The Role

The role

As a Lead Data Engineer, both directly and via your team of data engineers, you will be responsible for developing, maintaining, and optimising our data pipeline infrastructure using our proprietary central data platform, which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis.

This role offers the chance to lead impactful projects, work with cutting-edge technologies, and shape the future of data engineering at ITV. If you’re passionate about data, leadership, and innovation, we’d love to hear from you!

Key responsibilities

• Lead and Inspire: Take charge of a talented team of data engineers, providing technical leadership, mentorship, and fostering a collaborative environment. Drive productivity, set clear objectives, and support professional growth while contributing to performance reviews and recruitment
  
• Design & Build Scalable Data Solutions: Own the end-to-end design, development, and maintenance of cutting-edge data pipelines and infrastructure using Databricks. Innovate with new approaches, ensuring efficient data ingestion, transformation, and storage at scale
  
• Collaborate Across Teams: Work closely with cross-functional teams, including data scientists, analysts, and commercial stakeholders, to define data product requirements and deliver impactful data visualisation solutions
  
• Performance & Quality Assurance: Monitor and optimise data pipelines for peak performance and reliability. Implement automated quality processes to ensure data accuracy and report outcomes through advanced observability tools
  
• Governance & Security: Champion data governance practices, ensuring compliance with data privacy and security regulations. Establish access controls, encryption, and retention policies to safeguard sensitive information
  
• Continuous Improvement: Stay ahead of industry trends, identifying opportunities for innovation and automation. Engage with the ITV Symphony team to drive feature enhancements and contribute to the broader data engineering community.

Qualifications and Skills

• Experience in leading a small team of data engineers
• Extensive knowledge as a Data Engineer
• Demonstrable success in designing and building Databricks/Snowflake/GCP Big Data/Hadoop (Cloudera/HortonWorks)/Spark data products
• Excellent problem-solving, analytical thinking, and troubleshooting skills
• Strong communication and collaboration abilities, with the ability to work effectively in a team environment
• Strong programming skills in languages such as Python (PySpark preferred), Scala, or SQL
• Experience in owning, designing and implementing data pipelines ingesting enterprise levels of data volume. In-depth knowledge of data engineering concepts, data integration, and ETL processes
• Ability to write production-grade code including automated testing
• Experience in deploying code via CI/CD platforms (e.g. Github Actions, Jenkins)
• Proficiency in working with distributed computing frameworks, such as Apache Spark
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and associated services (e.g., S3, Redshift, BigQuery)
• Familiarity with data modelling, database systems, and SQL optimisation techniques

Other things we’re looking for (key criteria)

• Knowledge of the UK broadcast industry & broadcast/OTT advertising market
• Knowledge of digital marketing and advertising industry
• Familiar with data design techniques for classification and regression as well as knowledge of A/B testing, experimental design, and general statistical modelling
• Familiarity with architecture disciplines such as Data Mesh Architecture, Data Architecture, BI Architecture and Enterprise Architecture
• Good understanding of data governance, data privacy, and security principles.
• Knowledge of tools across other data management capabilities – data engineering, data governance, data quality, metadata management and master data management
• Knowledge of designing and building out data visualisation solutions based on technology such as: Tableau, AWS QuickSight, Looker, ThoughtSpot