Specialist IT Big Data

Key Skills

Agile methodologies, Apache Hadoop, Apache Hive, AWS, Azure, Big Data Technology, Business Analytics, Data Modeling, Data pipelines, Data Warehousing, ETL frameworks, Google BigQuery, Google Cloud Platform (GCP), Google Dataflow, HBase, IT, Java Programming, Python Programming, Scala Programming, SPARK Programming, SQL

Job Description

Our team will be responsible for migrating legacy analytic & business intelligence solutions and their associated data pipelines across Vodafone onto the Google Cloud Platform, enhancing and developing new solutions.

The Data Engineer delivers data pipelines and data service components. This will be at industrial scale delivering 20+ countries local data to a central “Data Ocean” hosted in the cloud. The data engineer will need to design highly efficient pipelines for performance and reliability. Custom components will need to be developed to ensure privacy and integrity of the data. As the data ocean is established the data engineer will also expose data with a common interface allowing the monetization of data across the whole of Vodafone in a controlled and secure fashion. Key focus for and engineer will be: 

1. Work with Business Analyst, Test Engineer and Product Owner in an Agile Squad to create data pipelines for each local market migration.
2. Integrate the necessary diverse and distributed data from multiple sources transforming this to a central data format for each country. Replacing existing EDW with a central solution on Google Cloud Platform. 
3. Build data pipelines that make use of large volumes of data and generate outputs that allow commercial and business actions that generate incremental value. 
4. Create semantic layers for Business Intelligence users and feature engineering for data science. 
5. Deliver and implement core capabilities (frameworks, platform, development infrastructure, documentation, guidelines and support) to speed up the Local Markets delivery in the BI and Analytics in the Cloud

As a Specialist IT Big Data, you´ll be in charge of the following responsibilities: 

• Designing and producing high performing stable end-to-end data pipelines to perform complex processing of batch and real-time massive volumes of data in a multi-tenancy Big Data environment and output insights back to business systems according to their requirements.
• Design and implement core platform capabilities, tools, processes, ways of working under agile development to support the integration of local markets data sourcing and use cases implementation, towards reusability, to easy up delivery and ensure standardisation across Local Markets deliverables in the platform.
• Working with the architecture team to define the strategy for evolving the data movement and transformation capability, including solution architectural decisions aligned with the platform architecture
• Defining the technologies to be used on the “Data Ocean” and investigating new technologies to identify where they can bring benefits

What do you bring?

• Experience in SQL (preferably knowledge of Big Query) and data modelling (DWH models)
• Experience with data pipelines tools (Data Flow, Data Fusion) in Cloud environments like GCP, Azure or AWS
• Knowledge of Hadoop ecosystem (Spark, Hive/Impala, HBase, Yarn)
• Experience in Java, Scala or Python programming languages; other functional languages desirable
• Worked in Agile environment 
• Experience working in EDW or Data Warehouse solutions and ETL
• Experience in designing, building and managing data pipelines to process large amounts of data in a Big Data ecosystem
• Experience in data pipelines and data layers on Google Cloud Platform