Data Engineer – Asia Data Office

Key Skills

Java Programming, Python Programming, SQL, DevOps, Azure, Apache Hadoop, Apache Kafka, Apache Hive, MapReduce, Google Cloud Platform (GCP), R Programming, SPARK Programming, HBase

Job Description

The Asia Data Office is looking for an experienced Data Engineer to join our team to support our regional partners. If you have passion for data and a desire to drive quality data solutions within a data driven culture, then read on to learn more about this opportunity!

We are looking for the right individual who would be interested and enjoy the opportunity to support data operations, strong willingness to learn and build data operations skills through team collaboration, desire to drive and execute on our data strategy, willing to help mature our agile practices, and is looking for an opportunity to embrace technical, operational, and informal leadership roles on the team.

Duties

• Selecting and integrating any Big Data tools and frameworks required providing requested capabilities
• Implement ETL/ELT flows via Big Data Solutions i.e Nifi, Spark, Python, Azure Data Factory, Azure Synapse, Azure DataBricks…etc
• To be able to monitor the performance and advice necessary infrastructure changes or performance tuning in codes.
• Support business users in use of the Enterprise Data Lake (EDL) as part of BAU
• Perform POC on new integration patterns and solutions.
• Maintain and monitor platform stability and performance of EDL
• Attends to Incidents and change requests
• Write and maintain technical documentation
• Perform unit tests and system integration tests
• Executes updates, patches, and other activities required to maintain and enhance the operations of the EDL
• Supports the Agile delivery squads when required

Skills and Qualifications

• Proficient understanding of distributed computing principles
• Good Knowledge of Hadoop cluster, Azure, with all included services
• Ability to solve any ongoing issues with operating the cluster
• Proficiency with Hadoop v2, MapReduce, HDFS
• Experience with Azure ADLSg2, Data Factory, DataBricks, Eventhub,
• Experience with data engineering projects
  i.e integration of data from multiple data sources to a target system
• Good knowledge of Big Data querying tools – Hive, HBase, SQL
• Knowledge in data streaming (e.g. Kafka, Spark)
• Knowledge in any server-side programming languages (e.g. Java, Python, Spark, R…etc)
• Knowledge in any Change Data Capture (CDC) solutions
• Experience in working on Agile projects is an advantage.
• Experience on Azure Cloud and/or other Cloud platforms
• Open Source RDBMS management including MongoDB & HBase
• Experience in managing Zeppelin, JupyterHub, R Studio, Neo4j are definitely added advantage

Education and experience

• Bachelor's degree preferably in Computer Science or Data/Engineering related courses
• Recognized qualification in Service Management helpful
• ITIL certification helpful
• 6+ years in one or more of the following

• Data Engineer
• Platform Engineer
• DevOps Engineer
• Systems Engineer
• Systems Administrator

• Project management experience is an advantage
• Understanding of the insurance and/or financial industry is an asset