Data Engineer

Key Skills

Apache Hadoop, Apache Hive, AWS, Azure, Big Data Technology, C Programming, C++, Docker, Effective communication skills, Google Cloud Platform (GCP), Java Programming, Kubernetes-K8s, MapReduce, Python Programming, Scala Programming, Software Development, Teamwork

Job Description

Responsibilities

• Develop and automate large scale, high-performance, scalable platform (batch and/or streaming) to drive faster analytics and support the operations of the machine learning models in production.
• Working with data scientists to create and improve data science workflows, i.e. machine learning lifecycle management.
• Ability to design large-scale, complex applications and frameworks with excellent run-time characteristics such as low-latency, fault-tolerance and availability.
• Partner with analytic consumers and data scientists to build and improve new/existing constructs and solve data engineering problems @ scale.

Key qualifications

• BSc/BEng degree in computer science, electrical engineering, or related fields; or equivalent technical proficiency
• Solid coding skills in at least two of the following languages: Go, Java, Scala, C/C++, Python
• 6+ years of experience in software and/or data engineering
• 3+ years of Spark development
• 3+ years of experience in Big Data Technologies (Hadoop, MapReduce, Hive etc…).
• Experience on Kubernetes, Docker preferred
• Experience with cloud services (GCP, AWS or Azure)
• Good knowledge of Data formats (Parquet, ORC etc.) and consensus management systems
• Deploy inclusive data quality checks to ensure high quality of data
• Experience building early stage robust and scalable products
• Attention to detail and effective verbal/written communication skills. Teamwork essential.