Data Engineer

Key Skills

Continuous Integration & Continuous Delivery - CI/CD, Data Analytics, Data Engineering, Data pipelines, Data science techniques, Design, DevOps, GitLab, Infrastructure as code, Java Programming, Jenkins, NoSQL, Python Programming, RDBMS, Rust, SPARK Programming, SQL

Job Description

What you will be working on:

• Designing, building, and maintaining highly available and scalable data infrastructure on the Cloud
• Designing, developing, and maintaining reliable data ingestion and transformation pipelines using modern workflow orchestrator
• Designing, developing, and maintaining robust code repository and reliable CI/CD pipelines
• Designing, developing, and maintaining software packages that abstract out the repetitive procedures for extracting, loading, and transforming raw data into valuable cleaned data
• Collaborating with Data Analysts, Data Scientists, ML Engineers, Software Engineers, Product Managers, and Business Stakeholders of varying functions to ensure the veracity and the timeliness of the data processed
• Publishing high-quality technical documentation in company's internal Wiki and conducting regular knowledge sharing, hands-on training, and mentoring for the less experienced team members
• Being the VP's and Manager's technical counsellor/researcher in designing a future-ready data infrastructure

Who you are:

• You have 5+ years of Data Engineering experience, or equivalent experience of handling 500+ daily data pipelines in Terabytes of scale
• You enjoy working at the intersection of Data Engineering and DevOps
• You have hands-on experience in managing Kubernetes cluster on the Cloud
• You are adept at setting up data pipeline CI/CD along with its tests using tools such as Jenkins, Gitlab CI, or equivalent
• You are fluent in Python, SQL, and Bash scripting for building data infrastructure and for developing data ingestion and transformation pipelines — fluency in Java/Go/Rust is a plus
• You know how to work with modern workflow orchestrators — Airflow is a must
• You are skillful at developing abstraction layer on top of workflow orchestrator to enable data transformation at scale
• You are familiar with various flavour of RDBMS and Columnar Query Engines — familiarity with NoSQL is a plus
• You know how to work with Apache Spark for both batch and stream data processing — familiarity with Apache Flink or Beam is a plus
• You are naturally inclined into optimising complex systems, both performance-wise and cost-wise