Senior Big Data Engineer

Key Skills

Apache Airflow, Azure, Big Data Technology, Confluence, Continuous Integration & Continuous Delivery - CI/CD, Data Lake, Data pipelines, Data science techniques, DataOps, DevOps, Grafana, Java Programming, JIRA, Kannada, Machine learning techniques, Python Programming, Scala Programming, Scrum Agile Methodology, SPARK Programming

Job Description

Here in Kaizen we are convinced that harnessing value from our data is the way forward. In order to achieve our data-first approach and enable all teams to make decisions and take actions based on them, we have built our Data Lake. 

Our new architecture enables us to host all our data under one hood, making them available in one place, supporting both structured and unstructured formats, batch and real-time ingestion and serving both analytics and machine learning workloads. 

As a result, we are providing a scalable, cost-effective single point of truth for all teams to work independently and create value for the company.

This initiative is driven by the Big Data Platform team, where a team of engineers is responsible for the design and expansion of the Data Lake. 

#ourmission

The Data Lake is used by many teams: the BI, Reporting, Data Science & Machine Learning teams are our best friends. As more and more teams attest to the value the Data Lake can unlock, our product teams are joining the ever increasing group, thus this is the reason our team needs to grow as well.

Part of our day to day tasks is to find how to bring new data sources to our infrastructure and enable the downstream users to create value from the delivered data.

We are also there to take care of our Data Lake. The stability of our pipelines as well as the scalability and the efficiency of our design is crucial for achieving our company goals. As a result, part of our job is also to design, develop and deploy all relevant tools that will make our and our data infrastructure users’ life easier.

Finally, we make our expertise in distributed computing available to any team that may need it, in order to accelerate Kaizen’s data-driven approach.

Tech Stack: 

Spark, Scala, Python, Delta, Apache Airflow, Azure Databricks, Azure DevOps for source code management and CI/CD, Grafana

Supportive Stack/Processes: Kanban/Scrum, Jira / Confluence, Pair programming

As part of the Big Data Platform team, you will:

• Help shape the new era of the company data wise;
• Implement our technology roadmap to the highest of quality standards;
• Affect the technological vision of the Big Data chapter;
• Always keep track of the latest developments in our domain;
• Work on challenging tasks and constantly hone your technical skills.

What you'll bring

Expectations:

• Very good knowledge of Spark, able to write and optimize code;
• Strong software engineering skills, versed in designing and writing production level code;
• Working experience with object-oriented programming (java, scala or python, not required)
• Working experience with data pipeline orchestration frameworks;
• Experience with DevOps (ideally DataOps);
• Ability to design, plan and deliver end to end data pipelines;
• Contribute to team culture with high quality coding standards (mentoring, code reviewing);
• Ability to understand and evaluate new technologies and take design decisions;
• Working experience with Databricks (not required).

Nice to have:

• Experience with Spark monitoring tools and frameworks

As a Senior Big Data Engineer, you will be called to help the existing team move forward technologically and help establish the high quality software engineering mentality we are pursuing. Concrete working experience and deep understanding of your craft is required and respective Spark programming skills are important. What really matters is your desire and ability to help the existing team progress technologically and to be part of this endeavor from the early stages by bringing passion and an attitude to level-up our pipelines.