Principal Data Engineer (ML Engineer) - QuantumBlack

Key Skills

Python Programming, PySpark, SQL, Apache Airflow, Databricks, Kedro, AWS, Google Cloud Platform (GCP), Azure, Scala Programming, RDBMS, PostgreSQL, Amazon RedShift, Teradata, Snowflake, NoSQL, Apache Cassandra, Neo4J, Titan, MongoDB

Job Description

Who You'll Work With

You will be based in Tokyo and be part of a global Data Engineering community and you will work in cross-functional and Agile project teams alongside project managers, data scientists, machine learning engineers, other data engineers, and industry experts.

You will work hands in hands with our clients, from data owners and users to C-level executives.

Who you are

You are a highly collaborative individual and enjoy solving problems that focus on adding business value. You have a sense of ownership and enjoy hands-on technical work. Our values resonate with yours.

What You'll Do

As a Data Engineer in Tokyo, you will...

• Partner with our clients, from data owners and users to C-level executives, to understand their needs and build impactful analytics solutions
• Design and build data pipelines to support data science projects following software engineering best practices
• Use state of the art technologies to acquire, ingest and transform big datasets
• Design and build Data/Analytcs platform
• Map data fields to hypothesis, curate, wrangle and prepare data to be used in advanced analytics models
• Create and manage data environments in the cloud or on premise
• Ensure information security standards are maintained at all time
• Contribute to cross-functional problem-solving sessions with your team and deliver presentations to colleagues and clients
• Be flexible to travel to our clients' offices to deliver presentations, gather information or share knowledge
• Have the opportunity to contribute to R&D and internal asset development projects

Our tech stack

While we advocate for using the right tech for the right task, we often leverage the following technologies: Python, PySpark, SQL, Airflow, Databricks, our own OSS called Kedro, container technologies such as Docker and Kubernetes, cloud solutions such as AWS, GCP or Azure, and more!

What you'll benefit from

• Real-World Impact – No project is ever the same. We work with top-tier clients across multiple sectors, providing unique learning and development opportunities internationally.
• Fusing Tech & Leadership – We work with the latest technologies and methodologies and offer first class learning programmes at all levels.
• Multidisciplinary Teamwork - Our teams include data scientists, engineers, project managers, UX and visual designers who work collaboratively to enhance performance.
• Innovative Work Culture – Creativity, insight and passion come from being balanced. We cultivate a modern work environment through an emphasis on wellness, insightful talks and training sessions.
• Striving for Diversity – With colleagues from over 40 nationalities, we recognise the benefits of working with people from all walks of life.

Visit our Careers site to watch our video and read about our interview processes and benefits.

Qualifications

• Degree educated in computer science, engineering, mathematics, or equivalent experience
• 5+ years of previous commercial experience as a leader in a data-driven role
• Ability to write clean, maintainable, and robust code in Python, Scala, Java or similar languages
• Deep understanding and expertise of software engineering concepts and best practices
• Deep knowledge and expertise of testing frameworks and libraries
• 2+ years of experience as a leader in Data/Analytics platform architecture design
• Hands-on experience of technologies like Spark, Hadoop/MapReduce 
• Deep familiarity with OSS, cloud, container, query languages and database technologies
• 2+ years of hands-on experience building data pipelines in production and ability to work across structured, semi-structured and unstructured data
• 2+ years of experience in ML pipeline for streaming/batch workflow 
• Commercial client-facing or C-level stakeholder management experience is desired 
• Collaborative - able to build strong relations that enable robust debate, and survive periodic disagreements regarding priorities.
• Ability to communicate effectively with technical and non-technical audience
• Strong command of both English and Japanese (N2 or above)