Principal Data Engineer - GEM Team

Key Skills

Apache Airflow, API, Databricks, ETL frameworks, ETL/ELT technologies, Spark Core, SQL

Job Description

Working at Atlassian

Atlassian can hire people in any country where we have a legal entity. Assuming you have eligible working rights and a sufficient time zone overlap with your team, you can choose to work remotely or from an office (unless it’s necessary for your role to be performed in the office). Interviews and onboarding are conducted virtually, a part of being a distributed-first company.

We're able to hire eligible candidates for this role from any location in Australia and New Zealand. If this sparks your interest, apply today and chat with our friendly Recruitment team.

Your future team

Developing Atlassian into a world-class engineering organisation is a high priority. The Engineering Excellence org focuses on improving all aspects of engineering including developer tooling, developer productivity and standards and practices. We're hiring a Principal Data Engineer to help build a team that acts as a centralised data engineering capability across Engineering Excellence. You will design and build data solutions to collect actionable insights aimed at making improvements in all of these crucial areas and collaborate with multiple teams.

You will report to the Engineering Manager of the Global Engineering Metrics team (GEM). The team uses data from our data lake to produce insights about the effectiveness of engineering teams and to help them raise the engineering bar. Our data is also used to make crucial decisions about tooling and standards. The GEM team produces dashboards to visualise the data so our teams can quickly and easily track their progress.

What you'll do

• Design and implement data pipelines to ETL data from multiple sources into a central data warehouse
• Maintain data processing infrastructure, including databases, data lakes, and data warehouses
• Ensure data quality and consistency across data systems
• Work closely with data scientists to build robust analytics models and define new metrics
• Collaborate with other teams such as software engineers, product managers, and data scientists to understand data needs and develop solutions to meet those needs
• Implement new technologies to improve data processing and analysis
• Coach junior data engineers to improve their skills
• Shape engineering process and set standards for the team (code quality, testing, code reviews)
• Set goals and success metrics for the team with an awareness of how the work relates to company-level OKRs

Your background

• Bachelors, Masters or PhD in Computer Science, Engineering, Information Management or other technical fields, and 15 or more years of data engineering experience
• Experience partnering across engineering teams
• Experience leading ambiguous and complex data engineering projects
• Experience partnering with cross-functional teams to accomplish org-level goals
• High proficiency in SQL, ETL, and scripting/programming to solve a wide array of data challenges
• Experience building scalable data pipelines in Spark using Airflow scheduler/executor framework or similar scheduling tools. Experience with Databricks and its APIs is a plus.
• Understanding of Data Engineering tools, frameworks and standards to improve the productivity and quality of output for data engineers in your team
• Industry experience working with large-scale, high-performance data processing systems (batch and streaming) with a Streaming First mindset to drive Atlassian's business growth and improve the product experience.

Our perks & benefits

To support you at work and play, our perks and benefits include ample time off, an annual education budget, paid volunteer days, and so much more.