DataBricks Engineer

Key Skills

Analytical and Problem solving, Automation, Azure, Big Data Technology, Data science techniques, Databricks, Docker, Git, IT, Jenkins, Linux Operating system, NoSQL, Python Programming, SPARK Programming, SQL, Unix Operating system

Job Description

About the Role

This position will be engaged in design and deploy of large-scale data processing frameworks and pipelines on Databricks to support enterprise-wide BI and data science initiatives. It will also provide user training and data coaching for effective Databricks usage.

Your Responsibilities

• Create MLOps frameworks for model development, training, deployment, and monitoring.
• Optimize Databricks clusters for performance, scalability, and resource efficiency.
• Monitor cluster health, performance metrics, and resource usage, adjusting cluster sizes as needed.
• Design efficient data processing systems and pipelines using Databricks, APIs, and other cloud services.
• Act as the Databricks expert and uphold platform standards.
• Collaborate with Data and Platform architects on defining the product roadmap.

Who we are looking for

• Bachelor's degree in Computer Science or related field.
• 7+ years of IT experience, including 5+ years in Databricks development and administration, Spark, and cloud platforms (preferably Azure).
• Proficiency in Databricks admin and development, with expertise in Resource and Performance Management.
• Expertise in test automation, continuous integration, Docker, GitHub, Jenkins, and Unix/Linux.
• Leadership in large project initiatives, willingness to mentor, and support for team members from different time zones.

Your preferred qualifications

• Experience with Kafka and Spark, and strong in Python and Pyspark.
• Hands-on experience in machine learning pipelines and scripting languages.
• Familiarity with Java, Python, SQL/NoSQL databases, and Big Data tools.
• Passion for new tech, continuous improvement, and collaborative problem-solving.
• Proactive, strong work ethic, and team player.

What we offer

At Magna, you can expect an engaging and dynamic environment where you can help to develop industry-leading automotive technologies. We invest in our employees, providing them with the support and resources they need to succeed. As a member of our global team, you can expect exciting, varied responsibilities as well as a wide range of development prospects. Because we believe that your career path should be as unique as you are.