Principal Big Data Engineer

Key Skills

Apache Airflow, Apache Kafka, Azure, Big Data Technology, Data Architecture, Data science techniques, Data structuring, Data Warehousing, Database, Databricks, Design, DevOps, Effective communication skills, ETL frameworks, Infrastructure as code, JIRA, Machine learning techniques, NoSQL, Python Programming, SPARK Programming, SQL, Terraform

Job Description

We are looking for a Principal Big Data Engineer to guide and influence our data segment in order to deliver scalable data and analytics solutions, implement our own data platform and participate in solution design and architecture discussions. The ideal candidate will have outstanding communication skills, shown data design and implementation capabilities and a drive to deliver results. The person in this role will be technically proficient and excel at collaboration and will stay current on emerging technologies, methodologies, and best practices. This person will be a self-starter, and will enjoy working in a fast-paced dynamic environment.

The Team

Our Data segment is part of our Data & AI tribe and consists of several data engineering team(s) with diverse backgrounds and expertise in several fields including, but not limited to, Spark / Scala / Python / Databricks / Delta lake / Azure Data Factory / Azure Data Warehouse / Azure DevOps / Terraform.

Responsibilities:

• Contribute to the design and development of a scalable and secure Big Data Platform aligned with the Data Mesh paradigm;
• Optimize data pipelines, ETL processes, and data storage for performance, cost-efficiency, and reliability;
• Assist in implementing data security and compliance policies in alignment with industry standards;
• Stay current on emerging technologies, methodologies, and industry best practices to ensure continuous improvement of the data platform;
• Communicate effectively and provide technical insights inside and outside of the organization on best practices for data architecture, technology selection, and solution design;
• Partner with management to find new potential talent.

Requirements:

• Minimum of 8 years of experience in designing, developing, and implementing large-scale data processing systems;
• Proven expertise with Azure and Databricks, including experience in designing and deploying data pipelines, analytics solutions, and storage architectures;
• Strong understanding of the Data Mesh paradigm and its implementation in large organizations;
• Familiarity with a wide range of big data technologies, such as Apache Spark, Delta Lake, Apache Kafka, Apache Flink and NoSQL databases;
• Knowledge of data modeling, data integration, data warehousing, and data quality management principles;
• Experience in implementing data security and compliance policies in alignment with industry standards;
• Strong communication skills, with the ability to effectively convey complex technical concepts to diverse audiences;
• Self-motivated and results-driven, with a passion for staying current on emerging technologies and industry trends.