Big Data Lead

Key Skills

Analytical and Problem solving, Azure, Big Data Technology, Data Lake, Data pipelines, Data Transformation, Databricks, Effective communication skills, ETL frameworks, Flax Programming Language, PowerShell Programming, Presentation skills, PySpark, SQL

Job Description

As a Senior Azure ADF Engineer, you will be responsible for designing, developing, and maintaining data pipelines using Azure Data Factory (ADF). You will work closely with customers to understand their business requirements and translate them into scalable, performant, and secure data integration solutions. You will be responsible for ensuring that the data pipelines integrate seamlessly with other Azure services, such as Azure Blob Storage, Azure Synapse Analytics , SQL Database, and Azure Data Lake Storage.

Key Responsibilities:

- Design, develop, and maintain data pipelines using Azure Data Factory (ADF).

- Work closely with customers to understand their business requirements and translate them into scalable, performant, and secure data integration solutions.

- Collaborate with other architects, engineers, and technical teams to ensure that data pipelines are aligned with overall solution architecture and best practices.

- Optimize data pipelines for performance, scalability, and cost efficiency.

- Develop and maintain technical documentation, including pipeline diagrams, data flow diagrams, and code documentation.

- Participate in data-related aspects of pre-sales activities, including solution design, proposal development, and customer presentations.

- Stay current with emerging technologies and industry trends related to data integration and management in Azure, and provide recommendations for improving existing solutions and implementing new ones.

 :

- Bachelor's or master's degree in computer science, engineering, or a related field.

- At least 7 years of experience in data engineering, including at least 3 years of experience with Azure Data Factory.

- Strong knowledge of Azure Data Factory, including data ingestion, transformation, and orchestration.

- Experience with data integration and ETL processes and writing SQL queries and SQL/Scripts.

- Experience with Azure Data lake and building pipelines for batch and CDC (change data capture).

- Experience with implementing ETL frameworks.

- Strong understanding of data security, including encryption, access control, and auditing.

- Excellent communication and presentation skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical audiences.

- Strong problem-solving and analytical skills, with the ability to identify and resolve complex technical issues.

Good to have Qualifications:

- Microsoft Certified: Azure Data Engineer Associate certification.

- Experience with other Azure data services, such as Azure Synapse Analytics and Azure Databricks.

- Experience with other data integration tools, such as Talend.

- Experience with programming languages, such as SQL, Powershell.