Big Data Lead

Key Skills

Azure, Big Data Technology, Continuous Integration & Continuous Delivery - CI/CD, Databricks, ETL/ELT technologies, Java Programming, PaaS, PySpark, Python Programming, Scala Programming, Snowflake, SQL

Job Description

Databricks Architect:

• 10- 12 years of experience in building solutions using Databricks on Azure or on Big Data, SQL Data, and other sources.
• In depth understanding of DataBricks Architecture including Lakehouse, Performance tuning, Compute and Storage
• 1-2 Migration project level experience from Hadoop ecosystem to Azure Databricks & related DBs along with data integration job migration.
• Good knowledge and Proven practical experience in migrating relational data base from on-prem to cloud OR to build cloud-based data lake using either Big Data / Azure Synapse / Databricks / Snowflake
• Experience in building data platforms using Azure services like Azure Data Factory, ADLS Gen2, Databricks, Azure IaaS/PaaS database and Azure Synapse data warehouse/analytical systems.
• Worked on the both the development and maintenance projects and followed the operational processes.
• Hands-on experience on selecting and integrating any Data related tools and frameworks required to provide requested capabilities.
• Responsible for end-to-end data platform design and architecture for DataBricks on Azure.
• Extensive experience writing and tuning SQL queries.
• In-depth knowledge in data virtualization using DataIQ.
• 3+ years programming in one or more languages (PySpark, Python, Java, Scala)
• Excellent in Written and Verbal Communication
• Experience with Agile process methodology, CI/CD.
• Hands-on experience in ETL/ELT tools like FiveTran, Talend and Informatica Power Centre.
• Certification on cloud or DataBricks (related to Data) would be preferred.
• Knowledge of Hadoop ecosystem will be an added advantage.