Azure Data Engineer With Pyspark

Key Skills

Java Programming, Python Programming, C++, C Programming, SQL, Cloud computing, Scala Programming, Machine learning techniques, Data science techniques, MATLAB Programming, PyTorch, TensorFlow, R Programming

Job Description

Tasks

• Implementing data ingestion pipelines from multiple data sources using Azure Data Factory, Azure Databricks and other ETL tools.
• Developing Big Data and non-Big Data cloud-based enterprise solutions in PySpark and SparkSQL and related frameworks/libraries.
• Developing scalable and re-usable, self-service frameworks for data ingestion and processing.
• Integrating end to end data pipelines to take data from data source to target data repositories ensuring the quality and consistency of data.
• Processing performance analysis and optimization.
• Bringing best practices in following areas:
• design & analysis,
• automation (Pipelining, IaC),
• testing,
• monitoring,

Requirements

Essential technical skills:

• At least over 2 year of experience in Azure development.
• Having PySpark exposure, or intent to learn it.
• Experience in building ETL/ Data Warehouse transformation process.
• Very good knowledge of SQL.
• Hands-on experience designing and delivering solutions using Azure DataLake.
• Direct experience of building data pipelines using Azure Data Factory, Azure Databricks or Azure Synapse.
• Experience working with structured and unstructured data.
• Experience working in Agile environment (preferably Scrum).
• Experience in cloud-based solutions.
• Knowledge of data management principles.

Required softskills:

• Self-driving, able to seek out new tasks when old ones complete.
• Able to manage own time, estimate when work will be done, fill gaps with related work.
• Understands when to provide opinion versus seek specialist domain knowledge.
• Eager to learn new tools.
• Able to communicate clearly amongst team members.

Nice to have:

• Big data related topics (Hadoop stack, HDInsight, Azure Databricks) knowledge.
• Data Science knowledge (AI, ML).

Offer

• Stable employment. On the market since 2008, 1200+ talents currently on board in 7 global sites.
• “Office as an option” model. You can choose to work remotely or in the office.
• Flexibility regarding working hours and your preferred form of contract.
• Comprehensive online onboarding program with a “Buddy” from day 1.
• Cooperation with top-tier engineers and experts.
• Certificate training programs. Lingarians earn 500+ technology certificates yearly.
• Upskilling support. Capability development programs, Competency Centers, knowledge sharing sessions, community webinars, 110+ training opportunities yearly.
• Grow as we grow as a company. 76% of our managers are internal promotions.
• A diverse, inclusive, and values-driven community.
• Autonomy to choose the way you work. We trust your ideas.
• Create our community together. Refer your friends to receive bonuses.
• Activities to support your well-being and health.
• Plenty of opportunities to donate to charities and support the environment.
• Modern office equipment. Purchased for you or available to borrow, depending on your location.