Big Data Engineer

Key Skills

Apache Airflow, Azure, Big Data Technology, Data pipelines, Data Visualization, Databricks, Optimization, Oracle, PySpark, Python Programming, RDBMS, SPARK Programming, SQL

Job Description

We are looking for a Big Data Engineer – for one of our large Global Investment Banking Customer for Raleigh, NC.

This role will be remote until the pandemic is over.

Job Description:

• Demonstrable experience designing and developing big data applications using Apache Spark and Airflow
• Experience leveraging Databricks to develop Spark based applications along with very good understanding about data engineering capabilities provided by the service
• Excellent Python programming skills and experience delivering projects using PySpark is a must. Scala programming experience is nice to have.
• Experience designing and developing Airflow DAGs, Operators etc…
• Experience leveraging Azure Storage and Data Lake Store Gen 2 in Azure based big data projects along with integration to other Azure Data services
• Experience designing and developing real time data processing applications using Spark streaming, Kafka, Event Hub etc..
• Experience using Cloud data warehouse technologies such as Snowflake is an value add
• Must have expertise on RDBMS solutions such as Oracle, SQL Server, Azure SQL DB etc..
• Understands the value of diversity in the workplace and is dedicated to fostering an inclusive culture in all aspects of working life so that people from all backgrounds receive equal treatment, realize their full potential and can bring their full, authentic selves to work

Qualifications

• Overview of business area or project We are building a multi-tenanted Big Data Platform for the bank running on Azure. This Big Data platform will be leveraged for running numerous Big Data Pipelines for key divisions within the bank, and will be building custom libraries/Artifacts to help its tenants process massive amounts of data at scale.

• Overview of department/team : We are a dynamic team of 30 developers, with 5-10 Data Engineers who help build big Data Pipelines leveraging Azure Databricks and orchestration tools like Airflow running on Azure. The platform team comprising of Data Engineers, Application engineer, Dev Ops/Cloud Automation Engineers, collectively work together to build capabilities on the Big Data platform that will be leveraged by tenants for consumption.
• Candidate Value Proposition : You will get to work on a cutting edge Big Data Multi tenanted Platform running on Azure, and leveraging the latest technology (Open source) stack which will expose you to Data Ingestion, Data Validation, Profiling, Anomaly detection and Visualization to see data with a different lens. If you are looking for continuous improvement and staying upto date on the latest technical stack from Big Data perspective, this team is the right fit for you. You will be required built custom libraries for platform in Python, create/augment/optimize Data bricks Notebooks and enhance the existing orchestration engine (running in Airflow).