Data Engineer

Key Skills

Analytical and Problem solving, Apache Hadoop, Apache Kafka, AWS, Azure, Bash scripting, Big Data Technology, Data Engineering, Data pipelines, Design, Effective communication skills, Google Cloud Platform (GCP), PySpark, Python Programming, SPARK Programming, SQL

Job Description

We’re looking for a dynamic Data engineer with Apache Spark and AWS experience to join the data analytics team at Gainwell Technologies. You will have the opportunity to work as part of a cross-functional team to define, design and deploy frameworks for data collection, normalization, transformation, storage and reporting on AWS to support the analytic missions of Gainwell and its clients. 

Your role in our mission

Essential Job Functions

• Design, develop and deploy data pipelines including ETL-processes for getting, processing and delivering data using Apache Spark Framework.
• Monitor, manage, validate and test data extraction, movement, transformation, loading, normalization, cleansing and updating processes. Build complex databases that are useful, accessible, safe and secure.
• Coordinates with users to understand data needs and delivery of data with a focus on data quality, data reuse, consistency, security, and regulatory compliance.
• Collaborate with team-members on data models and schemas in our data warehouse.
• Collaborate with team-members on documenting source-to-target mapping.
• Conceptualize and visualize data frameworks
• Communicate effectively with various internal and external stakeholders.
• Mentor and guide junior data engineers, fostering a culture of knowledge sharing and continuous learning.

What we're looking for

• 15+ years of experience in data engineering or a related field
• Technical Skills - SQL, Spark/Pyspark, Databricks, Python.
• Proven experience designing and building large-scale data pipelines
• In-depth knowledge of big data technologies (e.g., Apache Spark, Kafka, Hadoop)
• Strong proficiency in SQL and scripting languages (e.g., Python, Bash)
• Experience with cloud platforms (e.g., AWS, GCP, Azure) a plus
• Excellent communication, collaboration, and problem-solving skills
• Ability to lead and mentor a team of engineers

What you should expect in this role

Fast-paced,challenging and rewarding work environment.

Hybrid Office environment.

Will require late evening work to overlap US work hours.