Lead Data Engineer

Key Skills

Java Programming, Design, Python Programming, SQL, DevOps, Big Data Technology, Apache Kafka, Apache Hive, Continuous Integration & Continuous Delivery - CI/CD, PySpark, SPARK Programming, Database testing, AWS Kinesis, ETL frameworks, VR/AR frameworks, Data Warehousing

Job Description

At Discover, be part of a culture where diversity, teamwork and collaboration reign. Join a company that is just as employee-focused as it is on its customers and is consistently awarded for both. We’re all about people, and our employees are why Discover is a great place to work. Be the reason we help millions of consumers build a brighter financial future and achieve yours along the way with a rewarding career.

Responsible for applying a hands-on approach with next-generation technologies to contribute to the team by designing and developing the latest data-driven platforms and next-generation analytic technologies.

Responsibilities

• Designs and develops data ingestion frameworks, real-time processing solutions, and data processing/transformation frameworks, leveraging open-source tools.
• Provides senior-level technical consulting to peer-data engineers during design and development for highly complex and critical data projects
• Deploys application codes and analytical models using CI/CD tools and techniques and provides support for deployed data applications and analytical models.

Strong knowledge and experience in designing/using of DW Cloud Database :

• Snowflake is preferred
• Strong knowledge and experience in designing and development of complex Data Warehouse methodologies
• Experience in processing of huge dataset of hundreds of millions of records in Cloud and as well as in on-premise.
• Strong knowledge of ETL technologies – Ab Initio on Cloud is preferred, or – DataStage, Informatica.
• Experience in on-premise ETL development using Data Warehouse DB as backend – Teradata is preferred.
• Experience in performance tuning - both in ETL processes and Database objects
• Experience in end to end DevOps – CI/CD
• Experience in creating technical design plans/STT, solutioning, designing and delivering complex Datawarehouse projects
• Experience in working closely with Business teams to get the understanding of their requirements.
• Experience in grooming, mentoring the teams in self-managed teams
• Experience in supporting live systems and business continuity.

Minimum Qualifications

At a minimum, here’s what we need from you:

• Bachelor's Degree in Information Technology, or related field
• 4+ years of experience in Data Platform Administration/Engineering
• Knowledge in AWS and any AWS certifications are preferred.
• Experience in Agile based environment
• Expertise in open-source, ETL scripting and related tools.
• Strong team player and willing to collaborate
• Strong analytical and problem-solving skills
• Strong communication and decision-making skills

Preferred Qualifications

If we had our say, we’d also look for:

• 6+ years of experience in Data Platform Administration/Engineering
• Good understanding of Big Data technology trends, with knowledge of technologies such as Kinesis, Kafka, Spark, Hive, pySpark.
• Experience with one of the ETL tools Abinitio, Informatica or Data Stage is a plus.
• Prior experience with Banking or Financial domain is a big plus.