Lead Data Engineer

Key Skills

Apache Hadoop, Big Data Technology, Data Warehousing, Design, Effective communication skills, Java Programming, Planning and organization, Python Programming, Scala Programming, SDLC, SLA, SPARK Programming

Job Description

• Successfully lead major projects and complex assignments with broad scope and long-term business implications. Create solutions with high level of innovation, cost effectiveness, high quality and faster time to market.
• Lead day to day system development and maintenance activities of the team to meet service level agreements (SLAs). Collaborate with business and technical partners to identify and scope the opportunities, outline potential value and ROI, identify risks, benefits and constraints. Contribute to the overall strategy and manage complex issues within functional area of expertise.
• Develop overall strategic technical plan and create architecture proposals based on identified solution gaps. As recognized subject matter expert, lead planning, design and implementation of technical solutions. Create solution definition and solution architecture. Assist Management in Business Case Development and Scenario Planning leading to an effective decision making process. Make presentations to the Management, Business Owners and other Stakeholders.
• Assist in examining commercially available and forthcoming industry technologies related to area of expertise. Research on current and upcoming technologies and application appropriate frameworks and provide recommendations to Management. Perform PoCs (Proof of Concept) and help Department in the selection of Technology. Proactively create and maintain solution and technology road maps.
• Partner with Project Managers in implementation of processes per the evolving needs of the team and the solutions. Create detailed cost and time estimates for implementing business solutions, define technical scope and requirements of the project and assist project manager with managing scope during the lifecycle of the project. Identify Quality methodology and define Quality Metrics for the Team. Define and Lead Quality Assurance and Quality Control processes in the department. Assist in vendor selection and RFI/RFP process.
• Lead Quality Assurance and Quality Control processes in the department. Ensure adequate test coverage in Unit Testing, System Testing/Integration Testing and Performance Testing. Perform Quality Inspections and Walkthroughs through out the SDLC including Requirements Review, Architecture Review, Design Review, Code Review and Security Review to ensure compliance with MasterCard standards. Co-ordinate offshore development and manage day to day activities. Support collection and reporting of project and software metrics. Actively look for opportunities to enhance standards and improve process efficiency. Mentor and guide other team members during all phases of the SDLC.
• Lead and guide team through all phases of the SDLC of an application. Successfully lead production incident and problem management, identification of root cause and remediation of the problems. Mentor, coach and assist project team members. Provide Brownbag sessions on new and upcoming technologies, methodologies and frameworks. Provide technical training to the team. Assist with selection of technical resources for the projects.
• Ensure compliance to the defined standards. Actively look for opportunities to modify and enhance standards per the needs of the department. Develop documentation templates, lead and enforce development of documentation through out the development lifecycle of the solution.
• Ensure compliance of Audit Requirements by proactively educating team on Compliance Requirements and Integrating Compliance Requirements in SDLC. Represent team during Compliance Audits.

Education:

• Bachelor's degree in Information Systems, Information Technology, Computer Science or Engineering or equivalent experience.

Knowledge / Experience:

• 10+ years of experience in related field, including 6-8 years of experience in delivering secure solutions in Financial Services Sector is preferred.
• Advanced knowledge and understanding of Software Engineering Concepts and Methodologies is required.
• Demonstrate MC Core Competencies.

About you:

• Ability to be high-energy, detail-oriented, proactive and able to function under pressure in an independent environment along with a high degree of initiative and self-motivation to drive results.
• Ability to quickly learn and implement new technologies, frameworks and tools and support multiple concurrent activities and to interface with external / internal resources, working as a member of a matrix based diverse and geographically distributed project team.
• 10+ years of experience in Data Warehouse related projects and implementing multiple end-to-end DW projects in Big Data environment
• Experience of building data pipelines through Spark with Scala/Python/Java on Hadoop or Object storage
• Experience of building Nifi pipelines
• Experience of working with Databases like Oracle, Netezza would be added advantage
• Strong analytical skills required for debugging production issues, providing root cause and implementing mitigation plan
• Strong communication skills - both verbal and written – and strong relationship, collaboration skills and organizational skills
• Ability to multi-task across multiple projects, interface with external / internal resources and provide technical leadership to junior team members
• Ability to be high-energy, detail-oriented, proactive and able to function under pressure in an independent environment along with a high degree of initiative and self-motivation to drive results
• Ability to quickly learn and implement new technologies, and perform POC to explore best solution for the problem statement
• Flexibility to work as a member of a matrix based diverse and geographically distributed project teams