Lead Data Engineer

Key Skills

Apache Hadoop, API, Kubernetes-K8s, NoSQL, Presto, SQL, Unix Operating system

Job Description

Lead Data Engineer, Open Source - Data Platform

• Palo Alto, CA, USA
• Full-time

Visa is seeking a Hadoop Open Source Lead Engineer in the Data Platform department to act as one of key technologist to build and manage Visa’s Open Source Hadoop in the Data Platform.

This individual will act as a trusted technical expert and strategic contributor to the Data Platform department. The successful candidate should be comfortable navigating challenging projects in big data and data platform areas and be comfortable in a fast-paced environment with significant changes; developing/implementing strategies and processes that enable the organization to innovate and grow fast. You will have the opportunity to lead, participate, guide, and mentor other people in the team on architecture and design in a hands-on manner. This position reports to the head of the Large-Scale Shared Platform team in Palo Alto. The ideal candidate will have very strong development skill while exposure Open Source Hadoop and big data / large-scale software platforms like Hadoop, Presto, Spark and Kubernetes are required.

Primary responsibilities will include:

• Provide development lead oversight on Visa’s Hadoop Open Source Platform. Develop and help drive Open Source Hadoop based products and services.
• Contribute and holistically offers the Open source Hadoop services that solves current and potential Use Case(s) for Visa Internal Business teams.
• Contribute to the DP strategies and Open Source roadmap development to meet business objectives with existing or emerging technologies. Ability to Understands and Solves Common and Un-common User Problems by Mapping and Onboarding New Use Cases to the Open source Big data services.
• Innovate and Re-pivot Problems with Models and Higher Dimension Concepts to Support Current and Potential Visa Business Use Cases.
• Work extensively on providing Open Source Spark as a Service on Kubernetes and Presto to enable users use a data processing service connecting to Hadoop.
• Apply creative thinking/approach to determine technical solutions that further business goals and align with corporate technology strategies, keeping in mind performance, reliability, scalability, usability, security, flexibility, and cost.
• Lead internal proof of concept initiatives with Architects and work on multiple enhancement on existing services.
• Must love coding – prepare to spend 80% of the time on hands-on development with development teams
• Provide mentorship and help team growth especially on technical side.

Qualifications

REQUIRED QUALIFICATIONS

• 10 years of work experience with a Bachelor’s Degree or at least 8 years of work experience with an Advanced Degree (e.g. Masters/MBA/JD/MD) or at least 3 years of work experience with a PhD
• Minimum 5 years of Open Source Hadoop and Big data products experience.

PREFERRED QUALIFICATIONS

• Understand all aspects of our distributed systems and learn select set of services in detail. Implement solutions to solve massively distributed technology problems using open source products.
• Work closely with service engineering, operations, and Hadoop users to engineer applications for Visa Business projects.
• Active development experience to provide software as service on-prem and in Hybrid-Cloud. 
• Experience in Kubernetes based services development and have strong full stack development background using UI and API frameworks
• Understanding best practices for Big Data (in Hadoop), data warehousing, consumer analytics, knowledge management and key understanding of streaming and other NoSQL databases
• Hands-on experience in developing and managing development of Data Integration applications for large corporations with experience in both batch and online systems.
• Experience in using Apache Open source projects, contributed to open source projects is a plus.
• Experience in logging, metering and alerting open source-based solution is a plus. 
• Can navigate a Linux terminal with ease
• Relational database and SQL development experience required
• In-depth knowledge of the software development life cycle required
• Outstanding verbal, written, presentation, facilitation, and interaction skills, including ability to effectively communicate architectural issues and concepts to multiple organization levels and executive management.
• Quick learner; self-starter, detailed and thorough
• Strong customer-centric mindset
• Strong leadership and team player.
• Strong skills on mentoring/growing junior people
• Payment industry experience is a plus.

Additional Information

Travel Requirements This position requires the incumbent to travel for work 5% of the time.

Mental/Physical Requirements This position will be performed in an office setting. The position will require the incumbent to sit and stand at a desk, communicate in person and by telephone, frequently operate standard office equipment, such as telephones and computers, and reach with hands and arms.