Senior Data Engineer Gen AI

Key Skills

Aartificial intelligence, Amazon RedShift, Amazon Simple Storage Service (S3), API, AWS, Azure, Big Data Technology, Data Lake, Data Migration, Data pipelines, Data science techniques, Data Warehousing, Database, ETL/ELT technologies, Lambda, Large Language Models - LLMs, Machine learning techniques, PySpark, Python Programming, REST API, SQL

Job Description

At the heart of New York Life is a commitment to be there for our customers when they need us—whether today or decades into the future. We have delivered on that promise for over 175 years by investing wisely, growing a portfolio of strategic businesses, and remaining true to our mission as a mutual company, accountable only to our customers, not to outside investors. For our customers, that means having the confidence that comes with knowing they can build a better future for themselves and those they love.

As a mutual, we are fully aligned with our policy owners, not with outside investors or Wall Street. This allows us to focus on delivering lifelong value to our clients, including sharing our success with eligible participating policy owners through non-guaranteed dividends, which we've paid for 166 consecutive years.

People come to work at New York Life to contribute to the financial goals of millions of families and businesses each day. We have a diverse, nationwide workforce that allows us to support the communities where we work. Our purpose-driven mind-set has attracted people who want to do good and has resulted in great success for our company and our customers.

The Foundational Business at New York Life provides life insurance and investment products to our customers through a network of over 12,000 licensed agents. Technology and digital are central to our agent, customer and employee needs and as these evolve our Technology organization is transforming rapidly.

New York Life is embarking on a significant digital evolution which demands robust digital capabilities, exceptional user experience and connected data. To execute on our strategic plans, we have launched Blueprint. At the heart of this program sits our Operational Model consisting of Value Streams (VS) and Centers for Enablement (C4E) powered by our Scaled-Agile delivery execution model.

Role Summary

We are seeking energetic and passionate senior data engineer to design and execute our company’s modern data management strategy. As a key member of Enterprise Data Platform team, you will work with talented engineers, architects, data scientists and end users to reimagine, design and build the next generation of Insurance solutions in the cloud leveraging ancient and modern technologies and platforms including AWS and Azure cloud, APIs, and AI. 

You will be responsible for the design, development and deployment of optimal data pipelines taking advantage of leading-edge technologies through experimentation and iterative refinement. You will assist other team members in building and executing automation that effectively and repeatably ensures quality, security, integrity, and maintainability of our solutions. You will also be responsible for the evaluation of existing data pipeline architecture, evolve and enhance the foundation, participate in product ideation and technology stack evaluation and recommendations.

Key Responsibilities

• Engages in collaborative relationships across architecture, product management, data, and product delivery teams to design, implement solutions based on defined business challenges and outcomes
• Participate on scrum teams working in short sprints and ensure on time delivery with high quality
• Build and maintain data engineering solutions on cloud platforms using AWS or Azure services
• Design, develop, and implement scalable data pipelines and ETL/ELT processes using Python, PySpark and API integrations
• Implement data quality and validation checks to ensure accuracy and consistency of data
• Collaborate with data scientists and analysts to understand data requirements and translate them into scalable, high performant data pipeline solutions.
• Develop and maintain data models and schemas to support data integration and analysis
• Monitor and troubleshoot data pipeline performance, identifying and resolving bottlenecks and issues
• Stay updated with the latest technologies and trends in big data, data engineering, data science, and REST API development, and provide recommendations for process improvements
• Use AWS Cloud technologies to support data needs for expansion of Machine Learning/Data Science capabilities, applications/mobile apps/systems, BI/analytics, and cross-functional teams
• Create methods and routines to transition data from on-premise systems to the AWS Cloud
• Store data in Cloud platform and develop transformational logic based on business rules
• Build the infrastructure required for optimal ETL/ELT (extraction, transformation, and loading) of data from a wide variety of data sources using SQL and AWS big data technologies
• Develop and debug code in PySpark/Spark/Python. Strong SQL and query optimization techniques
• Maintaining high degree of knowledge and expertise in cloud-based technologies contributing to go forward best practices, standards, and patterns
• Active participant in relevant communities of practice to exchange ideas and knowledge

Required Skills

• Experience with database technologies for structured and unstructured data both for storage and optimal retrieval
• Experience in data management of Content and Content data stores
• Experience with AWS Cloud tools like Amazon Relational Database Service (AWS RDS), Redshift, AWS Database Migration Service (AWS DMS), AWS Glue, Amazon Simple Storage Service (AWS S3), AWS Lambda triggers & functions, etc.

• Strong, demonstratable hands-on experience with data engineering and data pipelines
• Proficiency with languages such as Python, PySpark, Java etc.
• Experience with developing Cloud-native applications and very good understanding of Cloud architecture and best practices
• Experience with relevant code repository and project tools such as GitHub, JIRA, Confluence
• Working experience with Continuous Integration, Continuous Deployment CI/CD
• Experience with highly scalable Data stores, Data Lake, Data Warehouse, Lakehouse, and unstructured datasets
• Experience with Data integration, data processing, data streaming, message queuing techniques
• Detail oriented individual with strong analytical, problem-solving, and organizational skills
• Ability to clearly communicate to both technical and business teams 

Desired Skills:

• Solid understanding of the insurance, wealth management or financial services industry
• Knowledge of scripting tools on multiple platforms preferred
• Extensive hands-on development experience with Cloud (AWS, Azure) and Storage services are desirable
• Experience in developing solutions leveraging LLMs a plus
• Knowledge of machine learning pipelines and frameworks like MLFlow a plus
• Strong hands-on experience and extensive development experience with API integrations a plus

Education / Training

• Bachelor’s degree in Computer Science or an Engineering discipline
• Certification in data management (structure and unstructured), and/or cloud architecture or engineering a plus
• Certification in delivery methodologies such as Scaled Agile a plus

Previous Experience

• 8+ years of experience in data engineering, machine learning, and cloud tech developing production grade software components and solutions

Additional Attributes

• Trusted and respected team member putting objectives of the team over individuals.
• Comfortable, experienced, and accomplished at working with business stakeholders, and able to debate and challenge in a professional and diplomatic way.
• Highly Collaborative and supportive of business and of its ideals and strategies.
• Highly innovative with aptitude for foresight, systems thinking and design thinking.
• More interested in achieving targeted business outcomes and what is best for the organization than in personal preferences, or in vested personal preferences of other business and IT leaders.
• Composed in the face of opposition to technology principles, governance, and standards.
• Practical in approach to problem solving and decision making, with a bias towards simplifying processes.