Senior Big Data Engineer

Key Skills

Java Programming, Python Programming, MongoDB, Database, SQL, Agile methodologies, DevOps, REST API, Big Data Technology, Apache Hadoop, Apache Hive, Docker, PostgreSQL, SPARK Programming, HBase, Effective communication skills

Job Description

Technology Services Group (TSG) powers BNY Mellon by providing scale, capacity, and capability to deliver the critical technology required to operate the global financial services markets.

TSG is defined by three core values:

• Creative innovators, delivering excellence and simplicity
• Always doing the right thing for the bank, colleagues, and clients
• Fostering a collaborative, diverse and inclusive environment, where we challenge each other to execute as one TSG team.

As part of TSG, our Architecture & Engineering team is

• Define enterprise technology architecture and engineering principles and best practices
• Develop next-generation engineering solutions focused on Resiliency and Innovation
• Engineer and integrate the critical systems and platforms used across the band and with external Clients

The Role

We’re looking for a Senior Big Data Engineer with proven Cloudera Hadoop product expertise and engineering skillset. TSG Database Engineering is building the next generation of big data platforms such as Cloudera CDP, MongoDB Ops Manager and PostgreSQL v13. Building on top of our DBaaS automation to efficiently deploy big data services, we are aggressively strengthening our database services and safeguard our data layer resiliencies.

Responsibilities:

• Work as a senior system engineer for the Hadoop enterprise data platform platforms
• As an Engineer you will design and create highly complex big data/Hadoop platforms. You will have expert knowledge in distributed datalake systems.
• You’ll configure and tune service components to optimize platform performance. You will design and implement backup and restore strategies and disaster recovery solutions. You will support and troubleshoot performance related issues.
• You’ll consult with storage and host engineering services to create optimized big data ecosystems.
• Assist in cross-team collaboration and design discussions
• Provide sandbox functionality to users of the datasets to assist application analytics
• Provide storage infrastructure system management including capacity planning, performance monitoring and tuning, security management etc.
• Responsible for infrastructure Support and projects including system design, planning, installations, configurations, upgrades and planning.
• Manage Tier-3 support following ITIL practice and incident management
• Manage system configurations and solutions, and monitor the full lifecycles of storage area network (SAN) administration, architecture, and support.
• Data storage tasks such as management, provisioning, troubleshooting and debugging
• Lead technical assessments of hardware, software, tools, applications, firmware, and operating systems to support business operations.
• You’ll develop trusted corporations with application tenants. You’ll advise applications with sounds data layer resiliencies.
• You’ll provide input into the selection of tools and any necessary migration into the company's environment.

Qualifications

• Bachelor's degree in computer science or a related discipline, or equivalent work experience required.
• 5-10 years of experience in Hadoop technologies, data lake design, experience in the securities or financial services industry is a plus.
• Excellent knowledge with Hadoop components for big data platforms related to data ingestion, storage, transformations and analytics
• Excellent DevOps skillsets and SDLC practices
• Excellent knowledge of automation tools such as Ansible
• Must have Python programming skills
• Excellent knowledge of Docker containers
• Collaborate with other developers and architects to enhance the capabilities of the platforms
• Assist in cross-team collaboration and design discussions
• Provide sandbox functionality to users of the datasets to assist application analytics
• Knowledges of Kerberos, REST APIs are a plus

Preferred Skills:

• 8+ years of experience on BigData Platform on Hadoop, Spark, Hive, HBase
• 3+ years’ experience on Python and DevOps tools such as Ansible
• Strong Agile/Scrum development experience
• Strong collaboration and communication skills within distributed project teams
• Excellent written and verbal communication skills
• Passion to stay on top of the latest happenings in the tech world and an attitude to discuss and bring those into play.
• Working knowledge with PostgreSQL or Mongo DB databases a plus .