Senior Data Engineer

Key Skills

Ansible, AWS, Azure, Big Data Technology, Continuous Integration & Continuous Delivery - CI/CD, Data Engineering, ETL/ELT technologies, Google Cloud Platform (GCP), Jenkins, Kanban, NoSQL, Python Programming, Scrum Agile Methodology

Job Description

Designs and develops highly scalable and distributed data platforms using open source frameworks to process high volume, high velocity, structured and unstructured data. Designs and architects cloud native application using AWS, Azure, and Google Cloud. Performs distribution application design and development using Java, Scala, and Python. Develops Extract Transform and Load/Extract Load and Transform (ETL/ELT) strategies. Develops Continuous Integration/Continuous Delivery (CI/CD) application pipelines using Maven, Jenkins, Stash, Ansible, and Docker. Deploys Machine Learning models with reinforcement learning in highly scalable environment. Delivers software solutions using Agile methodologies, Kanban, and SCRUM. Crafts, builds and supports mission critical applications with data architecture patterns – – Lambda and Kappa.

Primary Responsibilities:

• Explores new technology trends and uses them to simplify the data ecosystem.

• Drives innovation and implements solutions.

• Collaborates with internal and external teams to deliver technology solutions for business needs.

• Resolves technical roadblocks to the team and mitigates potential risks.

• Delivers system automation by setting up CI/CD pipelines.

• Analyzes information to determine, recommend, and plan computer software specifications on major projects and proposes modifications and improvements based on user need.

• Develops software system testing and validation procedures, programming, and documentation.

Education and Experience:

Bachelor’s degree (or foreign education equivalent) in Computer Science, Engineering, Information Technology, Information Systems, Mathematics, Physics, or a closely related field and three (3) years of experience in the job offered or three (3) years of experience implementing back-end and middle-tier applications.

Or, alternatively, Master’s degree (or foreign education equivalent) in Computer Science, Engineering, Information Technology, Information Systems, Mathematics, Physics, or a closely related field and one (1) year of experience in the job offered or one (1) year of experience implementing back-end and middle-tier applications.

.

Skills and Knowledge:

Candidate must also possess:

• Demonstrated Expertise (“DE”) designing and developing event-driven, micro-service-based applications, using stream-based technologies (Spring Boot, Kafka, Kafka Connect, Snowflake, Kafka Streams, and Flume) and programming languages (Java and Scala); and creating, managing, modeling, and performance tuning SQL queries and AWS Redshift cluster.

• DE designing and developing Continuous Integration/Continuous Delivery (CI/CD) pipelines in Jenkins, using Git, Python, Shell scripting, Docker, Kubernetes, and Maven technologies; and deploying Cloud compliant applications to AWS services — EC2, RDS, EKS, Kinesis, Ehcache, and IAM.

• DE performing database design and development, using relational (PostGreSQL, Oracle) and NoSQL (Cassandra and DynamoDB) databases; developing Java middleware applications using data integration tools — Amazon Web Services (AWS) Services, Apache Kafka, ActiveMQ, and RESTful Web services; and batch processing using big data technologies — AWS BigData Services, Spark, and Flink.

• DE participating in Agile/SCRUM activities — daily stand-ups, retrospectives, iteration planning, estimate/reporting, acceptance criteria definition, and logging and tracking defects/user stories (using JIRA) and collaborating with cross-functional, globally distributed teams to deliver software solutions, using Agile, Waterfall, or Kanban methodologies.