Senior Big Data Engineer (AWS+Python+Hadoop)

Key Skills

Amazon Athena, Amazon Elastic Compute Cloud-EC2, Amazon RedShift, Amazon Simple Storage Service (S3), AWS, AWS Glue, Data Architecture, Data Lake, Data Modeling, Databricks, ETL frameworks, Lambda, PySpark, Python Programming, Scala Programming, Snowflake

Job Description

NorthBay Solutions is looking for a highly skilled and motivated Senior Big Data Engineer (AWS+Python+Hadoop) with solid experience in data engineering and a passion for Big Data. The ideal candidate will have experience of over 3+ years to 6 years specifically in Data Engineering.

You will have the chance to join a global, multi-cultural team of Data Engineers, and be part of the growth and development of it, participate in a large variety of exciting and challenging projects and work with our exceptional team of Technology professionals.

Technology Stack Used & Required Knowledge:

• Over 3+ years of experience on ETL Data Engineer, Python, Java, Bigdata (Hadoop, spark, hive)
• Excellent understanding of ETL cycle.
• Experience in creating and driving large-scale ETL pipelines in AWS-based environment
• Experience with integration of data from multiple data sources.
• Good knowledge and understanding of Big data engineering, ETL, Data Modeling, and Data Architecture.
• Strong software development and programming skills with a focus on data using Python/ PySpark and/or Scala for data engineering.
• Experience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Lambda, Athena, and Redshift will be a plus
• Understanding of data types/ handling of different data models.
• Experience with the AWS data management tools such as Data lake and Databricks or AWS Snowflake is a plus
• Understanding of descriptive and exploratory statistics, predictive modelling, evaluation metrics, decision trees, machine learning algorithms is a plus
• Good scripting and programming skills.
• Should have experience in Jira and Agile development methodologies.
• Must be open to learn and the ability to grasp new technologies quickly.
• Be motivated and passionate about technology.
• Team player with strong integrity and professionalism who can work well with others in the group.
• Self-starter with a proven ability to take initiative and strong problem-solving skills.

To keep it short, below are some key responsibilities:

• Design and develop using Big Data technologies.
• Design and develop cutting edge Big Data, and Cloud technologies to build critical, highly complex distributed systems from scratch
• Design, develop and manage complex ETL jobs and pipelines.
• Act as a team member, consulting other teammates as your new systems transform the way this entire firm leverages data
• Deploying data models into production environments. This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.
• Support and helping them learn and improve their skills, as well as innovate and iterate on best practices.
• Solve complex issues and provide guidance to the team members when needed.
• Make improvement and process recommendations that have an impact on the business

About NorthBay

NorthBay, an 8+ Year Award Winning AWS Premier Consulting Partner, is an Global Analytics, DB/App Migration, App Dev and DevOps professional services company. NorthBay's intense focus is unparalleled in the areas of Big Data (Data Lakes, Data Platforms, Enterprise Data Warehouse), Database & Application Migrations and Machine Learning/AI. NorthBay has earned the Analytics, DevOps, Mobile; Education Competency and is also a Public Sector Partner and we are also certified for AWS Well Architected Reviews and Database Freedom Migrations.

What We Offer

• Consulting environment with a hyper-growth company
• Multi-cultural environment
• An opportunity to solve complex business challenges and problems for large corporations
• International Exposure and Experience.
• Competitive salary and benefits
• Paid holidays and vacations
• Medical OPD & Health Insurance
• Fuel Allowance
• Employee Life Insurance
• GYM/Health Club Fee Reimbursement
• Mobile & Internet Allowance
• Opportunities to make a difference in a highly productive environment