Data Engineer-Big Data – Sydney – 16 months FTH

IBM
Apply Now

Job Description

Your Role and Responsibilities

We are currently recruiting a Lead Data Engineer – Data Harbour SME; Data & Technology Transformation – Data Platforms for an initial period of 16 Months, based in Sydney, Australia.

The Data Engineers in the Nexus-Ops team are responsible for supporting the Data Harbour program, through working with stakeholders including Architects, technical teams, Business users, and data producers, to understand the target state requirements including upstream and downstream use cases. As a senior member of the Data Engineer team, the Lead Data Engineer will be accountable for end-to-end architecture, design, and development process, including providing technical and architecture advisory support to the implementation team and the wider Macquarie stakeholders. As the Lead Data Engineer your responsibilities include –

  • Architect, Design and Solutioning cloud based big data engineering and analytics projects.
  • Architect, Design data pipeline for batch, Streaming, Realtime data using AWS and Spark.
  • Creating architecture, design, technical specifications artefacts.
  • Participate in depth architectural discussions to build confidence and ensure customer success when building new solutions and migrating existing data applications on the AWS platform.
  • Apply in depth experience in AWS cloud services and hands on implementation experience using data-based services on AWS.
  • Work with a variety of cloud-based storage services: AWS S3, Elastic Storage, RDS, AWS Data pipeline, AWS Glue and Data analytics services.
  • Abide by team coding standards, development processes and best practices.

Required Technical and Professional Expertise

To ensure success in the role you will possess the following skills –

  • Must have been through several full-lifecycle Big Data engineering implementations and involved in availability, scalability and performance related design aspects in Data processing and Reporting.
  • Strong hands-on experience working in AWS Serverless (Lambda), Coding against API and SQL.
  • Must be hands-on with writing Python or Scala Scripts development.
  • Hands on experience designing of orchestration using Airflow.
  • Well versed with Spark development (SQL and/or Spark).
  • Hands on experience in AWS Service EC2, S3, SNS, SQS, Lambda, Hue, Presto, Spark, Glue, Kinesis, EMR, ECS, EKS.
  • Sound knowledge in Hive, Redshift, AWS RDS, My SQL and in any NoSQL database e.g., Cassandra, MongoDB.
  • Experience in architecting and designing batch or real time data streaming.
  • Ability to dynamically adapt to conventional big data frameworks and opensource tools if project demands.
  • Knowledge of design strategies for developing scalable, resilient, always-on data lake.
  • Strong development and automation skills.
  • Good understanding in data architecture principles, including data access patterns, data modelling and strong in Data base concepts.
  • Previous experience in Managed Services related reporting in a financial services organization is an advantage.

Company Info.

IBM

IBM is a leading cloud platform and cognitive solutions company. Restlessly reinventing since 1911, we are the largest technology and consulting employer in the world, with more than 290,000 employees serving clients in 177 countries. IBM Research provides unparalleled insight into business, industry and society by leveraging advanced computing architectures and methodologies to solve some of the world’s most pressing challenges.

  • Industry
    Information Technology,Computer software,Computer hardware
  • No. of Employees
    292,500
  • Location
    New Orchard Road, Armonk, New York, NY 10504, USA
  • Website
  • Jobs Posted

Get Similar Jobs In Your Inbox

IBM is currently hiring Big Data Engineer Jobs in Sydney NSW, Australia with average base salary of $30 - $60 / Hour.

Similar Jobs View More