Tech Lead /Architect - Big Data

Key Skills

Amazon Athena, Amazon RedShift, Ansible, Apache Hadoop, Apache Hive, AWS Glue, Databricks, JavaScript, Lambda, PySpark, Python Programming, Spark-SQL

Job Description

You will have the chance to join a global, multi-cultural team of Data Engineers, and be part of the growth and development of it, participate in a large variety of exciting and challenging projects and work with our exceptional team of Technology professionals.

Technology Stack Used & Required Knowledge:

• Must have experience of developing and implementing pipelines on AWS cloud that extract, transform, and load data into an information product that helps the organization reach its strategic goals
• Must have experience on ETL Data Engineer, Python, Java, Bigdata (Hadoop, PySpark, Spark SQL, Hive)
• Effective time management skills, including demonstrated ability to manage and prioritize multiple tasks of project.
• Experience in creating and driving large-scale ETL pipelines in AWS-based environment
• Experience with integration of data from multiple data sources.
• Overall 8+ years of relevant work experience in Big data engineering, ETL, Data Modeling, and Data Architecture.
• Experience with DevOps and Continuous Integration/Delivery (CI/CD) concepts and tools such as Bitbucket, Bamboo, Ansible, Sonar, and the Atlassian tool suite
• Strong software development and programming skills with a focus on data using Python/ PySpark and/or Scala for data engineering.
• Experience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Lambda, Athena, and Redshift will be a plus
• Understanding of data types/ handling of different data models.
• Experience with the AWS data management tools such as Data lake and Databricks or AWS Snowflake is a plus
• Understanding of descriptive and exploratory statistics, predictive modelling, evaluation metrics, decision trees, machine learning algorithms is a plus
• Good scripting and programming skills.

To keep it short, below are some key responsibilities:

• Design and develop using Big Data technologies.
• Design and develop cutting edge Big Data, and Cloud technologies to build critical, highly complex distributed systems from scratch
• Design, develop and manage complex ETL jobs and pipelines.
• Act as a critical thought leader, consulting and mentoring other teams as your new systems transform the way this entire firm leverages data
• Deploying data models into production environments. This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.
• Mentor and train colleagues where necessary by helping them learn and improve their skills, as well as innovate and iterate on best practices.
• Solve complex issues and provide guidance to the team members when needed.
• Make improvement and process recommendations that have an impact on the business

The Rest of the qualities, you know them:

• Leading and organizing teams of various sizes
• Client Facing Skills
• Flexibility in using different technologies/ platforms
• Ability to shift gears quickly and cope with change
• Analytical & Detail Oriented
• Confident, proactive, and self-motivated
• Disciplined and “Team first approach” oriented