PySpark Data Engineer

Key Skills

Apache Hadoop, C Programming, C++, Data science techniques, Java Programming, Machine learning techniques, MapReduce, Python Programming, PyTorch, R Programming, Scala Programming, SQL, TensorFlow

Job Description

Mactores is a team of emerging technologists, data engineers, and data scientists driven by excellence in technology to solve critical business problems. We help bridging complex business challenges with technical expertise. Specializing in AI, Fast Data, Industrial IoT and Cloud helps us collaborate with organizations that view technology as a strategic driver to success.

As a PySpark Data Engineer, you are a full-stack data engineer that loves solving business problems. You work with business leads, analysts, and data scientists to understand the business domain and engage with fellow engineers to build data products that empower better decision-making. You are passionate about the data quality of our business metrics and the flexibility of your solution that scales to respond to broader business questions.

What will you do

• Write efficient code in the chosen technology for the project - For example - Spark, Apache Beam
• Provide top-quality solution design and execution
• Provide support in defining the scope and sizing of work
• Provide alternative design solutions and project estimates
• Work with domain experts to put together a delivery plan and stay on track
• Stay current on the latest technology to ensure maximum ROI for clients
• Collaborate with many teams - engineering and business, to build better data products and services
• Deliver the projects along with the team collaboratively and manage updates to customers on time

What we are looking for

• 4 to 7 years of experience in Apache Spark.
• 2+ years of experience in writing ETL jobs using PySpark, and SparkSQL
• 2+ years of experience in SQL queries and stored procedures
• Have a deep understanding of all the Dataframe API with all the transformation functions supported by Spark 2.7+
• You will be preferred if
• Prior experience in working on AWS EMR, Apache Airflow
• Certifications AWS Certified Big Data – Specialty OR Cloudera Certified Big Data Engineer OR Hortonworks Certified Big Data Engineer
• Understanding of DataOps Engineering