Data Engineer

Key Skills

SQL, Python Programming, R Programming, AWS, Tableau, Scala Programming

Job Description

The Disney Decision Science and Integration (DDSI) analytics consulting team is responsible for supporting clients across The Walt Disney Company including Direct-to-Consumer & International, Media Networks (e.g., ABC, ESPN), Studio Entertainment (e.g., The Walt Disney Studios, Disney Theatrical Group) and Parks, Experiences & Consumer Products. DDSI leverages technology, data analytics, optimization, statistical and econometric modeling to explore opportunities, shape business decisions and drive business value.
The Data Engineering team (within DDSI) is looking to fill a Data Integration Manager role responsible for designing and implementing ETL/ELT data pipelines, designing and implementing database schema/tables/views, data quality validation, and participating in the architectural design and implementation of our next generation data platform to fulfill the needs of our applications, data services, ad-hoc analytics and self-service/POC initiatives.

Responsibilities:

As an individual contributor in the DDSI team you will be responsible for the following:

• Partner with our MSI, Decision Science and Technology team members in various activities around data requirements gathering, data validation scripting and review, developing and monitoring ETL/ELT data pipelines
• Designing and implementing database schema/tables/views
• Implementing data services API’s
• Evolving our data analytics platform
• Participate in architectural evolution of data engineering patterns, frameworks, systems, and platforms including defining best practices, standards, principles, and policies
• Leverage a multitude of technologies to fulfill the work including, but not limited to SQL, Python, Spark, PySpark, SparkSQL, Hadoop/Hive, Docker, Gitlab, Airflow, Kafka, Lambda, Snowflake, and PostgreSQL.
• Participate in driving best practices around data engineering software development processes

Basic Qualifications:

• 6+ years’ experience in a Technical/Developer role
• 5+ years’ experience with Python or Java
• 5+ years’ experience with SQL
• 5+ years’ experience designing, building and maintaining ETL/ELT data pipelines
• Knowledge utilizing PySpark and/or Scala
• Experience utilizing Hadoop, Hive, Spark, and Presto
• 2+ years’ experience with cloud based technologies, preferably AWS EMR, EC2, and S3
• Strong understanding of relational and non-relational database design
• Experience leveraging containerization technologies such as Docker, Nomad or Kubernetes
• Understanding differences between: data warehouses and data lakes, schema-on-read and schema-on-write, relational and non-relational databases, batch and stream processing
• Experience managing and deploying code using Gitlab
• Hands-on knowledge of Airflow, Kafka, and Glue
• Prior experience with NoSQL systems such as MongoDB, DynamoDB, Neo4J or Redis
• Proficiency with relational database technologies such as PostgreSQL, MariaDB, or Teradata
• Experience working with large datasets and big data technologies, preferably cloud-based, such as Snowflake, Redshift, Databricks, or similar
• Experience in stream data processing and real time analytics of data generated from user interaction with applications is a plus.
• SQL, Python, Spark, Docker, Gitlab, Airflow, AWS S3 and Relational Databases (PostgreSQL, MariaDB, Snowflake, Teradata)
• 2+ years’ experience working on a cloud platform
• 2+ years’ experience working with data lakes, data warehouses and application databases

Preferred Qualifications:

• 10+ years’ experience in a Technical/Developer role
• Experience in Java, PySpark, SparkSQL, Hadoop/Hive, Glue, Kafka, Lambda, BigQuery, Databricks

Required Education

• Bachelor degree in Computer Science, Mathematics, Engineering, or a related field

Preferred Education

• Master’s degree in Computer Science, Mathematics, Engineering, or a related field