Senior Software Development Engineer - Data

Key Skills

Amazon EMR, Amazon Simple Storage Service (S3), Apache Airflow, Apache Hadoop, Apache Hive, Apache Kafka, Big Data Technology, Data Lake, Databricks, Effective communication skills, GoLang, Java Programming, Machine learning techniques, Parquet, Python Programming, REDIS, Scala Programming, Software Development, SQL

Job Description

As a Senior Software Engineer in the Data Platform Team, you will play a pivotal role in designing, building, and maintaining scalable and high-performance Data systems to support Hotstar’s data-driven initiatives. Collaborating with cross-functional teams, you will contribute to the architecture, development, and deployment of data solutions, enabling us to provide personalized, seamless, and delightful experiences to millions of users worldwide.

About the team

Business Team: Data Platform

The Data Platform at Hotstar team runs a gamut of platform services and tools which enable the organization to ingest and analyze data at petabyte scale. All engineering and product teams derive important insights on user behavior from this data, to iterate and deliver a better experience to Hotstar users every day. Few of the services owned by Data Platform team include a multi million RPS scale Data ingestion service, multiple high throughput Apache Kafka clusters, a suite of Kafka connects to connect various data sources and sinks, a query platform on top of our Data Lake with multiple query engines like Apache Hive, Presto, Apache Spark, Apache Flink, tools for real time platform monitoring, and Apache Airflow for orchestration.

Reporting to: Engineering Manager, Data Platform

Key Responsibilities

• You will combine your knowledge of Data Platform technologies with your product sense to solve high scale, high impact and Data relevant use-cases for other product and engineering teams at Hotstar.
• You will work closely with Software Engineers & ML engineers to build Data platform abstractions that fuel the needs of multiple teams, systems and products.
• You will optimize data-delivery and build the infrastructure required for optimal extraction, transformation and loading of data required for a wide variety of use-cases using SQL/Spark.
• You will build stream processing pipelines and tools to support a vast variety of analytics and audit use-cases.
• You will continuously evaluate new and relevant technologies and initiate adoption of new best practices.
• You will influence and drive architecture and design discussions in the Data Platform team.
• You will work in a cross-functional team and collaborate with peers during the entire software development life cycle with a strong focus on continuous deliveryYou will act as a technical mentor and guide for junior team members, fostering a collaborative and growth-oriented work environment.
• Ability to support hyper growth systems across multiple geographies and regions

Skills & Attributes for success

• In-depth understanding and experience in working with high scale Data lakehouse systems.
• Hands-on experience in Big Data technologies like Apache Spark, SQL, Hadoop, Hive, Airflow, Kafka and Kafka connects.
• Hands-on experience with AWS stack: S3, EMR, RDS etc.
• Strong programming skills in 1 or more of the following languages: Java, Scala, Go, PythonStrong understanding of Database and Data storage fundamentalsStrong understanding of Data storage formats like Parquet, Hudi, Delta etc.
• Excellent teamwork and communication skills, with the ability to work effectively in a fast-paced, agile environment.
• Good to have experience in real-time streaming data pipelines using Spark Streaming or Flink.
• Familiarity with Data lakehouse solutions like Databricks.

Preferred Education & Experience

• B.Tech/B.E/Masters/PhD in Computer Science from a reputed college or related technical discipline
• 4-7 years of experience in software development, preferably in Data related teams

About Us

Disney+ Hotstar is India’s leading streaming platform that has changed the way Indians watch their entertainment - from their favorite TV shows and movies to sporting extravaganzas. With the widest range of content in India, Disney+ hotstar offers more than 100,000 hours of TV Shows and Movies in 18 languages and coverage of major global sporting events.