Data Engineer

Key Skills

Java Programming, Python Programming, SQL, AWS, Apache Hadoop, Scala Programming, Data Modelling, Apache Pig, Amazon RedShift, Amazon EMR, SPARK Programming, HBase, Lambda, ETL frameworks, Data Architecture

Job Description

Job summary

Kumo (pronounced ‘Koo-mo’, the Japanese word for cloud) is the global product and engineering organization for AWS Support, a multi-billion $ business. Kumo’s products, used by AWS customers and the internal Amazonian teams that help them, are vital to ensuring exceptional customer experiences on AWS. Kumo has a broad portfolio of customer-facing and internal services. Our public-facing properties include AWS Trusted Advisor, AWS Health, Automation, Recommendations & Insights, AWS Support Center, AWS Knowledge center, and many more. Our internal AWS Support teams rely on Kumo services to interact with AWS customers, diagnose and resolve customer issues, and safely share information across AWS teams. These services must work seamlessly, and be available when AWS services are not. Any changes to these tools has a force multiplicative effect on how customers experience AWS and the efficiencies of the internal workforce.

We are looking for an exceptional Data Engineer who is passionate about data and the insights that large amounts of data sets can provide. The ideal candidate will possess both a data engineering background and a strong business acumen that enables him/her to think strategically. He/she will experience a wide range of problem solving situations requiring extensive use of data collection and analysis. The successful candidate will work in lock-step with BI Engineers, Data scientists, ML scientists, Business analysts, Product Managers and other stakeholders across organization. He/she will:

• Develop and improve the current data architecture, data quality, monitoring and data availability.
• Collaborate with Data Scientists to implement advanced analytics algorithms that exploit our rich data sets for statistical analysis, prediction, clustering and machine learning
• Partner with BAs across teams to build and verify hypothesis to improve the AWS Support business.
• Help continually improve ongoing reporting and analysis processes, simplifying self-service support for customers
• Keep up to date with advances in big data technologies and run pilots to design the data architecture to scale with the increased data sets of customer experience on AWS.

BASIC QUALIFICATIONS

• Bachelor’s degree in Computer Science or related technical field, or equivalent work experience.
• 4+ years of work experience with ETL, Data Modeling, optimizing ETL, and Data Architecture.
• 2+ years of scripting experience in Python, Scala or other languages.
• 2+ experience with AWS services including S3, Redshift, EMR, Kinesis and RDS.
• 2+ years of work experience with Big Data Technologies (Hadoop, Hive, Hbase, Pig, Spark, etc.)
• Knowledge of distributed systems as it pertains to data storage and computing

PREFERRED QUALIFICATIONS

• Experience with ETL optimization, designing, coding, and tuning big data processes using Apache Spark or similar technologies.
• Experience with building data pipelines and applications to stream and process datasets at low latencies.
• Experience handling data - tracking data lineage, ensuring data quality, and improving discoverability of data.
• Knowledge of distributed systems and data architecture (lambda)- design and implement batch and stream data processing pipelines, knows how to optimize the distribution, partitioning, and MPP of high-level data structures.
• Experience with native AWS technologies for data and analytics such as Redshift Spectrum, Athena, S3, Lambda, Glue, EMR, Kinesis, SNS, CloudWatch, etc.