Lead Data Engineer

Key Skills

Java Programming, Design, Python Programming, MongoDB, SQL, Azure, AWS, PowerBI, REST API, Big Data Technology, Apache Hadoop, Scala Programming, NoSQL, QlikView, Tableau, Presto, Apache Hive, MapReduce, TextBlob, Apache Impala, Data Analytics, PostgreSQL, Sqoop, SPARK Programming, Databricks, AWS Kinesis, Data Lake, ETL frameworks

Job Description

If you desire to be part of something special, to be part of a winning team, to be part of a fun team – winning is fun. We are looking forward to a Lead Data Engineer in EATON India Innovation Centre, based in Pune, India. In Eaton, making our work exciting, engaging, meaningful; ensuring safety, health, wellness; and being a model of inclusion & diversity are already embedded in who we are - it’s in our values, part of our vision, and our clearly defined aspirational goals. Your essential responsibilities include:

• The candidate will demonstrate exceptional impact in delivering projects in terms of architecture, technical deliverables and project delivery throughout the project lifecycle. The candidate is expected to be conversant with Agile methodologies and tools and have a track record of delivering products in a production environment
• Work with a team of experts in deep learning, machine learning, distributed systems, program management, and product teams, and work on all aspects of design, development and delivery of deep learning enabled end-to-end pipelines and solutions.
• Design and develop ETL and machine learning pipelines on premise or public cloud infrastructures
• Lead the development of technical solutions and implement architectures for project and products across data engineering and data science teams
• Lead the architecture, design, and development of new intelligent power technology products and production quality end to end systems
• Experience on ML/AI Software development environment
• Work with your team and others, defining the architecture, design, and management of secure, large-scale, highly-distributed, geo-redundant, multi-tenant data stores
• Knowledge with microservices, cloud APIS (e.g. AWS and MS Azure)
• Is accountable for end-to end delivery of solutions from requirements gathering to production
• Author high-quality, highly-performance, unit-tested code to extract and transform data based on business and data science needs

Qualifications

• PhD, Masters or Bachelor’s degree in computer science or software engineering
• 8+ years of progressive experience in delivering technology solutions in a production environment
• 6+ years of experience in the software industry as a developer, with a proven track record of shipping high quality products
• 6+ years working with customers (internal and external) on developing requirements and working as a solutions architect to deliver
• PhD, Masters or Bachelor’s degree in Computer Science or Software Engineering or Information Technoogy
• Experience on Cloud Development Platforms - Azure & AWS, releated services on data storage, machine learning pipeline services, etc.
• Cloud based Analytics (AWI, REST API, Microservices)
• Knowledge of IoT technologies, including cloud processing, like Azure IoT Hub.
• Experience on CI/CD (Continuous Integration/Delivery) i.e. Jenkins, GIT, Travis-CI
• Virtual build environments (Container, VMs and Microservices) and Container orchestration - Docker Swarm, Kubernetes/Red Hat Openshift.
• Relational & non-relational database systems - SQL, Postgres SQL, NoSQL, MongoDB, CosmosDB, DocumentDB
• Data Warehousing & ETL - Write complex queries that are accessible, secure and perform in optimized manner that outputs to different consumers and systems
• ETL on Big Data Technologies - HDFS, Hive, Impala, SQOOP
• Progamming Knowledge - Java and/or Python and associated IDE's (Eclipse, IntelliJ, PyCharm etc.)
• APIs development to support data consumption needs of internal and external stakeholders
• Data visualization tools such as Power BI, QlikView and/or Tableau
• Data pipelining, scripting, reporting
• In-depth knowledge of SOA (Service Oriented Architecture)
• Experience in Azure Tools - Blob, SQL, Data Lake, Hive, Hadoop, Data Factory , Databricks, Azure Functions
• SW Development life-cycle process & tools
• Agile development methodologies and concepts including handson with Jira, bitbucket and confluence.
• Knowledge of streaming technologies like Apache Kafka, AWS Kinesis, Azure EventHubs
• Knowledge of Cloudera Hadoop, ML ops
• Knowledge of data analysis tools, like Apache Presto, Hive, Azure Data Lake Analytics, AWS Athena, Zeppelin
• Ability to specify and write code that is accessible, secure, and performs in an optimized manner with an ability to output to different types of consumers and systems
• Experience in Design Thinking or human-centered methods to identify and creatively solve customer needs, through a holistic understanding of customer’s problem area
• Knowledgeable in leveraging multiple data transit protocols and technologies (MQTT, Rest API, JDBC, etc)
• Knowledge of Hadoop and MapReduce/Spark or related frameworks
• Knowledge of Scala