Manager - Data Engineering - USA

Key Skills

Java Programming, Python Programming, C++, C Programming, SQL, Cloud computing, Scala Programming, Machine learning techniques, Data science techniques, MATLAB Programming, PyTorch, TensorFlow, R Programming

Job Description

Responsibilities:

• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities.
• Develop and maintain data pipelines implementing ETL process, monitoring performance and advising any necessary infrastructure changes.
• Translate complex technical and functional requirements into detailed designs.
• Investigate and analyze alternative solutions to data storing, processing etc. to ensure most streamlined approaches are implemented.
• Serve as a mentor to junior staff by conducting technical training sessions and reviewing project outputs.

Roles & Responsibilities

The organization is looking for a wide range of skills, including:

• Strong understanding of data warehousing and data modeling techniques.
• Proficient understanding of distributed computing principles - Hadoop v2, MapReduce, HDFS
• Experience with building stream-processing systems, using solutions such as Storm or Spark Streaming
• Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
• Experience with Spark, SQL, and Linux.
• Knowledge of various ETL techniques and frameworks, such as Flume, Apache NiFi, or DBT.
• Experience with various messaging systems, such as Kafka or RabbitMQ.
• Good understanding of Lambda Architecture, along with its advantages and drawbacks

Experience

Qualifications and Education Requirements

• Bachelor's and/or master’s degree