Data Engineer

Key Skills

Apache Hadoop, Azure, C# Programming, Data Engineering, Data Lake, Data pipelines, Data science techniques, DevOps, Effective communication skills, IT, NoSQL, NumPy, Optimization, Oracle, Pandas, PySpark, Python Programming, Rust, SPARK Programming, SQL

Job Description

Duties:

• Build a deep understanding of client needs to identify the best data solution across many industries and specific challenges
• Create, redevelop, and maintain scalable data systems and pipelines
• Identify and implement optimization opportunities, such as re-architecture and automation wherever helpful
• Perform root cause analysis on internal and external data processes to answer business questions and identify areas for improvement
• Bring together complex data sets in formats that are functional for data scientists and client goals
• Identify data source errors and issues and investigate and execute data quality enhancements
• Collaborate with data scientists, business analysts, policy analysts, and project managers to prioritize tasks, understand client pain points and challenges, and provide usable, relevant solutions
• Function as part of a rapid response team to learn quickly and respond to ad hoc needs
• Provide consistency in data architecture across projects at the same client
• Work flexibly and creatively in a fast-paced environment

Desired Skills & Experience

• Experience with Azure Data Services, such as Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake Storage.
• Proficient in Python and SQL.
• Familiarity with Python libraries, such as Pandas, NumPy, PyTest, and PySpark.
• Familiarity with other cloud analytics systems such as AWS, Snowflake, or Google Cloud services is a plus.
• Familiarity with Apache Spark and/or Hadoop.
• Familiarity with other programming languages such as Go, Rust, C#, is a plus.
• Experience with SQL and NoSQL on various relational database platforms such as Microsoft, Oracle, Hana, Postgres, and Cassandra
• Experience with Azure SQL Data Warehouse and dimensional modeling concepts.
• Proficient in version control tools such as Azure DevOps or Git.
• Strong verbal and communication skills.
• Collaborative team player who is detailed oriented and focused on solution quality and execution.
• Comfortable working across a wide range of project sizes and industries. 

Education

• Bachelor’s degree in information technology, computer science or business administration required.