Data Engineer

Key Skills

Ruby on Rails, Java Programming, Planning and organization, Design of Experiment, Python Programming, MongoDB, Database, SQL, DevOps, AWS, Big Data Technology, Apache Hadoop, Apache Kafka, Scala Programming, NoSQL, Apache Hive, Cloudera, REDIS, RESTful, MySQL, SPARK Programming, HBase, Web APIs, Apache Oozie, C# Programming

Job Description

Primary job responsibilities involve preparing data for analytical or operational uses across broad line of business needs areas and enterprise data domains. The data engineer often works as a dedicated member of delivery teams, focused on providing data in a ready-to-use services form for analytics groups and data scientists who are interrogating information for predictive analytics, machine learning and data mining purposes. In many cases, the data engineer also works with business units and departments integrating, consolidating and cleansing data, and structuring aggregations for use in individual analytics applications for business analysts, leadership groups, and other end users to aid in ongoing operational insights.

An independent & self-motivated lead Data Engineer must be versed in broad approaches to data architecture and applications, and will develop components/applications by studying operations and designing, and developing, reusable services and solutions that support the automated ingesting, profiling, and analysis of structured and unstructured data.

Primary Purpose:

Primary job responsibilities involve preparing data for analytical or operational uses across broad line of business needs areas and enterprise data domains. The data engineer often works as a dedicated member of delivery teams, focused on providing data in a ready-to-use services form for analytics groups and data scientists who are interrogating information for predictive analytics, machine learning and data mining purposes. In many cases, the data engineer also works with business units and departments integrating, consolidating and cleansing data, and structuring aggregations for use in individual analytics applications for business analysts, leadership groups, and other end users to aid in ongoing operational insights.

An independent & self-motivated lead Data Engineer must be versed in broad approaches to data architecture and applications, and will develop components/applications by studying operations and designing, and developing, reusable services and solutions that support the automated ingesting, profiling, and analysis of structured and unstructured data.

Essential Duties and Responsibilities:

Following is a summary of the essential functions for this job. Other duties may be performed, both m or and minor, which are not mentioned below. Specific activities may change from time to time.

• 4-6 years software development/ programming experience in enterprise/ web/ cloud applications
• 5+ years of experience in Data Modeling, data warehousing and data design
• Fluent with functional, imperative and object-oriented languages and methodologies.

• Experience with BIG DATA and Cloudera Hadoop.
• Experience with Cloud Infrastructure (AWS) and Snowflake
• Experience with SQL (MySQL, Postgres) and NoSQL(MongoDB/HBase/ReDis) database is expected.
• Proficiency with various operating systems (Linux/ Windows)

• Strong knowledge of programming languages/tools including: C#, Java, Python, Ruby, Scala, SQL and scripting (Java, Python, Spark, SQL, Hive, JavaScript, Shell). Scripts
• Experience with Big Data approaches and technologies including: Hadoop, Cloudera utilities, Spark, Kafka, Hive, Oozie (experience with Angular JS/HTML5/Node JS are big plus).
• Experience implementing and consuming large-scale web services (RESTful APIs)

• Has led, or been directly involved with, the investigation and resolution of complex data, system, and software systems requiring solutioning.
• Experience working in an agile environment.

• Explores, examines and interprets large volumes of data in various forms and recommends additional sources of data for improvements.
• Very strong knowledge of design patterns; logical, physical, dimensional and 3NF modeling skills.
• Experience in data management best practices, real-time and batch data integration, and data rationalization. 
• Ability to prioritize well, communicate clearly, have a consistent track record of commitment and accountability for delivery, as well as excellent software engineering and troubleshooting skills.
• Must be able to work across multiple phases of the project (e.g. initiation, planning, requirements, design, etc.) and manage multiple responsibilities.
• Creates and evaluates the data needs of assigned projects and assures the integrity of the data.

• Translates business and data requirements into data models in support of enterprise data modellings, data definition and warehousing, and analytical systems.
• Understand the relationships across business information and units of data; collaborate with business and other departments to identify data usage patterns and to formulate business names, definitions and data quality rules for data elements.
• Maintain data mappings documents, business matrix and other data design artifacts that define technical data specifications and transformation rules.
• Understand database performance factors and trends pertaining to very large database design and collaborate with DBAs to implement mitigating physical modeling solutions; provide data structures optimized for information entry and retrieval;
• Effectively communicate data design to other members of the team to ensure proper implementation and utilization of the data structures; support development team during process design and performance tuning.
• Adopt quality assurance practices to include: following an appropriate modeling methodology, helping to establish department standards and procedures, reviewing and critiquing data models produced by others, participating in walkthroughs and audits, ensuring appropriate documentation is produced at all points in the process.
• Pursue continuous improvements based on lessons learned and industry best practices.
• Understand the goals and risks associated with the business and technical requirements, and offer counsel on risk mitigation and the alignment of data solution with objectives.
• Demonstrate a team orientation by working closely and effectively with business partners, development teams and outside services.

• Ability to apply systems thinking for solutions by considering broad potential alternatives and impact areas.
• Ability to travel as needed, occasionally overnight.

Desired Skills:

Knowledge of, and experience working in, DevOps environments is desirable. Previous experience in the financial services industry and Internal Audit is a plus. Experience with performance tuning and documenting changes. Exposure to container technologies (Docker or similar) and orchestration is a plus. Experience with metadata capture, management, and platforms.