Senior Data Engineer

Key Skills

Agile methodologies, Analytical and Problem solving, Apache Airflow, Apache Cassandra, Apache Hadoop, Apache Kafka, AWS, Azure, Data Modeling, Data pipelines, Data structuring, Effective communication skills, Metadata, NoSQL, Optimization, Postgres, Python Programming, Scala Programming, Spark Core, SQL Server, Teamwork

Job Description

A Sr. Data Engineer designs and leads the implementation of data flows to connect operational systems and capture data for analytic and business intelligence systems. Key responsibilities include but are not limited to, recognizing opportunities to re-use existing data flows; leading the building of data streaming systems; optimizing the code to ensure processes perform optimally and leading routine work on database management.

As a senior member of the team, the incumbent will work on the most complex data systems and provide guidance to more junior members of the team, when needed.

Primary Accountabilities:

Technical (100%)

• Understand information security and the types of security controls that can be used to mitigate security threats within solutions and services.
• Able to effectively translate and accurately communicate across technical and non-technical stakeholders, as well as facilitating discussions within a multidisciplinary team, with potentially difficult dynamics.
• Able to advocate for the team externally and manage differing perspectives.
• Undertake data profiling and source system analysis and present clear insights to colleagues to support the end use of the data.
• Lead the design, building and testing of data products that are complex or large scale; build teams to complete data integration services.
• Select and implement the appropriate technologies to deliver resilient, scalable and future-proof data solutions.
• Understand the concepts and principles of data modeling and is able to produce relevant data models across multiple subject areas; reverse engineer data models from a live system.
• Understand industry recognized data modeling patterns and standards and when to apply them; compare and align different data models.
• Understand the impact on the organization of emerging trends in data tools, analysis techniques and data usage.
• Design an appropriate metadata repository and present changes to existing metadata repositories; understand a range of tools for storing and working with metadata; provide oversight and advice to more inexperienced members of the team.
• Own an issue until a new owner has been found or the problem has been mitigated and / or resolved.
• Initiate and monitor actions to investigate patterns and trends to resolve problems, including consulting specialists where required; determine the appropriate remedy and assists with implementation of these as well as preventative measures.
• Use agreed upon standard and tool to design, code, test, correct and document moderate to complex programs and scripts from agreed specifications and subsequent iterations; collaborate with others to review specifications where appropriate.
• Take inputs and establish coherent frameworks that work.
• Understand core technical concepts related to their role and apply them with guidance.
• Be seen as a SME for the Data team; mentor junior members of the team, when needed.

Required Qualifications:

• Bachelor of Science Degree in Computer Science, or a Bachelor of Arts Degree in a related technical field,
• 10+ years related work experience in data engineering or similar role
• Or any equivalent combination of experience and training/certification in platform maintenance and development
• Experience building and optimizing big data pipeline architectures and data sets
• Experience with object oriented/object function scripting languages such as Scala and/orPython
• Experience with public cloud technologies in Google (preferred), AWS and Azure
• Experience with relational and NoSQL databases such as SQL Server, Postgres, Cassandra, etc.
• Working knowledge of message queuing, stream processing and highly scalable big data stores
• Experience with financial/ERP systems such as Oracle Cloud ERP
• Experience with big data applications such as Hadoop, Spark, Kafka, etc.
• You have built processes that support data transformation, data structures, metadata, dependency and workload management
• A successful history of manipulation, processing and extracting value from data sets
• Familiarity with Agile methodologies and development processes
• You have performed root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
• Experience supporting and working with cross-functional teams in a dynamic environment
• Experience with data pipeline and workflow management tools such as Airflow

Individual Competencies:

• Integrity: Gains the trust of others by taking responsibility for your own actions and telling the truth.
• Teamwork: Builds relationships and works cooperatively with others, inside and outside the organization, to accomplish objectives to build and maintain mutually-beneficial partnerships, leverage information and achieve results.
• Adaptable: Responds to change with a willingness to learn new ways to accomplish work objectives with a positive attitude.
• Innovative: Ability to develop, sponsor, or support the introduction of new and improved methods, products, procedures or technologies.
• Curious: A desire to inquire and learn, to seek new knowledge and wisdom, and to listen to the contributions of others with a genuine interest to better self, the team, and the organization.
• Analytical and Critical Thinking: Ability to tackle a problem by using a logical, systematic, sequential approach.
• Problem Solving: Gathers and analyzes information to generate and evaluate potential solutions to problems, issues and challenges while weighing the accuracy and relevance of the facts, data and information.