Senior Big Data Engineer

Key Skills

Agile methodologies, Amazon Simple Storage Service (S3), AWS, Big Data Technology, Continuous Integration & Continuous Delivery - CI/CD, Data Analytics, Data management, Data science techniques, PySpark, Scala Programming, SDLC, Software Development, SPARK Programming, SQL

Job Description

Responsibilities

• Technical Leadership: Provide technical leadership and guidance to junior data engineers, mentoring them on best practices, design principles, and emerging technologies in the field of data engineering.
• Data Architecture: Design and implement scalable, robust, and efficient data architectures, including data lakes, data warehouses, and data pipelines, to support the organization's data requirements.
• Data Pipeline Development: Develop and maintain ETL (Extract, Transform, Load) processes and data pipelines to ingest, process, and transform large volumes of data from various sources, ensuring data quality and reliability.
• Data Modeling: Design and implement data models and schemas to organize and structure data for efficient storage, retrieval, and analysis, considering business requirements and data governance principles.
• Data Integration: Integrate data from diverse sources, such as databases, APIs, files, and streaming platforms, ensuring seamless data flow and interoperability across systems.
• Data Transformation and Enrichment: Implement data transformation and enrichment processes to derive insights, perform aggregations, and create derived datasets for analytics, reporting, and machine learning.
• Data Governance and Compliance: Establish and enforce data governance policies, standards, and best practices to ensure data integrity, security, privacy, and compliance with regulatory requirements.
• Performance Optimization: Optimize data pipelines and queries for performance, scalability, and efficiency, leveraging techniques such as partitioning, indexing, and query optimization.
• Collaboration: Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver data solutions that meet business needs.

Required Experience

• 7+ Years of overall software development experience in enterprise systems.
• 6+ years of Data Management experience in an enterprise environment.
• 6+ years of experience with SQL.
• 5+ years of experience with AWS S3, Spark, and Glue using PySpark (Scala is nice to have).
• 5+ years working with Snowflake, Databricks
• Strong knowledge of metadata management, data governance, and data lineage.
• In-depth knowledge of Agile Software Development practices and how to leverage these to streamline project execution.
• Possess the ability to decompose business problems into smaller parts to correctly solution, taking into consideration the impact on other subsystems.
• Team player with excellent problem-solving and troubleshooting skills.
• Familiar with software engineering practices, including CI/CD pipelines, Automated Testing, and Performance Engineering.
• Working knowledge of SDLC, especially regarding improving agile delivery metrics.
• Strong communication and team/technical leadership skills.
• Bachelor’s degree in Computer Science, MIS, or a related field.

Preferred Experience

• AWS certification preferred
• Financial services experience
• Consulting experience.

Doran Jones Inc. (DJI) is the leading financial services Data Engineering and Application Development firm, specializing in Capital Markets, Risk, and Regulatory Compliance. We are US-based, helping our clients fill gaps in capacity and expertise, reduce risk, and accelerate change. Our leaders average over 25 years of experience in Financial Services and Technology. DJI is repeatedly engaged by the largest and most complex clients for our Agile transformation expertise and practical understanding of the critical relationship between data, architecture, and application development.

DJI has a mission to place more people from non-traditional backgrounds into sustainable technology careers. Through partnerships with non-profit technology programs in underserved communities and Veteran organizations, candidates transition from tech training programs into real IT careers at DJI. Our unique recruitment policy allows us to create exceptional teams, bringing a broad spectrum of experience to our company and creating anything but a traditional consulting firm.