Big Data Engineer

Key Skills

Agile methodologies, Analytical and Problem solving, AWS, Big Data Technology, Continuous Integration & Continuous Delivery - CI/CD, Data Architecture, Data Governance, Data management, Data pipelines, Data science techniques, Design, Effective communication skills, Leadership Skill, Optimization, PySpark, Scala Programming, Snowflake, Software Development, SQL

Job Description

Responsibilities

• Data Architecture: Design and implement scalable, robust, and efficient data architectures, including data lakes, data warehouses, and data pipelines, to support the organization's data requirements.
• Data Pipeline Development: Develop and maintain ETL (Extract, Transform, Load) processes and data pipelines to ingest, process, and transform large volumes of data from various sources, ensuring data quality and reliability.
• Data Modeling: Design and implement data models and schemas to organize and structure data for efficient storage, retrieval, and analysis, considering business requirements and data governance principles.
• Data Integration: Integrate data from diverse sources, such as databases, APIs, files, and streaming platforms, ensuring seamless data flow and interoperability across systems.
• Data Transformation and Enrichment: Implement data transformation and enrichment processes to derive insights, perform aggregations, and create derived datasets for analytics, reporting, and machine learning.
• Data Governance and Compliance: Establish and enforce data governance policies, standards, and best practices to ensure data integrity, security, privacy, and compliance with regulatory requirements.
• Performance Optimization: Optimize data pipelines and queries for performance, scalability, and efficiency, leveraging techniques such as partitioning, indexing, and query optimization.
• Collaboration: Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver data solutions that meet business needs.

Required Experience

• 3+ years of Data Management experience in an enterprise environment.
• 2+ years of experience with SQL.
• 1+ years of experience with AWS S3, Spark and Glue.
• Knowledge of metadata management, data governance, and data lineage.

Preferred Experience

• Good understanding of Agile Software Development practices and how to leverage these to streamline project execution.
• The ability to decompose business problems into smaller parts and solve them correctly, considering the impact on other subsystems.
• AWS certification.
• Team player with excellent problem-solving and troubleshooting skills.
• Familiar with software engineering practices, including CI/CD pipelines, Automated Testing, and Performance Engineering.
• Working knowledge of SDLC, especially regarding improving agile delivery metrics.
• Strong communication skills.
• Bachelor’s degree in Computer Science, MIS, or a related field.
• Financial services experience.
• Consulting experience.

Doran Jones Inc. (DJI) is the leading financial services Data Engineering and Application Development firm, specializing in Capital Markets, Risk, and Regulatory Compliance. We are US-based, helping our clients fill gaps in capacity and expertise, reduce risk, and accelerate change. Our leaders average over 25 years of experience in Financial Services and Technology. DJI is repeatedly engaged by the largest and most complex clients for our Agile transformation expertise and practical understanding of the critical relationship between data, architecture, and application development.