Senior Big Data Developer

Key Skills

Aartificial intelligence, Analytical and Problem solving, Apache Airflow, Apache Hadoop, Apache NiFi, AWS, Azure, Big Data Technology, Data Analytics, Data cleaning / Data cleansing, Data Engineering, Data Lake, Data science techniques, Data Warehousing, Effective communication skills, ETL frameworks, Google Cloud Platform (GCP), NoSQL, SPARK Programming, SQL

Job Description

The Senior Data Engineer will be responsible for crafting, deploying, and maintaining the data pipeline architecture that is central to our analytics and trading platforms. The ideal candidate will bridge the gap between data science and data management, ensuring our data is accurate, accessible, and efficiently processed for various trading and investing applications.

What you'll do:

• Develop and maintain scalable and reliable data pipelines that can ingest and process large volumes of data with high velocity and variety
• Work closely with data scientists, analysts, and other stakeholders to understand data requirements and implement systems for data integration, data cleansing, and data analytics
• Build and optimize data models, ETL processes, and storage solutions (data lakes, data warehouses)
• Ensure data quality and compliance with data governance and security policies
• Monitor performance and advise on necessary infrastructure changes
• Define data retention policies and implement disaster recovery systems
• Troubleshoot and resolve any issues in the data pipeline
• Stay abreast of industry trends and best practices in data engineering and introduce new technology and tools where applicable

What you'll bring:

• Bachelor's or master's degree in Computer Science, Engineering, or a related field
• Proven experience as a Data Engineer or in a similar role
• Expertise in SQL and experience with relational databases, as well as NoSQL databases
• Familiarity with cloud services like AWS, Azure, or GCP, and experience with their data-related services
• Experience with big data tools (e.g., Hadoop, Spark) and data pipeline/workflow management tools (e.g., Airflow, NiFi)
• Ability to build and optimize data sets, 'big data' data pipelines and architectures
• Knowledge of programming languages such as Python, Java, or Scala
• Understanding of financial market data and experience with financial data platforms is highly desirable
• Excellent analytical and problem-solving abilities
• Strong communication and collaboration skills to effectively interface with various teams

What we offer:

• A chance to be part of a leading AI firm with a commitment to innovation in the financial sector
• A competitive salary and a comprehensive benefits package
• A culture of continuous learning and development with opportunities for career progression
• Open and inclusive workplace environment
• Engagement in cutting-edge work that has a real-world impact on trading and investing