Cloud Data Engineer

Key Skills

Java Programming, Python Programming, SQL, Agile methodologies, AWS, RDBMS, Big Data Technology, Apache Kafka, Google BigQuery, Snowflake, Google Cloud Platform (GCP), HDFS, Amazon RedShift, Linux Operating system, SPARK Programming, Parquet, ETL frameworks, Avro, Data Analysis

Job Description

ICE Data Services (an Intercontinental Exchange company) is seeking strong candidates to join its Technology team. This team supports a variety of reference data, index, and alternative data products. This role presents a unique opportunity to support innovative big data applications and new products aimed at helping the financial sector respond to the interrelated crises of climate change and widening socioeconomic inequality. The successful candidate will be able to multitask in a dynamic team-based environment, whilst demonstrating strong problem-solving, extensive technical skills, decision-making abilities, and the highest level of professionalism.

Responsibilities

• Develop ETL applications to interface between cloud-based data analytics and storage platforms and on-prem databases and data storage and transfer systems
• Optimize and maintain data pipelines for speed, robustness, integrity, and security.
• Improve production pipelines with better unit testing, validation, alerting, and ongoing performance profiling.
• Create robust ingestion pipelines for both commercial and publicly available third-party datasets
• Explore opportunities and make recommendations for using cloud technologies to optimize business outcomes (faster time to market for new products, cost savings, improved security, and operations, etc.).

Knowledge and Experience

• Experience using the Google Cloud Platform for big data applications, with a focus on BigQuery and Google Cloud Storage. Comparable experience with analogues in the AWS and Apache big data stacks is helpful (e.g., Spark, Parquet, Avro, HDFS, Redshift, Kafka, etc).
• 5+ years in an enterprise environment working with data-intensive applications in both on-premises and cloud database and storage applications.
• Experience coding data manipulation and transformation processes across multiple platforms and formats, preferably in python, using libraries like pandas, PyArrow, google cloud (or other vendor-specific libraries and APIs) or frameworks like DBT
• Ability to research, evaluate, compare, and adopt relevant open-source libraries or cloud tools
• Experience designing fault-tolerant and idempotent processes that can be incorporated into scheduling and orchestration pipelines
• Knowledge of SQL, ideally in the context both traditional RDBMSs and cloud datastores (BigQuery, Snowflake). Familiarity with the fundamental principles of optimizing performance a plus.
• Experience working on Linux platforms.
• Experience using modern source control and agile practices.
• Familiarity with testing methodologies, strategies, and automation tools.
• Curiosity about and knowledge of financial applications, geospatial analysis, and alternative data sources is a plus.