Senior Data Engineer (Cyber Defence System)

Key Skills

Apache Airflow, Apache Hadoop, Apache Kafka, Big Data Technology, Java Programming, Machine learning techniques, MLflow, MLOps tools, NoSQL, Python Programming, Spark Core, SparkML, Splunk, SQL

Job Description

As a Senior Data Engineer in the Cyber Defence System product group, you will be involved in creating, maintaining and scaling optimal data pipeline infrastructure, as well as architecting and implementing infrastructure and services to enable MLOps. This will lay the critical foundation upon which various Cyber Defence applications and workflows will rely on to deliver upon the mission and objectives effectively and efficiently, thereby strengthening our Cyber Defence Capabilities.

Responsibilities

• Create, maintain and scale optimal data pipeline infrastructure at Terabyte scale, ensuring both present and future data engineering requirements are met
• Building and optimizing 'big data' data pipelines, architectures and data sets.
• Architect and implement infrastructure and services to enable MLOps and work with both data scientists and threat researchers in making ML models and heuristics to streamline deployments to production
• Coach and develop junior data engineers to be more effective individual contributors

Requirements

• Master's or Bachelor's Degree in Computer/Electrical Engineering, Computer Science or equivalent qualifications
• Passion for data, new data technologies, and discovering new and interesting solutions to smeet the organisation's data needs
• Experience in designing and operating Terabyte scaled data infrastructure that can efficiently and reliably process both structured and unstructured datasets
• Proficient with modern data engineering practices e.g. building fault-tolerant pipelines, handling and logging errors, monitoring the system, understanding how to scale up, addressing continuous integration, knowledge of administration, ensuring a reliable and deterministic pipeline
• Team player with excellent communication skills to communicate with the fellow engineers to coordinate development of various data products and pipelines, and launch of any new product features to data scientist and threat researchers
• We are looking for a candidate with 5+ years of experience in a Data Engineer role who have experience architecting and operating a big data infrastructure.
• They should also have experience using the following software/tools:
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational SQL and NoSQL databases, including Postgres and Elasticsearch
• Experience with monitoring tools like ELK and Splunk
• Experience with data pipeline and workflow management tools: Airflow etc.
• Experience with object-oriented/object function languages: Python, Java, Scala etc.
• Experience with ML and MLOps frameworks and with e.g. SparkML, MLflow

As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.