Data Engineer (Information Management Systems)

Key Skills

Apache Airflow, Apache Kafka, Apache NiFi, Apache Solr, Elasticsearch, ElasticStack, ETL frameworks, ETL/ELT technologies, Grafana, Java Programming, Mentoring, MongoDB, RabbitMQ, REST API, Scrum Agile Methodology

Job Description

CSIT develops digital products to advance the national security interests of Singapore. We are looking for talented software engineers with a growth mindset and a passion for creating great products to improve the effectiveness and productivity of the organisation.

About this role

As a data engineer in the Information Management Systems team, you will be responsible for developing and managing data pipelines that deliver data to downstream business-critical systems. You will have opportunities to create/improve ETL (extract, transform, load) flows that process large volumes of data. You will also build/enhance APIs to serve data to downstream services and end-users.

You will have the autonomy to find the best way to solve problems, such as finding the most suitable technologies to use.

Responsibilities 

• Write high-quality code with unit, functional and end-to-end tests
• Design, build, and deploy ETL solutions that are scalable and maintainable
• Triage system issues to ensure the smooth running of data pipelines and systems (detecting/debugging/tracking/resolving production incidents)
• Enable easy access to data for end-users (data scientists, analysts, and software engineers) via APIs with appropriate access control mechanisms

Requirements

Minimum

• Bachelor's degree in Computer Science, Computer Engineering, Information Systems or a related field
• At least 2 years of hands-on experience in Java programming
• (Fresh grads – project/internship experience will be considered)
• Experience in building ETL flows (e.g. NiFi, Airflow)
• Experience in back-end development using frameworks such as SpringBoot
• Familiarity with microservices architecture & RESTful API design

Preferred

• Software development experience with Agile SCRUM practises, testing, and production error/metrics monitoring
• Experience in building scalable and fault-tolerant data pipelines that are maintainable
• Familiarity with databases (e.g. MongoDB) and object stores (e.g. MinIO)

Experience in the following would be advantageous:

• Search engines (e.g. Elasticsearch, Solr)
• Messaging systems (e.g. RabbitMQ, Kafka)
• Metrics/logging systems (E.g. Elastic Stack, Prometheus/Grafana)

Considerations for senior positions

• Candidates with at least 3 years of experience in software development with proven track record of the above qualifications
• Experience in providing technical mentorship to junior software engineers
• Experience in collaborating across teams to develop and deliver complex systems/products (e.g. communicating and negotiating architectural designs and technical solutions)

As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.