Data Engineer II (Python and API's)

Key Skills

Python Programming, Database, SQL, AWS, Big Data Technology, PyTorch, Apache Airflow, ScyllaDB, Flask, FastAPI

Job Description

With 1,000 intelligence professionals, over $300M in sales, and serving over 1,800 clients worldwide, Recorded Future is the world’s most advanced, and largest, intelligence company!

Data Engineer II, Attack Surface Module:

We are looking for a responsible Data Engineer who will help us build internet inventory, robust APIs, and Data Services for security teams. The successful candidate will be responsible for implementing and testing data pipelines that support our organization's business objectives. You'll join a small team of highly skilled professionals, bring new knowledge, and have the opportunity to shape our culture.

What you’ll do as a Data Engineer:

• Assist in solving data engineering problems relating to parsing global whois data, processing large volumes of internet scan data, indexing that data so that it is searchable, and building systems to monitor these processes
• Build high value analytics on raw data from several scanning pipelines
• Within your first three months, you will contribute code that will improve the the breadth and depth of what we scan across the internet

What you will bring to the Data Engineer role:

• 3+ years experience in software engineering using Python
• Great problem solving capabilities and experience troubleshooting data issues.
• Proactive communication across teams to identify and solve problems and make a project successful.
• Additional beneficial experience:

• Cloud computing experience is an asset to the role.
• Experience with Flask or FastAPI
• Experience with Spark, Scylla, Airflow, and other big data tools
• Elasticsearch experience is a plus, as well as work with Clickhouse / column oriented databases