Data Engineering & Web Crawling

Key Skills

Aartificial intelligence, Algorithms, C++, CUDA kernels, Data Engineering, Effective communication skills, GPUs, Python Programming, Rust

Job Description

About xAI

xAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge.

Our team is small, highly motivated, and focused on engineering excellence. This is an organization for those who appreciate challenging themselves and who thrive on curiosity. Engineers are encouraged to work across multiple areas of the company, and as a result all engineers and researchers share the title “Member of Technical Staff”.

We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company’s mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important.

All engineers and researchers are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.

xAI does not have recruiters. Every application is reviewed directly by a technical member of the team.

Tech Stack

• Rust
• Python
• Spark

Focus

• Building petabyte-scale, high-throughput data processing systems.
• Managing workloads across large cloud compute clusters.
• Pre-processing datasets for AI training.

Ideal Experience

• Building bespoke data processing libraries from scratch.
• Designing and implementing distributed systems in Rust.
• Keeping up with state-of-the-art techniques for preparing AI training data.
• Organizing and meticulously bookkeeping data across multiple clouds, of multiple modalities, and from many sources.