Engineering Manager, Data Team

Key Skills

Apache Hadoop, C Programming, C++, Data science techniques, Java Programming, Machine learning techniques, MapReduce, Python Programming, PyTorch, R Programming, Scala Programming, SQL, TensorFlow

Job Description

Semantic Scholar (semanticscholar.org) is a free, AI-powered research tool for scientific literature. We have grown from a groundbreaking research project at the Allen Institute for Artificial Intelligence to a product millions of scholars use every week. Semantic Scholar uses artificial intelligence to extract the semantics, or meaning, of scientific literature and surface it to our users through a discovery experience that helps them understand science more effectively than any other tool.

Since our launch in 2015, we have indexed more than 180 million research papers and built a user base of millions of scholars in over 100 countries. Our work has been featured in the New York Times, Wired, Nature, Science, and many other publications. In March 2020, Semantic Scholar released the CORD-19 dataset, a free resource of over 130,000 scholarly articles about COVID-19, SARS-CoV-2, and related coronaviruses for use by the global research community. The dataset is now updated daily and supports dozens of research initiatives around the world.

The Data Team software development manager owns the team and systems that fetch hundreds of millions of research paper data from myriad sources and create high quality, information- rich structured representations of research papers. These representations power the Semantic Scholar research knowledge graph, rich website experience, and a host of AI features. The quality and timeliness of this data is crucial to enabling Semantic Scholar’s users to easily find and understand the latest research, which has a large multiplier effect in their ability to generate their own cutting-edge research. This is critical to ensure Semantic Scholar can continue to magnify its role as a go-to destination for finding the latest relevant research in a highly accessible manner. Reporting to our senior technical leader, the Director of Engineering, you will manage a team of highly skilled, mission-driven engineers.

Responsibilities:

• Manage and develop a 10+ person engineering team focused on the Semantic Scholar data pipeline.
• Develop and maintain a roadmap of investments to improve standards of site reliability, availability, scalability and data quality in support of business goals.
• Define the metrics and goals for the Data team in support of business objectives.
• Set and maintain a high bar for processing quality & reliability.
• Make judicious tradeoffs among new feature development, tech debt, system migration, operational improvements. 
• Support diversity, equity, and inclusion in day-to-day management, hiring, professional development, and promotion. 
• Maintain our strong collaborative, transparent, mission-oriented team culture.

Required Experience:

• 5+ years experience as a software developer
• 2+ years managing software developers
• Experience managing large, high-throughput data workflows and pipelines
• Experience designing and implementing architectural improvements
• B.S. in Software Engineering or related discipline

Desired Experience:

• 7+ years experience as a software developer
• 3+ years managing software developers
• Experience managing and and scaling pipelines to hundreds of millions of events weekly; experience with AWS systems a plus
• Familiarity with machine learning infrastructure and pipelines, understanding of data clustering and information extraction techniques
• Experience collaborating with or supporting research teams
• Experience building or managing an account-based ingestion pipeline framework a plus
• Experience implementing diversity, equity, and inclusion initiatives in management, hiring, professional development, and promotion
• M.S. in Software Engineering or related discipline