About the role
The team builds and maintains web-crawling systems for data collection at internet scale. The engineer in this role designs, builds, and owns the crawler infrastructure and data processing pipelines. This involves ensuring the reliability and efficiency of systems that handle petabyte-scale data.
Our summary of the posting; the original is on the Thinking Machines Lab careers page.
What the posting requires
- Required
- Python
- Distributed systems
- Web crawling
- Filtering
- Infrastructure
- Pipelines
- Web crawler
- Specialized crawlers
- Reliability
- Preferred
- Search engine
- Indexing
- AI lab
- Machine learning
- Open-source
- Crawl selection
These tags are our reading of the posting. They can miss something; the original posting is the reference.