Building RAG-backed scraping infrastructure

Problem

At Edra Labs, I built the admin dashboard, search engine, and scraping pipeline for the launches of allmachines.com and coveted.com. Gathering information from websites was one of the product workflows that needed less manual effort.

For an original equipment manufacturer (OEM) website, the scraping task took roughly 2 days before the generic LLM-driven scraper brought it down to 2 hours.

Constraints

The work was a generic web scraper, rather than a one-off extraction for a single page. The pipeline needed a way to index page content and retrieve information from the resulting chunks.

The recorded timing improvement applies to an OEM website, rather than being a benchmark across all websites.

Architecture

The pipeline used retrieval-augmented generation (RAG) techniques to index web pages. Qdrant served as the vector store, and the indexed chunks were used for information retrieval.

Decisions

• Make indexing and retrieval explicit. Page content became indexed chunks that the pipeline could retrieve from, rather than leaving information retrieval as an unspecified LLM step.
• Use Qdrant for the vector index. The vector store was a concrete part of the implemented retrieval architecture.
• Keep the index in the system boundary. RAG adds stored content to manage alongside LLM-backed processing; the scraper was not just a standalone prompt.

Execution

I developed the generic LLM-driven scraper as part of the scraping infrastructure. The implemented pipeline combined RAG-based web-page indexing, Qdrant, and information retrieval over the indexed chunks.

This technical work sat alongside the admin dashboard and search engine I built for the product launches. Product engineering, workflow automation, and leadership contributions are covered separately.

Impact

Scraping an OEM website went from roughly 2 days to 2 hours, reducing manual effort for that workflow.