Every blueprint ships with source code, sample data, and step-by-step instructions — ready to run on your Cloudera Platform
-
Blueprint Card
Overview
Source Ledger turns a search prompt into a queryable table of source URLs — so a search becomes a solid resource rather than disappearing after a try. In a web browser, a search gives us URLs to click, but it does not store those URLs or the search parameters behind them. We also lose track of which engine we used (e.g., DuckDuckGo, Yahoo, Wikipedia, arXiv), the query, and other constraints (e.g., region, time, safe search) that scopes the results.
The problem is that search results are usually blown away after the initial search. We do not remember what we searched for, or how it was searched.
We introduce Source Ledger, which tabulates every search result as a governed record — the engine used, the query, the URLs, and the search parameters applied — across multiple search engines and domains. The program converts individual, siloed searching into a shared, queryable table the whole team can draw on. You can turn hours of repeated ad hoc research into reusable, AI-ready data and cut redundant search work.
Demo
Run one query across the open web, Wikipedia, arXiv, and OpenAlex. Source Ledger saves every result, with the query, engine, and settings used, into a queryable table that's ready for AI pipelines. Watch the demo
Key Features
Every result knows how it was found. Each URL is stored with the query, search engine, and hyperparameters that produced the results, so a source table can be audited, reproduced, or extended later.
One question, multiple corpora. Reach the open web, an encyclopedia, a scholarly record, and preprints through a single call, without writing per-source code
Links only, never page content. Source Ledger stores links and search settings, not page content, which keeps its data footprint small by design. Individual URLs are not copyrighted.
Use Case
Record. Every search is captured, not lost.
Enables team work. A search becomes something a team can pick up, review, or continue. It does not stop one person's private activity that disappears after a search.
Same format, queryable. Every search lands in the same governed schema, so the whole history of searches — anyone's, any time — can be queried, filtered. It is not scattered across individual trials.
Architecture
A query first passes through search engine selection, which routes it to one of three corpus types: web search (e.g., DuckDuckGo, Yahoo, Startpage, and Yandex queried in parallel), encyclopedia (e.g., Wikipedia), or academia (e.g., arXiv and OpenAlex). The selected engines run the search under the user's hyperparameters (e.g., region, safe search, time limit, result count), and the returned URLs are aggregated together with the original query and the parameters that were actually applied. Each aggregated result is written as a governed record to a relational database.
Your form submission has failed.
This may have been caused by one of the following:
- Your request timed out
- A plugin/browser extension blocked the submission. If you have an ad blocking plugin please disable it and close this message to reload the page.