pi-web-access: Search Fallbacks, PDF & YouTube

Overview

pi-web-access adds web_search, fetch_content and follow-up content retrieval to Pi. Use the installation command below, restart Pi, then begin with a public URL or a small search.

Zero setup applies to the keyless Exa MCP search path. It does not promise free access to every provider, offline operation, authenticated websites or YouTube analysis without an available backend.

How search falls back

Automatic search starts with configured SearXNG. When the active model is openai-codex it tries authenticated OpenAI first; otherwise Exa precedes OpenAI. Keyed Exa uses the direct API and unkeyed Exa uses MCP.

The remaining eligible sequence in this revision is Brave → Parallel → TinyFish → Search1API → Searchinfinity → Querit → Tavily → Firecrawl → Jina → SERPdive → Kagi → Bocha → Ollama → Perplexity → Gemini API/Web. Unavailable services are skipped. Cookie-based Gemini Web is opt-in.

DuckDuckGo is explicit-only. A named provider stays strict; searchRouting.providers can define an ordered route, and fallbackOn controls which failure types advance it. Do not assume every error silently selects another backend.

How URL extraction differs

A GitHub repository can be cloned for local file access. Ordinary HTML first uses direct HTTP, Readability and an RSC parser. Configured self-hosted Firecrawl or Crawl4AI may follow; third-party hosted extractors require fetchRouting.allowRemoteHostedProviders: true for remote HTTP(S) targets.

That opt-in lets a remote service receive the target URL and perform its own fetch. It is separate from API search configuration. Neither the fallback chain nor “intelligent extraction” guarantees access to login-only, paywalled or anti-bot content.

PDF dependencies and data paths

PDF extraction tries configured Datalab, then configured Gemini, then local unpdf/pdf.js. The local path does not need pdftotext, a browser or an API key, but reads only embedded text and has no OCR. Remote PDF tiers receive the PDF bytes.

Set pdf.provider to unpdf for local extraction; scanned pages may then return little or no text. Defaults are 20 MB and the first 100 pages. Remote errors can fall back locally, while cancellation and credential/configuration errors are not treated as ordinary recoverable failures.

{
  "pdf": { "provider": "unpdf", "maxSizeMB": 20, "maxPages": 100 }
}

YouTube analysis versus frame extraction

YouTube understanding tries Gemini Web when browser cookies are explicitly allowed, then Gemini API, then Perplexity. Configure an available authorized backend; private and age-restricted videos can fail across the chain.

ffmpeg and yt-dlp are optional system binaries for retrieving YouTube frames as images. They are not prerequisites for remote transcript/visual descriptions. Local video frame extraction needs ffmpeg; video analysis can upload the file to Gemini.

Configure and verify

Optional provider settings live in ~/.pi/agent/web-search.json. For example, braveApiKey can use "$BRAVE_API_KEY" to resolve that environment variable. Merge the PDF example above into the same JSON object when needed.

After restarting Pi, request a search and fetch as shown. Check source URLs and provider metadata, then try a known text PDF. These are reproducible verification steps, not a claim that this project ran each upstream service or measured extraction accuracy.

web_search({ query: "Pi coding agent documentation", numResults: 3 })
fetch_content({ url: "https://pi.dev/" })