Metadata-Version: 2.5
Name: fossick
Version: 0.1.9
Summary: Web search, fetch, crawl, and browser automation for humans and agents
Project-URL: Repository, https://github.com/vedicreader/fossick
Project-URL: Documentation, https://vedicreader.github.io/fossick/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.12
Requires-Dist: certifi>=2026.7.22
Requires-Dist: curl-cffi>=0.15.0
Requires-Dist: ddgs>=9.14.4
Requires-Dist: diskcache>=5.6.3
Requires-Dist: fastcdp>=0.0.5
Requires-Dist: fastcore>=1.12.31
Requires-Dist: html2text>=2025.4.15
Requires-Dist: liteparse>=2.1.1
Requires-Dist: mcp<2,>=1.2.0
Requires-Dist: pdf-oxide>=0.3.67
Requires-Dist: readability-lxml>=0.8.4.1
Requires-Dist: scrapling[fetchers]>=0.4.8
Requires-Dist: yt-dlp>=2026.3.17
Provides-Extra: rerank
Requires-Dist: flashrank>=0.2.10; extra == 'rerank'
Description-Content-Type: text/markdown

# fossick


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

fossick covers the web-interaction stack that keeps coming up in Python and agent workflows: private local search, reading any page from a static blog to a JS-heavy SPA behind bot detection, and direct Chrome automation for authenticated sites, form filling, and multi-step flows.

The modules compose naturally. [`search()`](https://vedicreader.github.io/fossick/cli.html#search) finds URLs; [`fetch()`](https://vedicreader.github.io/fossick/cli.html#fetch) reads them — handling JavaScript rendering and anti-bot evasion automatically. For YouTube, arXiv papers, or GitHub repos, dedicated readers return structured data. When the page requires a real login session or interactive interaction, [`cdp_connect()`](https://vedicreader.github.io/fossick/cdp.html#cdp_connect) attaches to a running Chrome browser: `pg.snapshot()` hands an agent a compact map of the page and `pg.fill_form()` / `pg.act()` drive it.

## Install

``` sh
uv add fossick
```

Text/image/news search work out of the box (no Docker). JS rendering, `stealthy` fetching, and [`google()`](https://vedicreader.github.io/fossick/search.html#google) use a bundled headless browser.

## Search

[`search()`](https://vedicreader.github.io/fossick/cli.html#search) queries many backends in parallel through [`ddgs`](https://github.com/deedy5/ddgs) (Google, Brave, DuckDuckGo, Startpage, Mojeek, Yahoo, …) — no API key, no Docker. It keeps each backend’s own ranking and fuses them with [reciprocal rank fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf), then reranks with BM25 over titles, snippets and url slugs. (ddgs’ own aggregation picks ~2 backends at random when you ask for few results, merges them by hit count rather than rank, and pushes every Wikipedia hit to the top — so [`search()`](https://vedicreader.github.io/fossick/cli.html#search) doesn’t use it.) [`images()`](https://vedicreader.github.io/fossick/cli.html#images), [`news()`](https://vedicreader.github.io/fossick/cli.html#news), and [`videos()`](https://vedicreader.github.io/fossick/cli.html#videos) return the corresponding media.

Pass `backend='google'` to use a single backend, or `method='flashrank'` for cross-encoder reranking (`pip install fossick[rerank]`). Fan-out doesn’t depend on `n`, so a large `n` costs no extra requests — and `pages=2` pulls a second result page from every backend when you want a wider candidate pool to rerank yourself (`method='none'` returns the fused list untouched).

For real Google ranking when you specifically need it, [`google()`](https://vedicreader.github.io/fossick/search.html#google) uses a stealth browser to bypass the JavaScript/bot wall that blocks plain-HTTP scraping. Every result is a plain dict, and all functions share a local TTL cache so repeated queries return instantly.

``` python
results = search('kosha fts+semantic+code graph for codebases', method='flashrank', n=5)
for r in results: print(r['score'], r['engines'], r['title'], r['href'])
```

    0.030579 ['brave', 'startpage'] Codebase knowledge graph: Code analysis with graphs https://neo4j.com/blog/developer/codebase-knowledge-graph/
    0.029031 ['brave', 'startpage'] Semantic Code Graph—An Information Model to Facilitate Software Comprehension | IEEE Journals & Magazine | IEEE Xplore https://ieeexplore.ieee.org/document/10385091/
    0.031054 ['brave', 'startpage'] Semantic Code Graph – an information model to facilitate software ... https://arxiv.org/html/2310.02128v2
    0.016393 ['startpage'] GitHub - vedicreader/kosha: kosha (कोश) — a treasury of your repo and ... https://github.com/vedicreader/kosha
    0.015625 ['startpage'] graph – koshas https://vedicreader.github.io/kosha/graph.html

[`research()`](https://vedicreader.github.io/fossick/cli.html#research) goes further: it searches, reads the top hits, and returns one cited markdown corpus. It only counts a source once it’s *readable* — a Cloudflare interstitial is an HTTP 200 whose markdown reads “Enable JavaScript and cookies to continue”, so those are dropped (with the reason, in `res['dropped']`) and the next hit is fetched in their place. Each page is then trimmed to the passages that answer the query rather than to its first `chars` characters, which is usually nav and cookie banners.

[`research()`](https://vedicreader.github.io/fossick/cli.html#research) closes the loop from question to answer: it searches, reads the top results in parallel (auto-escalating past bot walls), and returns one cited markdown corpus — {query, sources, digest} — ready to hand to an LLM.

``` python
res = research('sqlite WAL mode vs journal mode performance', n=5)
print(res['digest'][:600])
for d in res['dropped']: print('skipped', d['href'], '—', d['reason'])
print('sources :')
print([s['href'] for s in res['sources']])
```

    ## blog.sqlite.ai › journal-modes-in-sqliteJournal Modes in SQLite
    https://blog.sqlite.ai/journal-modes-in-sqlite

    SQLite has been the go-to single-user database for over 20 years. Recent advancements in hardware and distributed consensus have opened up SQLite to more use cases - in particular, SQLite’s ease of use and performance characteristics make it an increasingly attractive option for web-scale applications.

    But if you try spinning up an app with SQLite’s default settings, you’ll hit a wall (pun intended) rather quickly. This is because the defaults in SQLite are geared towards its ori
    skipped https://superuser.com/questions/1938008/why-is-sqlite-wal-mode-so-much-faster-than-default-delete-mode-for-concurrent-wr — fetch failed
    skipped https://databaseschool.com/series/high-performance-sqlite/videos/27 — no readable content
    skipped https://javascript.plainenglish.io/stop-the-sqlite-performance-wars-your-database-can-be-10x-faster-and-its-not-magic-156022addc75 — fetch failed
    sources :
    ['https://blog.sqlite.ai/journal-modes-in-sqlite', 'https://gist.github.com/promto-c/531e3d3321f1c2fa66487054b2e040c2', 'https://sqlite.org/forum/info/117c91891cf7ac15', 'https://sqlite.org/wal.html', 'https://mohit-bhalla.medium.com/understanding-wal-mode-in-sqlite-boosting-performance-in-sql-crud-operations-for-ios-5a8bd8be93d2']

## Fetch and read

[`fetch()`](https://vedicreader.github.io/fossick/cli.html#fetch) returns a page dict with the raw HTML, parsed JSON if the response was JSON, and any XHR calls the page made. [`to_md()`](https://vedicreader.github.io/fossick/core.html#to_md) converts HTML to clean markdown, optionally narrowing to a CSS selector first. Pass `heavy=True` for JavaScript-rendered pages or `stealthy=True` for sites with anti-bot detection.

fossick also pulls transcripts and metadata from YouTube, papers and PDFs from arXiv, and files from GitHub repos.

Pass `auto=True` to let [`fetch()`](https://vedicreader.github.io/fossick/cli.html#fetch) escalate on its own — plain HTTP first, then a headless browser, then the stealth fetcher, then the logged-in debug Chrome — stopping at the first response that isn’t a bot wall (the winning tier is on `page.tier`). Pass `session=True` to route the fetch through your persistent debug Chrome so it reuses that browser’s logged-in cookies — read authenticated pages with no login code.

``` python
# extract just the lead paragraphs — no nav, ads, or sidebars
page = fetch('https://en.wikipedia.org/wiki/Web_scraping', verify=False)
print(to_md(page)[:400])
```

    The legality of web scraping varies across the world. In general, web scraping may be against the terms of service of some websites, but the enforceability of these terms is unclear.[11]

    In the United States, website owners can use three major legal claims to prevent undesired web scraping: (1) copyright infringement (compilation), (2) violation of the Computer Fraud and Abuse Act ("CFAA"), and (

[`crawl()`](https://vedicreader.github.io/fossick/cli.html#crawl) follows links from a start URL, returning a list of Page dicts — useful for documentation sites, blogs, and any multi-page content.

``` python
pages = crawl('https://docs.python.org/3/library/functions.html',
              follow_sel='a.reference.internal', same_domain=True, max_pages=3, verify=False)
print(f'{len(pages)} pages crawled')
```

    3 pages crawled

[`fetch_all()`](https://vedicreader.github.io/fossick/core.html#fetch_all) fetches multiple URLs in parallel — faster than sequential [`fetch()`](https://vedicreader.github.io/fossick/cli.html#fetch) calls.

``` python
urls = ['https://httpbun.com/get', 'https://httpbun.com/status/200', 'https://httpbun.com/json']
pages = fetch_all(urls, verify=False)
print([p.status for p in pages])
```

    [200, 200, 404]

[`read_arxiv()`](https://vedicreader.github.io/fossick/cli.html#read_arxiv) fetches paper metadata and converts the full PDF to markdown. Pass an arxiv ID, abstract URL, or PDF URL. The PDF is saved to `save_dir`; results are cached in-process.

``` python
paper = read_arxiv('2306.14881', save_dir='.', verify=False)
print(paper['title'])
print()
print(paper['summary'][:400])
```

    Modeling the molecular gas content and CO-to-H2 conversion factors in low-metallicity star-forming dwarf galaxies

    Low-metallicity dwarf galaxies often show no or little CO emission, despite the intense star formation observed in local samples. Both simulations and resolved observations indicate that molecular gas in low-metallicity galaxies may reside in small dense clumps, surrounded by a substantial amount of more diffuse gas, not traced by CO. Constraining the relative importance of CO-bright versus CO-dar

[`read_gh_repo()`](https://vedicreader.github.io/fossick/cli.html#read_gh_repo) clones or fetches a GitHub repo and returns `{path: content}` for matched files. [`read_gh_file()`](https://vedicreader.github.io/fossick/cli.html#read_gh_file) reads a single file from a GitHub blob URL.

``` python
files = read_gh_repo('https://github.com/vedicreader/kosha', globs=('README*',))
for path, content in files.items(): print(path.split('/')[-1], f'({len(content)} chars)')
```

    README.md (166202 chars)

``` python
txt = read_gh_file('https://github.com/vedicreader/kosha/blob/main/pyproject.toml'); txt[:300]
```

    '[build-system]\nrequires = ["hatchling"]\nbuild-backend = "hatchling.build"\n\n[project]\nname = "koshas"\ndynamic = ["version"]\ndescription = "kosha (कोश) — a treasury of your repo and environment context for coding agents. FTS5 + vector search + call graph, no LLMs required."\nreadme = "README.md"\nrequir'

[`search_yt()`](https://vedicreader.github.io/fossick/cli.html#search_yt) searches YouTube and returns metadata for each result. [`read_yt()`](https://vedicreader.github.io/fossick/cli.html#read_yt) fetches the full English transcript and metadata for a known video URL.

``` python
for h in search_yt('3blue1brown neural networks', n=3, verify=False): print(h['title'], '\n  ', h['url'])
```

    But what is a neural network? | Deep learning chapter 1 
       https://www.youtube.com/watch?v=aircAruvnKk
    Backpropagation, intuitively | Deep Learning Chapter 3 
       https://www.youtube.com/watch?v=Ilg3gGewQ5U
    Gradient descent, how neural networks learn | Deep Learning Chapter 2 
       https://www.youtube.com/watch?v=IHZwWFHWa-w

``` python
video = read_yt('https://www.youtube.com/watch?v=aircAruvnKk', verify=False)
print(video['title'])
print(video['source'][:300])
```

    But what is a neural network? | Deep learning chapter 1
    [Music] This is a three. It's sloppily written and rendered at an extremely low resolution of 28x 28 pixels. But your brain has no trouble recognizing it as a three. And I want you to take a moment to appreciate how crazy it is that brains can do this so effortlessly. I mean this, this, and this are

``` python
download_yt('https://www.youtube.com/watch?v=aircAruvnKk', format='audio', save_dir='.', verify=False)
```

                                                             

    Path('But what is a neural network？ ｜ Deep learning chapter 1.mp3')

[`url2nb()`](https://vedicreader.github.io/fossick/cli.html#url2nb) converts any URL — HTML page, arXiv paper, or PDF — to a Jupyter notebook. [`pdf2nb()`](https://vedicreader.github.io/fossick/cli.html#pdf2nb) converts a PDF (local path or URL) to a notebook where each page becomes a markdown cell with an empty code cell below for annotations.

``` python
url2nb('https://squiddev.medium.com/continuing-continuations-cps-in-python-47bba90c8d1e', verify=False)
```

    Path('continuing-continuations-cps-in-python-47bba90c8d1.ipynb')

``` python
nb = pdf2nb('https://selfdeterminationtheory.org/SDT/documents/2000_RyanDeci_SDT.pdf', 'sdt.ipynb', verify=False)
print(nb)   # Path('/path/to/sdt.ipynb')
```

    /Users/71293/code/personal/orgs/fossick/nbs/sdt.ipynb

## Discover hidden APIs

Most modern sites serve their data through undocumented JSON APIs rather than HTML. [`find_xhr()`](https://vedicreader.github.io/fossick/cli.html#find_xhr) visits a page with a headless browser and captures every network call the browser makes. [`paginate_api()`](https://vedicreader.github.io/fossick/cli.html#paginate_api) replays the request across all pages and collects the results.

For sites behind a login, pass `find_xhr(url, session=True)` to capture the calls through your authenticated debug Chrome. Each result carries a `capture` you can hand to [`replay_xhr()`](https://vedicreader.github.io/fossick/core.html#replay_xhr), which re-issues the request as a fast plain-HTTP call — reusing the browser’s cookies — turning any logged-in site’s internal API into a data feed.

``` python
posts = paginate_api(
    'https://jsonplaceholder.typicode.com/posts',
    payload={'_page': 1, '_limit': 10},
    page_field='_page',
    size_field='_limit',
    method='GET',
)
print(f'{len(posts)} posts collected')
print(posts[0]['title'])
```

    100 posts collected
    sunt aut facere repellat provident occaecati excepturi optio reprehenderit

``` python
# on JavaScript-heavy sites, intercept the hidden API with a headless browser
# Woolworths uses a GraphQL endpoint — find_xhr captures whatever calls the page makes
apis = find_xhr('https://www.woolworths.com.au/shop/browse/fruit-veg', pattern='*woolworths.com.au/graphql*')
print(f'{len(apis)} GraphQL calls captured')
for a in apis:
    data = a.get('data', {}).get('data', a.get('data', {}))
    print(a['url'], list(data.keys()) if isinstance(data, dict) else type(data).__name__)
```

    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Response body is unavailable for redirect responses
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Response body is unavailable for redirect responses
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Response body is unavailable for redirect responses
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Response body is unavailable for redirect responses
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Response body is unavailable for redirect responses
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier
    [2026-08-09 20:09:48] ERROR: Error getting page content in async: Response.body: Protocol error (Network.getResponseBody): No data found for resource with given identifier

    2 GraphQL calls captured
    https://www.woolworths.com.au/graphql ['products']
    https://www.woolworths.com.au/graphql ['products']

## Browser automation

[`cdp_connect()`](https://vedicreader.github.io/fossick/cdp.html#cdp_connect) attaches to a running Chrome browser over the DevTools Protocol. The browser uses a persistent debug profile so cookies and SSO sessions survive across runs. Log in once to any enterprise or authenticated site; every subsequent call reuses that session.

`pg.snapshot()` gives an LLM a compact, interactive-elements-only view of the page — one `[#id] role "name"` line per button, link, and field — far smaller than the full `ax_tree()`. Act on it with `pg.fill_form({label: value}, submit='Sign in')`, `pg.click_settle()`, or the CSS bridges `pg.click_sel()` / `pg.fill_sel()`. `pg.act([...])` runs a whole declarative flow (`goto` / `fill` / `click` / `select` / `read`) in one call, and `ax_diff(before, after)` shows exactly what an action changed. `pg.md()` pulls the live, post-JavaScript page straight into fossick’s markdown pipeline. [`syncy()`](https://vedicreader.github.io/fossick/core.html#syncy) runs any async CDP call synchronously.

`snapshot()` re-reads the tree for you, so IDs are always fresh. When you need the raw tree, print the full `ax_tree()` output — never truncate — and re-read it after each navigation, since backend IDs change.

``` python
from fossick.cdp import cdp_connect, syncy

cdp = syncy(cdp_connect())
pg  = syncy(cdp.open_page('https://the-internet.herokuapp.com/login'))

# snapshot() gives an LLM-readable view of interactive elements — always fresh after navigation
print(syncy(pg.snapshot()))
# [#4] textbox "Username"
# [#5] textbox "Password"
# [#6] button " Login"

# fill fields by their visible label and submit
syncy(pg.fill_form({'Username': 'tomsmith', 'Password': 'SuperSecretPassword!'}, submit='Login'))

# act() runs a declarative multi-step flow; ('read', sel) captures the page section as markdown
out = syncy(pg.act([
    ('goto', 'https://the-internet.herokuapp.com/login'),
    ('fill', 'Username', 'tomsmith'),
    ('fill', 'Password', 'SuperSecretPassword!'),
    ('click', 'Login'),
    ('read', '#flash'),
]))
print(out['#flash'])    # You logged into a secure area!
```

    # The Internet
    https://the-internet.herokuapp.com/login

    [#28] link "Fork me on GitHub"
    [#3] textbox "Username"
    [#4] textbox "Password"
    [#5] button " Login"
    [#49] link "Elemental Selenium"
    You logged into a secure area! ×

## Shopping carts

`fossick.shop` is the cart layer on top of that browser: an agent finds products, adds them, and — the part that usually goes wrong — *knows whether the add worked*. There are no per-site selectors to write. Products are found by deriving card boundaries from the page’s own link structure, so the same code reads a Shopify collection, an Amazon results page and a bespoke Rails storefront; where a site has a real cart API (Shopify’s `/cart.js`) it is used instead of clicking.

Every mutating call reads the cart before and after and reports the evidence: `ok=True` with `how='count'`, `'subtotal'`, `'lines'` or a confirmation toast. `ok=None` means the page exposes no cart signal to check against — an agent should never turn that into “added it”. Ambiguity is handed back rather than guessed: an unmatched name raises with the titles that were on offer, and a product needing a size returns `need='variant'` with the variants that are actually in stock.

``` python
from fossick.shop import shop

s = shop('https://members.ceresfairfood.org.au')   # opens in the persistent debug Chrome
# if s.blockers() shows 'login-required', log in by hand first — session persists in the debug Chrome

results = s.search('apples')
print(results[:2])
# [{'i': 0, 'title': 'Apples Fuji IPM 500g', 'price': 5.5, ...},
#  {'i': 1, 'title': 'Apples Fuji IPM 1kg', 'price': 9.95, ...}]

r = s.add('Apples Fuji Organic 500g', qty=2)
print(r)
# {'ok': True, 'how': 'count', 'qty': 2, 'qty_set': '2',
#  'before': {'count': 0, ...}, 'after': {'count': 1, 'subtotal': 13, ...}}

if not r.get('ok'):
    print('add failed:', r.get('error'), '| blockers:', s.blockers()); raise SystemExit

cart = s.cart_page()
# {'count': 1, 'subtotal': 13, 'lines': [{'i': 0, 'title': 'Apples Fuji Organic 500g', 'qty': 2, ...}]}

line = cart['lines'][0]['i']   # actual line id from the live cart (not hardcoded)
s.set_qty(line, 3)             # bump qty to 3
s.remove(line)                 # then remove it
```

    [{'i': 0, 'title': 'Apples Fuji IPM 500g', 'price': 5.5, 'url': 'https://members.ceresfairfood.org.au/products/4388-apples-fuji-ipm-500g', 'add': True, 'add_label': '', 'qty': 'select', 'vid': None, 'related': None, 'oos': None}, {'i': 1, 'title': 'Apples Fuji IPM 1kg', 'price': 9.95, 'url': 'https://members.ceresfairfood.org.au/products/4389-apples-fuji-ipm-1kg', 'add': True, 'add_label': '', 'qty': 'select', 'vid': None, 'related': None, 'oos': None}]
    {'ok': True, 'how': 'count', 'item': {'i': 2, 'title': 'Apples Fuji Organic 500g', 'price': 6.5, 'url': 'https://members.ceresfairfood.org.au/products/2252-apples-fuji-organic-500g', 'add': True, 'add_label': 'Add', 'qty': 'select', 'vid': None, 'related': None, 'oos': None}, 'qty': 2, 'clicks': 1, 'qty_set': '2', 'before': {'count': 5, 'subtotal': None, 'badges': [{'n': 5, 'tier': 0, 'src': 'top-navigation__top-bar-cart-counter'}], 'url': 'https://members.ceresfairfood.org.au/products/search?q=apples', 'title': 'Searched on: apples | CERES Fair Food', 'source': 'dom'}, 'after': {'count': 6, 'subtotal': None, 'badges': [{'n': 6, 'tier': 0, 'src': 'top-navigation__top-bar-cart-counter'}], 'url': 'https://members.ceresfairfood.org.au/products/search?q=apples', 'title': 'Searched on: apples | CERES Fair Food', 'source': 'dom'}}

    {'ok': True,
     'how': 'count',
     'line': {'i': 0,
      'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
      'price': 6.5,
      'qty': None,
      'qty_kind': None,
      'remove': True},
     'before': {'count': 6,
      'subtotal': 78,
      'badges': [{'n': 6,
        'tier': 0,
        'src': 'top-navigation__top-bar-cart-counter'}],
      'url': 'https://members.ceresfairfood.org.au/cart',
      'title': 'Shopping Cart | CERES Fair Food',
      'lines': [{'i': 0,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 1,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 2,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 3,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 4,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 5,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True}],
      'source': 'dom'},
     'after': {'count': 5,
      'subtotal': 65,
      'badges': [{'n': 5,
        'tier': 0,
        'src': 'top-navigation__top-bar-cart-counter'}],
      'url': 'https://members.ceresfairfood.org.au/cart',
      'title': 'Shopping Cart | CERES Fair Food',
      'lines': [{'i': 0,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 1,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 2,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 3,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True},
       {'i': 4,
        'text': 'Apples Fuji Organic 500g 2 bags $6.50 EDIT $13.00',
        'price': 6.5,
        'qty': None,
        'qty_kind': None,
        'remove': True}],
      'source': 'dom'}}

``` python
s = shop('https://www.coles.com.au')
results = s.search('milk')
print(results[:2])
# [{'i': 0, 'title': 'Coles Full Cream Milk | 3L', 'price': 5.15, ...},
#  {'i': 1, 'title': 'a2 Milk Full Cream Uht Milk | 1L', 'price': 3.9, ...}]

s.add('Coles Full Cream Milk | 3L')
# {'ok': True, 'how': 'count', 'qty': 1, 'qty_set': None,
#  'before': {'count': 0, ...}, 'after': {'count': 1, 'subtotal': 5.15, ...}}
```

    [{'i': 0, 'title': 'Coles Full Cream Milk | 3L', 'price': 5.15, 'url': 'https://www.coles.com.au/product/coles-full-cream-milk-3l-8150288', 'add': False, 'add_label': None, 'qty': 'input', 'vid': None, 'related': None, 'oos': None}, {'i': 1, 'title': 'Nescafe Pistachio Latte Sachets | 8 Pack', 'price': 8, 'url': 'https://www.coles.com.au/product/nescafe-pistachio-latte-sachets-8-pack-1800545', 'add': True, 'add_label': 'Add to trolley: Nescafe Pistachio Latte ', 'qty': None, 'vid': None, 'related': None, 'oos': None}]

    {'ok': False,
     'error': 'add control vanished before the click',
     'item': {'i': 0,
      'title': 'Coles Full Cream Milk | 3L',
      'price': 5.15,
      'url': 'https://www.coles.com.au/product/coles-full-cream-milk-3l-8150288',
      'add': True,
      'add_label': 'Add to trolley: Coles Lite Reduced Fat M',
      'qty': 'input',
      'vid': None,
      'related': None,
      'oos': True}}

Checkout forms are read before they are filled. `s.fields()` returns every visible input with its label, `autocomplete` token, current value and `<select>` options — ground truth, so nothing has to be guessed from a field name. `s.fill(profile)` maps a plain dict onto them (by autocomplete token first, since that is a spec rather than a hunch), then re-reads the form to report which values actually stuck. It will not press a payment button: `submit=` accepts a button name, but anything that reads like paying needs `confirm=True` as well.

`s.blockers()` names what is standing in the way — `cookie-banner`, `location-required`, `login-required`, `captcha` — and `s.dismiss()` clicks through the consent ones. Logging in and choosing a delivery store stay a human’s job, done once; the persistent profile keeps them.

## Setup — the persistent debug Chrome

`fetch(url, session=True)` and the CDP tools use a long-lived Chrome with remote debugging on port 9223 — you don’t have to start it yourself. `fetch(session=True)`, [`cdp_ws()`](https://vedicreader.github.io/fossick/cdp.html#cdp_ws), and [`cdp_connect()`](https://vedicreader.github.io/fossick/cdp.html#cdp_connect) launch one **headless** on first use, on a persistent profile (`~/.cache/fastcdp/cdp-chrome`), so cookies and SSO sessions survive across runs.

To log in by hand (Cloudflare / SSO / Anubis) you need a **headed** window. A running Chrome can’t switch modes, so quit it and relaunch with `headless=False` — see *Managing the debug Chrome* in the [cdp docs](cdp.html). To keep one always-on at login, install the bundled service file for your OS (`chrome_debug.plist` for launchd, `chrome_debug.service` for systemd, `chrome_debug_task.xml` for Task Scheduler).

`cdp_app(url)` opens a URL as a chromeless app window — no tab strip, no address bar — in that same persistent browser. It reuses a running debug Chrome when there is one, so a window per local web app still costs a single browser process.

## CLI

fossick ships a `fossick` command for shell scripts and agent harnesses that don’t have a Python kernel. All commands accept `--as_json` to return JSON instead of markdown.

``` sh
# fetch a page as markdown (--auto escalates plain->heavy->stealthy->session; --session uses the logged-in Chrome)
fossick fetch https://en.wikipedia.org/wiki/Web_scraping --sel '.mw-parser-output > p'

# research: search, then read the top results into one cited markdown corpus
fossick research "retrieval augmented generation best practices" --n 5

# compact, agent-ready accessibility snapshot of a live page in the debug Chrome
fossick ax https://example.com

# web search
fossick search "fasthtml python framework" --n 5

# read an arxiv paper (summary only by default; --source for full text)
fossick read-arxiv 2306.14881

# YouTube transcript
fossick read-yt https://www.youtube.com/watch?v=aircAruvnKk

# search YouTube
fossick search-yt "3blue1brown neural networks" --n 3

# download YouTube audio or video
fossick download-yt https://www.youtube.com/watch?v=aircAruvnKk --format audio

# convert a URL, PDF, or arXiv paper to a Jupyter notebook
fossick url2nb https://arxiv.org/abs/2306.14881

# capture outgoing network requests fired by a page (uses real Chrome session)
fossick calls https://example.com --pattern '*api*' --as_json

# interactive screenshot capture — overlays a button in Chrome, ✓ Done to finish
fossick collect https://example.com --save_dir shots

# click elements to annotate them with AX role + selector; saves labeled screenshot
fossick annotate https://example.com --save_dir shots

# drive a shopping cart — list, add (verified), read the cart. Each call reuses the tab
# the last one left open, so --add acts on the page --search landed you on.
fossick shop https://members.ceresfairfood.org.au --search apples
fossick shop https://members.ceresfairfood.org.au --add 'Apples Fuji Organic 500g' --qty 2
fossick shop https://members.ceresfairfood.org.au --cart

# install SKILL.md to .agents/skills/fossick/ and .claude/skills/fossick/
fossick install
```

``` python
from fossick.cdp import cdp_setup, cdp_connect, syncy, _debug_running
# Start a persistent debug Chrome you can log into; fetch(url, session=True) reuses it afterwards.
if _debug_running(9223):                       # a headless instance may already be running
    cdp = syncy(cdp_connect(port=9223))
    try: syncy(cdp.quit())                     # quit() drops the socket -> ConnectionClosedOK
    except Exception: pass
syncy(cdp_setup(9223, headless=False))         # headed: log in by hand, then use session=True
```

## MCP server

`fossick-mcp` exposes the whole toolkit over the Model Context Protocol, so Claude Code, Claude Desktop, Codex, and any other MCP client can drive fossick directly — search, fetch, readers, hidden-API discovery, the logged-in debug Chrome ([`browse`](https://vedicreader.github.io/fossick/mcp.html#browse) / `page_*` tools), and shopping carts ([`shop_open`](https://vedicreader.github.io/fossick/mcp.html#shop_open), [`shop_search`](https://vedicreader.github.io/fossick/mcp.html#shop_search), [`shop_products`](https://vedicreader.github.io/fossick/mcp.html#shop_products), [`shop_add`](https://vedicreader.github.io/fossick/mcp.html#shop_add), [`shop_cart`](https://vedicreader.github.io/fossick/mcp.html#shop_cart), [`shop_line`](https://vedicreader.github.io/fossick/mcp.html#shop_line), [`shop_fields`](https://vedicreader.github.io/fossick/mcp.html#shop_fields), [`shop_fill`](https://vedicreader.github.io/fossick/mcp.html#shop_fill), [`shop_dismiss`](https://vedicreader.github.io/fossick/mcp.html#shop_dismiss)).

``` sh
uv add 'fossick'        # or: pip install 'fossick'
```

**Claude Code**

``` sh
claude mcp add fossick -- uvx --from 'fossick' fossick-mcp
```

**Codex** (`~/.codex/config.toml`)

``` toml
[mcp_servers.fossick]
command = "uvx"
args = ["--from", "fossick", "fossick-mcp"]
```

**Claude Desktop** (`claude_desktop_config.json`)

``` json
{"mcpServers": {"fossick": {"command": "uvx", "args": ["--from", "fossick", "fossick-mcp"]}}}
```

The server speaks stdio by default (`fossick-mcp --http` for Streamable HTTP). Tools mirror the Python/CLI API — see the [mcp docs](https://vedicreader.github.io/fossick/mcp.html) for the full list.
