Metadata-Version: 2.4
Name: dork-client
Version: 0.2.0
Summary: Compose, validate, run and export Google-style search-operator queries (dorks), and rewrite them for other engines.
License: MIT License
        
        Copyright (c) 2026 Google Dork Client contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/NOKREM/dork-client
Project-URL: Issues, https://github.com/NOKREM/dork-client/issues
Project-URL: Changelog, https://github.com/NOKREM/dork-client/blob/main/CHANGELOG.md
Keywords: osint,google-dork,search-operators,security,cli,serp
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Security
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic-settings>=2.3
Provides-Extra: search
Requires-Dist: httpx>=0.27; extra == "search"
Provides-Extra: dev
Requires-Dist: pytest>=8.2; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Dynamic: license-file

# Google Dork Client

A command-line tool for composing, validating, running and exporting
search-operator queries ("dorks"), and for rewriting them into the dialects
other search engines speak.

It is meant for OSINT research, checking how much of **your own** infrastructure
is publicly indexed, and general information gathering during authorised
security assessments.

> **Version 0.1.0 — feature complete.** Every command works, configuration
> comes from `.env` and the environment, every run is logged, and the suite
> covers 100% of the code with 743 tests that make no network request.
> See the [changelog](CHANGELOG.md) for what shipped.

---

## Scope and boundaries

The application only **composes queries, displays public search results,
filters, stores and exports them**. It deliberately does not, and will not:

- run exploits or automated vulnerability attacks,
- attempt logins, password guessing or authentication bypass,
- bypass CAPTCHAs, bot detection or access control,
- collect credentials, API keys or other secrets,
- evade or exceed a search engine's rate limits.

The backends that fetch results -- `google` and `brave` -- use those
providers' **official APIs** with your own credentials. Neither fetches or
parses a results page: scraping one would breach the provider's terms and
would require exactly the bot-detection evasion this project refuses to
implement. That is also why the other engines get a query and a URL rather
than a backend. The cost of the choice is honest and visible — you bring
your own API key and live within your own quota.

Use it against systems you own or are explicitly authorised to assess, and
follow the terms of service of whichever search provider you configure.

---

## Installation

Python 3.12 or newer, on Windows or Linux.

```bash
pip install dork-client
```

That gives you the `dork-client` command and everything except the ability to
send a request:

```bash
dork-client build --site example.com --filetype pdf
```

### To actually run searches

The `google` backend needs one extra dependency, `httpx`:

```bash
pip install "dork-client[search]"
```

No HTML parser is in that list on purpose: the backend uses the official JSON
API, so there is no results page to scrape.

### Without touching your system Python

[pipx](https://pipx.pypa.io) installs the command into its own environment:

```bash
pipx install "dork-client[search]"
```

Or run it once without installing anything:

```bash
uvx --from "dork-client[search]" dork-client build --site example.com --filetype pdf
```

### From source

For development, or to run against unreleased changes:

```bash
git clone https://github.com/NOKREM/dork-client.git
cd dork-client
python -m venv .venv
```

Activate it — Windows (PowerShell):

```bash
.venv\Scripts\Activate.ps1
```

Linux / macOS:

```bash
source .venv/bin/activate
```

Then install it editable, with the test suite and the linters:

```bash
pip install -e ".[search,dev]"
```

In a checkout the tool also runs straight from the entry point, without being
installed at all:

```bash
python main.py --help
```

---

## Configuration

Copy the sample file and edit it:

```bash
cp .env.example .env
```

### Google API credentials

The `google` backend needs two values, read from the environment:

| Variable | Where it comes from |
| --- | --- |
| `GOOGLE_API_KEY` | An API key from the Google Cloud console with the Custom Search API enabled |
| `GOOGLE_CSE_ID` | The "Search engine ID" of an engine you create at [programmablesearchengine.google.com](https://programmablesearchengine.google.com) |

```bash
export GOOGLE_API_KEY=...        # Linux / macOS
export GOOGLE_CSE_ID=...
```

```powershell
$env:GOOGLE_API_KEY = "..."      # Windows PowerShell
$env:GOOGLE_CSE_ID = "..."
```

The key is never logged and never appears in an error message; the credentials
object prints as `GoogleCredentials(api_key='***', ...)`.

Never commit your `.env`.

### Every setting

`.env`, environment variables and the command line all feed the same object.
Precedence: **command line > environment variable > `.env` > default**.

| Variable | Default | Meaning |
| --- | --- | --- |
| `SEARCH_ENGINE` | `preview` | Backend used when `--engine` is omitted |
| `DEFAULT_RESULT_LIMIT` | `10` | Default for `--limit` (1–100) |
| `REQUEST_TIMEOUT` | `10` | Per-request timeout in seconds |
| `REQUEST_DELAY` | `2.0` | Minimum seconds between two requests |
| `MAX_RETRIES` | `3` | Retries allowed for a transient failure |
| `BACKOFF_FACTOR` | `2.0` | Multiplier per retry, capped at 60 s |
| `USER_AGENT` | `dork-client/<version>` | Sent with every request |
| `GOOGLE_API_KEY` | — | Programmable Search API key |
| `GOOGLE_CSE_ID` | — | Programmable Search engine id |
| `BRAVE_API_KEY` | — | Brave Search API key |
| `DATABASE_PATH` | `data/history.db` | Where the history lives |
| `LOG_PATH` | `logs/app.log` | Where the log is written |
| `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` |

Everything is validated once at start-up, so a typo produces one readable
message naming every offending variable instead of a failure halfway through a
search:

```text
error: invalid configuration: REQUEST_TIMEOUT: Input should be greater than 0
```

### Checking what a run would use

```bash
dork-client config
```

```text
Effective configuration:
  SEARCH_ENGINE         preview
  DEFAULT_RESULT_LIMIT  10
  ...
  GOOGLE_API_KEY        ***
  GOOGLE_CSE_ID         my-search-engine
```

The key is shown as `***`; `--json` gives the same data for scripts.

---

## Logging

Every run appends to `logs/app.log` (a rotating file: 1 MB, three backups):
start-up, query creation, search requests and their status, parsing failures,
database failures, exports and the exit code.

```text
2026-08-09 10:16:08 INFO     dork-client.cli: started: command=search version=0.1.0
2026-08-09 10:16:08 INFO     dork-client.cli: query created: site:example.com filetype:pdf
2026-08-09 10:16:08 INFO     dork-client.search.google: search request: engine=google start=1 num=10 query=site:example.com filetype:pdf
2026-08-09 10:16:09 INFO     dork-client.search.google: parsed 8 result(s)
2026-08-09 10:16:09 INFO     dork-client.database.repository: saved search 3 (8 results, engine google)
2026-08-09 10:16:09 INFO     dork-client.cli: finished: command=search exit=0
```

`--verbose` mirrors the log to the error stream while a command runs.

**Secrets never reach the file.** Call sites already avoid logging the key —
the Google backend never logs its request parameters — and a
`SecretRedactingFilter` is the second line of defence: it rewrites any
configured secret to `***` before a handler formats the record, so even a
careless call site cannot leak it.

```text
INFO dork-client.example: calling https://api.example/search?key=***
```

```bash
python examples/07_config_and_logging.py
```

---

## CLI usage

The examples below use the installed command. In a checkout without an
install, `python main.py <command>` does exactly the same thing.

```bash
dork-client --help
```

| Command | Purpose | Status |
| --- | --- | --- |
| `build` | Compose a query from operators and print it | Available |
| `interactive` | Compose a query by answering questions | Available |
| `operators` | List the supported operators | Available |
| `templates` | List the ready-made templates | Available |
| `template` | Render one template for a domain | Available |
| `search` | Run a query against a backend and filter the results | Available |
| `history` | Past searches from SQLite | Available |
| `export` | Write a stored search as JSON / CSV / TXT | Available |
| `providers` | List the engines a query can be rewritten for | Available |
| `config` | Show the effective configuration | Available |

### `build`

```bash
dork-client build --site example.com --filetype pdf
```

```bash
dork-client build --site example.com --intitle "annual report" --exclude confidential
```

Add `--url` to also print a ready-to-open search URL (nothing is requested), or
`--json` for machine-readable output:

```bash
dork-client build --site example.com --filetype pdf --url
```

```bash
dork-client build --site example.com --filetype pdf --json
```

```json
{
  "query": "site:example.com filetype:pdf",
  "terms": ["site:example.com", "filetype:pdf"],
  "search_url": "https://www.google.com/search?q=site%3Aexample.com+filetype%3Apdf"
}
```

**Terms are rendered in the order you type them**, and every option may be
repeated:

```bash
dork-client build --inurl admin --inurl docs --site example.com
# inurl:admin inurl:docs site:example.com
```

Options: `--site`, `--any-site`, `--intitle`, `--inurl`, `--intext`,
`--filetype`, `--ext`, `--any-filetype`, `--related`, `--cache`, `--keyword`,
`--phrase`, `--any-of`, `--exclude`, `--exclude-site`, `--wildcard`.
The `--any-*` options take comma separated values:

```bash
dork-client build --any-site example.com,example.org --any-filetype pdf,docx
# (site:example.com OR site:example.org) (filetype:pdf OR filetype:docx)
```

### Other search engines

A dork written for Google does not run unchanged anywhere else. `--for`
restates the query in another engine's dialect and points the URL at it:

```bash
dork-client build --site example.com --filetype pdf --intext budget --for yandex --url
```

```text
note: Yandex cannot express intext:budget; dropped from its query
Generated query:
  site:example.com filetype:pdf intext:budget
For Yandex:
  site:example.com mime:pdf
Search URL:
  https://yandex.com/search/?text=site%3Aexample.com+mime%3Apdf
```

Yandex spells `filetype:` as `mime:` and `intitle:` as `title:`; Bing and
Brave use `inbody:` where Google uses `intext:`; Naver, Seznam and Ask.com
handle little beyond `site:`. **When an engine cannot express an operator the
term is dropped and reported**, never rewritten into something that means
something else — a query that quietly searched for the wrong thing would be
worse than one that admits what it lost.

```bash
dork-client providers
```

| Provider | Notes |
| --- | --- |
| `google` | The full operator set |
| `bing`, `ecosia` | `intext:` becomes `inbody:`; no `related:` or `cache:` |
| `brave` | Operators are experimental and may be ignored |
| `duckduckgo` | Operator handling was cut back in 2023 and only partly restored |
| `yandex` | Its own dialect: `mime:`, `title:`, `url:` |
| `yahoo`, `aol` | Basic operators only |
| `baidu` | Strongest on Chinese-language pages |
| `naver`, `seznam`, `ask` | Little beyond `site:` |
| `perplexity`, `chatgpt` | Answer engines: the text arrives as a prompt, not as syntax |

`--for` produces a query and a URL. It sends nothing: only providers with an
official API get a backend that fetches results (see below).

```bash
python examples/08_other_engines.py
```

### `interactive`

```bash
dork-client interactive
```

```text
Leave a field blank to skip it. Press Ctrl-C to abort.
Common file extensions: csv, doc, docx, json, md, odp, ods, odt, ...

Target domain: example.com
Search term:
URL keyword:
Title keyword:
File extension: pdf
Exact phrase: annual report
Exclude term:

Generated query:
  site:example.com filetype:pdf "annual report"
```

Blank fields are skipped, invalid answers are reported and asked again, and
`--url` / `--json` work here too. Prompts and complaints go to **stderr**, so
the composed query is the only thing on stdout and the output stays pipeable:

```bash
dork-client interactive --json 2>/dev/null > query.json
```

### `operators`

```bash
dork-client operators
```

Lists every supported operator with its description; `--json` gives the same
data as a machine-readable list.

### `templates` and `template`

```bash
dork-client templates
```

```text
Public PDF documents:
  public-pdf
      PDF documents a domain has published and search engines indexed.
      site:{site} filetype:pdf
  ...
```

Filter by category, or get the raw records:

```bash
dork-client templates --category "Public subdomains"
```

```bash
dork-client templates --json
```

Render one template for a domain:

```bash
dork-client template public-subdomains --site example.com
```

```text
Template: public-subdomains
Category: Public subdomains
  Indexed subdomains other than www, to review your footprint.

Generated query:
  site:*.example.com -site:www.example.com
```

`--url` and `--json` work here too; the JSON keeps both `template_query` (with
the placeholder) and the rendered `query`.

### `search`

Composes a query exactly like `build`, runs it through a backend and filters
what comes back.

```bash
dork-client search --site example.com --filetype pdf --url
```

```text
note: preview: the preview engine sends no request and returns no results; open the search URL yourself
Engine:  preview
Query:   site:example.com filetype:pdf
Results: 0
URL:     https://www.google.com/search?q=site%3Aexample.com+filetype%3Apdf

(no results)
```

The query can come from operators, from a ready-made string or from a
template:

```bash
dork-client search --query 'site:example.com "annual report"'
```

```bash
dork-client search --template public-subdomains --site example.com
```

Result filters run after the search and can be combined; repeated URLs are
dropped by default:

| Option | Effect |
| --- | --- |
| `--filter-domain DOMAIN` | Keep only this domain and its subdomains (repeatable) |
| `--exclude-domain DOMAIN` | Drop this domain and its subdomains (repeatable) |
| `--filter-url TEXT` | Keep results whose URL contains TEXT |
| `--filter-title TEXT` | Keep results whose title contains TEXT |
| `--filter-ext EXT` | Keep results with this file extension (repeatable) |
| `--keep-duplicates` | Do not drop repeated URLs |

```bash
dork-client search --site example.com --filter-domain example.com --filter-ext pdf
```

Other options: `--limit N` (1–100, default 10), `--engine NAME`, `--json`,
`--url`. With `--json` the output is a `SearchResponse` record plus
`requested_count`, `filters` and `search_url`.

To run a real search, pick the Google backend:

```bash
dork-client search --site example.com --filetype pdf --engine google
```

`--engine` defaults to `preview` so no command spends API quota by accident.
Set `SEARCH_ENGINE=google` in `.env` to change the default.

Every search is recorded in the history unless you pass `--no-save`;
`--database PATH` chooses a different file.

### `history`

```bash
dork-client history
```

```text
ID | QUERY                                    | DATE (UTC)       | RESULTS | ENGINE
-----------------------------------------------------------------------------------
 2 | site:*.example.org -site:www.example.org | 2026-08-09 06:51 |       4 | google
 1 | site:example.com filetype:pdf            | 2026-08-09 06:51 |      12 | google

Showing 2 of 2 searches.
Use 'dork-client history --show ID' to see the stored results.
```

Long queries are shortened in the table; `--json` and `--show` give the full
text. To see what one past search returned:

```bash
dork-client history --show 1
```

Options: `--limit N` (default 20), `--show ID`, `--database PATH`, `--json`.

### `export`

Writes a stored search — the most recent one unless you name another — to a
file:

```bash
dork-client export --format json --output results.json
```

```bash
dork-client export --format csv --output results.csv
```

The format can be left out when the file name says it:

```bash
dork-client export --output results.csv
```

```text
Exported 12 result(s) from history entry 3 to results.csv (csv).
```

An existing file is never replaced silently:

```text
error: results.csv already exists; pass --force to overwrite it
```

Options: `--output PATH` (required), `--format {json,csv,txt}`,
`--search-id ID`, `--force`, `--database PATH`, `--json`.

### Exit codes

| Code | Meaning |
| --- | --- |
| 0 | Success |
| 1 | Unexpected internal error |
| 2 | Malformed command line (argparse) |
| 3 | Input could not become a valid query |
| 130 | Interrupted with Ctrl-C |

---

## Building dorks in Python

`DorkBuilder` is immutable and chainable: every method returns a **new**
builder, so a partially built query can be reused as a template.

```python
from dork_client.dork import DorkBuilder

query = (
    DorkBuilder()
    .site("example.com")
    .filetype("pdf")
    .intitle("report")
    .exclude("confidential")
    .exact_phrase("annual report")
    .build()
)
# site:example.com filetype:pdf intitle:report -"confidential" "annual report"
```

### Supported operators

| Method | Renders | Notes |
| --- | --- | --- |
| `.site("example.com")` | `site:example.com` | Scheme, port and path are stripped; `*.example.com` allowed |
| `.intitle("annual report")` | `intitle:"annual report"` | Multi-word values are quoted automatically |
| `.inurl("login")` | `inurl:login` | |
| `.intext("documentation")` | `intext:documentation` | |
| `.filetype("pdf")` | `filetype:pdf` | A leading dot and the case are normalised |
| `.ext("docx")` | `ext:docx` | Alias of `filetype:` |
| `.related("example.com")` | `related:example.com` | |
| `.cache("https://example.com/x")` | `cache:https://example.com/x` | Deprecated by Google, kept for other engines |
| `.exact_phrase("annual report")` | `"annual report"` | `*` inside the phrase acts as a wildcard |
| `.keyword("report", "public")` | `report public` | Single words only |
| `.exclude("confidential")` | `-"confidential"` | `quoted=False` gives `-confidential` |
| `.exclude_operator("site", "blog.example.com")` | `-site:blog.example.com` | |
| `.any_of("login", "sign in")` | `(login OR "sign in")` | |
| `.any_site("example.com", "example.org")` | `(site:example.com OR site:example.org)` | |
| `.any_filetype("pdf", "docx")` | `(filetype:pdf OR filetype:docx)` | |
| `.wildcard()` | `*` | Standalone placeholder |
| `.operator("site", "example.com")` | `site:example.com` | Operator chosen at runtime, by name |

### Inspecting the result

`build()` returns the query string; `to_query()` returns a `DorkQuery` value
object:

```python
query = DorkBuilder().site("example.com").filetype("pdf").to_query()

query.query          # 'site:example.com filetype:pdf'
query.terms          # ('site:example.com', 'filetype:pdf')
query.to_dict()      # JSON-serialisable
query.as_search_url()  # renders a URL; sends no request
```

Terms are rendered in the order they were added, and identical terms are
collapsed, so `.site("example.com").site("EXAMPLE.com")` yields one `site:`
term.

### Runnable examples

```bash
python examples/01_build_queries.py
```

### Adding a new operator

1. Add one `OperatorSpec` row to `OPERATORS` in `dork_client/dork/operators.py`
   (keyword, value kind, description).
2. Add the matching method to `DorkBuilder` — a one-liner delegating to
   `OperatorTerm.create`.
3. Add one `DorkOption` row to `DORK_OPTIONS` in `dork_client/cli.py`.

That third row is enough for the CLI: the parser, the `build`/`search`
commands and — if you give it a `prompt` — the interactive wizard are all
generated from the same table.

---

## Template system

A template is the four-field record from the specification, so it round-trips
through JSON unchanged:

```json
{
  "name": "public-pdf",
  "description": "PDF documents a domain has published and search engines indexed.",
  "category": "Public PDF documents",
  "query": "site:{site} filetype:pdf"
}
```

**Every built-in template is scoped to one domain**: its query contains the
`{site}` placeholder and rendering fails without a value for it. Templates
therefore answer "what of *this* domain is publicly indexed?" instead of
sweeping the web, and none of them targets credentials, secrets or protected
resources.

### Built-in templates

| Category | Templates |
| --- | --- |
| Public PDF documents | `public-pdf`, `public-reports` |
| Public presentations | `public-presentations`, `public-spreadsheets` |
| Public documentation | `public-documentation`, `public-api-documentation` |
| Public subdomains | `public-subdomains` |
| Public login pages | `public-login-pages` |
| Public error pages | `public-error-pages` |
| Public technology information | `public-technology-information` |
| Public indexed directories | `public-indexed-directories` |
| Public configuration documentation | `public-configuration-documentation` |
| Public security contacts | `public-security-contact` |
| Public contact pages | `public-contact-pages` |
| Public policies | `public-policies` |
| Public status pages | `public-status-pages` |
| Public downloads | `public-downloads` |
| Public non-production hosts | `public-non-production-hosts` |
| Public media | `public-media-assets` |
| Related sites | `similar-sites` |

Twenty templates in all. `dork-client templates` prints them with their
queries; `--category` narrows the list.

A few are worth calling out because they answer different questions:

- **`public-security-contact`** finds the `security.txt` or disclosure policy
  that says how to report a vulnerability to a domain — the first thing to
  look for before reporting anything.
- **`public-non-production-hosts`** looks for indexed staging, dev and test
  hosts. This is the self-audit case: those are usually meant not to be
  indexed at all, and this is how you find out that they are.
- **`similar-sites`** is the one template that is not a footprint check. It
  uses `related:` on its own, as a starting point for finding adjacent sites
  rather than for inspecting one you already know.

### Templates in Python

```python
from dork_client.dork import DorkTemplate, default_registry

registry = default_registry()

registry.get("public-pdf").render(site="example.com").query
# 'site:example.com filetype:pdf'

[t.name for t in registry.by_category("Public documentation")]
# ['public-documentation', 'public-api-documentation']

registry.grouped()      # ((category, (template, ...)), ...)
registry.to_list()      # JSON-serialisable records
```

`render()` validates the domain, fills the placeholders and returns the same
`DorkQuery` object the builder produces — quoted phrases and OR groups stay
intact as single terms.

### Your own templates

Keep them in a JSON file holding a list of those records and pass it with
`--templates-file`; an entry reusing a built-in name replaces that built-in.
See [examples/templates.example.json](examples/templates.example.json).

```bash
dork-client templates --templates-file examples/templates.example.json
```

```python
registry = default_registry("my-templates.json")
```

```bash
python examples/02_templates.py
```

---

## Search backends

`SearchEngine` is the only thing the rest of the application knows about
searching, so a new provider changes nothing above this layer.

```python
from dork_client.search import SearchEngine, RequestPolicy, register_engine

@register_engine
class MyEngine(SearchEngine):
    name = "my-engine"

    def search(self, query: str, limit: int = 10) -> list[SearchResult]:
        self.rate_limiter.wait()
        ...
```

Registering makes the backend selectable as `--engine my-engine`;
`available_engines()` lists what is registered and `create_engine(name)`
instantiates it. `execute(query, limit)` is the concrete wrapper around your
`search()`: it validates the limit, truncates the results and returns a
`SearchResponse`.

### Politeness is part of the contract

Every engine carries a `RequestPolicy` and a matching `RateLimiter`:

| Setting | Default | Meaning |
| --- | --- | --- |
| `delay` | 2.0 s | Minimum interval between two requests |
| `timeout` | 10.0 s | Per-request timeout |
| `max_retries` | 3 | Retries allowed for a transient failure |
| `backoff_factor` | 2.0 | Multiplier per retry, capped at 60 s |
| `user_agent` | `dork-client/<version>` | Identifies the client honestly |

The retry rules exist to survive transient failures, never to work around a
provider's rate limit or any access control. A backend that is told to slow
down must slow down.

### The shipped backends

| Name | What it does |
| --- | --- |
| `preview` | Renders the query and a search URL. No network I/O, no results, no credentials. The default. |
| `google` | Calls the official Programmable Search JSON API with your credentials. |
| `brave` | Calls the official Brave Search API, which runs its own index. Needs `BRAVE_API_KEY`. |

Those are the providers with an official API. The other eleven in
`dork-client providers` can be rewritten for and opened, but not fetched from:
reaching them programmatically would mean scraping their results pages, which
their terms forbid and this project does not do.

`preview` exists so the whole pipeline is usable and testable without
credentials, and so no command spends quota by accident.

### The `google` backend

```bash
dork-client search --site example.com --filetype pdf --engine google --limit 20
```

What it does, and does not, do:

- **One API call per ten results.** The API caps a call at ten items, so
  `--limit 20` costs two calls; the rate limiter spaces them by `delay`. A
  short page ends the paging instead of spending another call on an empty
  answer.
- **Permanent failures are never retried.** Bad credentials (401/403), a
  rejected query (400) and an exhausted quota stop immediately with the API's
  own message.
- **Transient failures are retried** with exponential backoff: timeouts,
  connection errors, 5xx.
- **429 is obeyed, not circumvented.** The client waits at least as long as
  the provider's `Retry-After` header asks, and once the retry budget is spent
  it stops and tells you to slow down or raise your quota.
- **A single malformed result item is skipped**, so one odd entry does not
  fail the whole search.
- **The API key never leaves the request.** Error messages carry only the
  status and the API's structured message, never the parameters.

```bash
python examples/04_google_backend.py    # drives the backend with a fake client
```

### Adding another provider

Subclass `SearchEngine`, register it, and it is selectable as `--engine`:
nothing above the search layer changes. Prefer an official API; if a provider
only offers HTML, that is a reason to skip it, not a reason to scrape.

### Result model

Every backend produces the record from the specification:

```python
SearchResult(
    title="Annual report 2024",
    url="https://example.com/reports/annual-2024.pdf",
    display_url="example.com/reports/annual-2024.pdf",  # derived when omitted
    snippet="...",
    source="preview",       # engine name
    query="site:example.com filetype:pdf",
    timestamp=...,          # aware UTC, defaults to now
)
```

`to_dict()` / `from_dict()` round-trip it through JSON, and `.domain` and
`.extension` are read from the URL. A `SearchResponse` groups the results of
one query with its engine, timestamp and count — the object the history and
the exporters persist.

### Filtering in Python

```python
from dork_client.search import build_filter_chain

chain = build_filter_chain(
    domains=["example.com"],
    exclude_domains=["blog.example.com"],
    url="/reports/",
    title="annual",
    extensions=["pdf"],
    unique=True,
)
chain.apply(response.results)
chain.description   # 'on domain example.com; ...; unique URLs'
```

Each filter is a small object with one `apply` method, so they compose in the
order you list them.

```bash
python examples/03_search_backends.py
```

---

## Search history

Every search is stored in SQLite at `data/history.db` (override with
`--database`, or skip recording with `--no-save`).

### Schema

`searches` holds exactly the columns the specification asks for:

| Column | Type | Notes |
| --- | --- | --- |
| `id` | INTEGER | Primary key |
| `query` | TEXT | The query as it was run |
| `created_at` | TEXT | ISO-8601 UTC, sorts correctly as text |
| `result_count` | INTEGER | How many results were **kept** after filtering |
| `search_engine` | TEXT | Backend name |

A second table, `results`, keeps the results of each search so a past search
can be re-read and exported. `PRAGMA user_version` records
the schema version so a future migration knows what it is looking at.

What is stored is what you saw: filters run before the save, so a search that
returned 12 results and kept 4 is recorded as 4.

### The history in Python

```python
from dork_client.database import HistoryRepository

with HistoryRepository("data/history.db") as history:
    record = history.save(response)      # a SearchResponse
    history.recent(20)                   # newest first
    history.get(record.id)               # one entry
    history.results_for(record.id)       # its stored results
    history.latest()                     # or None when empty
    history.count()
```

Every `sqlite3` failure becomes a `DatabaseError` with a readable message, so
callers never see a driver exception.

```bash
python examples/05_history.py
```

To delete the history, delete the database file — the tool has no command that
destroys stored data.

---

## Export formats

All three write UTF-8 with LF line endings, so a file exported on Windows and
one exported on Linux are byte-for-byte identical.

**JSON** — the whole response as one object, and `SearchResponse.from_dict()`
reads it straight back:

```json
{
  "query": "site:example.com filetype:pdf",
  "engine": "google",
  "created_at": "2024-05-01T12:30:00+00:00",
  "result_count": 2,
  "results": [ { "title": "...", "url": "...", "...": "..." } ]
}
```

**CSV** — a header line and one row per result, with the seven result fields
(`title, url, display_url, snippet, source, query, timestamp`). Commas and
quotes inside a title are escaped by the `csv` module, so the file survives a
round trip through a spreadsheet.

**TXT** — a readable report: the query, engine, date and count, then the
numbered results.

### Exporting in Python

```python
from dork_client.export import create_exporter, exporter_for_path

create_exporter("csv").write(response, "results.csv")
exporter_for_path("results.json").render(response)   # returns a string
```

Adding a format means writing one class with a `render()` method and
registering it with `@register_exporter`; `--format` picks it up
automatically.

```bash
python examples/06_export.py
```

---

## Validation

Invalid input raises `DorkValidationError`, which carries the offending field
name so the CLI can point at the right argument:

```text
error: [site] 'not a domain' is not a valid domain name (expected e.g. 'example.com')
```

Checked at input time: empty values, invalid domains, invalid file extensions,
unknown operators, control characters or line breaks, embedded double quotes,
and values longer than 256 characters. Checked at `build()` time: empty
queries, queries over 2048 characters, and queries over 32 words (search
engines silently drop the remainder).

---

## Error handling

All application errors derive from `GoogleDorkError`, so the CLI can translate
any failure with a single handler:

| Exception | Raised when |
| --- | --- |
| `DorkValidationError` | User input cannot become a valid query |
| `SearchEngineError` | A search backend fails permanently |
| `TransientSearchError` | A backend failure worth retrying (timeout, 5xx) |
| `RateLimitError` | The provider asked the client to slow down (429) |
| `ConfigurationError` | Configuration or credentials are missing or invalid |
| `DatabaseError` | A persistence operation fails |
| `ExportError` | Results cannot be written to disk |

---

## Project layout

```text
dork-client/
├── main.py                 # thin entry point, forwards to dork_client.cli
├── pyproject.toml
├── README.md
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── .env.example
├── .gitignore
│
├── dork_client/
│   ├── __init__.py
│   ├── py.typed             # the package ships its type hints
│   ├── cli.py              # option table, parser, command handlers
│   ├── config.py           # Settings, loaded from .env and the environment
│   ├── models.py           # SearchResult, SearchResponse
│   ├── exceptions.py       # shared exception hierarchy
│   ├── dork/
│   │   ├── builder.py      # DorkBuilder, DorkQuery
│   │   ├── operators.py    # operator registry and term types
│   │   └── templates.py    # DorkTemplate, TemplateRegistry
│   ├── search/
│   │   ├── base.py         # SearchEngine ABC, RequestPolicy, retries
│   │   ├── transport.py    # HTTP client seam, no httpx at import time
│   │   ├── preview.py      # the network-free backend
│   │   ├── google.py       # Programmable Search JSON API backend
│   │   └── filters.py      # result filters
│   ├── database/
│   │   └── repository.py   # HistoryRepository, schema
│   ├── export/
│   │   └── exporter.py     # JSON / CSV / TXT exporters
│   └── utils/
│       ├── validators.py   # domain / extension / text / query validation
│       └── logging.py      # rotating log file, secret redaction
│
├── .github/workflows/ci.yml
├── data/                   # SQLite database (git-ignored)
├── logs/                   # application log (git-ignored)
├── examples/               # eight runnable scripts, none of them networked
└── tests/
```

### Where files are written

`DATABASE_PATH` and `LOG_PATH` default to **relative** paths, so the history
and the log land under the directory you run the command from. Set absolute
paths in `.env` if you want one history wherever you are:

```bash
DATABASE_PATH=/home/you/.local/share/dork-client/history.db
LOG_PATH=/home/you/.local/state/dork-client/app.log
```

---

## Running the tests

```bash
pip install -e ".[search,dev]"
```

```bash
pytest
```

**743 tests, 100% statement and branch coverage**, in about nine seconds.
The threshold is enforced, so a new branch without a test fails the run:

```bash
pytest --cov
```

Static checks:

```bash
ruff check . && mypy
```

### What the tests do not do

- **No test opens a socket.** The backends take an injected HTTP client and
  the tests supply a fake one, so the suite is deterministic and costs no API
  quota.
- **No test waits.** The rate limiter and the retry backoff take an injected
  sleep function.
- **No test touches your machine.** An autouse fixture gives every test its
  own configuration, database and log file under `tmp_path`, clears the
  configuration environment variables, and ignores any `.env` in the
  checkout — so a run can neither read your credentials nor write into `data/`
  or `logs/`.

### Layout

| File | Covers | Tests |
| --- | --- | --- |
| `test_builder.py` | Operators, term rendering, query validation | 53 |
| `test_validators.py` | Domain, extension, text and URL validation | 40 |
| `test_templates.py` | Template records, rendering, the registry | 85 |
| `test_models.py` | `SearchResult`, `SearchResponse`, timestamps | 52 |
| `test_search.py` | The `SearchEngine` contract, policy, rate limiting | 45 |
| `test_filters.py` | Every result filter and the chain | 28 |
| `test_google.py` | The Google backend: parsing, paging, errors, retries | 46 |
| `test_brave.py` | The Brave backend: header auth, parsing, errors | 51 |
| `test_providers.py` | Operator dialects, dropped terms, provider URLs | 61 |
| `test_repository.py` | The SQLite schema and round trips | 26 |
| `test_exporter.py` | JSON, CSV and TXT output and file writing | 40 |
| `test_config.py` | Settings, precedence, validation, secrets | 39 |
| `test_logging.py` | Log setup, levels, secret redaction | 27 |
| `test_cli.py` | Every command, end to end | 110 |
| `test_edge_cases.py` | Fallbacks and error branches the above skip | 40 |

### The tests the specification asks for

| Required test | Where |
| --- | --- |
| site operator | `test_builder.py::test_site_operator` |
| filetype operator | `test_builder.py::test_filetype_operator` |
| intitle operator | `test_builder.py::test_intitle_operator_quotes_multiword_values` |
| inurl operator | `test_builder.py::test_inurl_operator` |
| exact phrase | `test_builder.py::test_exact_phrase_is_always_quoted` |
| exclude operator | `test_builder.py::test_exclude_operator` |
| multiple operators | `test_builder.py::test_multiple_operators_keep_insertion_order` |
| empty query | `test_builder.py::test_empty_query_raises` |
| invalid operator | `test_builder.py::test_unknown_operator_raises` |
| duplicate result | `test_filters.py::test_unique_filter_removes_repeated_urls` |
| JSON export | `test_exporter.py::test_json_export_round_trips` |
| CSV export | `test_exporter.py::test_csv_export_writes_one_row_per_result` |
| SQLite repository | `test_repository.py::test_save_stores_the_specified_columns` |

### Continuous integration

[`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs the linter, the
type checker and the suite on Ubuntu and Windows with Python 3.12 and 3.13 for
every push and pull request.

---

## Development

### Building a distribution

```bash
pip install build
```

```bash
python -m build
```

That produces a wheel and an sdist in `dist/`. The wheel contains only the
`dork_client` package: `main.py` stays in the checkout rather than being installed as
a top-level module, where it would collide with any other project shipping the
same name. The console script points at `dork_client.cli:main`, so an installed copy
runs as `dork-client` from anywhere.

### Adding to the tool

Each extension point is one row or one class:

| To add | Do this |
| --- | --- |
| A search operator | One `OperatorSpec` row, one `DorkBuilder` method, one `DorkOption` row |
| A template | One `DorkTemplate`, or a record in a `--templates-file` |
| A search backend | Subclass `SearchEngine`, decorate with `@register_engine` |
| An export format | Subclass `Exporter`, decorate with `@register_exporter` |
| A result filter | Subclass `ResultFilter`, add it to `build_filter_chain` |

The CLI reads those registries, so a new entry appears in `--help`, in the
`--engine` and `--format` choices and in the interactive wizard without any
further wiring.

### House rules

- `ruff` and `mypy --strict` must pass.
- Coverage is enforced at 100%; a new branch needs a test.
- No test may open a socket, sleep on a real clock, or write outside
  `tmp_path`.

---

## Troubleshooting

**`ModuleNotFoundError: No module named 'dork_client'`**
You are running from a checkout without installing it. Either
`pip install -e .`, or run `python main.py` from the repository root. The
example scripts add the root to `sys.path` themselves.

**`pytest` is not recognised / uses the wrong Python**
The virtual environment is probably not active. Use `python -m pytest`, or on
Windows call the interpreter directly: `.venv\Scripts\python.exe -m pytest`.

**PowerShell refuses to run the activation script**
Allow local scripts for the current user:
`Set-ExecutionPolicy -Scope CurrentUser RemoteSigned`.

**"query is too long" / "too many words"**
Search engines ignore terms past roughly 32 words. Split the query into several
narrower ones.

**"template 'x' needs --site"**
Every built-in template is scoped to a domain on purpose. Pass the domain you
own or are authorised to assess: `dork-client template x --site example.com`.

**`search` always reports 0 results**
`--engine` defaults to `preview`, which sends no request by design. Open the
URL it prints, or pass `--engine google` with credentials configured.

**"set GOOGLE_API_KEY and GOOGLE_CSE_ID to use the google backend"**
Put both in `.env` or export them in your shell, then check with
`dork-client config`.

**"the search API refused the credentials (403)"**
Either the key is wrong, the Custom Search API is not enabled for that Cloud
project, or the daily quota is spent. The API's own message is included.

**"the search API asked the client to slow down"**
You hit the provider's rate limit. The client already waited as long as it was
asked to and then stopped. Raise `REQUEST_DELAY`, lower `--limit`, or request
a higher quota — do not work around the limit.

**"this backend needs httpx"**
Install the extra: `pip install -e ".[search]"`.

**"invalid configuration: ..."**
A value in `.env` or the environment is out of range or misspelt. The
message names every offending variable; `dork-client config` shows what is
in effect.

**A setting in `.env` seems to be ignored**
An environment variable of the same name wins over the file, and a command
line option wins over both. Check with `dork-client config`.

**"cannot open the log file ..."**
The path in `LOG_PATH` is not writable. Point it somewhere else.

**"no history entry with id N"**
Run `dork-client history` to see which ids exist. Ids are per database, so
check you are pointing `--database` at the right file.

**The history is in the wrong place / I want to start over**
Pass `--database PATH`, or delete `data/history.db`. Nothing else in the tool
removes stored data.

**"results.csv already exists; pass --force to overwrite it"**
Exports never replace a file silently. Add `--force`, or choose another name.

**"cannot tell the format from 'results.dat'"**
The suffix is not one of `.json`, `.csv`, `.txt`. Pass `--format` explicitly.

**"the history is empty; run a search before exporting"**
`export` reads from the history. Run a `search` first, or point `--database`
at the file that holds your searches.

**A quoted value is rejected**
Values must not contain a literal `"`. Use `.exact_phrase("annual report")`
instead of embedding quotes in an operator value.

**`cache:` returns nothing**
Google retired the operator. Keep using it only with engines that still
support it.

---

## License

MIT — see [LICENSE](LICENSE).
