Metadata-Version: 2.4
Name: tellaro-query-language
Version: 2.0.0
Summary: A flexible, human-friendly query language for searching and filtering structured data
License: Proprietary
License-File: LICENSE
Keywords: query,language,opensearch,elasticsearch,search,filter,tql
Author: Justin Henderson
Author-email: justin@tellaro.io
Requires-Python: >=3.11,<3.14
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Provides-Extra: opensearch
Requires-Dist: dnspython (>=2.8.0,<3.0.0)
Requires-Dist: maxminddb (>=3.0.0,<4.0.0)
Requires-Dist: opensearch-dsl (>=2.1.0,<3.0.0) ; extra == "opensearch"
Requires-Dist: opensearch-py (>=2.8.0,<3.0.0) ; extra == "opensearch"
Requires-Dist: pyparsing (>=3.3.0,<4.0.0)
Requires-Dist: urllib3 (>=2.6.0,<3.0.0)
Project-URL: Documentation, https://github.com/tellaro/tellaro-query-language/tree/main/docs
Project-URL: Homepage, https://github.com/tellaro/tellaro-query-language
Project-URL: Repository, https://github.com/tellaro/tellaro-query-language
Description-Content-Type: text/markdown

---
title: Tellaro Query Language (TQL)
class: repo-spec
audience: in-repo
status: current
owner: tql-team
last_verified: '2026-09-04'
verification_note: 'CONTRADICTED on SEMANTICS. The 2026-09-04 pass corrected twelve TQL literals
  that did not parse and recorded that "PARSING is not EVALUATING" -- then still missed three
  breaking changes that leave every query parsing exactly as before, which is the failure that
  warning describes. Corrected here: (1) `in` / `in_cs` becoming element-wise `eq_ci` / `eq` was
  filed under the heading "Also user-visible, NOT BREAKING" and described only as
  case-insensitivity, when its own commit says `f in [''5'']` no longer matches a stored "05" in
  Python -- it is now numbered section 8; (2) the `matches`/`regexp` half of the scalar-operand
  fix (a non-string operand is now a translated pattern where it was raw JSON) appeared
  nowhere -- now section 9; the "35 of 216 operator x literal combinations" this note used to
  quote was removed in round 4 as unreproducible, since 18 x 6 is 108 and the pinned matrix in
  tests/unit/test_scalar_operand_parity.py is 8 x 6; (3) the
  operator-less `field | <predicate>` form, which used to parse to `exists` and match every
  record that merely HAD the field -- now section 10. The intro said "six operator families"
  above SEVEN numbered sections; it now says the count IS the section count, so
  the next addition cannot silently drift it. A fourth false claim was found and fixed: the Type
  Hints parity warning said a hint and a mutator chain "are written in opposite orders in the two
  runtimes, and neither accepts the other''s spelling" -- closed by cea6024 on this same branch.
  Executed, in BOTH engines, rather than read: the in/eq matrix over
  [{f:5},{f:"5"},{f:"05"},{f:5.0}] for `in [''5'']`, `in [5]`, `in_cs [''5'']`, `eq ''5''` and
  `eq 5` (all five agree across engines, and reproduce the commit''s recorded before/after); the
  emitted DSL for `matches`/`regexp` over boolean and numeric operands; `ip | is_loopback`,
  `ip | is_private` and the `ip | lowercase` projection control; both hint/mutator orders on
  three mutator+hint shapes; empty-paren mutators (accepted by both, EXCEPT geoip_lookup/geo,
  which is now documented); and the null/exists matrix that section 1 rests on.
  Every TQL-shaped literal in a fenced block was re-extracted and re-parsed: 248 of 255 parse
  across the nine docs touched, and all seven that do not are deliberate -- five are the
  "WRONG" examples in TROUBLESHOOTING.md and two are Rust-only collection spellings on the
  crates.io page, each already carrying a parity warning and each verified to run under the Rust
  CLI.
  Round 3 of the doc audit moved the Rust-only pointer up by one; nothing else on this
  page changed, because the change behind it (a non-list operand to `in`) is Rust-only and its
  section lives on the crates.io page. Round 4 added a shared section for the `top`/`bottom`
  count Python refused inconsistently (ef65d17), which is what moved the shared count.
  NO COUNT IS RESTATED IN THIS NOTE, deliberately: every number on this page is derived from
  the headings by tests/unit/test_breaking_change_counts.py, and a count typed here is a fifth
  copy of the figure that test exists to compute -- this note carried a stale "ten" for exactly
  that reason. Round 4''s claims were re-measured by EXECUTION at ef65d17^ and at HEAD rather
  than read off the commit message: `validate()`, `to_opensearch()` and `query()` were run over
  five groups at both commits, reproducing the emitted
  {"terms": {"size": 99999999999999999999}} that passed validate(), and the four buckets
  returned with health_status "green" for `stats count() by g top -1`.
  Not checked here: the stats/aggregation surface, the cache and pagination examples, and the
  performance figures (already marked unsubstantiated).'
verified_by: tql-docs-parity-campaign@2026-09-04
source_refs:
  - tellaro-query-language:src/tql/__init__.py
  - tellaro-query-language:pyproject.toml
  - tellaro-query-language:src/tql/mutators/__init__.py
  - tellaro-query-language:src/tql/parser_components/grammar.py
  - tellaro-query-language:tql/src/parser/grammar.pest
  - tellaro-query-language:src/tql/evaluator_components/field_access.py
  - tellaro-query-language:src/tql/opensearch_components/query_converter.py
  - tellaro-query-language:src/tql/opensearch_components/field_mapping.py
---

# Tellaro Query Language (TQL)

[![PyPI version](https://badge.fury.io/py/tellaro-query-language.svg)](https://badge.fury.io/py/tellaro-query-language)
[![Tests Status](./badges/test-badge.svg?dummy=8484744)](./reports/pytest/junit.xml)
[![Coverage Status](./badges/coverage-badge.svg?dummy=8484744)](./reports/coverage/index.html)
[![Python 3.11-3.13](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org/)
[![License: Source Available](https://img.shields.io/badge/License-Source%20Available-blue.svg)](LICENSE)

**A flexible, human-friendly query language for searching and filtering structured data across files, databases, and search engines.**

TQL provides a unified, readable syntax for expressing complex queries that works seamlessly with:
- **Files**: Query JSON, JSONL, CSV files directly with CLI or Python API
- **OpenSearch/Elasticsearch**: Convert TQL to DSL queries automatically
- **In-Memory Data**: Filter Python dictionaries and lists
- **Statistical Analysis**: Built-in aggregations and grouping

```python
# Query JSON files directly
results = tql.query("logs.jsonl", "status = 200 AND response_time > 500")

# Query OpenSearch with automatic DSL translation
results = tql.execute_opensearch(client, "events-*",
    "user.role = 'admin' AND timestamp > '2024-01-01'")

# Aggregate data with stats
results = tql.query("sales.json", "region = 'west' | stats sum(revenue) by product")
```

---

## 🚀 Quick Start

### Installation

```bash
# Install from PyPI (Python package)
pip install tellaro-query-language

# Install with OpenSearch support
pip install tellaro-query-language[opensearch]

# Or install Rust CLI (300x faster for large files)
cargo install tellaro-query-language
```

### Query Files with CLI

TQL includes a blazing-fast command-line interface for querying files:

```bash
# Query JSON/JSONL files
tql 'status = "active"' users.json
tql 'age > 25 AND city = "NYC"' data.jsonl

# Query CSV files (auto-detects headers)
tql 'price > 100 AND category = "electronics"' products.csv

# Statistical aggregations
tql '| stats count() by status' events.jsonl
tql 'status = 200 | stats average(response_time) by endpoint' logs.jsonl

# Process folders recursively
tql 'level = "ERROR"' logs/ --pattern "*.jsonl" --recursive

# Pipe data from stdin
cat data.jsonl | tql 'score > 90'
```

**Performance**: The Rust CLI processes 50MB files in milliseconds vs. seconds for Python implementations.

### Query Files with Python API

```python
from tql import TQL

tql = TQL()

# Query JSON files directly
results = tql.query("data.json", "user.role = 'admin' AND status = 'active'")

# Query with field transformations
results = tql.query("logs.jsonl", "email | lowercase contains '@example.com'")

# Statistical analysis
results = tql.query("sales.json", "| stats sum(revenue), avg(price) by category")
```

### Query In-Memory Data

```python
from tql import TQL

tql = TQL()
data = [
    {'name': 'Alice', 'age': 30, 'city': 'NYC'},
    {'name': 'Bob', 'age': 25, 'city': 'LA'},
    {'name': 'Charlie', 'age': 35, 'city': 'NYC'}
]

# Simple queries. `query()` returns a result DICT, not a list -- the matching
# records are under 'results', alongside 'total' and the query health report.
results = tql.query(data, 'age > 27')
# Returns: {'total': 2, 'post_processing_applied': False, 'health_status': 'green',
#           'health_reasons': [],
#           'results': [{'name': 'Alice', 'age': 30, 'city': 'NYC'},
#                       {'name': 'Charlie', 'age': 35, 'city': 'NYC'}]}
for record in results['results']:
    ...

# Logical operators
results = tql.query(data, 'age >= 30 AND city = "NYC"')
# results['results'] -> [{'name': 'Alice', ...}, {'name': 'Charlie', ...}]

# Field transformations
results = tql.query(data, 'name | lowercase = "alice"')
```

### Query OpenSearch

```python
from opensearchpy import OpenSearch
from tql import TQL

# Initialize OpenSearch client
client = OpenSearch(
    hosts=['localhost:9200'],
    http_auth=('admin', 'admin'),
    use_ssl=True,
    verify_certs=False
)

# Initialize TQL with field mappings
mappings = {
    'user.name': {'type': 'keyword'},
    'user.email': {'type': 'text'},
    'timestamp': {'type': 'date'}
}
tql = TQL(mappings)

# Execute queries with automatic DSL translation
results = tql.execute_opensearch(
    opensearch_client=client,
    index='users-*',
    query='user.name = "admin" AND status = "active"'
)

# Complex queries with mutators and post-processing
results = tql.execute_opensearch(
    opensearch_client=client,
    index='logs-*',
    query='email | lowercase contains "@example.com" AND level = "ERROR"'
)
# TQL automatically applies post-processing for mutators
```

---

## ⚠️ Upgrading: breaking changes in this release

This release makes **eleven** breaking changes to what a query means. Each change is
deliberate, applied identically in the Python, Rust and OpenSearch backends, and
motivated by the same failure: a query that returned zero hits with no error,
which in a detection rule is indistinguishable from "nothing happened".

Read this section before upgrading a saved query, a detection rule, or anything
that builds TQL from a template.

> **The count is the number of numbered sections below — nothing else, and it is
> now asserted rather than asked for.** It said "six operator families" while
> seven sections followed it, because a family was added without the sentence
> being re-counted; the recount that fixed that asked, in prose, that every new
> breaking change get a new numbered section — and the next commit added one
> without recounting anyway. `tests/unit/test_breaking_change_counts.py` now
> derives both numbers here from the headings themselves. Seven further
> breaking changes affect the **Rust runtime only** and are not counted here;
> they are in [`tql/README.md`](tql/README.md), the crates.io page.

### 1. `is null` and `not exists` now mean HAS NO VALUE

**What changed.** `is null` and `not exists` both now mean *absent **or**
explicitly null*. `is not null` and `exists` are their exact complement. The
in-memory evaluators previously implemented a three-state model in which
`is null` meant only "present and null" and `not exists` meant only "absent
key" — so a record whose field was absent satisfied neither `f is null` nor
`f is not null`, and a record whose field was null satisfied neither
`f exists` nor `f not exists`.

**Why.** No backend could ever honour the three-state model. OpenSearch does not
index JSON nulls, so `must_not: {exists: {field}}` — which both DSL backends
already emitted for *both* spellings — cannot tell an absent field from a null
one. The evaluators were answering a question their own translation cannot ask.

**What to check.** Any saved query using `is null`, `is not null` or
`not exists` over a field that is sometimes absent. `exists` is unchanged.

**What to write instead.** Nothing, in most cases — the new answer is the one
the OpenSearch path was already giving. If you specifically wanted "present and
explicitly null", write `f eq null`, which selects only the explicitly-null
records (`f is null` selects those *and* the absent ones).

See the table under [Null and existence semantics](#null-and-existence-semantics)
for the full matrix, including the `{"f": []}` case (an empty array is a
**present** value) and the one residual in-memory-vs-cluster divergence.

### 2. `matches` / `regexp` searches unless you anchor it

**What changed.** Against OpenSearch, `f matches 'abc'` and `f matches '^abc$'`
emitted **identical** DSL, and both meant "the whole field value is exactly
abc" — while the in-memory evaluator used `re.search` and matched `abc`
anywhere. The same query meant two different things depending on where it ran.
An unanchored pattern is now wrapped in `.*` so it searches; `^` and `$` are
still translated rather than wrapped, so a deliberately anchored pattern keeps
full-match semantics.

**What to check.** Any OpenSearch query with an **unanchored** pattern. It will
now match strictly more documents than it did.

**What to write instead.** If you relied on the old implicit anchoring, add the
anchors: `f matches '^abc$'`.

### 3. A LIST operand to `any` / `none` is refused

**What changed.** `f any ['a','b']`, `f none ['a','b']` and every spelling of
them now raise `TQLTypeError` instead of answering — on translation **and in the
in-memory evaluator**. The **scalar** form is unchanged, and a one-element list
still unwraps to it.

**Why.** `f any ['a','b']` does not ask "is any element of `f` one of these two
values" — it asks whether a single element of `f` equals the two-element list,
which no scalar element can. The clause previously produced either an HTTP 400
that failed the *whole* search, or a silent `match_none`, which made
`f none [...]` an exclusion clause that excluded nothing.

The evaluator was raised to the translator in a second step, once the two were
measured selecting different **record sets** rather than different kinds of the
same answer. A translator refusal is not local to the clause — it fails the
whole query — so `code eq 1 OR f any ['a','b']` returned rows in memory and
nothing through OpenSearch, and `f none ['a','b']` returned *every* record in
memory and nothing through OpenSearch. The refusal is decided from the query
alone, never from the data, so short-circuiting `and` / `or` cannot make a
well-typed query out of an ill-typed one for some corpora and not others.

**What to write instead.** `f in ['a','b']` — which is what the query meant.
The error message names it.

`all` / `not_all` are deliberately exempt: they are answered by a script query
where a list is a valid (never-matching) operand.

### 4. Case-sensitive operators refuse a field that cannot answer them

**What changed.** `contains_cs`, `startswith_cs`, `endswith_cs`, `in_cs` and
their `not_` forms now raise `TQLTypeError` against an **analyzed `text` field
with no case-preserving subfield**, instead of silently matching nothing.

**Why.** The analyzer lowercased every token it indexed, so the case the query
asks about is not in the index at all. Measured on OpenSearch 2.19.4 against an
analyzed field holding `"A MiXeD Value"`: `wildcard *MiXeD*` returned 0 hits,
`must_not(wildcard *MiXeD*)` returned **every** document, and the control
`wildcard *mixed*` returned every document. The negated direction is the
dangerous one — an exclusion clause that excludes nothing.

**What to check.** Any `_cs` query against a `text`-mapped field.

**What to write instead.** Either add a `.keyword` subfield to the mapping, or
use the case-insensitive operator (`contains`, `in`, …). The error names both
remedies. A `.keyword` or `wildcard` subfield, a `keyword` base field, and
non-string types (`long`, `ip`, `date`, `boolean`) all still translate, as does
an unmapped field.

### 5. A type hint that cannot read a value skips the record, not the query

*(Migration note reproduced verbatim from the commit that made the change.)*

> A query that previously ERRORED on a heterogeneous field now returns results.
> Nothing that previously returned results changes -- with one exception, and it is
> the one to check: a query using a NEGATED comparator with a type hint over a
> field whose values do not all read as the hinted type. Those records used to
> abort the query; they now answer false rather than being swept in. If any saved
> query relied on the abort as a data-quality alarm, that alarm is gone.
>
> An author who wanted the strict behaviour should assert the shape explicitly
> rather than leaning on the hint's failure mode -- `f::int exists and f::int > 75`
> selects records readable as an int, and a separate rule over the un-hinted field
> finds the ones that are not. There is no flag to restore the raise, deliberately:
> two evaluators with two modes is how the engines drifted before.

An unknown hint **name** is still a hard error; only an unreadable *value* is a
skip. See [Type Hints](#type-hints) for the boundaries, including what a skip
means under negation and under the existence operators.

### 6. `::string` renders a boolean lowercase

**What changed.** `flag::string eq 'true'` over `{"flag": true}` previously
matched nothing, because the hint applied Python's `str()` and `str(True)` is
`"True"`. It now matches; `flag::string eq 'True'` no longer does.

**Why.** JSON writes `true`/`false`, TQL's own boolean literals are
`true`/`false`, and OpenSearch stores `true`/`false`. The capitalised spelling
was the only one in the stack, in the one place a value has been explicitly cast
to a string in order to be compared as text.

**What to check.** Any query comparing a boolean field cast with `::string` or
`::str` against the literal `'True'` or `'False'`.

**Scope.** Only a **top-level** boolean, and only under `string`/`str`. A
boolean nested in a list or object still renders `True`.

### 7. The string operators render a boolean lowercase too

**What changed.** `f contains_cs true`, `startswith_cs`, `endswith_cs`,
`matches` / `regexp` and their `not_` forms compare against `true` / `false`
rather than `True` / `False` — on **both sides**, the operand written in the
query and the value read out of the record. `f contains_cs true` over
`{"f": "x true y"}` now matches and over `{"f": "x True y"}` no longer does;
`f contains_cs 'ru'` over `{"f": true}` now matches.

**Why.** The evaluator and the OpenSearch translator were selecting **opposite
documents** for the same query. Measured over
`[{"f": "x true y"}, {"f": "x True y"}]`:

| | selects |
|---|---|
| evaluator (both engines) | the `"x True y"` record |
| translator → `wildcard *true*` | the `"x true y"` document |

Two execution paths, different answers, no error on either. Section 6 moved
`::string` to the lowercase spelling and a separate change moved the translator;
this is the remaining half. OpenSearch settles it outright for a boolean-mapped
field — `term b: "True"` is **rejected** (`only [true] or [false] are allowed`)
while `term b: "true"` matches.

**What to check.** Any query comparing a boolean field, or a text field holding
the word, against the literal `true` / `false` with a **case-sensitive** string
operator or with `matches` / `regexp`. The case-**insensitive** forms
(`contains`, `startswith`, `endswith`) lowercase both sides and are unaffected.

**Also fixed by the same change.** In Python the positive and negated spellings
disagreed with each other — `f contains_cs true` rendered `True` while
`f not contains_cs true` rendered `true`, because the negated spelling arrives
from the parser one list-level deeper and dodged the boolean coercion. Both now
render `true`.

### 8. `in` / `in_cs` are element-wise `eq_ci` / `eq`

**What changed.** `f in [a, b]` **is** `f eq_ci a or f eq_ci b`, and `f in_cs` is
the `eq` twin. Both engines re-implemented equality inside the membership arm
instead of delegating to it, and drifted from it in **opposite** directions. They
now delegate, so `in` answers exactly what the equivalent chain of `eq_ci` /
`eq` clauses answers.

`in` and `not_in` remain the case-**insensitive** forms; `in_cs` / `not_in_cs`
are the case-sensitive ones. That part is unchanged.

**What to check.** Any `in` / `not_in` / `in_cs` / `not_in_cs` query whose list
holds a **numeric-looking** value, quoted or unquoted, over a field that stores
numbers as text (or the reverse). Nothing else moves. Measured over
`[{"f": 5}, {"f": "5"}, {"f": "05"}, {"f": 5.0}]`:

| Query | Before | Now (both engines) |
|---|---|---|
| `f in ['5']` | Python also matched `"05"` | `5`, `"5"`, `5.0` |
| `f in [5]` | Rust did **not** match `"05"` | `5`, `"5"`, `"05"`, `5.0` |

So this is breaking, narrowly and in both directions: `f in ['5']` no longer
matches a stored `"05"` in Python, and `f in [5]` now does match it in Rust.

**Why.** Quoting is the author's type declaration, and `eq` has honoured it in
both engines for some time; `in` is the operator that never got the rule. Python
coerced every element numerically *regardless of quoting* — a list operand can
never be a quoted literal internally, so the guard that preserves the
distinction was always off for these operators. Rust went the other way and
compared a string field against pre-lowercased element *text*, so `f in [5]` was
false on the stored `"05"` while `f eq_ci 5` — the same question with one
element — was true.

**What to write instead.** Nothing, if the operand's quoting already says what
you meant. If you want the numeric reading, write the element unquoted (`[5]`);
if you want the text reading, quote it (`['5']`).

**The OpenSearch half is inert.** A `terms` clause is resolved by the *field's*
mapping, not by the literal's JSON type — measured on OpenSearch 2.19.4, a
`long` field holding `5` is selected by `["5"]`, `[5]`, `["05"]` and `["5.0"]`
alike. Only the in-memory evaluators, which have no mapping to consult, could
select different records.

### 9. `matches` / `regexp` translate a non-string operand into a pattern

**What changed.** `f matches true` and `f regexp 5` passed the operand straight
through to OpenSearch as raw JSON — `{"regexp": {"f": true}}`. They now render it
as a translated pattern, exactly as a string operand is:
`{"regexp": {"f": {"value": ".*true.*", "flags": "NONE"}}}`.

**Why.** The reasoning for passing it through was that OpenSearch would report
the type error. It does not. Measured on OpenSearch 2.19.4 against a field
holding the word:

```text
{"regexp": {"f": true}}        -> 0 hits, NO ERROR
{"regexp": {"f": ".*true.*"}}  -> 1 hit
```

The boolean is coerced to the string `"true"` and Lucene's regexp engine anchors
implicitly, so the pattern must equal the *whole* value — a silent zero, which is
the outcome this translator exists to prevent. `{"regexp": {"f": null}}` is
worse: a hard HTTP 400 (`value cannot be null`) that fails the **entire** search,
including every unrelated clause in it.

**What to check.** Any `matches` / `regexp` clause whose operand is not quoted —
a bare `true`, `false` or number. Note that this is one visible symptom of a
wider fix: the parser hands the **negated** spelling of a scalar operator its
operand wrapped in a one-element list, and the Python converter unwrapped that
list *after* type inference had already run, so the two arms of one operator
disagreed about what the operand was (`f contains_cs true` rendered `*True*`
while `f not contains_cs true` rendered `*true*`). It spans `contains`,
`startswith`, `endswith`, all three `_cs` twins, `matches`, `regexp`, `cidr`,
`any` and `all`, in both the positive and the negated spelling.

No combination COUNT is given, deliberately. This page carried "35 of 216
combinations over an 18-operator × 6-literal matrix" until round 4, and it does
not reproduce on any axis: 18 × 6 is 108, not 216; the operators the sentence
itself names are eleven, not eighteen; and the matrix the suite actually pins,
`SCALAR_OPERATORS × LITERALS` in `tests/unit/test_scalar_operand_parity.py`, is
8 × 6 = 48. A product of two derived sets means nothing without both of them
named — the same rule that retired the pushdown release's `373`. Read the
matrix off that test, which computes it.

**What to write instead.** Quote the operand if you meant the literal text
(`f matches 'true'`). The rendered pattern is the same either way; quoting just
makes the intent explicit.

### 10. `field | <predicate>` with no operator is a FILTER, not `exists`

**What changed.** When the last mutator in a chain returns a boolean —
`is_private`, `is_global`, `is_loopback`, `is_multicast`, `is_link_local` — the
operator-less form now means `eq true`. It used to parse to `exists`, so
`ip | is_loopback` matched **every record that merely had an `ip` field** while
reading as a filter.

**What to check.** Any saved query using one of the five IP predicates with no
comparison operator. The explicit spellings — `ip | is_loopback = true`,
`ip | is_private eq false` — are unchanged, and every example in this README
already used one of those.

**Why this was the worst shape.** An unconditional-true clause dressed as a
predicate cannot fail loudly; it quietly stops filtering and keeps returning
plausible results. Both engines had it by **opposite** routes: Python carried the
predicate names as a literal list written out in three places and only ever
listed two of the five, so two predicates filtered and three did not; Rust
enumerated nothing at all and emitted `exists` for every mutator, so all five
matched everything. The set is now derived on both sides — Rust asks
`Mutator::returns_boolean`, Python reads `BOOLEAN_PREDICATE_MUTATORS` — and a
test pins the two equal, so a sixth predicate needs no parser edit.

**A transforming mutator is unchanged.** `ip | lowercase` is a *projection*:
keep every record that has the field and apply the mutator on the way out. It
still parses to `exists`.

### 11. A `top` / `bottom` count the engine cannot honour is refused at every surface

**What changed.** A bucket count that no engine can honour — negative, or past
`usize::MAX` — is now refused by `validate()`, `evaluate()`, `to_opensearch()`
and `query()` alike. Previously each surface answered differently, and the two
signs failed in different directions.

**Overflow was refused nowhere.** `stats count() by g top 99999999999999999999`
passed `validate()` and emitted
`{"terms": {"field": "g", "size": 99999999999999999999}}`. OpenSearch parses
`size` into a 32-bit int, so that is not a large query but an **unservable**
one — while the same saved query on the Rust detection engine was refused when
it was written.

**A negative count was refused only by the surfaces that CHECK a query.**
`query()` — the one surface that actually runs a query against records — routes
through neither `validate()` nor `to_opensearch()`, so it saw no refusal at all.
Measured over five groups, before the fix:

| Query | `validate()` | `query()` |
|---|---|---|
| `stats count() by g top -1` | refused | **4 buckets**, `health_status: "green"` |
| `stats sum(n) top -1 by g` | refused | **4 buckets**, `health_status: "green"` |

Four of five. `results[:-1]` does not fail and does not empty the result — it
silently drops the **last** bucket and returns an answer nothing downstream can
distinguish from a real one. A truncated aggregate reported healthy is worse
than a raise, which at least surfaces.

**`top 0` is unaffected.** Zero *is* representable, so it still parses, still
validates, and is still refused only where it always was — in the translator,
for the shapes a `terms` aggregation cannot express. Both engines agree on it.

**What to check.** Anything that computes a count — `top <count - 1>` against an
empty count, say — or renders one from a template where the value can arrive
negative or unbounded. Under `query()` a negative count was returning one bucket
short and reporting green; it now raises `TQLError`. A positive or absent count
is unchanged.

### Also user-visible, not breaking

- **`cidr` translates to OpenSearch** rather than being refused, and now iterates
  array-valued fields in both evaluators — `related.ip cidr '10.0.0.0/8'` matches
  `["10.1.2.3", "8.8.8.8"]`. It is refused on a `keyword`-mapped field, where
  `term` would compare the literal string `"10.0.0.0/8"`; map the field as `ip`.
- **Word operators have boundary guards.** `f nonexists` — the obvious typo for
  `f not exists` — used to parse as `none` with the value `'xists'` and return a
  match. It is now a syntax error, as are `f containszq`, `f eqzq`, `f isnull`
  and `f::intzq`.
- **Four mutator names are rejected at parse.** `count`, `unique`, `first` and
  `last` parsed and silently returned nothing; they now raise `TQLSyntaxError`.
- **Rust type hints are load-bearing.** They were parsed, stored on the AST, and
  read by nothing, so `f::int eq 'Hello'` answered identically to the un-hinted
  query. Rust also gained the `date`, `geo`, `object` and `ip` hint names, which
  Python already had — the two runtimes now accept the same sixteen.
- **An empty list literal is refused at parse, in every runtime.** `f in []` was
  already a `TQLSyntaxError` here and in the editor, and this is unchanged for
  Python — but the Rust runtime used to parse it, so the same query was green
  while you typed it, refused when you saved it, and valid in the agent's
  detection engine. The refusal now covers `in`, `not_in`, `in_cs`, `not_in_cs`,
  `any`, `all`, `none` and `between`. A non-empty list is untouched, and
  `f in ['']` is still a list containing one empty string.

---

## 🎯 Core Features

### 🔍 **Unified Query Syntax**
Write one query, run it anywhere - files, OpenSearch, in-memory data:

```python
# Same query works everywhere
query = 'status = "active" AND age > 25'

# Query files
tql.query("users.json", query)

# Query OpenSearch
tql.execute_opensearch(client, "users-*", query)

# Query Python data
tql.query(python_list, query)
```

### 📁 **First-Class File Support**
Query files as easily as databases:

```python
# JSON/JSONL files
tql.query("logs.jsonl", "level = 'ERROR'")

# CSV files with automatic header detection
tql.query("products.csv", "price > 100 AND stock < 10")

# Folders with glob patterns
tql.query("logs/2024/*.jsonl", "status = 500", recursive=True)

# Streaming for large files (CLI)
$ tql 'status = 200' large-file.jsonl  # Processes without loading to memory
```

### 🔄 **25+ Field Mutators**
Transform data inline before comparison:

```python
# String transformations
'email | lowercase | trim = "admin@example.com"'
'name | uppercase = "JOHN DOE"'

# Encoding/decoding
'data | b64decode | lowercase = "secret"'
'password | md5 = "5f4dcc3b5aa765d61d8327deb882cf99"'

# Network operations
'ip | is_private = true'           # Check if IP is RFC 1918
'domain | defang = "hxxp://evil[.]com"'  # Security analysis

# DNS lookups
'hostname | nslookup contains "8.8.8.8"'

# GeoIP enrichment. A mutator name cannot contain a dot, so `| geoip.country_name`
# does NOT parse. `| geoip_lookup` enriches the record in place; the enriched
# `geo.*` fields are then filtered as ordinary fields.
'ip | geoip_lookup exists AND geo.country_name = "United States"'

# List operations
'scores | avg > 80'
'prices | sum between [100, 500]'
```

### 📊 **Statistical Aggregations**
Analyze data with built-in stats functions:

```python
# Simple aggregations
tql.query(data, '| stats count(), sum(revenue), avg(price)')

# Grouped analysis
tql.query(data, '| stats count() by status, region')

# Top N analysis
tql.query(data, '| stats sum(sales, top 10) by product')

# Combined filtering and stats
tql.query(data, 'region = "west" | stats avg(revenue) by category')
```

### 🔧 **OpenSearch Integration**
Seamless OpenSearch/Elasticsearch integration:

- **Automatic DSL Translation**: TQL queries → OpenSearch Query DSL
- **Smart Field Mapping**: Handles keyword vs text fields automatically
- **Post-Processing**: Apply mutators that OpenSearch can't handle
- **Pagination Support**: Handle large result sets efficiently

```python
# TQL handles field mapping automatically
mappings = {'user.email': {'type': 'text', 'fields': {'keyword': {'type': 'keyword'}}}}
tql = TQL(mappings)

# Exact match uses .keyword automatically
query = 'user.email = "admin@example.com"'  # Uses user.email.keyword

# Mutators trigger post-processing when needed
query = 'user.email | lowercase contains "admin"'  # Post-processes results
```

---

## 📖 Syntax Guide

### Comparison Operators

```python
# Equality
'status = "active"'           # Exact match (alias: eq)
'status != "inactive"'        # Not equal (alias: ne)

# Numeric comparisons
'age > 25'                    # Greater than
'age >= 18'                   # Greater or equal
'age < 65'                    # Less than
'age <= 100'                  # Less or equal

# String operations (case-INSENSITIVE)
'email contains "@example.com"'      # Substring
'name startswith "John"'            # Prefix
'filename endswith ".pdf"'          # Suffix

# Case-SENSITIVE twins, and their negations
'name contains_cs "John"'           # also startswith_cs, endswith_cs
'name not_contains_cs "John"'       # also not_startswith_cs, not_endswith_cs
'role in_cs ["Admin"]'              # also not_in_cs

# Regex. The pattern SEARCHES unless you anchor it -- see "Regex semantics".
'email matches "@example\\.com"'     # matches anywhere in the value
'email regexp "^\\w+@\\w+\\.\\w+$"'  # ^...$ means the WHOLE value

# Range and membership
'age between [18, 65]'              # Inclusive range
'status in ["active", "pending"]'   # Value in list (case-insensitive)
'status not in ["deleted"]'         # Value not in list (case-insensitive)

# Existence checks -- `exists` and `is not null` are the SAME question,
# and so are `not exists` and `is null`. See the matrix just below.
'field exists'                      # Field is present AND non-null
'field is not null'                 # identical to `field exists`
'field not exists'                  # Field is absent OR present-and-null
'field is null'                     # identical to `field not exists`

# Network operations
'ip cidr "192.168.0.0/16"'          # IP in CIDR range
```

There is **no `range` operator** in either runtime, despite older revisions of
this page listing one as an alias for `between`. Use `between`.

#### Null and existence semantics

`exists` / `is not null` and `not exists` / `is null` are exact complements.
Every record satisfies exactly one side, on every backend:

| document      | `is null` | `not exists` | `is not null` | `exists` |
|---------------|-----------|--------------|---------------|----------|
| `{}`          | match     | match        | --            | --       |
| `{"f": null}` | match     | match        | --            | --       |
| `{"f": "x"}`  | --        | --           | match         | match    |
| `{"f": []}`   | --        | --           | match         | match    |

An **empty array is a present value**: the producer deliberately wrote a field
with zero elements, so `[]` satisfies `exists` and `is not null`, not `is null`.
`[null]` and `["a", null]` follow the same rule.

> **Residual divergence, stated rather than left to be found.** OpenSearch does
> not index a JSON null and does not index an empty array, so against a cluster
> `{"f": []}` and `{"f": [null]}` DO match `not exists` / `is null`. The
> in-memory evaluators can see `_source` and answer per the table above. This is
> the one document shape where the two execution paths differ.

#### Regex semantics

`matches` and `regexp` are the same operator, and an **unanchored pattern
searches**: `f matches 'abc'` matches any value containing `abc`. An anchored
pattern still means a whole-value match — `^` and `$` are honoured, not wrapped:

```python
'f matches "abc"'      # matches "xxabcxx"
'f matches "^abc$"'    # matches only "abc"
```

Against OpenSearch this is translated by wrapping an unanchored pattern in `.*`,
because Lucene's `regexp` query anchors implicitly. Anchored patterns are passed
through unwrapped.

### Logical Operators

```python
# AND (all conditions must be true)
'age > 25 AND city = "NYC"'
'status = "active" AND role in ["admin", "moderator"]'

# OR (any condition must be true)
'city = "NYC" OR city = "LA"'
'status = "admin" OR role = "superuser"'

# NOT (negates condition)
'NOT (age < 18)'
'NOT status = "deleted"'

# Complex expressions with parentheses
'(age > 25 AND city = "NYC") OR (status = "vip" AND score > 90)'
```

### Collection Operators

Collection operators are written **field-first**, with the operator between the
field and the value:

```python
# any - at least one array element matches
'tags any "premium"'
'user.roles any "admin"'

# all - every array element equals the value
'status all "active"'
'scores all 80'

# none - no array element matches
'flags none "spam"'
'violations.severity none "critical"'

# Negated spellings
'tags not_any "premium"'
'status not_all "active"'
'flags not_none "spam"'
```

> The operator-first spelling — `ANY tags = "premium"` — parses in the **Rust**
> runtime only. The Python parser raises `TQLSyntaxError` on it. Use the
> field-first form above for queries that must run under both.

**The operand is a scalar.** `tags any ['a', 'b']` asks whether some single
element of `tags` equals the two-element *list*, which no scalar element can, so
both the OpenSearch translator and the in-memory evaluator raise `TQLTypeError`
and name `in` as the operator you wanted:

```python
'tags any ["a", "b"]'      # TQLTypeError -- ill-typed operand
'tags in ["a", "b"]'       # what that query meant
'tags any "a"'             # fine; a one-element list unwraps to this
```

`all` / `not_all` are the deliberate exception: they are answered by a script
query that compares each element to the operand, where a list is a valid
(never-matching) value.

### Nested Field Access

```python
# Dot notation for nested objects
'user.profile.email contains "@example.com"'
'metadata.tags.priority = "high"'

# Array indexing also uses dot notation -- bracket syntax is NOT accepted
'tags.0 = "urgent"'
'history.5.status = "completed"'
```

### Field Mutators Reference

#### **String Mutators**
- `lowercase`, `uppercase` - Case conversion
- `trim` - Remove whitespace
- `split(delimiter)` - Split string into array
- `length` - Get string length
- `replace(old, new)` - Replace substring

#### **Encoding Mutators**
- `b64encode`, `b64decode` - Base64 encoding/decoding
- `urldecode` - URL decode
- `hexencode`, `hexdecode` - Hex encoding/decoding
- `md5`, `sha256` - Cryptographic hashing

#### **Network/Security Mutators**
- `refang` - Convert defanged indicators (hxxp → http)
- `defang` - Defang URLs for safe display
- `is_private` - Check if IP is private (RFC 1918)
- `is_global` - Check if IP is globally routable
- `is_loopback` - Check if IP is loopback (`127.0.0.0/8`, `::1`)
- `is_multicast` - Check if IP is multicast (`224.0.0.0/4`, `ff00::/8`)
- `is_link_local` - Check if IP is link-local (`169.254.0.0/16`, `fe80::/10`)

  All five return a boolean, take no arguments, and match if **any** element of
  a multi-valued field qualifies. They are post-processed rather than pushed
  down to OpenSearch, because the value they compute is not the value that is
  indexed.

  "Take no arguments" means exactly that. Both parsers now also tolerate empty
  parens on a mutator — `ip | is_private()` parses and evaluates identically to
  `ip | is_private` in both runtimes — but the bare name is the spelling to
  write. The **GeoIP** mutators are the exception, and the one place the parens
  are not portable: `ip | geo() exists` and `ip | geoip_lookup() exists` parse
  under the Rust runtime and raise `TQLSyntaxError` in Python.

  **With no comparison operator, the bare form is a filter.** `ip | is_loopback`
  means `ip | is_loopback = true`. It used to parse to `exists` and match every
  record that merely *had* an `ip` field — see
  [breaking change 10](#10-field--predicate-with-no-operator-is-a-filter-not-exists).

#### **DNS Mutators**
- `nslookup` - Resolve hostname to IP addresses

#### **GeoIP Mutators**
- `geoip_lookup` (aliases `geo`, `geoip`) - Enrich IP with geolocation data.
  All three spellings work in every runtime. Earlier releases documented
  `geoip` as Rust-only and advised `geoip_lookup` "for queries that run under
  both"; that constraint is **gone** — Python accepts `geoip` as of this
  release, so a query written against either name behaves identically. The same
  applies to `is_loopback`, `is_multicast` and `is_link_local` above, and to the
  `not_any` / `not_all` / `not_none` operator spellings, all of which Rust
  accepted while Python did not.
- Returns: `geo.country_name`, `geo.city_name`, `geo.location`, `geo.continent_code`, etc.

#### **List Mutators**
- `any`, `all` - Boolean aggregations
- `avg`, `average` - Calculate mean
- `sum` - Calculate sum
- `min`, `max` - Find min/max values

A mutator name TQL does not know is rejected **at parse time**, not at
evaluation. `count`, `unique`, `first` and `last` used to parse and then quietly
return nothing; they now raise `TQLSyntaxError`. There is no public registration
API — the permitted set is `ALLOWED_MUTATORS` in `src/tql/mutators/__init__.py`,
and it is the same set in all three runtimes.

### Type Hints

A `::hint` suffix tells TQL **how to read a field's value** before the
comparison runs.

```python
'value::number > 75'          # read as a number, then compare
'flag::string eq "true"'      # read as a string, then compare
'addr::ip eq "10.1.2.3"'      # read as an IP address
```

> **A hint and a mutator chain may be written in EITHER order.**
> `f::number | trim > 75` and `f | trim::number > 75` both parse, in both
> runtimes and in the TypeScript editor, and select the same records. This was a
> parity divergence — each grammar rejected the other's spelling — and it was
> closed in this release. If you find a page still describing one order as
> runtime-specific, that page is stale.

Both runtimes accept the same sixteen names, and every one of them reaches a
decision:

| Hint | Behaviour |
|---|---|
| `string`, `str` | render the value as a string |
| `integer`, `int`, `number`, `decimal`, `float`, `double` | numeric conversion |
| `boolean`, `bool` | boolean conversion |
| `ip` | validate as an IP address (and as a CIDR under the `cidr` operator) |
| `array`, `list`, `date`, `geo`, `object` | assert the field's **shape**; no value conversion is performed |

Three rules govern what happens when a hint meets a value:

1. **A value the hint cannot read skips the record — it does not abort the
   query.** `value::number > 75` over `[{v: 80}, {v: "abc"}, {v: 90}]` matches
   80 and 90; `"abc"` simply does not match.
2. **A skipped record matches nothing, positively or negatively.**
   `f::number != 75` over `"abc"` is **false**, not true. The same holds for
   `not_contains`, `not_startswith`, `not_endswith`, `not_in` and `not_cidr`,
   and for all four existence operators: `f::int exists`, `f::int not exists`,
   `f::int is null` and `f::int is not null` are all false over `{"f": "Hello"}`
   — `f` does not exist *as an int*, which is what the hint asked. An absent or
   null field is not a read failure, so `f::int not exists` over `{}` and
   `f::int is null` over `{"f": null}` are unchanged.
3. **A query-level `not (...)` does still invert a skip.** `not` is a boolean
   operator over the comparison's answer, the same way it already behaves for an
   absent field, and the same way `must_not` answers on the cluster path.

An **unknown hint name** is a hard error — `f::bogus` is rejected by the parser.

`::string` renders a boolean as `true` / `false`, not Python's `True` / `False`:

```python
'flag::string eq "true"'   # matches {"flag": true}
'flag::string eq "True"'   # matches nothing
```

That divergence from `str()` is deliberate and applies to a **top-level boolean
under `string`/`str` only**. A boolean nested in a list or object still renders
`True`, so `f::string eq "['a', True]"` is how you match `{"f": ["a", true]}`.

> Type hints are honoured by the **in-memory evaluators**. The OpenSearch
> execution path ignores them: the cluster reads the value as the mapping says
> it is stored.

---

## 📊 Statistical Aggregations

TQL includes a powerful stats engine for data analysis:

### Available Functions

```python
# Counting
'| stats count()'              # Count all records
'| stats count(field)'         # Count non-null values
'| stats unique_count(field)'  # Count distinct values

# Numeric aggregations
'| stats sum(revenue)'         # Calculate sum
'| stats avg(price)'           # Calculate average (aliases: average, mean)
'| stats min(age), max(age)'   # Find min/max values
'| stats median(score)'        # Calculate median

# Statistical measures
'| stats std(values)'          # Standard deviation
# Full name: '| stats standard_deviation(values)'. Note that neither `stdev`
# nor `stddev` works in either runtime, despite `stddev` appearing in the Rust
# grammar -- the PEG matches the shorter `std` first, so the longer spelling is
# unreachable. The same shadowing affects `percentiles`, `percentile_rank(s)`
# and `pct_rank(s)` in Rust.
'| stats percentile(score, 95)' # Calculate percentile

# Value extraction
'| stats values(category)'     # Get unique values
```

### Grouping and Top N

```python
# Group by single field
'| stats count() by status'

# Group by multiple fields
'| stats sum(revenue) by region, category'

# Top N analysis. Both spellings are portable since 2026-09-04 -- both runtimes
# accept `| stats sum(sales, top 10) by product` and
# `| stats sum(sales) top 10 by product`, and both now return the SAME ten
# buckets. This modifier ranks by the AGGREGATE value.
'| stats sum(sales, top 10) by product'

# The GROUP-BY bucket limit is a different modifier with the same keyword: it
# ranks by doc_count, not by the aggregate. `by product top 10` and
# `sum(sales) top 10 by product` will disagree on which ten unless doc_count and
# the sum happen to rank the products identically.
'| stats sum(sales) by product top 10'

# Multiple aggregations
'| stats count(), sum(revenue), avg(price) by status'
```

### Combined Filtering and Stats

```python
# Filter then aggregate
'status = "success" AND region = "west" | stats avg(revenue) by category'

# Complex analytics
'timestamp > "2024-01-01" | stats count(), sum(bytes), avg(response_time) by endpoint'
```

---

## 🔌 OpenSearch Integration Guide

### Setup

```python
from opensearchpy import OpenSearch
from tql import TQL

# Create OpenSearch client
client = OpenSearch(
    hosts=['localhost:9200'],
    http_auth=('admin', 'admin'),
    use_ssl=True,
    verify_certs=False
)

# Get index mappings
response = client.indices.get_mapping(index='users-*')
mappings = response['users-2024']['mappings']['properties']

# Initialize TQL with mappings
tql = TQL(mappings)
```

### Query Translation

TQL automatically translates queries to OpenSearch DSL:

```python
# TQL Query
query = 'age > 25 AND status = "active"'

# Translates to OpenSearch DSL:
{
    "query": {
        "bool": {
            "must": [
                {"range": {"age": {"gt": 25}}},
                {"term": {"status.keyword": "active"}}
            ]
        }
    }
}

# Execute seamlessly
results = tql.execute_opensearch(client, 'users-*', query)
```

### Field Mapping Intelligence

TQL automatically handles field types:

```python
# Text field with keyword subfield
mappings = {
    'email': {
        'type': 'text',
        'fields': {
            'keyword': {'type': 'keyword'}
        }
    }
}

# Exact match - uses .keyword automatically
'email = "admin@example.com"'  # → term query on email.keyword

# Substring - also uses .keyword, with case_insensitive: true
'email contains "example"'      # → wildcard on email.keyword

# Mutators trigger post-processing
'email | lowercase = "admin@example.com"'  # → fetch + filter
```

#### A field that cannot answer the question is refused, not answered with zero

Some operators need information the index does not contain. TQL raises
`TQLTypeError` rather than emitting a query that can never match — a clause that
returns zero hits with no error is indistinguishable from "nothing happened", and
in the negated direction it is an exclusion that silently excludes nothing.

| Query | Field mapping | Result |
|---|---|---|
| `f contains_cs "A"` (also `startswith_cs`, `endswith_cs`, `in_cs`, and their `not_` forms) | analyzed `text`, no case-preserving subfield | `TQLTypeError` — the analyzer lowercased every indexed token, so the case being asked about is not in the index |
| `f cidr "10.0.0.0/8"` | `keyword` | `TQLTypeError` — `term` expands CIDR notation only on an `ip` field; on a `keyword` it compares the literal string `"10.0.0.0/8"` |
| `f any ["a", "b"]` | any | `TQLTypeError` — the operand is ill-typed; use `in` |

Each error names the remedy: add a `.keyword` subfield or use the
case-insensitive operator; map the field as `ip` or give it an `ip` subfield;
use `in`.

**An UNMAPPED field is still allowed through** in every one of these cases. With
no mappings TQL cannot tell an analyzed field from a keyword one, and refusing
would break every query against an index whose mappings could not be fetched.
So do not read a successful translation as proof the field can answer — read it
as proof TQL knows of no reason it cannot.

### Post-Processing

When OpenSearch can't handle operations, TQL applies post-processing:

```python
# Mutators that require post-processing
'email | lowercase contains "admin"'    # Post-process: case conversion
'data | b64decode contains "secret"'    # Post-process: decode
'ip | geoip_lookup exists'             # Post-process: GeoIP lookup

# TQL automatically:
# 1. Executes base query in OpenSearch
# 2. Fetches results
# 3. Applies mutators in Python
# 4. Filters results
# 5. Returns final matches
```

### Query Analysis

Analyze queries before execution to understand performance implications:

```python
# Analyze query health
analysis = tql.analyze_query('email | lowercase contains "admin"', context='opensearch')

print(f"Health: {analysis['health']['status']}")  # 'fair' (post-processing)
print(f"Score: {analysis['health']['score']}")    # 85
print(f"Post-processing: {analysis['mutator_health']['requires_post_processing']}")  # True

# Recommendations for optimization
for issue in analysis['health']['issues']:
    print(f"Issue: {issue['message']}")
    print(f"Fix: {issue['recommendation']}")
```

---

## 📚 Documentation

Comprehensive documentation is available in the [docs/](./docs/) directory:

User-facing TQL documentation — getting started, query basics, the operator and
mutator references, OpenSearch integration, stats, the API reference and the
cookbook — lives in the **`tellaro-docs`** repository under `public/tql/` and is
published from there.

What stays in this repo is contributor material: see
[docs/README.md](./docs/README.md).

---

## ⚡ Performance

### Benchmarks

> **Unsubstantiated.** These figures predate the current code and could not be
> reproduced: the repo ships no benchmark harness (no `benches/`, no
> `[[bench]]` target, no `pytest-benchmark`). Treat them as rough historical
> claims rather than measurements.

**Python Implementation:**
- In-memory queries: ~10,000 records/sec
- File parsing (JSON): ~5MB/sec
- OpenSearch queries: Limited by network latency

**Rust CLI (300x faster):**
- In-memory queries: ~3,000,000 records/sec
- File parsing (JSON): ~150MB/sec
- Large file streaming: Process 50MB in ~200ms

### Optimization Tips

```python
# Use CLI for large files (300x faster)
$ tql 'status = 200' 50MB-file.jsonl  # ✓ Fast (Rust)
$ python -m tql 'status = 200' 50MB-file.jsonl  # ✗ Slow (Python)

# Pre-compile queries for reuse
ast = tql.parse('age > 25 AND status = "active"')
results1 = tql.evaluate(ast, dataset1)
results2 = tql.evaluate(ast, dataset2)

# Use OpenSearch for large datasets
tql.execute_opensearch(client, 'huge-index-*', query)  # Leverages OpenSearch's speed

# Minimize post-processing
'email.keyword = "admin@example.com"'  # ✓ Fast (OpenSearch only)
'email | lowercase = "admin@example.com"'  # ✗ Slower (post-processing)
```

---

## 🛠️ Development

### Installation

```bash
# Clone repository
git clone https://github.com/tellaro/tellaro-query-language.git
cd tellaro-query-language

# Install dependencies (recommended)
uv sync

# Or with pip
pip install -e .
```

### Testing

```bash
# Run all tests
uv run tql-tests

# Run specific test file
uv run pytest tests/unit/test_parser.py -v

# Run with coverage
uv run tql-cov

# Run integration tests (requires OpenSearch)
cp .env.example .env  # Configure OpenSearch connection
uv run pytest tests/integration/test_opensearch_integration.py -v
```

### Code Quality

```bash
# Format + lint in one step
uv run tql-lint-all

# Format code (ruff replaces black, isort and flake8 -- none is a dependency)
uv run ruff format src tests

# Type checking
uv run pyright

# Linting
uv run ruff check src tests

# Security checks
uv run bandit -c bandit.yml -r src/
```

---

## 🗺️ Roadmap

### ✅ Implemented Features
- ✅ Core query engine with all operators
- ✅ 25+ field mutators (string, encoding, network, DNS, GeoIP, list)
- ✅ Statistical aggregations with grouping
- ✅ File support (JSON, JSONL, CSV)
- ✅ OpenSearch/Elasticsearch backend
- ✅ Intelligent post-processing
- ✅ Rust CLI for performance
- ✅ Mutator caching for GeoIP/DNS
- ✅ Query health analysis

### 🚧 In Progress
- 🚧 OpenSearch stats aggregation translation
- 🚧 Additional hash functions (SHA1, SHA512)
- 🚧 JSON parsing mutator
- 🚧 Timestamp conversion mutators

### 📋 Planned Features
- 📋 ElasticSearch backend support
- 📋 PostgreSQL/MySQL backends
- 📋 Query optimization engine
- 📋 Custom mutator plugins
- 📋 GraphQL-style field selection
- 📋 Parallel record evaluation
- 📋 Incremental file processing

### 🔮 Future Considerations
- 🔮 Time-series specific operators
- 🔮 Machine learning integration
- 🔮 Distributed query execution
- 🔮 Query caching layer

---

## 🤝 Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](./docs/developer/contributing.md) for guidelines.

### How to Contribute

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Make your changes
4. Run tests (`uv run tql-tests`)
5. Run linters (`uv run tql-lint`)
6. Commit changes (`git commit -m 'Add amazing feature'`)
7. Push to branch (`git push origin feature/amazing-feature`)
8. Open a Pull Request

---

## 📄 License

**Tellaro Query Language (TQL)** is **source-available** software with specific usage terms:

✅ **Permitted Uses:**
- Personal use (individual, non-commercial)
- Organizational use (within your company/organization)
- Integration into your applications and services
- Internal tools and automation

❌ **Restricted Uses:**
- Creating derivative query language products
- Commercial redistribution or resale
- Offering TQL-based commercial services to third parties
- Using source code to build competing products

For commercial licensing inquiries, contact: **support@tellaro.io**

See [LICENSE](LICENSE) for complete terms and conditions.

---

## 🔗 Related Projects

- **[TQL Rust](https://crates.io/crates/tellaro-query-language)** - High-performance Rust implementation
- **[Tellaro Platform](https://github.com/tellaro)** - Security operations platform using TQL

---

## 💬 Support

- **Issues**: [GitHub Issues](https://github.com/tellaro/tellaro-query-language/issues)
- **Documentation**: [Full Documentation](./docs/)
- **Examples**: the cookbook in `tellaro-docs/public/tql/cookbook/`
- **Email**: support@tellaro.io

---

## 🌟 Quick Examples

### Security Log Analysis

```python
# Find high-severity events from private IPs
query = '''
    source_ip | is_private = true AND
    severity in ["high", "critical"] AND
    (tags any "malware" OR url | defang contains "suspicious")
'''
results = tql.query("security-logs.jsonl", query)
```

### E-commerce Analytics

```python
# Analyze sales by region for premium products
query = '''
    product_tier = "premium" AND
    order_date > "2024-01-01" |
    stats sum(revenue), avg(order_value), count() by region
'''
results = tql.query("sales.json", query)
```

### System Monitoring

```python
# Find servers with high resource usage
query = '''
    hostname | nslookup exists AND
    (cpu_usage > 80 OR memory_usage > 90) AND
    status = "production"
'''
results = tql.execute_opensearch(client, "metrics-*", query)
```

---

**Made with ❤️ by the Tellaro Team**

