Metadata-Version: 2.4
Name: tgdata
Version: 0.0.9
Summary: A production-grade Python library for extracting and processing Telegram group and channel messages
Author: enes kuzucu
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: Telethon<2.0,>=1.45.0
Requires-Dist: pandas>=1.0
Provides-Extra: proxy
Requires-Dist: python-socks[asyncio]>=2.0; extra == "proxy"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# tgdata

A production-grade Python library for extracting and processing Telegram group and channel messages. Designed for ETL pipelines, data analysis, and archival purposes.

> **New in 0.0.8 — group discovery.** tgdata can now *find* groups and channels, not only read
> them: Telegram's name search, its "similar channels" recommendations, and links mined from
> posts, on the same persistent session, paced, flood-safe and budgeted. Four calls —
> `search_groups`, `similar_groups`, `linked_groups`, `discover_groups` — return candidate rooms
> in the `list_groups` row shape, already usable by `get_messages`. Quick start below under
> [Group discovery](#group-discovery--finding-groups-you-are-not-in); the full guide to using it
> properly is [`devdocs/guides/group_discovery.md`](devdocs/guides/group_discovery.md).

## Features

- 🚀 **Production-Ready**: Built for reliability and scale in ETL pipelines
- 🆔 **Flexible Identification**: Use numeric IDs or usernames (@channelname) for all operations
- 📊 **Efficient Data Extraction**: Fetch messages with automatic rate limit handling
- 🔄 **Incremental Updates**: Fetch only new messages with `after_id` parameter
- 📈 **Progress Tracking**: Monitor long-running operations with real-time progress
- 🔌 **Clean Architecture**: Focused on data extraction with minimal dependencies
- 🛡️ **Robust Error Handling**: Automatic retries with exponential backoff
- 📁 **Multiple Export Formats**: Export to CSV, JSON, or integrate with your data pipeline
- 🔔 **Real-time Updates**: Listen for new messages with event handlers
- ⏱️ **Polling Support**: Poll for new messages at configurable intervals
- 🎯 **Batch Processing**: Handle large groups with configurable batch sizes and delays
- 🖼️ **Media Detection & Download**: Flag photos/videos per message (no download needed) and pull the files on demand or during the fetch
- 🔎 **Group Discovery**: Find rooms you are not in — Telegram's name search, its "similar channels", and links mined from posts — paced, flood-safe, budgeted
- 🛡️ **Proxy Support**: An optional per-account SOCKS5/SOCKS4/HTTP proxy for every connection, with `require_proxy` and no silent fallback to a direct line
- 🪪 **Fixed Device Identity**: Optionally pin the device model, system and app version an account presents, so it looks the same from every machine and after Telethon upgrades
- 🩺 **Account Health Events**: Every wait, logout, ban, restriction or lost group is reported as a plain-data event — through one callback, a log line and a per-account summary — and "ok" when it ends
- **Per-account read budgets**: An optional shared SQLite ledger limits message pulls over a rolling 24 hours, with persistent warm-up steps and recoverable partial results
- **Versioned message batches**: Read bounded raw snapshots with canonical JSON, replayable batch IDs, safe continuation cursors and optional content-addressed media

## Installation

```bash
pip install tgdata
```


## Authentication

tgdata supports 1 authentication at the moment. 

(2 other are being implemented)

1. get telegram development credentials in telegram API Development Tools from [https://my.telegram.org/apps](https://my.telegram.org/apps)

2. Create a `config.ini` file with your Telegram API credentials like this:

```ini
[Telegram]

# you can get telegram development credentials in telegram API Development Tools
api_id = 1234566
api_hash = a24adjfakjdfakjshdflkajsbdflk
phone = +905064004949 # use full phone number including + and country code

# Where the Telethon session lives (".session" is appended automatically).
# Use an ABSOLUTE path — a bare name creates the file in whatever directory
# the process happens to run from. Do not quote values: INI values are raw
# text, so quotes become part of the value (tgdata strips them defensively
# since 0.0.4, but don't rely on it).
session_file = /absolute/path/to/my_session
```

**Session naming precedence** (since 0.0.4): an explicit `session_file`
always wins. A legacy `username` key is used as the session name only when
no `session_file` is set — configs that relied on username-named sessions
keep working, but new configs should set `session_file` and omit `username`
(it plays no part in authentication).

### Logging in

The first run of a new session, started by hand in a terminal, asks for the
code Telegram sends you. After that, tgdata never asks Telegram for a login
code on its own: before connecting it checks with Telegram whether the session
is logged in, and if it is not, it raises `AuthRequiredError` instead of
prompting.

- `.reason` is Telegram's own name for the problem — `AUTH_KEY_UNREGISTERED`
  or `SESSION_REVOKED` when Telegram logged the account out,
  `USER_DEACTIVATED_BAN` when it is banned. `.banned` is true when logging in
  again cannot help.
- To log back in after Telegram logged the account out, run once by hand in a
  terminal with `TgData("config.ini", interactive_login=True)`.
  `interactive_login=False` forbids even the first-run prompt.
- A background job should stop and alert someone on `AuthRequiredError`.
  Retrying cannot log it back in — and, unlike before, it no longer sends the
  owner a fresh login code on every retry.

### Keeping sessions out of files

Pass a store once to keep each account's session in your application's storage:

```python
import sqlite3
from tgdata import TgData

class SessionStore:
    def __init__(self, path):
        self.db = sqlite3.connect(path)
        with self.db:
            self.db.execute(
                "CREATE TABLE IF NOT EXISTS sessions "
                "(name TEXT PRIMARY KEY, data TEXT NOT NULL)"
            )

    def load(self, name):
        row = self.db.execute(
            "SELECT data FROM sessions WHERE name = ?", (name,)
        ).fetchone()
        return row[0] if row else None

    def save(self, name, data):
        with self.db:
            self.db.execute(
                "INSERT OR REPLACE INTO sessions (name, data) VALUES (?, ?)",
                (name, data),
            )

    def delete(self, name):            # optional; used on Telegram log-out
        with self.db:
            self.db.execute("DELETE FROM sessions WHERE name = ?", (name,))

store = SessionStore("accounts.sqlite3")
tg = TgData("config.ini", session_store=store)
# Use tg as usual. When finished, await tg.close(), then store.db.close().
```

This example keeps sessions in a small SQLite table across restarts. Supply
your own store to use another database or storage service. The methods are
synchronous, not `async def`, and run on the
event loop: keep them quick, for example with a local database or an in-memory
cache that writes behind. tgdata ships no storage backend.

- **Names:** `session_file` is the store key, used as configured; no `.session`
  suffix is added. The existing fallback to `username`, then `telegram_session`,
  still applies. Persistent and short-lived clients share that name; pool
  connections use `<name>_1`, `<name>_2`, and so on.
- **Contents:** one opaque, versioned string containing the login key, data
  centre, update states, groups and channels with their access hashes, and the
  account's own identity rows. Message senders are excluded from this session
  cache. The temporary key and sent-file cache are not persisted.
- **Keep the string as text.** It is not Telethon's `StringSession` format or a
  JSON document for the store to interpret. Its encoding is not encryption;
  access control and any encryption belong to your store.
- **Save points:** whenever Telethon saves (connect, data-centre changes, new
  login keys, and periodically while connected), plus disconnect/`close()`.
  Unchanged state causes no write. Always close the client when finished.
- **Failures:** a store load error or unreadable saved string raises before a
  connection opens. A failed save is logged at ERROR and retried at the next
  save point; failed deletes are logged too. Save/delete logs contain the
  session name and error type, never the error text or traceback, which could
  contain the login key.
- **Two clients:** with the same login, the later save wins for the cache.
  Before a changed save or deletion, a client re-reads the stored key. If it
  changed or was removed, the client skips the mutation and warns once. Logging
  out an older client therefore preserves a newer login saved by another client.
  This check and save are separate operations, not an atomic guarantee across
  processes. Telethon's session copies for side connections stay in memory and
  never write to the store.
- **Login rules stay the same:** `None` from `load` means a new session; an
  existing session that Telegram logged out still raises `AuthRequiredError`.
  If `delete` is provided, Telegram log-out removes the stored record only when
  it still holds the login that client last loaded or saved.

Without `session_store`, existing `.session` files work exactly as before.
There is no automatic migration from files to a store. This option is designed
and tested against Telethon 1.45.0.

### Connecting through a proxy (optional)

With no `proxy` key, tgdata connects directly, exactly as before. Add one, and
**every** connection to Telegram goes through it — the persistent client, each
pool connection, and every short lookup. All of them are built by one function
in the connection engine, so there is no second way in.

```ini
[Telegram]
api_id = 1234566
api_hash = a24adjfakjdfakjshdflkajsbdflk
session_file = /absolute/path/to/my_session
proxy = socks5://user:password@203.0.113.7:1080
require_proxy = true
```

- **Forms:** `socks5://host:port`, `socks5://user:pass@host:port`,
  `socks4://…`, `http://…` (an HTTP CONNECT proxy). Percent-encode special
  characters in the user name or password (`@` as `%40`, `:` as `%3A`).
- **`require_proxy = true`** refuses to connect when no proxy is configured.
  Use it on accounts that must never touch Telegram from the machine's own
  IP.
- **No fallback, ever.** A malformed URL, a missing proxy library, or
  `require_proxy` without a proxy raises `tgdata.ProxyConfigError` before any
  connection opens. A proxy that is down or refuses the tunnel makes the
  connection fail with `ConnectionError`. Nothing retries without the proxy.
- **Install the extra:** `pip install 'tgdata[proxy]'`, which pulls in
  `python-socks[asyncio]`.
- **Logs and health checks show the proxy masked**
  (`socks5://user:***@203.0.113.7:1080`); the password never appears in a
  log line or a repr. The engine logs once whether connections are direct or
  via a proxy.
- **Not supported:** MTProto proxies (`tg://proxy?…`, `t.me/proxy?…`); they
  are refused with a clear error.
- One `proxy` per config file, and the account pool has one config per
  account, so each account gets its own proxy.
- An existing session keeps working through a proxy; a session file is not
  tied to an IP.

### A fixed device identity (optional)

Every connection tells Telegram what device it is running on — a device
model, a system version, an app version and language codes. Telegram shows
them in the account's active-sessions list. Unless you pin them, Telethon
derives them from the machine and from its own version, so the same account
presents a different device on another machine, after an OS update, and after
every Telethon upgrade. On a Mac today that is `arm64`, `25.6.0`, `1.40.0`.

Pin them in the account's config, and every connection presents exactly
that, from any machine:

```ini
[Telegram]
device_model = PC 64bit
system_version = Windows 11
app_version = 4.16.8 x64
lang_code = en
system_lang_code = en
```

- **All five keys are optional.** No keys means Telethon's defaults, exactly
  as before; a key you leave out keeps its default. `lang_code` alone also
  sets `system_lang_code`. Values may contain spaces; quotes and an inline
  `; comment` are stripped.
- **You can add it later.** The identity is sent on every connection, not
  only at login, so an existing session picks it up on its next connection —
  no new login.
- **To keep what an account shows today**, pin its current values instead of
  inventing new ones (switching identity is itself a visible change). Run
  this on the machine the account normally runs on and paste the output into
  its config:

```python
print(TgData("config.ini").device_identity()['config_lines'])
```

  `device_identity()` needs no network and no session; it also returns the
  presented values and which ones are pinned. `health_check()` reports the
  presented identity too.
- Whether Telegram cares about a changing identity has not been measured;
  pinning is a precaution for accounts that move between machines.


## Per-account read budgets

Pass a `ReadBudget` to limit explicit message pulls across jobs for the same
Telegram account. Every persistent, pooled and short-lived client built by
that `TgData` uses it. Without `read_budget`, existing behavior is unchanged.

```python
import asyncio
from tgdata import TgData, ReadBudget, ReadBudgetExceeded

async def main():
    budget = ReadBudget("/absolute/path/read-budgets.sqlite3")
    tg = TgData("config.ini", read_budget=budget)
    try:
        # Policy setup: get the authenticated account ID using metadata only.
        client = await tg.connection_engine.get_client()
        me = await client.get_me()
        budget.configure(
            account_id=me.id,
            daily_limit=2000,
            warmup=[(0, 100), (2, 300), (7, 2000)],
        )

        try:
            messages = await tg.get_messages("@channelname", limit=2000)
        except ReadBudgetExceeded as error:
            messages = error.partial_result
            print(error.status.to_dict())
            print("Retry after seconds:", error.retry_after)

        if messages is not None:
            print(messages)
        print((await tg.get_read_budget()).to_dict())
    finally:
        await tg.close()

asyncio.run(main())
```

The numbers above illustrate configuration; they are **not Telegram-safe rate
recommendations**. Warm-up days are completed 24-hour periods since enrollment,
not calendar days. The example starts at 100, rises to 300 after two days, and
to 2,000 after seven. Without `warmup`, the full `daily_limit` applies immediately.
A zero cap pauses reads. A curve starts at day zero, has unique nonnegative
integer days and nondecreasing caps, and never exceeds `daily_limit`.

`configure()` deliberately replaces the limit and curve, while retaining usage
and the original start time. Omit it in later jobs to keep the stored policy.
Pass `started_at` explicitly to set a known enrollment time: a timezone-aware
datetime or UTC timestamp, no later than now. The last curve cap continues
until the policy changes; it does not automatically rise to `daily_limit`.

- **Shared identity and storage:** use the same SQLite file on the same host
  for every reader that should share an allowance. Keys are authenticated
  Telegram account IDs, not session names or pool suffixes. Different accounts
  have separate policies. A missing policy stops message reads. The parent
  directory must exist; `:memory:` is not supported. The ledger stores account
  IDs, policies and counters, never login keys or message content.
- **What spends the allowance:** history, message search, replies, explicit
  message-ID lookups, discovery post reading, and message re-fetches during
  media operations. Repeated reads, service/empty message slots and messages
  later filtered out still count. A count operation that fetches a message
  counts too. User/group/dialog metadata and incidental dialog previews,
  passive updates and update recovery, and downloading file bytes are excluded.
- **When it counts:** before each actual send, tgdata reserves the request's
  maximum number of messages. Successful replies settle to the returned slot
  count and release unused capacity. Failed, cancelled or crashed attempts
  retain their full claim; each retry needs another reservation. Cached flood
  waits do not spend capacity before a send. An oversized reply is charged at
  its actual count and raises a local error.
- **Rolling expiry:** each claim expires 24 hours after its send admission,
  regardless of when the reply arrives. There is no midnight reset. A persisted
  clock high-water mark prevents backward clock changes from restoring spent
  capacity. Keep the host clock accurate; forward jumps advance expiry and
  warm-up. No SQLite transaction stays open while awaiting Telegram. Operations
  are synchronous and normally short, but contention can block the event loop
  for up to the five-second SQLite busy timeout before raising.
- **Status and stopping:** `budget.status(account_id)` is local;
  `await tg.get_read_budget()` verifies the active account and returns its
  status, or `None` without connecting when disabled. `used` includes outstanding
  `reserved` capacity; `remaining` is the available allowance. `to_dict()` adds
  ISO timestamps and `retry_after` for one message. An exhaustion exception
  also has `requested`, `account_id`, `next_available_at` and `retry_after` for
  that request. Retry times are estimates assuming no new claims or policy
  changes; `None` means a smaller request or policy change is needed.
- **Partial results:** fetch/search/discovery errors carry the processed
  DataFrame in `partial_result`; media-by-ID errors carry completed entries
  in a dictionary. Batch callbacks may already have seen some of those rows.
  Polling delivers and deduplicates its partial frame, then raises exhaustion
  instead of automatically retrying. Other `ReadBudgetError` subclasses also
  stop polling. These are local policy/storage errors, not Telegram health
  verdicts; actual Telegram errors keep their existing health reporting.
- **Request boundaries:** history/search pages shrink to the available
  allowance while preserving their cursors. ID chunks and raw request bounds
  are not silently truncated, so a chunk larger than the remaining allowance
  is refused. Use smaller ID lists when needed. Unbounded or unsupported raw
  message-returning calls and message-bearing batches raise
  `UnsupportedBudgetRequest`; metadata-only batches still work. Supported
  invoke wrappers retain the inner message bound.
- **Identity verification:** each message send uses a fresh Telegram self
  lookup, including retries; page sizing and status can add another lookup.
  Cached session identities are not billing authority. Storage/configuration
  errors fail closed before further message sends; failed settlement retains
  the prior full claim. Clients created separately, other ledger files and
  direct use of private transport internals are outside this guard.

The budget suite exercises real Telethon request/iterator code with scripted
replies on 1.45.0. It does not test live Telegram response bounds or
whether any chosen limit prevents account restrictions.

## Group lookup, access and joining

These operations each open, authenticate and disconnect one short-lived client.
They use the configured proxy, device identity and session store, never prompt
for login, and do not change `current_group` or use the persistent client/pool.
Log the account in first as described under Authentication. Tested with
**Telethon 1.45.0**; the package now requires at least that version.

```python
from tgdata import TgData, JoinBudget, JoinBudgetExceeded

budget = JoinBudget("/absolute/path/join-budgets.sqlite3")
tg = TgData("config.ini", join_budget=budget)

# In your async function: enroll the actual account once, then keep the policy.
async with tg.connection_engine.ephemeral_client() as client:
    me = await client.get_me()
budget.configure(me.id, daily_limit=3)  # Illustrative policy, not a safe rate.

details = await tg.lookup_group("@group_handle")
access = await tg.check_group_access("@group_handle")
print(details.to_dict(), access.to_dict())

# When your caller explicitly chooses to join:
try:
    result = await tg.join_group("@group_handle")
    print(result.status, result.to_dict())
except JoinBudgetExceeded as error:
    print(error.status.to_dict(), error.retry_after)

print((await tg.get_join_budget()).to_dict())
```

`lookup_group` and `check_group_access` also work without a join budget. Accepted
references are bare or `@` handles, `https://t.me/handle` (also `telegram.me`),
and invite links `https://t.me/+HASH` or `https://t.me/joinchat/HASH`. Optional
`http://`, `www.`, or a bare Telegram host are accepted. Handles use 4–32 ASCII
letters, digits or underscores and begin with a letter. Invite hashes retain
their case. Links with extra paths, query strings, fragments, userinfo or ports,
phone numbers and non-Telegram URLs are rejected before opening a client.

Lookup/access can also take a nonzero integer ID or its decimal string when the
session already knows that group. A marked channel ID such as `-1000000000007`
identifies its peer kind; a positive ID is accepted only when unambiguous among
cached users/chats/channels. Unknown IDs do not trigger a dialog scan. Joining
accepts handles or invite links. References resolving to users are rejected.

| Operation | Result | What it establishes |
|---|---|---|
| `lookup_group(target)` | `GroupLookup` | Available metadata, observed `member`, whether approval/payment is indicated, and any temporary preview expiry |
| `check_group_access(target)` | `GroupAccess` | `readable` after a successful history request, `denied` after a Telegram group-access error, or `unprobed` when no usable peer is available |
| `join_group(target)` | `GroupJoin` | `joined`, `already_joined`, `requested`, `interaction_required`, or `payment_required` |

The immutable results have `to_dict()` methods containing only JSON-ready data.
`group` contains `id`, marked `peer_id`, `title`, `username` (without `@`), `kind`
(`group`, `megagroup`, `channel` or `unknown`) and `participants_count`. Missing
details are `None`: an unjoined invite preview has no invented group ID. An access
denial during resolution can have `group=None`. Existing discovery `GroupInfo`
objects retain their previous format.

Lookup never reads history or joins. Membership can be unknown; it does not prove
read access. Access issues one logical history request with limit1 when possible,
including for nonmembers of public groups. Telethon may retry that request;
each actual send is bounded and metered by an optional ReadBudget. An empty
successful reply is still readable.
`GroupAccess.readable` is `True`, `False` or `None`; `reason` is a safe denial name
or `NO_PEER`. Authentication, transport, read-quota and local validation/storage
failures raise their actual error instead of becoming a denied result. An optional
`ReadBudget` meters this history request independently of the join allowance.

Joining requires a configured `JoinBudget` and account policy, even if preflight
finds existing membership. That observation returns `already_joined` without a
mutation or charge. `requested` means approval was requested; `interaction_required`
includes `bot_id` and `query_id`, without inventing a webview URL. Paid previews
return `payment_required` without sending a join. tgdata does not pay, open a
webview or wait for approval. Only `joined` and `already_joined` have `member=True`;
other join outcomes have `member=None`. None of these replaces a separate read check.

After a join acknowledgment, available returned details are cached locally. Missing
details or failed optional cache writes do not turn that acknowledgment into failed
joining; a cache failure logs its error type only. No extra lookup is required after
the acknowledgment. An unrecognized reply raises `GroupResponseError` and retains
the attempt charge.

The join allowance has these rules:

- **Unit:** one actual admitted join request. Every SDK retry costs another
  attempt. Error replies (including already-participant or approval-requested),
  transport loss, cancellation and crashes retain the charge. No uncertainty
  refund is made. Preflight metadata and already-observed membership cost zero.
- **Storage:** all processes/clients sharing an allowance must use the same
  SQLite file on the same host. Policies are keyed by fresh authenticated account
  ID, never a session name or cached identity. The parent directory must exist;
  `:memory:` is rejected. Only IDs, policy limits and admission times are stored.
  ReadBudget can share the file; its tables and units remain separate.
- **Policy:** `configure(account_id, daily_limit)` requires explicit nonnegative
  integer limits, preserves existing usage, and provides no default rate. Zero
  stops new attempts. Lowering a limit does not erase prior attempts. The example
  limit is not a claim about avoiding Telegram restrictions.
- **Time:** each attempt expires 24 hours after admission, with no midnight reset.
  Backward clock movement cannot restore spent capacity; forward jumps advance
  expiry, so keep the host clock accurate. SQLite claims are atomic and no lock
  is held while awaiting Telegram. Synchronous operations are normally short;
  contention can block the event loop for up to the five-second busy timeout.
- **Status:** local `budget.status(account_id)` or ephemeral
  `await tg.get_join_budget()` returns `JoinBudgetStatus`. Without a budget the
  latter returns `None` without connecting. Status reports `limit`, `used`,
  `remaining`, observation/next-availability times and `retry_after` for one attempt.
  Availability assumes no new claims/policy changes; a zero cap has no timed retry.
- **Failures and boundary:** missing policy, exhausted allowance and storage
  failure stop before mutation (`JoinBudgetConfigError`, `JoinBudgetExceeded`,
  `JoinBudgetStorageError`). Configured factory clients also guard raw join calls;
  join-bearing batches and unknown wrappers raise `UnsupportedJoinRequest`.
  Separately constructed clients, other ledger files, direct private transport
  use and delayed approvals are outside this local admission guarantee.

Health events use the ephemeral call's fresh identity and a token-free invite
label. Metadata and join results cannot clear a prior group read-access denial;
a successful access read can. Local failures do not inherit unrelated Telegram
errors from the caller. New operation requests raise flood waits immediately;
the existing connection-authorization check keeps its usual wait behavior.

`test_20_group_operations` verifies SDK dispatch, results, lifecycle, health and
real SQLite concurrency offline with sockets blocked. It does not establish live
Telegram acceptance or a safe joining rate.

## Quick Start

### List All Group Chats 

This is required for finding chat id for the chat of interest. 

```python
from tgdata import TgData
import asyncio

async def main():
    # Initialize the client
    tg = TgData("config.ini")
    
    # List available groups and channels
    groups = await tg.list_groups()

    print(groups)

```

Outputs:

```
Python Devs 10012312313
```

### Group discovery — finding groups you are not in

`list_groups` shows what the account already follows. To find *new* rooms there are three
routes, all returning the same columns as `list_groups` plus `FoundVia` and `FoundBy`.
This is the quick start; the rules for running discovery on a real account without earning
a block — pacing, budgets, resumable runs, reading the output — are in
[`devdocs/guides/group_discovery.md`](devdocs/guides/group_discovery.md).

```python
from tgdata import TgData, build_search_queries

tg = TgData("config.ini")

# 1. Telegram's global search — matches room NAMES and @usernames only, never post text
df = await tg.search_groups("Анталия аренда")

# 2. Telegram's "similar channels" — rooms whose subscribers overlap the seeds', no keyword needed
df = await tg.similar_groups(["@antalyadaa", "@sprout_antalya"], rounds=2)

# 3. t.me links found in a room's recent posts — usernames only (free), or resolved
df = await tg.linked_groups(group_id, posts=100, resolve=False)

# All three on one session, deduplicated by id, in cost order
df = await tg.discover_groups(
    seeds=["@antalyadaa"],
    queries=build_search_queries(["Анталия", "Antalya"], topics=["чат", "аренда", "новости"],
                                 prefixes=["Турция"]),
    mine_links=True,
    heartbeat=lambda phase: print(phase),      # liveness ticks: search / similar / links / pause / flood-wait Ns
    found_callback=lambda row: rows.append(row),  # each room as it is found — persist as you go
)
```

**The name-only rule.** Telegram's search matches names and @usernames, never what a room
posts. A perfect room under an unrelated name is invisible to every query — that is what
`similar_groups` and `linked_groups` are for. Rooms name themselves "place + topic", so
`build_search_queries` combines your place spellings with topic words, in both word orders.

| Column | Meaning |
|---|---|
| `GroupID` … `ParticipantsCount` | the `list_groups` columns; `ParticipantsCount` is present on search and similar-channel results, absent (`NA`) on rooms resolved from links — Telegram's lookup reply does not carry it |
| `FoundVia` | `search` · `similar` · `link` — the cheapest route that found the room |
| `FoundBy` | the query, the seed's `@name`, or the source room |

Unresolved link rows carry only `Username`/`Identifier`; `GroupID` and the flags are `NA`
(the columns are nullable, so dtypes never drift). Every row with an id is already usable
by `get_messages(row['GroupID'])` — Telegram results land in the session cache with their
access hash, no dialog sync needed.

**Pacing and waits.** Every request is followed by `pace` seconds (default 2, the proven
pace for a personal account). A rate-limit wait up to `max_flood_wait` is obeyed exactly,
with heartbeat ticks so a watchdog sees life; a longer one raises `DiscoveryInterrupted`,
which carries `.found` (everything collected so far) and `.retry_after`. Two steps keep
Telethon's own handling: reading a room's posts for links, and the dialog sync for a numeric
id — a wait of up to a minute there is slept without ticks. A dropped connection
is waited out up to `max_offline` and the same request retried. Nothing found is ever lost.

**The budget.** Resolving a username to a room is the request Telegram punishes hardest —
a production account lost 8 and 21 hours of lookups to it. So every call has `max_resolve`
(default 100), `discover_groups` mines links from the first `link_sources` rooms only
(default 50), `resolve_links` is off by default (mined names come back as text, which costs
nothing), and a wait on a lookup is skipped or stopped, never waited out — for a
`linked_groups` source it ends the call with `DiscoveryInterrupted`. A truncating cap
logs a warning telling you which parameter lifts it.

**Similar channels** work on channels and megagroups, not basic groups; two rounds by
default (a third drifts off topic); the seeds themselves are never returned. Yield depends
on the account — a Premium account got 64 recommendations from one seed, a normal account
reportedly about 10.

**Links** are read from the visible text, from hyperlinked text (where channels put nearly
all of them) and from URL buttons. Accepted: `t.me/name`, `t.me/name/123`, `t.me/s/name`,
`t.me/boost/name`, `telegram.me/name`. Skipped: invite links (`t.me/+…`), `t.me/c/…`,
reserved paths, and bots (`…bot`). Discovery reads; it never joins.


### Get all messages of a chat 

You can use either the numeric chat ID or the username (if the chat has one).

**Note:** For large groups (2500+ messages), use batch processing with rate limit protection. 


```python
from tgdata import TgData
import asyncio

async def main():
    # Initialize the client
    tg = TgData("config.ini")

    # Fetch messages using numeric ID
    messages = await tg.get_messages(
        group_id=-1001234567890,  # Numeric ID
        limit=1000,
        with_progress=True
    )
    
    # OR using username (if the chat has one)
    messages = await tg.get_messages(
        group_id="@channelname",  # Username with @
        limit=1000,
        with_progress=True
    )
    
    # Export to CSV
    tg.export_messages(messages, "messages.csv")

asyncio.run(main())
```

### Get message count of a chat 



```python
from tgdata import TgData
import asyncio

async def main():
    # Initialize the client
    tg = TgData("config.ini")

    # Get count using numeric ID or username
    message_count = await tg.get_message_count(
        group_id=-1001234567890,  # Numeric ID
        # OR: group_id="@channelname"  # Username
    )

    print(message_count)

asyncio.run(main())
```




## Advanced Usage

### Using Usernames vs Numeric IDs

```python
# Both approaches work identically:

# Option 1: Using username (recommended if available)
messages = await tg.get_messages(
    group_id="@channelname",
    limit=100
)

# Option 2: Using numeric ID
messages = await tg.get_messages(
    group_id=-1001234567890,
    limit=100
)
```

A numeric id resolves only if the session can actually see the group. On a
cache miss tgdata syncs the account's chat list once and retries — so fresh
sessions work for groups the account belongs to. If the group still can't be
found it raises `tgdata.GroupAccessError` (the account is not a member, or
the id is wrong) instead of Telethon's generic "Could not find the input
entity" ValueError.

### get_messages with start_date parameter

```python
    tg = TgData("config.ini")
    # Fetch recent messages for ETL processing
    yesterday = datetime.now() - timedelta(days=1)
    messages = await tg.get_messages(
        group_id="@channelname",  # Can use username
        start_date=yesterday,
        with_progress=True
    )
    

```

### Incremental Message Fetching

```python
from tgdata import TgData

async def incremental_fetch():
    tg = TgData("config.ini")
    
    # Get the latest message ID from your storage, or db
    last_processed_id = load_checkpoint()  # Your implementation
    
    # Fetch only messages after that ID
    new_messages = await tg.get_messages(
        group_id="@channelname",  # Can use username or numeric ID
        after_id=last_processed_id
    )

    save_checkpoint(new_messages['MessageId'].max())   # Your implementation


asyncio.run(incremental_fetch())
```

### Stable message batches for storage and delivery

Use `get_message_batch` for a versioned raw snapshot that can be saved and
replayed. It keeps senderless and service messages, reads oldest first after an
exclusive cursor, and uses the same client and optional read budget. Designed
and tested with Telethon **1.45.0**.

```python
from tgdata import TgData, MessageBatch

async def prepare_batch():
    tg = TgData("config.ini")
    try:
        batch = await tg.get_message_batch(
            "@channelname", after_id=100, limit=200,
            download_media_to="out/media",  # omit for references only
        )
        manifest = batch.save("out/batches")
        replay = MessageBatch.from_json(manifest.read_bytes())
        assert replay.batch_id == batch.batch_id
        return manifest, batch.next_after_id
    finally:
        await tg.close()
```

Send the saved batch using `batch_id` for receiver deduplication, and advance
your stored cursor only after acknowledgment. If acknowledgment is lost, replay
that file: a fresh Telegram fetch can produce a different observation. Requested
photo/document files are named by their content hash; saving the manifest does
not copy its media. Blob paths are relative to `out/media` in this example.

Ordinary failures after group resolution carry the completed prefix as a
`MessageBatch` in `error.partial_result` and still raise. `next_after_id` never
passes an unfinished record/file. Local file errors keep their type without
inventing a Telegram health verdict. A secondary cleanup failure is logged by
operation/type and preserves the original error or cancellation; storage that
refuses cleanup may leave a private temporary file. The exact schema, limits,
interruption rules
and delivery protocol are in [Message batch v1](docs/message_batch_v1.md).

### Detecting & Downloading Media (photos / videos)

Every fetched message carries media-reference columns, so you can tell whether a
message has a photo **without downloading anything**:

| Column | Meaning |
|---|---|
| `MediaType` | `'photo'`, `'video'`, `'document'`, `'webpage'`, … or `None` (text-only) |
| `GroupedId` | album tag — rows sharing the same value are one multi-photo post (else `None`) |
| `MessageLink` | `t.me` deep link to the original message |
| `MediaPath` | local file path once media is downloaded (else `None`) |

> ⚠️ **`'webpage'` is NOT a real photo.** It is a link-preview thumbnail, not an
> attached image. Filter on the actual attachment type (`'photo'` / `'video'`) —
> **never** on "`MediaType` is not null" — or you will count URL previews as media.

```python
messages = await tg.get_messages(group_id="@channelname", limit=100)

# ✅ correct — real attached photos only
photos = messages[messages['MediaType'] == 'photo']

# ❌ wrong — also catches 'webpage' link previews, which are NOT real images
# media = messages[messages['MediaType'].notna()]

# how many photos each album/listing has
photos.groupby('GroupedId').size()
```

Get the actual media in one of three ways:

```python
# A) On demand — fetch references cheaply, then download only what you keep.
#    Pass every MessageId sharing a GroupedId to grab a whole album.
paths = await tg.download_media_by_id("@channelname", [12345, 12346], output_dir="media")
# -> {12345: "media/photo_....jpg", 12346: None}   (None = nothing downloadable, e.g. a webpage)

# B) During the fetch, to disk — one call returns the data AND writes the files.
messages = await tg.get_messages(
    group_id="@channelname",
    limit=100,
    download_media_to="media",   # each file's path is recorded in the MediaPath column
)

# C) During the fetch, in memory — bytes land in the DataFrame itself (MediaData column).
#    Great for small scrapes; holds every file in RAM, so scope it.
messages = await tg.get_messages(
    group_id="@channelname",
    limit=100,
    include_media=True,          # MediaData = raw bytes per row (None for text/webpage)
)
# Note: CSV export drops MediaData; JSON stores a "[Binary data]" placeholder.
```

Downloaded files (modes A and B) are named `<ChatId>_<MessageId>.<ext>` — e.g.
`1707717812_183019.jpg` — so every file is traceable to its exact message and
group from the filename alone (and won't collide across groups sharing a folder).
The precise path is also recorded per row in the `MediaPath` column.

**Re-scraping is idempotent:** if a message's file is already on disk, it is
reused, not re-downloaded — so pulling the same (or an overlapping) range again
never duplicates files or re-fetches bytes. Each message maps to exactly one file.

### Progress Monitoring

```python
async def monitor_extraction():
    tg = TgData()
    
    def progress_callback(current, total, rate):
        percent = (current / total * 100) if total else 0
        print(f"Progress: {current}/{total} ({percent:.1f}%) - {rate:.1f} msg/s")
    
    messages = await tg.get_messages(
        group_id=-1001234567890,
        limit=10000,
        progress_callback=progress_callback
    )
```

### Batch Processing for Large Groups

```python
# For groups with 100k+ messages, use batch processing with rate limit protection
async def process_large_group():
    tg = TgData("config.ini")
    
    async def save_batch(batch_df, batch_info):
        # Process each batch (e.g., save to database)
        print(f"Batch {batch_info['batch_num']}: {len(batch_df)} messages")
        batch_df.to_csv(f"batch_{batch_info['batch_num']}.csv")
    
    await tg.get_messages(
        group_id="@largechannel",  # Works with username
        batch_size=500,  # Process 500 messages at a time
        batch_callback=save_batch,
        batch_delay=2.0,  # Wait 2 seconds between batches
        rate_limit_strategy='exponential'  # Handle rate limits gracefully
    )
```

### Custom callback 




### Polling for New Messages

```python
async def poll_messages():
    tg = TgData()
    
    # Define callback for new messages
    async def process_batch(messages_df):
        print(f"Got {len(messages_df)} new messages")
        # Process messages here
    
    # Poll every 30 seconds
    await tg.poll_for_messages(
        group_id="@channelname",  # Works with username
        interval=30,
        callback=process_batch,
        max_iterations=10  # Stop after 10 polls
    )
```

Polling stops, raising the error, when polling again cannot help:
- the account is logged out, banned or restricted;
- it cannot read the group (`CHANNEL_PRIVATE`, `GroupAccessError`, ...);
- the session is not logged in (`AuthRequiredError`);
- the proxy setting is unusable (`ProxyConfigError`).

Network errors, rate-limit waits and server errors are logged and retried at
the next interval, as before. An exception raised by your callback now reaches
you; it used to be logged, and those messages were skipped.


### Real-time Message Events

```python
# Monitor messages in real-time
tg = TgData("config.ini")

@tg.on_new_message(group_id="@channelname")  # Works with username
async def handle_message(event):
    print(f"New message: {event.message.text}")

await tg.run_with_event_loop()
```


### Account health events

Every time Telegram says no to an account, tgdata tells you:
- which verdict;
- about what — the account, one kind of request, or one group;
- from which call;
- for how long;
- when it ended.

It reports and never acts. No exception changes, and nothing is paused or
retried because of a verdict.

```python
def on_health(event):                # or an async function
    print(event['verdict'], event['scope'], event['error'], event['call'])

tg = TgData("config.ini", health_callback=on_health, account_label="reader-7")
```

| Verdict | About | Telegram's names |
|---|---|---|
| `waiting` | one request type | `FLOOD_WAIT_X`, `FLOOD_PREMIUM_WAIT_X`, `SLOWMODE_WAIT_X`, `FLOOD_TEST_PHONE_WAIT_X` — with the seconds |
| `logged out` | the account | `AUTH_KEY_UNREGISTERED`, `SESSION_REVOKED`, `SESSION_EXPIRED`, `AUTH_KEY_INVALID`, `AUTH_KEY_DUPLICATED` |
| `banned` | the account | `USER_DEACTIVATED_BAN`, `USER_DEACTIVATED`, `PHONE_NUMBER_BANNED` |
| `restricted` | the account | `FROZEN_METHOD_INVALID`, `FROZEN_PARTICIPANT_MISSING`, `USER_RESTRICTED`, `PEER_FLOOD` |
| `no access` | one group | `CHANNEL_PRIVATE`, `CHAT_FORBIDDEN`, `CHANNEL_INVALID`, `USER_BANNED_IN_CHANNEL`, `CHANNEL_PUBLIC_GROUP_NA`, `CHANNEL_BANNED`, and tgdata's `GroupAccessError` |
| `unclassified` | — | any other Telegram error in the 401, 403, 406 or 420 categories, with its name (file-reference errors excepted) |
| `ok` | what recovered | — |

Nothing else is an event: bad requests (400), server errors, network and proxy
errors, a lookup that found nothing, or a session that was never logged in.

An event is a plain dict that survives `json.dumps`:

```python
{'kind': 'health', 'time': '2026-10-03T14:02:11.418523+00:00',
 'account': {'label': 'reader-7', 'session': 'reader7', 'user_id': 123456789},
 'verdict': 'no access', 'scope': 'group', 'group': 'somegroup',
 'call': 'get_messages', 'request': 'GetHistoryRequest',
 'wait_seconds': None, 'error': 'CHANNEL_PRIVATE', 'source': 'error'}
```

- **`group`** is the group as you named it: an int stays an int; `@Name` and
  t.me links become `name`.
- **`call`** is the `TgData` method; **`request`** is Telegram's request type.
- **`source`** says where it came from:
  - `error` — it ended the call, and you also get the exception;
  - `sleep` — Telethon slept through a wait of up to a minute, silently;
  - `handled` — tgdata waited it out itself;
  - `swallowed` — tgdata skipped the item and carried on, such as a
    discovery seed or link;
  - `recovery` — an `ok`.

**"ok" only when Telegram shows a condition ended:**
- **a wait** — the call that waited completes successfully;
- **logged out or banned** — a call completes that Telegram answered after
  the problem was recorded;
- **restricted** — the same, from the method that was refused;
- **no access** — the same, from a call naming that group.

A call that merely completes proves nothing. A call Telegram answered before
the problem was seen, or one that sent no request at all, never sends `ok`.

**Delivery.** The event that ends a call is delivered before its exception
reaches you. The callback may be a function or a coroutine function. If it
raises, the error is logged — first at WARNING with its traceback, then at
DEBUG — and nothing else changes. A wait Telethon sleeps through is delivered
from inside the request: a function runs there, a coroutine is scheduled.
The callback may call tgdata itself. Events caused by those calls are logged
and counted, but not delivered back to it, so it never calls itself.

**Summary.** `(await tg.health_check())['health']` holds, for this `TgData`,
in memory:
- the account's verdict, and since when;
- the open waits per request type;
- the groups without access;
- the wait count and total seconds;
- the event count;
- the last unclassified name.

**Log lines.** Each event is also logged on `tgdata.tgdata.health`, so
`log_file=` receives it: WARNING for logged out, banned, restricted and
unclassified; INFO for waiting, no access and ok. Your own logging output is
otherwise unchanged. The INFO lines in which Telethon logs its silent sleeps
stay hidden unless you asked for them.

**`validate_connection()` and `health_check()` ask Telegram directly.**
Telethon's `get_me()` answers a logout or a ban with `None` instead of an
error, so both now confirm with Telegram whether the session is logged in.
- `validate_connection()` still returns `True` or `False`. On `False` it says
  why: at WARNING with Telegram's name when it is a verdict, and as an event.
- `health_check()` marks such a connection unhealthy and names the verdict in
  `errors`, e.g. `logged out (AUTH_KEY_UNREGISTERED)`. It reports it as an
  event, and the same result's `['health']` already includes it.

**Not covered yet.**
- Signals that Telethon's background update loop only logs: an account banned
  in one channel, and waits while catching up.
- `restricted` follows Telegram's documentation; how a frozen account behaves
  when it only reads has not been observed.
- Capturing Telethon's silent sleeps relies on Telethon creating its INFO
  record: `logging.disable(logging.INFO)` or higher turns the capture off, and
  a level set on `telethon.client.users` during a call applies from the next
  call.

### Performance Tips

- The client authenticates once and keeps a single **persistent connection**,
  reused across all calls; release it with `await tg.close()` or use
  `async with TgData("config.ini") as tg:`
- Implement checkpoint logic for incremental processing
- Implement progress callbacks for visibility
- Export data incrementally for large datasets

## Requirements

- Python 3.7+
- Telethon 1.45.0 or newer within 1.x (verification target: 1.45.0)
- Telegram API credentials (not bot tokens)
- Group/channel membership

## License

MIT License - see LICENSE file for details

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.
