Metadata-Version: 2.5
Name: x-api-scraper-haystack
Version: 0.1.0
Summary: Haystack RAG components for Twitter search and public user timelines through TwexAPI. Not affiliated with X Corp.
Project-URL: Documentation, https://github.com/twexapi-dev/x-api-scraper-haystack
Project-URL: Homepage, https://docs.twexapi.io
Project-URL: Issues, https://github.com/twexapi-dev/x-api-scraper-haystack/issues
Project-URL: Source, https://github.com/twexapi-dev/x-api-scraper-haystack
Author: TwexAPI
License-Expression: MIT
License-File: LICENSE
Keywords: haystack,haystack-ai,rag,tweet-search,twexapi,twitter-api,twitter-search,user-timeline,x-api
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: haystack-ai>=2.12.0
Requires-Dist: httpx>=0.28.1
Description-Content-Type: text/markdown

# Twitter search and user timeline components for Haystack

TwexAPI is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.

Search Twitter and fetch public user timelines in Haystack RAG pipelines. Each TwexAPI result becomes a Haystack `Document`.

## Components

| Task | Haystack component | TwexAPI route | Output |
| --- | --- | --- | --- |
| Search tweets for RAG | `TwexApiTweetSearch` | `POST /twitter/advanced_search/page` | Matching posts as `Document` objects |
| Retrieve a user's timeline | `TwexApiUserTweetsFetcher` | `POST /twitter/{screen_name}/timeline/page` | Recent user posts as `Document` objects |

```python
from haystack_integrations.components.websearch.x_api_scraper import (
    TwexApiTweetSearch,
    TwexApiUserTweetsFetcher,
)
```

Both components read `X_API_SCRAPER_KEY` by default and accept a Haystack `Secret`. They send `Authorization: Bearer`. Set `base_url` for another TwexAPI-compatible endpoint. Results include `documents`, `links`, `has_more`, and `next_cursor`.

Search returns up to 20 tweets per page. Timeline `top_k` maps to the page `count` (1-100). Use REST or an SDK for follower pagination or approved posting.

## Install

```bash
pip install x-api-scraper-haystack
```

## Build a Haystack RAG pipeline with Twitter search

### Search posts

```python
from haystack import Pipeline
from haystack.utils import Secret
from haystack_integrations.components.websearch.x_api_scraper import TwexApiTweetSearch

search = TwexApiTweetSearch(api_key=Secret.from_env_var("X_API_SCRAPER_KEY"), top_k=10)

pipeline = Pipeline()
pipeline.add_component("x_search", search)

result = pipeline.run({"x_search": {"query": "haystack ai"}})
documents = result["x_search"]["documents"]
```

### Fetch user posts

```python
from haystack.utils import Secret
from haystack_integrations.components.websearch.x_api_scraper import (
    TwexApiUserTweetsFetcher,
)

fetcher = TwexApiUserTweetsFetcher(api_key=Secret.from_env_var("X_API_SCRAPER_KEY"))

result = fetcher.run(screen_name="elonmusk")
documents = result["documents"]
```

## Document mapping

Each tweet becomes a Haystack `Document` with this mapping.

- `Document.content`: `full_text` or `text`, or an empty string when both are missing
- `Document.meta["endpoint"]`: `search` or `timeline`
- `Document.meta`: available `id`, `url`, and `created_at` source values
- `Document.meta["author"]`: available author identity and verification data
- `Document.meta`: available like, retweet, reply, quote, view, and bookmark counts

## License

MIT. TwexAPI is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.
