Metadata-Version: 2.5
Name: pinecone-datasets
Version: 1.1.0
Summary: Load datasets to explore Pinecone
Author-email: Pinecone Support <support@pinecone.io>
Maintainer-email: Amnon Catav <amnon@pinecone.io>, Roy Miara <miararoy@gmail.com>, Jen Hamon <jhamon@pinecone.io>
License-File: LICENSE
Requires-Python: <3.16,>=3.11
Requires-Dist: fsspec>=2025.2.0
Requires-Dist: gcsfs>=2025.2.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: pyarrow>=18.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: s3fs>=2025.2.0
Requires-Dist: tenacity>=9.0.0
Provides-Extra: dev
Requires-Dist: ipykernel>=6.21.1; extra == 'dev'
Requires-Dist: mypy>=1.0.1; extra == 'dev'
Requires-Dist: pdoc>=13.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest-html>=3.2.0; extra == 'dev'
Requires-Dist: pytest-xdist>=3.3.1; extra == 'dev'
Requires-Dist: pytest>=7.2.2; extra == 'dev'
Requires-Dist: ruff>=0.8.0; extra == 'dev'
Requires-Dist: tqdm>=4.67.1; extra == 'dev'
Requires-Dist: tuna>=0.5.11; extra == 'dev'
Description-Content-Type: text/markdown

# Pinecone Datasets

## Install

```bash
pip install pinecone-datasets
```

For development, see [MAINTAINERS.md](MAINTAINERS.md) for setup instructions using uv.

### Loading public datasets

Pinecone hosts a public datasets catalog, you can load a dataset by name using `list_datasets` and `load_dataset` functions. This will use the default catalog endpoint (currently GCS) to list and load datasets.

```python
from pinecone_datasets import list_datasets, load_dataset

list_datasets()
# ["quora_all-MiniLM-L6-bm25", ... ]

dataset = load_dataset("quora_all-MiniLM-L6-bm25")

dataset.head()

# Prints
# ┌─────┬───────────────────────────┬─────────────────────────────────────┬───────────────────┬──────┐
# │ id  ┆ values                    ┆ sparse_values                       ┆ metadata          ┆ blob │
# │     ┆                           ┆                                     ┆                   ┆      │
# │ str ┆ list[f32]                 ┆ struct[2]                           ┆ struct[3]         ┆      │
# ╞═════╪═══════════════════════════╪═════════════════════════════════════╪═══════════════════╪══════╡
# │ 0   ┆ [0.118014, -0.069717, ... ┆ {[470065541, 52922727, ... 22364... ┆ {2017,12,"other"} ┆ .... │
# │     ┆ 0.0060...                 ┆                                     ┆                   ┆      │
# └─────┴───────────────────────────┴─────────────────────────────────────┴───────────────────┴──────┘
```


## Usage - Accessing data

Each dataset has three main attributes, `documents`, `queries`, and `metadata` which are lazily loaded the first time they are accessed. You may notice a delay as the underlying parquet files are being downloaded the first time these attributes are accessed.

Pinecone Datasets is build on top of pandas. `documents` and `queries` are lazily-loaded pandas dataframes. This means that you can use all the pandas API to access the data. In addition, we provide some helper functions to access the data in a more convenient way. 

accessing the documents and queries dataframes is done using the `documents` and `queries` properties. These properties are lazy and will only load the data when accessed. 

```python
from pinecone_datasets import list_datasets, load_dataset

dataset = load_dataset("quora_all-MiniLM-L6-bm25")

document_df: pd.DataFrame = dataset.documents

query_df: pd.DataFrame = dataset.queries
```


## Usage - Iterating over documents

The `Dataset` class has helpers for iterating over your dataset. This is useful for upserting a dataset to an index, or for benchmarking.

```python
# List Iterator, where every list of size N Dicts with ("id", "values", "sparse_values", "metadata")
dataset.iter_documents(batch_size=n)

# Dict Iterator, where every dict has ("vector", "sparse_vector", "filter", "top_k")
dataset.iter_queries()
```

### Upserting to Index

To upsert data to the index, you should install the [Pinecone SDK](https://github.com/pinecone-io/pinecone-python-client)

```python
from pinecone import Pinecone, ServerlessSpec
from pinecone_datasets import load_dataset, list_datasets

# See what datasets are available
for ds in list_datasets():
    print(ds)

# Download embeddings data
dataset = load_dataset(dataset_name)

# Instantiate a Pinecone client using API key from app.pinecone.io
pc = Pinecone(api_key="key")

# Create a Pinecone index
index_config = pc.create_index(
    name="demo-index",
    dimension=dataset.metadata.dense_model.dimension,
    spec=ServerlessSpec(cloud="aws", region="us-east1"),
)

# Instantiate an index client
index = pc.Index(host=index_config.host)

# Upsert data from the dataset
index.upsert_from_dataframe(df=dataset.documents)
```
