Metadata-Version: 2.4
Name: space-memory
Version: 0.1.0
Summary: Personal memory system — semantic search over 149K messages. pip install, importable, no cloud required.
Author-email: Abhishek Srivastava <bitsabhiphiinverse@gmail.com>
License: MIT
Keywords: memory,semantic-search,personal-data,sqlite,rag,embeddings
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: sentence-transformers
Requires-Dist: fastapi
Requires-Dist: uvicorn[standard]
Dynamic: requires-python

# digital-history — Space Memory System

> **Memory is reconstruction, not retrieval.**

149,515 messages across WhatsApp, Telegram, Claude, ChatGPT, Gmail — plus GitHub commits, articles, legal records. All searchable. All reconstructable. Fully offline once the server is up.

---

## Prerequisites

Two things must be running:

```bash
# 1. Ollama (local LLM — run once, stays up)
ollama serve

# 2. Query server (D1 semantic search — run once per session)
dh server
# or: cd ~/Documents/digital-history && python3.13 -m uvicorn query_server:app --host 127.0.0.1 --port 8000
```

The query server loads ~250K vectors from D1 on startup (~60 sec first time), then stays in memory. HF_TOKEN is loaded automatically from `~/.secret_tokens`.

---

## Quick commands — use `dh`

```bash
dh reconstruct "what was happening with amrita late 2024"   # full pipeline: CARM + tone + Ollama
dh reconstruct -i                                           # interactive mode
dh chat "what kind of projects do I keep building?"         # fine-tuned Phi-3 + SQLite search
dh query "amrita conversations"                             # raw D1 semantic search
dh server                                                   # start query server on :8000
```

## Or via bazinga --kb-d1

```bash
bazinga --kb-sources                                                         # check D1 reachable
bazinga --kb "your query" --kb-d1                                           # raw D1 search
bazinga --kb "your query" --kb-d1 --summarize --local                       # LLM summary
bazinga --kb "your query" --kb-d1 --summarize --voice --local               # reconstruction in your voice
bazinga --kb "your query" --kb-d1 --summarize --voice --memory-model        # fine-tuned Phi-3
```

### Good queries

```bash
dh reconstruct "what was I building in early 2025"
dh reconstruct "what was happening with amrita late 2024"
dh reconstruct "ZK proofs consciousness substrate"
dh reconstruct "darmiyan lambda-g work"
dh reconstruct "what was I going through mid 2024"
dh reconstruct "phi crystal fractal dimension"
```

---

## How it works (the pipeline)

```
Your query
    │
    ▼
query_server (localhost:8000)
    └── Roberts-16D cosine search over 496K embeddings in D1
    └── Returns top-15 real messages with dates and sources
    │
    ▼
8-axis tonal scoring (deterministic, no LLM)
    └── assertion, concrete, self, present, warmth, certainty, action, technical
    └── Produces tonal target for generation
    │
    ▼
Generator (--local = Ollama llama3, --memory-model = fine-tuned Phi-3)
    └── Prompt = fragments + tonal target + reconstruction role
    └── "Speak in his voice. Not a summary. A memory."
    │
    ▼
Reconstructed memory in Space's voice
```

---

## Data sources — what each command searches

| Command | Data source | Notes |
|---------|-------------|-------|
| `dh reconstruct` | D1 digital-history only | 149K messages, semantic search, clean |
| `dh chat` | local_copy.sqlite (keyword) | offline, same data, no embeddings |
| `dh query` | D1 digital-history only | raw semantic search |
| `bazinga --kb-d1` | D1 digital-history only | via query_server HTTP |
| `bazinga --kb` (no --kb-d1) | ~/.bazinga index | Gmail exports, GDrive, Mac files — noisy |
| `bazinga --ask / --chat` | ~/.bazinga RAG + cloud LLMs | general purpose, not memory-specific |

**Rule:** for memory reconstruction, always use `--kb-d1` or `dh reconstruct`. Never use plain `bazinga --kb` without `--kb-d1` for personal memory queries — the `~/.bazinga` index contains noisy local file scraps that are already in D1 at higher quality.

---

## Lower-level interface — query.py directly

For raw search without bazinga (needs wrangler + Cloudflare auth):

```bash
cd ~/Documents/digital-history

python3 query.py "when did I talk about moving cities"
python3 query.py "amrita conversations about future" --k 30
python3 query.py "authentication bug" --source commit
python3 query.py "ZK proof ideas" --source message
python3 query.py -i    # interactive mode
```

---

## What's in the database

| Table | Rows | What |
|-------|------|------|
| messages | 149,515 | WhatsApp, Telegram, Claude, ChatGPT, Gmail |
| entity_embeddings | 496,665 | 384-dim semantic vectors |
| conversations | 4,790 | Thread groupings |
| github_commits | 3,067 | Every git commit |
| github_repos | 79 | Repos |
| authored_content | 78 | Articles written |
| transactions | 63 | Financial records |
| legal_records | 30 | Legal docs |
| people | 10 | Amrita, Aishwarya, Manasi, Mini, Asmita... |

**DB size:** 1.23 GB on Cloudflare D1 (paid tier, 10GB limit)

### Message sources

| source_id | Source |
|-----------|--------|
| 1 | Gmail |
| 2 | WhatsApp |
| 3 | Telegram |
| 4 | Claude |
| 5 | ChatGPT |

---

## Files

| File | What |
|------|------|
| `query.py` | Direct D1 semantic search (Roberts-16D + 384D refine) |
| `query_server.py` | FastAPI wrapper — serves query.py over localhost:8000 |
| `chat.py` | Prototype RAG + fine-tuned model (superseded by bazinga --kb-d1) |
| `finetune.py` | QLoRA training script (Phi-3-mini, Apple MPS) |
| `build_dataset_v2.py` | Dataset builder (pulls from D1, saves local SQLite) |
| `local_copy.sqlite` | Offline copy of all 149K messages (176MB, not in git) |
| `finetune_dataset_v2.jsonl` | Training dataset (1,133 weekly windows, not in git) |
| `abhishek-memory-model/` | Fine-tuned LoRA adapter (34MB, not in git) |
| `PRD.md` | Full product spec |

---

## Local vs Cloud

| Operation | Internet needed? |
|-----------|-----------------|
| `bazinga --kb-d1 --voice --local` | Yes — D1 embeddings fetched at server startup |
| After query_server loads | No — vectors in RAM, Ollama is local |
| `--memory-model` (fine-tuned Phi-3) | No — model is local |
| Re-training | No — dataset and model are local |
| Adding new data to D1 | Yes |

---

## The fine-tuned model

- **Base:** Phi-3-mini-4k-instruct (Microsoft, 3.8B params)
- **Adapter:** QLoRA rank=16, 8.9M trainable params, 34MB on disk
- **Trained on:** 1,133 weekly windows — all messages, chronological, cross-source
- **Training:** ~7 hours on Apple MPS
- **Use:** `bazinga --kb "..." --kb-d1 --summarize --voice --memory-model`
- **Note:** loads the 7GB base model + 34MB adapter. Takes ~30 sec on first use.

---

## Repo

**bazinga integration:** `~/github-repos-bitsabhi/bazinga-indeed/bazinga/kb.py` + `cli/_core.py`
**GitHub:** https://github.com/0x-auth/digital-history (private)
**bazinga-indeed:** https://github.com/0x-auth/bazinga-indeed

---

*φ = 1.618 · Genesis prime: 137 · Memory is reconstruction, not retrieval*
