Metadata-Version: 2.4
Name: opencode-pine2pyne
Version: 0.2.1
Summary: Pine Script v6 to PyneCore Python transpiler
Author: rubycell
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/rubycell/pine2pyne
Project-URL: Repository, https://github.com/rubycell/pine2pyne
Keywords: pinescript,tradingview,transpiler,pynecore,backtesting
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: runtime
Requires-Dist: opencode-pyneruntime[cli]>=6.9.1; extra == "runtime"
Dynamic: license-file

# PyneCore - Pine Script to Python Transpiler

> **Backlog & known bugs:** [Projects board #10](https://github.com/users/rubycell/projects/10)
> — open work is in **Backlog** (prioritised P0/P1/P2), shipped work in **Done**.
> Confirmed unfixed transpiler bugs live there too; several are pinned by strict
> `xfail` tests (see [Running Tests](#running-tests)).

## Installation (New Computer)

### Fastest: the sandbox-setup skill

From the repo root, run the bundled setup script (idempotent — safe to re-run):

```bash
bash ~/.claude/skills/opencode-pine2pyne-setup/setup.sh   # from inside this clone; the skill lives in ~/.claude/skills
```

It builds `.venv`, installs everything below, detects `PINE_CPU_COUNT` /
`PINE_MAX_WORKERS`, seeds a demo dataset, and runs a smoke test that ends in
`smoke test : PASS` / `READY`. In Claude Code you can also invoke it as
`/sandbox-setup`.

### Manual (equivalent steps)

```bash
# From the repo root:
python3 -m venv .venv                          # 1. virtualenv
.venv/bin/pip install -e .                      # 2. this transpiler (editable)
.venv/bin/pip install 'opencode-pyneruntime[cli]>=6.9.1'  # 3. PyneCore runtime + `pyne` CLI
.venv/bin/pip install optuna numpy              # 4. optimizer deps (not pulled in by the runtime)
source .venv/bin/activate                       # 5. activate (Linux/macOS)
# .venv\Scripts\activate                        #    (Windows)
```

Verify the install:

```bash
.venv/bin/python -c "import pynecore, pine2pyne, optuna, numpy; print('all ok')"
```

**Note**: All commands below assume you are at the repo root with the venv activated.

## How to Convert Pine Script to Python

### Single File Conversion

```bash
python -m pine2pyne path/to/script.pine -o path/to/output.py
```

### Batch Conversion

```bash
python -m pine2pyne "sample/pinescript/*.pine" -o workdir/scripts/
```

## Running Tests

There are **three** independent test systems. They answer different questions,
so a green run of one says nothing about the others.

| System | Question it answers | Runtime |
|---|---|---|
| **1. Unit tests** (`tests/`, pytest) | Are the transpiler's internals correct? | ~5 s |
| **2. Sample suites** (`tools/test_all_*.py`) | Does every corpus sample transpile and run? | ~35 s |
| **3. Ground truth** (`tools/*_ground_truth*`) | Does it produce the *same numbers* as before? | minutes |

### 1. Unit tests — pytest

2,000+ tests over `pine2pyne/` (93% line / 86% branch coverage).

```bash
.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py -q

# with coverage
.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py \
    --cov=pine2pyne --cov-report=term-missing
```

> `--ignore` is required: `tests/test_screener_regression.py` imports a
> `screener` module from a **sibling repo** that is absent here, and fails at
> collection — which aborts the whole run. Tracked as a P0 card on
> [board #10](https://github.com/users/rubycell/projects/10).

Some tests are `@pytest.mark.xfail(strict=True)` and pin **confirmed unfixed
bugs**, each with a card on [board #10](https://github.com/users/rubycell/projects/10).
Strict means fixing a bug turns its marker into a failure until the marker is
removed, so nothing regresses silently in either direction.

The former P0 — codegen dropping Pine grouping, so `10 - (5 - 3)` became
`10 - 5 - 3` — is fixed (card #6) and guarded by
`tests/test_codegen_operator_grouping.py`. See [docs/test_plan.md](docs/test_plan.md) for the full bug inventory and the
measured coverage/mutation results.

### 2. Sample suites — end-to-end transpile + run

Transpiles every `.pine` in the corpus and runs it through `pyne`.

```bash
python tools/test_all_samples.py          # sample/pinescript  (~411 files)
python tools/test_all_sample_pds.py       # sample_pds         (~328 files)

python tools/test_all_samples.py "ex_001*"      # filter by pattern
python tools/test_all_samples.py --timeout 30   # default 15s
python tools/test_all_samples.py --verbose      # stderr on failure
python tools/test_all_samples.py -j 0           # all CPU cores
python tools/test_all_samples.py --run-dir /tmp/run1   # private scripts/, output/ and report files (card #70)
```

- Source: `sample/pinescript/`, `sample_pds/` · Output: `workdir/scripts/`
- `--run-dir DIR` keeps a run's transpiled scripts (`DIR/scripts/`), pyne outputs (`DIR/output/`, passed to `pyne`
  as explicit `--plot/--strat/--trade` paths) and `test_results*.{json,txt}` reports in `DIR`, so two runs, or a run
  and an old report, cannot read each other's files. Without it nothing changes: the shared `workdir/` paths and
  the repo-root reports. `--retry-failed` with `--run-dir` reads `DIR`'s own report.
- Data: `workdir/data/demo.ohlcv` · Results: `test_results{,_pds}.{json,txt}`

**Per-sample expectations** live in `tools/sample_expectations.toml`:

```toml
[expds_156_Out_of_bounds_index]
expect_error = "index_error"                    # MUST raise; running clean = FAIL

[ex_300_ticker_modify]
requires_feeds = { D = "demo", "2D" = "demo_2D" }   # -> --security D=demo …
```

`expect_error` marks Pine documentation samples that exist to *demonstrate* an
error — they are reported as `XFAIL` and count as passes. `requires_feeds`
supplies extra OHLCV feeds for samples calling `request.security()` at a
timeframe other than the chart's; pynecore deliberately will not resample the
chart feed, so the feed must be explicit. Generate one with:

```bash
cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D
```

Shared logic for both suites lives in `tools/_suite_common.py` — edit it there,
not in each harness.

### 3. Ground truth — numeric regression

Checks every freshly generated output file (~1,650 across ~550 samples) against
its recorded digest. This is the only system that catches *silently wrong
numbers*: output that runs fine but computes something different.

```bash
python tools/generate_ground_truth.py     # (re)build baselines
python tools/compare_ground_truth.py      # check current output vs baselines
python tools/compare_ground_truth.py --suite sample_pds -j 12
```

**The baseline is `ground_truth/digests.json`** (card #62). It holds a sha256 and a
line count for each output file, plus the sha256 of the data file. Git tracks
only it and `ground_truth/manifest.json` (under 1 MB together). The output
CSVs themselves (~920 MB) are gitignored: the generator still copies them to
`ground_truth/{sample,sample_pds}/` on your machine for diffing. Before writing a
new baseline, it moves the previous CSVs to `backup/deleteable/ground_truth_csv_<time>/`
(nothing is deleted; clear old ones out by hand). `manifest.json` lists its samples sorted by
suite, then filename (card #68), so a one-suite rebuild (`--suite sample`) and a full rebuild on the same data
give the same samples in the same order, and a rebuild never reorders the tracked file; only `generated_at`
changes by design.

- **Exit codes.** The compare exits 1 when `digests.json` is missing or empty, and
  2 with `DATA CHANGED` when the data file is not the one the baseline was made
  on (restore it, see *Test data* below, or rebaseline).
- **Every sample runs in its own directory** (card #70): the generator and the compare make one fresh run
  root, `backup/deleteable/ground_truth_runs/<tool>-<time>/` (or an empty `--run-dir DIR`; a directory that already
  holds files is refused), with `<suite>/<sample>/<data file>/` inside it holding the script, its 10B settings and the three
  CSVs, and collect only from there. An earlier run's file of the same name cannot be read as this run's; a run
  that exits 0 and writes nothing is `no_output` (a failure), never `ok` or `NEW PASS`. Run roots are never
  deleted (see the paragraph below on where they go).
- **Where run roots go, and the disk they use.** The default run root of the generator, the compare and the
  checker (`ground_truth_check.py capture` and `compare` each make one) is
  `backup/deleteable/ground_truth_runs/<tool>-<time>/`, next to the other disposable output, and each run leaves
  one tree of roughly 0.9 GB there (script, settings and three CSVs per sample). A generate run also copies the
  collected files into `ground_truth/<suite>/` (gitignored). Nothing here is deleted by a tool: when you want the
  space back, move old run roots (and the `gt_compare_<time>/` scratch copies) to the USB backup disk, never `rm`
  them. An explicit `--run-dir DIR` is used as given (it must be empty).
- **A `REGRESSION` line** names the sample and file whose digest changed. To see
  *what* changed, diff that sample's CSV in the run root's `<suite>/<sample>/<data file>/`
  against a baseline copy. That copy is in `ground_truth/<suite>/<sample>/` if you
  generated on the old commit, or, after a later generate, in the newest
  `backup/deleteable/ground_truth_csv_<time>/`.
- **`--suite sample` or `--suite sample_pds`** rebuilds one suite and keeps the
  other suite's digests and manifest entries. It refuses (exit 1) unless a
  baseline made on the same data already exists.
- **Expected noise:** 11 `sample_pds` samples call unseeded `math.random()` and
  differ on every run (expds_022, 037, 039, 043, 045, 081, 091, 092, 104, 114,
  115).

The old end-to-end shell wrapper is retired (card #66): it read `ground_truth/samples/`,
`ground_truth/samples_pds/` and `ground_truth/strategies/`, which no tool writes, so its
`--quick` mode passed with zero checks. Its default and `--samples` modes are now
`python tools/compare_ground_truth.py` (`--suite sample`, `sample_pds` or `both`; it reads
`workdir/data/VN30F1M_15m.ohlcv` by default, see *Test data*); its `--quick` and
`--strategies` modes are `python tools/ground_truth_check.py capture`, then `compare`.

`tools/ground_truth_check.py` (`capture` / `compare`) is a separate, local check of the
bigtest strategies and the sample suites. It writes only to `ground_truth_check/` at the
repo root (gitignored), never to `ground_truth/`, so it cannot change the tracked baseline
above. `capture --force` moves the old `ground_truth_check/` to
`backup/deleteable/ground_truth_check_<time>/` (nothing is deleted) and touches nothing else. Both actions run the
bigtest strategies and the two suite harnesses (`--run-dir`) in one fresh run root under
`backup/deleteable/ground_truth_runs/` (`--run-dir DIR` names an empty one), read only that run's files, and fail the capture when a suite wrote no report
or a bigtest file did not transpile. `compare` writes its
scratch copy of the current outputs to `backup/deleteable/gt_compare_<time>/` and never deletes it (card #67); each
run leaves one such folder, which you move to the backup disk when you want the space back. A capture made
before card #63 sits in `ground_truth/` (`transpiled/`, `strategy_runs/`, `test_suite_*`);
`compare` no longer reads it and says so. Run `capture` again.

> Regenerate baselines only when you have *confirmed* the new output is correct
> — otherwise a bug gets frozen in as the expected result.

Both tools run every sample on `workdir/data/VN30F1M_15m.ohlcv`, with a
`<script>.toml` that sets `initial_capital` to 10,000,000,000, and with one
per-sample timeout (120 s). Both values live in `tools/_ground_truth_common.py`.
At pynecore's default capital of 1,000,000, one VN30F1M contract costs more than
equity, so nearly every strategy would record zero trades (card #57).

### Test data

`workdir/data/` is gitignored, so a fresh clone has none and the sample suites
abort with `Data file not found`. Restore the exact dataset the recorded results
were measured against:

```bash
mkdir -p workdir/data
# Sample suites: CCXT BTCUSDT 1D, plus the 2D/5D feeds `requires_feeds` names
git show 227f3d2^:workdir/data/demo.ohlcv > workdir/data/demo.ohlcv
cp tests/fixtures/data/demo.toml workdir/data/demo.toml
(cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D && pyne data aggregate data/demo.ohlcv -tf 5D)
# Ground truth: VN30F1M 15m
git show 5d63d21:workdir/data/VN30F1M_15m.ohlcv > workdir/data/VN30F1M_15m.ohlcv
cp tests/fixtures/data/VN30F1M_15m.toml workdir/data/VN30F1M_15m.toml
```

The `.ohlcv` files come from those commits (sha256 `67e827cc1a5f71ed…` for
demo, `e9ef984b09eaa6d2…` for VN30F1M). Take the `.toml` files from
`tests/fixtures/data/`, not from the commits. The committed ones number weekdays
1..7, and pynecore rejects day 7 (it numbers 0 = Monday … 6 = Sunday). They also
declare one session per week, which leaves session-anchored values such as
`ta.vwap` NaN or weekly (card #57).

(`/sandbox-setup` instead seeds a small *synthetic* fixture — fine for a smoke
test, but results will not match the recorded rates.)

## For Claude Code / LLM Development

See `CLAUDE.md` in this directory for transpiler architecture, output format specs, optimizer documentation, and CSV rounding rules. See `pine2pyne/README.md` for transpiler internals and transformation rules.

## Optimizing Strategies

Quick start: `pyne optimize script.py data.ohlcv params.json -n 20`

See [Optimizer.md](./Optimizer.md) for full documentation (parameter JSON format, parallel execution, output files).

### Distributed Optimization (multi-machine)

Two tools distribute `pyne optimize` across SSH clusters:

| Tool | Model | Best for |
|------|-------|----------|
| `pyne-dynamic.sh` | Flat-queue (on-demand dispatch) | Multi-variant runs, heterogeneous clusters, long jobs |
| `pyne-parallel.sh` | Static pre-assignment | Quick jobs on similar-speed machines |

```bash
# Dynamic (recommended): flat-queue, pre-syncs once, auto-adapts to machine speed
# Run from workdir/ directory:
../tools/pyne-dynamic.sh scripts/strategy.py data/data.ohlcv optimize_variants/ \
  -H ../tools/machines.txt -C 24 --name my_run --output-dir runs/output/

# Static: pre-assigns chunks by core count
./tools/pyne-parallel.sh scripts/strategy.py data/data.ohlcv optimize.json \
  -H tools/machines.txt --sync

# Check progress / collect results
../tools/pyne-dynamic.sh --status
../tools/pyne-dynamic.sh --collect
```

Both use the same `machines.txt` format. See `CLAUDE.md` for cluster setup, machine file format, and troubleshooting.
