Metadata-Version: 2.5
Name: vquanty
Version: 0.2.1
Summary: Client for a Taiwan market data API (wide date x security frames with local cache), with a backtester and performance attribution
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: pyarrow>=14
Requires-Dist: requests>=2.31
Provides-Extra: fast
Requires-Dist: numba>=0.59; extra == 'fast'
Provides-Extra: plot
Requires-Dist: matplotlib>=3.8; extra == 'plot'
Provides-Extra: truststore
Requires-Dist: truststore>=0.9; extra == 'truststore'
Description-Content-Type: text/markdown

# vquanty

Client for a personal Taiwan market data API. Returns wide
frames (index = date, columns = security code) and caches them locally;
a frame is downloaded again only when the server data changes.

It also ships `vquanty.backtest`, a Taiwan-equity backtester with
Brinson-Fachler performance attribution that runs on any data source
(see [Backtest and attribution](#backtest-and-attribution); the quickest way
in is [`examples/backtest.py`](examples/backtest.py)).

Access requires an API key issued by the server owner.

```bash
pip install vquanty
```

```python
import vquanty

vquanty.login("vquant-...")  # or set VQUANTY_API_KEY
from vquanty import data

close = data.get("price:收盤價")
```

Environment variables: `VQUANTY_API_KEY`, `VQUANTY_BASE_URL`,
`VQUANTY_CACHE_DIR`, `VQUANTY_CF_CLIENT_ID`, `VQUANTY_CF_CLIENT_SECRET`,
`VQUANTY_PROXY`, `VQUANTY_TRUSTSTORE` (`1` to enable).

### Proxy

```bash
pip install 'vquanty[truststore]'   # only needed for truststore
```

```python
vquanty.set_proxy("http://user:pass@proxy.corp:8080")  # every later request
vquanty.set_proxy({"https": "http://proxy.corp:8080"}, save=True)  # also saved
data.get("price:收盤價", proxy="http://proxy.corp:8080")  # this call only
vquanty.login("vquant-...", proxy="http://proxy.corp:8080")  # login through it
vquanty.set_proxy(None)  # back to direct (HTTP(S)_PROXY env vars apply again)
```

Or without code, in the environment or `.env`:

```bash
VQUANTY_PROXY=http://proxy.corp:8080
VQUANTY_TRUSTSTORE=1
```

A string proxy is used for both http and https; a dict is passed to requests
as `proxies`. A proxy set here wins over `HTTP_PROXY`/`HTTPS_PROXY`.
`data.table(..., proxy=...)` takes a per-call proxy too. `set_proxy` covers
everything that fetches through vquanty: `data`, `finlab_compat`,
`VquantySource` and `PortfolioAnalyze(data="vquanty")`. Changing the proxy
does not touch the cache; frames already on disk are only revalidated.

`set_proxy` and `login(proxy=...)` call `truststore.inject_into_ssl()` by
default, so TLS is verified against the OS certificate store, where a
company proxy's root CA usually lives (certifi does not have it). Pass
`truststore=False` to skip it, or call `vquanty.use_truststore()` on its own.
The injection is process-wide and stays on after `set_proxy(None)`.

### Caching

Each key is stored once on disk in `~/.cache/vquanty` (or
`VQUANTY_CACHE_DIR`) as `<sha1(key)>.parquet` plus a `.json` sidecar holding
the server's ETag, a fingerprint of the source table's write state.

- A later `data.get(key)` sends that ETag as `If-None-Match`. When the data
  has not changed the server answers `304 Not Modified` with no body and the
  local file is used; only a changed fingerprint transfers the frame again.
- Within one process a frame is not revalidated again for 5 minutes
  (`Config.revalidate_after`), so repeated calls make no request at all.
- If the server is unreachable or busy (503) the cached copy is returned
  with a warning.
- `data.get(key, force_download=True)` skips the cache;
  `data.clear_cache()` deletes it.

The cache is per key string, so `price:收盤價` and `price:close_d` are
cached separately even though they are the same column.

Two options ask the server for a smaller frame; each is cached as its own
entry:

```python
data.get("price:收盤價", start="2015-01-01")   # rows from 2015 only
data.get("stock_attr:stktp_c", latest=True)   # one row: each security's last value
```

Text columns arrive with empty strings as missing values.

### Long tables

Data that cannot pivot into a wide frame (versions, events, security masters)
comes back as a long table:

```python
data.tables()                          # every table, with keys and date column
data.table_info("TWN_AINVFINB")        # columns of one table
data.table("TWN_AINVFINB", columns=["coid", "mdate", "key3", "no", "a3950"])
data.table("TWN_APISALE1", start="2020-01-01")
```

`start` works on tables with a date in their primary key. Row order is not
guaranteed. Caching works as for wide frames.

### Moving off FinLab

`finlab_compat.get` takes FinLab keys and returns daily market frames with index
columns, market closures and prices on no-trade days removed:

```python
from vquanty import finlab_compat

close = finlab_compat.get("price:收盤價")
```

`monthly_revenue:當月營收` is dated at the filing deadline (the 10th of the next
month, or the next trading day), so drop `.index_str_to_date()` from FinLab code.
Companies that file late already have a value on that date, and 27% to 90% of
companies filed after it for January 2019, January 2020, January 2022 and March
2023 revenue; use `data.table("TWN_APISALE1")` when the actual filing time matters.
`scripts/compare_finlab.py` compares these frames with FinLab's cell by cell.

## Backtest and attribution

`vquanty.backtest` is a Taiwan-equity backtester (fees, tax, round lots, limit
up/down, suspensions, delisting, cash dividends, stops, phase-in rebalancing)
with Brinson-Fachler attribution linked over time with Frongello (Carino
optional). It is a port of
the `tejbt` engine from the `tej_data` project with the TEJ DuckDB loader
replaced by a small data-source contract, so any source that returns wide
frames works: the vquanty API, local files, or another vendor. On the same
data it gives the same results as `tejbt` (see [Testing](#testing)), with two
exceptions: `Portfolio(frame)` with sparse dates, where `tejbt` keeps tickers
missing from a date and holds weights fixed between dates, and the new
`empty_group="neutral"` and `PortfolioAnalyze`, which `tejbt` does not have.

```bash
pip install "vquanty[fast,plot]"   # numba JIT and matplotlib are optional
```

```python
from vquanty import backtest as bt
from vquanty.backtest.attribution import SectorScheme, TopN

data = bt.load_market_data(bt.VquantySource(), universe=["普通股"], start="2020-01-01")

wp = bt.WeightPipeline(data)
wp.shift_to_trading_day(keep="last")        # signal dates -> trading days
wp.shift_n_trading_days(1)                  # trade the day after the signal
wp.sell_no_target_weight()
wp.sell_delisted()
wp.cancel_entry_if_suspend()
weights = wp.run(df_weight)                 # long frame: date, ticker, weight

cfg = bt.SimConfig.create(
    fee=0.001425, tax=0.003, buy_at="open", sell_at="open",
    initial_value=10_000_000, round_shares=1000,
    stop_loss=0.10, trail_stop=0.15,
    limit_up_entry="defer", limit_down_exit="defer",
    liquidity_limit=bt.LiquidityLimit(days=5, method="mean", multiplier=0.3),
)
report = bt.sim(weights, data, cfg, audit=wp.audit)
report.equity                               # daily NAV, tr and pr tracks
report.trades                               # fills with reason, MAE / MFE, realized P&L
report.positions, report.weights            # daily shares / realized weights incl. cash
report.metrics()                            # CAGR, volatility, Sharpe, MDD, ...

res = report.attribution(benchmark=TopN(50, weighting="cap"), scheme=SectorScheme.auto())
res.effects                                 # allocation / selection / interaction per sector
res.total["reconciliation_error"]           # ~0: effects + residual = active return
res.plot()
```

### Fastest path from the API to a result

[`examples/backtest.py`](examples/backtest.py) runs the whole flow (load,
momentum signal, weight pipeline, three simulations, Brinson attribution)
and prints how long each step takes.

```bash
pip install "vquanty[fast,plot]"
echo "VQUANTY_API_KEY=vquant-..." > .env      # or vquanty.login("vquant-...") once
# behind a company proxy also add VQUANTY_PROXY=... and VQUANTY_TRUSTSTORE=1 (see Proxy)
python examples/backtest.py
```

Measured with 2015 onwards, all 普通股 (2,881 days × 2,143 stocks), client
and server on one machine:

| step | first run | later runs |
|---|---|---|
| `load_market_data(VquantySource())` | 28 s (17 frames, about 215 MB) | 8 s (17 `304 Not Modified` checks, then building the arrays) |
| `load_market_data(VquantySource(start="2014-01-01"))` | 19 s (about 155 MB) | 5 s |
| first `sim` | 4.8 s (numba compiles the engine, cached on disk) | 0.4 s |
| each further `sim` | 0.2–0.3 s | 0.2–0.3 s |
| `report.attribution(...)` | 0.1 s | 0.1 s |
| `data.select(tickers=..., start=...)` | 0.02 s | 0.02 s |

Most of a first run is the server building each frame once; it keeps the
result, so after anyone has loaded a frame a new device only pays the
download (9 s instead of 28 s above, plus transfer time over the network).

What keeps it fast:

1. **Load once, run many.** In a notebook or script, build `data` once and
   pass it to every `bt.sim` and `report.attribution`. For a sub-period or
   a sub-universe use `data.select(start=..., end=..., tickers=...)`, which
   slices in memory, instead of loading again.
2. **Keep the cache.** Frames live in `~/.cache/vquanty`
   (`VQUANTY_CACHE_DIR`); only frames whose server fingerprint changed are
   downloaded again (see [Caching](#caching)). Avoid `force_download=True`
   and `data.clear_cache()` in normal use.
3. **Install `vquanty[fast]`.** Without numba the engine runs as plain
   Python and each `sim` is much slower.
4. **Fetch only the history you need.** `VquantySource(start=...)` downloads
   price and flag rows from that date on. Start it about a year before the
   backtest so forward-filling and the first dividend yields have earlier
   rows to use. Against the full history (2015 backtest, `start="2014-01-01"`):
   147 stocks with no price after 2014 drop out, and two stocks suspended
   since before 2014 (2432, 4415) have no price until they trade again.
   The default `start=None` gives the same result as `tejbt`.
5. **Skip fields you do not use.** Each price field is one frame of about
   20 MB:

   ```python
   src = bt.VquantySource(
       keys={"volume": None, "amount": None},  # needed by LiquidityLimit, buy_at="vwap"
   )
   ```

   Keep `adjfac_pr` and the dividends (`dividend_keys`) whenever you use
   `round_shares`: without them lots are priced on back-adjusted prices and
   the result is too optimistic. All seven trading flags come in one small
   frame, so skipping flags saves nothing.
6. **Reuse a loaded `MarketData` offline.** It pickles (0.1 s to load back,
   about 700 MB for the table above):

   ```python
   import pickle
   pickle.dump(data, open("md.pkl", "wb"), protocol=5)
   data = pickle.load(open("md.pkl", "rb"))
   ```

   A pickle is a snapshot: it does not check the server for new data, so
   reload from the API when you want the latest prices.

### Data-source contract

A source is any object with `frame(field) -> DataFrame | None`. Every frame is
wide: index = date, columns = ticker. Return `None` for fields you do not
have. `bt.FIELDS` lists them:

| field | meaning | needed by |
|---|---|---|
| `close` | raw close | **required** |
| `open`, `high`, `low` | raw OHLC (default: close) | `buy_at`/`sell_at`, touched stops |
| `adj_close` or `adjfac` | total-return adjusted close / factor | `tr` track, attribution |
| `adj_close_pr` or `adjfac_pr` | ex-rights-only adjustment | used with cash dividends |
| `volume` (thousand shares), `amount`, `vwap` | liquidity | `LiquidityLimit`, `buy_at="vwap"` |
| `mktcap` | market cap | `TopN(weighting="cap")`, `SelfReweight("cap")` |
| `limit_up`, `limit_down`, `limit_locked_open`, `suspend`, `attention`, `disposal`, `full_delivery` | bool flags | trade blocking, weight steps |
| `listed` | bool flag (default: has a close price) | informational |

Optional methods:

- `securities()`: one row per ticker (`ticker` column or index). Optional
  columns: `sector` (used by `SectorScheme.auto()`), `tejind`/`tejindnm`/
  `tseind`/`tseindnm`, `stktp_c` (for `universe=`), `delist_date`.
- `cash_dividends()`: long frame `ticker, ex_date, dividend[, pay_date, prev_close]`.
- `trading_days()`: exchange calendar; dates outside it are dropped.

Loading rules (the same as `tejbt`'s artifact builder):

- `tr` = raw × `adjfac`. Without cash dividends, the engine's trading basis
  `pr` equals `tr` (dividends already in the price). With `cash_dividends()`,
  `pr` = raw × `adjfac_pr`, falling back to `adjfac` where `adjfac_pr` is
  missing, and dividends are paid as cash per `SimConfig(div_reinvest=...)`.
- A raw price of 0 means no trade. Prices, volume and market cap are
  forward-filled over gaps (suspensions) but stay NaN before a ticker's
  first and after its last data row.
- Without a `delist_date` column, a ticker whose data stops before the last
  date is treated as delisted on its last data day (`infer_delist=False`
  turns this off). A non-null `delist_date` force-closes the position on the
  last data day.
- `start` / `end` slice after filling, so a window starting mid-suspension
  still carries the last price.

Three ways to plug in data:

```python
# 1. frames in memory
src = bt.FrameSource(close=close, adj_close=adj, mktcap=cap, securities=sec)

# 2. the vquanty API, with any key overridden or switched off
src = bt.VquantySource(keys={"amount": None}, securities=my_sector_table)

# 3. your own class
from pathlib import Path
import pandas as pd

class ParquetSource:
    def __init__(self, root):
        self.root = Path(root)

    def frame(self, field):
        path = self.root / f"{field}.parquet"
        return pd.read_parquet(path) if path.exists() else None

    def securities(self):
        return pd.read_parquet(self.root / "securities.parquet")

data = bt.load_market_data(ParquetSource("~/tw_data"))
```

`VquantySource` defaults: `price:close_d`, `open_d`, `high_d`, `low_d`,
`adjfac`, `adjfac_a`, `vol`, `amt`, `mktcap`; all seven flags from one
bitmask frame, `etl:trade_flags`, built on the server from `stock_attr:*_fg`
(`flags_key=None` reads the seven `stock_attr` string frames instead);
`stktp_c` and `sector` (`main_ind_c`) from `stock_attr` with `latest=True`.
The API serves only tables keyed by (security, date), so three inputs that
`tejbt` reads from other TEJ tables are rebuilt:

| input | tejbt source | `VquantySource` | switch |
|---|---|---|---|
| cash dividends | `TWN_ADIV` | `TWN_APIDV1` 除息日 / 現金股利 / 股息發放日; a re-announced distribution (same 盈餘分配迄日 and type) counts once; missing pay date = ex-date + 1 month | `dividend_keys=None` |
| trading calendar | `TWN_TRADEDAY_TWSE` | dates with at least 20 tickers priced (drops TWSE holidays on which only foreign-listed securities trade) | `min_tickers_per_day` |
| industry | `TWN_ANPRCSTD` TEJ / TSE industry | TEJ 主產業別 with TWSE and OTC labels merged (`M2300 電子工業` + `OTC23 OTC 電子類` → `電子工業`); raw label in `sector_raw` | `merge_markets=False` |
| delisting | `TWN_ANPRCSTD` delist date | inferred from where prices stop | `infer_delist=False` |

Checked against `tejbt` on the same database (普通股, 2005 onwards): dates,
tickers, every price matrix, the trading flags and delist positions are
identical. Of the cash-dividend events that can reach a position (ex-date
on or after the stock's first priced day), 40 of about 23,500 differ: 31
because `TWN_ADIV` divides by its own 前一日收盤價 while the loader uses the
previous close in the price table, and 9 that `TWN_APIDV1` does not list.
(`TWN_ADIV` also has yields on ex-dates before a stock's first priced day,
which no backtest can hold.) Backtests of 14 configurations match `tejbt`
to within 0.016% a year, most exactly; from 2019 a monthly top-50
market-cap strategy matches exactly, and from 2005 its daily NAV differs by
at most 0.001%.
`SectorScheme.tse()` and `.coarse3()` need TSE industry columns and so need
a `securities` frame you supply.

The first load downloads 17 frames (about 215 MB); later loads only
revalidate them (see [Fastest path](#fastest-path-from-the-api-to-a-result)).

### Attribution

- Benchmarks: `TopN(n, weighting="cap"|"equal")`, `Portfolio(weights frame or
  another Report)`, `SelfReweight("equal"|"cap")` (same holdings, different
  weighting).
- `Portfolio(frame)` takes target weights, long (`date, ticker, weight`) or
  wide. Each date is a full target: a ticker not listed on it has weight 0, and
  holdings drift with prices until the next date. A date that is not a trading
  day takes effect on the next one. Weights above 1 are rescaled with a warning.
- Grouping: `SectorScheme.auto()` (default: `sector` column, then TEJ industry,
  else per-security contribution), `.column(name)`, `.tej()`, `.tse()`,
  `.coarse3()` (金融 / 電子 / 非金電), `.custom(mapping)`, `.security()`.
- Linking: `linking="frongello"` (default, the same as FactSet PA) or
  `"carino"` (what `tejbt` uses; the parity script passes it). Both make the
  linked effects plus residual add up to the geometric active return. Carino
  scales each day's effect by a log-return ratio; Frongello grows each day's
  effect by the portfolio return before it, so effects on later days of a
  rising period count for more.
- Effects use start-of-day weights; the gap between buy-and-hold and the real
  NAV (costs, intraday fills) goes to a single `residual`, so linked effects
  plus residual reconcile to the NAV's active return.
- A group only one side holds has no return on the other side. By default
  (`empty_group="zero"`, same as `tejbt`) that return is 0, which puts large
  offsetting amounts in the effects of a concentrated portfolio: their sum is
  right, the split is not meaningful. `report.attribution(..., empty_group="neutral")`
  uses the benchmark's group return for a group the portfolio skips (allocation
  only) and the benchmark's total return for a group the benchmark skips (all
  in interaction). Cash stays at 0, so holding cash stays an allocation effect.
  The sum of the effects, the residual and `res.group_returns()` do not change.
- Sector labels are a static snapshot, not point-in-time; `res.warnings` says so.

```python
res.effects                  # per group: allocation, selection, interaction, total
res.total                    # active / portfolio / benchmark return, effect sums, residual
res.timeseries("selection")  # daily linked effect per group ("total", "residual" also work)
res.cumulative("total")      # cumulative contribution per group
res.group_returns()          # w_p, w_b, r_p, r_b per group and day
res.stock_effects            # per-security contribution (SectorScheme.security())
res.plot(); res.plot_group_weights(side="b")
```

### Attribution from two weight frames

`PortfolioAnalyze` compares two sets of weights without running a backtest:

```python
import vquanty
from vquanty.backtest.attribution import PortfolioAnalyze

vquanty.login("vquant-...")

pa = PortfolioAnalyze()
pa.data = "vquanty"
pa.portfolio = df_port             # date, ticker, weight
pa.benchmark = df_bench
pa.group = None                    # per security

res = pa.run(start_date="2026-01-01", end_date="2026-10-02")
res.effects                        # contribution per security, plus cash
res.stock_contributions()          # per day and security
res.total                          # active / portfolio / benchmark return
```

- `pa.data`: `"vquanty"` (loads prices for the tickers in the two frames, any
  security type), a wide price frame, a `MarketData` (fastest when running many
  times; it must cover the tickers), or a source. A
  price frame needs no login and must be adjusted for dividends, because
  returns come from those prices alone. A raw close shows every ex-dividend
  day as a loss.
- `pa.group`: `None` per security, a `securities` column such as `"sector"`
  (raises if the data has none), a `{ticker: group}` mapping, or a `SectorScheme`.
- Each date in a frame is a full target held from that day's close: a ticker
  not listed has weight 0, weights below 1 leave cash, and holdings drift with
  prices until the next date.
- `start_date` is the first day whose return counts. Holdings on that day come
  from the last earlier target, drifted to that day. Without it, the period
  starts from the close of the first date both frames cover.
- `pa.empty_group` defaults to `"neutral"` (see [Attribution](#attribution)).
- There are no costs, round lots or trading limits, so the residual is 0. For
  those, run `bt.sim` and use `report.attribution(benchmark=Portfolio(bench))`.

### Other tools

- Rules instead of config: `bt.sim(..., rules=[bt.StopLoss(0.1),
  bt.TrailingStop(0.15, activation=0.05), bt.TakeProfit(0.3),
  bt.SignalExit(bool_frame), bt.PortfolioRiskOff(exposure_series)])`.
- Phase-in rebalancing: `SimConfig.create(transition_progress=3,
  transition_type="waf"|"blend", transition_block="none"|"postpone"|"catchup")`.
- Weight transforms on long frames: `bt.factor_enhanced`, `bt.capped_weights`,
  `bt.top_n_max_weight`. They keep `tejbt`'s behaviour exactly, including two
  quirks: with `factor_ascending=False` a larger factor value gets a smaller
  tilt, and `capped_weights` pools the excess over every date in the frame
  (call it one date at a time to keep per-date sums at 1).
- Set `NUMBA_DISABLE_JIT=1` to run the engine as plain Python for debugging.

## Testing

```bash
uv run pytest                                  # unit tests (numba JIT)
NUMBA_DISABLE_JIT=1 uv run pytest              # same tests on the pure-Python engine
```

Parity with the original `tejbt` on real TEJ data (needs a `tej_data`
checkout with its DuckDB; about 20 s):

```bash
VQUANTY_PARITY=1 TEJ_DATA=/path/to/tej_data uv run --with duckdb pytest tests/test_parity_tejbt.py
```

`scripts/parity_tejbt.py` reads one DuckDB snapshot, builds `tejbt`'s
MarketData through its own artifact pipeline and vquanty's through
`FrameSource`, then compares every MarketData array, the weight pipeline,
14 backtests (2 signals × 7 configs covering lots, stops, liquidity limits,
vwap/close fills, every dividend mode and phase-in mode), declarative
rules, the weight transforms, and 8 attribution setups (every benchmark
type and grouping scheme, `tr` and `pr` bases). Result on 2019 to the
latest local snapshot: 259 checks, 0 differences.
