Metadata-Version: 2.5
Name: ad-data-pf
Version: 0.6.0
Summary: Tables and figures for a job-ad amenities project, run on Statistics Denmark microdata
Author-email: AskerNC <hms467@econ.ku.dk>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: loguru>=0.7
Requires-Dist: matplotlib>=3.10
Requires-Dist: numpy>=2.3
Requires-Dist: polars>=1.43
Requires-Dist: pyfixest>=0.60
Requires-Dist: pytest>=9.1.1
Requires-Dist: scikit-learn>=1.7
Description-Content-Type: text/markdown

# ad-data-pf

Tables (and later figures) for a research project on job-ad amenities. The functions are written and tested
locally against simulated data, then `pip install`ed on and run on the project data..

All numbers in this package come from simulated data.

## Use

```python
from ad_data_pf import validation as val, save_tex, save_fig, setup_log

setup_log(log_folder, 'descriptives')      # logs the package version first
frames = {'All AKU\nrespondents': aku, 'All\njob ads': jobads,
          'Linked\njob ads': jobads_linked, 'Job ads linked\nwith AKU response': aku_linked}
save_tex(val.desc_table(frames), out / 'tab_desc.tex')
save_tex(val.omission_table(aku_linked, ci='wilson'), out / 'tab_omission_wilson.tex')
save_fig(val.prevalence_figure(aku_linked, aku), out / 'fig_sensitivity_occ_prevalence.png')
save_fig(val.coverage_figure({'All job ads': jobads,
                              'Job ads with text': jobads.filter(c.has_text),
                              'New hires (BFL)': new_hires}),
         out / 'fig_postings_coverage.png')
```

`new_hires` has one row per new employment spell in BFL, with an integer
`year` (of the start of the spell) and `public` (Boolean, null where the
sector is unknown).

Tables return a bare `tabular`; figures return a matplotlib `Figure`.
Figures are drawn in `style/paper.mplstyle` with the colours of
`style/ggplot2.mplstyle`. `save_fig(fig, path, slides=True)` draws a
package figure again in `style/slides.mplstyle`, for slides. Draw a
figure of your own inside `with figure_style():` (or
`figure_style(slides=True)`) to match. While a style is in use,
matplotlib and fontTools log only warnings and errors.
Defaults (variables, labels, panel titles) sit at the top of each module and
in `ad_data_pf.labels`, and can be overridden per call. See the docstrings,
and `examples/` for every exhibit and for custom tables.

## Arguments

Each function fixes the layout of its exhibit: the panels, and the statistic
each one computes. The arguments say only which columns go in and what they
are called, in three plain forms:

- `{column: label}`, one row per column:
  `t_vars={'any_T': 'Any non-standard hours', 'work_night': 'Night work'}`.
- `{column: splits}`, one row per group of a column:
  - `{code: label}` matches the leading characters of a string column
    (`{'1': 'Managers', '2': 'Managers'}` on the 6-digit `disco`) and the
    value of any other (`{True: 'Public', False: 'Private'}`). Codes may
    share a label.
  - `[(upper bound, label), ...]` bins a numeric column, left-closed, `None`
    for no bound (`SIZE`).
- `{label: frame}` where the columns, rows or points are samples
  (`desc_table`, `robustness_table`, `ladder_figure`). The label is the
  column header, row label or axis label, and `\n` breaks it into lines.
  Make each frame with polars before the call:
  `al.filter(c.months_since_hire <= 3)`, `al.with_columns(any_T=c.any_T_sometimes)`.

Panel titles can be renamed and `{}` drops a panel. There are no
package-specific spec objects and no polars expressions as arguments.

`validation` makes these exhibits:

| Exhibit | Function |
| --- | --- |
| Job ads and new hires per year, share public | `coverage_figure` (one line per frame) |
| Samples and base rates | `desc_table` (panels units, T, M, composition; any number of frames) |
| Confusion table | `confusion_table` (shares of N by default, or counts) |
| Validation measures by type | `omission_table` (sensitivity, false omission and discovery rates, Youden's J, F1, prevalence; Wilson or cluster-bootstrap intervals) |
| Measure by occupational prevalence | `prevalence_figure` (`measure='sens'` or `'fom'`) |
| Measures by group | `heterogeneity_table` (one panel per split column) |
| Measures by linkage criterion | `ladder_figure` |
| Robustness | `robustness_table` (one row per sample) |

Inputs are checked. The following raise an error:

- T or M that is not Boolean, or M with nulls;
- a row whose columns exist in no frame;
- a split that no row falls in, or a group with no pairs;
- T and M labels that do not pair up.

A null T is a question not asked or answered, not a no: the pair is left
out of that T's measures and shares (logged), so each measure is of the
pairs whose T is known. On the server, cut the AKU answers with
`c(col) > cut`, which keeps nulls, not with `bool_cut`, which makes them
False.

Cells resting on fewer than 5 observations are left empty (DST disclosure).
In splits, a null or NaN counts as missing: in polars NaN compares above
every number, so it would otherwise land in the top bin.

`predictive` makes the exhibits of `text/Notes/20261016_predictive_power_mockup.tex`:
how well posting topics predict what workers report in AKU, who is hired, and
how posting varies within and between workplaces.

| Exhibit | Function |
| --- | --- |
| First stage of T on M, fixed effects added step by step | `first_stage_table` |
| First stage of every T/M pair | `first_stage_figure` |
| Out-of-sample AUC of four nested models | `auc_figure` |
| Topics that predict T (post-LASSO) | `top_topics_table` |
| First stage at the workplace level, between and within | `workplace_figure`, `workplace_table` |
| Topic groups and the gender of the hire | `gender_table` |
| Topic groups and four types of hire (mothers, ...) | `hire_types_figure` |
| Where variation in posting lies | `variance_figure` |
| Persistence of workplace posting shares | `persistence_table` |
| Workplaces around posting shocks, against placebo shocks | `shock_figure` |

Regressions are OLS (linear probability models for Boolean outcomes) by
pyfixest, with fixed effects absorbed and standard errors clustered on the
workplace (`cvrnr` x `prodnr`). A row with a null or NaN in any variable of a
regression drops from it, and so do singletons of its fixed effects; the N
printed is the N used. The tables report the R2 of the fixed effects alone,
the partial R2 of the topics (their share of what the fixed effects leave),
and the joint F of the topics. The AUC models are linear probability models
cross-fitted by workplace; the full-text one is a LASSO (`LassoCV`), and
categorical variables are target-encoded within each training fold.

Topics are the compacted topics of `topic_hierarchy/topics_compaction.py`,
one Boolean column per topic named as the topic with spaces as underscores
(`labels.TOPICS`). `pr.add_flags(df)` adds the flags of `labels.TOPIC_FLAGS`,
a flag being any of its topics: the six non-overlapping groups of
`labels.TOPIC_GROUPS` (flexible hours, part-time hours, positive work
environment, irregular hours, greedy job, physical work) and the topics
matched to AKU variables (`labels.OTHER_M_VARS`).

The frames are the four the validation exhibits take, with more columns.
`predictive.FRAMES` lists what each exhibit reads, and
`ad_data_pf.check_frames(frames)` logs every missing column and wrong type
at once. After prep:

| Frame | One row is | Columns |
| --- | --- | --- |
| `aku` | an AKU respondent | `cvrnr`, `prodnr`, `year`, `disco`, the T flags of `T_VARS` and `OTHER_T_VARS` (Boolean) |
| `jobads` | a posting | `cvrnr`, `prodnr`, `adnr`, `year`, `disco`, every topic of `TOPICS` and `wfh_dummy` (Boolean, no nulls) |
| `jobads_linked` | a posting-hire pair | as `jobads`, and `pnr`, `public`, `empl_sum`, `nace`, `female`, `child_u6` |
| `aku_linked` | a posting-respondent pair | as `aku`, and the topics, `wfh_dummy`, `disco_jobad`, `public`, `empl_sum`, `nace`, the controls `female`, `age` |

The T flags of `OTHER_T_VARS` are columns of the cleaned AKU (`load_aku.py`
on the server), which codes each answer from 0 to 1. A flag is True at the
top answer: `flexibility` is `VARITIME` 1, deciding fully when the working
day starts and ends (asked from 2021); `work_home` is `HOMEWORK` (`HOMEWK`
before 2021) "usually"; and `supervisor` is `SUPVISOR`. Cut them with the
hours flags, keeping nulls (a cut from 0.5 to below 1 keeps the top answer):

```python
*[(c(k) > par.aku_d_cut).alias(k) for k in [*AKU_VARIABLES, *OTHER_T_VARS]]
```

On the server:

```python
import ad_data_pf
from ad_data_pf import predictive as pr

ad_data_pf.check_frames({'aku': aku, 'jobads': jobads, 'jobads_linked': jobads_linked,
                         'aku_linked': aku_linked, 'new_hires': new_hires})
jobads = pr.add_flags(jobads)
jl = pr.add_flags(jobads_linked).with_columns(mother_u6=c.female & c.child_u6,
                                              father_u6=~c.female & c.child_u6)
al = pr.add_flags(aku_linked)
pairs = al.filter(c.unique_spell_link_3d == 1)     # the analysis link
save_tex(pr.first_stage_table(pairs, al), out / 'tab_first_stage.tex')
save_tex(pr.gender_table(jl), out / 'tab_gender_hire.tex')
save_fig(pr.shock_figure(jobads, aku, jl), out / 'fig_posting_shocks.png')
```

`examples/predictive/code/run_predictive.py` makes every exhibit. On the
full data `auc_figure` fits a LASSO 5 times for each of 10 outcomes, on at
most `max_n=100_000` rows each; lower `max_n` or `folds` for a quick run.

 [CHANGELOG.md](CHANGELOG.md) lists what to
retype when moving to a new version.

## Tests on the server data

The tests are installed with the package, and pytest with them. After prep,
run them from the server script:

```python
ad_data_pf.run_tests({'aku': aku, 'jobads': jobads, 'jobads_linked': jobads_linked,
                      'aku_linked': aku_linked, 'new_hires': new_hires})
```

It logs pytest's output as it comes, so it shows on screen and is in the log
file of `setup_log`, and returns 0 if every test passes. The summary at the
end names each failed and skipped test, with the reason for a skip. The frames are those after prep with all links kept, named as
`sim.simulate()` names them. They are written to a temporary folder, deleted
afterwards, and the tests run in a new process. Further arguments go to
pytest: `run_tests(frames, '-m', 'data')` runs the data tests only, and `'-x'`
stops at the first failure. It skips the test that compiles the tables with
latexmk, since LaTeX on the server differs; that test runs locally.

The validation data tests are skipped when one of the four frames is not
passed, and those of `coverage_figure` when `new_hires` is not. The
predictive data tests start with `check_frames`: while the frames lack a
column predictive reads, that one test fails with the list and the others
are skipped. To run one module's tests only, add `'-k', 'validation'` (or
`'predictive'`).

The data tests (`tests/test_validation_data.py`, marked `data`) run on these
frames, and on simulated ones when there are none, as in CI. They check what
the functions assume of their input but cannot check themselves, and build
every exhibit with its defaults:

- T and M flags are Boolean, M without nulls and T known in most rows (a
  failure shows the null share by year), and `any_T` and `any_M` are the
  any of the flags in `labels.py`;
- `disco` and `disco_jobad` are 6-digit strings;
- `public` is Boolean and `empl_sum` a number of at least 0;
- every pair has a `cvrnr` and a `prodnr`, the bootstrap's clusters;
- `jobads_linked` is part of `jobads`, and `aku_linked` of `aku` and of
  `jobads_linked`;
- every default exhibit builds, its counts add up, and its intervals
  contain its estimates.

`tests/test_predictive_data.py` checks that:

- the frames have every column of `predictive.FRAMES`, of the right type;
- no topic of `labels.TOPICS` has nulls in `jobads`, `jobads_linked` and
  `aku_linked` (the full-text models would drop such a row without a word);
- `flexibility` is null before 2021 and mostly known after;
- `disco` and `disco_jobad` are 6-digit strings, and workplace and year are
  never null;
- `jobads_linked` is part of `jobads`, and `aku_linked` of `jobads_linked`;
- every exhibit builds, the first stage without controls is the difference
  in shares, the four hire types sum to zero, and the R2 of the fixed
  effects is at most the R2 of the model.

The other tests check the code on fake data. They give the same result
anywhere, so on the server they test the environment.

A failed data test shows the offending values: the most common bad codes,
or the first rows without a match.

## Layout

```text
src/ad_data_pf/
├── functions/          one module per topic: validation.py, predictive.py, ...
│                       imported from the top: from ad_data_pf import validation
├── style/              matplotlib styles: paper, slides, ggplot2 (colours)
├── simulation/         fake data per module (validation.py, predictive.py)
│                       and the files in data/
├── examples/
│   ├── validation/     code/ runs every table on the fake data;
│   │                   logs/ and output/ hold what it writes
│   └── predictive/
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames
```

`simulation/`, `examples/` and `tests/` are installed with the package. To find them:

```bash
python -c "import ad_data_pf, pathlib; print(pathlib.Path(ad_data_pf.__file__).parent)"
python <that folder>/examples/validation/code/run_validation.py <output folder>
```

Every example also runs in the VS Code Interactive window (or any Jupyter
kernel), whole or cell by cell (`# %%`). It writes to its own `logs/` and
`output/`, or to `./ad_data_pf_examples/<name>/` in the working directory if
the installed package is read-only.

## Adding a module

1. `functions/<name>.py`: importable at once as `ad_data_pf.<name>`;
   arguments in the forms of [Arguments](#arguments)
2. `simulation/<name>.py`: `simulate()`, `save()`, `load()`
3. `examples/<name>/code/run_<name>.py`, plus `logs/.gitkeep`. Take the
   output folder from `root = example_root('<name>')`, never from `sys.argv`
   or `__file__`, and split sections with `# %%`. `tests/test_examples.py`
   runs every example as a script and as in a Jupyter kernel.
4. `tests/test_<name>.py` for the code, and `tests/test_<name>_data.py`,
   marked `data`, for what the functions assume of real frames but cannot
   check themselves (see [Tests on the server data](#tests-on-the-server-data)).
   Its frames fixture takes `data_folder`: `sim.load(data_folder)`, or
   `sim.simulate()` when it is None.

## Development

The package lives in `ad_data_pf/` of the (private) project repository; run
everything below from that folder.

```bash
uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/ad_data_pf/simulation/validation.py   # save the fake data to simulation/data
uv run pytest --data src/ad_data_pf/simulation/data     # data tests on saved frames
```

The lower bounds in `pyproject.toml` are older versions. To
test against exactly those, use a separate environment so `uv.lock` is not
rewritten:

```bash
cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock
```

GitHub Actions does both on every push that touches `ad_data_pf/`
(`.github/workflows/ad_data_pf_test.yml` at the repository root).

## Release

Bump `version` in `pyproject.toml` and add an entry to
[CHANGELOG.md](CHANGELOG.md). For each change to a call, give the old and
the new call, since the server script is retyped by hand. Commit, then
push a tag `ad_data_pf-v<version>`; GitHub Actions checks that the tag matches the
version, runs the tests, builds and publishes to PyPI (trusted publishing,
`.github/workflows/ad_data_pf_publish.yml`):

```bash
git tag ad_data_pf-v0.1.1 && git push origin ad_data_pf-v0.1.1
```

The same tag runs `.github/workflows/ad_data_pf_publish_1.yml`, which publishes the
same code under a second name, `ad_data_pf_1`. The DST server caches a package when it
is installed and won't update that name for 24 hours. On the server, install an exact
version under whichever name hasn't been installed in the last 24 hours:

```bash
pip install ad-data-pf==X.Y.Z
pip install ad_data_pf_1==X.Y.Z
```

Both install the import `ad_data_pf`, and the most recent install is the one used.
