Metadata-Version: 2.5
Name: occu-choice-pf
Version: 0.9.0
Summary: Tables for a survey experiment on perceived wage differences and job search, run on Statistics Denmark microdata
Author-email: AskerNC <hms467@econ.ku.dk>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: loguru>=0.7
Requires-Dist: matplotlib>=3.9
Requires-Dist: numpy>=2.3
Requires-Dist: pandas>=2.2
Requires-Dist: polars>=1.43
Requires-Dist: pyfixest>=0.60.0
Requires-Dist: pytest>=9.1.1
Requires-Dist: scipy>=1.18.1
Description-Content-Type: text/markdown

# occu-choice-pf

Tables for the survey experiment of *Perceived Wage Differences and Early
Career Job Search Choices*.
The functions are written and tested locally against simulated data, then run on the actual data.
All numbers in this package come from simulated data.

## Use

```python
from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS
```

Tables return a bare `tabular` in the style of the paper's tables (the
`\sym` stars of esttab included); `save_tex` writes it and logs it, so it can
be read before an export request. Defaults (variables, labels) sit in
`occu_choice_pf.labels` and can be overridden per call. See the docstrings,
and `examples/` for every exhibit and for custom tables.

## The table of controls

`reg_control_table(long)` regresses each outcome on `T_vj_vij_job`, one
panel per outcome, with one table column per entry of `cols_control`. Every
column has `coefs` (reported), `common_control`, and the controls of its
entry. The defaults (`labels.COLS_CONTROL`, `labels.COMMON_CONTROL`) give:

| Column | Controls | `cols_control` entry |
| --- | --- | --- |
| (1) | `vj_vij_job` and the person fixed effect (`common_control`) | `{}` |
| (2) | + `vj_vij_job` squared and cubed | `POWERS` |
| (3) | + pre-treatment controls times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS}` |
| (4) | (3) + education field dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **EDUC_FIELD}` |
| (5) | (3) + DISCED (`disced_6d`) dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **DISCED}` |

A control is `{check-mark row label: terms}`, and a term is a column
(`'age'`), a column made into a dummy for each value (`'i.age'`, as Stata's
`i.`), or a polars expression (`c.vj_vij_job ** 2`). Your own columns:

```python
from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])
```

Every term of `cols_control` is times `vj_vij_job`, except one wrapped in
`reg.plain()`, which enters its column as it is; an `i.<column>` there is a
fixed effect absorbed in that column only:

```python
from occu_choice_pf.labels import JOB
reg.reg_control_table(long, cols_control=[{}, {'Job type FE': reg.plain('i.' + JOB)}])
```

The arguments, the check-mark rows and the input checks are in
`help(reg.reg_control_table)`; the estimation, and what drops out of a
column (all of it logged), in `help(reg)`. `reg.reg_control_fits(long, ...)`
returns the pyfixest fits behind a table, `{outcome: [fit per column]}`, to
look into with `.summary()` or `pf.etable`. `examples/` has custom calls: a
subsample, other regressors, clustering and a second fixed effect.

The columns of a panel can rest on different rows: the Poisson regression
with the DISCED dummies drops the separated rows of persons alone in their
DISCED, which the other columns keep. `common_sample=True` estimates every
column on the rows all of them keep, so that the columns differ in their
controls only; `reg.reg_control_samples(long, ...)`, with the arguments of
the table, returns `long` with a Boolean column `in_<outcome>_<column>` per
regression, True in the rows it is estimated on:

```python
save_tex(reg.reg_control_table(long, common_sample=True), out / 'tab_poisson_common.tex')
s = reg.reg_control_samples(long)
s.filter(c.in_appl_job_5)                      # the rows of panel B, column (5)
```

A separated row leaves the person's other rows in: with `vj_vij_job`
winsorized, a person alone in their DISCED whose rows with applications are
tied at the bound keeps those. `drop_persons='separated'` drops a person
with a separated row whole, and `drop_persons='any'` a person who loses any
row, for a missing value, separation or pyfixest; both take the arguments
above too:

```python
save_tex(reg.reg_control_table(long, drop_persons='separated'), out / 'tab_poisson_sep.tex')
save_tex(reg.reg_control_table(long, drop_persons='any'), out / 'tab_poisson_whole.tex')
```

`share_range=(low, high)` keeps the persons of a major (`share_educ`, by
default `disced_6d`) only where its treated share among the persons of the
regression, after separation and pyfixest's drops, is from `low` to `high`.
The persons of the other majors are dropped whole and the regression refit
until every major left is in the range. Each table column and outcome has
its own sample, and so its own shares:

```python
save_tex(reg.reg_control_table(long, share_range=(0.1, 0.9)), out / 'tab_poisson_share.tex')
```

`totals={outcome: row label}` adds a row between Individuals and N in that
outcome's panel: the outcome summed over the rows each column is estimated
on, left empty where fewer than 5 persons (`min_n`) have an outcome above 0:

```python
reg.reg_control_table(long, totals={'appl_job': 'Applications'})   # the applications made
```

## The treated share within majors

With the major dummies times `vj_vij_job` (columns 4 and 5), the treatment
effect is estimated within majors: a major whose persons are all treated, or
all in the control group, adds next to nothing, and a major weighs more the
closer its treated share is to one half. Without them, a treated share that
differs between majors moves the estimate where the slope on `vj_vij_job`
differs between majors. `treatment_shares` shows how large the persons'
majors are and how far the treated share of their major is from what random
assignment gives (`T` permuted across the persons, 1000 draws, no strata).
Three table columns or a plot per column of `labels.EDUC`: major field and
DISCED. The functions take the persons frame, one row per person, and raise
on a person with more than one row:

```python
from occu_choice_pf import save_fig
from occu_choice_pf import treatment_shares as ts

save_tex(ts.percentile_table(persons), out / 'tab_share_percentiles.tex')
save_fig(ts.share_figure(persons), out / 'fig_share_hist.png')
ts.share_frame(persons, 'disced_6d')  # one row per major: on the server only
```

- `percentile_table`: percentiles over persons of the size of their major
  and of its treated share (data and random assignment). Each percentile is
  the mean of the 9 persons around it (`window`), never one person's value.
  Below them, the standard deviation over persons of the treated share of
  their major (data and random assignment) and its *p*-value under random
  assignment: a permutation chi-squared test of treatment by major.
- `share_figure`: a histogram of the persons by the treated share of their
  major (0, ten intervals, 1), with the mean and the 95% range of random
  assignment; a matplotlib figure, the numbers plotted in the log.
  `save_fig` writes it, a PNG at 300 dpi unless the path ends in `.pdf`.

The persons are those of the frame given, so the persons of a regression,
`persons.join(s.filter(c.in_appl_job_5), on='pnr', how='semi')` with
`s = reg.reg_control_samples(long)`, give the shares within it.

Disclosure: no number shown rests on fewer than `min_n` (5) persons, nor can
one be worked out from the others, the persons treated (in the paper's
balance table) included. A count below 5, 0 included, reads <5. Percentiles
are left empty where their 9-person windows overlap or leave 1-4 persons
outside; the shares, the test and random assignment, which depends on the
persons treated, with fewer than 5 treated or untreated. A bar of fewer than 5
persons reads <5; the bars at 0 and 1 are left out where the persons in
all-treated or all-control majors, or the treated or untreated of the
others, are 1-4; where the bars left out add up to 1-4 persons, the smallest
interval bar is left out too. A 0 can still be worked out. `share_frame`
lists every major, however small, and is for the server only.

## The attrition table

`attrition.attrition_table(df)` asks whether getting to the end of the
survey depends on the treatment, alone, by major or by misperception. T
is drawn when a person starts the survey, and the pages before the
treatment page are the same in both arms, so the attrition that can bias
the estimates is among those who reached it; reaching it, among everyone who
started, is a placebo. It takes one row per person:

```python
from occu_choice_pf import attrition as at
from occu_choice_pf.labels import COLS_PLACEBO, TEST_PLACEBO

save_tex(at.attrition_table(started.filter(c.T_page == 1)), out / 'tab_attrition.tex')
save_tex(at.attrition_table(started, {'T_page': 'Reached the treatment page'}, TEST_PLACEBO,
                            COLS_PLACEBO), out / 'tab_attrition_placebo.tex')
```

Linear probability models (pyfixest `feols`, standard errors clustered on
the person), one panel per outcome (`labels.ATTRITION_OUTCOMES`:
`full_answer` and `primary_sample`, each 0 or 1 for every person). Each
table column has two entries, one in each of two lists of the same length:

- `test_coefs=[[terms], ...]`: the terms the column tests. They are the top
  rows of the table, a number by its coefficient and a fixed effect by a
  check mark, and under them the p-value of the test that they are all 0
  and which test it is.
- `cols_control=[{label: terms}, ...]`: the column's background controls,
  check-mark rows at the bottom, as in `reg_control_table`.

Every term enters as it is written, and nothing is added: a column, a
polars expression, `'i.<column>'` for a fixed effect per value, or
`'i.<a>#<b>'` for one per value of a and b together (`'i.disced_6d#T'`, the
major-by-arm cells). Interactions with T are written out, and their main
effects go in the background. The defaults (`labels.TEST_COEFS`,
`labels.COLS_ATTRITION`):

| Column | Tested (top) | Background (bottom) |
| --- | --- | --- |
| (1) | `T` | none |
| (2) | `T` | major FE |
| (3) | `T` | major FE, pre-treatment controls |
| (4) | `'i.disced_6d#T'`, major by arm | T, major FE, pre-treatment controls |
| (5) | T times the misperception, and times its absolute value | T, the misperception and its absolute value, major FE, pre-treatment controls |

`test_labels={term: row label}` (`labels.TEST_LABELS`) names the top rows,
a column or fixed effect by how it is written, an expression by its alias.
Pre-treatment controls as fixed effects are `i.<column>` terms:

```python
from occu_choice_pf.labels import MAJOR_FE
controls = {**MAJOR_FE, 'Pre-treatment controls': [*linear, 'i.age']}
save_tex(at.attrition_table(exposed, test_coefs=[['T']], cols_control=[controls]), out / 'tab.tex')
# do the majors predict attrition? Their joint F test
at.attrition_table(exposed, test_coefs=[['i.disced_6d']], cols_control=[{}])
```

The joint test is one of three:

- **Wald**, tested terms that are all numbers: pyfixest's Wald test with
  the robust variance (chi-squared; one term, its t-test, as its stars).
- **F**, tested terms with a fixed effect: the classical F test of the fit
  with them against the background alone, each fit's parameters the rank of
  its regressors with a dummy per fixed-effect group, so that a fixed effect
  nested in or overlapping another counts once (the robust variance is
  degenerate where a group has one person).
- **Permutation**: T permuted within majors (`strata`, DISCED by default)
  `draws` times (1000), every term written with T computed anew, the
  statistic the residual sum of squares without the tested terms over that
  with them. Exact under the sharp null that T changes no one's outcome. The
  tested terms must be written with T (`T`, an expression of T, `i.<a>#T`): a
  column made from T before the call would not change with the draws.

Where the tested terms add nothing beside the background controls (pyfixest
drops them as collinear), no test is run and its cells are empty; the log
says so.

`permutation` chooses: `None` (the default) gives the permutation test to a
column whose tested terms include a fixed effect written with T, the
major-by-arm cells, where the analytical tests over-reject: some major has
one person per arm. With no effect of T, the robust Wald test of the
major-by-arm dummies rejected at 5% in 10 to 23% of simulated draws, the
classical F test in up to 9%, and the robust Wald test of the misperception
times T in 2.5 to 7%. `True` or `False` sets every column, a list each.
The permutation takes a few minutes per outcome at 1000 draws.

Every column shows its R² and adjusted R² (pyfixest's), so column (2)
against (1) says how much the majors predict attrition. With hundreds of
small majors the fixed effects raise the R² by about majors / persons even
where they predict nothing, which the adjusted R² takes out: it counts each
fixed-effect group as a parameter. A background control constant within the
groups of a tested fixed effect (the major FE and T within the major-by-arm
cells) is left out of the fit with the tested terms, so that it counts once;
pyfixest otherwise counts overlapping fixed effects twice. Persons alone in
a fixed effect are kept, so the persons of a column are the same with and
without the tested terms. A person missing a term that enters as a number
leaves that column (the log says how many); in a fixed effect,
`i.<column>`, a missing value (null or NaN) is a group of its own and the
person stays. The log has every tested coefficient.

Disclosure: a column is left empty unless 5 persons or more (`min_n`) of
each arm have each value of the outcome, as the control mean and a
coefficient on T alone give the share of each arm; Individuals reads <5
below 5. A tested fixed effect's check mark shows anyway.

## The table of wage gap underestimation

`underestimation_gap.gap_table(rows)` is Table 3 of the paper: one row per
subsample, `rows={label: frame}`, each frame the extra-long frame or a
subsample of it, made with polars before the call.

```python
from occu_choice_pf import underestimation_gap as ug

clean = {'d_vj_vij': c.d_vj_vij_clean, 'd_Vj_Vij_rate': c.d_Vj_Vij_rate_clean}
rows = {
    'Full sample': elong,
    **{f'Actual log wage gap $>$ {x}': elong.filter(c.d_vj > x) for x in [0.05, 0.1]},
    'Full sample, correcting reporting errors': elong.with_columns(**clean),
    'Reweighted for non-participation': elong,
}
weights = {'Reweighted for non-participation': 'weight_invited'}
save_tex(ug.gap_table(rows, weights), out / 'tab_estimation_gaps.tex')
save_tex(ug.gap_table(rows, weights, bootstrap=1000), out / 'tab_estimation_gaps_se.tex')
```

The rows are numbered. `bootstrap=B` adds standard errors from B draws of
persons with replacement. `compare=[(label, label)]` puts the p-value of the
difference between two rows under them (`$p$-value of difference (1)-(2)`),
and a line under that. The rows need not be next to each other, and a row
may be in several pairs: `compare=[('Men', 'Women'), ('Men', 'Women, reweighted to (1)')]`
puts both p-values under the three rows (the rows from the first to the last
of a pair are a block, joined with the blocks it overlaps). `p_at_end=True`
puts all the p-values in the last rows of the table instead.
`share_column` sets column 7: `'underestimator'` (the default), the share of
individuals who are underestimators; `'zero_gap'`, the share of job pairs
with no perceived wage gap (`d_Vij == 0`); a list of both,
`['underestimator', 'zero_gap']`, columns 7 and 8 in that order; or `None`,
no column 7. The frames need not have a column not shown. A frame with NaN
raises, since a filter such as `c.d_vj > 0.05` keeps NaN: make it missing in
prep, `elong = elong.with_columns(pl.col(pl.Float64).fill_nan(None))`. `ps_group`
reweights one group to the other with a propensity score model estimated
anew in each draw, as Table C1 of the parental transmission paper: it adds
`ps_weight` (reweighted to the other group) and `ps_support` (1 for persons
with a weight) to every row, to name in `weights`:

```python
no, yes = elong.filter(c.par_anyjobtype == 0), elong.filter(c.par_anyjobtype == 1)
rows = {'Neither': no, 'Either': yes, 'Neither, weightable': no, 'Either, reweighted to (3)': yes}
ug.gap_table(rows, {'Neither, weightable': 'ps_support', 'Either, reweighted to (3)': 'ps_weight'},
             bootstrap=1000, ps_group='par_anyjobtype',
             compare=[('Neither', 'Either'), ('Neither, weightable', 'Either, reweighted to (3)')])
```

The model is a logit of `ps_group` on `ps_xs` with the fixed effects
`ps_fes` (majors, `disced_6d`, by default), in pyfixest; with one fixed
effect and no `ps_xs` it is the group's share in each major. The cells, the
weights and what is left out are in `help(ug.gap_table)` and
`help(ug.ps_weights)`.

## The long frame

One row per person and job type (`pnr` x `type_sizerank_job`), with the columns
of `misc_docs/all_waves_long_variable_definitions.txt`: `T`, `vj_vij_job`,
`T_vj_vij_job`, the pre-treatment variables in `labels.PRETREATMENT`,
`educ_field`, `disced_6d`, and the outcomes in `labels.OUTCOMES`:
`spell_1_new_job` (1 for the job type of the first new job after the survey)
and `appl_job` (the number of applications to the job type).

## The persons frame

One row per person (`pnr`), with the columns of the long frame that take one
value per person: `T`, the pre-treatment variables and the education
columns. `treatment_shares` uses `T`, `educ_field` and `disced_6d`.

## The extra-long frame

One row per person and ordered pair of job types (`pnr` x
`type_sizerank_job` x `type_sizerank_jobj`), each pair in both orders; a
`d_` column is job j minus job j'. `gap_table` uses `d_Vj_Vij_rate`,
`d_vj_vij`, `d_v_vi_max_notmaxV_rate` (one value per person) and, to
reweight, `disced_6d`.

## The frame of everyone who started

One row per person who started the survey (`pnr`): `T` for everyone, the
last page reached `index_in_pages`, `T_page` (it is 14 or more: the
treatment page) and `full_answer` (it is 30: finished), `primary_sample`
(in the study sample: finished, and no expected wage of 0), education, the
pre-treatment controls and `d_v_vi_max_notmaxV`. The answers before the
treatment page are there for everyone who reached it.

## Tests on the server data

The tests are installed with the package, and pytest with them. After prep,
run them from the server script:

```python
occu_choice_pf.run_tests({'long': long, 'persons': persons, 'elong': elong, 'attrition': started})
```

The data tests of a frame left out are skipped.

It logs pytest's output as it comes, so it shows on screen and is in the log
file of `setup_log`, and returns 0 if every test passes. The frame is written
to a temporary folder, deleted afterwards, and the tests run in a new
process. Further arguments go to pytest: `run_tests(frames, '-m', 'data')`
runs the data tests only. It skips the test that compiles the tables with
latexmk, since LaTeX on the server differs.

The data tests (`tests/test_regressions_data.py`, marked `data`) check what
the table assumes of its input but cannot check itself, and build both
default tables:

- one row per person and job type;
- the outcomes, coefficients and controls are numbers or Booleans;
- `T`, the pre-treatment controls and the education columns take one value
  per person;
- `T` is 0 or 1, and `T_vj_vij_job` is `T * vj_vij_job`, missing (null or
  NaN) where `vj_vij_job` is;
- the pre-treatment controls have no missing values, so that the columns of
  a panel rest on the same rows but for what the Poisson regression drops;
- the outcomes are at least 0;
- both default tables build and keep `T_vj_vij_job` and `vj_vij_job` in
  every column, and pyfixest's estimation sample, whose persons the
  Individuals row and `min_n` count, has the fit's *N* rows and carries the
  row numbers that `common_sample` and `reg_control_samples` read (a check
  on the server's pyfixest version).

`tests/test_treatment_shares_data.py` checks that `persons` has one row per
person, `T` and the education columns, and, with `long` given too, the same
persons as `long` with the same `T` and education (the exhibits read them
from `persons`, the regressions from `long`); and it builds the exhibits of
`treatment_shares`.

`tests/test_attrition_data.py` checks that `attrition` has one row per
person; that `T`, `T_page`, `full_answer` and the outcomes are 0 or 1 for
everyone; that `T_page` and `full_answer` are the page reached (14 or more,
30); that `primary_sample` is 1 only for persons who finished; it warns,
with the number, where a pre-treatment control or the misperception is
missing for persons who reached the treatment page (the table leaves them
out of the columns where it enters as a number; as a fixed effect, missing
is a group of its own); with `persons` given too, that each person of
the study sample is in it, finished, with the same `T` and education; and
it builds both tables.

A failed data test shows the offending values. The other tests check the code
on fake data; they give the same result anywhere, so on the server they test
the environment.

## Layout

```text
src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       underestimation_gap.py, treatment_shares.py, attrition.py;
│                       imported from the top:
│                       from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames
```

`simulation/`, `examples/` and `tests/` are installed with the package. To find them:

```bash
python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>
```

Every example also runs in the VS Code Interactive window, whole or cell by
cell (`# %%`).

## Adding a module

1. `functions/<name>.py`: importable at once as `occu_choice_pf.<name>`;
   arguments as `{column: label}` dicts, with defaults in `labels.py`
2. `simulation/<name>.py`: `simulate()`, `save()`, `load()`
3. `examples/<name>/code/run_<name>.py`, plus `logs/.gitkeep`. Take the
   output folder from `root = example_root('<name>')`, and split sections
   with `# %%`. `tests/test_examples.py` runs every example.
4. `tests/test_<name>.py` for the code, and `tests/test_<name>_data.py`,
   marked `data`, for what the functions assume of real frames.

## Development

```bash
uv sync                                          # environment in .venv
uv run pytest -n auto                            # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames
```

`-n auto` runs the tests on all cores (pytest-xdist, in the dev group only).
The lower bounds in `pyproject.toml` are the server's versions for loguru,
numpy, polars and scipy (as for ad-data-pf). To test against exactly those,
in a separate environment so `uv.lock` is not rewritten:

```bash
cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest -n auto
cp "$TEMP/uv.lock" uv.lock
```

GitHub Actions does both on every push (`.github/workflows/test.yml`).

## Release

Bump `version` in `pyproject.toml`, run `uv lock`, and turn the
`## Unreleased` section of [CHANGELOG.md](CHANGELOG.md) into the version's
entry. For each change to a call, give the old and the new call, since the
server script is retyped by hand. Commit, then push a tag
`occu_choice_pf-v<version>`; GitHub Actions checks that the tag matches the
version, runs the tests, builds and publishes to PyPI (trusted publishing,
`.github/workflows/publish.yml`):

```bash
git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0
```

The same tag runs `.github/workflows/publish_1.yml` and `publish_2.yml`,
which publish the same code as `occu_choice_pf_1` and `occu_choice_pf_2`.
The DST server caches a package when it is installed and won't update that
name for 24 hours, so install an exact version under whichever name hasn't
been installed in the last 24 hours:

```bash
pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
pip install occu_choice_pf_2==X.Y.Z
```

All three install the import `occu_choice_pf`, and the most recent install is
the one used.

Before a name's first release, register it as a pending trusted publisher on
pypi.org (Account settings, Publishing): owner `AskerNC`, repository
`occu_choice_pf`, environment `pypi`, and workflow `publish.yml` for
`occu-choice-pf`, `publish_1.yml` for `occu_choice_pf_1`, `publish_2.yml` for
`occu_choice_pf_2`.
