Metadata-Version: 2.5
Name: occu_choice_pf_2
Version: 0.8.0
Summary: Tables for a survey experiment on perceived wage differences and job search, run on Statistics Denmark microdata
Author-email: AskerNC <hms467@econ.ku.dk>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: loguru>=0.7
Requires-Dist: matplotlib>=3.9
Requires-Dist: numpy>=2.3
Requires-Dist: pandas>=2.2
Requires-Dist: polars>=1.43
Requires-Dist: pyfixest>=0.60.0
Requires-Dist: pytest>=9.1.1
Requires-Dist: scipy>=1.18.1
Description-Content-Type: text/markdown

# occu-choice-pf

Tables for the survey experiment of *Perceived Wage Differences and Early
Career Job Search Choices*.
The functions are written and tested locally against simulated data, then run on the actual data.
All numbers in this package come from simulated data.

## Use

```python
from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS
```

Tables return a bare `tabular` in the style of the paper's tables (the
`\sym` stars of esttab included); `save_tex` writes it and logs it, so it can
be read before an export request. Defaults (variables, labels) sit in
`occu_choice_pf.labels` and can be overridden per call. See the docstrings,
and `examples/` for every exhibit and for custom tables.

## The table of controls

`reg_control_table(long)` regresses each outcome on `T_vj_vij_job`, one
panel per outcome, with one table column per entry of `cols_control`. Every
column has `coefs` (reported), `common_control`, and the controls of its
entry. The defaults (`labels.COLS_CONTROL`, `labels.COMMON_CONTROL`) give:

| Column | Controls | `cols_control` entry |
| --- | --- | --- |
| (1) | `vj_vij_job` and the person fixed effect (`common_control`) | `{}` |
| (2) | + `vj_vij_job` squared and cubed | `POWERS` |
| (3) | + pre-treatment controls times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS}` |
| (4) | (3) + education field dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **EDUC_FIELD}` |
| (5) | (3) + DISCED (`disced_6d`) dummies times `vj_vij_job` | `{**POWERS, **PRE_CONTROLS, **DISCED}` |

A control is `{check-mark row label: terms}`, and a term is a column
(`'age'`), a column made into a dummy for each value (`'i.age'`, as Stata's
`i.`), or a polars expression (`c.vj_vij_job ** 2`). Your own columns:

```python
from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])
```

Every term of `cols_control` is times `vj_vij_job`, except one wrapped in
`reg.plain()`, which enters its column as it is; an `i.<column>` there is a
fixed effect absorbed in that column only:

```python
from occu_choice_pf.labels import JOB
reg.reg_control_table(long, cols_control=[{}, {'Job type FE': reg.plain('i.' + JOB)}])
```

The arguments, the check-mark rows and the input checks are in
`help(reg.reg_control_table)`; the estimation, and what drops out of a
column (all of it logged), in `help(reg)`. `reg.reg_control_fits(long, ...)`
returns the pyfixest fits behind a table, `{outcome: [fit per column]}`, to
look into with `.summary()` or `pf.etable`. `examples/` has custom calls: a
subsample, other regressors, clustering and a second fixed effect.

The columns of a panel can rest on different rows: the Poisson regression
with the DISCED dummies drops the separated rows of persons alone in their
DISCED, which the other columns keep. `common_sample=True` estimates every
column on the rows all of them keep, so that the columns differ in their
controls only; `reg.reg_control_samples(long, ...)`, with the arguments of
the table, returns `long` with a Boolean column `in_<outcome>_<column>` per
regression, True in the rows it is estimated on:

```python
save_tex(reg.reg_control_table(long, common_sample=True), out / 'tab_poisson_common.tex')
s = reg.reg_control_samples(long)
s.filter(c.in_appl_job_5)                      # the rows of panel B, column (5)
```

A separated row leaves the person's other rows in: with `vj_vij_job`
winsorized, a person alone in their DISCED whose rows with applications are
tied at the bound keeps those. `drop_persons='separated'` drops a person
with a separated row whole, and `drop_persons='any'` a person who loses any
row, for a missing value, separation or pyfixest; both take the arguments
above too:

```python
save_tex(reg.reg_control_table(long, drop_persons='separated'), out / 'tab_poisson_sep.tex')
save_tex(reg.reg_control_table(long, drop_persons='any'), out / 'tab_poisson_whole.tex')
```

`share_range=(low, high)` keeps the persons of a major (`share_educ`, by
default `disced_6d`) only where its treated share among the persons of the
regression, after separation and pyfixest's drops, is from `low` to `high`.
The persons of the other majors are dropped whole and the regression refit
until every major left is in the range. Each table column and outcome has
its own sample, and so its own shares:

```python
save_tex(reg.reg_control_table(long, share_range=(0.1, 0.9)), out / 'tab_poisson_share.tex')
```

`totals={outcome: row label}` adds a row between Individuals and N in that
outcome's panel: the outcome summed over the rows each column is estimated
on, left empty where fewer than 5 persons (`min_n`) have an outcome above 0:

```python
reg.reg_control_table(long, totals={'appl_job': 'Applications'})   # the applications made
```

## The treated share within majors

With the major dummies times `vj_vij_job` (columns 4 and 5), the treatment
effect is estimated within majors: a major whose persons are all treated, or
all in the control group, adds next to nothing, and a major weighs more the
closer its treated share is to one half. Without them, a treated share that
differs between majors moves the estimate where the slope on `vj_vij_job`
differs between majors. `treatment_shares` shows how large the persons'
majors are and how far the treated share of their major is from what random
assignment gives (`T` permuted across the persons, 1000 draws, no strata).
Three table columns or a plot per column of `labels.EDUC`: major field and
DISCED. The functions take the persons frame, one row per person, and raise
on a person with more than one row:

```python
from occu_choice_pf import save_fig
from occu_choice_pf import treatment_shares as ts

save_tex(ts.percentile_table(persons), out / 'tab_share_percentiles.tex')
save_fig(ts.share_figure(persons), out / 'fig_share_hist.png')
ts.share_frame(persons, 'disced_6d')  # one row per major: on the server only
```

- `percentile_table`: percentiles over persons of the size of their major
  and of its treated share (data and random assignment). Each percentile is
  the mean of the 9 persons around it (`window`), never one person's value.
  Below them, the standard deviation over persons of the treated share of
  their major (data and random assignment) and its *p*-value under random
  assignment: a permutation chi-squared test of treatment by major.
- `share_figure`: a histogram of the persons by the treated share of their
  major (0, ten intervals, 1), with the mean and the 95% range of random
  assignment; a matplotlib figure, the numbers plotted in the log.
  `save_fig` writes it, a PNG at 300 dpi unless the path ends in `.pdf`.

The persons are those of the frame given, so the persons of a regression,
`persons.join(s.filter(c.in_appl_job_5), on='pnr', how='semi')` with
`s = reg.reg_control_samples(long)`, give the shares within it.

Disclosure: no number shown rests on fewer than `min_n` (5) persons, nor can
one be worked out from the others, the persons treated (in the paper's
balance table) included. A count below 5, 0 included, reads <5. Percentiles
are left empty where their 9-person windows overlap or leave 1-4 persons
outside; the shares, the test and random assignment, which depends on the
persons treated, with fewer than 5 treated or untreated. A bar of fewer than 5
persons reads <5; the bars at 0 and 1 are left out where the persons in
all-treated or all-control majors, or the treated or untreated of the
others, are 1-4; where the bars left out add up to 1-4 persons, the smallest
interval bar is left out too. A 0 can still be worked out. `share_frame`
lists every major, however small, and is for the server only.

## The attrition table

`attrition.attrition_table(df)` asks whether getting to the end of the
survey depends on the treatment, alone or by education or misperception. T
is drawn when a person starts the survey, and the pages before the
treatment page are the same in both arms, so the attrition that can bias
the estimates is among those who reached it; reaching it, among everyone who
started, is a placebo. It takes one row per person:

```python
from occu_choice_pf import attrition as at
from occu_choice_pf.labels import COLS_PLACEBO

save_tex(at.attrition_table(started.filter(c.T_page == 1)), out / 'tab_attrition.tex')
save_tex(at.attrition_table(started, {'T_page': 'Reached the treatment page'},
                            cols_control=COLS_PLACEBO), out / 'tab_attrition_placebo.tex')
```

Linear probability models (pyfixest `feols`, standard errors clustered on
the person), one panel per outcome (`labels.ATTRITION_OUTCOMES`:
`full_answer` and `primary_sample`, each 0 or 1 for every person) and one
column per entry of `cols_control`, as in `reg_control_table`: a term in
`plain()` enters as it is, the others as they are and times T (Stata's
`##`). Without `cols_control` (`labels.cols_attrition`; `pre` is
`labels.pre_controls()`):

| Column | Controls | `cols_control` entry |
| --- | --- | --- |
| (1) | T alone | `{}` |
| (2) | + pre-treatment controls and major FE | `{**pre, **MAJOR_FE}` |
| (3) | (2) + major field FE and times T | `{**pre, **MAJOR_FE, **FIELD_T}` |
| (4) | (2) + major times T | `{**pre, **MAJOR_FE, **MAJOR_T}` |
| (5) | (2) + the misperception and its absolute value, and times T | `{**pre, **MAJOR_FE, **MISPERCEPTION_T}` |

The pre-treatment controls of these columns are `pretreatment`, entering
linearly (`labels.PRETREATMENT` by default), and `pretreatment_fe`, entering
as fixed effects, a dummy per value (`labels.PRETREATMENT_FE`, none by
default). With your own `cols_control`, give its columns
`labels.pre_controls(linear, fe)` instead:

```python
save_tex(at.attrition_table(exposed, pretreatment=linear, pretreatment_fe=['age']), out / 'tab_attrition.tex')
cols = [{}, {**pre_controls(linear, ['age']), **MAJOR_FE}]
```

A column without terms times T shows the coefficient on T; one with them
shows the p-value of the test that they are jointly 0 instead. That test is
a permutation test: T is permuted within majors (`strata`, DISCED by
default) `draws` times (1000), and the statistic is the residual sum of
squares without the terms times T over that with them. It is exact under the
sharp null that T changes no one's outcome, so it holds with hundreds of
small majors. The major-by-arm cells are absorbed as a fixed effect, but the
main table still takes some 6 minutes at 1000 draws. Persons alone in a
fixed effect are kept, so every column of a panel has the same persons
unless a control is missing (the log says how many leave). The misperception
is `labels.MISPERCEPTION`, `d_v_vi_max_notmaxV`; the log has its
coefficients times T, to read the sign off.

Disclosure: a column is left empty unless 5 persons or more (`min_n`) of
each arm have each value of the outcome, as the control mean and column
(1) give the share of each arm; Individuals reads <5 below 5.

## The table of wage gap underestimation

`underestimation_gap.gap_table(rows)` is Table 3 of the paper: one row per
subsample, `rows={label: frame}`, each frame the extra-long frame or a
subsample of it, made with polars before the call.

```python
from occu_choice_pf import underestimation_gap as ug

clean = {'d_vj_vij': c.d_vj_vij_clean, 'd_Vj_Vij_rate': c.d_Vj_Vij_rate_clean}
rows = {
    'Full sample': elong,
    **{f'Actual log wage gap $>$ {x}': elong.filter(c.d_vj > x) for x in [0.05, 0.1]},
    'Full sample, correcting reporting errors': elong.with_columns(**clean),
    'Reweighted for non-participation': elong,
}
weights = {'Reweighted for non-participation': 'weight_invited'}
save_tex(ug.gap_table(rows, weights), out / 'tab_estimation_gaps.tex')
save_tex(ug.gap_table(rows, weights, bootstrap=1000), out / 'tab_estimation_gaps_se.tex')
```

The rows are numbered. `bootstrap=B` adds standard errors from B draws of
persons with replacement. `compare=[(label, label)]` puts the p-value of the
difference between two rows under them (`$p$-value of difference (1)-(2)`),
and a line under that. The rows need not be next to each other, and a row
may be in several pairs: `compare=[('Men', 'Women'), ('Men', 'Women, reweighted to (1)')]`
puts both p-values under the three rows (the rows from the first to the last
of a pair are a block, joined with the blocks it overlaps). `p_at_end=True`
puts all the p-values in the last rows of the table instead.
`share_column` sets column 7: `'underestimator'` (the default), the share of
individuals who are underestimators; `'zero_gap'`, the share of job pairs
with no perceived wage gap (`d_Vij == 0`); a list of both,
`['underestimator', 'zero_gap']`, columns 7 and 8 in that order; or `None`,
no column 7. The frames need not have a column not shown. A frame with NaN
raises, since a filter such as `c.d_vj > 0.05` keeps NaN: make it missing in
prep, `elong = elong.with_columns(pl.col(pl.Float64).fill_nan(None))`. `ps_group`
reweights one group to the other with a propensity score model estimated
anew in each draw, as Table C1 of the parental transmission paper: it adds
`ps_weight` (reweighted to the other group) and `ps_support` (1 for persons
with a weight) to every row, to name in `weights`:

```python
no, yes = elong.filter(c.par_anyjobtype == 0), elong.filter(c.par_anyjobtype == 1)
rows = {'Neither': no, 'Either': yes, 'Neither, weightable': no, 'Either, reweighted to (3)': yes}
ug.gap_table(rows, {'Neither, weightable': 'ps_support', 'Either, reweighted to (3)': 'ps_weight'},
             bootstrap=1000, ps_group='par_anyjobtype',
             compare=[('Neither', 'Either'), ('Neither, weightable', 'Either, reweighted to (3)')])
```

The model is a logit of `ps_group` on `ps_xs` with the fixed effects
`ps_fes` (majors, `disced_6d`, by default), in pyfixest; with one fixed
effect and no `ps_xs` it is the group's share in each major. The cells, the
weights and what is left out are in `help(ug.gap_table)` and
`help(ug.ps_weights)`.

## The long frame

One row per person and job type (`pnr` x `type_sizerank_job`), with the columns
of `misc_docs/all_waves_long_variable_definitions.txt`: `T`, `vj_vij_job`,
`T_vj_vij_job`, the pre-treatment variables in `labels.PRETREATMENT`,
`educ_field`, `disced_6d`, and the outcomes in `labels.OUTCOMES`:
`spell_1_new_job` (1 for the job type of the first new job after the survey)
and `appl_job` (the number of applications to the job type).

## The persons frame

One row per person (`pnr`), with the columns of the long frame that take one
value per person: `T`, the pre-treatment variables and the education
columns. `treatment_shares` uses `T`, `educ_field` and `disced_6d`.

## The extra-long frame

One row per person and ordered pair of job types (`pnr` x
`type_sizerank_job` x `type_sizerank_jobj`), each pair in both orders; a
`d_` column is job j minus job j'. `gap_table` uses `d_Vj_Vij_rate`,
`d_vj_vij`, `d_v_vi_max_notmaxV_rate` (one value per person) and, to
reweight, `disced_6d`.

## The frame of everyone who started

One row per person who started the survey (`pnr`): `T` for everyone, the
last page reached `index_in_pages`, `T_page` (it is 14 or more: the
treatment page) and `full_answer` (it is 30: finished), `primary_sample`
(in the study sample: finished, and no expected wage of 0), education, the
pre-treatment controls and `d_v_vi_max_notmaxV`. The answers before the
treatment page are there for everyone who reached it.

## Tests on the server data

The tests are installed with the package, and pytest with them. After prep,
run them from the server script:

```python
occu_choice_pf.run_tests({'long': long, 'persons': persons, 'elong': elong, 'attrition': started})
```

The data tests of a frame left out are skipped.

It logs pytest's output as it comes, so it shows on screen and is in the log
file of `setup_log`, and returns 0 if every test passes. The frame is written
to a temporary folder, deleted afterwards, and the tests run in a new
process. Further arguments go to pytest: `run_tests(frames, '-m', 'data')`
runs the data tests only. It skips the test that compiles the tables with
latexmk, since LaTeX on the server differs.

The data tests (`tests/test_regressions_data.py`, marked `data`) check what
the table assumes of its input but cannot check itself, and build both
default tables:

- one row per person and job type;
- the outcomes, coefficients and controls are numbers or Booleans;
- `T`, the pre-treatment controls and the education columns take one value
  per person;
- `T` is 0 or 1, and `T_vj_vij_job` is `T * vj_vij_job`, missing (null or
  NaN) where `vj_vij_job` is;
- the pre-treatment controls have no missing values, so that the columns of
  a panel rest on the same rows but for what the Poisson regression drops;
- the outcomes are at least 0;
- both default tables build and keep `T_vj_vij_job` and `vj_vij_job` in
  every column, and pyfixest's estimation sample, whose persons the
  Individuals row and `min_n` count, has the fit's *N* rows and carries the
  row numbers that `common_sample` and `reg_control_samples` read (a check
  on the server's pyfixest version).

`tests/test_treatment_shares_data.py` checks that `persons` has one row per
person, `T` and the education columns, and, with `long` given too, the same
persons as `long` with the same `T` and education (the exhibits read them
from `persons`, the regressions from `long`); and it builds the exhibits of
`treatment_shares`.

`tests/test_attrition_data.py` checks that `attrition` has one row per
person; that `T`, `T_page`, `full_answer` and the outcomes are 0 or 1 for
everyone; that `T_page` and `full_answer` are the page reached (14 or more,
30); that `primary_sample` is 1 only for persons who finished; that the
pre-treatment controls and the misperception are there for everyone who
reached the treatment page; with `persons` given too, that each person of
the study sample is in it, finished, with the same `T` and education; and
it builds both tables.

A failed data test shows the offending values. The other tests check the code
on fake data; they give the same result anywhere, so on the server they test
the environment.

## Layout

```text
src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       underestimation_gap.py, treatment_shares.py, attrition.py;
│                       imported from the top:
│                       from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames
```

`simulation/`, `examples/` and `tests/` are installed with the package. To find them:

```bash
python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>
```

Every example also runs in the VS Code Interactive window, whole or cell by
cell (`# %%`).

## Adding a module

1. `functions/<name>.py`: importable at once as `occu_choice_pf.<name>`;
   arguments as `{column: label}` dicts, with defaults in `labels.py`
2. `simulation/<name>.py`: `simulate()`, `save()`, `load()`
3. `examples/<name>/code/run_<name>.py`, plus `logs/.gitkeep`. Take the
   output folder from `root = example_root('<name>')`, and split sections
   with `# %%`. `tests/test_examples.py` runs every example.
4. `tests/test_<name>.py` for the code, and `tests/test_<name>_data.py`,
   marked `data`, for what the functions assume of real frames.

## Development

```bash
uv sync                                          # environment in .venv
uv run pytest -n auto                            # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames
```

`-n auto` runs the tests on all cores (pytest-xdist, in the dev group only).
The lower bounds in `pyproject.toml` are the server's versions for loguru,
numpy, polars and scipy (as for ad-data-pf). To test against exactly those,
in a separate environment so `uv.lock` is not rewritten:

```bash
cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest -n auto
cp "$TEMP/uv.lock" uv.lock
```

GitHub Actions does both on every push (`.github/workflows/test.yml`).

## Release

Bump `version` in `pyproject.toml`, run `uv lock`, and turn the
`## Unreleased` section of [CHANGELOG.md](CHANGELOG.md) into the version's
entry. For each change to a call, give the old and the new call, since the
server script is retyped by hand. Commit, then push a tag
`occu_choice_pf-v<version>`; GitHub Actions checks that the tag matches the
version, runs the tests, builds and publishes to PyPI (trusted publishing,
`.github/workflows/publish.yml`):

```bash
git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0
```

The same tag runs `.github/workflows/publish_1.yml` and `publish_2.yml`,
which publish the same code as `occu_choice_pf_1` and `occu_choice_pf_2`.
The DST server caches a package when it is installed and won't update that
name for 24 hours, so install an exact version under whichever name hasn't
been installed in the last 24 hours:

```bash
pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
pip install occu_choice_pf_2==X.Y.Z
```

All three install the import `occu_choice_pf`, and the most recent install is
the one used.

Before a name's first release, register it as a pending trusted publisher on
pypi.org (Account settings, Publishing): owner `AskerNC`, repository
`occu_choice_pf`, environment `pypi`, and workflow `publish.yml` for
`occu-choice-pf`, `publish_1.yml` for `occu_choice_pf_1`, `publish_2.yml` for
`occu_choice_pf_2`.
