Metadata-Version: 2.4
Name: gazoo-research-utils
Version: 2.0.1
Summary: Utilities for the Analysis of Gazoo Research Data
Author: Andrew Lim MD, Megan Lim MD, Christopher Lim MD, Robert Lim MD
License-Expression: MIT
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Dist: lifelines>=0.28.0
Requires-Dist: pandas>=3.0.0
Requires-Dist: pydantic>=2.12.5
Requires-Dist: pytest>=7.4.0 ; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0 ; extra == 'dev'
Requires-Python: >=3.11
Project-URL: Homepage, https://gazooresearch.com/
Provides-Extra: dev
Description-Content-Type: text/markdown

# GazooResearchUtils 
GazooResearchUtils is a python library whose purpose is to analyze medical data from GazooResearch. 


## Example output from GazooResearch prostate-data.csv
The data is denormalized on fields, but ultimately has this hierarchy.  
id -> collection -> tag -> (field, data_type, value, phi)


### Column descriptions are as follows
id: patient identification
collection: A collection of related tags
tag: name of tag (eg. dob, psa)
tag_id: Unique identifier
field: tag name (date, value, date)
data_type: data type held in the value column. Options are: text, categorical, number, date). dates will be of this form YYYY-MM-DD
value: value of field
phi: 0 or 1, depending on if the data element is protected health information

### Issues
The value field may be empty, or will contain data conforming to the data_type structure. 

```csv
id,collection,tag,tag_id,field,data_type,value,phi
111111,0,id,#6bed55ed-f844-4c88-ab74-af0e74aa2547,mrn,text,111111,1
111111,0,last-name,bd7e898c-9732-4320-8227-a1b90cc8c80e,value,text,Bing,1
111111,0,first-name,#4ff0ace3-e804-4d55-97a0-22cc6f927327,value,text,Chandler,1
111111,0,dob,#5578f001-b34b-4465-b699-8c8ff6c11d89,date,date,1952-12-17,0
111111,0,document,#ce5ab218-8e98-417b-b843-965ea637ca57,date,date,2023-12-06,0
111111,0,document,#ce5ab218-8e98-417b-b843-965ea637ca57,type,categorical,clinical,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,date,date,2010-01-01,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,type,categorical,prostatectomy-lymphadenectomy,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,target-id,text,prostate,0
111111,c61,R,#358ebce7-9e7d-441c-a976-38d02d3da58e,date,date,2010-01-01,0
111111,c61,R,#358ebce7-9e7d-441c-a976-38d02d3da58e,R,categorical,0,0
111111,c61,pT,#2485755d-aa2a-4fa9-a263-4d11c9370145,date,date,2010-01-01,0
111111,c61,pT,#2485755d-aa2a-4fa9-a263-4d11c9370145,T,categorical,2c,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,date,date,2010-01-01,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,N,categorical,0,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,involved-lymph-nodes,number,,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,total-lymph-nodes,number,,0
111111,c61,gleason-grade-group,#55109b94-41fe-4b11-a47a-824970ab9dfb,date,date,2010-01-01,0
111111,c61,gleason-grade-group,#55109b94-41fe-4b11-a47a-824970ab9dfb,value,categorical,2,0
111111,c61,gleason-grade-group,#55109b94-41fe-4b11-a47a-824970ab9dfb,target-id,text,prostate,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,date,date,2019-02-22,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,value,number,0.28,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,units,categorical,ng/ml,0
111111,c61,psa,91a5221d-de16-4af9-acb6-82b7d37ed118,date,date,2019-06-12,0
111111,c61,psa,91a5221d-de16-4af9-acb6-82b7d37ed118,value,number,0.31,0
111111,c61,psa,91a5221d-de16-4af9-acb6-82b7d37ed118,units,categorical,ng/ml,0
111111,c61,psa,6ed2dc02-af27-4395-a73e-36af30b7fed6,date,date,2023-12-06,0
111111,c61,psa,6ed2dc02-af27-4395-a73e-36af30b7fed6,value,number,0.01,0
111111,c61,psa,6ed2dc02-af27-4395-a73e-36af30b7fed6,units,categorical,ng/ml,0
111111,c61,androgen-deprivation-therapy,#ed2d15d6-2650-41fe-af7d-e9b6ac1140d6,start-date,date,2019-07-15,0
111111,c61,androgen-deprivation-therapy,#ed2d15d6-2650-41fe-af7d-e9b6ac1140d6,end-date,date,2019-12-06,0
111111,c61,androgen-deprivation-therapy,#ed2d15d6-2650-41fe-af7d-e9b6ac1140d6,medication,categorical,leuprolide,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,start-date,date,2019-08-15,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,end-date,date,2019-10-03,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,field,categorical,prostate-bed-pelvic-lymph-nodes,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,dose,number,,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,unit,categorical,Gy,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,fractions,number,,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,technique,categorical,unknown,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,target-id,text,prostate,0
111111,c61,biochemical-progression,#e6458863-ceba-4194-a361-df96407d5c3a,date,date,,0
111111,c61,biochemical-progression,#e6458863-ceba-4194-a361-df96407d5c3a,type,categorical,aua,0
111111,0,last-name,#3dd60c0d-d51d-4c4a-8250-715a8f415207,value,text,Bing,1
111111,0,id,#1c9d95e8-7d8b-4102-990b-02ebea1e9a14,mrn,text,111111,1
111111,0,dob,#47a681ae-23a8-41e8-b569-32039d3f47b4,date,date,1952-12-17,0
111111,c61,psa,#b482b981-cf18-420f-b518-d3f6279f03c5,date,date,2019-02-22,0
111111,c61,psa,#b482b981-cf18-420f-b518-d3f6279f03c5,value,number,0.28,0
111111,c61,psa,#b482b981-cf18-420f-b518-d3f6279f03c5,units,categorical,ng/ml,0
333333,0,id,c918371b-01f7-49ee-acd8-b00dee6e49e9,mrn,text,333333,1
333333,0,first-name,dc110270-d5c1-431c-8f90-86459bed3a36,value,text,Joey,1
333333,0,last-name,2d9ae449-8d3f-4493-82a8-75321f4df6cd,value,text,Tribbiani,1
333333,0,dob,d1044870-b049-4465-b306-e0eb1a20b8c2,date,date,1948-01-22,0
333333,c61,surgery,#a5b2b0b0-1752-401f-a4d6-63f41dec128b,date,date,2019-08-31,0
333333,c61,surgery,#a5b2b0b0-1752-401f-a4d6-63f41dec128b,type,categorical,prostatectomy,0
333333,c61,surgery,#a5b2b0b0-1752-401f-a4d6-63f41dec128b,target-id,text,prostate,0
333333,c61,pT,#3d0cf1cf-3608-4f98-95f9-7569d0f57a63,date,date,2019-08-31,0
333333,c61,pT,#3d0cf1cf-3608-4f98-95f9-7569d0f57a63,T,categorical,3a,0
```


## Anchors
Often when calculating time based analysis, we need a starting point. An anchor is used to define that starting point. An anchor json follow the same structure as a filter json. **Anchors should have a date field**.
```json
anchor = {'collection':str, 'tag': str, 'field': str, 'exact': [str, str,...], 'between': [float, float]}
```

### Tag Sequence
Sometimes you want to find an ordered scenario, for example you may want to find patients that first have surgery and then adjuvant radiation. Your anchor may have this structure:
```json
anchor = [{'tag': 'surgery'},{'tag':'radiation'}]
```
Note that you are not interested in the type of surgery, or type of radiation.

### Time Delta
time_delta represents the maximum number of **days** between anchor tags. This is only relavant if there is >1 anchor tag.

# Instance
There may be multiple instances of an anchor sequence. 'instance' defines which one matters.
instance = 0 - Means that the 1st instance of this anchor sequence will be used.

# Time Delta
Represents the maximum allowable days between anchors.
```json
anchor = [{'tag': 'surgery'},{'tag':'radiation'}]
time_delta = 60
# the maximum time between surgery and radaition is 60 days. Useful if you don't want to indentify patients with salvage radiotherapy that typically occurs many months to years later.
```

## Data Filtering
A filter json will be used to find patients that have matching tags. The filter does not need to include all fields. When you want to find patients matching multiple critera specify 'filters' which is an array of filter json. 
```json
filter = {'collection':str, 'tag': str, 'field': str, 'exact': [str, str,...], 'between': [float, float]}
```
It may include:
* collection alone
```json
filter = {'collection':'c61'}
```
* tag alone
```json
filter = {'tag':'psa'}
```
* field alone
```json
filter = {'tag':'psa'}
```
* field with a value that exactly matches
```json
filter = {'field':'histology', 'exact': ['melanoma', 'squamous-cell-carcinoma']}
```
* field with a value that is a number and is between two values
```json
filter = {'field':'psa', 'between': [0.2, 0.5]}
```
* or a combination of the above
```json
filter = {'tag':'psa', 'field':'psa', 'between': [0.2, 0.5]}
```
When you want to find patients matching multiple critera specify 'filters' which is an array of filter json.<br>The below filters finds patients who have both gleason grade group 4 OR 5, AND psa between 0.2-0.5. 
```json
filters = [{'field':'gleason-grade-group', 'exact': [4,5]}, {'tag':'psa', 'field':'psa', 'between': [0.2, 0.5]}]
```

### Search Range
The represents the time period before and after the anchor should be examined for the filters criteria (units: days).
<br>
For example, imagine you are researching prostate cancer patients getting radiation. You are interested in looking at patients with a pre-RT psa >=20 ng/mL.
```json
anchors = [{'tag':'radiation'}]
filters = [{'tag':'psa', 'field':'psa', 'between': [20, 10000]}]
search_range = [-90, 1] # Include patients who had a PSA >20 ng/mL 90 days before starting radiation, or 1 day after radiotherapy.

```
