Starting scenario 4, validation against site 1
2022-03-01 13:17:48.806464: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:17:48.806497: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Cross validation site: 1
Training sites: [0, 2, 3, 4, 5, 6, 7, 8]
Number of files: 8
Number of east files: 4
Number of west files: 4
Source files: ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
Full config: {'clean_training_data_kwargs': {'filter_clear': False, 'nan_option': 'interp'}, 'epochs_a': 100, 'epochs_b': 100, 'features': ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo'], 'hidden_layers': [{'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}], 'learning_rate': 0.0005, 'loss_weights_a': [1, 0], 'loss_weights_b': [0.5, 0.5], 'metric': 'relative_mae', 'n_batch': 32, 'one_hot_categories': {'flag': ['clear', 'ice_cloud', 'water_cloud', 'bad_cloud']}, 'p_fun': 'p_fun_all_sky', 'p_kwargs': {'loss_terms': ['mae_ghi']}, 'phygnn_seed': 0, 'surfrad_window_minutes': 15, 'y_labels': ['cld_opd_dcomp', 'cld_reff_dcomp']}
INFO - 2022-03-01 13:17:57,448 [trainer.py:40] : Trainer: Training on sites [0, 2, 3, 4, 5, 6, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:17:57,448 [trainer.py:49] : Trainer: Training on sites [0, 2, 3, 4, 5, 6, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:17:57,448 [data_handlers.py:60] : Loading training data
DEBUG - 2022-03-01 13:17:57,448 [data_handlers.py:78] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:17:57,449 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5
DEBUG - 2022-03-01 13:17:58,623 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:17:58,627 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:17:58,628 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:17:58,633 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:17:59,336 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:59,339 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:18:00,016 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:00,020 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:18:00,720 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:00,724 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:18:01,408 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:01,411 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:18:02,094 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:02,098 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:18:02,795 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:02,799 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:18:03,484 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:03,488 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:18:04,194 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:04,194 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5
DEBUG - 2022-03-01 13:18:05,218 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:18:05,223 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:05,223 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:05,226 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:18:05,890 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:05,894 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:18:06,558 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:06,561 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:18:07,227 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:07,231 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:18:07,892 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:07,896 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:18:08,562 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:08,566 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:18:09,230 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:09,233 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:18:09,889 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:09,892 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:18:10,559 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:10,560 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5
DEBUG - 2022-03-01 13:18:11,592 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:11,597 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:11,597 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:11,600 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:18:12,278 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:12,281 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:18:12,954 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:12,958 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:18:13,630 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:13,633 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:18:14,301 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:14,305 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:18:14,979 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:14,982 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:18:15,648 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:15,652 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:18:16,322 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:16,325 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:18:16,998 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:16,998 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5
DEBUG - 2022-03-01 13:18:18,033 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:18,037 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:18,037 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:18,040 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:18:18,702 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:18,705 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:18:19,348 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:19,351 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:18:20,000 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:20,003 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:18:20,641 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:20,645 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:18:21,293 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:21,296 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:18:21,946 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:21,950 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:18:22,611 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:22,614 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:18:23,261 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:23,261 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5
DEBUG - 2022-03-01 13:18:29,622 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:18:29,643 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:18:29,643 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:29,646 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:18:30,312 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:30,316 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:18:30,993 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:30,996 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:18:31,713 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:31,717 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:18:32,392 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:32,396 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:18:33,072 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:33,075 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:18:33,751 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:33,754 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:18:34,432 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:34,436 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:18:35,124 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:35,124 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5
DEBUG - 2022-03-01 13:18:36,360 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:36,365 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:36,365 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:36,368 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:18:37,018 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:37,022 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:18:37,671 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:37,674 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:18:38,326 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:38,329 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:18:38,980 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:38,983 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:18:39,632 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:39,635 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:18:40,286 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:40,289 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:18:40,944 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:40,948 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:18:41,602 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:41,602 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5
DEBUG - 2022-03-01 13:18:48,220 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:18:48,240 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:18:48,240 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:48,244 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:18:48,922 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:48,926 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:18:49,603 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:49,606 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:18:50,285 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:50,288 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:18:50,970 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:50,973 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:18:51,673 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:51,676 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:18:52,359 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:52,363 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:18:53,068 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:53,071 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:18:53,771 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:53,771 [data_handlers.py:85] : Loading data for site(s) [0, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5
DEBUG - 2022-03-01 13:18:57,035 [data_handlers.py:103] : 	Shape temp_raw=(420480, 19), temp_all_sky=(420480, 14)
DEBUG - 2022-03-01 13:18:57,045 [data_handlers.py:106] : 	Time step is 10 minutes
DEBUG - 2022-03-01 13:18:57,045 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:57,048 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:18:57,723 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:57,727 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:18:58,389 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:58,393 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:18:59,068 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:59,072 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:18:59,735 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:59,738 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:19:00,417 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:00,420 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:19:01,085 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:01,089 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:19:01,772 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:01,776 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:19:02,445 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:02,445 [data_handlers.py:136] : Data load complete. Shape df_raw=(2803968, 19), df_all_sky=(2803968, 14), df_surf=(2803968, 5)
DEBUG - 2022-03-01 13:19:03,308 [data_handlers.py:159] : Extracting 2D arrays to run rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:19:14,956 [data_handlers.py:176] : Running rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:21:09,145 [data_handlers.py:194] : Completed rest2 run for clearsky PhyGNN inputs.
INFO - 2022-03-01 13:21:10,829 [data_handlers.py:62] : Prepping training data
DEBUG - 2022-03-01 13:21:10,830 [data_handlers.py:214] : Training data clean kwargs: {'filter_daylight': True, 'filter_clear': False, 'add_cloud_flag': True, 'sza_lim': 89, 'nan_option': 'interp'}
DEBUG - 2022-03-01 13:21:10,830 [data_handlers.py:215] : Shape before cleaning: df_raw=(2803968, 19)
INFO - 2022-03-01 13:21:11,104 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:21:11,108 [data_cleaners.py:38] : 51.91% of daylight timesteps are cloudy
INFO - 2022-03-01 13:21:11,113 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:21:11,118 [data_cleaners.py:42] : 34.62% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:21:11,122 [data_cleaners.py:44] : 34.84% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:21:11,122 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:21:11,125 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,131 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,135 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,140 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,144 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.62% NaN values
DEBUG - 2022-03-01 13:21:11,148 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.62% NaN values
DEBUG - 2022-03-01 13:21:11,152 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.63% NaN values
DEBUG - 2022-03-01 13:21:11,156 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:21:11,159 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:21:11,163 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.61% NaN values
DEBUG - 2022-03-01 13:21:11,167 [data_cleaners.py:50] : 	"cloud_probability" has 3.61% NaN values
DEBUG - 2022-03-01 13:21:11,171 [data_cleaners.py:50] : 	"cloud_fraction" has 3.61% NaN values
DEBUG - 2022-03-01 13:21:11,175 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,179 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,183 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,187 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,190 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:11,194 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 83.13% NaN values
DEBUG - 2022-03-01 13:21:11,198 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.19% NaN values
DEBUG - 2022-03-01 13:21:11,198 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:21:14,418 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:21:14,697 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393391 after filters (49.69% of original)
DEBUG - 2022-03-01 13:21:14,822 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:21:14,823 [data_cleaners.py:107] : Cleaning took 4.0 seconds
DEBUG - 2022-03-01 13:21:14,823 [data_handlers.py:218] : Shape after cleaning: df_train=(1393391, 20)
DEBUG - 2022-03-01 13:21:14,823 [data_handlers.py:221] : Cleaning df_all_sky training data (for pfun).
DEBUG - 2022-03-01 13:21:14,823 [data_handlers.py:222] : Shape before cleaning: df_all_sky=(2803968, 25)
INFO - 2022-03-01 13:21:15,173 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:21:15,178 [data_cleaners.py:38] : 51.91% of daylight timesteps are cloudy
INFO - 2022-03-01 13:21:15,183 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:21:15,188 [data_cleaners.py:42] : 34.62% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:21:15,192 [data_cleaners.py:44] : 34.84% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:21:15,192 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:21:15,195 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,199 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,203 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,207 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,212 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,216 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 83.13% NaN values
DEBUG - 2022-03-01 13:21:15,220 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.19% NaN values
DEBUG - 2022-03-01 13:21:15,224 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,228 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,231 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,235 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,238 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,242 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,247 [data_cleaners.py:50] : 	"surfrad_dhi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,252 [data_cleaners.py:50] : 	"surfrad_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,257 [data_cleaners.py:50] : 	"surfrad_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,260 [data_cleaners.py:50] : 	"doy" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,265 [data_cleaners.py:50] : 	"radius" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,271 [data_cleaners.py:50] : 	"Tuuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,276 [data_cleaners.py:50] : 	"clearsky_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,281 [data_cleaners.py:50] : 	"clearsky_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,286 [data_cleaners.py:50] : 	"Ruuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,291 [data_cleaners.py:50] : 	"Tddclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,297 [data_cleaners.py:50] : 	"Tduclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,302 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:15,302 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:21:17,883 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:21:18,161 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393391 after filters (49.69% of original)
DEBUG - 2022-03-01 13:21:18,317 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:21:18,318 [data_cleaners.py:107] : Cleaning took 3.5 seconds
DEBUG - 2022-03-01 13:21:18,319 [data_handlers.py:226] : Shape after cleaning: df_all_sky=(1393391, 26)
DEBUG - 2022-03-01 13:21:18,414 [data_handlers.py:240] : **Shape: df_train=(1393391, 17)
DEBUG - 2022-03-01 13:21:18,443 [data_handlers.py:250] : Shapes: x=(1393391, 15), y=(1393391, 2), p=(1393391, 26)
DEBUG - 2022-03-01 13:21:18,443 [data_handlers.py:253] : Training features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
DEBUG - 2022-03-01 13:21:18,443 [trainer.py:67] : Building PHYGNN model
INFO - 2022-03-01 13:21:18,443 [trainer.py:70] : Using p_fun: <function p_fun_all_sky at 0x2b27a2a728b0>
INFO - 2022-03-01 13:21:18,443 [base.py:152] : Active python environment versions: 
{   'numpy': '1.22.2',
    'pandas': '1.2.4',
    'phygnn': '0.0.14',
    'python': '3.8.8 (default, Feb 24 2021, 21:46:12) \n[GCC 7.3.0]',
    'sklearn': '0.24.1',
    'tensorflow': '2.8.0'}
INFO - 2022-03-01 13:21:18,460 [base.py:111] : Successfully initialized model with 17 layers
INFO - 2022-03-01 13:21:18,460 [trainer.py:84] : Training part A - pure data. Loss is [1, 0]
2022-03-01 13:21:27.640666: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.641709: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.642576: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.643283: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.643982: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.644713: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.645489: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.646158: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:27.646177: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-03-01 13:21:27.646642: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO - 2022-03-01 13:21:35,939 [phygnn.py:576] : Epoch 0 train loss: 7.01e-01 val loss: 6.88e-01 for "phygnn"
INFO - 2022-03-01 13:21:44,458 [phygnn.py:576] : Epoch 1 train loss: 6.43e-01 val loss: 6.26e-01 for "phygnn"
INFO - 2022-03-01 13:21:53,013 [phygnn.py:576] : Epoch 2 train loss: 5.69e-01 val loss: 5.56e-01 for "phygnn"
INFO - 2022-03-01 13:22:01,399 [phygnn.py:576] : Epoch 3 train loss: 5.38e-01 val loss: 5.20e-01 for "phygnn"
INFO - 2022-03-01 13:22:09,963 [phygnn.py:576] : Epoch 4 train loss: 5.18e-01 val loss: 4.98e-01 for "phygnn"
INFO - 2022-03-01 13:22:18,794 [phygnn.py:576] : Epoch 5 train loss: 5.06e-01 val loss: 4.87e-01 for "phygnn"
INFO - 2022-03-01 13:22:27,212 [phygnn.py:576] : Epoch 6 train loss: 4.94e-01 val loss: 4.82e-01 for "phygnn"
INFO - 2022-03-01 13:22:35,566 [phygnn.py:576] : Epoch 7 train loss: 4.92e-01 val loss: 4.77e-01 for "phygnn"
INFO - 2022-03-01 13:22:43,897 [phygnn.py:576] : Epoch 8 train loss: 4.87e-01 val loss: 4.72e-01 for "phygnn"
INFO - 2022-03-01 13:22:52,428 [phygnn.py:576] : Epoch 9 train loss: 4.88e-01 val loss: 4.70e-01 for "phygnn"
INFO - 2022-03-01 13:23:01,032 [phygnn.py:576] : Epoch 10 train loss: 4.90e-01 val loss: 4.66e-01 for "phygnn"
INFO - 2022-03-01 13:23:09,857 [phygnn.py:576] : Epoch 11 train loss: 4.76e-01 val loss: 4.66e-01 for "phygnn"
INFO - 2022-03-01 13:23:18,267 [phygnn.py:576] : Epoch 12 train loss: 4.78e-01 val loss: 4.62e-01 for "phygnn"
INFO - 2022-03-01 13:23:26,891 [phygnn.py:576] : Epoch 13 train loss: 4.77e-01 val loss: 4.62e-01 for "phygnn"
INFO - 2022-03-01 13:23:35,276 [phygnn.py:576] : Epoch 14 train loss: 4.77e-01 val loss: 4.59e-01 for "phygnn"
INFO - 2022-03-01 13:23:43,751 [phygnn.py:576] : Epoch 15 train loss: 4.77e-01 val loss: 4.58e-01 for "phygnn"
INFO - 2022-03-01 13:23:52,139 [phygnn.py:576] : Epoch 16 train loss: 4.64e-01 val loss: 4.57e-01 for "phygnn"
INFO - 2022-03-01 13:24:00,870 [phygnn.py:576] : Epoch 17 train loss: 4.68e-01 val loss: 4.55e-01 for "phygnn"
INFO - 2022-03-01 13:24:09,285 [phygnn.py:576] : Epoch 18 train loss: 4.67e-01 val loss: 4.54e-01 for "phygnn"
INFO - 2022-03-01 13:24:17,641 [phygnn.py:576] : Epoch 19 train loss: 4.63e-01 val loss: 4.51e-01 for "phygnn"
INFO - 2022-03-01 13:24:26,150 [phygnn.py:576] : Epoch 20 train loss: 4.64e-01 val loss: 4.51e-01 for "phygnn"
INFO - 2022-03-01 13:24:34,949 [phygnn.py:576] : Epoch 21 train loss: 4.60e-01 val loss: 4.50e-01 for "phygnn"
INFO - 2022-03-01 13:24:43,548 [phygnn.py:576] : Epoch 22 train loss: 4.63e-01 val loss: 4.48e-01 for "phygnn"
INFO - 2022-03-01 13:24:51,936 [phygnn.py:576] : Epoch 23 train loss: 4.61e-01 val loss: 4.49e-01 for "phygnn"
INFO - 2022-03-01 13:25:00,369 [phygnn.py:576] : Epoch 24 train loss: 4.61e-01 val loss: 4.47e-01 for "phygnn"
INFO - 2022-03-01 13:25:08,971 [phygnn.py:576] : Epoch 25 train loss: 4.59e-01 val loss: 4.48e-01 for "phygnn"
INFO - 2022-03-01 13:25:17,468 [phygnn.py:576] : Epoch 26 train loss: 4.57e-01 val loss: 4.42e-01 for "phygnn"
INFO - 2022-03-01 13:25:26,250 [phygnn.py:576] : Epoch 27 train loss: 4.59e-01 val loss: 4.43e-01 for "phygnn"
INFO - 2022-03-01 13:25:34,948 [phygnn.py:576] : Epoch 28 train loss: 4.50e-01 val loss: 4.41e-01 for "phygnn"
INFO - 2022-03-01 13:25:43,698 [phygnn.py:576] : Epoch 29 train loss: 4.56e-01 val loss: 4.38e-01 for "phygnn"
INFO - 2022-03-01 13:25:52,102 [phygnn.py:576] : Epoch 30 train loss: 4.48e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2022-03-01 13:26:00,452 [phygnn.py:576] : Epoch 31 train loss: 4.47e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2022-03-01 13:26:08,751 [phygnn.py:576] : Epoch 32 train loss: 4.42e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2022-03-01 13:26:17,412 [phygnn.py:576] : Epoch 33 train loss: 4.45e-01 val loss: 4.36e-01 for "phygnn"
INFO - 2022-03-01 13:26:26,085 [phygnn.py:576] : Epoch 34 train loss: 4.46e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2022-03-01 13:26:34,462 [phygnn.py:576] : Epoch 35 train loss: 4.53e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:26:42,997 [phygnn.py:576] : Epoch 36 train loss: 4.45e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:26:51,533 [phygnn.py:576] : Epoch 37 train loss: 4.43e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:26:59,970 [phygnn.py:576] : Epoch 38 train loss: 4.45e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:27:08,423 [phygnn.py:576] : Epoch 39 train loss: 4.37e-01 val loss: 4.32e-01 for "phygnn"
INFO - 2022-03-01 13:27:16,885 [phygnn.py:576] : Epoch 40 train loss: 4.44e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:27:25,488 [phygnn.py:576] : Epoch 41 train loss: 4.44e-01 val loss: 4.29e-01 for "phygnn"
INFO - 2022-03-01 13:27:33,828 [phygnn.py:576] : Epoch 42 train loss: 4.39e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:27:42,520 [phygnn.py:576] : Epoch 43 train loss: 4.33e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:27:51,026 [phygnn.py:576] : Epoch 44 train loss: 4.39e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:27:59,502 [phygnn.py:576] : Epoch 45 train loss: 4.39e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:28:08,194 [phygnn.py:576] : Epoch 46 train loss: 4.37e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:28:16,939 [phygnn.py:576] : Epoch 47 train loss: 4.38e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:28:25,618 [phygnn.py:576] : Epoch 48 train loss: 4.42e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:28:34,241 [phygnn.py:576] : Epoch 49 train loss: 4.33e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:28:42,752 [phygnn.py:576] : Epoch 50 train loss: 4.36e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:28:51,471 [phygnn.py:576] : Epoch 51 train loss: 4.34e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:28:59,856 [phygnn.py:576] : Epoch 52 train loss: 4.29e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:29:08,167 [phygnn.py:576] : Epoch 53 train loss: 4.31e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:29:16,505 [phygnn.py:576] : Epoch 54 train loss: 4.27e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:29:25,348 [phygnn.py:576] : Epoch 55 train loss: 4.28e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:29:34,004 [phygnn.py:576] : Epoch 56 train loss: 4.35e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:29:42,355 [phygnn.py:576] : Epoch 57 train loss: 4.30e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:29:50,894 [phygnn.py:576] : Epoch 58 train loss: 4.32e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:29:59,633 [phygnn.py:576] : Epoch 59 train loss: 4.30e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:30:08,245 [phygnn.py:576] : Epoch 60 train loss: 4.32e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:30:16,848 [phygnn.py:576] : Epoch 61 train loss: 4.31e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:30:25,595 [phygnn.py:576] : Epoch 62 train loss: 4.32e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:30:33,972 [phygnn.py:576] : Epoch 63 train loss: 4.28e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:30:42,534 [phygnn.py:576] : Epoch 64 train loss: 4.27e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:30:51,058 [phygnn.py:576] : Epoch 65 train loss: 4.24e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:30:59,612 [phygnn.py:576] : Epoch 66 train loss: 4.29e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:31:08,181 [phygnn.py:576] : Epoch 67 train loss: 4.27e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:31:16,582 [phygnn.py:576] : Epoch 68 train loss: 4.24e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:31:25,031 [phygnn.py:576] : Epoch 69 train loss: 4.25e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:31:33,495 [phygnn.py:576] : Epoch 70 train loss: 4.31e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:31:42,045 [phygnn.py:576] : Epoch 71 train loss: 4.23e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:31:50,535 [phygnn.py:576] : Epoch 72 train loss: 4.29e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:31:59,036 [phygnn.py:576] : Epoch 73 train loss: 4.24e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:32:07,485 [phygnn.py:576] : Epoch 74 train loss: 4.21e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:32:16,360 [phygnn.py:576] : Epoch 75 train loss: 4.20e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:32:25,072 [phygnn.py:576] : Epoch 76 train loss: 4.20e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:32:33,728 [phygnn.py:576] : Epoch 77 train loss: 4.22e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:32:42,512 [phygnn.py:576] : Epoch 78 train loss: 4.21e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:32:51,342 [phygnn.py:576] : Epoch 79 train loss: 4.21e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:33:00,159 [phygnn.py:576] : Epoch 80 train loss: 4.18e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:33:08,578 [phygnn.py:576] : Epoch 81 train loss: 4.14e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:33:16,879 [phygnn.py:576] : Epoch 82 train loss: 4.14e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:33:25,482 [phygnn.py:576] : Epoch 83 train loss: 4.25e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:33:33,971 [phygnn.py:576] : Epoch 84 train loss: 4.11e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:33:42,658 [phygnn.py:576] : Epoch 85 train loss: 4.20e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:33:51,177 [phygnn.py:576] : Epoch 86 train loss: 4.22e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:33:59,638 [phygnn.py:576] : Epoch 87 train loss: 4.23e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:34:08,279 [phygnn.py:576] : Epoch 88 train loss: 4.17e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:17,191 [phygnn.py:576] : Epoch 89 train loss: 4.12e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:34:25,739 [phygnn.py:576] : Epoch 90 train loss: 4.15e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:34,242 [phygnn.py:576] : Epoch 91 train loss: 4.18e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:34:42,458 [phygnn.py:576] : Epoch 92 train loss: 4.13e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:34:50,801 [phygnn.py:576] : Epoch 93 train loss: 4.16e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:34:59,235 [phygnn.py:576] : Epoch 94 train loss: 4.14e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:35:07,472 [phygnn.py:576] : Epoch 95 train loss: 4.17e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:35:15,845 [phygnn.py:576] : Epoch 96 train loss: 4.13e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:35:24,357 [phygnn.py:576] : Epoch 97 train loss: 4.15e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:35:32,626 [phygnn.py:576] : Epoch 98 train loss: 4.14e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2022-03-01 13:35:41,172 [phygnn.py:576] : Epoch 99 train loss: 4.17e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:35:41,855 [trainer.py:92] : Training part B - data and phygnn. Loss is [0.5, 0.5]
INFO - 2022-03-01 13:36:04,670 [phygnn.py:576] : Epoch 100 train loss: 2.87e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:36:18,027 [phygnn.py:576] : Epoch 101 train loss: 2.81e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:36:31,896 [phygnn.py:576] : Epoch 102 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:36:44,933 [phygnn.py:576] : Epoch 103 train loss: 2.84e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:36:57,924 [phygnn.py:576] : Epoch 104 train loss: 2.79e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:37:10,731 [phygnn.py:576] : Epoch 105 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:37:23,566 [phygnn.py:576] : Epoch 106 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:37:36,649 [phygnn.py:576] : Epoch 107 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:37:49,673 [phygnn.py:576] : Epoch 108 train loss: 2.84e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:38:02,991 [phygnn.py:576] : Epoch 109 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:38:16,465 [phygnn.py:576] : Epoch 110 train loss: 2.83e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:38:30,237 [phygnn.py:576] : Epoch 111 train loss: 2.83e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:38:43,469 [phygnn.py:576] : Epoch 112 train loss: 2.83e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:38:57,253 [phygnn.py:576] : Epoch 113 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:39:10,221 [phygnn.py:576] : Epoch 114 train loss: 2.83e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:39:24,299 [phygnn.py:576] : Epoch 115 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:39:37,674 [phygnn.py:576] : Epoch 116 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:39:51,125 [phygnn.py:576] : Epoch 117 train loss: 2.80e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:40:05,075 [phygnn.py:576] : Epoch 118 train loss: 2.78e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:40:18,600 [phygnn.py:576] : Epoch 119 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:40:31,993 [phygnn.py:576] : Epoch 120 train loss: 2.78e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:40:45,610 [phygnn.py:576] : Epoch 121 train loss: 2.83e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:40:58,719 [phygnn.py:576] : Epoch 122 train loss: 2.79e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:41:12,144 [phygnn.py:576] : Epoch 123 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:41:25,463 [phygnn.py:576] : Epoch 124 train loss: 2.79e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:41:38,863 [phygnn.py:576] : Epoch 125 train loss: 2.83e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:41:52,484 [phygnn.py:576] : Epoch 126 train loss: 2.81e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:05,940 [phygnn.py:576] : Epoch 127 train loss: 2.82e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:19,734 [phygnn.py:576] : Epoch 128 train loss: 2.83e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:33,306 [phygnn.py:576] : Epoch 129 train loss: 2.83e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:46,430 [phygnn.py:576] : Epoch 130 train loss: 2.81e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:59,552 [phygnn.py:576] : Epoch 131 train loss: 2.77e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:43:13,243 [phygnn.py:576] : Epoch 132 train loss: 2.77e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:43:26,828 [phygnn.py:576] : Epoch 133 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:43:40,529 [phygnn.py:576] : Epoch 134 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:43:53,492 [phygnn.py:576] : Epoch 135 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:44:06,739 [phygnn.py:576] : Epoch 136 train loss: 2.79e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:44:19,619 [phygnn.py:576] : Epoch 137 train loss: 2.76e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:44:32,462 [phygnn.py:576] : Epoch 138 train loss: 2.82e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:44:46,030 [phygnn.py:576] : Epoch 139 train loss: 2.78e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:44:59,341 [phygnn.py:576] : Epoch 140 train loss: 2.76e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:45:12,958 [phygnn.py:576] : Epoch 141 train loss: 2.75e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:45:26,566 [phygnn.py:576] : Epoch 142 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:45:40,656 [phygnn.py:576] : Epoch 143 train loss: 2.77e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:45:54,140 [phygnn.py:576] : Epoch 144 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:46:07,436 [phygnn.py:576] : Epoch 145 train loss: 2.75e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:46:20,777 [phygnn.py:576] : Epoch 146 train loss: 2.78e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:46:34,068 [phygnn.py:576] : Epoch 147 train loss: 2.76e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:46:48,187 [phygnn.py:576] : Epoch 148 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:47:01,559 [phygnn.py:576] : Epoch 149 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:47:14,668 [phygnn.py:576] : Epoch 150 train loss: 2.75e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:47:27,909 [phygnn.py:576] : Epoch 151 train loss: 2.76e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:47:41,378 [phygnn.py:576] : Epoch 152 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:47:54,587 [phygnn.py:576] : Epoch 153 train loss: 2.84e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:48:08,145 [phygnn.py:576] : Epoch 154 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:48:21,579 [phygnn.py:576] : Epoch 155 train loss: 2.77e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:48:35,080 [phygnn.py:576] : Epoch 156 train loss: 2.77e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:48:48,397 [phygnn.py:576] : Epoch 157 train loss: 2.81e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:49:01,610 [phygnn.py:576] : Epoch 158 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:49:14,913 [phygnn.py:576] : Epoch 159 train loss: 2.76e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:49:28,334 [phygnn.py:576] : Epoch 160 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:49:41,903 [phygnn.py:576] : Epoch 161 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:49:55,008 [phygnn.py:576] : Epoch 162 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:50:08,340 [phygnn.py:576] : Epoch 163 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:50:21,644 [phygnn.py:576] : Epoch 164 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:50:34,312 [phygnn.py:576] : Epoch 165 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:50:47,865 [phygnn.py:576] : Epoch 166 train loss: 2.78e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:51:01,547 [phygnn.py:576] : Epoch 167 train loss: 2.80e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:51:14,542 [phygnn.py:576] : Epoch 168 train loss: 2.79e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:51:27,620 [phygnn.py:576] : Epoch 169 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:51:40,920 [phygnn.py:576] : Epoch 170 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:51:53,699 [phygnn.py:576] : Epoch 171 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:52:06,376 [phygnn.py:576] : Epoch 172 train loss: 2.74e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:52:19,756 [phygnn.py:576] : Epoch 173 train loss: 2.76e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:52:32,942 [phygnn.py:576] : Epoch 174 train loss: 2.83e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:52:45,458 [phygnn.py:576] : Epoch 175 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:52:58,151 [phygnn.py:576] : Epoch 176 train loss: 2.73e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:10,686 [phygnn.py:576] : Epoch 177 train loss: 2.74e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:23,666 [phygnn.py:576] : Epoch 178 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:36,700 [phygnn.py:576] : Epoch 179 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:50,065 [phygnn.py:576] : Epoch 180 train loss: 2.76e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:54:03,040 [phygnn.py:576] : Epoch 181 train loss: 2.74e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:54:15,613 [phygnn.py:576] : Epoch 182 train loss: 2.74e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:54:28,445 [phygnn.py:576] : Epoch 183 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:54:41,477 [phygnn.py:576] : Epoch 184 train loss: 2.71e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:54:53,954 [phygnn.py:576] : Epoch 185 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:55:07,673 [phygnn.py:576] : Epoch 186 train loss: 2.73e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:55:20,799 [phygnn.py:576] : Epoch 187 train loss: 2.72e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:55:33,148 [phygnn.py:576] : Epoch 188 train loss: 2.69e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:55:45,692 [phygnn.py:576] : Epoch 189 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:55:58,275 [phygnn.py:576] : Epoch 190 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:11,225 [phygnn.py:576] : Epoch 191 train loss: 2.73e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:23,544 [phygnn.py:576] : Epoch 192 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:36,208 [phygnn.py:576] : Epoch 193 train loss: 2.72e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:48,797 [phygnn.py:576] : Epoch 194 train loss: 2.70e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:01,731 [phygnn.py:576] : Epoch 195 train loss: 2.72e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:14,529 [phygnn.py:576] : Epoch 196 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:57:27,171 [phygnn.py:576] : Epoch 197 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:57:39,753 [phygnn.py:576] : Epoch 198 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:52,455 [phygnn.py:576] : Epoch 199 train loss: 2.74e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:53,270 [trainer.py:102] : Training complete
INFO - 2022-03-01 13:57:53,318 [base.py:496] : Saved model to: /home/gbuster/code/mlclouds/mlclouds/model/k_fold/outputs/model_1.pkl
DEBUG - 2022-03-01 13:57:53,318 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 13:57:53,318 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:57:53,355 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:57:54,573 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:57:54,573 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:57:55,736 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:57:55,736 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:57:56,989 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:57:56,989 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:57:58,211 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:57:58,211 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:05,028 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 13:58:05,029 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:06,532 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:58:06,533 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:13,866 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 13:58:13,867 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:17,523 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 13:58:17,523 [data_handlers.py:413] : Shape df_raw=(3154464, 19), df_all_sky=(3154464, 14)
DEBUG - 2022-03-01 13:58:17,523 [data_handlers.py:420] : Shape after reset_index: df_raw=(3154464, 19), df_all_sky=(3154464, 14)
INFO - 2022-03-01 13:58:17,906 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:58:17,912 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 13:58:17,918 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:58:17,923 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:58:17,928 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:58:17,928 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:58:17,931 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:17,937 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:17,942 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:17,947 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:17,952 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.45% NaN values
DEBUG - 2022-03-01 13:58:17,956 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.45% NaN values
DEBUG - 2022-03-01 13:58:17,960 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.28% NaN values
DEBUG - 2022-03-01 13:58:17,965 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 13:58:17,969 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 13:58:17,973 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.27% NaN values
DEBUG - 2022-03-01 13:58:17,978 [data_cleaners.py:50] : 	"cloud_probability" has 3.27% NaN values
DEBUG - 2022-03-01 13:58:17,982 [data_cleaners.py:50] : 	"cloud_fraction" has 3.27% NaN values
DEBUG - 2022-03-01 13:58:17,986 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:17,990 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:17,995 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:17,999 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:18,003 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:18,008 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 13:58:18,012 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 13:58:18,012 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:58:21,914 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 13:58:22,316 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 13:58:22,316 [data_cleaners.py:107] : Cleaning took 4.8 seconds
INFO - 2022-03-01 13:58:22,684 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:58:22,689 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 13:58:22,694 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:58:22,700 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:58:22,705 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:58:22,705 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:58:22,708 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,714 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,719 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,723 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,727 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,733 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,737 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 13:58:22,742 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 13:58:22,746 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,750 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,755 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,759 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,762 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,766 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:58:22,766 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:58:24,980 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 13:58:25,438 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 13:58:25,438 [data_cleaners.py:107] : Cleaning took 3.1 seconds
DEBUG - 2022-03-01 13:58:25,440 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 13:58:25,483 [data_handlers.py:463] : Mask: shape=(3154464,), sum=1567353
DEBUG - 2022-03-01 13:58:25,663 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 13:58:25,663 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 13:58:29,540 [validator.py:110] : Predicted data shape is (1567353, 2)
DEBUG - 2022-03-01 13:58:30,059 [validator.py:158] : shapes: df_feature_val=(3154464, 20), df_all_sky_val=(3154464, 15)
INFO - 2022-03-01 13:58:30,371 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 13:58:30,374 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:30,374 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 13:58:30,374 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:58:30,806 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 13:58:31,537 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:58:31,576 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 13:58:32,384 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:58:32,422 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 13:58:33,199 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:58:33,237 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 13:58:33,991 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:58:34,030 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 13:58:34,823 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:58:34,862 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 13:58:35,634 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:58:35,673 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 13:58:36,416 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:58:36,455 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 13:58:37,200 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:58:37,238 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 13:58:37,980 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 13:58:37,980 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:58:38,507 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 13:58:39,151 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:58:39,189 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 13:58:39,833 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:58:39,869 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 13:58:40,512 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:58:40,548 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 13:58:41,195 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:58:41,231 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 13:58:41,870 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:58:41,908 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 13:58:42,552 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:58:42,589 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 13:58:43,230 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:58:43,267 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 13:58:43,906 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:58:43,943 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 13:58:44,613 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 13:58:44,613 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:58:45,070 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 13:58:45,907 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:58:45,943 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 13:58:46,700 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:58:46,737 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 13:58:47,492 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:58:47,529 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 13:58:48,297 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:58:48,334 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 13:58:49,036 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:58:49,071 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 13:58:49,818 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:58:49,854 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 13:58:50,635 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:58:50,671 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 13:58:51,371 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:58:51,408 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 13:58:52,154 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 13:58:52,154 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:58:52,662 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 13:58:53,312 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:58:53,348 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 13:58:53,990 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:58:54,026 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 13:58:54,669 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:58:54,705 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 13:58:55,350 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:58:55,386 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 13:58:56,027 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:58:56,062 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 13:58:56,715 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:58:56,750 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 13:58:57,393 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:58:57,428 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 13:58:58,078 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:58:58,114 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 13:58:58,758 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 13:58:58,758 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:58:59,406 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 13:59:00,206 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:00,319 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 13:59:01,146 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:01,260 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 13:59:02,035 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:02,152 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 13:59:02,892 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:03,008 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 13:59:03,811 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:03,924 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 13:59:04,725 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:04,839 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 13:59:05,631 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:05,744 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 13:59:06,504 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:06,617 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 13:59:07,414 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 13:59:07,414 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:07,859 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 13:59:08,517 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:08,553 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 13:59:09,212 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:09,248 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 13:59:09,905 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:09,941 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 13:59:10,603 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:10,639 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 13:59:11,296 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:11,332 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 13:59:11,990 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:12,026 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 13:59:12,684 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:12,720 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 13:59:13,385 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:13,421 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 13:59:14,081 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 13:59:14,081 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:14,638 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 13:59:15,378 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:15,491 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 13:59:16,292 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:16,405 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 13:59:17,216 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:17,329 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 13:59:18,192 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:18,308 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 13:59:19,070 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:19,185 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 13:59:20,101 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:20,214 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 13:59:20,982 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:21,097 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 13:59:21,828 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:21,941 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 13:59:22,728 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 13:59:22,728 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:23,209 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 13:59:23,905 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:23,976 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 13:59:24,652 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:24,720 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 13:59:25,411 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:25,479 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 13:59:26,171 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:26,239 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 13:59:26,933 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:27,001 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 13:59:27,703 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:27,770 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 13:59:28,465 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:28,532 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 13:59:29,242 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:29,309 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 13:59:30,019 [validator.py:187] : Shapes: df_base_full=(3154464, 6), df_surf_full=(3154464, 4)
DEBUG - 2022-03-01 13:59:30,024 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 13:59:30,062 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 13:59:45,885 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 13:59:45,923 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:00:01,696 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:00:01,734 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:00:17,429 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:00:17,467 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:00:33,270 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:00:33,309 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:00:49,271 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:00:49,309 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:05,187 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:01:05,225 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:21,059 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:01:21,097 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:37,022 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:01:37,060 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
INFO - 2022-03-01 14:01:52,853 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:01:52,867 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:01:52,867 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:01:52,870 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:01:53,907 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:01:53,907 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:01:54,961 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:01:54,962 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:01,511 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:02:01,511 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:08,310 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:02:08,310 [data_handlers.py:413] : Shape df_raw=(2207952, 19), df_all_sky=(2207952, 14)
DEBUG - 2022-03-01 14:02:08,310 [data_handlers.py:420] : Shape after reset_index: df_raw=(2207952, 19), df_all_sky=(2207952, 14)
INFO - 2022-03-01 14:02:08,542 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:02:08,546 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:02:08,550 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:02:08,554 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:02:08,557 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:02:08,557 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:02:08,560 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,564 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,567 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,571 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,574 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 49.99% NaN values
DEBUG - 2022-03-01 14:02:08,577 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 49.99% NaN values
DEBUG - 2022-03-01 14:02:08,580 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 0.35% NaN values
DEBUG - 2022-03-01 14:02:08,584 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:02:08,587 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:02:08,590 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 0.34% NaN values
DEBUG - 2022-03-01 14:02:08,593 [data_cleaners.py:50] : 	"cloud_probability" has 0.34% NaN values
DEBUG - 2022-03-01 14:02:08,596 [data_cleaners.py:50] : 	"cloud_fraction" has 0.34% NaN values
DEBUG - 2022-03-01 14:02:08,599 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,602 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,605 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,609 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,612 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:08,615 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:02:08,618 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:02:08,618 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:02:11,072 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:02:11,348 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:02:11,348 [data_cleaners.py:107] : Cleaning took 3.0 seconds
INFO - 2022-03-01 14:02:11,582 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:02:11,586 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:02:11,590 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:02:11,593 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:02:11,597 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:02:11,597 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:02:11,599 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,604 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,607 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,610 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,613 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,617 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,620 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:02:11,623 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:02:11,627 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,630 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,633 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,636 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,638 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,641 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:11,641 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:02:13,089 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:02:13,366 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:02:13,366 [data_cleaners.py:107] : Cleaning took 2.0 seconds
DEBUG - 2022-03-01 14:02:13,368 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:02:13,390 [data_handlers.py:463] : Mask: shape=(2207952,), sum=1097157
DEBUG - 2022-03-01 14:02:13,472 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:02:13,472 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:02:16,096 [validator.py:110] : Predicted data shape is (1097157, 2)
DEBUG - 2022-03-01 14:02:16,414 [validator.py:158] : shapes: df_feature_val=(2207952, 20), df_all_sky_val=(2207952, 15)
INFO - 2022-03-01 14:02:16,625 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:02:16,629 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:16,629 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:02:16,629 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:02:16,664 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:02:17,306 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:02:17,343 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:02:17,980 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:02:18,017 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:02:18,654 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:02:18,691 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:02:19,336 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:02:19,374 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:02:20,012 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:02:20,049 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:02:20,691 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:02:20,729 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:02:21,364 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:02:21,401 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:02:22,034 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:02:22,073 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:02:22,730 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:02:22,730 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:02:22,766 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:02:23,411 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:02:23,449 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:02:24,086 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:02:24,124 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:02:24,761 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:02:24,799 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:02:25,438 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:02:25,474 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:02:26,113 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:02:26,151 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:02:26,797 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:02:26,832 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:02:27,470 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:02:27,506 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:02:28,149 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:02:28,185 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:02:28,825 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:02:28,825 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:02:28,938 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:02:29,587 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:02:29,701 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:02:30,354 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:02:30,466 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:02:31,120 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:02:31,236 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:02:31,892 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:02:32,007 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:02:32,663 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:02:32,775 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:02:33,435 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:02:33,549 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:02:34,213 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:02:34,325 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:02:34,982 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:02:35,095 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:02:35,759 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:02:35,760 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:02:35,875 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:02:36,542 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:02:36,655 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:02:37,321 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:02:37,437 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:02:38,117 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:02:38,231 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:02:38,901 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:02:39,014 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:02:39,694 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:02:39,807 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:02:40,478 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:02:40,590 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:02:41,278 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:02:41,392 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:02:42,083 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:02:42,198 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:02:42,887 [validator.py:187] : Shapes: df_base_full=(2207952, 6), df_surf_full=(2207952, 4)
DEBUG - 2022-03-01 14:02:42,891 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:02:42,918 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:02:54,399 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:02:54,426 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:03:05,783 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:03:05,810 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:03:17,091 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:03:17,118 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:03:28,460 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:03:28,487 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:03:39,884 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:03:39,911 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:03:51,328 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:03:51,355 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:02,729 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:04:02,756 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:14,160 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:04:14,187 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
INFO - 2022-03-01 14:04:25,856 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:04:25,884 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:04:25,884 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:04:25,887 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:26,966 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:26,966 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:28,094 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:28,095 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:29,231 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:29,231 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:32,517 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 14:04:32,517 [data_handlers.py:413] : Shape df_raw=(946512, 19), df_all_sky=(946512, 14)
DEBUG - 2022-03-01 14:04:32,517 [data_handlers.py:420] : Shape after reset_index: df_raw=(946512, 19), df_all_sky=(946512, 14)
INFO - 2022-03-01 14:04:32,622 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:04:32,624 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:04:32,626 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:04:32,628 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:04:32,629 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:04:32,629 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:04:32,630 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,632 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,634 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,636 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,637 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 54.84% NaN values
DEBUG - 2022-03-01 14:04:32,639 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 54.84% NaN values
DEBUG - 2022-03-01 14:04:32,640 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 10.12% NaN values
DEBUG - 2022-03-01 14:04:32,642 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:04:32,644 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:04:32,645 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 10.10% NaN values
DEBUG - 2022-03-01 14:04:32,647 [data_cleaners.py:50] : 	"cloud_probability" has 10.10% NaN values
DEBUG - 2022-03-01 14:04:32,648 [data_cleaners.py:50] : 	"cloud_fraction" has 10.10% NaN values
DEBUG - 2022-03-01 14:04:32,650 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,651 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,653 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,654 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,656 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:32,657 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:04:32,659 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:04:32,659 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:04:33,718 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:04:33,843 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:04:33,844 [data_cleaners.py:107] : Cleaning took 1.3 seconds
INFO - 2022-03-01 14:04:33,943 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:04:33,945 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:04:33,947 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:04:33,949 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:04:33,950 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:04:33,950 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:04:33,952 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,954 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,955 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,957 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,958 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,960 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,962 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:04:33,963 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:04:33,965 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,966 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,968 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,969 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,971 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,972 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:04:33,972 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:04:34,567 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:04:34,691 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:04:34,691 [data_cleaners.py:107] : Cleaning took 0.8 seconds
DEBUG - 2022-03-01 14:04:34,691 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:04:34,704 [data_handlers.py:463] : Mask: shape=(946512,), sum=470196
DEBUG - 2022-03-01 14:04:34,740 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:04:34,740 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:04:35,850 [validator.py:110] : Predicted data shape is (470196, 2)
DEBUG - 2022-03-01 14:04:35,963 [validator.py:158] : shapes: df_feature_val=(946512, 20), df_all_sky_val=(946512, 15)
INFO - 2022-03-01 14:04:36,052 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:04:36,055 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:36,055 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:04:36,055 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:04:36,104 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:04:36,748 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:04:36,784 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:04:37,421 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:04:37,458 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:04:38,098 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:04:38,136 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:04:38,781 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:04:38,818 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:04:39,457 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:04:39,494 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:04:40,137 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:04:40,174 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:04:40,814 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:04:40,853 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:04:41,489 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:04:41,527 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:04:42,169 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:04:42,169 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:04:42,217 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:04:42,859 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:04:42,898 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:04:43,566 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:04:43,602 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:04:44,267 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:04:44,303 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:04:44,948 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:04:44,983 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:04:45,625 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:04:45,661 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:04:46,313 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:04:46,350 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:04:46,992 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:04:47,028 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:04:47,686 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:04:47,722 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:04:48,365 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:04:48,365 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:04:48,413 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:04:49,055 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:04:49,091 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:04:49,731 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:04:49,768 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:04:50,421 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:04:50,457 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:04:51,098 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:04:51,134 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:04:51,774 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:04:51,810 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:04:52,460 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:04:52,495 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:04:53,132 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:04:53,168 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:04:53,814 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:04:53,850 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:04:54,493 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:04:54,493 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:04:54,589 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:04:55,250 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:04:55,318 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:04:55,981 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:04:56,049 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:04:56,716 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:04:56,784 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:04:57,443 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:04:57,511 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:04:58,164 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:04:58,231 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:04:58,878 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:04:58,944 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:04:59,590 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:04:59,657 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:05:00,316 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:00,383 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:05:01,041 [validator.py:187] : Shapes: df_base_full=(946512, 6), df_surf_full=(946512, 4)
DEBUG - 2022-03-01 14:05:01,046 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:05:01,058 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:06,634 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:05:06,646 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:12,196 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:05:12,208 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:17,723 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:05:17,735 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:23,279 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:05:23,291 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:28,862 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:05:28,874 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:34,454 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:05:34,466 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:40,029 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:05:40,041 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:45,612 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:05:45,624 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
INFO - 2022-03-01 14:05:51,168 [validator.py:292] : Finished computing stats.
