Starting scenario 4, validation against site 5
2022-03-01 13:17:48.848562: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:17:48.848591: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Cross validation site: 5
Training sites: [0, 1, 2, 3, 4, 6, 7, 8]
Number of files: 8
Number of east files: 4
Number of west files: 4
Source files: ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
Full config: {'clean_training_data_kwargs': {'filter_clear': False, 'nan_option': 'interp'}, 'epochs_a': 100, 'epochs_b': 100, 'features': ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo'], 'hidden_layers': [{'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}], 'learning_rate': 0.0005, 'loss_weights_a': [1, 0], 'loss_weights_b': [0.5, 0.5], 'metric': 'relative_mae', 'n_batch': 32, 'one_hot_categories': {'flag': ['clear', 'ice_cloud', 'water_cloud', 'bad_cloud']}, 'p_fun': 'p_fun_all_sky', 'p_kwargs': {'loss_terms': ['mae_ghi']}, 'phygnn_seed': 0, 'surfrad_window_minutes': 15, 'y_labels': ['cld_opd_dcomp', 'cld_reff_dcomp']}
INFO - 2022-03-01 13:17:55,728 [trainer.py:40] : Trainer: Training on sites [0, 1, 2, 3, 4, 6, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:17:55,728 [trainer.py:49] : Trainer: Training on sites [0, 1, 2, 3, 4, 6, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:17:55,728 [data_handlers.py:60] : Loading training data
DEBUG - 2022-03-01 13:17:55,728 [data_handlers.py:78] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:17:55,728 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5
DEBUG - 2022-03-01 13:17:56,901 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:17:56,905 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:17:56,906 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:17:56,911 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:17:57,608 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:57,612 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:17:58,290 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:58,294 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:17:58,982 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:58,985 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:17:59,678 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:59,682 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:18:00,366 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:00,369 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:18:01,050 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:01,054 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:18:01,736 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:01,739 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:18:02,438 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:02,438 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5
DEBUG - 2022-03-01 13:18:03,491 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:18:03,496 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:03,496 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:18:03,499 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:18:04,161 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:04,165 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:18:04,825 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:04,829 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:18:05,488 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:05,492 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:18:06,156 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:06,160 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:18:06,822 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:06,825 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:18:07,483 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:07,486 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:18:08,145 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:08,149 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:18:08,811 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:08,811 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5
DEBUG - 2022-03-01 13:18:09,853 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:09,857 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:09,857 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:18:09,861 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:18:10,565 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:10,569 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:18:11,232 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:11,235 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:18:11,897 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:11,900 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:18:12,564 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:12,567 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:18:13,229 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:13,233 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:18:13,895 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:13,898 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:18:14,566 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:14,569 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:18:15,242 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:15,243 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5
DEBUG - 2022-03-01 13:18:16,315 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:16,320 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:16,320 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:18:16,324 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:18:16,981 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:16,984 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:18:17,628 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:17,631 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:18:18,273 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:18,276 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:18:18,924 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:18,927 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:18:19,563 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:19,566 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:18:20,201 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:20,205 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:18:20,849 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:20,853 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:18:21,495 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:21,495 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5
DEBUG - 2022-03-01 13:18:27,705 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:18:27,725 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:18:27,725 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:18:27,728 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:18:28,429 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:28,432 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:18:29,100 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:29,103 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:18:29,776 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:29,779 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:18:30,449 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:30,452 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:18:31,122 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:31,125 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:18:31,835 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:31,839 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:18:32,507 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:32,510 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:18:33,267 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:33,267 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5
DEBUG - 2022-03-01 13:18:34,502 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:34,506 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:34,506 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:18:34,510 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:18:35,154 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:35,158 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:18:35,822 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:35,826 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:18:36,485 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:36,488 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:18:37,136 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:37,139 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:18:37,788 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:37,791 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:18:38,438 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:38,442 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:18:39,094 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:39,097 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:18:39,749 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:39,749 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5
DEBUG - 2022-03-01 13:18:46,175 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:18:46,196 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:18:46,196 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:18:46,200 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:18:46,874 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:46,878 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:18:47,693 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:47,697 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:18:48,373 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:48,376 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:18:49,062 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:49,065 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:18:49,761 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:49,764 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:18:50,500 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:50,504 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:18:51,205 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:51,209 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:18:51,937 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:51,937 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5
DEBUG - 2022-03-01 13:18:55,314 [data_handlers.py:103] : 	Shape temp_raw=(420480, 19), temp_all_sky=(420480, 14)
DEBUG - 2022-03-01 13:18:55,324 [data_handlers.py:106] : 	Time step is 10 minutes
DEBUG - 2022-03-01 13:18:55,324 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 6, 7, 8]
DEBUG - 2022-03-01 13:18:55,328 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:18:56,000 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:56,004 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:18:56,678 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:56,682 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:18:57,356 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:57,360 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:18:58,036 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:58,039 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:18:58,718 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:58,721 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:18:59,395 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:59,399 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:19:00,074 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:00,078 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:19:00,756 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:00,756 [data_handlers.py:136] : Data load complete. Shape df_raw=(2803968, 19), df_all_sky=(2803968, 14), df_surf=(2803968, 5)
DEBUG - 2022-03-01 13:19:01,636 [data_handlers.py:159] : Extracting 2D arrays to run rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:19:13,690 [data_handlers.py:176] : Running rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:21:07,139 [data_handlers.py:194] : Completed rest2 run for clearsky PhyGNN inputs.
INFO - 2022-03-01 13:21:08,800 [data_handlers.py:62] : Prepping training data
DEBUG - 2022-03-01 13:21:08,800 [data_handlers.py:214] : Training data clean kwargs: {'filter_daylight': True, 'filter_clear': False, 'add_cloud_flag': True, 'sza_lim': 89, 'nan_option': 'interp'}
DEBUG - 2022-03-01 13:21:08,800 [data_handlers.py:215] : Shape before cleaning: df_raw=(2803968, 19)
INFO - 2022-03-01 13:21:09,062 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 13:21:09,067 [data_cleaners.py:38] : 50.38% of daylight timesteps are cloudy
INFO - 2022-03-01 13:21:09,071 [data_cleaners.py:40] : 2.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:21:09,076 [data_cleaners.py:42] : 31.42% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:21:09,081 [data_cleaners.py:44] : 31.65% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:21:09,081 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:21:09,083 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,089 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,093 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,098 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,102 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 50.84% NaN values
DEBUG - 2022-03-01 13:21:09,105 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 50.84% NaN values
DEBUG - 2022-03-01 13:21:09,109 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 2.08% NaN values
DEBUG - 2022-03-01 13:21:09,113 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 1.98% NaN values
DEBUG - 2022-03-01 13:21:09,117 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 1.98% NaN values
DEBUG - 2022-03-01 13:21:09,121 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 2.06% NaN values
DEBUG - 2022-03-01 13:21:09,125 [data_cleaners.py:50] : 	"cloud_probability" has 2.06% NaN values
DEBUG - 2022-03-01 13:21:09,128 [data_cleaners.py:50] : 	"cloud_fraction" has 2.06% NaN values
DEBUG - 2022-03-01 13:21:09,132 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,136 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,140 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,144 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,148 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:09,152 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.83% NaN values
DEBUG - 2022-03-01 13:21:09,155 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.89% NaN values
DEBUG - 2022-03-01 13:21:09,155 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:21:12,279 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:21:12,554 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393022 after filters (49.68% of original)
DEBUG - 2022-03-01 13:21:12,677 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:21:12,677 [data_cleaners.py:107] : Cleaning took 3.9 seconds
DEBUG - 2022-03-01 13:21:12,677 [data_handlers.py:218] : Shape after cleaning: df_train=(1393022, 20)
DEBUG - 2022-03-01 13:21:12,677 [data_handlers.py:221] : Cleaning df_all_sky training data (for pfun).
DEBUG - 2022-03-01 13:21:12,677 [data_handlers.py:222] : Shape before cleaning: df_all_sky=(2803968, 25)
INFO - 2022-03-01 13:21:13,013 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 13:21:13,018 [data_cleaners.py:38] : 50.38% of daylight timesteps are cloudy
INFO - 2022-03-01 13:21:13,022 [data_cleaners.py:40] : 2.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:21:13,027 [data_cleaners.py:42] : 31.42% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:21:13,032 [data_cleaners.py:44] : 31.65% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:21:13,032 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:21:13,035 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,038 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,042 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,046 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,051 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,055 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.83% NaN values
DEBUG - 2022-03-01 13:21:13,059 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.89% NaN values
DEBUG - 2022-03-01 13:21:13,063 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,067 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,071 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,074 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,077 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,081 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,086 [data_cleaners.py:50] : 	"surfrad_dhi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,091 [data_cleaners.py:50] : 	"surfrad_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,096 [data_cleaners.py:50] : 	"surfrad_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,099 [data_cleaners.py:50] : 	"doy" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,104 [data_cleaners.py:50] : 	"radius" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,109 [data_cleaners.py:50] : 	"Tuuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,114 [data_cleaners.py:50] : 	"clearsky_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,119 [data_cleaners.py:50] : 	"clearsky_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,125 [data_cleaners.py:50] : 	"Ruuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,130 [data_cleaners.py:50] : 	"Tddclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,135 [data_cleaners.py:50] : 	"Tduclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,140 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,140 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:21:15,686 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:21:15,962 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393022 after filters (49.68% of original)
DEBUG - 2022-03-01 13:21:16,115 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:21:16,115 [data_cleaners.py:107] : Cleaning took 3.4 seconds
DEBUG - 2022-03-01 13:21:16,116 [data_handlers.py:226] : Shape after cleaning: df_all_sky=(1393022, 26)
DEBUG - 2022-03-01 13:21:16,211 [data_handlers.py:240] : **Shape: df_train=(1393022, 17)
DEBUG - 2022-03-01 13:21:16,238 [data_handlers.py:250] : Shapes: x=(1393022, 15), y=(1393022, 2), p=(1393022, 26)
DEBUG - 2022-03-01 13:21:16,239 [data_handlers.py:253] : Training features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
DEBUG - 2022-03-01 13:21:16,239 [trainer.py:67] : Building PHYGNN model
INFO - 2022-03-01 13:21:16,239 [trainer.py:70] : Using p_fun: <function p_fun_all_sky at 0x2b782889f8b0>
INFO - 2022-03-01 13:21:16,239 [base.py:152] : Active python environment versions: 
{   'numpy': '1.22.2',
    'pandas': '1.2.4',
    'phygnn': '0.0.14',
    'python': '3.8.8 (default, Feb 24 2021, 21:46:12) \n[GCC 7.3.0]',
    'sklearn': '0.24.1',
    'tensorflow': '2.8.0'}
INFO - 2022-03-01 13:21:16,255 [base.py:111] : Successfully initialized model with 17 layers
INFO - 2022-03-01 13:21:16,256 [trainer.py:84] : Training part A - pure data. Loss is [1, 0]
2022-03-01 13:21:25.290919: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.291838: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.292676: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.293626: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.294374: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.295122: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.295915: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.296687: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:25.296704: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-03-01 13:21:25.297063: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO - 2022-03-01 13:21:33,889 [phygnn.py:576] : Epoch 0 train loss: 6.89e-01 val loss: 6.76e-01 for "phygnn"
INFO - 2022-03-01 13:21:42,630 [phygnn.py:576] : Epoch 1 train loss: 6.28e-01 val loss: 6.15e-01 for "phygnn"
INFO - 2022-03-01 13:21:51,342 [phygnn.py:576] : Epoch 2 train loss: 5.42e-01 val loss: 5.30e-01 for "phygnn"
INFO - 2022-03-01 13:21:59,858 [phygnn.py:576] : Epoch 3 train loss: 5.12e-01 val loss: 4.92e-01 for "phygnn"
INFO - 2022-03-01 13:22:08,288 [phygnn.py:576] : Epoch 4 train loss: 4.91e-01 val loss: 4.70e-01 for "phygnn"
INFO - 2022-03-01 13:22:17,340 [phygnn.py:576] : Epoch 5 train loss: 4.76e-01 val loss: 4.59e-01 for "phygnn"
INFO - 2022-03-01 13:22:26,351 [phygnn.py:576] : Epoch 6 train loss: 4.69e-01 val loss: 4.53e-01 for "phygnn"
INFO - 2022-03-01 13:22:35,097 [phygnn.py:576] : Epoch 7 train loss: 4.64e-01 val loss: 4.47e-01 for "phygnn"
INFO - 2022-03-01 13:22:43,496 [phygnn.py:576] : Epoch 8 train loss: 4.60e-01 val loss: 4.43e-01 for "phygnn"
INFO - 2022-03-01 13:22:51,843 [phygnn.py:576] : Epoch 9 train loss: 4.61e-01 val loss: 4.40e-01 for "phygnn"
INFO - 2022-03-01 13:23:00,160 [phygnn.py:576] : Epoch 10 train loss: 4.51e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2022-03-01 13:23:08,583 [phygnn.py:576] : Epoch 11 train loss: 4.50e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2022-03-01 13:23:17,244 [phygnn.py:576] : Epoch 12 train loss: 4.47e-01 val loss: 4.34e-01 for "phygnn"
INFO - 2022-03-01 13:23:25,566 [phygnn.py:576] : Epoch 13 train loss: 4.47e-01 val loss: 4.32e-01 for "phygnn"
INFO - 2022-03-01 13:23:34,142 [phygnn.py:576] : Epoch 14 train loss: 4.45e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:23:43,223 [phygnn.py:576] : Epoch 15 train loss: 4.43e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:23:51,847 [phygnn.py:576] : Epoch 16 train loss: 4.43e-01 val loss: 4.28e-01 for "phygnn"
INFO - 2022-03-01 13:24:00,356 [phygnn.py:576] : Epoch 17 train loss: 4.41e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:24:08,865 [phygnn.py:576] : Epoch 18 train loss: 4.39e-01 val loss: 4.24e-01 for "phygnn"
INFO - 2022-03-01 13:24:17,642 [phygnn.py:576] : Epoch 19 train loss: 4.32e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:24:26,631 [phygnn.py:576] : Epoch 20 train loss: 4.32e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:24:35,719 [phygnn.py:576] : Epoch 21 train loss: 4.38e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:24:44,463 [phygnn.py:576] : Epoch 22 train loss: 4.36e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:24:52,988 [phygnn.py:576] : Epoch 23 train loss: 4.31e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:25:01,865 [phygnn.py:576] : Epoch 24 train loss: 4.32e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:25:10,424 [phygnn.py:576] : Epoch 25 train loss: 4.31e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:25:19,037 [phygnn.py:576] : Epoch 26 train loss: 4.30e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:25:27,630 [phygnn.py:576] : Epoch 27 train loss: 4.27e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:25:36,532 [phygnn.py:576] : Epoch 28 train loss: 4.32e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:25:45,249 [phygnn.py:576] : Epoch 29 train loss: 4.29e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:25:54,270 [phygnn.py:576] : Epoch 30 train loss: 4.33e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:26:02,843 [phygnn.py:576] : Epoch 31 train loss: 4.24e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:26:11,367 [phygnn.py:576] : Epoch 32 train loss: 4.21e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:26:19,780 [phygnn.py:576] : Epoch 33 train loss: 4.19e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:26:28,717 [phygnn.py:576] : Epoch 34 train loss: 4.26e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:26:37,483 [phygnn.py:576] : Epoch 35 train loss: 4.27e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:26:45,820 [phygnn.py:576] : Epoch 36 train loss: 4.18e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:26:54,667 [phygnn.py:576] : Epoch 37 train loss: 4.19e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:27:03,021 [phygnn.py:576] : Epoch 38 train loss: 4.18e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:27:11,359 [phygnn.py:576] : Epoch 39 train loss: 4.23e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:27:20,100 [phygnn.py:576] : Epoch 40 train loss: 4.20e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:27:28,612 [phygnn.py:576] : Epoch 41 train loss: 4.18e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:27:37,199 [phygnn.py:576] : Epoch 42 train loss: 4.21e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:27:46,387 [phygnn.py:576] : Epoch 43 train loss: 4.22e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:27:54,780 [phygnn.py:576] : Epoch 44 train loss: 4.23e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:28:03,410 [phygnn.py:576] : Epoch 45 train loss: 4.16e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:28:12,009 [phygnn.py:576] : Epoch 46 train loss: 4.15e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:28:20,562 [phygnn.py:576] : Epoch 47 train loss: 4.12e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:28:28,914 [phygnn.py:576] : Epoch 48 train loss: 4.21e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:28:37,296 [phygnn.py:576] : Epoch 49 train loss: 4.15e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:28:45,692 [phygnn.py:576] : Epoch 50 train loss: 4.19e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:28:54,761 [phygnn.py:576] : Epoch 51 train loss: 4.17e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:29:03,457 [phygnn.py:576] : Epoch 52 train loss: 4.22e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:29:11,886 [phygnn.py:576] : Epoch 53 train loss: 4.13e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:29:20,183 [phygnn.py:576] : Epoch 54 train loss: 4.12e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:29:28,438 [phygnn.py:576] : Epoch 55 train loss: 4.14e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:29:37,017 [phygnn.py:576] : Epoch 56 train loss: 4.12e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:29:46,206 [phygnn.py:576] : Epoch 57 train loss: 4.14e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:29:54,891 [phygnn.py:576] : Epoch 58 train loss: 4.10e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:30:03,855 [phygnn.py:576] : Epoch 59 train loss: 4.15e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:30:12,893 [phygnn.py:576] : Epoch 60 train loss: 4.11e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:30:21,578 [phygnn.py:576] : Epoch 61 train loss: 4.07e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:30:30,588 [phygnn.py:576] : Epoch 62 train loss: 4.14e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:30:39,150 [phygnn.py:576] : Epoch 63 train loss: 4.12e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:30:47,918 [phygnn.py:576] : Epoch 64 train loss: 4.10e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:30:56,844 [phygnn.py:576] : Epoch 65 train loss: 4.07e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:31:05,172 [phygnn.py:576] : Epoch 66 train loss: 4.15e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:31:13,755 [phygnn.py:576] : Epoch 67 train loss: 4.12e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2022-03-01 13:31:22,586 [phygnn.py:576] : Epoch 68 train loss: 4.01e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2022-03-01 13:31:31,131 [phygnn.py:576] : Epoch 69 train loss: 4.12e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:31:39,806 [phygnn.py:576] : Epoch 70 train loss: 4.11e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2022-03-01 13:31:48,645 [phygnn.py:576] : Epoch 71 train loss: 4.13e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2022-03-01 13:31:57,560 [phygnn.py:576] : Epoch 72 train loss: 4.08e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:32:06,642 [phygnn.py:576] : Epoch 73 train loss: 4.08e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2022-03-01 13:32:15,593 [phygnn.py:576] : Epoch 74 train loss: 4.08e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:32:24,038 [phygnn.py:576] : Epoch 75 train loss: 4.05e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:32:32,559 [phygnn.py:576] : Epoch 76 train loss: 4.07e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:32:41,099 [phygnn.py:576] : Epoch 77 train loss: 4.06e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2022-03-01 13:32:49,959 [phygnn.py:576] : Epoch 78 train loss: 4.03e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:32:58,527 [phygnn.py:576] : Epoch 79 train loss: 4.10e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2022-03-01 13:33:07,037 [phygnn.py:576] : Epoch 80 train loss: 4.09e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:33:15,659 [phygnn.py:576] : Epoch 81 train loss: 4.00e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:33:24,619 [phygnn.py:576] : Epoch 82 train loss: 4.11e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:33:33,464 [phygnn.py:576] : Epoch 83 train loss: 4.08e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:33:41,786 [phygnn.py:576] : Epoch 84 train loss: 4.02e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:33:50,557 [phygnn.py:576] : Epoch 85 train loss: 4.06e-01 val loss: 3.94e-01 for "phygnn"
INFO - 2022-03-01 13:33:58,870 [phygnn.py:576] : Epoch 86 train loss: 4.04e-01 val loss: 3.95e-01 for "phygnn"
INFO - 2022-03-01 13:34:07,226 [phygnn.py:576] : Epoch 87 train loss: 4.04e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:34:16,382 [phygnn.py:576] : Epoch 88 train loss: 4.04e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:34:25,236 [phygnn.py:576] : Epoch 89 train loss: 4.04e-01 val loss: 3.94e-01 for "phygnn"
INFO - 2022-03-01 13:34:33,948 [phygnn.py:576] : Epoch 90 train loss: 4.00e-01 val loss: 3.93e-01 for "phygnn"
INFO - 2022-03-01 13:34:42,355 [phygnn.py:576] : Epoch 91 train loss: 3.99e-01 val loss: 3.95e-01 for "phygnn"
INFO - 2022-03-01 13:34:51,215 [phygnn.py:576] : Epoch 92 train loss: 4.05e-01 val loss: 3.93e-01 for "phygnn"
INFO - 2022-03-01 13:35:00,138 [phygnn.py:576] : Epoch 93 train loss: 4.01e-01 val loss: 3.94e-01 for "phygnn"
INFO - 2022-03-01 13:35:08,793 [phygnn.py:576] : Epoch 94 train loss: 4.05e-01 val loss: 3.95e-01 for "phygnn"
INFO - 2022-03-01 13:35:17,546 [phygnn.py:576] : Epoch 95 train loss: 4.08e-01 val loss: 3.95e-01 for "phygnn"
INFO - 2022-03-01 13:35:26,290 [phygnn.py:576] : Epoch 96 train loss: 4.00e-01 val loss: 3.94e-01 for "phygnn"
INFO - 2022-03-01 13:35:35,031 [phygnn.py:576] : Epoch 97 train loss: 4.01e-01 val loss: 3.94e-01 for "phygnn"
INFO - 2022-03-01 13:35:43,857 [phygnn.py:576] : Epoch 98 train loss: 4.01e-01 val loss: 3.95e-01 for "phygnn"
INFO - 2022-03-01 13:35:52,361 [phygnn.py:576] : Epoch 99 train loss: 3.99e-01 val loss: 3.92e-01 for "phygnn"
INFO - 2022-03-01 13:35:53,082 [trainer.py:92] : Training part B - data and phygnn. Loss is [0.5, 0.5]
INFO - 2022-03-01 13:36:16,773 [phygnn.py:576] : Epoch 100 train loss: 2.69e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:36:31,760 [phygnn.py:576] : Epoch 101 train loss: 2.73e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:36:45,860 [phygnn.py:576] : Epoch 102 train loss: 2.78e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:36:59,994 [phygnn.py:576] : Epoch 103 train loss: 2.71e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:37:12,919 [phygnn.py:576] : Epoch 104 train loss: 2.73e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:37:26,740 [phygnn.py:576] : Epoch 105 train loss: 2.72e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:37:39,818 [phygnn.py:576] : Epoch 106 train loss: 2.71e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:37:53,603 [phygnn.py:576] : Epoch 107 train loss: 2.73e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:38:07,671 [phygnn.py:576] : Epoch 108 train loss: 2.74e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:38:21,273 [phygnn.py:576] : Epoch 109 train loss: 2.72e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:38:34,592 [phygnn.py:576] : Epoch 110 train loss: 2.74e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:38:48,473 [phygnn.py:576] : Epoch 111 train loss: 2.69e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:39:02,292 [phygnn.py:576] : Epoch 112 train loss: 2.73e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:39:16,382 [phygnn.py:576] : Epoch 113 train loss: 2.72e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:39:29,755 [phygnn.py:576] : Epoch 114 train loss: 2.71e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:39:44,925 [phygnn.py:576] : Epoch 115 train loss: 2.72e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:39:59,716 [phygnn.py:576] : Epoch 116 train loss: 2.70e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:40:13,337 [phygnn.py:576] : Epoch 117 train loss: 2.72e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:40:26,380 [phygnn.py:576] : Epoch 118 train loss: 2.73e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:40:39,406 [phygnn.py:576] : Epoch 119 train loss: 2.71e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:40:53,413 [phygnn.py:576] : Epoch 120 train loss: 2.69e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:41:07,924 [phygnn.py:576] : Epoch 121 train loss: 2.74e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:41:21,778 [phygnn.py:576] : Epoch 122 train loss: 2.72e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:41:35,933 [phygnn.py:576] : Epoch 123 train loss: 2.73e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:41:50,167 [phygnn.py:576] : Epoch 124 train loss: 2.71e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:42:03,698 [phygnn.py:576] : Epoch 125 train loss: 2.70e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:42:16,492 [phygnn.py:576] : Epoch 126 train loss: 2.72e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:42:30,821 [phygnn.py:576] : Epoch 127 train loss: 2.71e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:42:44,427 [phygnn.py:576] : Epoch 128 train loss: 2.66e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:42:59,335 [phygnn.py:576] : Epoch 129 train loss: 2.66e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:43:13,673 [phygnn.py:576] : Epoch 130 train loss: 2.71e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:43:27,973 [phygnn.py:576] : Epoch 131 train loss: 2.70e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:43:41,704 [phygnn.py:576] : Epoch 132 train loss: 2.67e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:43:55,743 [phygnn.py:576] : Epoch 133 train loss: 2.67e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:44:09,935 [phygnn.py:576] : Epoch 134 train loss: 2.69e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:44:23,379 [phygnn.py:576] : Epoch 135 train loss: 2.70e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:44:37,249 [phygnn.py:576] : Epoch 136 train loss: 2.70e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:44:52,041 [phygnn.py:576] : Epoch 137 train loss: 2.68e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:45:06,142 [phygnn.py:576] : Epoch 138 train loss: 2.71e-01 val loss: 2.64e-01 for "phygnn"
INFO - 2022-03-01 13:45:20,516 [phygnn.py:576] : Epoch 139 train loss: 2.70e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:45:34,523 [phygnn.py:576] : Epoch 140 train loss: 2.68e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:45:49,085 [phygnn.py:576] : Epoch 141 train loss: 2.69e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:46:03,371 [phygnn.py:576] : Epoch 142 train loss: 2.74e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:46:17,447 [phygnn.py:576] : Epoch 143 train loss: 2.70e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:46:31,639 [phygnn.py:576] : Epoch 144 train loss: 2.69e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:46:44,928 [phygnn.py:576] : Epoch 145 train loss: 2.68e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:46:58,509 [phygnn.py:576] : Epoch 146 train loss: 2.65e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:47:13,108 [phygnn.py:576] : Epoch 147 train loss: 2.71e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:47:28,206 [phygnn.py:576] : Epoch 148 train loss: 2.70e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:47:42,727 [phygnn.py:576] : Epoch 149 train loss: 2.69e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:47:56,271 [phygnn.py:576] : Epoch 150 train loss: 2.69e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:48:09,585 [phygnn.py:576] : Epoch 151 train loss: 2.70e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:48:22,494 [phygnn.py:576] : Epoch 152 train loss: 2.67e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:48:37,080 [phygnn.py:576] : Epoch 153 train loss: 2.68e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:48:51,343 [phygnn.py:576] : Epoch 154 train loss: 2.65e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:49:05,382 [phygnn.py:576] : Epoch 155 train loss: 2.69e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:49:19,726 [phygnn.py:576] : Epoch 156 train loss: 2.70e-01 val loss: 2.63e-01 for "phygnn"
INFO - 2022-03-01 13:49:32,786 [phygnn.py:576] : Epoch 157 train loss: 2.68e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:49:46,451 [phygnn.py:576] : Epoch 158 train loss: 2.67e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:50:01,328 [phygnn.py:576] : Epoch 159 train loss: 2.65e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:50:14,968 [phygnn.py:576] : Epoch 160 train loss: 2.70e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:50:29,083 [phygnn.py:576] : Epoch 161 train loss: 2.68e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:50:43,020 [phygnn.py:576] : Epoch 162 train loss: 2.68e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:50:56,758 [phygnn.py:576] : Epoch 163 train loss: 2.66e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:51:10,796 [phygnn.py:576] : Epoch 164 train loss: 2.65e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:51:23,492 [phygnn.py:576] : Epoch 165 train loss: 2.71e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:51:36,562 [phygnn.py:576] : Epoch 166 train loss: 2.67e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:51:49,142 [phygnn.py:576] : Epoch 167 train loss: 2.69e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:52:02,374 [phygnn.py:576] : Epoch 168 train loss: 2.67e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:52:14,933 [phygnn.py:576] : Epoch 169 train loss: 2.68e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:52:27,629 [phygnn.py:576] : Epoch 170 train loss: 2.69e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:52:40,357 [phygnn.py:576] : Epoch 171 train loss: 2.65e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:52:54,570 [phygnn.py:576] : Epoch 172 train loss: 2.64e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:53:08,127 [phygnn.py:576] : Epoch 173 train loss: 2.66e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:53:21,495 [phygnn.py:576] : Epoch 174 train loss: 2.65e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:53:34,357 [phygnn.py:576] : Epoch 175 train loss: 2.68e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:53:47,510 [phygnn.py:576] : Epoch 176 train loss: 2.68e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:54:00,741 [phygnn.py:576] : Epoch 177 train loss: 2.68e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:54:13,780 [phygnn.py:576] : Epoch 178 train loss: 2.68e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:54:26,432 [phygnn.py:576] : Epoch 179 train loss: 2.67e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:54:38,771 [phygnn.py:576] : Epoch 180 train loss: 2.65e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:54:51,823 [phygnn.py:576] : Epoch 181 train loss: 2.67e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:55:04,822 [phygnn.py:576] : Epoch 182 train loss: 2.68e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:55:17,895 [phygnn.py:576] : Epoch 183 train loss: 2.68e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:55:32,134 [phygnn.py:576] : Epoch 184 train loss: 2.67e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:55:44,904 [phygnn.py:576] : Epoch 185 train loss: 2.65e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:55:57,413 [phygnn.py:576] : Epoch 186 train loss: 2.67e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:56:10,638 [phygnn.py:576] : Epoch 187 train loss: 2.64e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:56:23,343 [phygnn.py:576] : Epoch 188 train loss: 2.67e-01 val loss: 2.62e-01 for "phygnn"
INFO - 2022-03-01 13:56:37,133 [phygnn.py:576] : Epoch 189 train loss: 2.67e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:56:50,220 [phygnn.py:576] : Epoch 190 train loss: 2.65e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:57:03,244 [phygnn.py:576] : Epoch 191 train loss: 2.65e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:57:15,410 [phygnn.py:576] : Epoch 192 train loss: 2.66e-01 val loss: 2.60e-01 for "phygnn"
INFO - 2022-03-01 13:57:27,788 [phygnn.py:576] : Epoch 193 train loss: 2.63e-01 val loss: 2.60e-01 for "phygnn"
INFO - 2022-03-01 13:57:41,079 [phygnn.py:576] : Epoch 194 train loss: 2.64e-01 val loss: 2.60e-01 for "phygnn"
INFO - 2022-03-01 13:57:53,819 [phygnn.py:576] : Epoch 195 train loss: 2.64e-01 val loss: 2.60e-01 for "phygnn"
INFO - 2022-03-01 13:58:07,720 [phygnn.py:576] : Epoch 196 train loss: 2.66e-01 val loss: 2.61e-01 for "phygnn"
INFO - 2022-03-01 13:58:21,737 [phygnn.py:576] : Epoch 197 train loss: 2.66e-01 val loss: 2.60e-01 for "phygnn"
INFO - 2022-03-01 13:58:34,417 [phygnn.py:576] : Epoch 198 train loss: 2.65e-01 val loss: 2.60e-01 for "phygnn"
INFO - 2022-03-01 13:58:47,460 [phygnn.py:576] : Epoch 199 train loss: 2.67e-01 val loss: 2.60e-01 for "phygnn"
INFO - 2022-03-01 13:58:48,276 [trainer.py:102] : Training complete
INFO - 2022-03-01 13:58:48,315 [base.py:496] : Saved model to: /home/gbuster/code/mlclouds/mlclouds/model/k_fold/outputs/model_5.pkl
DEBUG - 2022-03-01 13:58:48,316 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 13:58:48,316 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:58:48,320 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:49,524 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:58:49,524 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:50,725 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:58:50,725 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:51,853 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:58:51,854 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:52,997 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:58:52,997 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:58:59,847 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 13:58:59,847 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:01,208 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:59:01,208 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:08,227 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 13:59:08,227 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:11,850 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 13:59:11,851 [data_handlers.py:413] : Shape df_raw=(3154464, 19), df_all_sky=(3154464, 14)
DEBUG - 2022-03-01 13:59:11,851 [data_handlers.py:420] : Shape after reset_index: df_raw=(3154464, 19), df_all_sky=(3154464, 14)
INFO - 2022-03-01 13:59:12,220 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:59:12,226 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 13:59:12,231 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:59:12,236 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:59:12,241 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:59:12,241 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:59:12,244 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,250 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,254 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,260 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,264 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.45% NaN values
DEBUG - 2022-03-01 13:59:12,268 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.45% NaN values
DEBUG - 2022-03-01 13:59:12,272 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.28% NaN values
DEBUG - 2022-03-01 13:59:12,277 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 13:59:12,281 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 13:59:12,285 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.27% NaN values
DEBUG - 2022-03-01 13:59:12,289 [data_cleaners.py:50] : 	"cloud_probability" has 3.27% NaN values
DEBUG - 2022-03-01 13:59:12,293 [data_cleaners.py:50] : 	"cloud_fraction" has 3.27% NaN values
DEBUG - 2022-03-01 13:59:12,298 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,302 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,306 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,310 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,314 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:12,319 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 13:59:12,323 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 13:59:12,323 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:59:16,195 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 13:59:16,598 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 13:59:16,598 [data_cleaners.py:107] : Cleaning took 4.7 seconds
INFO - 2022-03-01 13:59:16,957 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:59:16,963 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 13:59:16,968 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:59:16,973 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:59:16,978 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:59:16,978 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:59:16,981 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:16,987 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:16,991 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:16,996 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,000 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,005 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,010 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 13:59:17,014 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 13:59:17,018 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,022 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,027 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,031 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,034 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,038 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:17,038 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:59:19,246 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 13:59:19,644 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 13:59:19,644 [data_cleaners.py:107] : Cleaning took 3.0 seconds
DEBUG - 2022-03-01 13:59:19,646 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 13:59:19,679 [data_handlers.py:463] : Mask: shape=(3154464,), sum=1567353
DEBUG - 2022-03-01 13:59:19,834 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 13:59:19,834 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 13:59:23,664 [validator.py:110] : Predicted data shape is (1567353, 2)
DEBUG - 2022-03-01 13:59:24,143 [validator.py:158] : shapes: df_feature_val=(3154464, 20), df_all_sky_val=(3154464, 15)
INFO - 2022-03-01 13:59:24,462 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 13:59:24,466 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:24,466 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 13:59:24,466 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:24,516 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 13:59:25,208 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:25,246 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 13:59:25,902 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:25,940 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 13:59:26,646 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:26,683 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 13:59:27,343 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:27,380 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 13:59:28,035 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:28,072 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 13:59:28,726 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:28,764 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 13:59:29,415 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:29,453 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 13:59:30,104 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:30,141 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 13:59:30,812 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 13:59:30,812 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:30,861 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 13:59:31,508 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:31,546 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 13:59:32,190 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:32,228 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 13:59:32,874 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:32,910 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 13:59:33,559 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:33,594 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 13:59:34,241 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:34,277 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 13:59:34,917 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:34,955 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 13:59:35,594 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:35,630 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 13:59:36,267 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:36,302 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 13:59:36,944 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 13:59:36,944 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:36,992 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 13:59:37,664 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:37,700 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 13:59:38,353 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:38,389 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 13:59:39,048 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:39,084 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 13:59:39,735 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:39,771 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 13:59:40,430 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:40,465 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 13:59:41,130 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:41,165 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 13:59:41,819 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:41,854 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 13:59:42,514 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:42,550 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 13:59:43,214 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 13:59:43,214 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:43,262 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 13:59:43,913 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:43,948 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 13:59:44,588 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:44,623 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 13:59:45,264 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:45,299 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 13:59:45,939 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:45,975 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 13:59:46,623 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:46,658 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 13:59:47,307 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:47,343 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 13:59:47,982 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:48,019 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 13:59:48,673 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:48,709 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 13:59:49,352 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 13:59:49,353 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:49,615 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 13:59:50,287 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:50,401 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 13:59:51,118 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:51,230 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 13:59:51,903 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:52,016 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 13:59:52,741 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:52,853 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 13:59:53,582 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 13:59:53,693 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 13:59:54,420 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 13:59:54,534 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 13:59:55,251 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 13:59:55,365 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 13:59:56,080 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 13:59:56,193 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 13:59:56,886 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 13:59:56,886 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 13:59:56,934 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 13:59:57,587 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 13:59:57,623 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 13:59:58,280 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 13:59:58,316 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 13:59:58,973 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 13:59:59,008 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 13:59:59,663 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 13:59:59,699 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:00:00,353 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:00,388 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:00:01,046 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:01,081 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:00:01,738 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:01,773 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:00:02,436 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:02,471 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:00:03,129 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:00:03,129 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:03,308 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:00:03,994 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:04,107 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:00:04,792 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:04,904 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:00:05,593 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:05,706 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:00:06,394 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:06,507 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:00:07,203 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:07,315 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:00:08,091 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:08,203 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:00:08,935 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:09,047 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:00:09,771 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:09,883 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:00:10,606 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:00:10,606 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:10,702 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:00:11,390 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:11,458 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:00:12,150 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:12,216 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:00:12,914 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:12,984 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:00:13,683 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:13,750 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:00:14,445 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:14,512 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:00:15,205 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:15,272 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:00:15,966 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:16,033 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:00:16,748 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:16,815 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:00:17,523 [validator.py:187] : Shapes: df_base_full=(3154464, 6), df_surf_full=(3154464, 4)
DEBUG - 2022-03-01 14:00:17,528 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:00:17,565 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:00:33,277 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:00:33,314 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:00:49,121 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:00:49,159 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:04,705 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:01:04,742 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:20,383 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:01:20,420 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:36,126 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:01:36,164 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:51,917 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:01:51,955 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:02:07,662 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:02:07,700 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:02:23,427 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:02:23,464 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
INFO - 2022-03-01 14:02:39,115 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:02:39,125 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:02:39,126 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:02:39,128 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:40,151 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:02:40,151 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:41,189 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:02:41,189 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:47,800 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:02:47,801 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:54,548 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:02:54,548 [data_handlers.py:413] : Shape df_raw=(2207952, 19), df_all_sky=(2207952, 14)
DEBUG - 2022-03-01 14:02:54,548 [data_handlers.py:420] : Shape after reset_index: df_raw=(2207952, 19), df_all_sky=(2207952, 14)
INFO - 2022-03-01 14:02:54,789 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:02:54,793 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:02:54,796 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:02:54,800 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:02:54,804 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:02:54,804 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:02:54,806 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,810 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,813 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,817 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,820 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 49.99% NaN values
DEBUG - 2022-03-01 14:02:54,823 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 49.99% NaN values
DEBUG - 2022-03-01 14:02:54,827 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 0.35% NaN values
DEBUG - 2022-03-01 14:02:54,830 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:02:54,833 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:02:54,836 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 0.34% NaN values
DEBUG - 2022-03-01 14:02:54,839 [data_cleaners.py:50] : 	"cloud_probability" has 0.34% NaN values
DEBUG - 2022-03-01 14:02:54,842 [data_cleaners.py:50] : 	"cloud_fraction" has 0.34% NaN values
DEBUG - 2022-03-01 14:02:54,845 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,848 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,851 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,854 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,857 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:54,860 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:02:54,863 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:02:54,864 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:02:57,386 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:02:57,660 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:02:57,660 [data_cleaners.py:107] : Cleaning took 3.1 seconds
INFO - 2022-03-01 14:02:57,889 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:02:57,893 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:02:57,896 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:02:57,900 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:02:57,904 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:02:57,904 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:02:57,906 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,910 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,913 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,917 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,920 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,924 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,927 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:02:57,930 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:02:57,933 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,936 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,939 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,942 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,944 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,947 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:57,947 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:02:59,385 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:02:59,658 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:02:59,658 [data_cleaners.py:107] : Cleaning took 2.0 seconds
DEBUG - 2022-03-01 14:02:59,660 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:02:59,681 [data_handlers.py:463] : Mask: shape=(2207952,), sum=1097157
DEBUG - 2022-03-01 14:02:59,774 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:02:59,774 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:03:02,391 [validator.py:110] : Predicted data shape is (1097157, 2)
DEBUG - 2022-03-01 14:03:02,712 [validator.py:158] : shapes: df_feature_val=(2207952, 20), df_all_sky_val=(2207952, 15)
INFO - 2022-03-01 14:03:02,920 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:03:02,923 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:03:02,923 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:03:02,923 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:02,958 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:03:03,599 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:03,637 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:03:04,273 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:04,311 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:03:04,949 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:04,986 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:03:05,632 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:05,669 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:03:06,306 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:06,344 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:03:06,984 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:07,021 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:03:07,658 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:07,696 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:03:08,335 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:08,372 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:03:09,011 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:03:09,011 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:09,049 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:03:09,694 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:09,731 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:03:10,367 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:10,404 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:03:11,041 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:11,076 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:03:11,713 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:11,750 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:03:12,386 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:12,421 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:03:13,064 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:13,099 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:03:13,736 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:13,771 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:03:14,415 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:14,450 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:03:15,088 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:03:15,088 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:15,199 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:03:15,848 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:15,963 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:03:16,615 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:16,727 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:03:17,381 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:17,494 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:03:18,149 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:18,263 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:03:18,920 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:19,033 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:03:19,689 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:19,803 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:03:20,463 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:20,576 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:03:21,242 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:21,359 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:03:22,048 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:03:22,048 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:22,164 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:03:22,850 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:22,966 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:03:23,656 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:23,772 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:03:24,464 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:24,581 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:03:25,270 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:25,387 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:03:26,078 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:26,194 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:03:26,885 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:27,002 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:03:27,708 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:27,825 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:03:28,522 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:28,639 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:03:29,363 [validator.py:187] : Shapes: df_base_full=(2207952, 6), df_surf_full=(2207952, 4)
DEBUG - 2022-03-01 14:03:29,367 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:03:29,397 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:03:40,841 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:03:40,870 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:03:52,207 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:03:52,234 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:03,487 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:04:03,514 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:14,823 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:04:14,850 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:26,215 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:04:26,241 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:37,623 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:04:37,651 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:49,003 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:04:49,030 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:05:00,398 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:05:00,425 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
INFO - 2022-03-01 14:05:11,736 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:05:11,759 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:05:11,759 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:05:11,763 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:12,880 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:05:12,881 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:14,028 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:05:14,028 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:15,169 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:05:15,169 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:18,360 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 14:05:18,361 [data_handlers.py:413] : Shape df_raw=(946512, 19), df_all_sky=(946512, 14)
DEBUG - 2022-03-01 14:05:18,361 [data_handlers.py:420] : Shape after reset_index: df_raw=(946512, 19), df_all_sky=(946512, 14)
INFO - 2022-03-01 14:05:18,465 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:05:18,466 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:05:18,468 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:05:18,470 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:05:18,471 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:05:18,472 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:05:18,473 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,475 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,476 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,478 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,480 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 54.84% NaN values
DEBUG - 2022-03-01 14:05:18,481 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 54.84% NaN values
DEBUG - 2022-03-01 14:05:18,483 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 10.12% NaN values
DEBUG - 2022-03-01 14:05:18,484 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:05:18,486 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:05:18,487 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 10.10% NaN values
DEBUG - 2022-03-01 14:05:18,489 [data_cleaners.py:50] : 	"cloud_probability" has 10.10% NaN values
DEBUG - 2022-03-01 14:05:18,491 [data_cleaners.py:50] : 	"cloud_fraction" has 10.10% NaN values
DEBUG - 2022-03-01 14:05:18,492 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,494 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,495 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,497 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,498 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:18,500 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:05:18,501 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:05:18,501 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:05:19,523 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:05:19,646 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:05:19,647 [data_cleaners.py:107] : Cleaning took 1.3 seconds
INFO - 2022-03-01 14:05:19,746 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:05:19,748 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:05:19,750 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:05:19,751 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:05:19,753 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:05:19,753 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:05:19,754 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,756 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,758 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,759 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,761 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,763 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,764 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:05:19,766 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:05:19,767 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,769 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,771 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,772 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,773 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,775 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:19,775 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:05:20,324 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:05:20,447 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:05:20,448 [data_cleaners.py:107] : Cleaning took 0.8 seconds
DEBUG - 2022-03-01 14:05:20,448 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:05:20,461 [data_handlers.py:463] : Mask: shape=(946512,), sum=470196
DEBUG - 2022-03-01 14:05:20,496 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:05:20,496 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:05:21,617 [validator.py:110] : Predicted data shape is (470196, 2)
DEBUG - 2022-03-01 14:05:21,727 [validator.py:158] : shapes: df_feature_val=(946512, 20), df_all_sky_val=(946512, 15)
INFO - 2022-03-01 14:05:21,815 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:05:21,818 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:21,818 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:05:21,819 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:21,867 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:05:22,507 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:22,545 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:05:23,185 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:23,222 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:05:23,859 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:23,897 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:05:24,541 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:24,579 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:05:25,216 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:25,251 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:05:25,890 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:25,928 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:05:26,562 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:26,600 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:05:27,235 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:27,273 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:05:27,911 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:05:27,911 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:27,959 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:05:28,606 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:28,644 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:05:29,275 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:29,310 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:05:29,942 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:29,980 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:05:30,611 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:30,646 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:05:31,277 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:31,312 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:05:31,959 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:31,996 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:05:32,628 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:32,663 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:05:33,304 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:33,340 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:05:33,974 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:05:33,974 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:34,021 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:05:34,656 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:34,691 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:05:35,336 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:35,371 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:05:36,008 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:36,043 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:05:36,687 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:36,722 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:05:37,358 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:37,395 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:05:38,034 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:38,069 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:05:38,711 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:38,746 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:05:39,392 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:39,427 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:05:40,067 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:05:40,067 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:40,162 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:05:40,811 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:40,878 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:05:41,526 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:41,593 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:05:42,243 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:42,309 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:05:42,957 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:43,024 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:05:43,669 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:43,736 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:05:44,383 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:44,450 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:05:45,107 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:45,173 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:05:45,832 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:45,899 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:05:46,555 [validator.py:187] : Shapes: df_base_full=(946512, 6), df_surf_full=(946512, 4)
DEBUG - 2022-03-01 14:05:46,559 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:05:46,571 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:52,068 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:05:52,080 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:05:57,546 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:05:57,557 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:03,004 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:06:03,016 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:08,476 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:06:08,488 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:13,971 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:06:13,983 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:19,476 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:06:19,488 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:24,963 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:06:24,974 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:30,454 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:06:30,466 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
INFO - 2022-03-01 14:06:35,925 [validator.py:292] : Finished computing stats.
