Starting scenario 4, validation against site 0
2022-03-01 13:17:48.710731: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:17:48.710761: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Cross validation site: 0
Training sites: [1, 2, 3, 4, 5, 6, 7, 8]
Number of files: 8
Number of east files: 4
Number of west files: 4
Source files: ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
Full config: {'clean_training_data_kwargs': {'filter_clear': False, 'nan_option': 'interp'}, 'epochs_a': 100, 'epochs_b': 100, 'features': ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo'], 'hidden_layers': [{'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}], 'learning_rate': 0.0005, 'loss_weights_a': [1, 0], 'loss_weights_b': [0.5, 0.5], 'metric': 'relative_mae', 'n_batch': 32, 'one_hot_categories': {'flag': ['clear', 'ice_cloud', 'water_cloud', 'bad_cloud']}, 'p_fun': 'p_fun_all_sky', 'p_kwargs': {'loss_terms': ['mae_ghi']}, 'phygnn_seed': 0, 'surfrad_window_minutes': 15, 'y_labels': ['cld_opd_dcomp', 'cld_reff_dcomp']}
INFO - 2022-03-01 13:17:55,686 [trainer.py:40] : Trainer: Training on sites [1, 2, 3, 4, 5, 6, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:17:55,686 [trainer.py:49] : Trainer: Training on sites [1, 2, 3, 4, 5, 6, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:17:55,686 [data_handlers.py:60] : Loading training data
DEBUG - 2022-03-01 13:17:55,686 [data_handlers.py:78] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:17:55,686 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5
DEBUG - 2022-03-01 13:17:56,831 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:17:56,835 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:17:56,835 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:17:56,840 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:17:57,526 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:57,530 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:17:58,199 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:58,203 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:17:58,884 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:58,888 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:17:59,560 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:17:59,563 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:18:00,396 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:00,400 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:18:01,069 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:01,072 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:18:01,749 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:01,752 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:18:02,446 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:02,446 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5
DEBUG - 2022-03-01 13:18:03,491 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:18:03,495 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:03,495 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:03,499 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:18:04,154 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:04,157 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:18:04,805 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:04,808 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:18:05,464 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:05,467 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:18:06,114 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:06,117 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:18:06,779 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:06,783 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:18:07,446 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:07,450 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:18:08,126 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:08,129 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:18:08,797 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:18:08,797 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5
DEBUG - 2022-03-01 13:18:09,945 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:09,950 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:09,950 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:09,954 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:18:10,633 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:10,636 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:18:11,321 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:11,324 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:18:12,005 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:12,009 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:18:12,687 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:12,690 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:18:13,382 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:13,386 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:18:14,064 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:14,067 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:18:14,748 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:14,751 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:18:15,435 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:15,436 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5
DEBUG - 2022-03-01 13:18:16,586 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:16,592 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:16,592 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:16,596 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:18:17,250 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:17,254 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:18:17,910 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:17,914 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:18:18,557 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:18,560 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:18:19,211 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:19,215 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:18:19,881 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:19,884 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:18:20,535 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:20,539 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:18:21,205 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:21,208 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:18:21,863 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:21,863 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5
DEBUG - 2022-03-01 13:18:28,503 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:18:28,526 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:18:28,526 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:28,529 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:18:29,217 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:29,220 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:18:29,900 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:29,904 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:18:30,596 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:30,599 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:18:31,302 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:31,305 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:18:31,996 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:31,999 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:18:32,688 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:32,692 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:18:33,391 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:33,395 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:18:34,104 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:34,104 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5
DEBUG - 2022-03-01 13:18:35,393 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:18:35,398 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:18:35,398 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:35,401 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:18:36,064 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:36,068 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:18:36,734 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:36,738 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:18:37,412 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:37,416 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:18:38,096 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:38,100 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:18:38,774 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:38,777 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:18:39,458 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:39,462 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:18:40,140 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:40,143 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:18:40,825 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:18:40,825 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5
DEBUG - 2022-03-01 13:18:47,378 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:18:47,399 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:18:47,399 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:47,403 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:18:48,089 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:48,093 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:18:48,791 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:48,795 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:18:49,487 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:49,491 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:18:50,192 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:50,195 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:18:50,885 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:50,889 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:18:51,650 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:51,653 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:18:52,363 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:52,367 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:18:53,076 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:18:53,076 [data_handlers.py:85] : Loading data for site(s) [1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5
DEBUG - 2022-03-01 13:18:56,469 [data_handlers.py:103] : 	Shape temp_raw=(420480, 19), temp_all_sky=(420480, 14)
DEBUG - 2022-03-01 13:18:56,480 [data_handlers.py:106] : 	Time step is 10 minutes
DEBUG - 2022-03-01 13:18:56,480 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:18:56,484 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:18:57,163 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:57,167 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:18:57,832 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:57,836 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:18:58,508 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:58,512 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:18:59,184 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:59,187 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:18:59,853 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:18:59,857 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:19:00,542 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:00,546 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:19:01,212 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:01,215 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:19:01,902 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:19:01,902 [data_handlers.py:136] : Data load complete. Shape df_raw=(2803968, 19), df_all_sky=(2803968, 14), df_surf=(2803968, 5)
DEBUG - 2022-03-01 13:19:02,753 [data_handlers.py:159] : Extracting 2D arrays to run rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:19:14,252 [data_handlers.py:176] : Running rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:21:11,344 [data_handlers.py:194] : Completed rest2 run for clearsky PhyGNN inputs.
INFO - 2022-03-01 13:21:13,041 [data_handlers.py:62] : Prepping training data
DEBUG - 2022-03-01 13:21:13,042 [data_handlers.py:214] : Training data clean kwargs: {'filter_daylight': True, 'filter_clear': False, 'add_cloud_flag': True, 'sza_lim': 89, 'nan_option': 'interp'}
DEBUG - 2022-03-01 13:21:13,042 [data_handlers.py:215] : Shape before cleaning: df_raw=(2803968, 19)
INFO - 2022-03-01 13:21:13,340 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:21:13,345 [data_cleaners.py:38] : 51.71% of daylight timesteps are cloudy
INFO - 2022-03-01 13:21:13,349 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:21:13,354 [data_cleaners.py:42] : 34.82% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:21:13,359 [data_cleaners.py:44] : 35.05% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:21:13,359 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:21:13,362 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,368 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,372 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,378 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,382 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.62% NaN values
DEBUG - 2022-03-01 13:21:13,387 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.62% NaN values
DEBUG - 2022-03-01 13:21:13,391 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.63% NaN values
DEBUG - 2022-03-01 13:21:13,395 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:21:13,399 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:21:13,403 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.61% NaN values
DEBUG - 2022-03-01 13:21:13,408 [data_cleaners.py:50] : 	"cloud_probability" has 3.61% NaN values
DEBUG - 2022-03-01 13:21:13,412 [data_cleaners.py:50] : 	"cloud_fraction" has 3.61% NaN values
DEBUG - 2022-03-01 13:21:13,416 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,420 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,424 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,429 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,433 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:13,437 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 83.25% NaN values
DEBUG - 2022-03-01 13:21:13,441 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.31% NaN values
DEBUG - 2022-03-01 13:21:13,441 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:21:16,805 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:21:17,131 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393162 after filters (49.69% of original)
DEBUG - 2022-03-01 13:21:17,264 [data_cleaners.py:105] : Feature flag column has these values: ['bad_cloud' 'clear' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:21:17,264 [data_cleaners.py:107] : Cleaning took 4.2 seconds
DEBUG - 2022-03-01 13:21:17,264 [data_handlers.py:218] : Shape after cleaning: df_train=(1393162, 20)
DEBUG - 2022-03-01 13:21:17,264 [data_handlers.py:221] : Cleaning df_all_sky training data (for pfun).
DEBUG - 2022-03-01 13:21:17,264 [data_handlers.py:222] : Shape before cleaning: df_all_sky=(2803968, 25)
INFO - 2022-03-01 13:21:17,649 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:21:17,655 [data_cleaners.py:38] : 51.71% of daylight timesteps are cloudy
INFO - 2022-03-01 13:21:17,660 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:21:17,664 [data_cleaners.py:42] : 34.82% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:21:17,669 [data_cleaners.py:44] : 35.05% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:21:17,669 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:21:17,672 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,676 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,681 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,685 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,691 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,695 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 83.25% NaN values
DEBUG - 2022-03-01 13:21:17,699 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.31% NaN values
DEBUG - 2022-03-01 13:21:17,703 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,707 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,711 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,715 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,718 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,722 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,727 [data_cleaners.py:50] : 	"surfrad_dhi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,733 [data_cleaners.py:50] : 	"surfrad_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,738 [data_cleaners.py:50] : 	"surfrad_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,740 [data_cleaners.py:50] : 	"doy" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,746 [data_cleaners.py:50] : 	"radius" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,751 [data_cleaners.py:50] : 	"Tuuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,756 [data_cleaners.py:50] : 	"clearsky_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,761 [data_cleaners.py:50] : 	"clearsky_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,766 [data_cleaners.py:50] : 	"Ruuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,772 [data_cleaners.py:50] : 	"Tddclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,777 [data_cleaners.py:50] : 	"Tduclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,783 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:21:17,783 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:21:20,412 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:21:20,731 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393162 after filters (49.69% of original)
DEBUG - 2022-03-01 13:21:20,897 [data_cleaners.py:105] : Feature flag column has these values: ['bad_cloud' 'clear' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:21:20,897 [data_cleaners.py:107] : Cleaning took 3.6 seconds
DEBUG - 2022-03-01 13:21:20,899 [data_handlers.py:226] : Shape after cleaning: df_all_sky=(1393162, 26)
DEBUG - 2022-03-01 13:21:21,006 [data_handlers.py:240] : **Shape: df_train=(1393162, 17)
DEBUG - 2022-03-01 13:21:21,039 [data_handlers.py:250] : Shapes: x=(1393162, 15), y=(1393162, 2), p=(1393162, 26)
DEBUG - 2022-03-01 13:21:21,039 [data_handlers.py:253] : Training features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
DEBUG - 2022-03-01 13:21:21,039 [trainer.py:67] : Building PHYGNN model
INFO - 2022-03-01 13:21:21,039 [trainer.py:70] : Using p_fun: <function p_fun_all_sky at 0x2acd59fda820>
INFO - 2022-03-01 13:21:21,040 [base.py:152] : Active python environment versions: 
{   'numpy': '1.22.2',
    'pandas': '1.2.4',
    'phygnn': '0.0.14',
    'python': '3.8.8 (default, Feb 24 2021, 21:46:12) \n[GCC 7.3.0]',
    'sklearn': '0.24.1',
    'tensorflow': '2.8.0'}
INFO - 2022-03-01 13:21:21,057 [base.py:111] : Successfully initialized model with 17 layers
INFO - 2022-03-01 13:21:21,057 [trainer.py:84] : Training part A - pure data. Loss is [1, 0]
2022-03-01 13:21:31.099080: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.100444: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.101302: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.102165: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.103031: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.103805: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.104548: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.105268: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:21:31.105289: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-03-01 13:21:31.105759: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO - 2022-03-01 13:21:39,511 [phygnn.py:576] : Epoch 0 train loss: 7.07e-01 val loss: 6.92e-01 for "phygnn"
INFO - 2022-03-01 13:21:48,086 [phygnn.py:576] : Epoch 1 train loss: 6.41e-01 val loss: 6.29e-01 for "phygnn"
INFO - 2022-03-01 13:21:56,429 [phygnn.py:576] : Epoch 2 train loss: 5.73e-01 val loss: 5.58e-01 for "phygnn"
INFO - 2022-03-01 13:22:05,195 [phygnn.py:576] : Epoch 3 train loss: 5.38e-01 val loss: 5.21e-01 for "phygnn"
INFO - 2022-03-01 13:22:13,686 [phygnn.py:576] : Epoch 4 train loss: 5.22e-01 val loss: 5.01e-01 for "phygnn"
INFO - 2022-03-01 13:22:21,976 [phygnn.py:576] : Epoch 5 train loss: 5.11e-01 val loss: 4.90e-01 for "phygnn"
INFO - 2022-03-01 13:22:30,534 [phygnn.py:576] : Epoch 6 train loss: 5.03e-01 val loss: 4.84e-01 for "phygnn"
INFO - 2022-03-01 13:22:39,148 [phygnn.py:576] : Epoch 7 train loss: 4.93e-01 val loss: 4.79e-01 for "phygnn"
INFO - 2022-03-01 13:22:47,434 [phygnn.py:576] : Epoch 8 train loss: 4.86e-01 val loss: 4.76e-01 for "phygnn"
INFO - 2022-03-01 13:22:55,923 [phygnn.py:576] : Epoch 9 train loss: 4.93e-01 val loss: 4.72e-01 for "phygnn"
INFO - 2022-03-01 13:23:04,343 [phygnn.py:576] : Epoch 10 train loss: 4.86e-01 val loss: 4.70e-01 for "phygnn"
INFO - 2022-03-01 13:23:13,050 [phygnn.py:576] : Epoch 11 train loss: 4.83e-01 val loss: 4.68e-01 for "phygnn"
INFO - 2022-03-01 13:23:21,697 [phygnn.py:576] : Epoch 12 train loss: 4.82e-01 val loss: 4.69e-01 for "phygnn"
INFO - 2022-03-01 13:23:30,507 [phygnn.py:576] : Epoch 13 train loss: 4.84e-01 val loss: 4.64e-01 for "phygnn"
INFO - 2022-03-01 13:23:39,391 [phygnn.py:576] : Epoch 14 train loss: 4.82e-01 val loss: 4.63e-01 for "phygnn"
INFO - 2022-03-01 13:23:48,040 [phygnn.py:576] : Epoch 15 train loss: 4.77e-01 val loss: 4.60e-01 for "phygnn"
INFO - 2022-03-01 13:23:56,929 [phygnn.py:576] : Epoch 16 train loss: 4.77e-01 val loss: 4.59e-01 for "phygnn"
INFO - 2022-03-01 13:24:05,691 [phygnn.py:576] : Epoch 17 train loss: 4.74e-01 val loss: 4.57e-01 for "phygnn"
INFO - 2022-03-01 13:24:14,466 [phygnn.py:576] : Epoch 18 train loss: 4.71e-01 val loss: 4.57e-01 for "phygnn"
INFO - 2022-03-01 13:24:23,109 [phygnn.py:576] : Epoch 19 train loss: 4.70e-01 val loss: 4.56e-01 for "phygnn"
INFO - 2022-03-01 13:24:31,879 [phygnn.py:576] : Epoch 20 train loss: 4.65e-01 val loss: 4.54e-01 for "phygnn"
INFO - 2022-03-01 13:24:40,811 [phygnn.py:576] : Epoch 21 train loss: 4.61e-01 val loss: 4.51e-01 for "phygnn"
INFO - 2022-03-01 13:24:49,656 [phygnn.py:576] : Epoch 22 train loss: 4.67e-01 val loss: 4.49e-01 for "phygnn"
INFO - 2022-03-01 13:24:58,418 [phygnn.py:576] : Epoch 23 train loss: 4.65e-01 val loss: 4.46e-01 for "phygnn"
INFO - 2022-03-01 13:25:07,071 [phygnn.py:576] : Epoch 24 train loss: 4.62e-01 val loss: 4.46e-01 for "phygnn"
INFO - 2022-03-01 13:25:15,843 [phygnn.py:576] : Epoch 25 train loss: 4.56e-01 val loss: 4.42e-01 for "phygnn"
INFO - 2022-03-01 13:25:24,628 [phygnn.py:576] : Epoch 26 train loss: 4.59e-01 val loss: 4.41e-01 for "phygnn"
INFO - 2022-03-01 13:25:33,524 [phygnn.py:576] : Epoch 27 train loss: 4.60e-01 val loss: 4.41e-01 for "phygnn"
INFO - 2022-03-01 13:25:42,264 [phygnn.py:576] : Epoch 28 train loss: 4.55e-01 val loss: 4.39e-01 for "phygnn"
INFO - 2022-03-01 13:25:50,967 [phygnn.py:576] : Epoch 29 train loss: 4.52e-01 val loss: 4.39e-01 for "phygnn"
INFO - 2022-03-01 13:25:59,497 [phygnn.py:576] : Epoch 30 train loss: 4.51e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2022-03-01 13:26:08,256 [phygnn.py:576] : Epoch 31 train loss: 4.55e-01 val loss: 4.36e-01 for "phygnn"
INFO - 2022-03-01 13:26:16,895 [phygnn.py:576] : Epoch 32 train loss: 4.50e-01 val loss: 4.34e-01 for "phygnn"
INFO - 2022-03-01 13:26:25,470 [phygnn.py:576] : Epoch 33 train loss: 4.55e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2022-03-01 13:26:34,228 [phygnn.py:576] : Epoch 34 train loss: 4.51e-01 val loss: 4.34e-01 for "phygnn"
INFO - 2022-03-01 13:26:43,001 [phygnn.py:576] : Epoch 35 train loss: 4.44e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:26:51,760 [phygnn.py:576] : Epoch 36 train loss: 4.46e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:27:00,550 [phygnn.py:576] : Epoch 37 train loss: 4.43e-01 val loss: 4.34e-01 for "phygnn"
INFO - 2022-03-01 13:27:09,123 [phygnn.py:576] : Epoch 38 train loss: 4.41e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:27:17,557 [phygnn.py:576] : Epoch 39 train loss: 4.42e-01 val loss: 4.29e-01 for "phygnn"
INFO - 2022-03-01 13:27:26,218 [phygnn.py:576] : Epoch 40 train loss: 4.51e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:27:34,887 [phygnn.py:576] : Epoch 41 train loss: 4.45e-01 val loss: 4.28e-01 for "phygnn"
INFO - 2022-03-01 13:27:43,414 [phygnn.py:576] : Epoch 42 train loss: 4.45e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:27:52,155 [phygnn.py:576] : Epoch 43 train loss: 4.39e-01 val loss: 4.28e-01 for "phygnn"
INFO - 2022-03-01 13:28:00,928 [phygnn.py:576] : Epoch 44 train loss: 4.40e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:28:09,569 [phygnn.py:576] : Epoch 45 train loss: 4.36e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:28:18,201 [phygnn.py:576] : Epoch 46 train loss: 4.41e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:28:26,915 [phygnn.py:576] : Epoch 47 train loss: 4.38e-01 val loss: 4.24e-01 for "phygnn"
INFO - 2022-03-01 13:28:35,383 [phygnn.py:576] : Epoch 48 train loss: 4.36e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:28:44,055 [phygnn.py:576] : Epoch 49 train loss: 4.42e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:28:52,871 [phygnn.py:576] : Epoch 50 train loss: 4.38e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:29:01,601 [phygnn.py:576] : Epoch 51 train loss: 4.42e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:29:10,473 [phygnn.py:576] : Epoch 52 train loss: 4.36e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:29:19,049 [phygnn.py:576] : Epoch 53 train loss: 4.38e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2022-03-01 13:29:27,869 [phygnn.py:576] : Epoch 54 train loss: 4.38e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:29:36,500 [phygnn.py:576] : Epoch 55 train loss: 4.34e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:29:45,383 [phygnn.py:576] : Epoch 56 train loss: 4.30e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:29:53,911 [phygnn.py:576] : Epoch 57 train loss: 4.35e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:30:02,672 [phygnn.py:576] : Epoch 58 train loss: 4.31e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:30:11,477 [phygnn.py:576] : Epoch 59 train loss: 4.31e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:30:20,172 [phygnn.py:576] : Epoch 60 train loss: 4.35e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:30:29,014 [phygnn.py:576] : Epoch 61 train loss: 4.33e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:30:37,666 [phygnn.py:576] : Epoch 62 train loss: 4.32e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:30:46,402 [phygnn.py:576] : Epoch 63 train loss: 4.35e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:30:54,937 [phygnn.py:576] : Epoch 64 train loss: 4.35e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:31:03,845 [phygnn.py:576] : Epoch 65 train loss: 4.25e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:31:12,672 [phygnn.py:576] : Epoch 66 train loss: 4.25e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:31:21,464 [phygnn.py:576] : Epoch 67 train loss: 4.28e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:31:30,178 [phygnn.py:576] : Epoch 68 train loss: 4.31e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:31:38,951 [phygnn.py:576] : Epoch 69 train loss: 4.33e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:31:47,817 [phygnn.py:576] : Epoch 70 train loss: 4.22e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:31:56,643 [phygnn.py:576] : Epoch 71 train loss: 4.26e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:32:05,402 [phygnn.py:576] : Epoch 72 train loss: 4.20e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:32:14,193 [phygnn.py:576] : Epoch 73 train loss: 4.28e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:32:22,985 [phygnn.py:576] : Epoch 74 train loss: 4.31e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:32:31,756 [phygnn.py:576] : Epoch 75 train loss: 4.25e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:32:40,554 [phygnn.py:576] : Epoch 76 train loss: 4.28e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:32:49,450 [phygnn.py:576] : Epoch 77 train loss: 4.23e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:32:57,897 [phygnn.py:576] : Epoch 78 train loss: 4.18e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:33:06,604 [phygnn.py:576] : Epoch 79 train loss: 4.24e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:33:15,446 [phygnn.py:576] : Epoch 80 train loss: 4.21e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:33:24,120 [phygnn.py:576] : Epoch 81 train loss: 4.19e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:33:32,741 [phygnn.py:576] : Epoch 82 train loss: 4.18e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:33:41,195 [phygnn.py:576] : Epoch 83 train loss: 4.21e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:33:49,974 [phygnn.py:576] : Epoch 84 train loss: 4.15e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:33:58,786 [phygnn.py:576] : Epoch 85 train loss: 4.20e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:34:07,547 [phygnn.py:576] : Epoch 86 train loss: 4.19e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:16,361 [phygnn.py:576] : Epoch 87 train loss: 4.22e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:24,987 [phygnn.py:576] : Epoch 88 train loss: 4.20e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:34:33,847 [phygnn.py:576] : Epoch 89 train loss: 4.21e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:42,426 [phygnn.py:576] : Epoch 90 train loss: 4.25e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:51,356 [phygnn.py:576] : Epoch 91 train loss: 4.25e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:35:00,004 [phygnn.py:576] : Epoch 92 train loss: 4.20e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:35:08,580 [phygnn.py:576] : Epoch 93 train loss: 4.14e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:35:17,422 [phygnn.py:576] : Epoch 94 train loss: 4.18e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:35:26,083 [phygnn.py:576] : Epoch 95 train loss: 4.15e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:35:34,716 [phygnn.py:576] : Epoch 96 train loss: 4.17e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:35:43,232 [phygnn.py:576] : Epoch 97 train loss: 4.13e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:35:52,114 [phygnn.py:576] : Epoch 98 train loss: 4.25e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:36:00,738 [phygnn.py:576] : Epoch 99 train loss: 4.19e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2022-03-01 13:36:01,561 [trainer.py:92] : Training part B - data and phygnn. Loss is [0.5, 0.5]
INFO - 2022-03-01 13:36:26,455 [phygnn.py:576] : Epoch 100 train loss: 2.87e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:36:40,841 [phygnn.py:576] : Epoch 101 train loss: 2.86e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:36:55,140 [phygnn.py:576] : Epoch 102 train loss: 2.86e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:37:09,187 [phygnn.py:576] : Epoch 103 train loss: 2.86e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:37:22,609 [phygnn.py:576] : Epoch 104 train loss: 2.83e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:37:36,658 [phygnn.py:576] : Epoch 105 train loss: 2.88e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:37:50,149 [phygnn.py:576] : Epoch 106 train loss: 2.85e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:38:03,732 [phygnn.py:576] : Epoch 107 train loss: 2.88e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:38:17,726 [phygnn.py:576] : Epoch 108 train loss: 2.89e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:38:31,469 [phygnn.py:576] : Epoch 109 train loss: 2.89e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:38:45,856 [phygnn.py:576] : Epoch 110 train loss: 2.84e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:38:59,865 [phygnn.py:576] : Epoch 111 train loss: 2.88e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:39:13,959 [phygnn.py:576] : Epoch 112 train loss: 2.85e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:39:27,513 [phygnn.py:576] : Epoch 113 train loss: 2.84e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:39:41,757 [phygnn.py:576] : Epoch 114 train loss: 2.84e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:39:55,870 [phygnn.py:576] : Epoch 115 train loss: 2.87e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:40:10,253 [phygnn.py:576] : Epoch 116 train loss: 2.87e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:40:23,818 [phygnn.py:576] : Epoch 117 train loss: 2.86e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:40:38,456 [phygnn.py:576] : Epoch 118 train loss: 2.81e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:40:52,906 [phygnn.py:576] : Epoch 119 train loss: 2.85e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:41:06,426 [phygnn.py:576] : Epoch 120 train loss: 2.85e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:41:20,105 [phygnn.py:576] : Epoch 121 train loss: 2.85e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:41:34,161 [phygnn.py:576] : Epoch 122 train loss: 2.84e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:41:48,534 [phygnn.py:576] : Epoch 123 train loss: 2.88e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:42:02,269 [phygnn.py:576] : Epoch 124 train loss: 2.88e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:42:15,993 [phygnn.py:576] : Epoch 125 train loss: 2.87e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:42:30,284 [phygnn.py:576] : Epoch 126 train loss: 2.86e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:42:44,278 [phygnn.py:576] : Epoch 127 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:42:58,256 [phygnn.py:576] : Epoch 128 train loss: 2.82e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:43:12,358 [phygnn.py:576] : Epoch 129 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:43:26,246 [phygnn.py:576] : Epoch 130 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:43:40,850 [phygnn.py:576] : Epoch 131 train loss: 2.82e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:43:55,580 [phygnn.py:576] : Epoch 132 train loss: 2.87e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:44:09,296 [phygnn.py:576] : Epoch 133 train loss: 2.86e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:44:23,658 [phygnn.py:576] : Epoch 134 train loss: 2.81e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:44:38,161 [phygnn.py:576] : Epoch 135 train loss: 2.79e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:44:52,110 [phygnn.py:576] : Epoch 136 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:45:07,023 [phygnn.py:576] : Epoch 137 train loss: 2.86e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:45:21,348 [phygnn.py:576] : Epoch 138 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:45:35,279 [phygnn.py:576] : Epoch 139 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:45:49,968 [phygnn.py:576] : Epoch 140 train loss: 2.80e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:46:03,751 [phygnn.py:576] : Epoch 141 train loss: 2.80e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:46:18,611 [phygnn.py:576] : Epoch 142 train loss: 2.83e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:46:33,553 [phygnn.py:576] : Epoch 143 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:46:48,312 [phygnn.py:576] : Epoch 144 train loss: 2.82e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:47:02,599 [phygnn.py:576] : Epoch 145 train loss: 2.80e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:47:17,223 [phygnn.py:576] : Epoch 146 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:47:31,627 [phygnn.py:576] : Epoch 147 train loss: 2.81e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:47:46,634 [phygnn.py:576] : Epoch 148 train loss: 2.82e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:48:01,134 [phygnn.py:576] : Epoch 149 train loss: 2.84e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:48:14,970 [phygnn.py:576] : Epoch 150 train loss: 2.83e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:48:29,530 [phygnn.py:576] : Epoch 151 train loss: 2.81e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:48:43,017 [phygnn.py:576] : Epoch 152 train loss: 2.81e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:48:57,041 [phygnn.py:576] : Epoch 153 train loss: 2.84e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:49:11,023 [phygnn.py:576] : Epoch 154 train loss: 2.80e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:49:24,352 [phygnn.py:576] : Epoch 155 train loss: 2.77e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:49:39,064 [phygnn.py:576] : Epoch 156 train loss: 2.80e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:49:53,658 [phygnn.py:576] : Epoch 157 train loss: 2.80e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:50:07,934 [phygnn.py:576] : Epoch 158 train loss: 2.81e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:50:21,841 [phygnn.py:576] : Epoch 159 train loss: 2.80e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:50:36,101 [phygnn.py:576] : Epoch 160 train loss: 2.80e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:50:50,858 [phygnn.py:576] : Epoch 161 train loss: 2.80e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:51:04,555 [phygnn.py:576] : Epoch 162 train loss: 2.79e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:51:19,104 [phygnn.py:576] : Epoch 163 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:51:32,796 [phygnn.py:576] : Epoch 164 train loss: 2.79e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:51:46,580 [phygnn.py:576] : Epoch 165 train loss: 2.78e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:52:00,558 [phygnn.py:576] : Epoch 166 train loss: 2.80e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:52:14,129 [phygnn.py:576] : Epoch 167 train loss: 2.83e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:52:27,531 [phygnn.py:576] : Epoch 168 train loss: 2.85e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:52:41,445 [phygnn.py:576] : Epoch 169 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:52:55,583 [phygnn.py:576] : Epoch 170 train loss: 2.81e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:53:09,852 [phygnn.py:576] : Epoch 171 train loss: 2.78e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:53:23,115 [phygnn.py:576] : Epoch 172 train loss: 2.81e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:53:36,768 [phygnn.py:576] : Epoch 173 train loss: 2.77e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:53:50,946 [phygnn.py:576] : Epoch 174 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:54:04,171 [phygnn.py:576] : Epoch 175 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:54:17,918 [phygnn.py:576] : Epoch 176 train loss: 2.78e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:54:31,232 [phygnn.py:576] : Epoch 177 train loss: 2.80e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:54:43,925 [phygnn.py:576] : Epoch 178 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:54:57,404 [phygnn.py:576] : Epoch 179 train loss: 2.83e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:55:10,807 [phygnn.py:576] : Epoch 180 train loss: 2.82e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:55:24,268 [phygnn.py:576] : Epoch 181 train loss: 2.80e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:55:37,714 [phygnn.py:576] : Epoch 182 train loss: 2.76e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:55:51,534 [phygnn.py:576] : Epoch 183 train loss: 2.78e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:56:05,196 [phygnn.py:576] : Epoch 184 train loss: 2.76e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:56:18,128 [phygnn.py:576] : Epoch 185 train loss: 2.76e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:56:31,668 [phygnn.py:576] : Epoch 186 train loss: 2.77e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:56:44,657 [phygnn.py:576] : Epoch 187 train loss: 2.77e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:56:57,461 [phygnn.py:576] : Epoch 188 train loss: 2.78e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:57:10,013 [phygnn.py:576] : Epoch 189 train loss: 2.75e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:57:23,228 [phygnn.py:576] : Epoch 190 train loss: 2.80e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:57:36,550 [phygnn.py:576] : Epoch 191 train loss: 2.76e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:57:49,927 [phygnn.py:576] : Epoch 192 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:58:03,075 [phygnn.py:576] : Epoch 193 train loss: 2.79e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:58:16,524 [phygnn.py:576] : Epoch 194 train loss: 2.77e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:58:29,814 [phygnn.py:576] : Epoch 195 train loss: 2.75e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:58:42,654 [phygnn.py:576] : Epoch 196 train loss: 2.75e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:58:55,865 [phygnn.py:576] : Epoch 197 train loss: 2.78e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:59:09,091 [phygnn.py:576] : Epoch 198 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:59:22,020 [phygnn.py:576] : Epoch 199 train loss: 2.78e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:59:22,705 [trainer.py:102] : Training complete
INFO - 2022-03-01 13:59:22,739 [base.py:496] : Saved model to: /home/gbuster/code/mlclouds/mlclouds/model/k_fold/outputs/model_0.pkl
DEBUG - 2022-03-01 13:59:22,740 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 13:59:22,740 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:59:22,745 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:23,882 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:59:23,882 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:25,075 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:59:25,075 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:26,244 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:59:26,244 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:27,419 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:59:27,419 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:34,110 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 13:59:34,110 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:35,568 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 13:59:35,568 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:43,070 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 13:59:43,070 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 13:59:46,984 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 13:59:46,985 [data_handlers.py:413] : Shape df_raw=(3154464, 19), df_all_sky=(3154464, 14)
DEBUG - 2022-03-01 13:59:46,985 [data_handlers.py:420] : Shape after reset_index: df_raw=(3154464, 19), df_all_sky=(3154464, 14)
INFO - 2022-03-01 13:59:47,373 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:59:47,379 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 13:59:47,384 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:59:47,390 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:59:47,395 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:59:47,395 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:59:47,398 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,405 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,410 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,416 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,420 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.45% NaN values
DEBUG - 2022-03-01 13:59:47,424 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.45% NaN values
DEBUG - 2022-03-01 13:59:47,429 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.28% NaN values
DEBUG - 2022-03-01 13:59:47,433 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 13:59:47,437 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 13:59:47,442 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.27% NaN values
DEBUG - 2022-03-01 13:59:47,446 [data_cleaners.py:50] : 	"cloud_probability" has 3.27% NaN values
DEBUG - 2022-03-01 13:59:47,450 [data_cleaners.py:50] : 	"cloud_fraction" has 3.27% NaN values
DEBUG - 2022-03-01 13:59:47,454 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,459 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,463 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,467 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,471 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:47,476 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 13:59:47,480 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 13:59:47,480 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:59:51,513 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 13:59:51,935 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 13:59:51,935 [data_cleaners.py:107] : Cleaning took 5.0 seconds
INFO - 2022-03-01 13:59:52,313 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:59:52,318 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 13:59:52,323 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:59:52,329 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:59:52,334 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:59:52,334 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:59:52,337 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,343 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,348 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,352 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,356 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,362 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,366 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 13:59:52,371 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 13:59:52,375 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,379 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,384 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,388 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,391 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,395 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:59:52,395 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:59:54,611 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 13:59:55,018 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 13:59:55,019 [data_cleaners.py:107] : Cleaning took 3.1 seconds
DEBUG - 2022-03-01 13:59:55,020 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 13:59:55,059 [data_handlers.py:463] : Mask: shape=(3154464,), sum=1567353
DEBUG - 2022-03-01 13:59:55,220 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 13:59:55,220 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 13:59:59,284 [validator.py:110] : Predicted data shape is (1567353, 2)
DEBUG - 2022-03-01 13:59:59,798 [validator.py:158] : shapes: df_feature_val=(3154464, 20), df_all_sky_val=(3154464, 15)
INFO - 2022-03-01 14:00:00,112 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:00:00,116 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:00:00,116 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:00:00,116 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:00,166 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:00:00,842 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:00,881 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:00:01,553 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:01,590 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:00:02,259 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:02,297 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:00:02,972 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:03,010 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:00:03,679 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:03,715 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:00:04,391 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:04,429 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:00:05,100 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:05,138 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:00:05,804 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:05,841 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:00:06,525 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:00:06,525 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:06,576 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:00:07,237 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:07,274 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:00:07,931 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:07,967 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:00:08,628 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:08,666 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:00:09,328 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:09,364 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:00:10,022 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:10,058 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:00:10,719 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:10,756 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:00:11,416 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:11,453 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:00:12,107 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:12,148 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:00:12,807 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:00:12,808 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:12,858 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:00:13,533 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:13,569 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:00:14,242 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:14,278 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:00:14,955 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:14,994 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:00:15,666 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:15,703 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:00:16,371 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:16,407 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:00:17,086 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:17,123 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:00:17,794 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:17,831 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:00:18,508 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:18,545 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:00:19,228 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:00:19,228 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:19,279 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:00:19,946 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:19,984 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:00:20,642 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:20,679 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:00:21,339 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:21,377 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:00:22,039 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:22,075 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:00:22,731 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:22,767 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:00:23,430 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:23,466 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:00:24,124 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:24,162 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:00:24,825 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:24,864 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:00:25,521 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:00:25,521 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:25,701 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:00:26,381 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:26,496 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:00:27,220 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:27,335 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:00:28,027 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:28,142 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:00:28,836 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:28,950 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:00:29,650 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:29,769 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:00:30,496 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:30,611 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:00:31,328 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:31,443 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:00:32,137 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:32,252 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:00:32,965 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:00:32,965 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:33,017 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:00:33,688 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:33,726 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:00:34,400 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:34,437 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:00:35,106 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:35,141 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:00:35,813 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:35,849 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:00:36,519 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:36,555 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:00:37,228 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:37,264 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:00:37,932 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:37,968 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:00:38,649 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:38,685 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:00:39,362 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:00:39,362 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:39,539 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:00:40,244 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:40,358 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:00:41,063 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:41,178 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:00:41,884 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:41,999 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:00:42,719 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:42,833 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:00:43,543 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:43,657 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:00:44,416 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:44,530 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:00:45,273 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:45,387 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:00:46,115 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:46,230 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:00:46,969 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:00:46,969 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:00:47,065 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:00:47,765 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:00:47,833 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:00:48,542 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:00:48,610 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:00:49,319 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:00:49,386 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:00:50,091 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:00:50,161 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:00:50,873 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:00:50,940 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:00:51,649 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:00:51,719 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:00:52,429 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:00:52,496 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:00:53,213 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:00:53,280 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:00:54,007 [validator.py:187] : Shapes: df_base_full=(3154464, 6), df_surf_full=(3154464, 4)
DEBUG - 2022-03-01 14:00:54,012 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:00:54,054 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:10,050 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:01:10,093 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:25,990 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:01:26,033 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:41,824 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:01:41,868 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:01:57,761 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:01:57,804 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:02:13,769 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:02:13,812 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:02:29,832 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:02:29,874 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:02:45,813 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:02:45,856 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:03:01,817 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:03:01,860 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
INFO - 2022-03-01 14:03:17,751 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:03:17,759 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:03:17,759 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:03:17,762 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:03:18,872 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:03:18,872 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:03:20,014 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:03:20,015 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:03:27,301 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:03:27,302 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:03:34,682 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:03:34,683 [data_handlers.py:413] : Shape df_raw=(2207952, 19), df_all_sky=(2207952, 14)
DEBUG - 2022-03-01 14:03:34,683 [data_handlers.py:420] : Shape after reset_index: df_raw=(2207952, 19), df_all_sky=(2207952, 14)
INFO - 2022-03-01 14:03:34,930 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:03:34,934 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:03:34,938 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:03:34,941 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:03:34,945 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:03:34,945 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:03:34,948 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:34,952 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:34,956 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:34,960 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:34,963 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 49.99% NaN values
DEBUG - 2022-03-01 14:03:34,966 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 49.99% NaN values
DEBUG - 2022-03-01 14:03:34,970 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 0.35% NaN values
DEBUG - 2022-03-01 14:03:34,973 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:03:34,976 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:03:34,979 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 0.34% NaN values
DEBUG - 2022-03-01 14:03:34,982 [data_cleaners.py:50] : 	"cloud_probability" has 0.34% NaN values
DEBUG - 2022-03-01 14:03:34,985 [data_cleaners.py:50] : 	"cloud_fraction" has 0.34% NaN values
DEBUG - 2022-03-01 14:03:34,988 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:34,991 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:34,995 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:34,998 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:35,001 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:35,004 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:03:35,007 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:03:35,007 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:03:37,603 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:03:37,884 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:03:37,884 [data_cleaners.py:107] : Cleaning took 3.2 seconds
INFO - 2022-03-01 14:03:38,121 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:03:38,125 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:03:38,129 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:03:38,132 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:03:38,136 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:03:38,136 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:03:38,138 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,143 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,147 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,150 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,153 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,157 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,160 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:03:38,163 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:03:38,166 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,170 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,173 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,176 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,178 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,181 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:03:38,181 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:03:39,645 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:03:39,924 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:03:39,925 [data_cleaners.py:107] : Cleaning took 2.0 seconds
DEBUG - 2022-03-01 14:03:39,926 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:03:39,947 [data_handlers.py:463] : Mask: shape=(2207952,), sum=1097157
DEBUG - 2022-03-01 14:03:40,048 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:03:40,048 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:03:42,799 [validator.py:110] : Predicted data shape is (1097157, 2)
DEBUG - 2022-03-01 14:03:43,139 [validator.py:158] : shapes: df_feature_val=(2207952, 20), df_all_sky_val=(2207952, 15)
INFO - 2022-03-01 14:03:43,349 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:03:43,353 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:03:43,353 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:03:43,353 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:43,390 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:03:44,050 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:44,089 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:03:44,743 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:44,781 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:03:45,435 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:45,474 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:03:46,133 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:46,172 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:03:46,823 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:46,862 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:03:47,517 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:47,556 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:03:48,211 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:48,249 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:03:48,902 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:48,941 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:03:49,599 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:03:49,599 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:49,638 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:03:50,298 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:50,335 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:03:50,988 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:51,025 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:03:51,676 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:51,715 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:03:52,365 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:52,403 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:03:53,056 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:53,091 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:03:53,750 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:53,785 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:03:54,439 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:54,474 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:03:55,129 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:55,164 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:03:55,814 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:03:55,815 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:55,930 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:03:56,594 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:56,709 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:03:57,374 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:57,489 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:03:58,156 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:58,271 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:03:58,939 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:59,055 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:03:59,729 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:59,844 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:04:00,517 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:04:00,632 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:04:01,315 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:04:01,431 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:04:02,098 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:04:02,212 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:04:02,889 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:04:02,889 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:04:03,004 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:04:03,675 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:04:03,790 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:04:04,469 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:04:04,585 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:04:05,265 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:04:05,380 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:04:06,056 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:04:06,171 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:04:06,855 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:04:06,970 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:04:07,648 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:04:07,762 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:04:08,452 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:04:08,569 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:04:09,271 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:04:09,386 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:04:10,088 [validator.py:187] : Shapes: df_base_full=(2207952, 6), df_surf_full=(2207952, 4)
DEBUG - 2022-03-01 14:04:10,093 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:04:10,122 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:21,944 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:04:21,973 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:33,680 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:04:33,710 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:45,105 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:04:45,135 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:04:56,542 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:04:56,571 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:05:08,037 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:05:08,067 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:05:19,655 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:05:19,685 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:05:31,154 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:05:31,184 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:05:42,671 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:05:42,701 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
INFO - 2022-03-01 14:05:54,102 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:05:54,127 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:05:54,127 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:05:54,131 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:55,347 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:05:55,347 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:56,575 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:05:56,576 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:57,821 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:05:57,821 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:06:01,347 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 14:06:01,347 [data_handlers.py:413] : Shape df_raw=(946512, 19), df_all_sky=(946512, 14)
DEBUG - 2022-03-01 14:06:01,347 [data_handlers.py:420] : Shape after reset_index: df_raw=(946512, 19), df_all_sky=(946512, 14)
INFO - 2022-03-01 14:06:01,454 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:06:01,456 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:06:01,458 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:06:01,460 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:06:01,461 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:06:01,461 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:06:01,462 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,465 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,466 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,468 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,470 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 54.84% NaN values
DEBUG - 2022-03-01 14:06:01,471 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 54.84% NaN values
DEBUG - 2022-03-01 14:06:01,473 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 10.12% NaN values
DEBUG - 2022-03-01 14:06:01,475 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:06:01,476 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:06:01,478 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 10.10% NaN values
DEBUG - 2022-03-01 14:06:01,480 [data_cleaners.py:50] : 	"cloud_probability" has 10.10% NaN values
DEBUG - 2022-03-01 14:06:01,481 [data_cleaners.py:50] : 	"cloud_fraction" has 10.10% NaN values
DEBUG - 2022-03-01 14:06:01,483 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,485 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,486 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,488 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,490 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:01,491 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:06:01,493 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:06:01,493 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:06:02,569 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:06:02,694 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:06:02,695 [data_cleaners.py:107] : Cleaning took 1.3 seconds
INFO - 2022-03-01 14:06:02,800 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:06:02,802 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:06:02,804 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:06:02,805 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:06:02,807 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:06:02,807 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:06:02,808 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,810 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,812 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,813 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,815 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,817 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,819 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:06:02,821 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:06:02,822 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,824 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,826 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,827 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,828 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,830 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:02,830 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:06:03,410 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:06:03,539 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:06:03,539 [data_cleaners.py:107] : Cleaning took 0.8 seconds
DEBUG - 2022-03-01 14:06:03,539 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:06:03,552 [data_handlers.py:463] : Mask: shape=(946512,), sum=470196
DEBUG - 2022-03-01 14:06:03,591 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:06:03,591 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:06:04,741 [validator.py:110] : Predicted data shape is (470196, 2)
DEBUG - 2022-03-01 14:06:04,850 [validator.py:158] : shapes: df_feature_val=(946512, 20), df_all_sky_val=(946512, 15)
INFO - 2022-03-01 14:06:04,943 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:06:04,946 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:06:04,946 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:06:04,947 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:04,996 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:06:05,648 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:05,686 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:06:06,337 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:06,375 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:06:07,031 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:07,070 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:06:07,728 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:07,767 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:06:08,419 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:08,459 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:06:09,113 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:09,152 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:06:09,805 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:09,844 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:06:10,494 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:10,534 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:06:11,189 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:06:11,189 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:11,239 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:06:11,901 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:11,938 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:06:12,591 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:12,628 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:06:13,281 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:13,318 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:06:13,971 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:14,008 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:06:14,659 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:14,696 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:06:15,354 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:15,391 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:06:16,044 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:16,082 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:06:16,736 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:16,774 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:06:17,425 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:06:17,425 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:17,475 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:06:18,131 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:18,168 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:06:18,821 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:18,859 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:06:19,515 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:19,553 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:06:20,209 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:20,248 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:06:20,901 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:20,938 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:06:21,594 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:21,631 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:06:22,282 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:22,320 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:06:22,980 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:23,018 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:06:23,671 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:06:23,671 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:23,770 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:06:24,427 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:24,496 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:06:25,158 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:25,226 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:06:25,889 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:25,958 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:06:26,618 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:26,687 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:06:27,350 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:27,419 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:06:28,086 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:28,154 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:06:28,820 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:28,888 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:06:29,554 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:29,623 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:06:30,289 [validator.py:187] : Shapes: df_base_full=(946512, 6), df_surf_full=(946512, 4)
DEBUG - 2022-03-01 14:06:30,294 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:06:30,307 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:36,371 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:06:36,385 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:42,447 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:06:42,460 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:48,225 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:06:48,237 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:53,876 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:06:53,888 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:06:59,500 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:06:59,513 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:07:05,130 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:07:05,142 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:07:10,745 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:07:10,758 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:07:16,369 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:07:16,381 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
INFO - 2022-03-01 14:07:22,541 [validator.py:292] : Finished computing stats.
