Starting scenario 4, validation against site 6
2022-03-01 13:20:25.063457: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:20:25.063490: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Cross validation site: 6
Training sites: [0, 1, 2, 3, 4, 5, 7, 8]
Number of files: 8
Number of east files: 4
Number of west files: 4
Source files: ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
Full config: {'clean_training_data_kwargs': {'filter_clear': False, 'nan_option': 'interp'}, 'epochs_a': 100, 'epochs_b': 100, 'features': ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo'], 'hidden_layers': [{'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}], 'learning_rate': 0.0005, 'loss_weights_a': [1, 0], 'loss_weights_b': [0.5, 0.5], 'metric': 'relative_mae', 'n_batch': 32, 'one_hot_categories': {'flag': ['clear', 'ice_cloud', 'water_cloud', 'bad_cloud']}, 'p_fun': 'p_fun_all_sky', 'p_kwargs': {'loss_terms': ['mae_ghi']}, 'phygnn_seed': 0, 'surfrad_window_minutes': 15, 'y_labels': ['cld_opd_dcomp', 'cld_reff_dcomp']}
INFO - 2022-03-01 13:20:33,783 [trainer.py:40] : Trainer: Training on sites [0, 1, 2, 3, 4, 5, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:20:33,784 [trainer.py:49] : Trainer: Training on sites [0, 1, 2, 3, 4, 5, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:20:33,784 [data_handlers.py:60] : Loading training data
DEBUG - 2022-03-01 13:20:33,784 [data_handlers.py:78] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:20:33,784 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5
DEBUG - 2022-03-01 13:20:34,982 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:20:34,987 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:20:34,987 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:20:34,991 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:20:35,726 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:35,729 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:20:36,435 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:36,438 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:20:37,146 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:37,150 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:20:37,897 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:37,901 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:20:38,620 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:38,623 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:20:39,331 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:39,335 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:20:40,089 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:40,093 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:20:40,873 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:40,873 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5
DEBUG - 2022-03-01 13:20:41,954 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:20:41,959 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:20:41,959 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:20:41,962 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:20:42,624 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:42,628 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:20:43,287 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:43,290 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:20:43,950 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:43,954 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:20:44,615 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:44,619 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:20:45,278 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:45,281 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:20:45,938 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:45,942 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:20:46,595 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:46,598 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:20:47,260 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:20:47,260 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5
DEBUG - 2022-03-01 13:20:48,336 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:20:48,341 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:20:48,341 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:20:48,345 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:20:49,061 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:49,064 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:20:49,776 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:49,780 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:20:50,467 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:50,470 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:20:51,194 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:51,197 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:20:51,912 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:51,915 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:20:52,690 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:52,693 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:20:53,389 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:53,392 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:20:54,136 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:54,136 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5
DEBUG - 2022-03-01 13:20:55,212 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:20:55,217 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:20:55,217 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:20:55,220 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:20:55,877 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:55,880 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:20:56,526 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:56,530 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:20:57,171 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:57,174 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:20:57,817 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:57,821 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:20:58,458 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:58,461 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:20:59,109 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:59,112 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:20:59,760 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:20:59,764 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:21:00,405 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:00,405 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5
DEBUG - 2022-03-01 13:21:06,610 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:21:06,630 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:21:06,631 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:21:06,634 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:21:07,346 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:07,349 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:21:08,044 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:08,048 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:21:08,793 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:08,796 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:21:09,501 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:09,504 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:21:10,243 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:10,247 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:21:10,948 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:10,952 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:21:11,662 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:11,665 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:21:12,463 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:12,463 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5
DEBUG - 2022-03-01 13:21:13,727 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:21:13,731 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:21:13,731 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:21:13,735 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:21:14,382 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:14,386 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:21:15,035 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:15,038 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:21:15,684 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:15,687 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:21:16,334 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:16,337 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:21:16,981 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:16,985 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:21:17,631 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:17,635 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:21:18,287 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:18,290 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:21:18,937 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:21:18,938 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5
DEBUG - 2022-03-01 13:21:25,312 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:21:25,333 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:21:25,333 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:21:25,336 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:21:26,042 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:26,045 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:21:26,893 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:26,896 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:21:27,617 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:27,620 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:21:28,353 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:28,357 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:21:29,074 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:29,077 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:21:29,807 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:29,811 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:21:30,533 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:30,540 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:21:31,303 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:21:31,303 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 7, 8], from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5
DEBUG - 2022-03-01 13:21:34,595 [data_handlers.py:103] : 	Shape temp_raw=(420480, 19), temp_all_sky=(420480, 14)
DEBUG - 2022-03-01 13:21:34,605 [data_handlers.py:106] : 	Time step is 10 minutes
DEBUG - 2022-03-01 13:21:34,606 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 7, 8]
DEBUG - 2022-03-01 13:21:34,609 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:21:35,280 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:35,284 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:21:35,955 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:35,959 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:21:36,632 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:36,636 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:21:37,310 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:37,314 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:21:37,989 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:37,992 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:21:38,665 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:38,668 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:21:39,349 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:39,353 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:21:40,031 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:21:40,031 [data_handlers.py:136] : Data load complete. Shape df_raw=(2803968, 19), df_all_sky=(2803968, 14), df_surf=(2803968, 5)
DEBUG - 2022-03-01 13:21:40,893 [data_handlers.py:159] : Extracting 2D arrays to run rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:21:52,803 [data_handlers.py:176] : Running rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:23:46,442 [data_handlers.py:194] : Completed rest2 run for clearsky PhyGNN inputs.
INFO - 2022-03-01 13:23:48,107 [data_handlers.py:62] : Prepping training data
DEBUG - 2022-03-01 13:23:48,107 [data_handlers.py:214] : Training data clean kwargs: {'filter_daylight': True, 'filter_clear': False, 'add_cloud_flag': True, 'sza_lim': 89, 'nan_option': 'interp'}
DEBUG - 2022-03-01 13:23:48,107 [data_handlers.py:215] : Shape before cleaning: df_raw=(2803968, 19)
INFO - 2022-03-01 13:23:48,378 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 13:23:48,383 [data_cleaners.py:38] : 51.77% of daylight timesteps are cloudy
INFO - 2022-03-01 13:23:48,388 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:23:48,392 [data_cleaners.py:42] : 34.39% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:23:48,397 [data_cleaners.py:44] : 34.61% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:23:48,397 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:23:48,400 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,405 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,409 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,414 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,418 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.62% NaN values
DEBUG - 2022-03-01 13:23:48,422 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.62% NaN values
DEBUG - 2022-03-01 13:23:48,426 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.62% NaN values
DEBUG - 2022-03-01 13:23:48,430 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:23:48,433 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:23:48,437 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.61% NaN values
DEBUG - 2022-03-01 13:23:48,441 [data_cleaners.py:50] : 	"cloud_probability" has 3.61% NaN values
DEBUG - 2022-03-01 13:23:48,445 [data_cleaners.py:50] : 	"cloud_fraction" has 3.61% NaN values
DEBUG - 2022-03-01 13:23:48,449 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,453 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,457 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,460 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,464 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:48,468 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 83.12% NaN values
DEBUG - 2022-03-01 13:23:48,472 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.18% NaN values
DEBUG - 2022-03-01 13:23:48,472 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:23:51,629 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:23:51,907 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393108 after filters (49.68% of original)
DEBUG - 2022-03-01 13:23:52,031 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:23:52,031 [data_cleaners.py:107] : Cleaning took 3.9 seconds
DEBUG - 2022-03-01 13:23:52,031 [data_handlers.py:218] : Shape after cleaning: df_train=(1393108, 20)
DEBUG - 2022-03-01 13:23:52,031 [data_handlers.py:221] : Cleaning df_all_sky training data (for pfun).
DEBUG - 2022-03-01 13:23:52,031 [data_handlers.py:222] : Shape before cleaning: df_all_sky=(2803968, 25)
INFO - 2022-03-01 13:23:52,377 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 13:23:52,382 [data_cleaners.py:38] : 51.77% of daylight timesteps are cloudy
INFO - 2022-03-01 13:23:52,387 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:23:52,391 [data_cleaners.py:42] : 34.39% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:23:52,396 [data_cleaners.py:44] : 34.61% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:23:52,396 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:23:52,399 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,403 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,407 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,410 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,416 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,420 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 83.12% NaN values
DEBUG - 2022-03-01 13:23:52,423 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.18% NaN values
DEBUG - 2022-03-01 13:23:52,427 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,431 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,435 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,439 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,441 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,445 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,450 [data_cleaners.py:50] : 	"surfrad_dhi" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,456 [data_cleaners.py:50] : 	"surfrad_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,461 [data_cleaners.py:50] : 	"surfrad_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,463 [data_cleaners.py:50] : 	"doy" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,468 [data_cleaners.py:50] : 	"radius" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,474 [data_cleaners.py:50] : 	"Tuuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,479 [data_cleaners.py:50] : 	"clearsky_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,484 [data_cleaners.py:50] : 	"clearsky_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,489 [data_cleaners.py:50] : 	"Ruuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,494 [data_cleaners.py:50] : 	"Tddclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,499 [data_cleaners.py:50] : 	"Tduclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,505 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:23:52,505 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:23:55,059 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:23:55,337 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393108 after filters (49.68% of original)
DEBUG - 2022-03-01 13:23:55,491 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:23:55,491 [data_cleaners.py:107] : Cleaning took 3.5 seconds
DEBUG - 2022-03-01 13:23:55,492 [data_handlers.py:226] : Shape after cleaning: df_all_sky=(1393108, 26)
DEBUG - 2022-03-01 13:23:55,587 [data_handlers.py:240] : **Shape: df_train=(1393108, 17)
DEBUG - 2022-03-01 13:23:55,616 [data_handlers.py:250] : Shapes: x=(1393108, 15), y=(1393108, 2), p=(1393108, 26)
DEBUG - 2022-03-01 13:23:55,616 [data_handlers.py:253] : Training features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
DEBUG - 2022-03-01 13:23:55,616 [trainer.py:67] : Building PHYGNN model
INFO - 2022-03-01 13:23:55,616 [trainer.py:70] : Using p_fun: <function p_fun_all_sky at 0x2b6f476e58b0>
INFO - 2022-03-01 13:23:55,616 [base.py:152] : Active python environment versions: 
{   'numpy': '1.22.2',
    'pandas': '1.2.4',
    'phygnn': '0.0.14',
    'python': '3.8.8 (default, Feb 24 2021, 21:46:12) \n[GCC 7.3.0]',
    'sklearn': '0.24.1',
    'tensorflow': '2.8.0'}
INFO - 2022-03-01 13:23:55,632 [base.py:111] : Successfully initialized model with 17 layers
INFO - 2022-03-01 13:23:55,632 [trainer.py:84] : Training part A - pure data. Loss is [1, 0]
2022-03-01 13:24:04.731410: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.732579: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.733316: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.734061: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.734786: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.735552: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.736301: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.737059: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:24:04.737077: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-03-01 13:24:04.737519: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO - 2022-03-01 13:24:13,569 [phygnn.py:576] : Epoch 0 train loss: 7.03e-01 val loss: 6.90e-01 for "phygnn"
INFO - 2022-03-01 13:24:22,166 [phygnn.py:576] : Epoch 1 train loss: 6.30e-01 val loss: 6.22e-01 for "phygnn"
INFO - 2022-03-01 13:24:31,014 [phygnn.py:576] : Epoch 2 train loss: 5.65e-01 val loss: 5.51e-01 for "phygnn"
INFO - 2022-03-01 13:24:39,507 [phygnn.py:576] : Epoch 3 train loss: 5.33e-01 val loss: 5.14e-01 for "phygnn"
INFO - 2022-03-01 13:24:48,188 [phygnn.py:576] : Epoch 4 train loss: 5.10e-01 val loss: 4.93e-01 for "phygnn"
INFO - 2022-03-01 13:24:56,901 [phygnn.py:576] : Epoch 5 train loss: 5.05e-01 val loss: 4.81e-01 for "phygnn"
INFO - 2022-03-01 13:25:05,538 [phygnn.py:576] : Epoch 6 train loss: 4.87e-01 val loss: 4.76e-01 for "phygnn"
INFO - 2022-03-01 13:25:14,509 [phygnn.py:576] : Epoch 7 train loss: 4.83e-01 val loss: 4.73e-01 for "phygnn"
INFO - 2022-03-01 13:25:23,315 [phygnn.py:576] : Epoch 8 train loss: 4.86e-01 val loss: 4.66e-01 for "phygnn"
INFO - 2022-03-01 13:25:32,148 [phygnn.py:576] : Epoch 9 train loss: 4.76e-01 val loss: 4.65e-01 for "phygnn"
INFO - 2022-03-01 13:25:41,002 [phygnn.py:576] : Epoch 10 train loss: 4.72e-01 val loss: 4.62e-01 for "phygnn"
INFO - 2022-03-01 13:25:49,767 [phygnn.py:576] : Epoch 11 train loss: 4.75e-01 val loss: 4.59e-01 for "phygnn"
INFO - 2022-03-01 13:25:58,492 [phygnn.py:576] : Epoch 12 train loss: 4.77e-01 val loss: 4.56e-01 for "phygnn"
INFO - 2022-03-01 13:26:07,388 [phygnn.py:576] : Epoch 13 train loss: 4.71e-01 val loss: 4.57e-01 for "phygnn"
INFO - 2022-03-01 13:26:16,238 [phygnn.py:576] : Epoch 14 train loss: 4.74e-01 val loss: 4.53e-01 for "phygnn"
INFO - 2022-03-01 13:26:25,094 [phygnn.py:576] : Epoch 15 train loss: 4.64e-01 val loss: 4.52e-01 for "phygnn"
INFO - 2022-03-01 13:26:34,021 [phygnn.py:576] : Epoch 16 train loss: 4.58e-01 val loss: 4.52e-01 for "phygnn"
INFO - 2022-03-01 13:26:42,727 [phygnn.py:576] : Epoch 17 train loss: 4.61e-01 val loss: 4.46e-01 for "phygnn"
INFO - 2022-03-01 13:26:51,609 [phygnn.py:576] : Epoch 18 train loss: 4.58e-01 val loss: 4.46e-01 for "phygnn"
INFO - 2022-03-01 13:27:00,454 [phygnn.py:576] : Epoch 19 train loss: 4.58e-01 val loss: 4.44e-01 for "phygnn"
INFO - 2022-03-01 13:27:09,208 [phygnn.py:576] : Epoch 20 train loss: 4.58e-01 val loss: 4.40e-01 for "phygnn"
INFO - 2022-03-01 13:27:17,748 [phygnn.py:576] : Epoch 21 train loss: 4.57e-01 val loss: 4.38e-01 for "phygnn"
INFO - 2022-03-01 13:27:26,590 [phygnn.py:576] : Epoch 22 train loss: 4.58e-01 val loss: 4.40e-01 for "phygnn"
INFO - 2022-03-01 13:27:35,274 [phygnn.py:576] : Epoch 23 train loss: 4.51e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2022-03-01 13:27:43,996 [phygnn.py:576] : Epoch 24 train loss: 4.53e-01 val loss: 4.34e-01 for "phygnn"
INFO - 2022-03-01 13:27:52,903 [phygnn.py:576] : Epoch 25 train loss: 4.43e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2022-03-01 13:28:01,822 [phygnn.py:576] : Epoch 26 train loss: 4.51e-01 val loss: 4.32e-01 for "phygnn"
INFO - 2022-03-01 13:28:10,895 [phygnn.py:576] : Epoch 27 train loss: 4.46e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2022-03-01 13:28:19,520 [phygnn.py:576] : Epoch 28 train loss: 4.44e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2022-03-01 13:28:28,448 [phygnn.py:576] : Epoch 29 train loss: 4.44e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:28:37,331 [phygnn.py:576] : Epoch 30 train loss: 4.39e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:28:46,368 [phygnn.py:576] : Epoch 31 train loss: 4.51e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:28:55,293 [phygnn.py:576] : Epoch 32 train loss: 4.37e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:29:04,192 [phygnn.py:576] : Epoch 33 train loss: 4.36e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:29:13,180 [phygnn.py:576] : Epoch 34 train loss: 4.41e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:29:22,236 [phygnn.py:576] : Epoch 35 train loss: 4.41e-01 val loss: 4.24e-01 for "phygnn"
INFO - 2022-03-01 13:29:31,135 [phygnn.py:576] : Epoch 36 train loss: 4.41e-01 val loss: 4.25e-01 for "phygnn"
INFO - 2022-03-01 13:29:39,921 [phygnn.py:576] : Epoch 37 train loss: 4.37e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:29:48,743 [phygnn.py:576] : Epoch 38 train loss: 4.37e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:29:57,739 [phygnn.py:576] : Epoch 39 train loss: 4.40e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:30:06,692 [phygnn.py:576] : Epoch 40 train loss: 4.30e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:30:15,543 [phygnn.py:576] : Epoch 41 train loss: 4.34e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:30:24,445 [phygnn.py:576] : Epoch 42 train loss: 4.31e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:30:33,457 [phygnn.py:576] : Epoch 43 train loss: 4.38e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:30:42,404 [phygnn.py:576] : Epoch 44 train loss: 4.36e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:30:51,095 [phygnn.py:576] : Epoch 45 train loss: 4.32e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:31:00,011 [phygnn.py:576] : Epoch 46 train loss: 4.33e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:31:09,123 [phygnn.py:576] : Epoch 47 train loss: 4.32e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:31:17,979 [phygnn.py:576] : Epoch 48 train loss: 4.36e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:31:27,047 [phygnn.py:576] : Epoch 49 train loss: 4.30e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:31:35,753 [phygnn.py:576] : Epoch 50 train loss: 4.27e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:31:44,649 [phygnn.py:576] : Epoch 51 train loss: 4.24e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:31:53,409 [phygnn.py:576] : Epoch 52 train loss: 4.30e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:32:02,468 [phygnn.py:576] : Epoch 53 train loss: 4.27e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:32:11,528 [phygnn.py:576] : Epoch 54 train loss: 4.23e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:32:20,264 [phygnn.py:576] : Epoch 55 train loss: 4.27e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:32:29,145 [phygnn.py:576] : Epoch 56 train loss: 4.19e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:32:37,949 [phygnn.py:576] : Epoch 57 train loss: 4.29e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:32:46,980 [phygnn.py:576] : Epoch 58 train loss: 4.21e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:32:56,028 [phygnn.py:576] : Epoch 59 train loss: 4.21e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:33:04,868 [phygnn.py:576] : Epoch 60 train loss: 4.28e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:33:13,993 [phygnn.py:576] : Epoch 61 train loss: 4.26e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:33:22,880 [phygnn.py:576] : Epoch 62 train loss: 4.28e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:33:31,847 [phygnn.py:576] : Epoch 63 train loss: 4.22e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:33:40,703 [phygnn.py:576] : Epoch 64 train loss: 4.30e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:33:49,521 [phygnn.py:576] : Epoch 65 train loss: 4.22e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:33:58,431 [phygnn.py:576] : Epoch 66 train loss: 4.17e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:34:07,431 [phygnn.py:576] : Epoch 67 train loss: 4.14e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:34:16,084 [phygnn.py:576] : Epoch 68 train loss: 4.22e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:25,144 [phygnn.py:576] : Epoch 69 train loss: 4.19e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:34,214 [phygnn.py:576] : Epoch 70 train loss: 4.25e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:34:43,266 [phygnn.py:576] : Epoch 71 train loss: 4.16e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:34:52,243 [phygnn.py:576] : Epoch 72 train loss: 4.19e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:35:01,137 [phygnn.py:576] : Epoch 73 train loss: 4.20e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:35:10,173 [phygnn.py:576] : Epoch 74 train loss: 4.18e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:35:19,301 [phygnn.py:576] : Epoch 75 train loss: 4.10e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:35:28,052 [phygnn.py:576] : Epoch 76 train loss: 4.18e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:35:36,967 [phygnn.py:576] : Epoch 77 train loss: 4.07e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:35:46,001 [phygnn.py:576] : Epoch 78 train loss: 4.15e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:35:54,853 [phygnn.py:576] : Epoch 79 train loss: 4.14e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:36:03,768 [phygnn.py:576] : Epoch 80 train loss: 4.15e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:36:12,714 [phygnn.py:576] : Epoch 81 train loss: 4.19e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2022-03-01 13:36:21,647 [phygnn.py:576] : Epoch 82 train loss: 4.16e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:36:30,540 [phygnn.py:576] : Epoch 83 train loss: 4.10e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2022-03-01 13:36:39,447 [phygnn.py:576] : Epoch 84 train loss: 4.23e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2022-03-01 13:36:48,224 [phygnn.py:576] : Epoch 85 train loss: 4.12e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2022-03-01 13:36:57,208 [phygnn.py:576] : Epoch 86 train loss: 4.19e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:37:05,990 [phygnn.py:576] : Epoch 87 train loss: 4.08e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:37:14,933 [phygnn.py:576] : Epoch 88 train loss: 4.08e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:37:24,000 [phygnn.py:576] : Epoch 89 train loss: 4.10e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2022-03-01 13:37:32,935 [phygnn.py:576] : Epoch 90 train loss: 4.13e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2022-03-01 13:37:41,604 [phygnn.py:576] : Epoch 91 train loss: 4.04e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2022-03-01 13:37:50,384 [phygnn.py:576] : Epoch 92 train loss: 4.16e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2022-03-01 13:37:59,454 [phygnn.py:576] : Epoch 93 train loss: 4.08e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2022-03-01 13:38:08,362 [phygnn.py:576] : Epoch 94 train loss: 4.08e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2022-03-01 13:38:16,870 [phygnn.py:576] : Epoch 95 train loss: 4.13e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:38:25,609 [phygnn.py:576] : Epoch 96 train loss: 4.09e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2022-03-01 13:38:34,428 [phygnn.py:576] : Epoch 97 train loss: 4.07e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2022-03-01 13:38:43,407 [phygnn.py:576] : Epoch 98 train loss: 4.07e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2022-03-01 13:38:52,227 [phygnn.py:576] : Epoch 99 train loss: 4.06e-01 val loss: 3.94e-01 for "phygnn"
INFO - 2022-03-01 13:38:53,097 [trainer.py:92] : Training part B - data and phygnn. Loss is [0.5, 0.5]
INFO - 2022-03-01 13:39:17,096 [phygnn.py:576] : Epoch 100 train loss: 2.77e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:39:31,257 [phygnn.py:576] : Epoch 101 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:39:44,909 [phygnn.py:576] : Epoch 102 train loss: 2.83e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:39:58,680 [phygnn.py:576] : Epoch 103 train loss: 2.83e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:40:12,372 [phygnn.py:576] : Epoch 104 train loss: 2.82e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:40:26,134 [phygnn.py:576] : Epoch 105 train loss: 2.81e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:40:39,932 [phygnn.py:576] : Epoch 106 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:40:54,026 [phygnn.py:576] : Epoch 107 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:41:08,470 [phygnn.py:576] : Epoch 108 train loss: 2.84e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:41:22,740 [phygnn.py:576] : Epoch 109 train loss: 2.82e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:41:36,684 [phygnn.py:576] : Epoch 110 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:41:50,946 [phygnn.py:576] : Epoch 111 train loss: 2.82e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:05,124 [phygnn.py:576] : Epoch 112 train loss: 2.85e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:19,071 [phygnn.py:576] : Epoch 113 train loss: 2.77e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:42:33,091 [phygnn.py:576] : Epoch 114 train loss: 2.83e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:42:47,135 [phygnn.py:576] : Epoch 115 train loss: 2.82e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:43:01,076 [phygnn.py:576] : Epoch 116 train loss: 2.77e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:43:15,226 [phygnn.py:576] : Epoch 117 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:43:29,430 [phygnn.py:576] : Epoch 118 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:43:43,183 [phygnn.py:576] : Epoch 119 train loss: 2.81e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:43:57,219 [phygnn.py:576] : Epoch 120 train loss: 2.84e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:44:10,870 [phygnn.py:576] : Epoch 121 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:44:24,791 [phygnn.py:576] : Epoch 122 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:44:38,809 [phygnn.py:576] : Epoch 123 train loss: 2.78e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:44:52,691 [phygnn.py:576] : Epoch 124 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:45:06,700 [phygnn.py:576] : Epoch 125 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:45:20,715 [phygnn.py:576] : Epoch 126 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:45:34,864 [phygnn.py:576] : Epoch 127 train loss: 2.83e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:45:48,840 [phygnn.py:576] : Epoch 128 train loss: 2.83e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:46:02,861 [phygnn.py:576] : Epoch 129 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:46:16,719 [phygnn.py:576] : Epoch 130 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:46:30,913 [phygnn.py:576] : Epoch 131 train loss: 2.78e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:46:44,927 [phygnn.py:576] : Epoch 132 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:46:59,075 [phygnn.py:576] : Epoch 133 train loss: 2.76e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:47:13,130 [phygnn.py:576] : Epoch 134 train loss: 2.76e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:47:27,520 [phygnn.py:576] : Epoch 135 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:47:41,604 [phygnn.py:576] : Epoch 136 train loss: 2.76e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:47:55,552 [phygnn.py:576] : Epoch 137 train loss: 2.81e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:48:09,196 [phygnn.py:576] : Epoch 138 train loss: 2.77e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:48:23,535 [phygnn.py:576] : Epoch 139 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:48:37,674 [phygnn.py:576] : Epoch 140 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:48:51,506 [phygnn.py:576] : Epoch 141 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:49:05,745 [phygnn.py:576] : Epoch 142 train loss: 2.76e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:49:19,485 [phygnn.py:576] : Epoch 143 train loss: 2.76e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:49:33,835 [phygnn.py:576] : Epoch 144 train loss: 2.77e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:49:47,689 [phygnn.py:576] : Epoch 145 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:50:01,936 [phygnn.py:576] : Epoch 146 train loss: 2.82e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:50:16,219 [phygnn.py:576] : Epoch 147 train loss: 2.77e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:50:29,929 [phygnn.py:576] : Epoch 148 train loss: 2.81e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:50:43,992 [phygnn.py:576] : Epoch 149 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:50:58,314 [phygnn.py:576] : Epoch 150 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:51:12,321 [phygnn.py:576] : Epoch 151 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:51:26,622 [phygnn.py:576] : Epoch 152 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:51:40,383 [phygnn.py:576] : Epoch 153 train loss: 2.77e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:51:54,179 [phygnn.py:576] : Epoch 154 train loss: 2.74e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:52:08,149 [phygnn.py:576] : Epoch 155 train loss: 2.74e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:52:21,951 [phygnn.py:576] : Epoch 156 train loss: 2.78e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:52:36,290 [phygnn.py:576] : Epoch 157 train loss: 2.76e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:52:50,195 [phygnn.py:576] : Epoch 158 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:04,007 [phygnn.py:576] : Epoch 159 train loss: 2.78e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:17,654 [phygnn.py:576] : Epoch 160 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:31,392 [phygnn.py:576] : Epoch 161 train loss: 2.73e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:45,498 [phygnn.py:576] : Epoch 162 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:53:59,249 [phygnn.py:576] : Epoch 163 train loss: 2.78e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:54:12,933 [phygnn.py:576] : Epoch 164 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:54:26,498 [phygnn.py:576] : Epoch 165 train loss: 2.73e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:54:40,217 [phygnn.py:576] : Epoch 166 train loss: 2.76e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:54:53,721 [phygnn.py:576] : Epoch 167 train loss: 2.74e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:55:07,550 [phygnn.py:576] : Epoch 168 train loss: 2.74e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:55:20,956 [phygnn.py:576] : Epoch 169 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:55:34,652 [phygnn.py:576] : Epoch 170 train loss: 2.73e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:55:48,517 [phygnn.py:576] : Epoch 171 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2022-03-01 13:56:02,494 [phygnn.py:576] : Epoch 172 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:16,229 [phygnn.py:576] : Epoch 173 train loss: 2.78e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:29,796 [phygnn.py:576] : Epoch 174 train loss: 2.76e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:43,161 [phygnn.py:576] : Epoch 175 train loss: 2.74e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:56:56,895 [phygnn.py:576] : Epoch 176 train loss: 2.76e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:10,572 [phygnn.py:576] : Epoch 177 train loss: 2.76e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:24,448 [phygnn.py:576] : Epoch 178 train loss: 2.78e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:37,824 [phygnn.py:576] : Epoch 179 train loss: 2.72e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:57:51,024 [phygnn.py:576] : Epoch 180 train loss: 2.71e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:58:04,265 [phygnn.py:576] : Epoch 181 train loss: 2.76e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:58:17,579 [phygnn.py:576] : Epoch 182 train loss: 2.76e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:58:31,182 [phygnn.py:576] : Epoch 183 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2022-03-01 13:58:44,691 [phygnn.py:576] : Epoch 184 train loss: 2.78e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:58:58,028 [phygnn.py:576] : Epoch 185 train loss: 2.70e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:59:11,749 [phygnn.py:576] : Epoch 186 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:59:24,928 [phygnn.py:576] : Epoch 187 train loss: 2.71e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 13:59:38,253 [phygnn.py:576] : Epoch 188 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 13:59:51,486 [phygnn.py:576] : Epoch 189 train loss: 2.74e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 14:00:04,516 [phygnn.py:576] : Epoch 190 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 14:00:17,791 [phygnn.py:576] : Epoch 191 train loss: 2.76e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 14:00:31,349 [phygnn.py:576] : Epoch 192 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 14:00:44,599 [phygnn.py:576] : Epoch 193 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 14:00:57,581 [phygnn.py:576] : Epoch 194 train loss: 2.70e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 14:01:10,732 [phygnn.py:576] : Epoch 195 train loss: 2.72e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 14:01:24,114 [phygnn.py:576] : Epoch 196 train loss: 2.73e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 14:01:37,280 [phygnn.py:576] : Epoch 197 train loss: 2.75e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 14:01:50,287 [phygnn.py:576] : Epoch 198 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2022-03-01 14:02:03,401 [phygnn.py:576] : Epoch 199 train loss: 2.74e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2022-03-01 14:02:04,192 [trainer.py:102] : Training complete
INFO - 2022-03-01 14:02:04,229 [base.py:496] : Saved model to: /home/gbuster/code/mlclouds/mlclouds/model/k_fold/outputs/model_6.pkl
DEBUG - 2022-03-01 14:02:04,230 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:02:04,230 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:02:04,234 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:05,412 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:02:05,413 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:06,752 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:02:06,752 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:08,016 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:02:08,016 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:09,305 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:02:09,305 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:16,833 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:02:16,833 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:18,298 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:02:18,298 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:26,068 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:02:26,068 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:30,152 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 14:02:30,152 [data_handlers.py:413] : Shape df_raw=(3154464, 19), df_all_sky=(3154464, 14)
DEBUG - 2022-03-01 14:02:30,152 [data_handlers.py:420] : Shape after reset_index: df_raw=(3154464, 19), df_all_sky=(3154464, 14)
INFO - 2022-03-01 14:02:30,533 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:02:30,539 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 14:02:30,544 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:02:30,549 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:02:30,555 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:02:30,555 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:02:30,558 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,565 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,570 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,576 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,580 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.45% NaN values
DEBUG - 2022-03-01 14:02:30,584 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.45% NaN values
DEBUG - 2022-03-01 14:02:30,589 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.28% NaN values
DEBUG - 2022-03-01 14:02:30,593 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 14:02:30,597 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 14:02:30,601 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.27% NaN values
DEBUG - 2022-03-01 14:02:30,606 [data_cleaners.py:50] : 	"cloud_probability" has 3.27% NaN values
DEBUG - 2022-03-01 14:02:30,610 [data_cleaners.py:50] : 	"cloud_fraction" has 3.27% NaN values
DEBUG - 2022-03-01 14:02:30,614 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,619 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,623 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,627 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,632 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:30,636 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 14:02:30,640 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 14:02:30,640 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:02:34,669 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:02:35,115 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:02:35,115 [data_cleaners.py:107] : Cleaning took 5.0 seconds
INFO - 2022-03-01 14:02:35,489 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:02:35,495 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 14:02:35,500 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:02:35,505 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:02:35,511 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:02:35,511 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:02:35,514 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,521 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,525 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,529 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,534 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,540 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,544 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 14:02:35,548 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 14:02:35,552 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,557 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,561 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,565 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,568 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,573 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:02:35,573 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:02:37,845 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:02:38,252 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:02:38,252 [data_cleaners.py:107] : Cleaning took 3.1 seconds
DEBUG - 2022-03-01 14:02:38,253 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:02:38,296 [data_handlers.py:463] : Mask: shape=(3154464,), sum=1567353
DEBUG - 2022-03-01 14:02:38,463 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:02:38,463 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:02:42,365 [validator.py:110] : Predicted data shape is (1567353, 2)
DEBUG - 2022-03-01 14:02:42,876 [validator.py:158] : shapes: df_feature_val=(3154464, 20), df_all_sky_val=(3154464, 15)
INFO - 2022-03-01 14:02:43,204 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:02:43,208 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:02:43,208 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:02:43,209 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:02:43,261 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:02:43,975 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:02:44,015 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:02:44,734 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:02:44,773 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:02:45,485 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:02:45,525 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:02:46,246 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:02:46,286 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:02:47,006 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:02:47,046 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:02:47,768 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:02:47,807 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:02:48,537 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:02:48,576 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:02:49,296 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:02:49,333 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:02:50,082 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:02:50,082 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:02:50,132 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:02:50,794 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:02:50,833 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:02:51,498 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:02:51,534 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:02:52,202 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:02:52,239 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:02:52,906 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:02:52,945 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:02:53,611 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:02:53,648 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:02:54,314 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:02:54,351 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:02:55,010 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:02:55,047 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:02:55,703 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:02:55,740 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:02:56,413 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:02:56,413 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:02:56,465 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:02:57,200 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:02:57,237 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:02:57,960 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:02:57,997 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:02:58,717 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:02:58,755 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:02:59,475 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:02:59,511 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:03:00,225 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:00,262 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:03:00,985 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:01,023 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:03:01,764 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:01,801 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:03:02,499 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:02,536 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:03:03,329 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:03:03,329 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:03,378 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:03:04,046 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:04,083 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:03:04,748 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:04,785 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:03:05,447 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:05,484 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:03:06,145 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:06,183 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:03:06,843 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:06,880 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:03:07,544 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:07,581 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:03:08,239 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:08,276 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:03:08,943 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:08,980 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:03:09,652 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:03:09,653 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:09,980 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:03:10,707 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:10,822 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:03:11,633 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:11,749 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:03:12,498 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:12,614 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:03:13,355 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:13,470 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:03:14,205 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:14,321 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:03:15,048 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:15,164 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:03:15,901 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:16,018 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:03:16,751 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:16,867 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:03:17,642 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:03:17,642 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:17,691 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:03:18,365 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:18,402 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:03:19,072 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:19,110 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:03:19,784 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:19,823 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:03:20,502 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:20,539 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:03:21,209 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:21,246 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:03:21,919 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:21,957 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:03:22,629 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:22,665 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:03:23,342 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:23,379 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:03:24,057 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:03:24,057 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:24,375 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:03:25,130 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:25,245 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:03:26,012 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:26,129 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:03:26,873 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:26,988 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:03:27,745 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:27,860 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:03:28,600 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:28,718 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:03:29,467 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:29,585 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:03:30,320 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:30,437 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:03:31,189 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:31,305 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:03:32,122 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:03:32,122 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:03:32,222 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:03:32,926 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:03:32,994 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:03:33,707 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:03:33,775 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:03:34,490 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:03:34,559 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:03:35,269 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:03:35,339 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:03:36,051 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:03:36,120 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:03:36,816 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:03:36,885 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:03:37,588 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:03:37,657 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:03:38,368 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:03:38,438 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:03:39,155 [validator.py:187] : Shapes: df_base_full=(3154464, 6), df_surf_full=(3154464, 4)
DEBUG - 2022-03-01 14:03:39,161 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:03:39,200 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:03:55,734 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:03:55,776 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:04:12,107 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:04:12,148 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:04:27,949 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:04:27,992 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:04:44,029 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:04:44,073 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:05:00,565 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:05:00,608 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:05:16,742 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:05:16,783 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:05:32,796 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:05:32,837 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:05:49,016 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:05:49,056 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
INFO - 2022-03-01 14:06:04,985 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:06:04,992 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:06:04,992 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:06:04,996 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:06:06,057 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:06:06,057 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:06:07,191 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:06:07,191 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:06:13,910 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:06:13,910 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:06:21,079 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:06:21,079 [data_handlers.py:413] : Shape df_raw=(2207952, 19), df_all_sky=(2207952, 14)
DEBUG - 2022-03-01 14:06:21,079 [data_handlers.py:420] : Shape after reset_index: df_raw=(2207952, 19), df_all_sky=(2207952, 14)
INFO - 2022-03-01 14:06:21,320 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:06:21,324 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:06:21,328 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:06:21,331 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:06:21,335 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:06:21,335 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:06:21,338 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,342 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,345 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,349 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,352 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 49.99% NaN values
DEBUG - 2022-03-01 14:06:21,356 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 49.99% NaN values
DEBUG - 2022-03-01 14:06:21,359 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 0.35% NaN values
DEBUG - 2022-03-01 14:06:21,362 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:06:21,365 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:06:21,368 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 0.34% NaN values
DEBUG - 2022-03-01 14:06:21,371 [data_cleaners.py:50] : 	"cloud_probability" has 0.34% NaN values
DEBUG - 2022-03-01 14:06:21,374 [data_cleaners.py:50] : 	"cloud_fraction" has 0.34% NaN values
DEBUG - 2022-03-01 14:06:21,378 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,381 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,384 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,387 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,390 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:21,393 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:06:21,396 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:06:21,396 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:06:23,949 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:06:24,226 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:06:24,226 [data_cleaners.py:107] : Cleaning took 3.1 seconds
INFO - 2022-03-01 14:06:24,462 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:06:24,466 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:06:24,470 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:06:24,474 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:06:24,477 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:06:24,477 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:06:24,480 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,484 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,487 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,490 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,494 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,498 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,501 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:06:24,504 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:06:24,507 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,510 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,513 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,517 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,519 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,522 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:06:24,522 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:06:25,980 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:06:26,254 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:06:26,254 [data_cleaners.py:107] : Cleaning took 2.0 seconds
DEBUG - 2022-03-01 14:06:26,256 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:06:26,277 [data_handlers.py:463] : Mask: shape=(2207952,), sum=1097157
DEBUG - 2022-03-01 14:06:26,372 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:06:26,373 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:06:29,040 [validator.py:110] : Predicted data shape is (1097157, 2)
DEBUG - 2022-03-01 14:06:29,362 [validator.py:158] : shapes: df_feature_val=(2207952, 20), df_all_sky_val=(2207952, 15)
INFO - 2022-03-01 14:06:29,594 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:06:29,598 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:06:29,598 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:06:29,598 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:29,634 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:06:30,279 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:30,318 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:06:30,960 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:30,998 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:06:31,645 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:31,683 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:06:32,335 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:32,373 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:06:33,016 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:33,054 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:06:33,698 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:33,736 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:06:34,378 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:34,416 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:06:35,061 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:35,096 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:06:35,746 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:06:35,746 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:35,781 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:06:36,431 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:36,466 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:06:37,107 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:37,142 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:06:37,785 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:37,821 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:06:38,460 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:38,496 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:06:39,136 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:39,171 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:06:39,820 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:39,855 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:06:40,495 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:40,531 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:06:41,176 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:41,212 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:06:41,856 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:06:41,856 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:41,969 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:06:42,623 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:42,737 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:06:43,394 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:43,507 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:06:44,164 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:44,277 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:06:44,938 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:45,051 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:06:45,713 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:45,827 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:06:46,491 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:46,604 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:06:47,287 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:47,399 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:06:48,067 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:48,179 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:06:48,859 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:06:48,859 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:48,972 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:06:49,645 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:49,758 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:06:50,436 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:50,549 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:06:51,236 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:51,349 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:06:52,032 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:52,146 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:06:52,833 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:52,948 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:06:53,638 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:53,752 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:06:54,438 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:54,553 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:06:55,253 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:55,368 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:06:56,088 [validator.py:187] : Shapes: df_base_full=(2207952, 6), df_surf_full=(2207952, 4)
DEBUG - 2022-03-01 14:06:56,093 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:06:56,119 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:07:07,779 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:07:07,809 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:07:19,712 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:07:19,741 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:07:31,117 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:07:31,145 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:07:43,142 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:07:43,173 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:07:55,214 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:07:55,245 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:08:06,854 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:08:06,883 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:08:18,992 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:08:19,023 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:08:30,933 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:08:30,963 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
INFO - 2022-03-01 14:08:42,970 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:08:43,009 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:08:43,009 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:08:43,012 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:44,224 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:08:44,224 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:45,470 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:08:45,470 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:46,723 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:08:46,723 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:50,381 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 14:08:50,381 [data_handlers.py:413] : Shape df_raw=(946512, 19), df_all_sky=(946512, 14)
DEBUG - 2022-03-01 14:08:50,381 [data_handlers.py:420] : Shape after reset_index: df_raw=(946512, 19), df_all_sky=(946512, 14)
INFO - 2022-03-01 14:08:50,490 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:08:50,492 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:08:50,494 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:08:50,496 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:08:50,497 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:08:50,497 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:08:50,499 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,501 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,502 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,505 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,506 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 54.84% NaN values
DEBUG - 2022-03-01 14:08:50,508 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 54.84% NaN values
DEBUG - 2022-03-01 14:08:50,510 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 10.12% NaN values
DEBUG - 2022-03-01 14:08:50,511 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:08:50,513 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:08:50,515 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 10.10% NaN values
DEBUG - 2022-03-01 14:08:50,516 [data_cleaners.py:50] : 	"cloud_probability" has 10.10% NaN values
DEBUG - 2022-03-01 14:08:50,518 [data_cleaners.py:50] : 	"cloud_fraction" has 10.10% NaN values
DEBUG - 2022-03-01 14:08:50,519 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,521 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,523 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,524 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,526 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:50,528 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:08:50,529 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:08:50,529 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:08:51,650 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:08:51,780 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:08:51,780 [data_cleaners.py:107] : Cleaning took 1.4 seconds
INFO - 2022-03-01 14:08:51,887 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:08:51,889 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:08:51,890 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:08:51,892 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:08:51,894 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:08:51,894 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:08:51,895 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,897 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,899 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,900 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,902 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,904 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,906 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:08:51,908 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:08:51,909 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,911 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,913 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,914 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,915 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,917 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:51,917 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:08:52,511 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:08:52,635 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:08:52,636 [data_cleaners.py:107] : Cleaning took 0.9 seconds
DEBUG - 2022-03-01 14:08:52,636 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:08:52,649 [data_handlers.py:463] : Mask: shape=(946512,), sum=470196
DEBUG - 2022-03-01 14:08:52,688 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:08:52,688 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:08:53,829 [validator.py:110] : Predicted data shape is (470196, 2)
DEBUG - 2022-03-01 14:08:53,947 [validator.py:158] : shapes: df_feature_val=(946512, 20), df_all_sky_val=(946512, 15)
INFO - 2022-03-01 14:08:54,039 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:08:54,042 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:54,042 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:08:54,042 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:08:54,092 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:08:54,750 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:08:54,789 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:08:55,447 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:08:55,486 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:08:56,142 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:08:56,180 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:08:56,843 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:08:56,881 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:08:57,538 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:08:57,578 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:08:58,236 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:08:58,275 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:08:58,933 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:08:58,971 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:08:59,628 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:08:59,667 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:09:00,329 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:09:00,329 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:09:00,379 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:09:01,029 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:09:01,066 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:09:01,705 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:09:01,745 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:09:02,387 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:09:02,426 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:09:03,068 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:09:03,105 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:09:03,755 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:09:03,792 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:09:04,460 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:09:04,496 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:09:05,149 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:09:05,185 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:09:05,852 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:09:05,889 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:09:06,535 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:09:06,536 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:09:06,584 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:09:07,234 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:09:07,272 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:09:07,912 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:09:07,948 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:09:08,588 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:09:08,624 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:09:09,265 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:09:09,301 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:09:09,941 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:09:09,975 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:09:10,616 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:09:10,651 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:09:11,290 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:09:11,325 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:09:11,979 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:09:12,014 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:09:12,655 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:09:12,655 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:09:12,751 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:09:13,398 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:09:13,465 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:09:14,110 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:09:14,177 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:09:14,826 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:09:14,892 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:09:15,538 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:09:15,605 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:09:16,256 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:09:16,322 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:09:16,978 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:09:17,044 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:09:17,694 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:09:17,760 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:09:18,421 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:09:18,488 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:09:19,152 [validator.py:187] : Shapes: df_base_full=(946512, 6), df_surf_full=(946512, 4)
DEBUG - 2022-03-01 14:09:19,156 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:09:19,168 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:09:24,888 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:09:24,899 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:09:30,580 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:09:30,592 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:09:36,233 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:09:36,245 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:09:41,918 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:09:41,930 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:09:47,623 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:09:47,634 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:09:53,338 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:09:53,350 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:09:59,037 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:09:59,049 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:10:04,747 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:10:04,759 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
INFO - 2022-03-01 14:10:10,428 [validator.py:292] : Finished computing stats.
