Starting scenario 4, validation against site 7
2022-03-01 13:26:07.786237: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:26:07.786263: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Cross validation site: 7
Training sites: [0, 1, 2, 3, 4, 5, 6, 8]
Number of files: 8
Number of east files: 4
Number of west files: 4
Source files: ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
Full config: {'clean_training_data_kwargs': {'filter_clear': False, 'nan_option': 'interp'}, 'epochs_a': 100, 'epochs_b': 100, 'features': ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo'], 'hidden_layers': [{'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}], 'learning_rate': 0.0005, 'loss_weights_a': [1, 0], 'loss_weights_b': [0.5, 0.5], 'metric': 'relative_mae', 'n_batch': 32, 'one_hot_categories': {'flag': ['clear', 'ice_cloud', 'water_cloud', 'bad_cloud']}, 'p_fun': 'p_fun_all_sky', 'p_kwargs': {'loss_terms': ['mae_ghi']}, 'phygnn_seed': 0, 'surfrad_window_minutes': 15, 'y_labels': ['cld_opd_dcomp', 'cld_reff_dcomp']}
INFO - 2022-03-01 13:26:16,070 [trainer.py:40] : Trainer: Training on sites [0, 1, 2, 3, 4, 5, 6, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:26:16,070 [trainer.py:49] : Trainer: Training on sites [0, 1, 2, 3, 4, 5, 6, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:26:16,070 [data_handlers.py:60] : Loading training data
DEBUG - 2022-03-01 13:26:16,070 [data_handlers.py:78] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:26:16,070 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5
DEBUG - 2022-03-01 13:26:17,173 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:26:17,176 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:26:17,177 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:26:17,181 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:26:17,900 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:17,903 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:26:18,586 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:18,590 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:26:19,274 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:19,277 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:26:19,982 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:19,985 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:26:20,663 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:20,666 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:26:21,373 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:21,376 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:26:22,129 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:22,132 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:26:22,854 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:22,854 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5
DEBUG - 2022-03-01 13:26:23,892 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:26:23,896 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:26:23,896 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:26:23,899 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:26:24,530 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:24,533 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:26:25,164 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:25,167 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:26:25,797 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:25,800 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:26:26,436 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:26,439 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:26:27,066 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:27,069 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:26:27,699 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:27,702 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:26:28,332 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:28,336 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2022-03-01 13:26:28,966 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:28,966 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5
DEBUG - 2022-03-01 13:26:29,955 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:26:29,960 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:26:29,960 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:26:29,963 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:26:30,651 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:30,654 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:26:31,332 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:31,335 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:26:31,998 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:32,001 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:26:32,703 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:32,706 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:26:33,360 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:33,363 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:26:34,108 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:34,111 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:26:34,774 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:34,777 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:26:35,490 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:35,490 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5
DEBUG - 2022-03-01 13:26:36,483 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:26:36,486 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:26:36,486 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:26:36,490 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:26:37,115 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:37,118 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:26:37,730 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:37,733 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:26:38,345 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:38,348 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:26:38,963 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:38,966 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:26:39,578 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:39,581 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:26:40,211 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:40,214 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:26:40,825 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:40,828 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2022-03-01 13:26:41,444 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:41,444 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5
DEBUG - 2022-03-01 13:26:47,049 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:26:47,065 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:26:47,066 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:26:47,069 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:26:47,879 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:47,882 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:26:48,565 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:48,568 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:26:49,245 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:49,248 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:26:49,927 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:49,930 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:26:50,596 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:50,599 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:26:51,330 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:51,333 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:26:52,015 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:52,018 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:26:52,779 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:26:52,779 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5
DEBUG - 2022-03-01 13:26:53,930 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:26:53,933 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:26:53,933 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:26:53,937 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:26:54,554 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:54,557 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:26:55,184 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:55,188 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:26:55,816 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:55,819 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:26:56,436 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:56,439 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:26:57,068 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:57,071 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:26:57,691 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:57,694 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:26:58,323 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:58,326 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2022-03-01 13:26:58,958 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:26:58,958 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5
DEBUG - 2022-03-01 13:27:04,659 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:27:04,676 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:27:04,676 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:27:04,679 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:27:05,395 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:05,398 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:27:06,147 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:06,150 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:27:06,828 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:06,831 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:27:07,511 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:07,514 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:27:08,210 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:08,214 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:27:08,904 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:08,908 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:27:09,630 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:09,634 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:27:10,403 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:10,404 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 8], from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5
DEBUG - 2022-03-01 13:27:13,824 [data_handlers.py:103] : 	Shape temp_raw=(420480, 19), temp_all_sky=(420480, 14)
DEBUG - 2022-03-01 13:27:13,835 [data_handlers.py:106] : 	Time step is 10 minutes
DEBUG - 2022-03-01 13:27:13,835 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 6, 8]
DEBUG - 2022-03-01 13:27:13,838 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:27:14,495 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:14,498 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:27:15,158 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:15,162 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:27:15,819 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:15,823 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:27:16,482 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:16,485 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:27:17,145 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:17,149 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:27:17,805 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:17,809 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:27:18,465 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:18,469 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2022-03-01 13:27:19,134 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:19,135 [data_handlers.py:136] : Data load complete. Shape df_raw=(2803968, 19), df_all_sky=(2803968, 14), df_surf=(2803968, 5)
DEBUG - 2022-03-01 13:27:19,957 [data_handlers.py:159] : Extracting 2D arrays to run rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:27:31,217 [data_handlers.py:176] : Running rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:29:19,543 [data_handlers.py:194] : Completed rest2 run for clearsky PhyGNN inputs.
INFO - 2022-03-01 13:29:21,026 [data_handlers.py:62] : Prepping training data
DEBUG - 2022-03-01 13:29:21,026 [data_handlers.py:214] : Training data clean kwargs: {'filter_daylight': True, 'filter_clear': False, 'add_cloud_flag': True, 'sza_lim': 89, 'nan_option': 'interp'}
DEBUG - 2022-03-01 13:29:21,026 [data_handlers.py:215] : Shape before cleaning: df_raw=(2803968, 19)
INFO - 2022-03-01 13:29:21,302 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 13:29:21,306 [data_cleaners.py:38] : 53.35% of daylight timesteps are cloudy
INFO - 2022-03-01 13:29:21,311 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:29:21,316 [data_cleaners.py:42] : 34.72% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:29:21,320 [data_cleaners.py:44] : 34.93% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:29:21,320 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:29:21,323 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,327 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,331 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,335 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,338 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.62% NaN values
DEBUG - 2022-03-01 13:29:21,342 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.62% NaN values
DEBUG - 2022-03-01 13:29:21,345 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.62% NaN values
DEBUG - 2022-03-01 13:29:21,349 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:29:21,352 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:29:21,355 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.61% NaN values
DEBUG - 2022-03-01 13:29:21,359 [data_cleaners.py:50] : 	"cloud_probability" has 3.61% NaN values
DEBUG - 2022-03-01 13:29:21,362 [data_cleaners.py:50] : 	"cloud_fraction" has 3.61% NaN values
DEBUG - 2022-03-01 13:29:21,366 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,369 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,372 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,376 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,379 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:21,383 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.70% NaN values
DEBUG - 2022-03-01 13:29:21,386 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.75% NaN values
DEBUG - 2022-03-01 13:29:21,386 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:29:24,278 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:29:24,550 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393004 after filters (49.68% of original)
DEBUG - 2022-03-01 13:29:24,661 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:29:24,661 [data_cleaners.py:107] : Cleaning took 3.6 seconds
DEBUG - 2022-03-01 13:29:24,661 [data_handlers.py:218] : Shape after cleaning: df_train=(1393004, 20)
DEBUG - 2022-03-01 13:29:24,661 [data_handlers.py:221] : Cleaning df_all_sky training data (for pfun).
DEBUG - 2022-03-01 13:29:24,662 [data_handlers.py:222] : Shape before cleaning: df_all_sky=(2803968, 25)
INFO - 2022-03-01 13:29:25,003 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 13:29:25,007 [data_cleaners.py:38] : 53.35% of daylight timesteps are cloudy
INFO - 2022-03-01 13:29:25,012 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:29:25,017 [data_cleaners.py:42] : 34.72% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:29:25,021 [data_cleaners.py:44] : 34.93% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:29:25,021 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:29:25,024 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,028 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,031 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,035 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,040 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,043 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.70% NaN values
DEBUG - 2022-03-01 13:29:25,047 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.75% NaN values
DEBUG - 2022-03-01 13:29:25,050 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,054 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,057 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,060 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,063 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,067 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,071 [data_cleaners.py:50] : 	"surfrad_dhi" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,075 [data_cleaners.py:50] : 	"surfrad_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,080 [data_cleaners.py:50] : 	"surfrad_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,082 [data_cleaners.py:50] : 	"doy" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,086 [data_cleaners.py:50] : 	"radius" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,091 [data_cleaners.py:50] : 	"Tuuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,095 [data_cleaners.py:50] : 	"clearsky_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,099 [data_cleaners.py:50] : 	"clearsky_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,104 [data_cleaners.py:50] : 	"Ruuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,108 [data_cleaners.py:50] : 	"Tddclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,112 [data_cleaners.py:50] : 	"Tduclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,117 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:29:25,117 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:29:27,377 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:29:27,626 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393004 after filters (49.68% of original)
DEBUG - 2022-03-01 13:29:27,779 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:29:27,780 [data_cleaners.py:107] : Cleaning took 3.1 seconds
DEBUG - 2022-03-01 13:29:27,780 [data_handlers.py:226] : Shape after cleaning: df_all_sky=(1393004, 26)
DEBUG - 2022-03-01 13:29:27,861 [data_handlers.py:240] : **Shape: df_train=(1393004, 17)
DEBUG - 2022-03-01 13:29:27,888 [data_handlers.py:250] : Shapes: x=(1393004, 15), y=(1393004, 2), p=(1393004, 26)
DEBUG - 2022-03-01 13:29:27,888 [data_handlers.py:253] : Training features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
DEBUG - 2022-03-01 13:29:27,888 [trainer.py:67] : Building PHYGNN model
INFO - 2022-03-01 13:29:27,888 [trainer.py:70] : Using p_fun: <function p_fun_all_sky at 0x2b732af378b0>
INFO - 2022-03-01 13:29:27,889 [base.py:152] : Active python environment versions: 
{   'numpy': '1.22.2',
    'pandas': '1.2.4',
    'phygnn': '0.0.14',
    'python': '3.8.8 (default, Feb 24 2021, 21:46:12) \n[GCC 7.3.0]',
    'sklearn': '0.24.1',
    'tensorflow': '2.8.0'}
INFO - 2022-03-01 13:29:27,905 [base.py:111] : Successfully initialized model with 17 layers
INFO - 2022-03-01 13:29:27,906 [trainer.py:84] : Training part A - pure data. Loss is [1, 0]
2022-03-01 13:29:36.316414: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:29:36.316623: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-03-01 13:29:36.316644: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (r7i6n14): /proc/driver/nvidia/version does not exist
2022-03-01 13:29:36.317041: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO - 2022-03-01 13:29:43,987 [phygnn.py:576] : Epoch 0 train loss: 7.00e-01 val loss: 6.92e-01 for "phygnn"
INFO - 2022-03-01 13:29:51,674 [phygnn.py:576] : Epoch 1 train loss: 6.47e-01 val loss: 6.35e-01 for "phygnn"
INFO - 2022-03-01 13:29:59,266 [phygnn.py:576] : Epoch 2 train loss: 5.70e-01 val loss: 5.62e-01 for "phygnn"
INFO - 2022-03-01 13:30:06,974 [phygnn.py:576] : Epoch 3 train loss: 5.49e-01 val loss: 5.28e-01 for "phygnn"
INFO - 2022-03-01 13:30:14,739 [phygnn.py:576] : Epoch 4 train loss: 5.24e-01 val loss: 5.07e-01 for "phygnn"
INFO - 2022-03-01 13:30:22,679 [phygnn.py:576] : Epoch 5 train loss: 5.07e-01 val loss: 4.97e-01 for "phygnn"
INFO - 2022-03-01 13:30:30,567 [phygnn.py:576] : Epoch 6 train loss: 5.08e-01 val loss: 4.91e-01 for "phygnn"
INFO - 2022-03-01 13:30:38,306 [phygnn.py:576] : Epoch 7 train loss: 5.03e-01 val loss: 4.85e-01 for "phygnn"
INFO - 2022-03-01 13:30:46,190 [phygnn.py:576] : Epoch 8 train loss: 4.96e-01 val loss: 4.86e-01 for "phygnn"
INFO - 2022-03-01 13:30:54,089 [phygnn.py:576] : Epoch 9 train loss: 4.94e-01 val loss: 4.82e-01 for "phygnn"
INFO - 2022-03-01 13:31:02,074 [phygnn.py:576] : Epoch 10 train loss: 4.88e-01 val loss: 4.78e-01 for "phygnn"
INFO - 2022-03-01 13:31:10,080 [phygnn.py:576] : Epoch 11 train loss: 4.86e-01 val loss: 4.76e-01 for "phygnn"
INFO - 2022-03-01 13:31:17,949 [phygnn.py:576] : Epoch 12 train loss: 4.89e-01 val loss: 4.73e-01 for "phygnn"
INFO - 2022-03-01 13:31:25,765 [phygnn.py:576] : Epoch 13 train loss: 4.88e-01 val loss: 4.71e-01 for "phygnn"
INFO - 2022-03-01 13:31:33,584 [phygnn.py:576] : Epoch 14 train loss: 4.82e-01 val loss: 4.68e-01 for "phygnn"
INFO - 2022-03-01 13:31:41,500 [phygnn.py:576] : Epoch 15 train loss: 4.81e-01 val loss: 4.68e-01 for "phygnn"
INFO - 2022-03-01 13:31:49,414 [phygnn.py:576] : Epoch 16 train loss: 4.76e-01 val loss: 4.65e-01 for "phygnn"
INFO - 2022-03-01 13:31:57,074 [phygnn.py:576] : Epoch 17 train loss: 4.73e-01 val loss: 4.64e-01 for "phygnn"
INFO - 2022-03-01 13:32:04,658 [phygnn.py:576] : Epoch 18 train loss: 4.73e-01 val loss: 4.63e-01 for "phygnn"
INFO - 2022-03-01 13:32:12,618 [phygnn.py:576] : Epoch 19 train loss: 4.71e-01 val loss: 4.61e-01 for "phygnn"
INFO - 2022-03-01 13:32:20,631 [phygnn.py:576] : Epoch 20 train loss: 4.69e-01 val loss: 4.60e-01 for "phygnn"
INFO - 2022-03-01 13:32:28,730 [phygnn.py:576] : Epoch 21 train loss: 4.70e-01 val loss: 4.58e-01 for "phygnn"
INFO - 2022-03-01 13:32:36,564 [phygnn.py:576] : Epoch 22 train loss: 4.70e-01 val loss: 4.59e-01 for "phygnn"
INFO - 2022-03-01 13:32:44,609 [phygnn.py:576] : Epoch 23 train loss: 4.65e-01 val loss: 4.59e-01 for "phygnn"
INFO - 2022-03-01 13:32:52,665 [phygnn.py:576] : Epoch 24 train loss: 4.68e-01 val loss: 4.58e-01 for "phygnn"
INFO - 2022-03-01 13:33:00,649 [phygnn.py:576] : Epoch 25 train loss: 4.65e-01 val loss: 4.55e-01 for "phygnn"
INFO - 2022-03-01 13:33:08,754 [phygnn.py:576] : Epoch 26 train loss: 4.63e-01 val loss: 4.52e-01 for "phygnn"
INFO - 2022-03-01 13:33:16,747 [phygnn.py:576] : Epoch 27 train loss: 4.60e-01 val loss: 4.50e-01 for "phygnn"
INFO - 2022-03-01 13:33:24,877 [phygnn.py:576] : Epoch 28 train loss: 4.57e-01 val loss: 4.48e-01 for "phygnn"
INFO - 2022-03-01 13:33:32,918 [phygnn.py:576] : Epoch 29 train loss: 4.58e-01 val loss: 4.49e-01 for "phygnn"
INFO - 2022-03-01 13:33:41,067 [phygnn.py:576] : Epoch 30 train loss: 4.60e-01 val loss: 4.46e-01 for "phygnn"
INFO - 2022-03-01 13:33:49,156 [phygnn.py:576] : Epoch 31 train loss: 4.54e-01 val loss: 4.44e-01 for "phygnn"
INFO - 2022-03-01 13:33:57,243 [phygnn.py:576] : Epoch 32 train loss: 4.54e-01 val loss: 4.42e-01 for "phygnn"
INFO - 2022-03-01 13:34:05,352 [phygnn.py:576] : Epoch 33 train loss: 4.61e-01 val loss: 4.45e-01 for "phygnn"
INFO - 2022-03-01 13:34:13,445 [phygnn.py:576] : Epoch 34 train loss: 4.52e-01 val loss: 4.41e-01 for "phygnn"
INFO - 2022-03-01 13:34:21,675 [phygnn.py:576] : Epoch 35 train loss: 4.56e-01 val loss: 4.41e-01 for "phygnn"
INFO - 2022-03-01 13:34:29,852 [phygnn.py:576] : Epoch 36 train loss: 4.48e-01 val loss: 4.39e-01 for "phygnn"
INFO - 2022-03-01 13:34:37,889 [phygnn.py:576] : Epoch 37 train loss: 4.48e-01 val loss: 4.38e-01 for "phygnn"
INFO - 2022-03-01 13:34:45,849 [phygnn.py:576] : Epoch 38 train loss: 4.50e-01 val loss: 4.38e-01 for "phygnn"
INFO - 2022-03-01 13:34:53,909 [phygnn.py:576] : Epoch 39 train loss: 4.55e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2022-03-01 13:35:02,081 [phygnn.py:576] : Epoch 40 train loss: 4.53e-01 val loss: 4.39e-01 for "phygnn"
INFO - 2022-03-01 13:35:10,121 [phygnn.py:576] : Epoch 41 train loss: 4.46e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2022-03-01 13:35:18,248 [phygnn.py:576] : Epoch 42 train loss: 4.44e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2022-03-01 13:35:26,071 [phygnn.py:576] : Epoch 43 train loss: 4.43e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2022-03-01 13:35:34,137 [phygnn.py:576] : Epoch 44 train loss: 4.46e-01 val loss: 4.34e-01 for "phygnn"
INFO - 2022-03-01 13:35:42,358 [phygnn.py:576] : Epoch 45 train loss: 4.48e-01 val loss: 4.36e-01 for "phygnn"
INFO - 2022-03-01 13:35:50,547 [phygnn.py:576] : Epoch 46 train loss: 4.48e-01 val loss: 4.32e-01 for "phygnn"
INFO - 2022-03-01 13:35:58,465 [phygnn.py:576] : Epoch 47 train loss: 4.49e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:36:06,678 [phygnn.py:576] : Epoch 48 train loss: 4.47e-01 val loss: 4.32e-01 for "phygnn"
INFO - 2022-03-01 13:36:14,710 [phygnn.py:576] : Epoch 49 train loss: 4.42e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:36:22,645 [phygnn.py:576] : Epoch 50 train loss: 4.43e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:36:30,554 [phygnn.py:576] : Epoch 51 train loss: 4.40e-01 val loss: 4.29e-01 for "phygnn"
INFO - 2022-03-01 13:36:38,602 [phygnn.py:576] : Epoch 52 train loss: 4.42e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:36:46,723 [phygnn.py:576] : Epoch 53 train loss: 4.41e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:36:54,692 [phygnn.py:576] : Epoch 54 train loss: 4.42e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:37:02,777 [phygnn.py:576] : Epoch 55 train loss: 4.32e-01 val loss: 4.28e-01 for "phygnn"
INFO - 2022-03-01 13:37:10,969 [phygnn.py:576] : Epoch 56 train loss: 4.38e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:37:19,046 [phygnn.py:576] : Epoch 57 train loss: 4.39e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2022-03-01 13:37:27,208 [phygnn.py:576] : Epoch 58 train loss: 4.35e-01 val loss: 4.25e-01 for "phygnn"
INFO - 2022-03-01 13:37:35,331 [phygnn.py:576] : Epoch 59 train loss: 4.38e-01 val loss: 4.24e-01 for "phygnn"
INFO - 2022-03-01 13:37:43,300 [phygnn.py:576] : Epoch 60 train loss: 4.40e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:37:51,543 [phygnn.py:576] : Epoch 61 train loss: 4.35e-01 val loss: 4.24e-01 for "phygnn"
INFO - 2022-03-01 13:37:59,753 [phygnn.py:576] : Epoch 62 train loss: 4.43e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:38:07,716 [phygnn.py:576] : Epoch 63 train loss: 4.34e-01 val loss: 4.24e-01 for "phygnn"
INFO - 2022-03-01 13:38:15,547 [phygnn.py:576] : Epoch 64 train loss: 4.29e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:38:23,569 [phygnn.py:576] : Epoch 65 train loss: 4.33e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:38:31,778 [phygnn.py:576] : Epoch 66 train loss: 4.32e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:38:39,819 [phygnn.py:576] : Epoch 67 train loss: 4.34e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:38:47,801 [phygnn.py:576] : Epoch 68 train loss: 4.30e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:38:56,011 [phygnn.py:576] : Epoch 69 train loss: 4.33e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2022-03-01 13:39:04,260 [phygnn.py:576] : Epoch 70 train loss: 4.31e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2022-03-01 13:39:12,431 [phygnn.py:576] : Epoch 71 train loss: 4.30e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:39:20,518 [phygnn.py:576] : Epoch 72 train loss: 4.36e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:39:28,787 [phygnn.py:576] : Epoch 73 train loss: 4.25e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2022-03-01 13:39:37,046 [phygnn.py:576] : Epoch 74 train loss: 4.33e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:39:45,170 [phygnn.py:576] : Epoch 75 train loss: 4.31e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:39:53,196 [phygnn.py:576] : Epoch 76 train loss: 4.17e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:40:01,318 [phygnn.py:576] : Epoch 77 train loss: 4.26e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:40:09,313 [phygnn.py:576] : Epoch 78 train loss: 4.23e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:40:17,151 [phygnn.py:576] : Epoch 79 train loss: 4.26e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:40:25,245 [phygnn.py:576] : Epoch 80 train loss: 4.26e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:40:33,495 [phygnn.py:576] : Epoch 81 train loss: 4.37e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:40:41,427 [phygnn.py:576] : Epoch 82 train loss: 4.26e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:40:49,579 [phygnn.py:576] : Epoch 83 train loss: 4.25e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:40:57,731 [phygnn.py:576] : Epoch 84 train loss: 4.29e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2022-03-01 13:41:05,982 [phygnn.py:576] : Epoch 85 train loss: 4.30e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:41:14,225 [phygnn.py:576] : Epoch 86 train loss: 4.24e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:41:22,427 [phygnn.py:576] : Epoch 87 train loss: 4.27e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:41:30,313 [phygnn.py:576] : Epoch 88 train loss: 4.22e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:41:38,372 [phygnn.py:576] : Epoch 89 train loss: 4.24e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:41:46,473 [phygnn.py:576] : Epoch 90 train loss: 4.25e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:41:54,554 [phygnn.py:576] : Epoch 91 train loss: 4.24e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:42:02,351 [phygnn.py:576] : Epoch 92 train loss: 4.22e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:42:10,203 [phygnn.py:576] : Epoch 93 train loss: 4.27e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:42:17,980 [phygnn.py:576] : Epoch 94 train loss: 4.24e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:42:25,789 [phygnn.py:576] : Epoch 95 train loss: 4.22e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:42:33,915 [phygnn.py:576] : Epoch 96 train loss: 4.27e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:42:41,975 [phygnn.py:576] : Epoch 97 train loss: 4.24e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:42:50,114 [phygnn.py:576] : Epoch 98 train loss: 4.21e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:42:58,188 [phygnn.py:576] : Epoch 99 train loss: 4.20e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:42:59,013 [trainer.py:92] : Training part B - data and phygnn. Loss is [0.5, 0.5]
INFO - 2022-03-01 13:43:19,822 [phygnn.py:576] : Epoch 100 train loss: 2.93e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:43:33,157 [phygnn.py:576] : Epoch 101 train loss: 2.90e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:43:46,052 [phygnn.py:576] : Epoch 102 train loss: 2.91e-01 val loss: 2.83e-01 for "phygnn"
INFO - 2022-03-01 13:43:59,353 [phygnn.py:576] : Epoch 103 train loss: 2.96e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:44:12,775 [phygnn.py:576] : Epoch 104 train loss: 2.91e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:44:26,184 [phygnn.py:576] : Epoch 105 train loss: 2.95e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:44:39,761 [phygnn.py:576] : Epoch 106 train loss: 2.91e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:44:53,045 [phygnn.py:576] : Epoch 107 train loss: 2.91e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:45:05,638 [phygnn.py:576] : Epoch 108 train loss: 2.92e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:45:18,742 [phygnn.py:576] : Epoch 109 train loss: 2.94e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:45:31,671 [phygnn.py:576] : Epoch 110 train loss: 2.90e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:45:44,590 [phygnn.py:576] : Epoch 111 train loss: 2.89e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:45:57,801 [phygnn.py:576] : Epoch 112 train loss: 2.88e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:46:11,058 [phygnn.py:576] : Epoch 113 train loss: 2.90e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:46:23,938 [phygnn.py:576] : Epoch 114 train loss: 2.89e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:46:37,250 [phygnn.py:576] : Epoch 115 train loss: 2.94e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:46:51,062 [phygnn.py:576] : Epoch 116 train loss: 2.92e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:47:04,442 [phygnn.py:576] : Epoch 117 train loss: 2.90e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:47:18,215 [phygnn.py:576] : Epoch 118 train loss: 2.93e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:47:30,617 [phygnn.py:576] : Epoch 119 train loss: 2.88e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:47:42,804 [phygnn.py:576] : Epoch 120 train loss: 2.86e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:47:55,510 [phygnn.py:576] : Epoch 121 train loss: 2.89e-01 val loss: 2.80e-01 for "phygnn"
INFO - 2022-03-01 13:48:08,854 [phygnn.py:576] : Epoch 122 train loss: 2.90e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:48:22,135 [phygnn.py:576] : Epoch 123 train loss: 2.88e-01 val loss: 2.81e-01 for "phygnn"
INFO - 2022-03-01 13:48:35,907 [phygnn.py:576] : Epoch 124 train loss: 2.89e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:48:49,028 [phygnn.py:576] : Epoch 125 train loss: 2.89e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:49:02,698 [phygnn.py:576] : Epoch 126 train loss: 2.84e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:49:16,443 [phygnn.py:576] : Epoch 127 train loss: 2.88e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:49:29,336 [phygnn.py:576] : Epoch 128 train loss: 2.90e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:49:42,396 [phygnn.py:576] : Epoch 129 train loss: 2.87e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:49:55,610 [phygnn.py:576] : Epoch 130 train loss: 2.89e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:50:07,828 [phygnn.py:576] : Epoch 131 train loss: 2.85e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:50:21,090 [phygnn.py:576] : Epoch 132 train loss: 2.88e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:50:34,562 [phygnn.py:576] : Epoch 133 train loss: 2.88e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:50:47,955 [phygnn.py:576] : Epoch 134 train loss: 2.87e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:51:00,333 [phygnn.py:576] : Epoch 135 train loss: 2.89e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:51:12,576 [phygnn.py:576] : Epoch 136 train loss: 2.87e-01 val loss: 2.79e-01 for "phygnn"
INFO - 2022-03-01 13:51:25,916 [phygnn.py:576] : Epoch 137 train loss: 2.88e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:51:39,397 [phygnn.py:576] : Epoch 138 train loss: 2.89e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:51:51,756 [phygnn.py:576] : Epoch 139 train loss: 2.86e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:52:04,548 [phygnn.py:576] : Epoch 140 train loss: 2.84e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:52:17,445 [phygnn.py:576] : Epoch 141 train loss: 2.87e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:52:31,053 [phygnn.py:576] : Epoch 142 train loss: 2.87e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:52:44,143 [phygnn.py:576] : Epoch 143 train loss: 2.85e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:52:56,639 [phygnn.py:576] : Epoch 144 train loss: 2.85e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:53:09,614 [phygnn.py:576] : Epoch 145 train loss: 2.85e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:53:22,726 [phygnn.py:576] : Epoch 146 train loss: 2.89e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:53:35,605 [phygnn.py:576] : Epoch 147 train loss: 2.91e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:53:49,160 [phygnn.py:576] : Epoch 148 train loss: 2.86e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:54:01,880 [phygnn.py:576] : Epoch 149 train loss: 2.86e-01 val loss: 2.78e-01 for "phygnn"
INFO - 2022-03-01 13:54:15,470 [phygnn.py:576] : Epoch 150 train loss: 2.85e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:54:28,545 [phygnn.py:576] : Epoch 151 train loss: 2.84e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:54:41,976 [phygnn.py:576] : Epoch 152 train loss: 2.84e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:54:55,523 [phygnn.py:576] : Epoch 153 train loss: 2.82e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:55:09,198 [phygnn.py:576] : Epoch 154 train loss: 2.83e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:55:22,227 [phygnn.py:576] : Epoch 155 train loss: 2.88e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:55:34,515 [phygnn.py:576] : Epoch 156 train loss: 2.84e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:55:47,984 [phygnn.py:576] : Epoch 157 train loss: 2.87e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:56:00,151 [phygnn.py:576] : Epoch 158 train loss: 2.81e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:56:13,315 [phygnn.py:576] : Epoch 159 train loss: 2.83e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:56:26,939 [phygnn.py:576] : Epoch 160 train loss: 2.85e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:56:39,533 [phygnn.py:576] : Epoch 161 train loss: 2.87e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:56:52,703 [phygnn.py:576] : Epoch 162 train loss: 2.82e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:57:04,857 [phygnn.py:576] : Epoch 163 train loss: 2.83e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:57:18,490 [phygnn.py:576] : Epoch 164 train loss: 2.85e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:57:31,824 [phygnn.py:576] : Epoch 165 train loss: 2.87e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:57:44,739 [phygnn.py:576] : Epoch 166 train loss: 2.87e-01 val loss: 2.77e-01 for "phygnn"
INFO - 2022-03-01 13:57:58,237 [phygnn.py:576] : Epoch 167 train loss: 2.87e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:58:11,598 [phygnn.py:576] : Epoch 168 train loss: 2.84e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:58:24,456 [phygnn.py:576] : Epoch 169 train loss: 2.83e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:58:36,906 [phygnn.py:576] : Epoch 170 train loss: 2.84e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:58:49,780 [phygnn.py:576] : Epoch 171 train loss: 2.81e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:59:03,398 [phygnn.py:576] : Epoch 172 train loss: 2.83e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:59:16,346 [phygnn.py:576] : Epoch 173 train loss: 2.83e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:59:28,396 [phygnn.py:576] : Epoch 174 train loss: 2.86e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:59:41,913 [phygnn.py:576] : Epoch 175 train loss: 2.83e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:59:54,430 [phygnn.py:576] : Epoch 176 train loss: 2.85e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:00:06,508 [phygnn.py:576] : Epoch 177 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:00:18,669 [phygnn.py:576] : Epoch 178 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:00:31,845 [phygnn.py:576] : Epoch 179 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:00:44,795 [phygnn.py:576] : Epoch 180 train loss: 2.81e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:00:58,251 [phygnn.py:576] : Epoch 181 train loss: 2.82e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 14:01:11,046 [phygnn.py:576] : Epoch 182 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:01:23,321 [phygnn.py:576] : Epoch 183 train loss: 2.83e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:01:36,682 [phygnn.py:576] : Epoch 184 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:01:49,096 [phygnn.py:576] : Epoch 185 train loss: 2.83e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:02:01,270 [phygnn.py:576] : Epoch 186 train loss: 2.81e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:02:14,689 [phygnn.py:576] : Epoch 187 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:02:27,192 [phygnn.py:576] : Epoch 188 train loss: 2.81e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:02:39,906 [phygnn.py:576] : Epoch 189 train loss: 2.85e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:02:53,204 [phygnn.py:576] : Epoch 190 train loss: 2.85e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 14:03:05,617 [phygnn.py:576] : Epoch 191 train loss: 2.83e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:03:18,362 [phygnn.py:576] : Epoch 192 train loss: 2.80e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:03:31,519 [phygnn.py:576] : Epoch 193 train loss: 2.79e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 14:03:43,503 [phygnn.py:576] : Epoch 194 train loss: 2.83e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:03:55,638 [phygnn.py:576] : Epoch 195 train loss: 2.80e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:04:08,381 [phygnn.py:576] : Epoch 196 train loss: 2.83e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:04:20,923 [phygnn.py:576] : Epoch 197 train loss: 2.83e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 14:04:33,194 [phygnn.py:576] : Epoch 198 train loss: 2.82e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 14:04:45,444 [phygnn.py:576] : Epoch 199 train loss: 2.79e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 14:04:46,182 [trainer.py:102] : Training complete
INFO - 2022-03-01 14:04:46,226 [base.py:496] : Saved model to: /home/gbuster/code/mlclouds/mlclouds/model/k_fold/outputs/model_7.pkl
DEBUG - 2022-03-01 14:04:46,226 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:04:46,226 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:04:46,231 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:47,281 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:47,281 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:48,320 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:48,320 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:49,408 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:49,408 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:50,469 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:50,469 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:56,711 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:04:56,711 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:04:57,910 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:04:57,911 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:04,327 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:05:04,328 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:07,737 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 14:05:07,737 [data_handlers.py:413] : Shape df_raw=(3154464, 19), df_all_sky=(3154464, 14)
DEBUG - 2022-03-01 14:05:07,737 [data_handlers.py:420] : Shape after reset_index: df_raw=(3154464, 19), df_all_sky=(3154464, 14)
INFO - 2022-03-01 14:05:08,055 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:05:08,060 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 14:05:08,065 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:05:08,071 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:05:08,076 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:05:08,076 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:05:08,079 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,085 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,089 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,093 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,097 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.45% NaN values
DEBUG - 2022-03-01 14:05:08,101 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.45% NaN values
DEBUG - 2022-03-01 14:05:08,105 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.28% NaN values
DEBUG - 2022-03-01 14:05:08,109 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 14:05:08,113 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.19% NaN values
DEBUG - 2022-03-01 14:05:08,117 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.27% NaN values
DEBUG - 2022-03-01 14:05:08,120 [data_cleaners.py:50] : 	"cloud_probability" has 3.27% NaN values
DEBUG - 2022-03-01 14:05:08,124 [data_cleaners.py:50] : 	"cloud_fraction" has 3.27% NaN values
DEBUG - 2022-03-01 14:05:08,128 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,132 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,136 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,140 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,143 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:08,147 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 14:05:08,151 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 14:05:08,151 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:05:11,514 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:05:11,885 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:05:11,885 [data_cleaners.py:107] : Cleaning took 4.1 seconds
INFO - 2022-03-01 14:05:12,199 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:05:12,205 [data_cleaners.py:38] : 52.32% of daylight timesteps are cloudy
INFO - 2022-03-01 14:05:12,210 [data_cleaners.py:40] : 3.25% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:05:12,215 [data_cleaners.py:42] : 34.00% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:05:12,220 [data_cleaners.py:44] : 34.22% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:05:12,220 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:05:12,223 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,228 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,232 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,236 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,240 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,245 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,249 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.84% NaN values
DEBUG - 2022-03-01 14:05:12,252 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.90% NaN values
DEBUG - 2022-03-01 14:05:12,256 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,260 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,264 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,268 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,271 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,275 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:05:12,275 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:05:14,184 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:05:14,522 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:05:14,522 [data_cleaners.py:107] : Cleaning took 2.6 seconds
DEBUG - 2022-03-01 14:05:14,523 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:05:14,559 [data_handlers.py:463] : Mask: shape=(3154464,), sum=1567353
DEBUG - 2022-03-01 14:05:14,700 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:05:14,700 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:05:18,113 [validator.py:110] : Predicted data shape is (1567353, 2)
DEBUG - 2022-03-01 14:05:18,526 [validator.py:158] : shapes: df_feature_val=(3154464, 20), df_all_sky_val=(3154464, 15)
INFO - 2022-03-01 14:05:18,782 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:05:18,785 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:05:18,785 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:05:18,785 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:18,835 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:05:19,520 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:19,557 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:05:20,234 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:20,272 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:05:20,950 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:20,985 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:05:21,673 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:21,711 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:05:22,388 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:22,426 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:05:23,120 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:23,157 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:05:23,835 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:23,870 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:05:24,544 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:24,579 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:05:25,283 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:05:25,284 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:25,330 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:05:25,957 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:25,992 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:05:26,614 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:26,649 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:05:27,271 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:27,306 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:05:27,935 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:27,970 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:05:28,589 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:28,623 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:05:29,244 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:29,279 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:05:29,899 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:29,934 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:05:30,552 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:30,589 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:05:31,208 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:05:31,209 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:31,256 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:05:31,939 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:31,973 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:05:32,647 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:32,682 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:05:33,368 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:33,403 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:05:34,077 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:34,113 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:05:34,791 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:34,826 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:05:35,512 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:35,547 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:05:36,213 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:36,249 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:05:36,910 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:36,946 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:05:37,644 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:05:37,644 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:37,691 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:05:38,320 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:38,355 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:05:38,977 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:39,013 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:05:39,630 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:39,665 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:05:40,287 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:40,322 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:05:40,947 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:40,982 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:05:41,610 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:41,645 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:05:42,261 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:42,295 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:05:42,917 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:42,951 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:05:43,569 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:05:43,570 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:43,881 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:05:44,571 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:44,681 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:05:45,367 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:45,477 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:05:46,170 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:46,281 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:05:46,977 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:47,087 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:05:47,777 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:47,888 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:05:48,575 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:48,687 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:05:49,375 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:49,484 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:05:50,196 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:50,305 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:05:51,042 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:05:51,042 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:51,087 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:05:51,717 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:51,751 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:05:52,381 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:52,416 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:05:53,047 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:53,082 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:05:53,712 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:05:53,746 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:05:54,370 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:05:54,404 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:05:55,035 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:05:55,069 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:05:55,693 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:05:55,727 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:05:56,365 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:05:56,400 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:05:57,033 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:05:57,033 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:05:57,345 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:05:58,056 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:05:58,167 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:05:58,876 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:05:58,988 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:05:59,691 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:05:59,801 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:06:00,509 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:00,625 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:06:01,337 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:01,448 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:06:02,165 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:02,276 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:06:02,998 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:03,109 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:06:03,813 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:03,927 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:06:04,694 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:06:04,695 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:06:04,790 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:06:05,455 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:06:05,520 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:06:06,178 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:06:06,244 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:06:06,906 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:06:06,971 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:06:07,627 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:06:07,693 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:06:08,353 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:06:08,418 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:06:09,081 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:06:09,147 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:06:09,816 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:06:09,880 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:06:10,557 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:06:10,622 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:06:11,296 [validator.py:187] : Shapes: df_base_full=(3154464, 6), df_surf_full=(3154464, 4)
DEBUG - 2022-03-01 14:06:11,300 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:06:11,332 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:06:26,270 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:06:26,302 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:06:41,141 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:06:41,173 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:06:55,970 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:06:56,002 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:07:10,849 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:07:10,880 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:07:25,825 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:07:25,857 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:07:40,796 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:07:40,828 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:07:55,741 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:07:55,774 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
DEBUG - 2022-03-01 14:08:10,684 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:08:10,716 [validator.py:209] : Shapes: df_baseline=(350496, 6), df_surf=(350496, 4)
INFO - 2022-03-01 14:08:25,575 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:08:25,584 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:08:25,585 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:08:25,587 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:26,590 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:08:26,590 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:27,660 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:08:27,660 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:33,920 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:08:33,921 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:40,445 [data_handlers.py:410] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2022-03-01 14:08:40,445 [data_handlers.py:413] : Shape df_raw=(2207952, 19), df_all_sky=(2207952, 14)
DEBUG - 2022-03-01 14:08:40,445 [data_handlers.py:420] : Shape after reset_index: df_raw=(2207952, 19), df_all_sky=(2207952, 14)
INFO - 2022-03-01 14:08:40,642 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:08:40,646 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:08:40,649 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:08:40,653 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:08:40,656 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:08:40,656 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:08:40,658 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,663 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,666 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,669 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,672 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 49.99% NaN values
DEBUG - 2022-03-01 14:08:40,675 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 49.99% NaN values
DEBUG - 2022-03-01 14:08:40,677 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 0.35% NaN values
DEBUG - 2022-03-01 14:08:40,680 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:08:40,683 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 0.26% NaN values
DEBUG - 2022-03-01 14:08:40,686 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 0.34% NaN values
DEBUG - 2022-03-01 14:08:40,688 [data_cleaners.py:50] : 	"cloud_probability" has 0.34% NaN values
DEBUG - 2022-03-01 14:08:40,691 [data_cleaners.py:50] : 	"cloud_fraction" has 0.34% NaN values
DEBUG - 2022-03-01 14:08:40,694 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,697 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,699 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,702 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,705 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:40,708 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:08:40,710 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:08:40,710 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:08:42,892 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:08:43,137 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:08:43,137 [data_cleaners.py:107] : Cleaning took 2.7 seconds
INFO - 2022-03-01 14:08:43,335 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 14:08:43,339 [data_cleaners.py:38] : 50.87% of daylight timesteps are cloudy
INFO - 2022-03-01 14:08:43,343 [data_cleaners.py:40] : 0.34% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:08:43,346 [data_cleaners.py:42] : 26.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:08:43,350 [data_cleaners.py:44] : 27.19% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:08:43,350 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:08:43,352 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,356 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,359 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,361 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,364 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,368 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,370 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2022-03-01 14:08:43,373 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2022-03-01 14:08:43,376 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,379 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,381 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,384 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,386 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,389 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:08:43,389 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:08:44,634 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:08:44,871 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'clear' 'bad_cloud']
INFO - 2022-03-01 14:08:44,871 [data_cleaners.py:107] : Cleaning took 1.7 seconds
DEBUG - 2022-03-01 14:08:44,872 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:08:44,889 [data_handlers.py:463] : Mask: shape=(2207952,), sum=1097157
DEBUG - 2022-03-01 14:08:44,957 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:08:44,958 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:08:47,267 [validator.py:110] : Predicted data shape is (1097157, 2)
DEBUG - 2022-03-01 14:08:47,541 [validator.py:158] : shapes: df_feature_val=(2207952, 20), df_all_sky_val=(2207952, 15)
INFO - 2022-03-01 14:08:47,720 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:08:47,723 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:08:47,723 [validator.py:346] : Loading data for 2016 / east
DEBUG - 2022-03-01 14:08:47,723 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:08:47,758 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:08:48,380 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:08:48,417 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:08:49,040 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:08:49,077 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:08:49,699 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:08:49,736 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:08:50,364 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:08:50,401 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:08:51,022 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:08:51,059 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:08:51,680 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:08:51,717 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:08:52,337 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:08:52,373 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:08:52,990 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:08:53,025 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:08:53,648 [validator.py:346] : Loading data for 2017 / east
DEBUG - 2022-03-01 14:08:53,648 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:08:53,685 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:08:54,308 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:08:54,345 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:08:54,963 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:08:55,000 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:08:55,616 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:08:55,651 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:08:56,271 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:08:56,305 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:08:56,921 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:08:56,956 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:08:57,581 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:08:57,616 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:08:58,235 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:08:58,270 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:08:58,894 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:08:58,929 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:08:59,547 [validator.py:346] : Loading data for 2018 / east
DEBUG - 2022-03-01 14:08:59,547 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:08:59,657 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:09:00,286 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:09:00,397 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:09:01,033 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:09:01,143 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:09:01,777 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:09:01,888 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:09:02,523 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:09:02,633 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:09:03,263 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:09:03,374 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:09:04,009 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:09:04,119 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:09:04,753 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:09:04,864 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:09:05,499 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:09:05,609 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:09:06,254 [validator.py:346] : Loading data for 2019 / east
DEBUG - 2022-03-01 14:09:06,254 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:09:06,363 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:09:07,009 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:09:07,119 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:09:07,760 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:09:07,870 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:09:08,514 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:09:08,624 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:09:09,265 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:09:09,376 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:09:10,034 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:09:10,143 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:09:10,789 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:09:10,898 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:09:11,563 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:09:11,673 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:09:12,329 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:09:12,440 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:09:13,115 [validator.py:187] : Shapes: df_base_full=(2207952, 6), df_surf_full=(2207952, 4)
DEBUG - 2022-03-01 14:09:13,119 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:09:13,140 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:09:23,867 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:09:23,889 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:09:34,533 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:09:34,556 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:09:45,126 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:09:45,148 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:09:55,796 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:09:55,818 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:10:06,524 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:10:06,545 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:10:17,256 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:10:17,278 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:10:27,962 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:10:27,984 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
DEBUG - 2022-03-01 14:10:38,678 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:10:38,700 [validator.py:209] : Shapes: df_baseline=(245328, 6), df_surf=(245328, 4)
INFO - 2022-03-01 14:10:49,338 [validator.py:292] : Finished computing stats.
DEBUG - 2022-03-01 14:10:49,364 [data_handlers.py:381] : Loading validation data
DEBUG - 2022-03-01 14:10:49,365 [data_handlers.py:387] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 14:10:49,367 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:10:50,444 [data_handlers.py:410] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:10:50,444 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:10:51,572 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:10:51,572 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:10:52,701 [data_handlers.py:410] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2022-03-01 14:10:52,702 [data_handlers.py:392] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:10:55,805 [data_handlers.py:410] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2022-03-01 14:10:55,805 [data_handlers.py:413] : Shape df_raw=(946512, 19), df_all_sky=(946512, 14)
DEBUG - 2022-03-01 14:10:55,805 [data_handlers.py:420] : Shape after reset_index: df_raw=(946512, 19), df_all_sky=(946512, 14)
INFO - 2022-03-01 14:10:55,895 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:10:55,897 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:10:55,898 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:10:55,900 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:10:55,902 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:10:55,902 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:10:55,903 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,905 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,906 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,908 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,909 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 54.84% NaN values
DEBUG - 2022-03-01 14:10:55,911 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 54.84% NaN values
DEBUG - 2022-03-01 14:10:55,912 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 10.12% NaN values
DEBUG - 2022-03-01 14:10:55,914 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:10:55,915 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 10.01% NaN values
DEBUG - 2022-03-01 14:10:55,917 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 10.10% NaN values
DEBUG - 2022-03-01 14:10:55,918 [data_cleaners.py:50] : 	"cloud_probability" has 10.10% NaN values
DEBUG - 2022-03-01 14:10:55,920 [data_cleaners.py:50] : 	"cloud_fraction" has 10.10% NaN values
DEBUG - 2022-03-01 14:10:55,921 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,923 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,924 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,926 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,927 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:55,929 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:10:55,931 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:10:55,931 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:10:56,853 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:10:56,969 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:10:56,969 [data_cleaners.py:107] : Cleaning took 1.2 seconds
INFO - 2022-03-01 14:10:57,058 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2022-03-01 14:10:57,060 [data_cleaners.py:38] : 55.72% of daylight timesteps are cloudy
INFO - 2022-03-01 14:10:57,062 [data_cleaners.py:40] : 10.05% of daylight timesteps are missing cloud type
INFO - 2022-03-01 14:10:57,063 [data_cleaners.py:42] : 49.09% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 14:10:57,065 [data_cleaners.py:44] : 49.21% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 14:10:57,065 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 14:10:57,066 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,068 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,069 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,071 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,072 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,074 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,076 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 85.91% NaN values
DEBUG - 2022-03-01 14:10:57,077 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 85.94% NaN values
DEBUG - 2022-03-01 14:10:57,078 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,080 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,081 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,082 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,083 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,085 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 14:10:57,085 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 14:10:57,576 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2022-03-01 14:10:57,690 [data_cleaners.py:105] : Feature flag column has these values: ['ice_cloud' 'water_cloud' 'night' 'bad_cloud' 'clear']
INFO - 2022-03-01 14:10:57,690 [data_cleaners.py:107] : Cleaning took 0.7 seconds
DEBUG - 2022-03-01 14:10:57,690 [data_handlers.py:453] : Prepping validation data
DEBUG - 2022-03-01 14:10:57,703 [data_handlers.py:463] : Mask: shape=(946512,), sum=470196
DEBUG - 2022-03-01 14:10:57,734 [data_handlers.py:474] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2022-03-01 14:10:57,734 [validator.py:107] : Predicting opd and reff
DEBUG - 2022-03-01 14:10:58,741 [validator.py:110] : Predicted data shape is (470196, 2)
DEBUG - 2022-03-01 14:10:58,836 [validator.py:158] : shapes: df_feature_val=(946512, 20), df_all_sky_val=(946512, 15)
INFO - 2022-03-01 14:10:58,914 [validator.py:179] : Calculating statistics
DEBUG - 2022-03-01 14:10:58,917 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2022-03-01 14:10:58,917 [validator.py:346] : Loading data for 2016 / west
DEBUG - 2022-03-01 14:10:58,917 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:10:58,964 [validator.py:386] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2022-03-01 14:10:59,582 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:10:59,617 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2022-03-01 14:11:00,233 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:11:00,270 [validator.py:386] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2022-03-01 14:11:00,885 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:11:00,922 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2022-03-01 14:11:01,544 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:11:01,581 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2022-03-01 14:11:02,196 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:11:02,233 [validator.py:386] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2022-03-01 14:11:02,850 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:11:02,885 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2022-03-01 14:11:03,500 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:11:03,534 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2022-03-01 14:11:04,145 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:11:04,179 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2022-03-01 14:11:04,794 [validator.py:346] : Loading data for 2017 / west
DEBUG - 2022-03-01 14:11:04,794 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:11:04,840 [validator.py:386] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2022-03-01 14:11:05,459 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:11:05,494 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2022-03-01 14:11:06,108 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:11:06,143 [validator.py:386] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2022-03-01 14:11:06,757 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:11:06,794 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2022-03-01 14:11:07,405 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:11:07,440 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2022-03-01 14:11:08,053 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:11:08,087 [validator.py:386] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2022-03-01 14:11:08,709 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:11:08,744 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2022-03-01 14:11:09,357 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:11:09,391 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2022-03-01 14:11:10,011 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:11:10,046 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2022-03-01 14:11:10,666 [validator.py:346] : Loading data for 2018 / west
DEBUG - 2022-03-01 14:11:10,666 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:11:10,712 [validator.py:386] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2022-03-01 14:11:11,324 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:11:11,359 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2022-03-01 14:11:11,975 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:11:12,010 [validator.py:386] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2022-03-01 14:11:12,624 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:11:12,658 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2022-03-01 14:11:13,274 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:11:13,310 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2022-03-01 14:11:13,924 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:11:13,958 [validator.py:386] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2022-03-01 14:11:14,576 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:11:14,610 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2022-03-01 14:11:15,225 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:11:15,259 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2022-03-01 14:11:15,883 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:11:15,918 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2022-03-01 14:11:16,535 [validator.py:346] : Loading data for 2019 / west
DEBUG - 2022-03-01 14:11:16,535 [validator.py:423] : 	Getting gid 0 from *.h5
DEBUG - 2022-03-01 14:11:16,601 [validator.py:386] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2022-03-01 14:11:17,222 [validator.py:423] : 	Getting gid 1 from *.h5
DEBUG - 2022-03-01 14:11:17,287 [validator.py:386] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2022-03-01 14:11:17,914 [validator.py:423] : 	Getting gid 2 from *.h5
DEBUG - 2022-03-01 14:11:17,979 [validator.py:386] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2022-03-01 14:11:18,602 [validator.py:423] : 	Getting gid 3 from *.h5
DEBUG - 2022-03-01 14:11:18,666 [validator.py:386] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2022-03-01 14:11:19,288 [validator.py:423] : 	Getting gid 4 from *.h5
DEBUG - 2022-03-01 14:11:19,353 [validator.py:386] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2022-03-01 14:11:19,980 [validator.py:423] : 	Getting gid 5 from *.h5
DEBUG - 2022-03-01 14:11:20,045 [validator.py:386] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2022-03-01 14:11:20,668 [validator.py:423] : 	Getting gid 6 from *.h5
DEBUG - 2022-03-01 14:11:20,732 [validator.py:386] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2022-03-01 14:11:21,356 [validator.py:423] : 	Getting gid 7 from *.h5
DEBUG - 2022-03-01 14:11:21,421 [validator.py:386] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2022-03-01 14:11:22,057 [validator.py:423] : 	Getting gid 8 from *.h5
DEBUG - 2022-03-01 14:11:22,123 [validator.py:386] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2022-03-01 14:11:22,750 [validator.py:187] : Shapes: df_base_full=(946512, 6), df_surf_full=(946512, 4)
DEBUG - 2022-03-01 14:11:22,754 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2022-03-01 14:11:22,765 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:11:27,975 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2022-03-01 14:11:27,986 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:11:33,163 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2022-03-01 14:11:33,175 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:11:38,323 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2022-03-01 14:11:38,334 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:11:43,515 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2022-03-01 14:11:43,526 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:11:48,727 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2022-03-01 14:11:48,739 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:11:54,367 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2022-03-01 14:11:54,378 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:11:59,571 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2022-03-01 14:11:59,582 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
DEBUG - 2022-03-01 14:12:04,777 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2022-03-01 14:12:04,788 [validator.py:209] : Shapes: df_baseline=(105168, 6), df_surf=(105168, 4)
INFO - 2022-03-01 14:12:09,964 [validator.py:292] : Finished computing stats.
