2023-04-13 13:35:13.961594: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-13 13:35:14.368309: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-13 13:35:14.368779: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-13 13:35:16.498829: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Number of files: 8
Number of east files: 4
Number of west files: 4
Source files: ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
Full config: {'clean_training_data_kwargs': {'filter_clear': False, 'nan_option': 'interp'}, 'epochs_a': 100, 'epochs_b': 90, 'features': ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo'], 'hidden_layers': [{'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}], 'learning_rate': 0.0005, 'loss_weights_a': [1, 0], 'loss_weights_b': [0.5, 0.5], 'metric': 'relative_mae', 'n_batch': 64, 'one_hot_categories': {'flag': ['clear', 'ice_cloud', 'water_cloud', 'bad_cloud']}, 'p_fun': 'p_fun_all_sky', 'p_kwargs': {'loss_terms': ['mae_ghi']}, 'phygnn_seed': 0, 'surfrad_window_minutes': 15, 'y_labels': ['cld_opd_dcomp', 'cld_reff_dcomp']}
INFO - 2023-04-13 13:35:22,420 [trainer.py:40] : Trainer: Training on sites all from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2023-04-13 13:35:22,427 [trainer.py:49] : Trainer: Training on sites [0, 1, 2, 3, 4, 5, 6, 7, 8] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2023-04-13 13:35:22,427 [data_handlers.py:60] : Loading training data
DEBUG - 2023-04-13 13:35:22,427 [data_handlers.py:78] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2023-04-13 13:35:22,427 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5
DEBUG - 2023-04-13 13:35:24,213 [data_handlers.py:103] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14)
DEBUG - 2023-04-13 13:35:24,216 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2023-04-13 13:35:24,216 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:35:24,219 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2023-04-13 13:35:24,931 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:24,933 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2023-04-13 13:35:25,643 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:25,645 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2023-04-13 13:35:26,351 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:26,354 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2023-04-13 13:35:27,072 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:27,075 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2023-04-13 13:35:27,797 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:27,800 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2023-04-13 13:35:28,519 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:28,523 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2023-04-13 13:35:29,236 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:29,239 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2023-04-13 13:35:29,961 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:29,964 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2023-04-13 13:35:30,702 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:30,702 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5
DEBUG - 2023-04-13 13:35:32,737 [data_handlers.py:103] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14)
DEBUG - 2023-04-13 13:35:32,740 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2023-04-13 13:35:32,740 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:35:32,743 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2023-04-13 13:35:33,392 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:33,395 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2023-04-13 13:35:34,044 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:34,046 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2023-04-13 13:35:34,702 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:34,705 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2023-04-13 13:35:35,353 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:35,355 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2023-04-13 13:35:36,001 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:36,004 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2023-04-13 13:35:36,653 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:36,656 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2023-04-13 13:35:37,310 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:37,313 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2023-04-13 13:35:37,968 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:37,971 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2016.h5
DEBUG - 2023-04-13 13:35:38,622 [data_handlers.py:135] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2023-04-13 13:35:38,622 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5
DEBUG - 2023-04-13 13:35:40,518 [data_handlers.py:103] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14)
DEBUG - 2023-04-13 13:35:40,522 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2023-04-13 13:35:40,522 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:35:40,525 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2023-04-13 13:35:41,231 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:41,234 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2023-04-13 13:35:41,950 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:41,953 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2023-04-13 13:35:42,662 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:42,665 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2023-04-13 13:35:43,384 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:43,387 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2023-04-13 13:35:44,098 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:44,101 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2023-04-13 13:35:44,803 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:44,806 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2023-04-13 13:35:45,528 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:45,531 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2023-04-13 13:35:46,226 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:46,229 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2023-04-13 13:35:46,977 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:46,977 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5
DEBUG - 2023-04-13 13:35:48,929 [data_handlers.py:103] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14)
DEBUG - 2023-04-13 13:35:48,932 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2023-04-13 13:35:48,932 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:35:48,935 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2023-04-13 13:35:49,583 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:49,586 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2023-04-13 13:35:50,236 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:50,239 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2023-04-13 13:35:50,889 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:50,892 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2023-04-13 13:35:51,539 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:51,542 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2023-04-13 13:35:52,194 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:52,197 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2023-04-13 13:35:52,842 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:52,846 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2023-04-13 13:35:53,489 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:53,492 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2023-04-13 13:35:54,137 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:54,140 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2017.h5
DEBUG - 2023-04-13 13:35:54,783 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:35:54,783 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5
DEBUG - 2023-04-13 13:36:05,820 [data_handlers.py:103] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14)
DEBUG - 2023-04-13 13:36:05,836 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2023-04-13 13:36:05,836 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:36:05,839 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2023-04-13 13:36:06,554 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:06,557 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2023-04-13 13:36:07,268 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:07,270 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2023-04-13 13:36:07,980 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:07,982 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2023-04-13 13:36:08,693 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:08,696 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2023-04-13 13:36:09,462 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:09,465 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2023-04-13 13:36:10,185 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:10,188 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2023-04-13 13:36:10,903 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:10,906 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2023-04-13 13:36:11,620 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:11,623 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2023-04-13 13:36:12,372 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:12,372 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5
DEBUG - 2023-04-13 13:36:14,221 [data_handlers.py:103] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14)
DEBUG - 2023-04-13 13:36:14,224 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2023-04-13 13:36:14,224 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:36:14,227 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2023-04-13 13:36:14,881 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:14,884 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2023-04-13 13:36:15,540 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:15,543 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2023-04-13 13:36:16,201 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:16,204 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2023-04-13 13:36:16,860 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:16,863 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2023-04-13 13:36:17,517 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:17,524 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2023-04-13 13:36:18,181 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:18,185 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2023-04-13 13:36:18,839 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:18,842 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2023-04-13 13:36:19,494 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:19,497 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2018.h5
DEBUG - 2023-04-13 13:36:20,151 [data_handlers.py:135] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2023-04-13 13:36:20,152 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5
DEBUG - 2023-04-13 13:36:30,948 [data_handlers.py:103] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14)
DEBUG - 2023-04-13 13:36:30,963 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2023-04-13 13:36:30,963 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:36:30,967 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2023-04-13 13:36:31,690 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:31,694 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2023-04-13 13:36:32,420 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:32,424 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2023-04-13 13:36:33,146 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:33,149 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2023-04-13 13:36:33,874 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:33,877 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2023-04-13 13:36:34,596 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:34,599 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2023-04-13 13:36:35,313 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:35,316 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2023-04-13 13:36:36,077 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:36,081 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2023-04-13 13:36:36,781 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:36,785 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2023-04-13 13:36:37,553 [data_handlers.py:135] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2023-04-13 13:36:37,554 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7, 8], from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5
DEBUG - 2023-04-13 13:36:42,804 [data_handlers.py:103] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14)
DEBUG - 2023-04-13 13:36:42,812 [data_handlers.py:106] : 	Time step is 10 minutes
DEBUG - 2023-04-13 13:36:42,812 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 13:36:42,815 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2023-04-13 13:36:43,485 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:43,488 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2023-04-13 13:36:44,159 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:44,162 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2023-04-13 13:36:44,827 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:44,831 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2023-04-13 13:36:45,498 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:45,501 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2023-04-13 13:36:46,170 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:46,173 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2023-04-13 13:36:46,836 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:46,839 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2023-04-13 13:36:47,519 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:47,523 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2023-04-13 13:36:48,203 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:48,206 [data_handlers.py:117] : 		Grabbing surface data for srrl from /projects/pxs/surfrad/h5/srrl_2019.h5
DEBUG - 2023-04-13 13:36:48,896 [data_handlers.py:135] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2023-04-13 13:36:48,896 [data_handlers.py:137] : Data load complete. Shape df_raw=(3154464, 19), df_all_sky=(3154464, 14), df_surf=(3154464, 5)
DEBUG - 2023-04-13 13:36:49,860 [data_handlers.py:285] : Creating test set; 20.0% of full data set
DEBUG - 2023-04-13 13:36:51,561 [data_handlers.py:301] : Train set shape: df_raw=(2523571, 19), df_all_sky=(2523571, 17)
DEBUG - 2023-04-13 13:36:51,673 [data_handlers.py:303] : Test set shape: df_raw=(630893, 19), df_all_sky=(630893, 17)
DEBUG - 2023-04-13 13:36:51,674 [data_handlers.py:162] : Extracting 2D arrays to run rest2 for clearsky PhyGNN inputs.
DEBUG - 2023-04-13 13:37:03,099 [data_handlers.py:179] : Running rest2 for clearsky PhyGNN inputs.
DEBUG - 2023-04-13 13:38:40,989 [data_handlers.py:197] : Completed rest2 run for clearsky PhyGNN inputs.
INFO - 2023-04-13 13:38:41,901 [data_handlers.py:62] : Prepping training data
DEBUG - 2023-04-13 13:38:41,901 [data_handlers.py:218] : Training data clean kwargs: {'filter_daylight': True, 'filter_clear': False, 'add_cloud_flag': True, 'sza_lim': 89, 'nan_option': 'interp'}
DEBUG - 2023-04-13 13:38:41,901 [data_handlers.py:219] : Shape before cleaning: df_raw=(2523571, 19)
INFO - 2023-04-13 13:38:42,738 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2023-04-13 13:38:42,741 [data_cleaners.py:38] : 52.35% of daylight timesteps are cloudy
INFO - 2023-04-13 13:38:42,743 [data_cleaners.py:40] : 3.26% of daylight timesteps are missing cloud type
INFO - 2023-04-13 13:38:42,746 [data_cleaners.py:42] : 34.01% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 13:38:42,748 [data_cleaners.py:44] : 34.23% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 13:38:42,748 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 13:38:42,750 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,753 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,755 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,759 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,761 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.45% NaN values
DEBUG - 2023-04-13 13:38:42,763 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.45% NaN values
DEBUG - 2023-04-13 13:38:42,766 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.29% NaN values
DEBUG - 2023-04-13 13:38:42,768 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.19% NaN values
DEBUG - 2023-04-13 13:38:42,770 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.19% NaN values
DEBUG - 2023-04-13 13:38:42,773 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.27% NaN values
DEBUG - 2023-04-13 13:38:42,775 [data_cleaners.py:50] : 	"cloud_probability" has 3.27% NaN values
DEBUG - 2023-04-13 13:38:42,777 [data_cleaners.py:50] : 	"cloud_fraction" has 3.27% NaN values
DEBUG - 2023-04-13 13:38:42,779 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,782 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,784 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,786 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,788 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:42,791 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.83% NaN values
DEBUG - 2023-04-13 13:38:42,793 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.89% NaN values
DEBUG - 2023-04-13 13:38:42,793 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 13:38:44,676 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2023-04-13 13:38:45,346 [data_cleaners.py:99] : Data reduced from 2523571 rows to 1253916 after filters (49.69% of original)
DEBUG - 2023-04-13 13:38:45,439 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2023-04-13 13:38:45,439 [data_cleaners.py:107] : Cleaning took 3.5 seconds
DEBUG - 2023-04-13 13:38:45,439 [data_handlers.py:222] : Shape after cleaning: df_train=(1253916, 20)
DEBUG - 2023-04-13 13:38:45,439 [data_handlers.py:225] : Cleaning df_all_sky training data (for pfun).
DEBUG - 2023-04-13 13:38:45,439 [data_handlers.py:226] : Shape before cleaning: df_all_sky=(2523571, 25)
INFO - 2023-04-13 13:38:46,390 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2023-04-13 13:38:46,392 [data_cleaners.py:38] : 52.35% of daylight timesteps are cloudy
INFO - 2023-04-13 13:38:46,395 [data_cleaners.py:40] : 3.26% of daylight timesteps are missing cloud type
INFO - 2023-04-13 13:38:46,397 [data_cleaners.py:42] : 34.01% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 13:38:46,399 [data_cleaners.py:44] : 34.23% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 13:38:46,399 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 13:38:46,401 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,403 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,406 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,408 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,412 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,414 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.83% NaN values
DEBUG - 2023-04-13 13:38:46,416 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.89% NaN values
DEBUG - 2023-04-13 13:38:46,419 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,421 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,424 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,426 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,428 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,430 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,433 [data_cleaners.py:50] : 	"surfrad_dhi" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,436 [data_cleaners.py:50] : 	"surfrad_dni" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,439 [data_cleaners.py:50] : 	"surfrad_ghi" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,441 [data_cleaners.py:50] : 	"doy" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,444 [data_cleaners.py:50] : 	"radius" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,447 [data_cleaners.py:50] : 	"Tuuclr" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,450 [data_cleaners.py:50] : 	"clearsky_ghi" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,453 [data_cleaners.py:50] : 	"clearsky_dni" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,456 [data_cleaners.py:50] : 	"Ruuclr" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,459 [data_cleaners.py:50] : 	"Tddclr" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,462 [data_cleaners.py:50] : 	"Tduclr" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,466 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 13:38:46,466 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 13:38:47,814 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2023-04-13 13:38:48,486 [data_cleaners.py:99] : Data reduced from 2523571 rows to 1253916 after filters (49.69% of original)
DEBUG - 2023-04-13 13:38:48,609 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2023-04-13 13:38:48,609 [data_cleaners.py:107] : Cleaning took 3.2 seconds
DEBUG - 2023-04-13 13:38:48,611 [data_handlers.py:230] : Shape after cleaning: df_all_sky=(1253916, 26)
DEBUG - 2023-04-13 13:38:48,702 [data_handlers.py:244] : **Shape: df_train=(1253916, 17)
DEBUG - 2023-04-13 13:38:48,728 [data_handlers.py:254] : Shapes: x=(1253916, 15), y=(1253916, 2), p=(1253916, 26)
DEBUG - 2023-04-13 13:38:48,728 [data_handlers.py:257] : Training features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2023-04-13 13:38:48,728 [data_handlers.py:332] : Saving training data to: ./mlclouds_training_data.csv
DEBUG - 2023-04-13 13:39:19,372 [trainer.py:67] : Building PHYGNN model
INFO - 2023-04-13 13:39:19,372 [trainer.py:70] : Using p_fun: <function p_fun_all_sky at 0x2b2dde8a5af0>
INFO - 2023-04-13 13:39:19,373 [base.py:90] : Active python environment versions: 
{   'nrel-rex': '0.2.80',
    'numpy': '1.23.5',
    'pandas': '2.0.0',
    'phygnn': '0.0.23',
    'python': '3.9.16 (main, Mar  8 2023, 14:00:05) \n[GCC 11.2.0]',
    'sklearn': '1.2.2',
    'tensorflow': '2.12.0'}
INFO - 2023-04-13 13:39:19,385 [base.py:108] : Successfully initialized model with 17 layers
INFO - 2023-04-13 13:39:19,433 [base_model.py:78] : Active python environment versions: 
{   'nrel-rex': '0.2.80',
    'numpy': '1.23.5',
    'pandas': '2.0.0',
    'phygnn': '0.0.23',
    'python': '3.9.16 (main, Mar  8 2023, 14:00:05) \n[GCC 11.2.0]',
    'sklearn': '1.2.2',
    'tensorflow': '2.12.0'}
INFO - 2023-04-13 13:39:19,433 [trainer.py:84] : Training part A - pure data. Loss is [1, 0]
/projects/pxs/gbuster/nsrdb_env_2023/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.
  warnings.warn(
INFO - 2023-04-13 13:39:35,029 [phygnn.py:594] : Epoch 0 train loss: 7.42e-01 val loss: 5.99e-01 for "phygnn"
INFO - 2023-04-13 13:39:44,225 [phygnn.py:594] : Epoch 1 train loss: 5.64e-01 val loss: 5.16e-01 for "phygnn"
INFO - 2023-04-13 13:39:53,669 [phygnn.py:594] : Epoch 2 train loss: 5.18e-01 val loss: 4.90e-01 for "phygnn"
INFO - 2023-04-13 13:40:02,959 [phygnn.py:594] : Epoch 3 train loss: 5.02e-01 val loss: 4.82e-01 for "phygnn"
INFO - 2023-04-13 13:40:12,085 [phygnn.py:594] : Epoch 4 train loss: 4.93e-01 val loss: 4.74e-01 for "phygnn"
INFO - 2023-04-13 13:40:21,281 [phygnn.py:594] : Epoch 5 train loss: 4.87e-01 val loss: 4.70e-01 for "phygnn"
INFO - 2023-04-13 13:40:30,555 [phygnn.py:594] : Epoch 6 train loss: 4.82e-01 val loss: 4.68e-01 for "phygnn"
INFO - 2023-04-13 13:40:39,671 [phygnn.py:594] : Epoch 7 train loss: 4.78e-01 val loss: 4.62e-01 for "phygnn"
INFO - 2023-04-13 13:40:48,803 [phygnn.py:594] : Epoch 8 train loss: 4.74e-01 val loss: 4.61e-01 for "phygnn"
INFO - 2023-04-13 13:40:58,258 [phygnn.py:594] : Epoch 9 train loss: 4.72e-01 val loss: 4.58e-01 for "phygnn"
INFO - 2023-04-13 13:41:07,695 [phygnn.py:594] : Epoch 10 train loss: 4.69e-01 val loss: 4.55e-01 for "phygnn"
INFO - 2023-04-13 13:41:16,777 [phygnn.py:594] : Epoch 11 train loss: 4.66e-01 val loss: 4.53e-01 for "phygnn"
INFO - 2023-04-13 13:41:25,900 [phygnn.py:594] : Epoch 12 train loss: 4.64e-01 val loss: 4.53e-01 for "phygnn"
INFO - 2023-04-13 13:41:35,018 [phygnn.py:594] : Epoch 13 train loss: 4.61e-01 val loss: 4.51e-01 for "phygnn"
INFO - 2023-04-13 13:41:44,134 [phygnn.py:594] : Epoch 14 train loss: 4.60e-01 val loss: 4.46e-01 for "phygnn"
INFO - 2023-04-13 13:41:53,309 [phygnn.py:594] : Epoch 15 train loss: 4.57e-01 val loss: 4.43e-01 for "phygnn"
INFO - 2023-04-13 13:42:02,292 [phygnn.py:594] : Epoch 16 train loss: 4.55e-01 val loss: 4.42e-01 for "phygnn"
INFO - 2023-04-13 13:42:11,233 [phygnn.py:594] : Epoch 17 train loss: 4.53e-01 val loss: 4.40e-01 for "phygnn"
INFO - 2023-04-13 13:42:20,345 [phygnn.py:594] : Epoch 18 train loss: 4.52e-01 val loss: 4.38e-01 for "phygnn"
INFO - 2023-04-13 13:42:29,418 [phygnn.py:594] : Epoch 19 train loss: 4.51e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2023-04-13 13:42:38,543 [phygnn.py:594] : Epoch 20 train loss: 4.49e-01 val loss: 4.38e-01 for "phygnn"
INFO - 2023-04-13 13:42:47,588 [phygnn.py:594] : Epoch 21 train loss: 4.48e-01 val loss: 4.34e-01 for "phygnn"
INFO - 2023-04-13 13:42:56,634 [phygnn.py:594] : Epoch 22 train loss: 4.47e-01 val loss: 4.37e-01 for "phygnn"
INFO - 2023-04-13 13:43:05,724 [phygnn.py:594] : Epoch 23 train loss: 4.46e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2023-04-13 13:43:14,826 [phygnn.py:594] : Epoch 24 train loss: 4.45e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2023-04-13 13:43:23,873 [phygnn.py:594] : Epoch 25 train loss: 4.44e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2023-04-13 13:43:32,942 [phygnn.py:594] : Epoch 26 train loss: 4.43e-01 val loss: 4.29e-01 for "phygnn"
INFO - 2023-04-13 13:43:42,049 [phygnn.py:594] : Epoch 27 train loss: 4.42e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2023-04-13 13:43:51,148 [phygnn.py:594] : Epoch 28 train loss: 4.41e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2023-04-13 13:44:00,190 [phygnn.py:594] : Epoch 29 train loss: 4.40e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2023-04-13 13:44:09,115 [phygnn.py:594] : Epoch 30 train loss: 4.39e-01 val loss: 4.28e-01 for "phygnn"
INFO - 2023-04-13 13:44:18,009 [phygnn.py:594] : Epoch 31 train loss: 4.38e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2023-04-13 13:44:26,913 [phygnn.py:594] : Epoch 32 train loss: 4.37e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2023-04-13 13:44:36,141 [phygnn.py:594] : Epoch 33 train loss: 4.37e-01 val loss: 4.27e-01 for "phygnn"
INFO - 2023-04-13 13:44:45,220 [phygnn.py:594] : Epoch 34 train loss: 4.36e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2023-04-13 13:44:54,409 [phygnn.py:594] : Epoch 35 train loss: 4.35e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2023-04-13 13:45:03,443 [phygnn.py:594] : Epoch 36 train loss: 4.34e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2023-04-13 13:45:12,358 [phygnn.py:594] : Epoch 37 train loss: 4.34e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2023-04-13 13:45:21,648 [phygnn.py:594] : Epoch 38 train loss: 4.33e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2023-04-13 13:45:30,803 [phygnn.py:594] : Epoch 39 train loss: 4.32e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2023-04-13 13:45:39,820 [phygnn.py:594] : Epoch 40 train loss: 4.31e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2023-04-13 13:45:48,868 [phygnn.py:594] : Epoch 41 train loss: 4.31e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2023-04-13 13:45:58,003 [phygnn.py:594] : Epoch 42 train loss: 4.30e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2023-04-13 13:46:07,153 [phygnn.py:594] : Epoch 43 train loss: 4.30e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2023-04-13 13:46:16,269 [phygnn.py:594] : Epoch 44 train loss: 4.29e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2023-04-13 13:46:25,359 [phygnn.py:594] : Epoch 45 train loss: 4.29e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2023-04-13 13:46:34,517 [phygnn.py:594] : Epoch 46 train loss: 4.28e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2023-04-13 13:46:43,855 [phygnn.py:594] : Epoch 47 train loss: 4.28e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2023-04-13 13:46:52,952 [phygnn.py:594] : Epoch 48 train loss: 4.27e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2023-04-13 13:47:02,007 [phygnn.py:594] : Epoch 49 train loss: 4.26e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2023-04-13 13:47:11,058 [phygnn.py:594] : Epoch 50 train loss: 4.26e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2023-04-13 13:47:20,206 [phygnn.py:594] : Epoch 51 train loss: 4.25e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2023-04-13 13:47:29,789 [phygnn.py:594] : Epoch 52 train loss: 4.25e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2023-04-13 13:47:38,882 [phygnn.py:594] : Epoch 53 train loss: 4.24e-01 val loss: 4.16e-01 for "phygnn"
INFO - 2023-04-13 13:47:47,821 [phygnn.py:594] : Epoch 54 train loss: 4.24e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2023-04-13 13:47:56,766 [phygnn.py:594] : Epoch 55 train loss: 4.23e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2023-04-13 13:48:05,897 [phygnn.py:594] : Epoch 56 train loss: 4.23e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2023-04-13 13:48:14,941 [phygnn.py:594] : Epoch 57 train loss: 4.23e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2023-04-13 13:48:24,060 [phygnn.py:594] : Epoch 58 train loss: 4.22e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2023-04-13 13:48:33,035 [phygnn.py:594] : Epoch 59 train loss: 4.21e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2023-04-13 13:48:42,036 [phygnn.py:594] : Epoch 60 train loss: 4.22e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2023-04-13 13:48:51,201 [phygnn.py:594] : Epoch 61 train loss: 4.21e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2023-04-13 13:49:00,165 [phygnn.py:594] : Epoch 62 train loss: 4.20e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2023-04-13 13:49:09,226 [phygnn.py:594] : Epoch 63 train loss: 4.20e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2023-04-13 13:49:18,240 [phygnn.py:594] : Epoch 64 train loss: 4.19e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2023-04-13 13:49:27,271 [phygnn.py:594] : Epoch 65 train loss: 4.19e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2023-04-13 13:49:36,282 [phygnn.py:594] : Epoch 66 train loss: 4.18e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2023-04-13 13:49:45,180 [phygnn.py:594] : Epoch 67 train loss: 4.18e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2023-04-13 13:49:54,135 [phygnn.py:594] : Epoch 68 train loss: 4.18e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2023-04-13 13:50:03,204 [phygnn.py:594] : Epoch 69 train loss: 4.17e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2023-04-13 13:50:12,249 [phygnn.py:594] : Epoch 70 train loss: 4.16e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2023-04-13 13:50:21,477 [phygnn.py:594] : Epoch 71 train loss: 4.16e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2023-04-13 13:50:30,543 [phygnn.py:594] : Epoch 72 train loss: 4.16e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2023-04-13 13:50:39,906 [phygnn.py:594] : Epoch 73 train loss: 4.16e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2023-04-13 13:50:49,006 [phygnn.py:594] : Epoch 74 train loss: 4.15e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2023-04-13 13:50:58,155 [phygnn.py:594] : Epoch 75 train loss: 4.15e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2023-04-13 13:51:07,419 [phygnn.py:594] : Epoch 76 train loss: 4.14e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2023-04-13 13:51:16,459 [phygnn.py:594] : Epoch 77 train loss: 4.14e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2023-04-13 13:51:25,485 [phygnn.py:594] : Epoch 78 train loss: 4.13e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2023-04-13 13:51:34,490 [phygnn.py:594] : Epoch 79 train loss: 4.13e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2023-04-13 13:51:43,418 [phygnn.py:594] : Epoch 80 train loss: 4.13e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2023-04-13 13:51:52,484 [phygnn.py:594] : Epoch 81 train loss: 4.12e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2023-04-13 13:52:01,623 [phygnn.py:594] : Epoch 82 train loss: 4.12e-01 val loss: 4.02e-01 for "phygnn"
INFO - 2023-04-13 13:52:10,646 [phygnn.py:594] : Epoch 83 train loss: 4.11e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2023-04-13 13:52:19,834 [phygnn.py:594] : Epoch 84 train loss: 4.12e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2023-04-13 13:52:28,774 [phygnn.py:594] : Epoch 85 train loss: 4.11e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2023-04-13 13:52:37,834 [phygnn.py:594] : Epoch 86 train loss: 4.10e-01 val loss: 4.00e-01 for "phygnn"
INFO - 2023-04-13 13:52:46,903 [phygnn.py:594] : Epoch 87 train loss: 4.10e-01 val loss: 4.01e-01 for "phygnn"
INFO - 2023-04-13 13:52:56,034 [phygnn.py:594] : Epoch 88 train loss: 4.10e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2023-04-13 13:53:05,177 [phygnn.py:594] : Epoch 89 train loss: 4.09e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2023-04-13 13:53:14,327 [phygnn.py:594] : Epoch 90 train loss: 4.09e-01 val loss: 3.99e-01 for "phygnn"
INFO - 2023-04-13 13:53:23,361 [phygnn.py:594] : Epoch 91 train loss: 4.09e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2023-04-13 13:53:32,345 [phygnn.py:594] : Epoch 92 train loss: 4.08e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2023-04-13 13:53:41,496 [phygnn.py:594] : Epoch 93 train loss: 4.08e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2023-04-13 13:53:50,544 [phygnn.py:594] : Epoch 94 train loss: 4.08e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2023-04-13 13:53:59,623 [phygnn.py:594] : Epoch 95 train loss: 4.07e-01 val loss: 3.97e-01 for "phygnn"
INFO - 2023-04-13 13:54:08,660 [phygnn.py:594] : Epoch 96 train loss: 4.07e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2023-04-13 13:54:17,777 [phygnn.py:594] : Epoch 97 train loss: 4.07e-01 val loss: 3.98e-01 for "phygnn"
INFO - 2023-04-13 13:54:26,826 [phygnn.py:594] : Epoch 98 train loss: 4.07e-01 val loss: 3.96e-01 for "phygnn"
INFO - 2023-04-13 13:54:35,740 [phygnn.py:594] : Epoch 99 train loss: 4.06e-01 val loss: 3.95e-01 for "phygnn"
INFO - 2023-04-13 13:54:36,283 [trainer.py:92] : Training part B - data and phygnn. Loss is [0.5, 0.5]
/projects/pxs/gbuster/nsrdb_env_2023/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.
  warnings.warn(
INFO - 2023-04-13 13:54:58,271 [phygnn.py:594] : Epoch 100 train loss: 2.83e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2023-04-13 13:55:13,725 [phygnn.py:594] : Epoch 101 train loss: 2.82e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:55:29,597 [phygnn.py:594] : Epoch 102 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:55:45,412 [phygnn.py:594] : Epoch 103 train loss: 2.81e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 13:56:00,699 [phygnn.py:594] : Epoch 104 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:56:16,522 [phygnn.py:594] : Epoch 105 train loss: 2.81e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 13:56:31,918 [phygnn.py:594] : Epoch 106 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:56:47,725 [phygnn.py:594] : Epoch 107 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:57:03,442 [phygnn.py:594] : Epoch 108 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:57:18,652 [phygnn.py:594] : Epoch 109 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:57:34,282 [phygnn.py:594] : Epoch 110 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 13:57:49,814 [phygnn.py:594] : Epoch 111 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 13:58:05,209 [phygnn.py:594] : Epoch 112 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:58:21,121 [phygnn.py:594] : Epoch 113 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:58:36,471 [phygnn.py:594] : Epoch 114 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2023-04-13 13:58:52,621 [phygnn.py:594] : Epoch 115 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 13:59:08,617 [phygnn.py:594] : Epoch 116 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 13:59:23,962 [phygnn.py:594] : Epoch 117 train loss: 2.80e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 13:59:39,706 [phygnn.py:594] : Epoch 118 train loss: 2.79e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2023-04-13 13:59:55,475 [phygnn.py:594] : Epoch 119 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:00:11,120 [phygnn.py:594] : Epoch 120 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:00:26,967 [phygnn.py:594] : Epoch 121 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:00:43,208 [phygnn.py:594] : Epoch 122 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:00:58,842 [phygnn.py:594] : Epoch 123 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:01:14,742 [phygnn.py:594] : Epoch 124 train loss: 2.79e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:01:31,008 [phygnn.py:594] : Epoch 125 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:01:46,759 [phygnn.py:594] : Epoch 126 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:02:02,360 [phygnn.py:594] : Epoch 127 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:02:17,977 [phygnn.py:594] : Epoch 128 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:02:34,122 [phygnn.py:594] : Epoch 129 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:02:49,666 [phygnn.py:594] : Epoch 130 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:03:05,104 [phygnn.py:594] : Epoch 131 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:03:20,516 [phygnn.py:594] : Epoch 132 train loss: 2.78e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2023-04-13 14:03:36,024 [phygnn.py:594] : Epoch 133 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:03:52,207 [phygnn.py:594] : Epoch 134 train loss: 2.78e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:04:08,251 [phygnn.py:594] : Epoch 135 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:04:23,992 [phygnn.py:594] : Epoch 136 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:04:40,418 [phygnn.py:594] : Epoch 137 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:04:55,979 [phygnn.py:594] : Epoch 138 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:05:12,058 [phygnn.py:594] : Epoch 139 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:05:27,801 [phygnn.py:594] : Epoch 140 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:05:44,136 [phygnn.py:594] : Epoch 141 train loss: 2.77e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:05:59,927 [phygnn.py:594] : Epoch 142 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:06:15,679 [phygnn.py:594] : Epoch 143 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:06:31,870 [phygnn.py:594] : Epoch 144 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:06:47,766 [phygnn.py:594] : Epoch 145 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:07:03,741 [phygnn.py:594] : Epoch 146 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:07:20,064 [phygnn.py:594] : Epoch 147 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:07:36,410 [phygnn.py:594] : Epoch 148 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:07:52,620 [phygnn.py:594] : Epoch 149 train loss: 2.76e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:08:08,908 [phygnn.py:594] : Epoch 150 train loss: 2.76e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:08:25,049 [phygnn.py:594] : Epoch 151 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:08:41,416 [phygnn.py:594] : Epoch 152 train loss: 2.76e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:08:57,877 [phygnn.py:594] : Epoch 153 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:09:13,392 [phygnn.py:594] : Epoch 154 train loss: 2.75e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:09:29,631 [phygnn.py:594] : Epoch 155 train loss: 2.75e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:09:45,734 [phygnn.py:594] : Epoch 156 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:10:01,808 [phygnn.py:594] : Epoch 157 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:10:17,979 [phygnn.py:594] : Epoch 158 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:10:33,999 [phygnn.py:594] : Epoch 159 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:10:50,215 [phygnn.py:594] : Epoch 160 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:11:06,251 [phygnn.py:594] : Epoch 161 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:11:22,209 [phygnn.py:594] : Epoch 162 train loss: 2.75e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:11:38,730 [phygnn.py:594] : Epoch 163 train loss: 2.74e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:11:54,954 [phygnn.py:594] : Epoch 164 train loss: 2.74e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:12:11,124 [phygnn.py:594] : Epoch 165 train loss: 2.74e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:12:27,414 [phygnn.py:594] : Epoch 166 train loss: 2.74e-01 val loss: 2.68e-01 for "phygnn"
INFO - 2023-04-13 14:12:43,280 [phygnn.py:594] : Epoch 167 train loss: 2.74e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:12:59,252 [phygnn.py:594] : Epoch 168 train loss: 2.74e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:13:15,369 [phygnn.py:594] : Epoch 169 train loss: 2.74e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:13:31,618 [phygnn.py:594] : Epoch 170 train loss: 2.74e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:13:47,792 [phygnn.py:594] : Epoch 171 train loss: 2.73e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:14:04,148 [phygnn.py:594] : Epoch 172 train loss: 2.73e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:14:20,321 [phygnn.py:594] : Epoch 173 train loss: 2.73e-01 val loss: 2.67e-01 for "phygnn"
INFO - 2023-04-13 14:14:36,384 [phygnn.py:594] : Epoch 174 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:14:52,065 [phygnn.py:594] : Epoch 175 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:15:07,718 [phygnn.py:594] : Epoch 176 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:15:23,715 [phygnn.py:594] : Epoch 177 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:15:40,033 [phygnn.py:594] : Epoch 178 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:15:56,369 [phygnn.py:594] : Epoch 179 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:16:12,565 [phygnn.py:594] : Epoch 180 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:16:28,616 [phygnn.py:594] : Epoch 181 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:16:44,331 [phygnn.py:594] : Epoch 182 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:17:00,711 [phygnn.py:594] : Epoch 183 train loss: 2.73e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:17:16,512 [phygnn.py:594] : Epoch 184 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:17:32,025 [phygnn.py:594] : Epoch 185 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:17:47,975 [phygnn.py:594] : Epoch 186 train loss: 2.72e-01 val loss: 2.65e-01 for "phygnn"
INFO - 2023-04-13 14:18:04,353 [phygnn.py:594] : Epoch 187 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:18:20,367 [phygnn.py:594] : Epoch 188 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:18:36,821 [phygnn.py:594] : Epoch 189 train loss: 2.72e-01 val loss: 2.66e-01 for "phygnn"
INFO - 2023-04-13 14:18:37,381 [trainer.py:102] : Training complete
INFO - 2023-04-13 14:18:37,432 [base.py:469] : Saved model to: /home/gbuster/code/mlclouds/mlclouds/model/production_model/outputs/mlclouds_model.pkl
Validating on 630893 out of 3154464 observations
DEBUG - 2023-04-13 14:18:38,715 [data_handlers.py:391] : Loading validation data
DEBUG - 2023-04-13 14:18:38,715 [data_handlers.py:397] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2023-04-13 14:18:38,744 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:18:40,586 [data_handlers.py:421] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:18:40,586 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:18:42,310 [data_handlers.py:421] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:18:42,310 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:18:44,001 [data_handlers.py:421] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:18:44,001 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:18:45,768 [data_handlers.py:421] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:18:45,769 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:18:57,370 [data_handlers.py:421] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2023-04-13 14:18:57,371 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:18:59,094 [data_handlers.py:421] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:18:59,094 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:19:10,830 [data_handlers.py:421] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2023-04-13 14:19:10,831 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:19:16,267 [data_handlers.py:421] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2023-04-13 14:19:16,267 [data_handlers.py:424] : Shape df_raw=(3154464, 19), df_all_sky=(3154464, 14)
DEBUG - 2023-04-13 14:19:16,267 [data_handlers.py:431] : Shape after reset_index: df_raw=(3154464, 19), df_all_sky=(3154464, 14)
DEBUG - 2023-04-13 14:19:16,368 [data_handlers.py:441] : Test set shape: df_raw=(630893, 19), df_all_sky=(630893, 14)
INFO - 2023-04-13 14:19:16,464 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2023-04-13 14:19:16,464 [data_cleaners.py:38] : 52.16% of daylight timesteps are cloudy
INFO - 2023-04-13 14:19:16,465 [data_cleaners.py:40] : 3.22% of daylight timesteps are missing cloud type
INFO - 2023-04-13 14:19:16,466 [data_cleaners.py:42] : 33.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 14:19:16,466 [data_cleaners.py:44] : 34.13% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 14:19:16,467 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 14:19:16,467 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,468 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,469 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,470 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,471 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.43% NaN values
DEBUG - 2023-04-13 14:19:16,471 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.43% NaN values
DEBUG - 2023-04-13 14:19:16,472 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.26% NaN values
DEBUG - 2023-04-13 14:19:16,473 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.17% NaN values
DEBUG - 2023-04-13 14:19:16,474 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.17% NaN values
DEBUG - 2023-04-13 14:19:16,475 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.25% NaN values
DEBUG - 2023-04-13 14:19:16,475 [data_cleaners.py:50] : 	"cloud_probability" has 3.25% NaN values
DEBUG - 2023-04-13 14:19:16,476 [data_cleaners.py:50] : 	"cloud_fraction" has 3.25% NaN values
DEBUG - 2023-04-13 14:19:16,477 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,478 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,479 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,479 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,480 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:16,481 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.88% NaN values
DEBUG - 2023-04-13 14:19:16,482 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.93% NaN values
DEBUG - 2023-04-13 14:19:16,482 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 14:19:16,889 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2023-04-13 14:19:17,007 [data_cleaners.py:105] : Feature flag column has these values: ['water_cloud' 'night' 'bad_cloud' 'ice_cloud' 'clear']
INFO - 2023-04-13 14:19:17,007 [data_cleaners.py:107] : Cleaning took 0.6 seconds
INFO - 2023-04-13 14:19:17,092 [data_cleaners.py:36] : 49.68% of timesteps are daylight
INFO - 2023-04-13 14:19:17,093 [data_cleaners.py:38] : 52.16% of daylight timesteps are cloudy
INFO - 2023-04-13 14:19:17,094 [data_cleaners.py:40] : 3.22% of daylight timesteps are missing cloud type
INFO - 2023-04-13 14:19:17,094 [data_cleaners.py:42] : 33.92% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 14:19:17,095 [data_cleaners.py:44] : 34.13% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 14:19:17,095 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 14:19:17,096 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,097 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,097 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,098 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,099 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,100 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,101 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.88% NaN values
DEBUG - 2023-04-13 14:19:17,101 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 82.93% NaN values
DEBUG - 2023-04-13 14:19:17,102 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,103 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,104 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,104 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,105 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,106 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 14:19:17,106 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 14:19:17,328 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2023-04-13 14:19:17,445 [data_cleaners.py:105] : Feature flag column has these values: ['water_cloud' 'night' 'bad_cloud' 'ice_cloud' 'clear']
INFO - 2023-04-13 14:19:17,445 [data_cleaners.py:107] : Cleaning took 0.4 seconds
DEBUG - 2023-04-13 14:19:17,446 [data_handlers.py:464] : Prepping validation data
DEBUG - 2023-04-13 14:19:17,483 [data_handlers.py:474] : Mask: shape=(630893,), sum=313437
DEBUG - 2023-04-13 14:19:17,507 [data_handlers.py:485] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2023-04-13 14:19:17,507 [validator.py:107] : Predicting opd and reff
/projects/pxs/gbuster/nsrdb_env_2023/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.
  warnings.warn(
DEBUG - 2023-04-13 14:19:18,147 [validator.py:110] : Predicted data shape is (313437, 2)
DEBUG - 2023-04-13 14:19:18,217 [validator.py:158] : shapes: df_feature_val=(630893, 20), df_all_sky_val=(630893, 15)
INFO - 2023-04-13 14:19:18,265 [validator.py:179] : Calculating statistics
DEBUG - 2023-04-13 14:19:18,268 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:19:18,268 [validator.py:339] : Loading data for 2016 / east
DEBUG - 2023-04-13 14:19:18,268 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:19:18,314 [validator.py:377] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2023-04-13 14:19:19,018 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:19:19,051 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2023-04-13 14:19:19,746 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:19:19,780 [validator.py:377] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2023-04-13 14:19:20,470 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:19:20,504 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2023-04-13 14:19:21,192 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:19:21,226 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2023-04-13 14:19:21,912 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:19:21,946 [validator.py:377] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2023-04-13 14:19:22,640 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:19:22,674 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2023-04-13 14:19:23,359 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:19:23,392 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2023-04-13 14:19:24,079 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:19:24,112 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2023-04-13 14:19:24,822 [validator.py:339] : Loading data for 2016 / west
DEBUG - 2023-04-13 14:19:24,823 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:19:24,868 [validator.py:377] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2023-04-13 14:19:25,495 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:19:25,528 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2023-04-13 14:19:26,158 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:19:26,192 [validator.py:377] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2023-04-13 14:19:26,820 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:19:26,854 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2023-04-13 14:19:27,481 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:19:27,512 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2023-04-13 14:19:28,142 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:19:28,176 [validator.py:377] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2023-04-13 14:19:28,806 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:19:28,838 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2023-04-13 14:19:29,471 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:19:29,504 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2023-04-13 14:19:30,129 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:19:30,161 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2023-04-13 14:19:30,790 [validator.py:339] : Loading data for 2017 / east
DEBUG - 2023-04-13 14:19:30,791 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:19:30,837 [validator.py:377] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2023-04-13 14:19:31,528 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:19:31,561 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2023-04-13 14:19:32,262 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:19:32,294 [validator.py:377] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2023-04-13 14:19:32,985 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:19:33,018 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2023-04-13 14:19:33,914 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:19:33,946 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2023-04-13 14:19:34,640 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:19:34,673 [validator.py:377] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2023-04-13 14:19:35,372 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:19:35,404 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2023-04-13 14:19:36,099 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:19:36,131 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2023-04-13 14:19:36,804 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:19:36,836 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2023-04-13 14:19:37,563 [validator.py:339] : Loading data for 2017 / west
DEBUG - 2023-04-13 14:19:37,563 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:19:37,715 [validator.py:377] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2023-04-13 14:19:38,355 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:19:38,387 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2023-04-13 14:19:39,031 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:19:39,063 [validator.py:377] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2023-04-13 14:19:39,706 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:19:39,738 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2023-04-13 14:19:40,377 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:19:40,409 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2023-04-13 14:19:41,050 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:19:41,083 [validator.py:377] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2023-04-13 14:19:41,722 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:19:41,755 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2023-04-13 14:19:42,391 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:19:42,424 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2023-04-13 14:19:43,058 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:19:43,090 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2023-04-13 14:19:43,731 [validator.py:339] : Loading data for 2018 / east
DEBUG - 2023-04-13 14:19:43,731 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:19:44,026 [validator.py:377] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2023-04-13 14:19:44,725 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:19:44,796 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2023-04-13 14:19:45,506 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:19:45,577 [validator.py:377] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2023-04-13 14:19:46,285 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:19:46,357 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2023-04-13 14:19:47,069 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:19:47,141 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2023-04-13 14:19:47,851 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:19:47,922 [validator.py:377] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2023-04-13 14:19:48,627 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:19:48,699 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2023-04-13 14:19:49,410 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:19:49,481 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2023-04-13 14:19:50,189 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:19:50,261 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2023-04-13 14:19:51,005 [validator.py:339] : Loading data for 2018 / west
DEBUG - 2023-04-13 14:19:51,005 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:19:51,051 [validator.py:377] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2023-04-13 14:19:51,700 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:19:51,732 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2023-04-13 14:19:52,383 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:19:52,415 [validator.py:377] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2023-04-13 14:19:53,064 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:19:53,095 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2023-04-13 14:19:53,745 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:19:53,776 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2023-04-13 14:19:54,428 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:19:54,468 [validator.py:377] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2023-04-13 14:19:55,120 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:19:55,152 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2023-04-13 14:19:55,803 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:19:55,835 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2023-04-13 14:19:56,484 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:19:56,515 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2023-04-13 14:19:57,167 [validator.py:339] : Loading data for 2019 / east
DEBUG - 2023-04-13 14:19:57,167 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:19:57,438 [validator.py:377] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2023-04-13 14:19:58,154 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:19:58,225 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2023-04-13 14:19:58,945 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:19:59,016 [validator.py:377] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2023-04-13 14:19:59,737 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:19:59,808 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2023-04-13 14:20:00,529 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:20:00,599 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2023-04-13 14:20:01,326 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:20:01,398 [validator.py:377] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2023-04-13 14:20:02,110 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:20:02,181 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2023-04-13 14:20:02,897 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:20:02,967 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2023-04-13 14:20:03,660 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:20:03,731 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2023-04-13 14:20:04,542 [validator.py:339] : Loading data for 2019 / west
DEBUG - 2023-04-13 14:20:04,542 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:20:04,620 [validator.py:377] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2023-04-13 14:20:05,292 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:20:05,339 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2023-04-13 14:20:06,022 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:20:06,072 [validator.py:377] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2023-04-13 14:20:06,755 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:20:06,802 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2023-04-13 14:20:07,482 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:20:07,529 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2023-04-13 14:20:08,210 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:20:08,258 [validator.py:377] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2023-04-13 14:20:08,925 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:20:08,973 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2023-04-13 14:20:09,653 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:20:09,700 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2023-04-13 14:20:10,373 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:20:10,420 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2023-04-13 14:20:11,107 [validator.py:187] : Shapes: df_base_full=(3154464, 6), df_surf_full=(3154464, 4)
DEBUG - 2023-04-13 14:20:11,153 [validator.py:194] : Test set shapes: df_base_full=(630893, 6), df_surf_full=(630893, 4)
DEBUG - 2023-04-13 14:20:11,157 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2023-04-13 14:20:11,163 [validator.py:209] : Shapes: df_baseline=(70126, 6), df_surf=(70126, 4)
DEBUG - 2023-04-13 14:20:15,176 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2023-04-13 14:20:15,182 [validator.py:209] : Shapes: df_baseline=(70295, 6), df_surf=(70295, 4)
DEBUG - 2023-04-13 14:20:19,145 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2023-04-13 14:20:19,152 [validator.py:209] : Shapes: df_baseline=(70062, 6), df_surf=(70062, 4)
DEBUG - 2023-04-13 14:20:23,084 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2023-04-13 14:20:23,090 [validator.py:209] : Shapes: df_baseline=(70216, 6), df_surf=(70216, 4)
DEBUG - 2023-04-13 14:20:27,039 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2023-04-13 14:20:27,046 [validator.py:209] : Shapes: df_baseline=(70196, 6), df_surf=(70196, 4)
DEBUG - 2023-04-13 14:20:31,014 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2023-04-13 14:20:31,021 [validator.py:209] : Shapes: df_baseline=(70109, 6), df_surf=(70109, 4)
DEBUG - 2023-04-13 14:20:34,995 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2023-04-13 14:20:35,001 [validator.py:209] : Shapes: df_baseline=(69998, 6), df_surf=(69998, 4)
DEBUG - 2023-04-13 14:20:38,961 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2023-04-13 14:20:38,967 [validator.py:209] : Shapes: df_baseline=(69862, 6), df_surf=(69862, 4)
DEBUG - 2023-04-13 14:20:42,888 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2023-04-13 14:20:42,894 [validator.py:209] : Shapes: df_baseline=(70029, 6), df_surf=(70029, 4)
INFO - 2023-04-13 14:20:46,812 [validator.py:285] : Finished computing stats.
Validating on 441941 out of 2207952 observations
DEBUG - 2023-04-13 14:20:47,344 [data_handlers.py:391] : Loading validation data
DEBUG - 2023-04-13 14:20:47,344 [data_handlers.py:397] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2023-04-13 14:20:47,347 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:20:49,037 [data_handlers.py:421] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:20:49,037 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:20:50,701 [data_handlers.py:421] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:20:50,701 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:21:01,733 [data_handlers.py:421] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2023-04-13 14:21:01,734 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:21:12,838 [data_handlers.py:421] : 	Shape temp_raw=(946080, 19), temp_all_sky=(946080, 14), & tstep=5 minutes
DEBUG - 2023-04-13 14:21:12,839 [data_handlers.py:424] : Shape df_raw=(2207952, 19), df_all_sky=(2207952, 14)
DEBUG - 2023-04-13 14:21:12,839 [data_handlers.py:431] : Shape after reset_index: df_raw=(2207952, 19), df_all_sky=(2207952, 14)
DEBUG - 2023-04-13 14:21:12,911 [data_handlers.py:441] : Test set shape: df_raw=(441941, 19), df_all_sky=(441941, 14)
INFO - 2023-04-13 14:21:12,977 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2023-04-13 14:21:12,977 [data_cleaners.py:38] : 50.80% of daylight timesteps are cloudy
INFO - 2023-04-13 14:21:12,978 [data_cleaners.py:40] : 0.33% of daylight timesteps are missing cloud type
INFO - 2023-04-13 14:21:12,979 [data_cleaners.py:42] : 26.84% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 14:21:12,979 [data_cleaners.py:44] : 27.10% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 14:21:12,979 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 14:21:12,980 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,981 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,981 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,982 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,983 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 49.99% NaN values
DEBUG - 2023-04-13 14:21:12,983 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 49.99% NaN values
DEBUG - 2023-04-13 14:21:12,984 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 0.34% NaN values
DEBUG - 2023-04-13 14:21:12,984 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 0.26% NaN values
DEBUG - 2023-04-13 14:21:12,985 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 0.26% NaN values
DEBUG - 2023-04-13 14:21:12,986 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 0.33% NaN values
DEBUG - 2023-04-13 14:21:12,986 [data_cleaners.py:50] : 	"cloud_probability" has 0.33% NaN values
DEBUG - 2023-04-13 14:21:12,987 [data_cleaners.py:50] : 	"cloud_fraction" has 0.33% NaN values
DEBUG - 2023-04-13 14:21:12,987 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,988 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,989 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,989 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,990 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:12,990 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2023-04-13 14:21:12,991 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2023-04-13 14:21:12,991 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 14:21:13,260 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2023-04-13 14:21:13,338 [data_cleaners.py:105] : Feature flag column has these values: ['water_cloud' 'night' 'bad_cloud' 'ice_cloud' 'clear']
INFO - 2023-04-13 14:21:13,339 [data_cleaners.py:107] : Cleaning took 0.4 seconds
INFO - 2023-04-13 14:21:13,395 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2023-04-13 14:21:13,396 [data_cleaners.py:38] : 50.80% of daylight timesteps are cloudy
INFO - 2023-04-13 14:21:13,397 [data_cleaners.py:40] : 0.33% of daylight timesteps are missing cloud type
INFO - 2023-04-13 14:21:13,397 [data_cleaners.py:42] : 26.84% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 14:21:13,398 [data_cleaners.py:44] : 27.10% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 14:21:13,398 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 14:21:13,398 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,399 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,400 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,400 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,401 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,402 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,402 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 81.53% NaN values
DEBUG - 2023-04-13 14:21:13,403 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 81.60% NaN values
DEBUG - 2023-04-13 14:21:13,403 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,404 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,404 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,405 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,405 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,406 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 14:21:13,406 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 14:21:13,562 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2023-04-13 14:21:13,640 [data_cleaners.py:105] : Feature flag column has these values: ['water_cloud' 'night' 'bad_cloud' 'ice_cloud' 'clear']
INFO - 2023-04-13 14:21:13,640 [data_cleaners.py:107] : Cleaning took 0.3 seconds
DEBUG - 2023-04-13 14:21:13,640 [data_handlers.py:464] : Prepping validation data
DEBUG - 2023-04-13 14:21:13,665 [data_handlers.py:474] : Mask: shape=(441941,), sum=219586
DEBUG - 2023-04-13 14:21:13,680 [data_handlers.py:485] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2023-04-13 14:21:13,680 [validator.py:107] : Predicting opd and reff
/projects/pxs/gbuster/nsrdb_env_2023/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.
  warnings.warn(
DEBUG - 2023-04-13 14:21:14,356 [validator.py:110] : Predicted data shape is (219586, 2)
DEBUG - 2023-04-13 14:21:14,403 [validator.py:158] : shapes: df_feature_val=(441941, 20), df_all_sky_val=(441941, 15)
INFO - 2023-04-13 14:21:14,434 [validator.py:179] : Calculating statistics
DEBUG - 2023-04-13 14:21:14,437 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:21:14,437 [validator.py:339] : Loading data for 2016 / east
DEBUG - 2023-04-13 14:21:14,437 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:21:14,470 [validator.py:377] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2023-04-13 14:21:15,090 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:21:15,122 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2023-04-13 14:21:15,749 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:21:15,783 [validator.py:377] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2023-04-13 14:21:16,412 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:21:16,445 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2023-04-13 14:21:17,072 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:21:17,105 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2023-04-13 14:21:17,735 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:21:17,769 [validator.py:377] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2023-04-13 14:21:18,397 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:21:18,431 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2023-04-13 14:21:19,059 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:21:19,091 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2023-04-13 14:21:19,716 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:21:19,748 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2023-04-13 14:21:20,373 [validator.py:339] : Loading data for 2017 / east
DEBUG - 2023-04-13 14:21:20,373 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:21:20,404 [validator.py:377] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2023-04-13 14:21:21,034 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:21:21,065 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2023-04-13 14:21:21,700 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:21:21,734 [validator.py:377] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2023-04-13 14:21:22,366 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:21:22,398 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2023-04-13 14:21:23,027 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:21:23,059 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2023-04-13 14:21:23,689 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:21:23,721 [validator.py:377] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2023-04-13 14:21:24,351 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:21:24,385 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2023-04-13 14:21:25,015 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:21:25,046 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2023-04-13 14:21:25,672 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:21:25,704 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2023-04-13 14:21:26,337 [validator.py:339] : Loading data for 2018 / east
DEBUG - 2023-04-13 14:21:26,337 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:21:26,407 [validator.py:377] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2023-04-13 14:21:27,055 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:21:27,126 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2023-04-13 14:21:27,774 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:21:27,845 [validator.py:377] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2023-04-13 14:21:28,490 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:21:28,561 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2023-04-13 14:21:29,209 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:21:29,280 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2023-04-13 14:21:29,925 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:21:29,996 [validator.py:377] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2023-04-13 14:21:30,645 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:21:30,716 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2023-04-13 14:21:31,369 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:21:31,440 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2023-04-13 14:21:32,086 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:21:32,157 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2023-04-13 14:21:32,810 [validator.py:339] : Loading data for 2019 / east
DEBUG - 2023-04-13 14:21:32,810 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:21:32,882 [validator.py:377] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2023-04-13 14:21:33,538 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:21:33,609 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2023-04-13 14:21:34,282 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:21:34,353 [validator.py:377] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2023-04-13 14:21:35,007 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:21:35,079 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2023-04-13 14:21:35,746 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:21:35,817 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2023-04-13 14:21:36,479 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:21:36,551 [validator.py:377] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2023-04-13 14:21:37,217 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:21:37,289 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2023-04-13 14:21:37,959 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:21:38,031 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2023-04-13 14:21:38,703 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:21:38,774 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2023-04-13 14:21:39,455 [validator.py:187] : Shapes: df_base_full=(2207952, 6), df_surf_full=(2207952, 4)
DEBUG - 2023-04-13 14:21:39,486 [validator.py:194] : Test set shapes: df_base_full=(441941, 6), df_surf_full=(441941, 4)
DEBUG - 2023-04-13 14:21:39,489 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2023-04-13 14:21:39,494 [validator.py:209] : Shapes: df_baseline=(48999, 6), df_surf=(48999, 4)
DEBUG - 2023-04-13 14:21:43,164 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2023-04-13 14:21:43,169 [validator.py:209] : Shapes: df_baseline=(49331, 6), df_surf=(49331, 4)
DEBUG - 2023-04-13 14:21:46,828 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2023-04-13 14:21:46,834 [validator.py:209] : Shapes: df_baseline=(49184, 6), df_surf=(49184, 4)
DEBUG - 2023-04-13 14:21:50,479 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2023-04-13 14:21:50,484 [validator.py:209] : Shapes: df_baseline=(49164, 6), df_surf=(49164, 4)
DEBUG - 2023-04-13 14:21:54,139 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2023-04-13 14:21:54,144 [validator.py:209] : Shapes: df_baseline=(49354, 6), df_surf=(49354, 4)
DEBUG - 2023-04-13 14:21:57,819 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2023-04-13 14:21:57,825 [validator.py:209] : Shapes: df_baseline=(49208, 6), df_surf=(49208, 4)
DEBUG - 2023-04-13 14:22:01,496 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2023-04-13 14:22:01,502 [validator.py:209] : Shapes: df_baseline=(48837, 6), df_surf=(48837, 4)
DEBUG - 2023-04-13 14:22:05,153 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2023-04-13 14:22:05,158 [validator.py:209] : Shapes: df_baseline=(48932, 6), df_surf=(48932, 4)
DEBUG - 2023-04-13 14:22:08,818 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2023-04-13 14:22:08,823 [validator.py:209] : Shapes: df_baseline=(48932, 6), df_surf=(48932, 4)
INFO - 2023-04-13 14:22:12,467 [validator.py:285] : Finished computing stats.
Validating on 188952 out of 946512 observations
DEBUG - 2023-04-13 14:22:13,107 [data_handlers.py:391] : Loading validation data
DEBUG - 2023-04-13 14:22:13,107 [data_handlers.py:397] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2023-04-13 14:22:13,110 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:22:14,825 [data_handlers.py:421] : 	Shape temp_raw=(158112, 19), temp_all_sky=(158112, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:22:14,825 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:22:16,570 [data_handlers.py:421] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:22:16,571 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:22:18,311 [data_handlers.py:421] : 	Shape temp_raw=(157680, 19), temp_all_sky=(157680, 14), & tstep=30 minutes
DEBUG - 2023-04-13 14:22:18,311 [data_handlers.py:402] : Loading validation data from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5 for gids [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:22:23,587 [data_handlers.py:421] : 	Shape temp_raw=(473040, 19), temp_all_sky=(473040, 14), & tstep=10 minutes
DEBUG - 2023-04-13 14:22:23,587 [data_handlers.py:424] : Shape df_raw=(946512, 19), df_all_sky=(946512, 14)
DEBUG - 2023-04-13 14:22:23,587 [data_handlers.py:431] : Shape after reset_index: df_raw=(946512, 19), df_all_sky=(946512, 14)
DEBUG - 2023-04-13 14:22:23,619 [data_handlers.py:441] : Test set shape: df_raw=(188952, 19), df_all_sky=(188952, 14)
INFO - 2023-04-13 14:22:23,652 [data_cleaners.py:36] : 49.67% of timesteps are daylight
INFO - 2023-04-13 14:22:23,652 [data_cleaners.py:38] : 55.35% of daylight timesteps are cloudy
INFO - 2023-04-13 14:22:23,653 [data_cleaners.py:40] : 10.00% of daylight timesteps are missing cloud type
INFO - 2023-04-13 14:22:23,653 [data_cleaners.py:42] : 49.12% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 14:22:23,653 [data_cleaners.py:44] : 49.24% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 14:22:23,653 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 14:22:23,654 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,654 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,654 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,655 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,655 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 54.80% NaN values
DEBUG - 2023-04-13 14:22:23,656 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 54.80% NaN values
DEBUG - 2023-04-13 14:22:23,656 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 10.10% NaN values
DEBUG - 2023-04-13 14:22:23,656 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 10.00% NaN values
DEBUG - 2023-04-13 14:22:23,657 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 9.99% NaN values
DEBUG - 2023-04-13 14:22:23,657 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 10.08% NaN values
DEBUG - 2023-04-13 14:22:23,657 [data_cleaners.py:50] : 	"cloud_probability" has 10.08% NaN values
DEBUG - 2023-04-13 14:22:23,658 [data_cleaners.py:50] : 	"cloud_fraction" has 10.08% NaN values
DEBUG - 2023-04-13 14:22:23,658 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,659 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,659 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,659 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,660 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,660 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 86.01% NaN values
DEBUG - 2023-04-13 14:22:23,660 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 86.05% NaN values
DEBUG - 2023-04-13 14:22:23,660 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 14:22:23,785 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2023-04-13 14:22:23,819 [data_cleaners.py:105] : Feature flag column has these values: ['water_cloud' 'night' 'bad_cloud' 'clear' 'ice_cloud']
INFO - 2023-04-13 14:22:23,819 [data_cleaners.py:107] : Cleaning took 0.2 seconds
INFO - 2023-04-13 14:22:23,847 [data_cleaners.py:36] : 49.67% of timesteps are daylight
INFO - 2023-04-13 14:22:23,847 [data_cleaners.py:38] : 55.35% of daylight timesteps are cloudy
INFO - 2023-04-13 14:22:23,847 [data_cleaners.py:40] : 10.00% of daylight timesteps are missing cloud type
INFO - 2023-04-13 14:22:23,848 [data_cleaners.py:42] : 49.12% of cloudy daylight timesteps are missing cloud opd
INFO - 2023-04-13 14:22:23,848 [data_cleaners.py:44] : 49.24% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2023-04-13 14:22:23,848 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2023-04-13 14:22:23,848 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,849 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,849 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,850 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,850 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,850 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,851 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 86.01% NaN values
DEBUG - 2023-04-13 14:22:23,851 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 86.05% NaN values
DEBUG - 2023-04-13 14:22:23,851 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,852 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,852 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,853 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,853 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,853 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2023-04-13 14:22:23,853 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2023-04-13 14:22:23,920 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
DEBUG - 2023-04-13 14:22:23,954 [data_cleaners.py:105] : Feature flag column has these values: ['water_cloud' 'night' 'bad_cloud' 'clear' 'ice_cloud']
INFO - 2023-04-13 14:22:23,954 [data_cleaners.py:107] : Cleaning took 0.1 seconds
DEBUG - 2023-04-13 14:22:23,954 [data_handlers.py:464] : Prepping validation data
DEBUG - 2023-04-13 14:22:23,963 [data_handlers.py:474] : Mask: shape=(188952,), sum=93851
DEBUG - 2023-04-13 14:22:23,971 [data_handlers.py:485] : Validation features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
INFO - 2023-04-13 14:22:23,971 [validator.py:107] : Predicting opd and reff
/projects/pxs/gbuster/nsrdb_env_2023/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.
  warnings.warn(
DEBUG - 2023-04-13 14:22:24,202 [validator.py:110] : Predicted data shape is (93851, 2)
DEBUG - 2023-04-13 14:22:24,225 [validator.py:158] : shapes: df_feature_val=(188952, 20), df_all_sky_val=(188952, 15)
INFO - 2023-04-13 14:22:24,238 [validator.py:179] : Calculating statistics
DEBUG - 2023-04-13 14:22:24,242 [validator.py:182] : Calcing stats for gids: [0, 1, 2, 3, 4, 5, 6, 7, 8]
DEBUG - 2023-04-13 14:22:24,242 [validator.py:339] : Loading data for 2016 / west
DEBUG - 2023-04-13 14:22:24,242 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:22:24,275 [validator.py:377] : 	Getting surfrad data for 0 from bon_2016.h5
DEBUG - 2023-04-13 14:22:24,900 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:22:24,931 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2016.h5
DEBUG - 2023-04-13 14:22:25,560 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:22:25,594 [validator.py:377] : 	Getting surfrad data for 2 from dra_2016.h5
DEBUG - 2023-04-13 14:22:26,225 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:22:26,259 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2016.h5
DEBUG - 2023-04-13 14:22:26,889 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:22:26,923 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2016.h5
DEBUG - 2023-04-13 14:22:27,557 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:22:27,592 [validator.py:377] : 	Getting surfrad data for 5 from psu_2016.h5
DEBUG - 2023-04-13 14:22:28,225 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:22:28,259 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2016.h5
DEBUG - 2023-04-13 14:22:28,893 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:22:28,926 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2016.h5
DEBUG - 2023-04-13 14:22:29,553 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:22:29,586 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2016.h5
DEBUG - 2023-04-13 14:22:30,221 [validator.py:339] : Loading data for 2017 / west
DEBUG - 2023-04-13 14:22:30,221 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:22:30,255 [validator.py:377] : 	Getting surfrad data for 0 from bon_2017.h5
DEBUG - 2023-04-13 14:22:30,890 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:22:30,923 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2017.h5
DEBUG - 2023-04-13 14:22:31,557 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:22:31,589 [validator.py:377] : 	Getting surfrad data for 2 from dra_2017.h5
DEBUG - 2023-04-13 14:22:32,226 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:22:32,258 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2017.h5
DEBUG - 2023-04-13 14:22:32,894 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:22:32,926 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2017.h5
DEBUG - 2023-04-13 14:22:33,561 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:22:33,591 [validator.py:377] : 	Getting surfrad data for 5 from psu_2017.h5
DEBUG - 2023-04-13 14:22:34,225 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:22:34,256 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2017.h5
DEBUG - 2023-04-13 14:22:34,893 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:22:34,925 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2017.h5
DEBUG - 2023-04-13 14:22:35,555 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:22:35,587 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2017.h5
DEBUG - 2023-04-13 14:22:36,221 [validator.py:339] : Loading data for 2018 / west
DEBUG - 2023-04-13 14:22:36,222 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:22:36,254 [validator.py:377] : 	Getting surfrad data for 0 from bon_2018.h5
DEBUG - 2023-04-13 14:22:36,888 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:22:36,920 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2018.h5
DEBUG - 2023-04-13 14:22:37,555 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:22:37,587 [validator.py:377] : 	Getting surfrad data for 2 from dra_2018.h5
DEBUG - 2023-04-13 14:22:38,225 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:22:38,256 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2018.h5
DEBUG - 2023-04-13 14:22:38,892 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:22:38,923 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2018.h5
DEBUG - 2023-04-13 14:22:39,557 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:22:39,589 [validator.py:377] : 	Getting surfrad data for 5 from psu_2018.h5
DEBUG - 2023-04-13 14:22:40,223 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:22:40,255 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2018.h5
DEBUG - 2023-04-13 14:22:40,892 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:22:40,924 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2018.h5
DEBUG - 2023-04-13 14:22:41,557 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:22:41,589 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2018.h5
DEBUG - 2023-04-13 14:22:42,226 [validator.py:339] : Loading data for 2019 / west
DEBUG - 2023-04-13 14:22:42,226 [validator.py:414] : 	Getting gid 0 from *.h5
DEBUG - 2023-04-13 14:22:42,272 [validator.py:377] : 	Getting surfrad data for 0 from bon_2019.h5
DEBUG - 2023-04-13 14:22:42,908 [validator.py:414] : 	Getting gid 1 from *.h5
DEBUG - 2023-04-13 14:22:42,955 [validator.py:377] : 	Getting surfrad data for 1 from tbl_2019.h5
DEBUG - 2023-04-13 14:22:43,602 [validator.py:414] : 	Getting gid 2 from *.h5
DEBUG - 2023-04-13 14:22:43,649 [validator.py:377] : 	Getting surfrad data for 2 from dra_2019.h5
DEBUG - 2023-04-13 14:22:44,295 [validator.py:414] : 	Getting gid 3 from *.h5
DEBUG - 2023-04-13 14:22:44,342 [validator.py:377] : 	Getting surfrad data for 3 from fpk_2019.h5
DEBUG - 2023-04-13 14:22:44,984 [validator.py:414] : 	Getting gid 4 from *.h5
DEBUG - 2023-04-13 14:22:45,030 [validator.py:377] : 	Getting surfrad data for 4 from gwn_2019.h5
DEBUG - 2023-04-13 14:22:45,671 [validator.py:414] : 	Getting gid 5 from *.h5
DEBUG - 2023-04-13 14:22:45,718 [validator.py:377] : 	Getting surfrad data for 5 from psu_2019.h5
DEBUG - 2023-04-13 14:22:46,350 [validator.py:414] : 	Getting gid 6 from *.h5
DEBUG - 2023-04-13 14:22:46,398 [validator.py:377] : 	Getting surfrad data for 6 from sxf_2019.h5
DEBUG - 2023-04-13 14:22:47,043 [validator.py:414] : 	Getting gid 7 from *.h5
DEBUG - 2023-04-13 14:22:47,090 [validator.py:377] : 	Getting surfrad data for 7 from sgp_2019.h5
DEBUG - 2023-04-13 14:22:47,730 [validator.py:414] : 	Getting gid 8 from *.h5
DEBUG - 2023-04-13 14:22:47,780 [validator.py:377] : 	Getting surfrad data for 8 from srrl_2019.h5
DEBUG - 2023-04-13 14:22:48,430 [validator.py:187] : Shapes: df_base_full=(946512, 6), df_surf_full=(946512, 4)
DEBUG - 2023-04-13 14:22:48,445 [validator.py:194] : Test set shapes: df_base_full=(188952, 6), df_surf_full=(188952, 4)
DEBUG - 2023-04-13 14:22:48,449 [validator.py:203] : Computing stats for gid: 0 bon
DEBUG - 2023-04-13 14:22:48,451 [validator.py:209] : Shapes: df_baseline=(21127, 6), df_surf=(21127, 4)
DEBUG - 2023-04-13 14:22:51,044 [validator.py:203] : Computing stats for gid: 1 tbl
DEBUG - 2023-04-13 14:22:51,047 [validator.py:209] : Shapes: df_baseline=(20964, 6), df_surf=(20964, 4)
DEBUG - 2023-04-13 14:22:53,624 [validator.py:203] : Computing stats for gid: 2 dra
DEBUG - 2023-04-13 14:22:53,627 [validator.py:209] : Shapes: df_baseline=(20878, 6), df_surf=(20878, 4)
DEBUG - 2023-04-13 14:22:56,199 [validator.py:203] : Computing stats for gid: 3 fpk
DEBUG - 2023-04-13 14:22:56,202 [validator.py:209] : Shapes: df_baseline=(21052, 6), df_surf=(21052, 4)
DEBUG - 2023-04-13 14:22:58,783 [validator.py:203] : Computing stats for gid: 4 gwn
DEBUG - 2023-04-13 14:22:58,785 [validator.py:209] : Shapes: df_baseline=(20842, 6), df_surf=(20842, 4)
DEBUG - 2023-04-13 14:23:01,362 [validator.py:203] : Computing stats for gid: 5 psu
DEBUG - 2023-04-13 14:23:01,365 [validator.py:209] : Shapes: df_baseline=(20901, 6), df_surf=(20901, 4)
DEBUG - 2023-04-13 14:23:03,946 [validator.py:203] : Computing stats for gid: 6 sxf
DEBUG - 2023-04-13 14:23:03,949 [validator.py:209] : Shapes: df_baseline=(21161, 6), df_surf=(21161, 4)
DEBUG - 2023-04-13 14:23:06,536 [validator.py:203] : Computing stats for gid: 7 sgp
DEBUG - 2023-04-13 14:23:06,539 [validator.py:209] : Shapes: df_baseline=(20930, 6), df_surf=(20930, 4)
DEBUG - 2023-04-13 14:23:09,118 [validator.py:203] : Computing stats for gid: 8 srrl
DEBUG - 2023-04-13 14:23:09,120 [validator.py:209] : Shapes: df_baseline=(21097, 6), df_surf=(21097, 4)
INFO - 2023-04-13 14:23:11,701 [validator.py:285] : Finished computing stats.
