Starting scenario 4, validation against site 8
2022-03-01 13:26:46.536910: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:26:46.536939: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Cross validation site: 8
Training sites: [0, 1, 2, 3, 4, 5, 6, 7]
Number of files: 8
Number of east files: 4
Number of west files: 4
Source files: ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
Full config: {'clean_training_data_kwargs': {'filter_clear': False, 'nan_option': 'interp'}, 'epochs_a': 100, 'epochs_b': 100, 'features': ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo'], 'hidden_layers': [{'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}, {'activation': 'relu', 'dropout': 0.1, 'units': 256}], 'learning_rate': 0.0005, 'loss_weights_a': [1, 0], 'loss_weights_b': [0.5, 0.5], 'metric': 'relative_mae', 'n_batch': 32, 'one_hot_categories': {'flag': ['clear', 'ice_cloud', 'water_cloud', 'bad_cloud']}, 'p_fun': 'p_fun_all_sky', 'p_kwargs': {'loss_terms': ['mae_ghi']}, 'phygnn_seed': 0, 'surfrad_window_minutes': 15, 'y_labels': ['cld_opd_dcomp', 'cld_reff_dcomp']}
INFO - 2022-03-01 13:26:56,349 [trainer.py:40] : Trainer: Training on sites [0, 1, 2, 3, 4, 5, 6, 7] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:26:56,349 [trainer.py:49] : Trainer: Training on sites [0, 1, 2, 3, 4, 5, 6, 7] from files ['/projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5', '/projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5', '/projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5', '/projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5', '/projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5', '/projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5', '/projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5', '/projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5']
INFO - 2022-03-01 13:26:56,350 [data_handlers.py:60] : Loading training data
DEBUG - 2022-03-01 13:26:56,350 [data_handlers.py:78] : Loading vars ['solar_zenith_angle', 'cloud_type', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'cld_opd_dcomp', 'cld_reff_dcomp']
DEBUG - 2022-03-01 13:26:56,350 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2016_east_v322/mlclouds_surfrad_east_2016.h5
DEBUG - 2022-03-01 13:26:57,402 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:26:57,405 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:26:57,406 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:26:57,411 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:26:58,080 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:58,083 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:26:58,729 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:58,732 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:26:59,381 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:26:59,384 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:27:00,041 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:00,045 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:27:00,693 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:00,696 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:27:01,346 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:01,349 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:27:01,998 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:02,001 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:27:02,912 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:02,912 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2016_west_v322/mlclouds_surfrad_west_2016.h5
DEBUG - 2022-03-01 13:27:03,898 [data_handlers.py:103] : 	Shape temp_raw=(140544, 19), temp_all_sky=(140544, 14)
DEBUG - 2022-03-01 13:27:03,902 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:27:03,902 [data_handlers.py:110] : 	Grabbing surface data for 2016 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:27:03,905 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2016.h5
DEBUG - 2022-03-01 13:27:04,533 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:04,536 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2016.h5
DEBUG - 2022-03-01 13:27:05,163 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:05,166 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2016.h5
DEBUG - 2022-03-01 13:27:05,797 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:05,800 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2016.h5
DEBUG - 2022-03-01 13:27:06,447 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:06,450 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2016.h5
DEBUG - 2022-03-01 13:27:07,084 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:07,087 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2016.h5
DEBUG - 2022-03-01 13:27:07,718 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:07,721 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2016.h5
DEBUG - 2022-03-01 13:27:08,352 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:08,355 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2016.h5
DEBUG - 2022-03-01 13:27:08,986 [data_handlers.py:134] : 	Shape: temp_surf=(17568, 5)
DEBUG - 2022-03-01 13:27:08,986 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2017_east_v322/mlclouds_surfrad_east_2017.h5
DEBUG - 2022-03-01 13:27:10,007 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:27:10,011 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:27:10,011 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:27:10,014 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:27:10,653 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:10,656 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:27:11,285 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:11,288 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:27:11,917 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:11,920 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:27:12,549 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:12,552 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:27:13,188 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:13,191 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:27:13,834 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:13,837 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:27:14,476 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:14,479 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:27:15,171 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:15,171 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2017_west_v322/mlclouds_surfrad_west_2017.h5
DEBUG - 2022-03-01 13:27:16,179 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:27:16,183 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:27:16,183 [data_handlers.py:110] : 	Grabbing surface data for 2017 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:27:16,186 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2017.h5
DEBUG - 2022-03-01 13:27:16,821 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:16,824 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2017.h5
DEBUG - 2022-03-01 13:27:17,440 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:17,443 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2017.h5
DEBUG - 2022-03-01 13:27:18,058 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:18,061 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2017.h5
DEBUG - 2022-03-01 13:27:18,674 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:18,677 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2017.h5
DEBUG - 2022-03-01 13:27:19,289 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:19,292 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2017.h5
DEBUG - 2022-03-01 13:27:19,911 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:19,914 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2017.h5
DEBUG - 2022-03-01 13:27:20,525 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:20,528 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2017.h5
DEBUG - 2022-03-01 13:27:21,143 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:21,143 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2018_east_v322/mlclouds_surfrad_east_2018.h5
DEBUG - 2022-03-01 13:27:26,823 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:27:26,840 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:27:26,840 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:27:26,843 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:27:27,477 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:27,480 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:27:28,328 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:28,331 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:27:28,971 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:28,974 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:27:29,670 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:29,674 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:27:30,395 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:30,398 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:27:31,068 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:31,071 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:27:31,755 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:31,758 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:27:32,419 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:32,419 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2018_west_v322/mlclouds_surfrad_west_2018.h5
DEBUG - 2022-03-01 13:27:33,595 [data_handlers.py:103] : 	Shape temp_raw=(140160, 19), temp_all_sky=(140160, 14)
DEBUG - 2022-03-01 13:27:33,598 [data_handlers.py:106] : 	Time step is 30 minutes
DEBUG - 2022-03-01 13:27:33,599 [data_handlers.py:110] : 	Grabbing surface data for 2018 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:27:33,602 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2018.h5
DEBUG - 2022-03-01 13:27:34,230 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:34,233 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2018.h5
DEBUG - 2022-03-01 13:27:34,853 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:34,857 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2018.h5
DEBUG - 2022-03-01 13:27:35,484 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:35,487 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2018.h5
DEBUG - 2022-03-01 13:27:36,108 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:36,111 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2018.h5
DEBUG - 2022-03-01 13:27:36,741 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:36,744 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2018.h5
DEBUG - 2022-03-01 13:27:37,365 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:37,368 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2018.h5
DEBUG - 2022-03-01 13:27:37,997 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:38,000 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2018.h5
DEBUG - 2022-03-01 13:27:38,624 [data_handlers.py:134] : 	Shape: temp_surf=(17520, 5)
DEBUG - 2022-03-01 13:27:38,624 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2019_east_v322/mlclouds_surfrad_east_2019.h5
DEBUG - 2022-03-01 13:27:44,500 [data_handlers.py:103] : 	Shape temp_raw=(840960, 19), temp_all_sky=(840960, 14)
DEBUG - 2022-03-01 13:27:44,517 [data_handlers.py:106] : 	Time step is 5 minutes
DEBUG - 2022-03-01 13:27:44,517 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:27:44,520 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:27:45,172 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:45,175 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:27:45,895 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:45,898 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:27:46,539 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:46,542 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:27:47,186 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:47,189 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:27:47,837 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:47,841 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:27:48,527 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:48,530 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:27:49,185 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:49,188 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:27:49,951 [data_handlers.py:134] : 	Shape: temp_surf=(105120, 5)
DEBUG - 2022-03-01 13:27:49,951 [data_handlers.py:85] : Loading data for site(s) [0, 1, 2, 3, 4, 5, 6, 7], from /projects/pxs/mlclouds/training_data/2019_west_v322/mlclouds_surfrad_west_2019.h5
DEBUG - 2022-03-01 13:27:52,952 [data_handlers.py:103] : 	Shape temp_raw=(420480, 19), temp_all_sky=(420480, 14)
DEBUG - 2022-03-01 13:27:52,961 [data_handlers.py:106] : 	Time step is 10 minutes
DEBUG - 2022-03-01 13:27:52,961 [data_handlers.py:110] : 	Grabbing surface data for 2019 and [0, 1, 2, 3, 4, 5, 6, 7]
DEBUG - 2022-03-01 13:27:52,964 [data_handlers.py:117] : 		Grabbing surface data for bon from /projects/pxs/surfrad/h5/bon_2019.h5
DEBUG - 2022-03-01 13:27:53,598 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:53,601 [data_handlers.py:117] : 		Grabbing surface data for tbl from /projects/pxs/surfrad/h5/tbl_2019.h5
DEBUG - 2022-03-01 13:27:54,227 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:54,230 [data_handlers.py:117] : 		Grabbing surface data for dra from /projects/pxs/surfrad/h5/dra_2019.h5
DEBUG - 2022-03-01 13:27:54,873 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:54,877 [data_handlers.py:117] : 		Grabbing surface data for fpk from /projects/pxs/surfrad/h5/fpk_2019.h5
DEBUG - 2022-03-01 13:27:55,503 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:55,505 [data_handlers.py:117] : 		Grabbing surface data for gwn from /projects/pxs/surfrad/h5/gwn_2019.h5
DEBUG - 2022-03-01 13:27:56,148 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:56,151 [data_handlers.py:117] : 		Grabbing surface data for psu from /projects/pxs/surfrad/h5/psu_2019.h5
DEBUG - 2022-03-01 13:27:56,780 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:56,783 [data_handlers.py:117] : 		Grabbing surface data for sxf from /projects/pxs/surfrad/h5/sxf_2019.h5
DEBUG - 2022-03-01 13:27:57,425 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:57,428 [data_handlers.py:117] : 		Grabbing surface data for sgp from /projects/pxs/surfrad/h5/sgp_2019.h5
DEBUG - 2022-03-01 13:27:58,074 [data_handlers.py:134] : 	Shape: temp_surf=(52560, 5)
DEBUG - 2022-03-01 13:27:58,074 [data_handlers.py:136] : Data load complete. Shape df_raw=(2803968, 19), df_all_sky=(2803968, 14), df_surf=(2803968, 5)
DEBUG - 2022-03-01 13:27:58,879 [data_handlers.py:159] : Extracting 2D arrays to run rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:28:10,437 [data_handlers.py:176] : Running rest2 for clearsky PhyGNN inputs.
DEBUG - 2022-03-01 13:29:58,441 [data_handlers.py:194] : Completed rest2 run for clearsky PhyGNN inputs.
INFO - 2022-03-01 13:29:59,894 [data_handlers.py:62] : Prepping training data
DEBUG - 2022-03-01 13:29:59,894 [data_handlers.py:214] : Training data clean kwargs: {'filter_daylight': True, 'filter_clear': False, 'add_cloud_flag': True, 'sza_lim': 89, 'nan_option': 'interp'}
DEBUG - 2022-03-01 13:29:59,894 [data_handlers.py:215] : Shape before cleaning: df_raw=(2803968, 19)
INFO - 2022-03-01 13:30:00,151 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:30:00,156 [data_cleaners.py:38] : 52.30% of daylight timesteps are cloudy
INFO - 2022-03-01 13:30:00,160 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:30:00,165 [data_cleaners.py:42] : 34.53% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:30:00,169 [data_cleaners.py:44] : 34.75% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:30:00,169 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:30:00,172 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,176 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,180 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,184 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,188 [data_cleaners.py:50] : 	"refl_0_65um_nom" has 51.62% NaN values
DEBUG - 2022-03-01 13:30:00,191 [data_cleaners.py:50] : 	"refl_0_65um_nom_stddev_3x3" has 51.62% NaN values
DEBUG - 2022-03-01 13:30:00,195 [data_cleaners.py:50] : 	"refl_3_75um_nom" has 3.63% NaN values
DEBUG - 2022-03-01 13:30:00,198 [data_cleaners.py:50] : 	"temp_3_75um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:30:00,202 [data_cleaners.py:50] : 	"temp_11_0um_nom" has 3.53% NaN values
DEBUG - 2022-03-01 13:30:00,205 [data_cleaners.py:50] : 	"temp_11_0um_nom_stddev_3x3" has 3.61% NaN values
DEBUG - 2022-03-01 13:30:00,209 [data_cleaners.py:50] : 	"cloud_probability" has 3.61% NaN values
DEBUG - 2022-03-01 13:30:00,212 [data_cleaners.py:50] : 	"cloud_fraction" has 3.61% NaN values
DEBUG - 2022-03-01 13:30:00,215 [data_cleaners.py:50] : 	"air_temperature" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,219 [data_cleaners.py:50] : 	"dew_point" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,222 [data_cleaners.py:50] : 	"relative_humidity" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,226 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,229 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:00,233 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.99% NaN values
DEBUG - 2022-03-01 13:30:00,236 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.04% NaN values
DEBUG - 2022-03-01 13:30:00,236 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:30:03,034 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:30:03,258 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393350 after filters (49.69% of original)
DEBUG - 2022-03-01 13:30:03,364 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:30:03,364 [data_cleaners.py:107] : Cleaning took 3.5 seconds
DEBUG - 2022-03-01 13:30:03,364 [data_handlers.py:218] : Shape after cleaning: df_train=(1393350, 20)
DEBUG - 2022-03-01 13:30:03,364 [data_handlers.py:221] : Cleaning df_all_sky training data (for pfun).
DEBUG - 2022-03-01 13:30:03,364 [data_handlers.py:222] : Shape before cleaning: df_all_sky=(2803968, 25)
INFO - 2022-03-01 13:30:03,657 [data_cleaners.py:36] : 49.69% of timesteps are daylight
INFO - 2022-03-01 13:30:03,661 [data_cleaners.py:38] : 52.30% of daylight timesteps are cloudy
INFO - 2022-03-01 13:30:03,666 [data_cleaners.py:40] : 3.59% of daylight timesteps are missing cloud type
INFO - 2022-03-01 13:30:03,671 [data_cleaners.py:42] : 34.53% of cloudy daylight timesteps are missing cloud opd
INFO - 2022-03-01 13:30:03,675 [data_cleaners.py:44] : 34.75% of cloudy daylight timesteps are missing cloud reff
DEBUG - 2022-03-01 13:30:03,675 [data_cleaners.py:47] : Column NaN values:
DEBUG - 2022-03-01 13:30:03,678 [data_cleaners.py:50] : 	"gid" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,681 [data_cleaners.py:50] : 	"alpha" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,685 [data_cleaners.py:50] : 	"aod" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,688 [data_cleaners.py:50] : 	"asymmetry" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,693 [data_cleaners.py:50] : 	"cloud_type" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,696 [data_cleaners.py:50] : 	"cld_opd_dcomp" has 82.99% NaN values
DEBUG - 2022-03-01 13:30:03,700 [data_cleaners.py:50] : 	"cld_reff_dcomp" has 83.04% NaN values
DEBUG - 2022-03-01 13:30:03,703 [data_cleaners.py:50] : 	"ozone" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,707 [data_cleaners.py:50] : 	"solar_zenith_angle" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,710 [data_cleaners.py:50] : 	"ssa" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,714 [data_cleaners.py:50] : 	"surface_albedo" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,716 [data_cleaners.py:50] : 	"surface_pressure" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,720 [data_cleaners.py:50] : 	"total_precipitable_water" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,724 [data_cleaners.py:50] : 	"surfrad_dhi" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,728 [data_cleaners.py:50] : 	"surfrad_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,733 [data_cleaners.py:50] : 	"surfrad_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,735 [data_cleaners.py:50] : 	"doy" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,740 [data_cleaners.py:50] : 	"radius" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,744 [data_cleaners.py:50] : 	"Tuuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,748 [data_cleaners.py:50] : 	"clearsky_ghi" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,753 [data_cleaners.py:50] : 	"clearsky_dni" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,757 [data_cleaners.py:50] : 	"Ruuclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,762 [data_cleaners.py:50] : 	"Tddclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,766 [data_cleaners.py:50] : 	"Tduclr" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,770 [data_cleaners.py:50] : 	"time_index" has 0.00% NaN values
DEBUG - 2022-03-01 13:30:03,770 [data_cleaners.py:53] : Interpolating opd and reff
DEBUG - 2022-03-01 13:30:06,050 [data_cleaners.py:79] : Adding cloud type flag (e.g. flag=[night, clear, ice_cloud, water_cloud, bad_cloud])
INFO - 2022-03-01 13:30:06,273 [data_cleaners.py:99] : Data reduced from 2803968 rows to 1393350 after filters (49.69% of original)
DEBUG - 2022-03-01 13:30:06,422 [data_cleaners.py:105] : Feature flag column has these values: ['clear' 'bad_cloud' 'water_cloud' 'ice_cloud']
INFO - 2022-03-01 13:30:06,423 [data_cleaners.py:107] : Cleaning took 3.1 seconds
DEBUG - 2022-03-01 13:30:06,423 [data_handlers.py:226] : Shape after cleaning: df_all_sky=(1393350, 26)
DEBUG - 2022-03-01 13:30:06,504 [data_handlers.py:240] : **Shape: df_train=(1393350, 17)
DEBUG - 2022-03-01 13:30:06,527 [data_handlers.py:250] : Shapes: x=(1393350, 15), y=(1393350, 2), p=(1393350, 26)
DEBUG - 2022-03-01 13:30:06,527 [data_handlers.py:253] : Training features: ['solar_zenith_angle', 'refl_0_65um_nom', 'refl_0_65um_nom_stddev_3x3', 'refl_3_75um_nom', 'temp_3_75um_nom', 'temp_11_0um_nom', 'temp_11_0um_nom_stddev_3x3', 'cloud_probability', 'cloud_fraction', 'air_temperature', 'dew_point', 'relative_humidity', 'total_precipitable_water', 'surface_albedo', 'flag']
DEBUG - 2022-03-01 13:30:06,527 [trainer.py:67] : Building PHYGNN model
INFO - 2022-03-01 13:30:06,527 [trainer.py:70] : Using p_fun: <function p_fun_all_sky at 0x2b12bea618b0>
INFO - 2022-03-01 13:30:06,528 [base.py:152] : Active python environment versions: 
{   'numpy': '1.22.2',
    'pandas': '1.2.4',
    'phygnn': '0.0.14',
    'python': '3.8.8 (default, Feb 24 2021, 21:46:12) \n[GCC 7.3.0]',
    'sklearn': '0.24.1',
    'tensorflow': '2.8.0'}
INFO - 2022-03-01 13:30:06,543 [base.py:111] : Successfully initialized model with 17 layers
INFO - 2022-03-01 13:30:06,543 [trainer.py:84] : Training part A - pure data. Loss is [1, 0]
2022-03-01 13:30:14.349249: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nopt/slurm/current/lib:/nopt/slurm/current/lib:
2022-03-01 13:30:14.349281: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-03-01 13:30:14.349300: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (r7i0n5): /proc/driver/nvidia/version does not exist
2022-03-01 13:30:14.349611: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO - 2022-03-01 13:30:21,573 [phygnn.py:576] : Epoch 0 train loss: 6.98e-01 val loss: 6.89e-01 for "phygnn"
INFO - 2022-03-01 13:30:29,292 [phygnn.py:576] : Epoch 1 train loss: 6.35e-01 val loss: 6.27e-01 for "phygnn"
INFO - 2022-03-01 13:30:37,081 [phygnn.py:576] : Epoch 2 train loss: 5.64e-01 val loss: 5.55e-01 for "phygnn"
INFO - 2022-03-01 13:30:45,065 [phygnn.py:576] : Epoch 3 train loss: 5.38e-01 val loss: 5.20e-01 for "phygnn"
INFO - 2022-03-01 13:30:52,484 [phygnn.py:576] : Epoch 4 train loss: 5.15e-01 val loss: 4.99e-01 for "phygnn"
INFO - 2022-03-01 13:30:59,799 [phygnn.py:576] : Epoch 5 train loss: 5.10e-01 val loss: 4.91e-01 for "phygnn"
INFO - 2022-03-01 13:31:07,175 [phygnn.py:576] : Epoch 6 train loss: 4.95e-01 val loss: 4.83e-01 for "phygnn"
INFO - 2022-03-01 13:31:15,123 [phygnn.py:576] : Epoch 7 train loss: 4.92e-01 val loss: 4.79e-01 for "phygnn"
INFO - 2022-03-01 13:31:22,957 [phygnn.py:576] : Epoch 8 train loss: 4.95e-01 val loss: 4.74e-01 for "phygnn"
INFO - 2022-03-01 13:31:30,434 [phygnn.py:576] : Epoch 9 train loss: 4.87e-01 val loss: 4.72e-01 for "phygnn"
INFO - 2022-03-01 13:31:38,078 [phygnn.py:576] : Epoch 10 train loss: 4.87e-01 val loss: 4.68e-01 for "phygnn"
INFO - 2022-03-01 13:31:45,708 [phygnn.py:576] : Epoch 11 train loss: 4.81e-01 val loss: 4.66e-01 for "phygnn"
INFO - 2022-03-01 13:31:53,373 [phygnn.py:576] : Epoch 12 train loss: 4.83e-01 val loss: 4.64e-01 for "phygnn"
INFO - 2022-03-01 13:32:01,049 [phygnn.py:576] : Epoch 13 train loss: 4.86e-01 val loss: 4.64e-01 for "phygnn"
INFO - 2022-03-01 13:32:08,613 [phygnn.py:576] : Epoch 14 train loss: 4.77e-01 val loss: 4.61e-01 for "phygnn"
INFO - 2022-03-01 13:32:15,970 [phygnn.py:576] : Epoch 15 train loss: 4.73e-01 val loss: 4.60e-01 for "phygnn"
INFO - 2022-03-01 13:32:23,775 [phygnn.py:576] : Epoch 16 train loss: 4.68e-01 val loss: 4.58e-01 for "phygnn"
INFO - 2022-03-01 13:32:31,181 [phygnn.py:576] : Epoch 17 train loss: 4.66e-01 val loss: 4.57e-01 for "phygnn"
INFO - 2022-03-01 13:32:38,710 [phygnn.py:576] : Epoch 18 train loss: 4.69e-01 val loss: 4.55e-01 for "phygnn"
INFO - 2022-03-01 13:32:46,456 [phygnn.py:576] : Epoch 19 train loss: 4.71e-01 val loss: 4.54e-01 for "phygnn"
INFO - 2022-03-01 13:32:53,863 [phygnn.py:576] : Epoch 20 train loss: 4.63e-01 val loss: 4.51e-01 for "phygnn"
INFO - 2022-03-01 13:33:01,580 [phygnn.py:576] : Epoch 21 train loss: 4.70e-01 val loss: 4.51e-01 for "phygnn"
INFO - 2022-03-01 13:33:09,153 [phygnn.py:576] : Epoch 22 train loss: 4.68e-01 val loss: 4.49e-01 for "phygnn"
INFO - 2022-03-01 13:33:16,839 [phygnn.py:576] : Epoch 23 train loss: 4.62e-01 val loss: 4.47e-01 for "phygnn"
INFO - 2022-03-01 13:33:24,701 [phygnn.py:576] : Epoch 24 train loss: 4.60e-01 val loss: 4.48e-01 for "phygnn"
INFO - 2022-03-01 13:33:32,216 [phygnn.py:576] : Epoch 25 train loss: 4.65e-01 val loss: 4.47e-01 for "phygnn"
INFO - 2022-03-01 13:33:39,814 [phygnn.py:576] : Epoch 26 train loss: 4.54e-01 val loss: 4.43e-01 for "phygnn"
INFO - 2022-03-01 13:33:47,171 [phygnn.py:576] : Epoch 27 train loss: 4.57e-01 val loss: 4.42e-01 for "phygnn"
INFO - 2022-03-01 13:33:54,582 [phygnn.py:576] : Epoch 28 train loss: 4.54e-01 val loss: 4.41e-01 for "phygnn"
INFO - 2022-03-01 13:34:02,085 [phygnn.py:576] : Epoch 29 train loss: 4.46e-01 val loss: 4.39e-01 for "phygnn"
INFO - 2022-03-01 13:34:09,846 [phygnn.py:576] : Epoch 30 train loss: 4.46e-01 val loss: 4.38e-01 for "phygnn"
INFO - 2022-03-01 13:34:17,547 [phygnn.py:576] : Epoch 31 train loss: 4.51e-01 val loss: 4.36e-01 for "phygnn"
INFO - 2022-03-01 13:34:25,134 [phygnn.py:576] : Epoch 32 train loss: 4.50e-01 val loss: 4.36e-01 for "phygnn"
INFO - 2022-03-01 13:34:33,027 [phygnn.py:576] : Epoch 33 train loss: 4.47e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2022-03-01 13:34:40,710 [phygnn.py:576] : Epoch 34 train loss: 4.45e-01 val loss: 4.35e-01 for "phygnn"
INFO - 2022-03-01 13:34:48,269 [phygnn.py:576] : Epoch 35 train loss: 4.50e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2022-03-01 13:34:55,853 [phygnn.py:576] : Epoch 36 train loss: 4.50e-01 val loss: 4.33e-01 for "phygnn"
INFO - 2022-03-01 13:35:03,637 [phygnn.py:576] : Epoch 37 train loss: 4.47e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:35:11,267 [phygnn.py:576] : Epoch 38 train loss: 4.43e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:35:18,919 [phygnn.py:576] : Epoch 39 train loss: 4.49e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:35:26,893 [phygnn.py:576] : Epoch 40 train loss: 4.46e-01 val loss: 4.31e-01 for "phygnn"
INFO - 2022-03-01 13:35:34,867 [phygnn.py:576] : Epoch 41 train loss: 4.39e-01 val loss: 4.30e-01 for "phygnn"
INFO - 2022-03-01 13:35:42,630 [phygnn.py:576] : Epoch 42 train loss: 4.43e-01 val loss: 4.28e-01 for "phygnn"
INFO - 2022-03-01 13:35:50,296 [phygnn.py:576] : Epoch 43 train loss: 4.40e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:35:58,252 [phygnn.py:576] : Epoch 44 train loss: 4.41e-01 val loss: 4.28e-01 for "phygnn"
INFO - 2022-03-01 13:36:05,977 [phygnn.py:576] : Epoch 45 train loss: 4.38e-01 val loss: 4.26e-01 for "phygnn"
INFO - 2022-03-01 13:36:13,907 [phygnn.py:576] : Epoch 46 train loss: 4.40e-01 val loss: 4.25e-01 for "phygnn"
INFO - 2022-03-01 13:36:21,345 [phygnn.py:576] : Epoch 47 train loss: 4.39e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:36:29,063 [phygnn.py:576] : Epoch 48 train loss: 4.35e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:36:36,783 [phygnn.py:576] : Epoch 49 train loss: 4.32e-01 val loss: 4.22e-01 for "phygnn"
INFO - 2022-03-01 13:36:44,191 [phygnn.py:576] : Epoch 50 train loss: 4.30e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:36:51,648 [phygnn.py:576] : Epoch 51 train loss: 4.34e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:36:59,204 [phygnn.py:576] : Epoch 52 train loss: 4.32e-01 val loss: 4.21e-01 for "phygnn"
INFO - 2022-03-01 13:37:06,568 [phygnn.py:576] : Epoch 53 train loss: 4.35e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:37:13,884 [phygnn.py:576] : Epoch 54 train loss: 4.35e-01 val loss: 4.23e-01 for "phygnn"
INFO - 2022-03-01 13:37:21,632 [phygnn.py:576] : Epoch 55 train loss: 4.35e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2022-03-01 13:37:29,422 [phygnn.py:576] : Epoch 56 train loss: 4.36e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2022-03-01 13:37:36,919 [phygnn.py:576] : Epoch 57 train loss: 4.37e-01 val loss: 4.20e-01 for "phygnn"
INFO - 2022-03-01 13:37:44,585 [phygnn.py:576] : Epoch 58 train loss: 4.34e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:37:52,399 [phygnn.py:576] : Epoch 59 train loss: 4.33e-01 val loss: 4.19e-01 for "phygnn"
INFO - 2022-03-01 13:37:59,990 [phygnn.py:576] : Epoch 60 train loss: 4.26e-01 val loss: 4.17e-01 for "phygnn"
INFO - 2022-03-01 13:38:07,570 [phygnn.py:576] : Epoch 61 train loss: 4.26e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:38:15,403 [phygnn.py:576] : Epoch 62 train loss: 4.31e-01 val loss: 4.18e-01 for "phygnn"
INFO - 2022-03-01 13:38:23,121 [phygnn.py:576] : Epoch 63 train loss: 4.28e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:38:30,488 [phygnn.py:576] : Epoch 64 train loss: 4.28e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:38:38,444 [phygnn.py:576] : Epoch 65 train loss: 4.25e-01 val loss: 4.14e-01 for "phygnn"
INFO - 2022-03-01 13:38:46,279 [phygnn.py:576] : Epoch 66 train loss: 4.26e-01 val loss: 4.15e-01 for "phygnn"
INFO - 2022-03-01 13:38:54,135 [phygnn.py:576] : Epoch 67 train loss: 4.26e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:39:01,672 [phygnn.py:576] : Epoch 68 train loss: 4.24e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:39:09,075 [phygnn.py:576] : Epoch 69 train loss: 4.21e-01 val loss: 4.13e-01 for "phygnn"
INFO - 2022-03-01 13:39:16,910 [phygnn.py:576] : Epoch 70 train loss: 4.28e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:39:24,818 [phygnn.py:576] : Epoch 71 train loss: 4.26e-01 val loss: 4.12e-01 for "phygnn"
INFO - 2022-03-01 13:39:32,340 [phygnn.py:576] : Epoch 72 train loss: 4.27e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:39:40,074 [phygnn.py:576] : Epoch 73 train loss: 4.17e-01 val loss: 4.11e-01 for "phygnn"
INFO - 2022-03-01 13:39:48,025 [phygnn.py:576] : Epoch 74 train loss: 4.23e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:39:55,865 [phygnn.py:576] : Epoch 75 train loss: 4.23e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:40:03,564 [phygnn.py:576] : Epoch 76 train loss: 4.29e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:40:11,361 [phygnn.py:576] : Epoch 77 train loss: 4.17e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:40:18,978 [phygnn.py:576] : Epoch 78 train loss: 4.24e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:40:26,341 [phygnn.py:576] : Epoch 79 train loss: 4.21e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:40:33,652 [phygnn.py:576] : Epoch 80 train loss: 4.25e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:40:41,385 [phygnn.py:576] : Epoch 81 train loss: 4.19e-01 val loss: 4.08e-01 for "phygnn"
INFO - 2022-03-01 13:40:49,354 [phygnn.py:576] : Epoch 82 train loss: 4.33e-01 val loss: 4.10e-01 for "phygnn"
INFO - 2022-03-01 13:40:57,320 [phygnn.py:576] : Epoch 83 train loss: 4.25e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:41:05,094 [phygnn.py:576] : Epoch 84 train loss: 4.16e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:41:12,707 [phygnn.py:576] : Epoch 85 train loss: 4.20e-01 val loss: 4.07e-01 for "phygnn"
INFO - 2022-03-01 13:41:20,124 [phygnn.py:576] : Epoch 86 train loss: 4.16e-01 val loss: 4.09e-01 for "phygnn"
INFO - 2022-03-01 13:41:27,756 [phygnn.py:576] : Epoch 87 train loss: 4.08e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:41:35,296 [phygnn.py:576] : Epoch 88 train loss: 4.22e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:41:42,785 [phygnn.py:576] : Epoch 89 train loss: 4.14e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:41:50,210 [phygnn.py:576] : Epoch 90 train loss: 4.20e-01 val loss: 4.05e-01 for "phygnn"
INFO - 2022-03-01 13:41:58,104 [phygnn.py:576] : Epoch 91 train loss: 4.21e-01 val loss: 4.06e-01 for "phygnn"
INFO - 2022-03-01 13:42:05,687 [phygnn.py:576] : Epoch 92 train loss: 4.13e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:42:13,475 [phygnn.py:576] : Epoch 93 train loss: 4.13e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:42:21,391 [phygnn.py:576] : Epoch 94 train loss: 4.19e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:42:29,002 [phygnn.py:576] : Epoch 95 train loss: 4.16e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:42:36,349 [phygnn.py:576] : Epoch 96 train loss: 4.18e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:42:43,946 [phygnn.py:576] : Epoch 97 train loss: 4.16e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:42:51,361 [phygnn.py:576] : Epoch 98 train loss: 4.16e-01 val loss: 4.03e-01 for "phygnn"
INFO - 2022-03-01 13:42:59,089 [phygnn.py:576] : Epoch 99 train loss: 4.13e-01 val loss: 4.04e-01 for "phygnn"
INFO - 2022-03-01 13:42:59,867 [trainer.py:92] : Training part B - data and phygnn. Loss is [0.5, 0.5]
INFO - 2022-03-01 13:43:21,966 [phygnn.py:576] : Epoch 100 train loss: 2.88e-01 val loss: 2.76e-01 for "phygnn"
INFO - 2022-03-01 13:43:35,002 [phygnn.py:576] : Epoch 101 train loss: 2.82e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:43:47,991 [phygnn.py:576] : Epoch 102 train loss: 2.82e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:44:00,932 [phygnn.py:576] : Epoch 103 train loss: 2.85e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:44:13,922 [phygnn.py:576] : Epoch 104 train loss: 2.84e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:44:26,701 [phygnn.py:576] : Epoch 105 train loss: 2.86e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:44:39,599 [phygnn.py:576] : Epoch 106 train loss: 2.83e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:44:52,483 [phygnn.py:576] : Epoch 107 train loss: 2.83e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:45:05,217 [phygnn.py:576] : Epoch 108 train loss: 2.86e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:45:17,700 [phygnn.py:576] : Epoch 109 train loss: 2.84e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:45:30,626 [phygnn.py:576] : Epoch 110 train loss: 2.83e-01 val loss: 2.75e-01 for "phygnn"
INFO - 2022-03-01 13:45:43,273 [phygnn.py:576] : Epoch 111 train loss: 2.84e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:45:55,874 [phygnn.py:576] : Epoch 112 train loss: 2.81e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:46:08,822 [phygnn.py:576] : Epoch 113 train loss: 2.81e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:46:21,663 [phygnn.py:576] : Epoch 114 train loss: 2.87e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:46:34,706 [phygnn.py:576] : Epoch 115 train loss: 2.82e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:46:47,641 [phygnn.py:576] : Epoch 116 train loss: 2.84e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:47:00,485 [phygnn.py:576] : Epoch 117 train loss: 2.82e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:47:13,240 [phygnn.py:576] : Epoch 118 train loss: 2.81e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:47:26,006 [phygnn.py:576] : Epoch 119 train loss: 2.83e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:47:38,763 [phygnn.py:576] : Epoch 120 train loss: 2.85e-01 val loss: 2.74e-01 for "phygnn"
INFO - 2022-03-01 13:47:51,671 [phygnn.py:576] : Epoch 121 train loss: 2.84e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:48:04,234 [phygnn.py:576] : Epoch 122 train loss: 2.81e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:48:17,196 [phygnn.py:576] : Epoch 123 train loss: 2.83e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:48:29,785 [phygnn.py:576] : Epoch 124 train loss: 2.81e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:48:42,698 [phygnn.py:576] : Epoch 125 train loss: 2.81e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:48:55,191 [phygnn.py:576] : Epoch 126 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:49:07,748 [phygnn.py:576] : Epoch 127 train loss: 2.85e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:49:20,732 [phygnn.py:576] : Epoch 128 train loss: 2.80e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:49:33,641 [phygnn.py:576] : Epoch 129 train loss: 2.80e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:49:46,240 [phygnn.py:576] : Epoch 130 train loss: 2.82e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:49:58,974 [phygnn.py:576] : Epoch 131 train loss: 2.83e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:50:11,583 [phygnn.py:576] : Epoch 132 train loss: 2.77e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:50:24,192 [phygnn.py:576] : Epoch 133 train loss: 2.80e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:50:36,613 [phygnn.py:576] : Epoch 134 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:50:49,537 [phygnn.py:576] : Epoch 135 train loss: 2.80e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:51:01,971 [phygnn.py:576] : Epoch 136 train loss: 2.83e-01 val loss: 2.73e-01 for "phygnn"
INFO - 2022-03-01 13:51:14,879 [phygnn.py:576] : Epoch 137 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:51:27,433 [phygnn.py:576] : Epoch 138 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:51:40,381 [phygnn.py:576] : Epoch 139 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:51:53,177 [phygnn.py:576] : Epoch 140 train loss: 2.82e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:52:05,759 [phygnn.py:576] : Epoch 141 train loss: 2.84e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:52:18,514 [phygnn.py:576] : Epoch 142 train loss: 2.78e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:52:31,504 [phygnn.py:576] : Epoch 143 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:52:43,939 [phygnn.py:576] : Epoch 144 train loss: 2.79e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:52:56,776 [phygnn.py:576] : Epoch 145 train loss: 2.77e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:53:09,635 [phygnn.py:576] : Epoch 146 train loss: 2.81e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:53:22,096 [phygnn.py:576] : Epoch 147 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:53:34,612 [phygnn.py:576] : Epoch 148 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:53:47,108 [phygnn.py:576] : Epoch 149 train loss: 2.78e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:53:59,999 [phygnn.py:576] : Epoch 150 train loss: 2.81e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:54:12,832 [phygnn.py:576] : Epoch 151 train loss: 2.78e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:54:25,415 [phygnn.py:576] : Epoch 152 train loss: 2.79e-01 val loss: 2.72e-01 for "phygnn"
INFO - 2022-03-01 13:54:38,255 [phygnn.py:576] : Epoch 153 train loss: 2.75e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:54:51,167 [phygnn.py:576] : Epoch 154 train loss: 2.82e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:55:03,656 [phygnn.py:576] : Epoch 155 train loss: 2.75e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:55:16,469 [phygnn.py:576] : Epoch 156 train loss: 2.80e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:55:29,376 [phygnn.py:576] : Epoch 157 train loss: 2.77e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:55:42,299 [phygnn.py:576] : Epoch 158 train loss: 2.77e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:55:55,171 [phygnn.py:576] : Epoch 159 train loss: 2.79e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:56:07,924 [phygnn.py:576] : Epoch 160 train loss: 2.81e-01 val loss: 2.71e-01 for "phygnn"
INFO - 2022-03-01 13:56:20,545 [phygnn.py:576] : Epoch 161 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:56:33,065 [phygnn.py:576] : Epoch 162 train loss: 2.75e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:56:45,746 [phygnn.py:576] : Epoch 163 train loss: 2.77e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:56:58,099 [phygnn.py:576] : Epoch 164 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:57:10,695 [phygnn.py:576] : Epoch 165 train loss: 2.77e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:57:23,289 [phygnn.py:576] : Epoch 166 train loss: 2.78e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:57:35,910 [phygnn.py:576] : Epoch 167 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:57:48,543 [phygnn.py:576] : Epoch 168 train loss: 2.80e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:58:01,358 [phygnn.py:576] : Epoch 169 train loss: 2.79e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:58:14,045 [phygnn.py:576] : Epoch 170 train loss: 2.77e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:58:26,860 [phygnn.py:576] : Epoch 171 train loss: 2.77e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:58:39,604 [phygnn.py:576] : Epoch 172 train loss: 2.81e-01 val loss: 2.70e-01 for "phygnn"
INFO - 2022-03-01 13:58:52,454 [phygnn.py:576] : Epoch 173 train loss: 2.75e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:59:05,269 [phygnn.py:576] : Epoch 174 train loss: 2.77e-01 val loss: 2.69e-01 for "phygnn"
INFO - 2022-03-01 13:59:18,137 [phygnn.py:576] : Epoch 175 train loss: 2.77e-01 val loss: 2.69e-01 for "phygnn"
ERROR - 2022-03-01 13:59:25,882 [phygnn.py:461] : phygnn calculated a NaN loss value!
Traceback (most recent call last):
  File "k_fold.py", line 50, in <module>
    t = Trainer(train_sites=train_sites, train_files=files, config=config)
  File "/home/gbuster/code/mlclouds/mlclouds/trainer.py", line 95, in __init__
    out = model.train_model(self.x, self.y, self.p,
  File "/home/gbuster/miniconda3/envs/nsrdb/lib/python3.8/site-packages/phygnn/model_interfaces/phygnn_model.py", line 187, in train_model
    diagnostics = self.model.fit(x, y, p,
  File "/home/gbuster/miniconda3/envs/nsrdb/lib/python3.8/site-packages/phygnn/phygnn.py", line 570, in fit
    tr_loss, tr_nn_loss, tr_p_loss = self.run_gradient_descent(
  File "/home/gbuster/miniconda3/envs/nsrdb/lib/python3.8/site-packages/phygnn/phygnn.py", line 482, in run_gradient_descent
    grad, loss, nn_loss, p_loss = self._get_grad(x, y_true, p, p_kwargs)
  File "/home/gbuster/miniconda3/envs/nsrdb/lib/python3.8/site-packages/phygnn/phygnn.py", line 473, in _get_grad
    loss, nn_loss, p_loss = self.calc_loss(y_true, y_predicted,
  File "/home/gbuster/miniconda3/envs/nsrdb/lib/python3.8/site-packages/phygnn/phygnn.py", line 462, in calc_loss
    raise ArithmeticError(msg)
ArithmeticError: phygnn calculated a NaN loss value!
