PowerGenome Clustering fails silently if wrong RESOURCE_GROUPS is used, and TBD other reason

I inadvertently had RESOURCE_GROUPS set to eastern-20-zone instead of ipm_regions and this caused all of the renewables to be omitted from the new generators df, without any error/warning, which took me a long time to figure out.

In this commit, I added a warning in the add_renewables_clusters function of nrelatb to call attention to this. Perhaps this would be helpful to merge to help future users. It might also be useful to include some more clear documentation / examples on how RESOURCE_GROUPS should be set. The example systems do not include this in the settings files since it's assumed to be in the user's .env file, and in the CI tests it's set to a github secret, and this lack of example contributed to my confusion.

It might be preferred to include a warning directly in the assign_site_cluster function of renewables.py instead of in nrelatb like I did, since that is where I traced it back to the actual string match not occurring, but I wasn't sure if this would produce false-positive warnings if a particular renewable was not intended to be built in a particular region. line 475 data = renew_data.loc[renew_data["region"].isin(regions), :]

Even with the correct RESOURCE_GROUPS setting, I still see the clustering have an issue for offshore wind (the warning I added being thrown), and I am wondering if you have any advice on what might still be wrong? My settings file is https://github.com/symbiotic-engineering/WEC-DECIDER/blob/becca-edit-khai-branch/modules/CEM/data_east/test_settings_2.yml. This is the log, with the relevant warnings bolded:

23:51:47 [    INFO] powergenome:180 Reading settings file
23:51:47 [ WARNING] powergenome.util:445 
    *****************************
    The ATB technology "LandbasedWind_LTRG10" listed in your settings file under 'atb_new_gen'
    is not fully specified in the 'cost_multiplier_technology_map' settings parameter.
    Part of the _ string might be included, but it is best practice to
    include the full name in this format. Check your settings file.
        
23:51:47 [ WARNING] powergenome.util:445 
    *****************************
    The ATB technology "OffShoreWind_OTRG10" listed in your settings file under 'atb_new_gen'
    is not fully specified in the 'cost_multiplier_technology_map' settings parameter.
    Part of the _ string might be included, but it is best practice to
    include the full name in this format. Check your settings file.
        
23:51:47 [ WARNING] powergenome.util:445 
    *****************************
    The ATB technology "UtilityPV_Chicago" listed in your settings file under 'atb_new_gen'
    is not fully specified in the 'cost_multiplier_technology_map' settings parameter.
    Part of the _ string might be included, but it is best practice to
    include the full name in this format. Check your settings file.
        
23:51:47 [    INFO] powergenome.util:533 The default interest compounding method for calculating annuities has changed from continuous to discrete. This method can be set with the parameter 'interest_compound_method', using values `discrete` or `continuous`.
This message will be removed after version 0.7.0.
23:51:47 [    INFO] powergenome:223 Sorted model regions are NENGREST, NENG_CT, NENG_ME
23:51:47 [    INFO] powergenome:271 

Starting year 2030 scenario case1


23:52:02 [    INFO] powergenome.generators:3166 Compiling existing generators and their attributes
2025-07-07 23:52:04 [    INFO] catalystcoop.pudl.transform.eia861:456 Started with 325 missing BA Codes out of 12670 records (2.57%)
2025-07-07 23:52:06 [    INFO] catalystcoop.pudl.transform.eia861:480 Ended with 325 missing BA Codes out of 12670 records (2.57%)
2025-07-07 23:52:06 [    INFO] catalystcoop.pudl.output.eia860:177 97.4% of plant records have consistently reported BA Codes
2025-07-07 23:52:06 [    INFO] catalystcoop.pudl.output.eia860:227 Before any filling treatment has been applied. 2.6% of records have no BA codes
2025-07-07 23:52:07 [    INFO] catalystcoop.pudl.output.eia860:227 Backfilling and consistent value is the same. Filled w/ most consistent BA code. 2.6% of records have no BA codes
2025-07-07 23:52:07 [    INFO] catalystcoop.pudl.output.eia860:227 SWPP is most consistent value. Filled w/ oldest BA code. 2.6% of records have no BA codes
2025-07-07 23:52:07 [    INFO] catalystcoop.pudl.output.eia860:227 NWMT is most consistent value. Filled w/ oldest BA code. 2.6% of records have no BA codes
2025-07-07 23:52:07 [    INFO] catalystcoop.pudl.output.eia860:227 Two or more years of oldest BA code. Filled w/ oldest BA code. 2.6% of records have no BA codes
2025-07-07 23:52:11 [    INFO] catalystcoop.pudl.transform.eia861:456 Started with 325 missing BA Codes out of 12670 records (2.57%)
2025-07-07 23:52:14 [    INFO] catalystcoop.pudl.transform.eia861:480 Ended with 325 missing BA Codes out of 12670 records (2.57%)
2025-07-07 23:52:14 [    INFO] catalystcoop.pudl.output.eia860:177 97.4% of plant records have consistently reported BA Codes
2025-07-07 23:52:14 [    INFO] catalystcoop.pudl.output.eia860:227 Before any filling treatment has been applied. 2.6% of records have no BA codes
2025-07-07 23:52:14 [    INFO] catalystcoop.pudl.output.eia860:227 Backfilling and consistent value is the same. Filled w/ most consistent BA code. 2.6% of records have no BA codes
2025-07-07 23:52:14 [    INFO] catalystcoop.pudl.output.eia860:227 SWPP is most consistent value. Filled w/ oldest BA code. 2.6% of records have no BA codes
2025-07-07 23:52:14 [    INFO] catalystcoop.pudl.output.eia860:227 NWMT is most consistent value. Filled w/ oldest BA code. 2.6% of records have no BA codes
2025-07-07 23:52:14 [    INFO] catalystcoop.pudl.output.eia860:227 Two or more years of oldest BA code. Filled w/ oldest BA code. 2.6% of records have no BA codes
23:52:17 [    INFO] powergenome.nrelatb:455 The EIA technologies ['Nuclear', 'Batteries', 'Solar Photovoltaic', 'Small Hydroelectric', 'Onshore Wind Turbine', 'Offshore Wind Turbine', 'Conventional Hydroelectric', 'Hydroelectric Pumped Storage'] do not have an ATB counterpart with a valid heat rate. Not all ATB technologies *should* have a valid heat rate (e.g. wind, solar, and hydro). Check the 'eia_atb_tech_map' parameter in your settings file(s) if you think one of these technologies should be mapped to an ATB technology with a valid heat rate.
23:52:17 [    INFO] powergenome.nrelatb:507 The EIA technologies ['Nuclear', 'Solar Photovoltaic', 'Onshore Wind Turbine', 'Offshore Wind Turbine'] do not have an ATB counterpart with valid fixed or variable O&M costs. All ATB technologies *should* have valid fixed/variable O&M costs. Check the 'eia_atb_tech_map' parameter in your settings file(s).
23:52:17 [    INFO] powergenome.generators:3474 Age-retired gens are included for clustering to keep consistent cluster assignments across periods. 
23:52:17 [    INFO] powergenome.generators:3510 Creating technology clusters by region
23:52:41 [ WARNING] powergenome.generators:1551 The model resource tags {'LDS', 'Rsv_Max', 'Hydro_level', 'Reg_Max'} are listed in the settings parameter 'model_tags_name' but are not assigned values for any resources
23:52:42 [    INFO] powergenome.generators:3880 Finished creating existing generator clusters
23:52:42 [    INFO] powergenome.generators:3885 Starting to build new generation resources
23:52:45 [ WARNING] powergenome.nrelatb:1646 Site assignment empty for renewables cluster in region NENG_CT for technology OffShoreWind_OTRG10_Mid with profile data/resource_profiles/interpolated_scaled_offshore_wind_profiles_20210713.parquet and site map None.
23:52:45 [ WARNING] powergenome.nrelatb:1714 Problem with renewables clustering for region NENG_CT. Expected to create 3 clusters, but actually made 2. Check that RESOURCE_GROUPS aligns with your region aggregations.
23:52:45 [ WARNING] powergenome.nrelatb:1646 Site assignment empty for renewables cluster in region NENG_ME for technology OffShoreWind_OTRG10_Mid with profile data/resource_profiles/interpolated_scaled_offshore_wind_profiles_20210713.parquet and site map None.
23:52:45 [ WARNING] powergenome.nrelatb:1714 Problem with renewables clustering for region NENG_ME. Expected to create 3 clusters, but actually made 2. Check that RESOURCE_GROUPS aligns with your region aggregations.
23:52:45 [    INFO] powergenome.external_data:173 Inflating external interconnect annuity costs from 2017 to 2020
23:52:45 [ WARNING] powergenome.generators:2605 Transmission investment costs are missing or zero for some resources and will not be included in the total investment costs.
23:52:45 [    INFO] powergenome.generators:3037 Creating flexible demand resources
23:52:45 [    INFO] powergenome.load_construction:293 Building demand profiles of electrified end-use technologies
23:52:57 [ WARNING] powergenome.generators:1551 The model resource tags {'LDS', 'Rsv_Max', 'Hydro_level', 'Reg_Max'} are listed in the settings parameter 'model_tags_name' but are not assigned values for any resources
23:52:57 [    INFO] powergenome.generators:3947 Finished creating new generation resources
23:52:57 [ WARNING] powergenome.GenX:545 The resources ['res_water_heat_1', 'trans_light_duty_1', 'wave_low_0'] are not included in your `misc_gen_inputs_fn` CSV file. This is a warning in case they should have parameters in that file.
23:52:57 [ WARNING] powergenome.GenX:1417 The variable resources ['NENGREST_offshore_wind_turbine_1'] have non-variable generation profiles.
23:52:57 [    INFO] powergenome.load_profiles:650 Creating hourly demand profiles
23:52:57 [    INFO] powergenome.load_profiles:665 Regional load data sources have not been specified. Defaulting to EFS load data. See documentation of the parameter 'regional_load_source' to use other data.
23:53:00 [    INFO] powergenome.load_construction:293 Building demand profiles of electrified end-use technologies
23:53:10 [    INFO] powergenome.load_profiles:783 Creating distributed generation profiles
23:53:10 [    INFO] powergenome.time_reduction:131 Reducing time domain from 8760 hours to representative periods
23:53:16 [ WARNING] powergenome.transmission:66 No transmission value column (e.g. firm vs non-firm) was specified in the settings. The column 'firm_ttc_mw' will be used as a default. This is a change from previous versions of PG, where 'nonfirm_ttc_mw' was used. Firm transmission capacity is lower or equal to non-firm capacity.
23:53:16 [ WARNING] powergenome.transmission:137 The user transmission capacity table duplicates some values from the database. Database values will be discarded in these cases.
23:53:16 [ WARNING] powergenome.GenX:1447 Upgrading the Can_Retire and New_Build interface
23:53:16 [    INFO] powergenome.GenX:1593 Creating policy data file for ESR_
23:53:16 [    INFO] powergenome.GenX:1593 Creating policy data file for Derating_factor_
23:53:16 [    INFO] powergenome.GenX:1593 Creating policy data file for Min_Cap_
23:53:16 [    INFO] powergenome.GenX:1593 Creating policy data file for Max_Cap_
23:53:16 [    INFO] powergenome.GenX:1671 Creating multistage data file
23:53:16 [    INFO] powergenome.GenX:1514 Creating resource data file for THERM
23:53:16 [    INFO] powergenome.GenX:1514 Creating resource data file for VRE
23:53:16 [    INFO] powergenome.GenX:1514 Creating resource data file for MUST_RUN
23:53:16 [    INFO] powergenome.GenX:1514 Creating resource data file for STOR
23:53:16 [    INFO] powergenome.GenX:1514 Creating resource data file for FLEX
23:53:16 [    INFO] powergenome.GenX:1514 Creating resource data file for HYDRO

Jul 08 '25 16:07 rebeccamccabe

I'm pushing through a big set of updates to code and tests but will check back on this tomorrow. Bump it if you haven't heard back by the day after.

Jul 08 '25 17:07 gschivley

Thanks! I am bumping the question.

Jul 10 '25 17:07 rebeccamccabe