`fetchRaCA()` database needs update

[potash]

The comments for fetchRACA() have some warnings, including a message (from 2014?) "Carbon concentration and stock values are probably wrong, or at least suspect. USE WITH CAUTION." Wondering what the status of this method is today, in particular how the resulting data compares to what I would download directly here:

[dylanbeaudette]

It is not and needs a serious overhaul. The original RaCA objectives (estimate SOC, SIC from VNIR spectra) did not work out so well. The second iteration (measure SOC, SIC) was completed but the database this function hits was not updated.

[potash]

Thanks. Is the dataset I linked to what you refer to as the first or second iteration?

[dylanbeaudette]

It uses the first iteration. I'd like to update sometime this calendar year.

[brownag]

I am going to re-open this issue and change the title as a reminder to get the SoilWeb side of this function updated this year.

[brownag]

Bump

[brownag]

@dylanbeaudette What is the status of updating the RaCA data in the SoilWeb snapshot?

Last week I had an inquiry about getting the latest RaCA data. At this point I am leaning towards replacing the "USE WITH CAUTION" message in this function with a deprecation warning that directs to the direct download of the ZIP file for whole database.

[dylanbeaudette]

Still waiting to hear back from Skye about what is planned for RaCA. Ideally, it will be merged back into the rest of the KSSL holdings and then we won't need a special function, link, or API. Also, it would be nice to get the coordinates un-fuzzed. Worst-case scenarios:

1. update the SoilWeb API
2. deprecate function with link to what is posted on our website

[brownag]

@dylanbeaudette I am inclined to deprecate fetchRaCA() in v2.8.0. Thoughts?

[dylanbeaudette]

Maybe a good idea.

• there appears to be very little interest in merging back into LDM
• the coordinates are still fuzzed, and require special permission to see the original
• the VISNIR spectra are not (maybe cannot be) normalized to a common baseline

If we deprecate then I'll likely retire the SoilWeb API endpoint for these data. Maybe to be resurrected someday.

[brownag]

OK. This will add a deprecation warning, but I would hold off on any changes to the API for at least several months.

The alternative would be to just update the SoilWeb API to use the current data as originally proposed. I think the coordinate fuzzing and VNIR baseline are not as important as getting the more modern KSSL data linked up with the spectra, so I personally would not let that be the reason the SoilWeb API does not get updated.

Since the update to SoilWeb hasn't been done in the time the combined database has been available, and we may not want to proliferate "unofficial" sources of data available elsewhere, I think deprecation with a link to official download is the best approach.

If RaCA were to be merged back into LDM we would also want to deprecate, as the data would then be accessible vis SDA_query/fetchLDM.

[dylanbeaudette]

RaCA will likely be integrated into some "future IRA GHG effort" (1-5 years hence). A new NCSS director could change that at any time. For now, the data will likely live in its current form (zip file of spread sheets and text files), spectral will remain nearly unusable, and coordinates fuzzed.

[potash]

A couple questions before you shut this down:

0. When going through ag data commons today to try to download RaCA I get pointed here which is broken. What is the current link for downloading RaCA?

1. fetchRaCA() currently includes a site table with taxonomy and topography for each pedon. Am I correct in thinking that this data is not currently included in the zip file? If so, it would be good to put it in the zip before turning off the API.

2. What is the best way to link from a RaCA pedon to a SSURGO component? Is the SSURGO areasymbol available somewhere? If not, are the coordinates fine enough to to look up the areasymbol through soilDB? Even then we just have the soil series and areasymbol which is not always enough to uniquely identify the component.

Thanks!

[brownag]

0. When going through ag data commons today to try to download RaCA I get pointed here which is broken. What is the current link for downloading RaCA?

Sorry about that! I put in a request that this link be updated over a year ago. I saw that the metadata had been updated in September, but must have made a mistake in that the link referenced is still the old one you point out above. It appears that it is still not fixed. The link should point to this page https://www.nrcs.usda.gov/resources/data-and-reports/rapid-carbon-assessment-raca. Which in turn links to Box for the actual download: https://nrcs.app.box.com/s/upx5xhlwis7saunfiysclfrhl5vxxudn . Will request this be fixed again.

I tried to put the above Box link in the docs but it did not pass CRAN check. However the Box link is given in the new deprecation message issued on 2.8.0+. Sorry for the confusion

1. fetchRaCA() currently includes a site table with taxonomy and topography for each pedon. Am I correct in thinking that this data is not currently included in the zip file? If so, it would be good to put it in the zip before turning off the API.

I believe you are correct, @dylanbeaudette can confirm. I agree that there are some value-added components to the fetchRaCA API currently unhandled by the direct download option.

2. What is the best way to link from a RaCA pedon to a SSURGO component? Is the SSURGO areasymbol available somewhere? If not, are the coordinates fine enough to to look up the areasymbol through soilDB? Even then we just have the soil series and areasymbol which is not always enough to uniquely identify the component.

I think that in general the coordinates are accurate enough for overlay with the SAPOLYGON layer to determine the areasymbol. It is true that this is still limited. In general the RaCA points should be within mapunits of the corresponding named series--as that is how the sampling was laid out--but it is true that there are instances where a soil that is different from the named series/component was encountered. Tying it to a specific component would likely require more accurate location information, but it might be adequate to tie it to the series concept (and associated components) from the survey area.

[potash]

@brownag Thanks for the quick reply! All makes sense. Two more questions:

3. I see that a possible alternative route to getting taxonomic data on the RaCA pedons is by querying the lab_combine_nasis_ncss table. According to the RaCA documentation, the upedonid in the RaCA_samples.csv file should allow me to lookup the same pedon in NASIS. However, I am not getting any results:

# 10 randomly sampled upedonids from RaCA_samples.csv
upedonids = "('C1201F43-1','F1408F065','RaCA11AZHH041','C0110X03-3','C1304C0403','RaCA11TXCV0049','C0404F04C','C1405W723','C0109R04_1','C1311F1803')"

pedons_RaCA = SDA_query(sprintf(
  "SELECT *
  from lab_combine_nasis_ncss
  where upedonid IN %s", upedonids))

returns

empty result set

4. In the RaCA_samples.csv file the only measured carbon I am finding is c_tot_ncs which apparently is total carbon not organic carbon. Am I correct in thinking then that organic carbon was not measured in RaCA?

[dylanbeaudette]

A couple comments and clarifications:

• The geographic coordinates that fetchRaCA() returns are rounded to 2 decimal places. This is not enough precision to correctly identify the sampled SSURGO MU polygon in most cases. I am not sure if this is the same level of "fuzz" applied to the coordinates in the .zip archive, but it is probably close.
• The RaCA points are not part of our LDM snapshot: this means that corresponding data (site, pedon, etc.) cannot be accessed via SDA or SQLite snapshots.
• The latest RaCA update should include total, organic, and inorganic carbon measurements. The values returned by fetchRaCA() should not be used.

If there is sufficient interest, I may be able to update the SoilWeb API. However, it has always been a very low priority due to the access limitations imposed on the source data. I'd suggest email Skye Wills for details.

[brownag]

• The latest RaCA update should include total, organic, and inorganic carbon measurements.

I think that for RaCA download available on Box the CaCO3 equivalent needs to be converted to %C, then subtracted from the dry combustion total carbon to calculate organic carbon.

[potash]

Got it, thanks to you both for this info