edgarWebR icon indicating copy to clipboard operation
edgarWebR copied to clipboard

Unable to find documentation to set User Agent

Open pietjonker opened this issue 4 years ago • 20 comments

When retrieving filings, I receive an error for being an Undeclared Automated Tool. For example, when using

latest_filings()

I receive error

No encoding supplied: defaulting to UTF-8.
Error in check_result(res) : 
  EDGAR request blocked from Undeclared Automated Tool.
Please visit https://www.sec.gov/developer for best practices.
See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
Consider also setting the environment variable 'EDGARWEBR_USER_AGENT

I found in the README the following information:

Because of abusive use of this library, the SEC is likely to block its use “as is” without setting a custom ‘User Agent’ identifier. Details for setting a custom agent are below.

However, no details were given below. Could anyone help me on setting the user agent?

pietjonker avatar Jul 05 '21 09:07 pietjonker

Within edarWebR you need to set an env variable. Sys.setenv('xxxx')

Then, i usually call EDGARWEBR_USER_AGENT <- Sys.getenv('xxxx')

Also, for those of us that are getting errors due to the SEC shutting it down (despite never exceeding the stated limit on their webiste!), I've been substituting read_html with this:

html_retry <-  function(url1,   N.TRIES = 3L) {

  require(rvest)
  
  N.TRIES <- as.integer(N.TRIES)
  stopifnot(length(N.TRIES) == 1L, !is.na(N.TRIES))
  stopifnot(N.TRIES > 0L)
  
  while (N.TRIES > 0L) {
    result <- tryCatch(read_html(
      x = url1
    ), error = identity)
    if (!inherits(result, "error")) {
      break
    }
    ## Wait 0.5 seconds
    Sys.sleep(0.5)
    N.TRIES <- N.TRIES - 1L
  }
  
  if (N.TRIES == 0L) {
    stop(
      "'html_retry()' failed:",
      "\n  URL: ", url1,
      "\n  error: ", conditionMessage(result)
    )
  }
  
  invisible(result)
}

xbrl-data avatar Jul 21 '21 03:07 xbrl-data

Hello,

thanks for developing this package. I set an env variable EDGARWEBR_USER_AGENT using Sys.setenv and tried 'filing_documents' function but I still receive the error message.

No encoding supplied: defaulting to UTF-8. Error in check_result(res) : EDGAR request blocked from Undeclared Automated Tool. Please visit https://www.sec.gov/developer for best practices. See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities Consider also setting the environment variable 'EDGARWEBR_USER_AGENT

could you help me how to proceed from here?

mirayoon avatar Jul 30 '21 10:07 mirayoon

@mirayoon : I have the same issue, but a try-catch block loop until successful like @xbrl-data mentioned will work.

YuMan-Tam avatar Jul 31 '21 22:07 YuMan-Tam

Has anyone gotten this to work lately? I've been setting the user agent correctly per the SEC website and I'm still not able to return anything other than the messages above. How has the try-catch routine as described been substituted for what xml2 has? Does all the underlying R code need to be editted? Also, does Accept-Encoding: gzip, deflate need to somehow be set? TIA, mconsidine

mconsidine avatar Sep 29 '21 20:09 mconsidine

Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with.

xbrl-data avatar Sep 29 '21 20:09 xbrl-data

I've managed to hack some of the code to work with a with_config wrapper around a GET call.  But things only break down again later...On Sep 29, 2021 4:59 PM, xbrl-data @.***> wrote: Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

mconsidine avatar Sep 30 '21 01:09 mconsidine

It seems like some formatting at the SEC site has changed. E.g. the /feed/company-info node seems empty. Cik_ search works though.

Does anyone have a workflow they can share for low frequency requests for data access/parsing/report analysis if this is now busted? I was hoping to pair this with finstr-upd/xbrl-upd to show a college class.

Mconsidine

mconsidine avatar Sep 30 '21 02:09 mconsidine

@xbrl-data : can you give me an idea how you have used those as a workaround? A PM would be fine, if you'd prefer.

mconsidine avatar Oct 01 '21 13:10 mconsidine

@mirayoon I believe this is the proper way to set the environment variable

Sys.setenv(EDGARWEBR_USER_AGENT = " yourcompanyname youremail")

where yourcompanyname and youremail are specified by the user.

You can check it is properly set using the following

Sys.getenv("EDGARWEBR_USER_AGENT")

IEORTools avatar Jan 19 '22 23:01 IEORTools

@mirayoon I believe this is the proper way to set the environment variable

Sys.setenv(EDGARWEBR_USER_AGENT = " yourcompanyname youremail")

where yourcompanyname and youremail are specified by the user.

You can check it is properly set using the following

Sys.getenv("EDGARWEBR_USER_AGENT")

For me neitherSys.setenv() nor usethis::edit_r_environ() works — try using the fork I've suggested (see above).

balthasars avatar Jan 23 '22 17:01 balthasars

Hi @balthasars thank you for sharing your solution but, unfortunately, it does not seem to work for me. Any other tips on how to solve it?

granatakorps avatar Apr 02 '22 18:04 granatakorps

This has worked for me:

library(tidyverse) library(httr) library(XBRL) library(edgarWebR)

edgar_agent <- Sys.getenv("EDGARWEBR_USER_AGENT", unset = "edgarWebR [email protected]" ) ua <- httr::user_agent(edgar_agent) options(HTTPUserAgent=ua$options$useragent)

Haven't tried to figure out if there's redundancy/unused vars being set in that ...

HTH, mconsidine

mconsidine avatar Apr 04 '22 12:04 mconsidine

Hello None of the above approaches works for me

snvv avatar May 25 '22 18:05 snvv

I used the version found here https://github.com/balthasars/edgarWebR/tree/fix-user-agent Also, this appeared to work, looking back over some code (thought I think its a kludge): with_config(user_agent(ua$options$useragent),search.result <- xml2::read_html(url)) mconsidine

mconsidine avatar May 25 '22 19:05 mconsidine

Sorry for only getting back to this now, everyone!

So it appears that the install of my proposed fork doesn't work when installing it using remotes::install_github():

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/Rtmp2UYpBk/remotes894768da6952/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
#> No encoding supplied: defaulting to UTF-8.
#> Error in `mutate()`:
#> ! Problem while computing `results = purrr::map(search_terms,
#>   edgarWebR::company_search, type = "13F-HR")`.
#> Caused by error in `check_result()`:
#> ! EDGAR request blocked from Undeclared Automated Tool.
#> Please visit https://www.sec.gov/developer for best practices.
#> See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
#> Consider also setting the environment variable 'EDGARWEBR_USER_AGENT
bank_search_results
#> Error in eval(expr, envir, enclos): object 'bank_search_results' not found

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

Created on 2022-05-25 by the reprex package (v2.0.1)

BUT: when I clone the repo, build the package in RStudio from there and then run my example, ta-dah:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

Created on 2022-05-25 by the reprex package (v2.0.1)

I am puzzled by this behavior and will need to read up some more on environmental variables.

UPDATE: The install using remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE) does also work if you set the environmental variable beforehand, using usethis::edit_r_environ(), such as EDGARWEBR_USER_AGENT = "Balthasar [email protected]:

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/RtmpvaBIOY/remotes937f2cfc9c69/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

# edgar_agent <- Sys.setenv(EDGARWEBR_USER_AGENT = "edgarWebR [email protected]")
# ua <- httr::user_agent(edgar_agent)
# options(HTTPUserAgent=ua$options$useragent)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] "Balthasar  [email protected]"

Created on 2022-05-25 by the reprex package (v2.0.1)

So something must be overriding the correct value for the user agent variable....

balthasars avatar May 25 '22 21:05 balthasars

I had been able to install and use the Balthazar version by specifying the @fix-user-agent branch in the "remotes" request. This was a few months ago and I'm not at my PC right now.

I'm not sure what if anything has changed lately, but a get company info request earlier today worked.

Mconsidine

mconsidine avatar May 25 '22 22:05 mconsidine

I had luck by changing the edgar_GET function

edgar_GET_V2 <- function(href, ua = httr::user_agent(Sys.getenv('EDGARWEBR_USER_AGENT'))){ res <- httr::GET(href, ua) check_result(res) return(res))

environment(edgar_GET_V2) <- asNamespace('edgarWebR')

assignInNamespace("edgar_GET", edgar_GET_V2, ns = "edgarWebR")

twood886 avatar Jun 01 '22 16:06 twood886

@twood886 solution worked for me

alejandroll10 avatar Nov 24 '22 06:11 alejandroll10

As of last week, @twood886 had the only solution that worked for me. As of today, it no longer works.

tripletvn avatar May 15 '23 01:05 tripletvn

Sorry for only getting back to this now, everyone!

So it appears that the install of my proposed fork doesn't work when installing it using remotes::install_github():

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/Rtmp2UYpBk/remotes894768da6952/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
#> No encoding supplied: defaulting to UTF-8.
#> Error in `mutate()`:
#> ! Problem while computing `results = purrr::map(search_terms,
#>   edgarWebR::company_search, type = "13F-HR")`.
#> Caused by error in `check_result()`:
#> ! EDGAR request blocked from Undeclared Automated Tool.
#> Please visit https://www.sec.gov/developer for best practices.
#> See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
#> Consider also setting the environment variable 'EDGARWEBR_USER_AGENT
bank_search_results
#> Error in eval(expr, envir, enclos): object 'bank_search_results' not found

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

Created on 2022-05-25 by the reprex package (v2.0.1)

BUT: when I clone the repo, build the package in RStudio from there and then run my example, ta-dah:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

Created on 2022-05-25 by the reprex package (v2.0.1)

I am puzzled by this behavior and will need to read up some more on environmental variables.

UPDATE: The install using remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE) does also work if you set the environmental variable beforehand, using usethis::edit_r_environ(), such as EDGARWEBR_USER_AGENT = "Balthasar [email protected]:

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/RtmpvaBIOY/remotes937f2cfc9c69/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

# edgar_agent <- Sys.setenv(EDGARWEBR_USER_AGENT = "edgarWebR [email protected]")
# ua <- httr::user_agent(edgar_agent)
# options(HTTPUserAgent=ua$options$useragent)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] "Balthasar  [email protected]"

Created on 2022-05-25 by the reprex package (v2.0.1)

So something must be overriding the correct value for the user agent variable....

spacecadet333 avatar Aug 30 '23 17:08 spacecadet333