Unable to find documentation to set User Agent

[pietjonker]

When retrieving filings, I receive an error for being an Undeclared Automated Tool. For example, when using

latest_filings()

I receive error

No encoding supplied: defaulting to UTF-8.
Error in check_result(res) : 
  EDGAR request blocked from Undeclared Automated Tool.
Please visit https://www.sec.gov/developer for best practices.
See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
Consider also setting the environment variable 'EDGARWEBR_USER_AGENT

I found in the README the following information:

Because of abusive use of this library, the SEC is likely to block its use “as is” without setting a custom ‘User Agent’ identifier. Details for setting a custom agent are below.

However, no details were given below. Could anyone help me on setting the user agent?

[xbrl-data]

Within edarWebR you need to set an env variable.
Sys.setenv('xxxx')

Then, i usually call
EDGARWEBR_USER_AGENT <- Sys.getenv('xxxx')

Also, for those of us that are getting errors due to the SEC shutting it down (despite never exceeding the stated limit on their webiste!), I've been substituting read_html with this:

html_retry <-  function(url1,   N.TRIES = 3L) {

  require(rvest)
  
  N.TRIES <- as.integer(N.TRIES)
  stopifnot(length(N.TRIES) == 1L, !is.na(N.TRIES))
  stopifnot(N.TRIES > 0L)
  
  while (N.TRIES > 0L) {
    result <- tryCatch(read_html(
      x = url1
    ), error = identity)
    if (!inherits(result, "error")) {
      break
    }
    ## Wait 0.5 seconds
    Sys.sleep(0.5)
    N.TRIES <- N.TRIES - 1L
  }
  
  if (N.TRIES == 0L) {
    stop(
      "'html_retry()' failed:",
      "\n  URL: ", url1,
      "\n  error: ", conditionMessage(result)
    )
  }
  
  invisible(result)
}

[mirayoon]

Hello,

thanks for developing this package. I set an env variable EDGARWEBR_USER_AGENT using Sys.setenv and tried 'filing_documents' function but I still receive the error message.

No encoding supplied: defaulting to UTF-8. Error in check_result(res) : EDGAR request blocked from Undeclared Automated Tool. Please visit https://www.sec.gov/developer for best practices. See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities Consider also setting the environment variable 'EDGARWEBR_USER_AGENT

could you help me how to proceed from here?

[YuMan-Tam]

@mirayoon : I have the same issue, but a try-catch block loop until successful like @xbrl-data mentioned will work.

[mconsidine]

Has anyone gotten this to work lately? I've been setting the user agent correctly per the SEC website and I'm still not able to return anything other than the messages above.
How has the try-catch routine as described been substituted for what xml2 has? Does all the underlying R code need to be editted?
Also, does Accept-Encoding: gzip, deflate need to somehow be set?
TIA,
mconsidine

[xbrl-data]

Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with.

[mconsidine]

I've managed to hack some of the code to work with a with_config wrapper around a GET call.  But things only break down again later...On Sep 29, 2021 4:59 PM, xbrl-data ***@***.***> wrote: Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with. —You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[mconsidine]

It seems like some formatting at the SEC site has changed. E.g. the /feed/company-info node seems empty. Cik_ search works though.

Does anyone have a workflow they can share for low frequency requests for data access/parsing/report analysis if this is now busted? I was hoping to pair this with finstr-upd/xbrl-upd to show a college class.

Mconsidine

[mconsidine]

@xbrl-data : can you give me an idea how you have used those as a workaround? A PM would be fine, if you'd prefer.

[IEORTools]

@mirayoon I believe this is the proper way to set the environment variable

Sys.setenv(EDGARWEBR_USER_AGENT = " yourcompanyname youremail")

where yourcompanyname and youremail are specified by the user.

You can check it is properly set using the following

Sys.getenv("EDGARWEBR_USER_AGENT")

[balthasars]

@mirayoon I believe this is the proper way to set the environment variable

Sys.setenv(EDGARWEBR_USER_AGENT = " yourcompanyname youremail")

where yourcompanyname and youremail are specified by the user.

You can check it is properly set using the following

Sys.getenv("EDGARWEBR_USER_AGENT")

For me neitherSys.setenv() nor usethis::edit_r_environ() works — try using the fork I've suggested (see above).

[granatakorps]

Hi @balthasars thank you for sharing your solution but, unfortunately, it does not seem to work for me.
Any other tips on how to solve it?

[mconsidine]

This has worked for me:

library(tidyverse)
library(httr)
library(XBRL)
library(edgarWebR)

edgar_agent <- Sys.getenv("EDGARWEBR_USER_AGENT",
unset = "edgarWebR edgarpackage@examplemail.com"
)
ua <- httr::user_agent(edgar_agent)
options(HTTPUserAgent=ua$options$useragent)

Haven't tried to figure out if there's redundancy/unused vars being set in that ...

HTH,
mconsidine

[snvv]

Hello
None of the above approaches works for me

[mconsidine]

I used the version found here
https://github.com/balthasars/edgarWebR/tree/fix-user-agent
Also, this appeared to work, looking back over some code (thought I think its a kludge):
with_config(user_agent(ua$options$useragent),search.result <- xml2::read_html(url))
mconsidine