Cannot Open URL

[georgeaj]

No finreportr functions work when year = 2019. Have tested on multiple companies and multiple years, problem is not company specific and only exists when year = 2019.

GetBalanceSheet('GOOG', 2019)

Error in fileFromCache(file) : Error in download.file(file, cached.file, quiet = !verbose) : cannot open URL 'https://www.sec.gov/Archives/edgar/data/1652044/000165204419000004/https://xbrl.sec.gov/dei/2018/dei-2018-01-31.xsd'

Session Info:

R version 3.6.0 (2019-04-26)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

Random number generation:
 RNG:     Mersenne-Twister 
 Normal:  Inversion 
 Sample:  Rounding 
 
locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.4.0    stringr_1.4.0    dplyr_0.8.1      purrr_0.3.2      readr_1.3.1      tidyr_0.8.3      tibble_2.1.3     ggplot2_3.1.1   
 [9] tidyverse_1.2.1  finreportr_1.0.1 lubridate_1.7.4  rvest_0.3.4      xml2_1.2.0       edgar_2.0.1     

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5  xfun_0.7          slam_0.1-45       NLP_0.2-0         haven_2.1.0       lattice_0.20-38   colorspace_1.4-1 
 [8] generics_0.0.2    yaml_2.2.0        XML_3.98-1.20     rlang_0.3.4       R.oo_1.22.0       pillar_1.4.1      withr_2.1.2      
[15] glue_1.3.1        R.utils_2.8.0     selectr_0.4-1     readxl_1.3.1      modelr_0.1.4      plyr_1.8.4        cellranger_1.1.0 
[22] munsell_0.5.0     gtable_0.3.0      R.methodsS3_1.7.1 XBRL_0.99.18      qdapRegex_0.7.2   knitr_1.23        tm_0.7-6         
[29] parallel_3.6.0    curl_3.3          broom_0.5.2       Rcpp_1.0.1        backports_1.1.4   scales_1.0.0      jsonlite_1.6     
[36] hms_0.4.2         stringi_1.4.3     grid_3.6.0        cli_1.1.0         tools_3.6.0       magrittr_1.5      lazyeval_0.2.2   
[43] crayon_1.3.4      pkgconfig_2.0.2   assertthat_0.2.1  httr_1.4.0        rstudioapi_0.10   R6_2.4.0          nlme_3.1-139     
[50] compiler_3.6.0   

[dchen728]

Hi author and georgeaj, is this issue resolved? I am having the same issue and trying to figure out why.
This is a great package and really helpful to pull annual data. Thanks.

[georgeaj]

I haven’t tried it in a few days. I was able to get one company’s 2019 data one time. I suspect it could possibly be that the SEC database can’t handle the amount of requests it gets for current data every day and so it returns nothing. If this is the case then there may not be a solution. After having this problem I wrote my own function to pull the data from the SEC’s excel files that are posted with every filing. I may make it into a package if I get the rest of the kinks out. What method does finreportr use to get the data?

…

On Jun 26, 2019, at 4:59 PM, dchen728 ***@***.***> wrote: Hi author and georgeaj, is this issue resolved? I am having the same issue and trying to figure out why. This is a great package and really helpful to pull annual data. Thanks. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[dchen728]

My understanding is that finreportr pulls the data in XML format from SEC and then parse and convert the data into dataframe in R. It would be great if you could make your function into a package since there are very few ways currently available to pull SEC data into R.

[dchen728]

I haven’t tried it in a few days. I was able to get one company’s 2019 data one time. I suspect it could possibly be that the SEC database can’t handle the amount of requests it gets for current data every day and so it returns nothing. If this is the case then there may not be a solution. After having this problem I wrote my own function to pull the data from the SEC’s excel files that are posted with every filing. I may make it into a package if I get the rest of the kinks out. What method does finreportr use to get the data?
…
On Jun 26, 2019, at 4:59 PM, dchen728 @.***> wrote: Hi author and georgeaj, is this issue resolved? I am having the same issue and trying to figure out why. This is a great package and really helpful to pull annual data. Thanks. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

My understanding is that finreportr pulls the data in XML format from SEC and then parse and convert the data into dataframe in R. It would be great if you could make your function into a package since there are very few ways currently available to pull SEC data into R.

[sewardlee337]

Hello @georgeaj @dchen728 ,

Thank you very much for reporting this issue. Much apologies for the late reply -- I have been very busy lately.

I will take a look this week, and will be in touch if I need help testing patches for this bug.

[sewardlee337]

A brief update:

From what I'm seeing, the underlying issue appears to be due to something about the way the XBRL package interfaces with EDGAR. When finreportr pulls and parses XBRL-format data from the U.S. Securities and Exchange Commission, it calls the XBRL package function xbrlDoAll().

For example, if you try to run:

## ORCL's 2019 financials
url <- "https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531.xml"

## Call xbrlDoAll(), in verbose mode
XBRL::xbrlDoAll(url, cache.dir='XBRLcache',prefix.out="out",verbose=TRUE)

The printout you receive is:

Downloading to cache dir...trying URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531.xml'
downloaded 6.2 MB

Schema:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531.xsd 
Level: 1 ==> https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531.xsd 
Downloading to cache dir...trying URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531.xsd'
downloaded 98 KB

Roles
Elements
XBRLcache/orcl-20190531.xsd  ==> Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_cal.xml 
Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_cal.xml 
Level: 2 ==> https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_cal.xml 
Downloading to cache dir...trying URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_cal.xml'
downloaded 119 KB

Calculations.
XBRLcache/orcl-20190531.xsd  ==> Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_def.xml 
Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_def.xml 
Level: 2 ==> https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_def.xml 
Downloading to cache dir...trying URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_def.xml'
downloaded 356 KB

Definitions.
XBRLcache/orcl-20190531.xsd  ==> Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_lab.xml 
Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_lab.xml 
Level: 2 ==> https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_lab.xml 
Downloading to cache dir...trying URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_lab.xml'
downloaded 879 KB

Labels.
XBRLcache/orcl-20190531.xsd  ==> Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_pre.xml 
Linkbase:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_pre.xml 
Level: 2 ==> https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_pre.xml 
Downloading to cache dir...trying URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-20190531_pre.xml'
downloaded 643 KB

Presentations.
XBRLcache/orcl-20190531.xsd  ==> Schema:  http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd 
Schema:  http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd 
Level: 2 ==> http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd 
Using file from cache dir...
Elements
XBRLcache/xbrl-instance-2003-12-31.xsd  ==> Schema:  http://www.xbrl.org/2003/xbrl-linkbase-2003-12-31.xsd 
Schema:  http://www.xbrl.org/2003/xbrl-linkbase-2003-12-31.xsd 
Level: 3 ==> http://www.xbrl.org/2003/xbrl-linkbase-2003-12-31.xsd 
Using file from cache dir...
Elements
XBRLcache/xbrl-linkbase-2003-12-31.xsd  ==> Schema:  http://www.xbrl.org/2003/xl-2003-12-31.xsd 
Schema:  http://www.xbrl.org/2003/xl-2003-12-31.xsd 
Level: 4 ==> http://www.xbrl.org/2003/xl-2003-12-31.xsd 
Using file from cache dir...
Elements
XBRLcache/xl-2003-12-31.xsd  ==> Schema:  http://www.xbrl.org/2003/xlink-2003-12-31.xsd 
Schema:  http://www.xbrl.org/2003/xlink-2003-12-31.xsd 
Level: 5 ==> http://www.xbrl.org/2003/xlink-2003-12-31.xsd 
Using file from cache dir...
Elements
XBRLcache/xbrl-linkbase-2003-12-31.xsd  ==> Schema:  http://www.xbrl.org/2003/xlink-2003-12-31.xsd 
Schema:  http://www.xbrl.org/2003/xlink-2003-12-31.xsd 
Already discovered. Skipping
XBRLcache/orcl-20190531.xsd  ==> Schema:  http://www.xbrl.org/2003/xbrl-linkbase-2003-12-31.xsd 
Schema:  http://www.xbrl.org/2003/xbrl-linkbase-2003-12-31.xsd 
Already discovered. Skipping
XBRLcache/orcl-20190531.xsd  ==> Schema:  http://www.xbrl.org/2005/xbrldt-2005.xsd 
Schema:  http://www.xbrl.org/2005/xbrldt-2005.xsd 
Level: 2 ==> http://www.xbrl.org/2005/xbrldt-2005.xsd 
Using file from cache dir...
Elements
XBRLcache/xbrldt-2005.xsd  ==> Schema:  http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd 
Schema:  http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd 
Already discovered. Skipping
XBRLcache/orcl-20190531.xsd  ==> Schema:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/https://xbrl.sec.gov/dei/2018/dei-2018-01-31.xsd 
Schema:  https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/https://xbrl.sec.gov/dei/2018/dei-2018-01-31.xsd 
Level: 2 ==> https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/https://xbrl.sec.gov/dei/2018/dei-2018-01-31.xsd 
Downloading to cache dir...trying URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/https://xbrl.sec.gov/dei/2018/dei-2018-01-31.xsd'
Error in fileFromCache(file) : 
  Error in download.file(file, cached.file, quiet = !verbose) : 
  cannot open URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/https://xbrl.sec.gov/dei/2018/dei-2018-01-31.xsd'

In addition: Warning message:
In download.file(file, cached.file, quiet = !verbose) :
  cannot open URL 'https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/https://xbrl.sec.gov/dei/2018/dei-2018-01-31.xsd': HTTP status was '403 Forbidden'

This issue appears to affect all packages and applications that use this function in the XBRL package. For example: bergant/finstr#12

I will update when I find out more. Thank you so much for your patience.

[dchen728]

Hey Seward, thanks for the update. Look forward to hearing more updates. I will keep an eye on the XBRL package as well. Thanks again.

[sewardlee337]

I have written to the author of the XBRL package to see if he can offer some guidance.

[dchen728]

Thanks for the update.

[enFinExplorer]

I am having issues with a company that got delisted with a subsequent symbol change (EPE to EPEG).
GetIncome('EPEG', 2019) does not work because because the XML tag has epeg at the end as opposed to the name when it was filed, epe.
'https://www.sec.gov/Archives/edgar/data/1584952/000158495219000003/epeg-20181231.xml'

[shyams80]

I have written to the author of the XBRL package to see if he can offer some guidance.

Any luck with this?

[selgamal]

I have written to the author of the XBRL package to see if he can offer some guidance.

Any luck with this?

Please see this SO question , might help hack a solution if this is urgent for you.

[mfarr76]

I was able to fix the XBRL package with the SO question from above but I ran into another problem where GetFiniancials for a 2019 report year would return the following error:
Error: Result must have length 1011, not 0

After doing some digging it appears that the descriptions of cash flow statements, balance sheets, and income have changed from previous years. The 2019 report I was looking at (symbol "SM") has the following:

CONSOLIDATED BALANCE SHEETS (in thousands, except share data)
CONSOLIDATED STATEMENTS OF CASH FLOWS (in thousands)
CONSOLIDATED STATEMENTS OF OPERATIONS (in thousands, except per share data)

For example, GetIncome only looks for these column headers:

income.descriptions <- c("CONSOLIDATED STATEMENTS OF INCOME", "CONSOLIDATED STATEMENT OF INCOME", "CONSOLIDATED STATEMENTS OF OPERATIONS", "CONSOLIDATED STATEMENT OF OPERATIONS", "CONSOLIDATED STATEMENT OF EARNINGS", "CONSOLIDATED STATEMENTS OF EARNINGS", "INCOME STATEMENTS", "CONSOLIDATED RESULTS OF OPERATIONS")

I made the correction to the descriptions and was able to download the data.

Just thought I would pass along.

[GreenGrassBlueOcean]

I am also working with this package and observed the same behavior. There are however two distinct problems arising at the same time.

1. XBRL is not generating correct URLs
2. The naming of xml files on the sec website has changed recently.

XBRL:
The bug in the XBRL package should be fixed as indicated above (I also found the same solution independently). When working working on windows it is best to rebuild the XBRL package from source with the changed file . The libxml file is not available for the compiler and should best be added according to this url: https://stackoverflow.com/questions/39568937/how-to-create-cran-ready-r-package-that-has-external-dependency-libxml2

SEC NAMES:
when running (with the fixed XBRL package) the following code:

Income <- finreportr::GetIncome("SBUX", 2019)

results in the following error

Error in fileFromCache(file.inst) : 
  Error in download.file(file, cached.file, quiet = !verbose) : 
  cannot open URL 'https://www.sec.gov/Archives/edgar/data/829224/000082922419000051/sbux-20190929.xml'

if I then check if this file is present on the Edgar website:
https://www.sec.gov/Archives/edgar/data/829224/000082922419000051/

I notice that the file sbux-20190929.xml is not present on EDGAR.
I checked this also for other companies like Google and Boeing and they all observer the same behavior.

When I then try to find what should be the correct name using the edgarWebR (https://github.com/mwaldstein/edgarWebR) package:

FilingsonEdgar <- edgarWebR::company_filings(x = "SBUX", type = "10-K")
DocumentsonEdgar <-  edgarWebR::filing_documents(x = test$href[1])
link <- DocumentsonEdgar[DocumentsonEdgar[5] == 'XML', 4]

I get the following URL:

https://www.sec.gov/Archives/edgar/data/12927/000001292719000077/a201909sep3010-q_htm.xml

When passing this URL to the revised XBRL package

xbrl.vars <- XBRL::xbrlDoAll(link, verbose=TRUE)

it downloads the data correctly.

CONCLUSION:
The finreportr::GetIncome function generates the wrong URL for using in XBRL.
When using the debug statement in Rstudio for finreportr::GetIncome you end up in the function finreportr::GetFinancial which has a in-function helper function called GetURL which generates a static URL. I would propose to replace the GetURL function with a slight adaptation of the above mentioned way to retrieve the right url by using the edgarWebR package

[Handiel]

Hello, I am trying to load J.P. Morgan income statement, but I get the following error, could you help me with some solution. Thanks in advance.

>GetIncome("JPM", 2019)

Error in fileFromCache(file.inst) : Error in download.file(file, cached.file, quiet = !verbose) : no fue posible abrir la URL 'https://www.sec.gov/Archives/edgar/data/19617/000001961719000054/jpm-20181231.xml'_