httr
httr copied to clipboard
r-lib
R/httr and R/curl much slower than download.file
When querying an endpoint of a REST API I set up with R/plumber, I experience substantial time differences between R/httr, R/curl and download.file. While download.file takes around 8 seconds, httr and curl take 25 seconds or more for exactly the same query using the same client and host server (both CentOS 7). With wget and curl on command line it also takes 8 seconds.
Querying the API
system.time({
r = GET("http://myserver.com/path/to/endpoint")
warn_for_status(r)
stop_for_status(r)
x = content(r)
})
# user system elapsed
# 9.329 0.345 32.946
system.time({
req = curl_fetch_memory("http://myserver.com/path/to/endpoint")
})
# user system elapsed
# 1.707 0.268 25.160
system.time({
curl_download("http://myserver.com/path/to/endpoint", tempfile())
})
# user system elapsed
# 1.608 0.417 25.237
system.time({
curl_fetch_disk("http://myserver.com/path/to/endpoint", tempfile())
})
# user system elapsed
# 1.656 0.467 25.468
system.time({
download.file("http://myserver.com/path/to/endpoint", tempfile())
})
# user system elapsed
# 0.980 0.469 8.260
Response header
> headers(r)
$server
[1] "nginx/1.21.1"
$date
[1] "Wed, 29 Sep 2021 14:51:36 GMT"
$`content-type`
[1] "text/tab-separated-values; charset=UTF-8"
$`transfer-encoding`
[1] "chunked"
$connection
[1] "keep-alive"
$`content-encoding`
[1] "gzip"
attr(,"class")
[1] "insensitive" "list"
sessionInfo()
> sessionInfo()
R version 4.0.1 (2020-06-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblasp-r0.3.3.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] curl_4.3.2 httr_1.4.2
loaded via a namespace (and not attached):
[1] webutils_1.1 tidyselect_1.1.1 remotes_2.4.0 purrr_0.3.4 lattice_0.20-44 swagger_3.33.1 vctrs_0.3.8 generics_0.1.0
[9] testthat_3.0.2 usethis_2.0.1 SnowballC_0.7.0 tidytext_0.3.1 utf8_1.2.2 blob_1.2.2 rlang_0.4.11 pkgbuild_1.2.0
[17] pillar_1.6.2 later_1.2.0 glue_1.4.2 withr_2.4.2 DBI_1.1.1 bit64_4.0.5 sessioninfo_1.1.1 lifecycle_1.0.0
[25] stringr_1.4.0 devtools_2.4.2 memoise_2.0.0 callr_3.7.0 fastmap_1.1.0 ps_1.6.0 fansi_0.5.0 tokenizers_0.2.1
[33] Rcpp_1.0.7 promises_1.2.0.1 cachem_1.0.5 desc_1.3.0 pkgload_1.2.1 jsonlite_1.7.2 fs_1.5.0 bit_4.0.4
[41] stringi_1.6.2 processx_3.5.2 dplyr_1.0.7 plumber_1.1.0 grid_4.0.1 rprojroot_2.0.2 cli_3.0.1 tools_4.0.1
[49] magrittr_2.0.1 RSQLite_2.2.7 tibble_3.1.3 janeaustenr_0.1.5 crayon_1.4.1 pkgconfig_2.0.3 ellipsis_0.3.2 Matrix_1.3-4
[57] data.table_1.14.0 prettyunits_1.1.1 assertthat_0.2.1 R6_2.5.0 compiler_4.0.1
libcurl version
libcurl 7.29.0 with NSS/3.53.1
Update 04-10-2021
There seems to be a discrepancy between result of headers() and what the actual header is:
> library(httr)
> system.time({
+ r = GET("http://127.0.0.1:4000/test")
+ warn_for_status(r)
+ stop_for_status(r)
+ x = content(r)
+ })
Rows: 350 Columns: 40
── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: "\t"
dbl (40): X1, X2, X3, X4, X5, X6, X7, X8, X9, X10, X11, X12, X13, X14, X15, ...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
user system elapsed
0.478 0.059 0.751
> r$headers
$date
[1] "Mon, 04 Oct 2021 07:47:47 GMT"
$`content-type`
[1] "text/tab-separated-values; charset=UTF-8"
$`content-encoding`
[1] "gzip"
$`transfer-encoding`
[1] "chunked"
attr(,"class")
[1] "insensitive" "list"
> r
Response [http://127.0.0.1:4000/test]
Date: 2021-10-04 07:47
Status: 200
Content-Type: text/tab-separated-values; charset=UTF-8
Size: 255 kB
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X2...
14.005301101133227 23.625352163799107 92.48203919269145 53.32668418996036 43....
49.71931471955031 93.39884805958718 11.632965900935233 85.32237017061561 91.9...
78.89712990727276 39.409890957176685 90.90405236929655 0.5622981814667583 15....
41.816325089894235 47.52457377035171 32.00235064141452 40.431985072791576 4.1...
0.009128847159445286 32.41147438529879 77.71695936098695 59.59550221450627 74...
36.60198471043259 15.195061289705336 25.27566181961447 42.5753008807078 85.85...
77.14250846765935 76.39204727020115 82.18081402592361 79.50914725661278 5.536...
80.04448262508959 65.86111697833985 12.915009562857449 22.876024013385177 10....
27.796625229530036 86.96586838923395 98.06580552831292 81.84361984021962 26.9...
...
