parse_aws_s3_response: print(out) only if isTRUE(verbose)
Hi,
thanks for this great package!
I had the following issue when bulk-downloading data from a bucket (with non-existing keys) using save_object() (related to #228 and #306)
As the current implementation uses print() in parse_aws_s3_response() for printing the http response to the terminal
in case of http errors , these options do not suppress this:
- wrapping the code into
suppressMessages() -
try(..., silent = TRUE)
When trying to download a lot of objects, the terminal is cluttered with all the printed statements.
library(aws.s3)
library(dplyr)
bucket = "dataforgood-fb-data"
prefix = "csv/month=2019-06/country=BEN"
content = aws.s3::get_bucket_df(bucket, prefix = prefix)
content = content %>%
mutate(
Size = R.utils::hsize(as.numeric(Size)),
Path = file.path(tempdir(), "BEN", basename(Key))
) %>%
select(Key, Size, Bucket, Path)
# keep only 1st entry, break 2nd key
content_sub = content %>%
slice_head(n = 2)
content_sub$Key[2] = file.path(dirname(content_sub$Key[2]), "abc.csv.gz")
content_sub
#> Key Size Bucket Path
#> 1 csv/month=2019-06/country=BEN/type=children_under_five/BEN_children_under_five.csv.gz 6.3 MiB dataforgood-fb-data /tmp/Rtmp1z0wTX/BEN/BEN_children_under_five.csv.gz
#> 2 csv/month=2019-06/country=BEN/type=elderly_60_plus/abc.csv.gz 6.3 MiB dataforgood-fb-data /tmp/Rtmp1z0wTX/BEN/BEN_elderly_60_plus.csv.gz
# save objects (try)
res = purrr::map2(
.x = content_sub$Key,
.y = content_sub$Path,
.f = ~try(
aws.s3::save_object(object = .x, bucket = bucket, file = .y),
silent = TRUE
)
)
#> List of 5
#> $ Code : chr "NoSuchKey"
#> $ Message : chr "The specified key does not exist."
#> $ Key : chr "csv/month=2019-06/country=BEN/type=elderly_60_plus/abc.csv.gz"
#> $ RequestId: chr "JBWBAGGY2235FR52"
#> $ HostId : chr "VzlvI8tyT0dMDhzCyQhnepo3aGQ3KKSc/MELh4fvAkVnGaDVgbLuxI1f8MRUdFmw9bmqLLDOxx0="
#> - attr(*, "headers")=List of 6
#> ..$ x-amz-request-id : chr "JBWBAGGY2235FR52"
#> ..$ x-amz-id-2 : chr "VzlvI8tyT0dMDhzCyQhnepo3aGQ3KKSc/MELh4fvAkVnGaDVgbLuxI1f8MRUdFmw9bmqLLDOxx0="
#> ..$ content-type : chr "application/xml"
#> ..$ transfer-encoding: chr "chunked"
#> ..$ date : chr "Mon, 13 Sep 2021 09:42:26 GMT"
#> ..$ server : chr "AmazonS3"
#> ..- attr(*, "class")= chr [1:2] "insensitive" "list"
#> - attr(*, "class")= chr "aws_error"
#> - attr(*, "request_canonical")= chr "GET\n/dataforgood-fb-data/csv/month%3D2019-06/country%3DBEN/type%3Delderly_60_plus/abc.csv.gz\n\nhost:s3.amazon"| __truncated__
#> - attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n20210913T094227Z\n20210913/us-east-1/s3/aws4_request\n38938aaacd4e153aa3d60ad428825a535958c7b"| __truncated__
#> - attr(*, "request_signature")= chr "AWS4-HMAC-SHA256 Credential=AKIA3ATRX55B3CXRVM35/20210913/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-da"| __truncated__
#> NULL
Created on 2021-09-13 by the reprex package (v2.0.1)
One option (at user side) would be to wrap it like
capture.output(aws.s3::save_object(object = key, bucket = bucket), file = 'NUL'),
but this changes to character string output.
What about print(out) only on isTRUE(verbose)?