Find a way to structure unit tests

[kamilzyla]

Problem

Due testthat, directories cannot be used to organize tests in the tests/testthat directory and tests must have names starting with test-. It might be hard to organize tests in complex projects using a flat directory structure. The test- prefix breaks the snake_case rule for filenames.

Ideas

1. Write unit tests as name.test.R next to name.R being tested.
2. Make it possible to use directories to organize tests under tests/testthat. We could then follow the Java convention of mirroring the source and test directory structure. It would be good to have an easy way to switch between the source and test file.
3. Consider using tinytest instead of testthat. Perhaps we should allow both options?

[telegott]

@kamilzyla referring to https://github.com/Appsilon/rhino/issues/319

After you mentioning tinytest I had a look at it and it seems much friendlier than testthat for this usecase since the result of tinytest::run_test_dir can be collected into a dataframe, which would allow running this function recursively on all folders, collecting all results of all directories into one dataframe, and return an exit code from the script, something along the lines of:

box::use(
  dplyr[...],
  purrr[imap_dfr, map],
  tibble[as_tibble],
)

dirs <- c(tests = fs::path('tests'), fs::dir_ls('tests', type = 'directory', recurse = TRUE))
tests <- map(dirs, tinytest::run_test_dir)

results <- tests |>
  imap_dfr(~as_tibble(.x) |> mutate(dir = .y)) |>
  mutate(path = fs::path(dir, file))

if (all(results$result)) {
  message('all tests passed')
  quit(status = 0)
} else {
  error_messages <- results |>
    filter(!result) |>
    mutate(across(c(call, diff), stringr::str_replace, pattern = '\n', replacement = '')) |>
    transmute(error_message = glue::glue('{path} | lines: {first}:{last} | call: {call} | error: {diff}')) |>
    pull(error_message) |>
    paste(collapse = '\n')
  stop('\n', error_messages)
}

Sample output:

> Rscript scripts/test.R                                                                                                                                                                                                                                    
test_helpers.R................    3 tests 1 fails 43ms
test_nested_function.R........    1 tests 1 fails 2ms
Error: 
tests/test_helpers.R | lines 6:6 | call: expect_equal(add(3, 5), 9) | error: Expected '9', got '8'
tests/nested/test_nested_function.R | lines 4:6 | call: expect_equal(nested_function(3), 10) | error:  Modes: numeric, character target is numeric, current is character
Execution halted

EDIT: Base R version:

test_dir <- function(directory) {
  message('Testing directory ', directory)
  test <- tinytest::run_test_dir(directory)
  frame <- as.data.frame(test)
  frame$path <- paste(directory, frame$file, sep = "/")
  frame
}

clean_characters <- \(vec) gsub("\r?\n|\r", "", vec)

dirs <- list.dirs(path = "tests", full.names = TRUE, recursive = TRUE)
tests <- do.call(what = 'rbind', args = lapply(dirs, test_dir))

if (all(tests$result)) {
  message('All tests passed')
  quit(status = 0)
} else {
  failed_tests <- tests[!tests$result,]
  for (column in c('call', 'diff')) {
    failed_tests[[column]] <- clean_characters(failed_tests[[column]])
  }
  error_messages <- paste(
    failed_tests$path,
    paste0('lines: ', failed_tests$first, ':', failed_tests$last),
    paste0('call: ', failed_tests$call),
    paste0('error: ', failed_tests$diff),
    sep = ' | '
  )
  stop('\n', paste(error_messages, collapse = '\n'))
}