ValueError: Invalid metric specification: bart_score_mt

[iglee]

Ran into value error when I tried running bart_score_mt.

Traceback:

Traceback (most recent call last):
  File "/Users/iglee/opt/anaconda3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/Users/iglee/opt/anaconda3/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/eaas/async_client.py", line 45, in fn
    self._results[request_id] = original_fn()
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/eaas/async_client.py", line 58, in <lambda>
    lambda: super(AsyncClient, self).score(inputs, metrics, calculate)
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/eaas/client.py", line 56, in score
    raise ValueError(f"Invalid metric specification: {metric}")
ValueError: Invalid metric specification: bart_score_mt
Traceback (most recent call last):
  File "/Users/iglee/opt/anaconda3/bin/explainaboard", line 8, in <module>
    sys.exit(main())
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/explainaboard_main.py", line 409, in main
    report = get_processor(task=task).process(
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/processors/processor.py", line 654, in process
    overall_statistics = self.get_overall_statistics(metadata, sys_output)
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/processors/processor.py", line 524, in get_overall_statistics
    overall_results = self.get_overall_performance(
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/processors/processor.py", line 458, in get_overall_performance
    metric_result = metric_func.evaluate_from_stats(
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/metric.py", line 214, in evaluate_from_stats
    agg_stats = self.aggregate_stats(stats)
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/metric.py", line 590, in aggregate_stats
    return np.mean(stats.get_data(), axis=0)
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/metric.py", line 543, in get_data
    self._fetch_results()
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/explainaboard/metric.py", line 531, in _fetch_results
    result = self.eaas_request.get_result()
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/eaas/async_client.py", line 20, in get_result
    self._result = self._eaas_client.wait_and_get_result(self._request_id)
  File "/Users/iglee/opt/anaconda3/lib/python3.9/site-packages/eaas/async_client.py", line 65, in wait_and_get_result
    result = self._results[request_id]
KeyError: '3f6e6f4a-c40e-11ec-aae7-acde48001122'

but I saw as a supported metric on task.py https://github.com/neulab/ExplainaBoard/blob/5055013af62dd9c50c46e389ed3c678031a3a319/explainaboard/tasks.py#L46

[neubig]

So I think this is probably because BARTScore is taking too long to compute and timing out.

• Short term fix: don't use bart_score, or use it only on smaller files.
• Long term fix: we will make bart_score computation more efficient on EaaS so this doesn't happen.

Also, we need to fix the error message, which is a known issue: https://github.com/inspired-co/eaas_client/issues/19

[iglee]

Hi, thank you! I'll try a smaller file, but probably will use other metrics in the short term.

[neubig]

yes, I think sticking with other metrics is fine for now