RFC: (Optionally?) hide gin from tracebacks

Open jlebar opened this issue 6 years ago • 1 comments

Right now wrapping a function with gin adds two lines to every traceback. For projects which use gin heavily, this gets to be...kind of a lot, see example at the bottom of this bug report.

Would you be willing to take a patch which (optionally? always?) hides gin from stack traces?

(I'm no Python expert, but I think it's possible to do this, see e.g. https://stackoverflow.com/questions/23765707/python-2-and-3-compatible-way-of-hiding-a-decorator-from-stacktrace)

Example of highly-decorated stack trace:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trainer.py", line 133, in <module>
    app.run(main)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trainer.py", line 129, in main
    trax.train(output_dir=output_dir)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
    six.raise_from(proxy.with_traceback(exception.__traceback__), None)
  File "<string>", line 3, in raise_from
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trax.py", line 878, in train
    save_steps=save_steps, has_weights=has_weights)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
    six.raise_from(proxy.with_traceback(exception.__traceback__), None)
  File "<string>", line 3, in raise_from
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trax.py", line 540, in __init__
    inputs = inputs(n_devices)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
    six.raise_from(proxy.with_traceback(exception.__traceback__), None)
  File "<string>", line 3, in raise_from
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 89, in inputs
    dataset_name, data_dir, input_name, n_devices)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 527, in _train_and_eval_batches
    dataset, data_dir)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
    six.raise_from(proxy.with_traceback(exception.__traceback__), None)
  File "<string>", line 3, in raise_from
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 217, in train_and_eval_dataset
    return _train_and_eval_dataset_v1(dataset_name[4:], data_dir)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 280, in _train_and_eval_dataset_v1
    train_dataset = problem.dataset(tf.estimator.ModeKeys.TRAIN, data_dir)
  File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/data_generators/problem.py", line 646, in dataset
    data_filepattern))
  File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/data/parallel_reader.py", line 316, in get_data_files
    raise ValueError('No data files found in %s' % (data_sources,))

Sep 10 '19 17:09 jlebar

If you want to give it a try and create a CL, we can review it.

On Tue, Sep 10, 2019, 10:45 AM Justin Lebar [email protected] wrote:

Right now wrapping a function with gin adds two lines to every traceback. For projects which use gin heavily, this gets to be...kind of a lot, see example at the bottom of this bug report.

Would you be willing to take a patch which (optionally? always?) hides gin from stack traces?

(I'm no Python expert, but I think it's possible to do this, see e.g. https://stackoverflow.com/questions/23765707/python-2-and-3-compatible-way-of-hiding-a-decorator-from-stacktrace )

Example of highly-decorated stack trace:

Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trainer.py", line 133, in app.run(main) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run _run_main(main, args) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trainer.py", line 129, in main trax.train(output_dir=output_dir) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper utils.augment_exception_message_and_reraise(e, err_str) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise six.raise_from(proxy.with_traceback(exception.traceback), None) File "", line 3, in raise_from File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper return fn(*new_args, **new_kwargs) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trax.py", line 878, in train save_steps=save_steps, has_weights=has_weights) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper utils.augment_exception_message_and_reraise(e, err_str) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise six.raise_from(proxy.with_traceback(exception.traceback), None) File "", line 3, in raise_from File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper return fn(*new_args, **new_kwargs) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/trax.py", line 540, in init inputs = inputs(n_devices) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper utils.augment_exception_message_and_reraise(e, err_str) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise six.raise_from(proxy.with_traceback(exception.traceback), None) File "", line 3, in raise_from File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper return fn(*new_args, **new_kwargs) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 89, in inputs dataset_name, data_dir, input_name, n_devices) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 527, in _train_and_eval_batches dataset, data_dir) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1073, in gin_wrapper utils.augment_exception_message_and_reraise(e, err_str) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise six.raise_from(proxy.with_traceback(exception.traceback), None) File "", line 3, in raise_from File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/gin/config.py", line 1050, in gin_wrapper return fn(*new_args, **new_kwargs) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 217, in train_and_eval_dataset return _train_and_eval_dataset_v1(dataset_name[4:], data_dir) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/trax/inputs.py", line 280, in _train_and_eval_dataset_v1 train_dataset = problem.dataset(tf.estimator.ModeKeys.TRAIN, data_dir) File "/usr/local/google/home/jlebar/code/tensor2tensor/tensor2tensor/data_generators/problem.py", line 646, in dataset data_filepattern)) File "/usr/local/google/home/jlebar/.local/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/data/parallel_reader.py", line 316, in get_data_files raise ValueError('No data files found in %s' % (data_sources,))

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/gin-config/issues/29?email_source=notifications&email_token=AANPI7GUBSHWJHTFUQUV4KDQI7MK3A5CNFSM4IVKVOW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HKQOEFA, or mute the thread https://github.com/notifications/unsubscribe-auth/AANPI7HTASWIHRT7CCNF363QI7MK3ANCNFSM4IVKVOWQ .

Sep 10 '19 23:09 sguada