LaunchPlatform
Hiccup with pushtag
Hi!
I'm running into a hiccup trying to import transactions into a project that contains an unrelated file that uses push/poptags directives.
If we define a Beancount file with `pushtag` and `poptag` directives:
pushtag #test
2000-01-01 open Assets:Test
2000-01-01 open Expenses:Test
2001-01-01 * "Test"
Expenses:Test 1 USD
Assets:Test
poptag #test
And provide a placeholder configuration:
inputs: []
imports: []
Running bh import fails with a parsing error:
bh import -c empty-config.yaml -b document-with-pushtag.beancount 2>&1
[22:22:59] INFO Loaded import doc from config.yaml import_cli.py:72
INFO Generated 0 transactions import_cli.py:109
INFO Deleted 0 transactions import_cli.py:110
INFO Skipped 0 transactions import_cli.py:111
INFO Collecting existing imported transactions import_cli.py:122
from Beancount books ...
Traceback (most recent call last):
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 665, in lex
yield lexer.next_token(lexer_state, parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 598, in next_token
raise UnexpectedCharacters(lex_state.text, line_ctr.char_pos, line_ctr.line, line_ctr.column,
lark.exceptions.UnexpectedCharacters: No terminal matches '#' in the current parser context, at line 1 col 9
pushtag #test
^
Expected one of:
* COLON
Previous tokens: Token('METADATA_KEY', 'pushtag')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/nix/store/49ks2fr4p22hddm76cjfky1hgy8qfb7c-python3.12-beanhub-cli-2.1.1/bin/.bh-wrapped", line 9, in <module>
sys.exit(cli())
^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/sq347ia5a61cpc6jildf6w93f853ikad-devenv-profile/lib/python3.12/site-packages/beanhub_cli/import_cli.py", line 126, in main
existing_txns = list(
^^^^^
File "/nix/store/psjfv2mma7my7lfb0bhchh0pcljwgvbw-python3.12-beanhub-import-1.2.0/lib/python3.12/site-packages/beanhub_import/post_processor.py", line 52, in extract_existing_transactions
for bean_path, tree in traverse(
^^^^^^^^^
File "/nix/store/4afpjpmfa6bjvihyhbbjlnl0r7b6j6a0-python3.12-beancount-parser-1.2.3/lib/python3.12/site-packages/beancount_parser/parser.py", line 54, in traverse
tree = parser.parse(current_file.read_text())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lark.py", line 655, in parse
return self.parser.parse(text, start=start, on_error=on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parser_frontends.py", line 104, in parse
return self.parser.parse(stream, chosen_start, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
return self.parser.parse(lexer, start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
return self.parse_from_state(parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 111, in parse_from_state
raise e
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
for token in state.lexer.lex(state):
^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 674, in lex
raise UnexpectedToken(token, e.allowed, state=parser_state, token_history=[last_token], terminals_by_name=self.root_lexer.terminals_by_name)
lark.exceptions.UnexpectedToken: Unexpected token Token('TAGS', '#test') at line 1, column 9.
Expected one of:
* COLON
Previous tokens: [Token('METADATA_KEY', 'pushtag')]
Using a hash-less tag fails in a similar way:
pushtag test
2000-01-01 open Assets:Test
2000-01-01 open Expenses:Test
2001-01-01 * "Test"
Expenses:Test 1 USD
Assets:Test
poptag test
bh import -c empty-config.yaml -b document-with-hashless-pushtag.beancount 2>&1
[22:35:55] INFO Loaded import doc from empty-config.yaml import_cli.py:72
INFO Generated 0 transactions import_cli.py:109
INFO Deleted 0 transactions import_cli.py:110
INFO Skipped 0 transactions import_cli.py:111
INFO Collecting existing imported transactions import_cli.py:122
from Beancount books ...
Traceback (most recent call last):
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 665, in lex
yield lexer.next_token(lexer_state, parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 598, in next_token
raise UnexpectedCharacters(lex_state.text, line_ctr.char_pos, line_ctr.line, line_ctr.column,
lark.exceptions.UnexpectedCharacters: No terminal matches 't' in the current parser context, at line 1 col 9
pushtag test
^
Expected one of:
* COLON
Previous tokens: Token('METADATA_KEY', 'pushtag')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/nix/store/49ks2fr4p22hddm76cjfky1hgy8qfb7c-python3.12-beanhub-cli-2.1.1/bin/.bh-wrapped", line 9, in <module>
sys.exit(cli())
^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/sq347ia5a61cpc6jildf6w93f853ikad-devenv-profile/lib/python3.12/site-packages/beanhub_cli/import_cli.py", line 126, in main
existing_txns = list(
^^^^^
File "/nix/store/psjfv2mma7my7lfb0bhchh0pcljwgvbw-python3.12-beanhub-import-1.2.0/lib/python3.12/site-packages/beanhub_import/post_processor.py", line 52, in extract_existing_transactions
for bean_path, tree in traverse(
^^^^^^^^^
File "/nix/store/4afpjpmfa6bjvihyhbbjlnl0r7b6j6a0-python3.12-beancount-parser-1.2.3/lib/python3.12/site-packages/beancount_parser/parser.py", line 54, in traverse
tree = parser.parse(current_file.read_text())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lark.py", line 655, in parse
return self.parser.parse(text, start=start, on_error=on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parser_frontends.py", line 104, in parse
return self.parser.parse(stream, chosen_start, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
return self.parser.parse(lexer, start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
return self.parse_from_state(parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 111, in parse_from_state
raise e
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
for token in state.lexer.lex(state):
^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 674, in lex
raise UnexpectedToken(token, e.allowed, state=parser_state, token_history=[last_token], terminals_by_name=self.root_lexer.terminals_by_name)
lark.exceptions.UnexpectedToken: Unexpected token Token('METADATA_KEY', 'test') at line 1, column 9.
Expected one of:
* COLON
Previous tokens: [Token('METADATA_KEY', 'pushtag')]
Is this expected?
Side-note: it'd be very useful to have a file-name on the error output!
Thanks for this project!
Hi @aisamu, thanks for reporting the issue. Unfortunately we don't support push tag and have no plan to support in the short term. Because push tag makes sorting and formatting way more complex. We think it's a bit unnecessary given that if our users adopt tools like our beanhub-import, they can easily manage tags for the transactions they want.
Noted!
Just to reinforce, I ran into this while trying to migrate my existing ledgers to beanhub-import! I'm more than happy to have something handle tags other than myself.
Do you have a suggestion on how to handle existing/old files?
Manual conversion would be a bit tricky given the volume, and excluding them (e.g. via a separate safe-main.beancount) would potentially make things like duplicate detection fail for those accounts/files, right?
Currently, there's no easy way provided to convert your legacy Beancount file with pushtag syntax to not using pushtax instead. But the beancount-parser for parsing the syntax into syntax tree + beancount-black for formatting syntax tree back into beancount file can help.
In beanhub-cli, we provided account renaming feature along with the format subcommand:
https://beanhub-cli-docs.beanhub.io/commands/format/
We actually implemented a transformer to transform tne syntax tree for renaming account or currency:
https://github.com/LaunchPlatform/beanhub-cli/blob/f9a97e92bcbdb7e3ff7fc51bbcee9657503edf34/beanhub_cli/format.py#L75-L86
I think it's possible to create a transformer to inject tags into your new beancount file. I would envision write a simple script to record the line numbers of push tag and the tags to push. Then, write it to a JSON file or something like that one the side. Then, with a custom transformer, read your beancount file after commenting all the push tag syntax and see if the entry are between the line numbers you have recorded. If so, inject tag into the syntax tree and let beancount-black serialize them back into a new beancount file.
Sorry I know this is less than idea. But I am thinking a better way to do it. Another approach might be asking a LLM model to do the changes for you.