neogeny
Cleanup recovery of grammar comments
# comments retain the # and are output as (* *) comments.
Prettification of comments is not well defined.
It's best to stick to Python-style (#) comments.
Comments can be added to the TatSu grammar for rule and option. The corresponding classes in grammars would have a comment field for the pretty-printer to use.
Some tweaks may be needed for comments right before a rule definition, and right after, before the first element.
PEG allows for this.
Complications would be on the pretty printer (most important), though the code generator could also try to do its part.
The current design for attempting recovery of comments sucks!
This needs a better strategy, or deletion by giving grammar writers the responsibility for comment recovery.
I dealt with this issue when parsing COBOL and NATURAL. I'm closing to think about it when I need to do something alike again. Not all grammars are easy when comment recovery is attempted.
I'd take a look at the new PEG Python grammar for Python.
My subconscious says that this would be manageable with tokenization, as was used in the PEG Python grammar, more so when now in TatSu Buffer is one implementation of Tokenizer.
This issue will come back. I just don't want to leave it open without a design, strategy, or pull request.