Support for `start/end_keyword` in listener version 3

[bergstenarn]

I would like to see support for start_keyword and end_keyword in listener version 3.

[pekkaklarck]

The problem implementing this is that on the execution side there currently isn't any place where we could call start/end_keywords so that they would get data and result objects similarly as other listener v3 methods get. Changing that would require quite big changes to the code and doing them just to get this feature implemented hasn't been considered worth the effort.

In RF 4.0 the plan is to add new features like IF/ELSE (#3074) that require big changes to the execution side. When doing them, this issue definitely needs to be taken into account. Thus assigned this one to the v4.0 scope.

[Snooz82]

i really like this feature. including line numbers for start and end keyword! This would be a huge enabler for debuggers!

[dakaizou]

Hi @bergstenarn and @Snooz82 , I'm not sure what those problems mentioned by @pekkaklarck are. Maybe there are some potential issues, but I patched the keyword runners to get something similar, and it worked fine for me:

from functools import wraps

import robot
from robot.running.librarykeywordrunner import LibraryKeywordRunner
from robot.running.userkeywordrunner import UserKeywordRunner

ROBOT_LISTENER_API_VERSION = 3


def start_keyword(kw, context):
    print(kw.lineno)
    print(kw.source)


def end_keyword(kw, context):
    print(kw)


def patch_kwrunner(runner):
    old_run = runner.run

    @wraps(old_run)
    def wrapper(self, kw, context):
        start_keyword(kw, context)
        try:
            if False:
                # bypassing this keyword
                return self.dry_run(kw, context)
            return old_run(self, kw, context)
        finally:
            end_keyword(kw, context)

    runner.run = wrapper


def start_suite(suite, result):
    patch_kwrunner(LibraryKeywordRunner)
    patch_kwrunner(UserKeywordRunner)


if __name__ == "__main__":
    robot.run("main_suite.robot", listener=__file__)

[mkorpela]

Related draft pull request #3672

[damies13]

Hi,

rfswarm agent uses the following version 2 api methods:

• start_suite
• log_message
• end_keyword

So not having "end_keyword" in the version 3 listener API is preventing me upgrading rfswarm from the version 2 API, fortunately the version 2 API works in Robot Framework V3 and hopefully won't be depreciated be in Robot Framework V4?

https://github.com/damies13/rfswarm/issues/67

For my project I would be happy to skip V3 and jump to V4 if end_keyword never gets implemented in V3 but does get implemented in V4 api methods.

[Snooz82]

@damies13 Afaik there are no plans to deprecate Listener v2!

However at the moment it looks like adding start/end_keyword in listener v3 has to be postponed to RF 5 or later RF 4 minor Release.

With RF4 there will definitely not be a Listener V4! The listener Version is not attached to the Robot Version number.

[damies13]

@Snooz82 Thanks, that's good news 👍🏼

[pekkaklarck]

This is, very unfortunately, too much work for RF 4.0. Recent model changes (#3749) have improved the code related to this a lot but also shown that there's still quite a bit to do. RF 4.0 is already quite badly late, and it has lot of awesome features we want to release officially, that spending time with this issue is not anymore possible.

I hope we can get this issue done in the next bigger RF release be it 4.1 or 5.0. We'll make sure that source and line number information is available also for keywords via the listener API v2 (#3538) already in RF 4.0.

[AVaitkunas]

I would be willing to work on adding start/end keyword for listener v3. Is there already some work done? Or maybe someone could provide a guidance to start?

[AVaitkunas]

@pekkaklarck I have made some progress on this topic and created draft pull request (#4344 ). I would like to get your feedback/guidance before proceeding.

[pekkaklarck]

There has been a huge amount of work, as expected, but this this feature is finally getting ready! Some of the things that have been done:

• Internal logging has been enhanced to make data and result objects available at the same time without using kludges like the old ModelCombiner.
• Listener code has been refactored to make it easier to add the new listener methods.
• start/end_xxx methods were added for all control structures so we now have methods like start_for and end_return. They get data and result objects the same way as earlier methods such as start_test.
• start/end_keyword methods that are called with all keywords, by default, have been added. Also they get data and result. Notice that the data object basically contains just the keyword name and arguments that were used in the data.
• Separate start/end_user_keyword, start/end_library_keyword and start/end_invalid_keyword were added to give listeners possibility to get access to the keyword implementation. As the names imply, the first two are called with user and library keywords, respectively. The third one is called if a keyword isn't found or the keyword itself is invalid. These methods are called with three arguments data, implementation and result. data and result are the same as with start/end_keyword, but implementation is keyword type dependent and makes it possible inspect and also modify executed keywords in interesting ways. If these methods are defined, matching start/end_keyword are not called.
• Model objects related to test libraries. library keywords and invalid keywords have been totally rewritten. These keyword objects are passed to listeners and they have references to libraries. Old code was horrible spaghetti and passing those model objects to listeners wouldn't have been a good idea. Related to this, also resource file and user keyword code has been enhanced. All this was a rather big task, but the code is now considerably better also otherwise and these APIs now ought to be stable.
• Test related results are preserved in memory until tests end to keep them available for listeners. All result objects created during execution also have correct parent relations.

There's still some work to do as well:

• [x] New ForIteraion and WhileIteration data objects for representing individual executed loop iterations. We already have matching result objects, but data objects haven't been needed earlier.
• [x] New start/end_body_item listener methods to call as a fallback with all keywords and control structures. This would allow having one method (or a pair of methods) to listen for all body items.
• [x] Possibility for start_invalid_keyword to "handle" error situations by setting implementation.error = None.
• [ ] Error codes or some other way to separate different error situations to ease deciding which errors to handle. This isn't a must in RF 7.0.
• [x] Added methods need to be added also to the optional robot.api.ListenerV3 base class with appropriate type hints and documentation.
• [x] Tests. Lot of them.
• [ ] Documentation in the User Guide. Can be done after the forthcoming beta release or even after the release candidate.

[pekkaklarck]

I hope that interested users can test the new functionality and report possible problems and enhancement ideas. Right now you need to use the code directly from the version control, but the plan is to create a beta release soon as well.

Here's an example listener you can try:

def start_test(data, result):
    print()


def start_user_keyword(data, implementation, result):
    # We get a new `implementation` instance every time. Modifying it does not change
    # the original keyword. It could be done via `implementation.owner` if needed, though.
    implementation.body.create_keyword('Log to console', ['It wroks!'])


def start_library_keyword(data, implementation, result):
    print(f"Starting library keyword '{implementation.name}'.")
    print(f"Keyword exists in library '{implementation.owner.name}' "
          f"on line {implementation.lineno}.")
    print(f'Library instance: {implementation.owner.instance}')
    print(f"Keyword's parent: {implementation.parent}")


def start_invalid_keyword(data, implementation, result):
    print('Something bad happened:', implementation.error)


def start_keyword(data, result):
    raise TypeError('Not called because all `start_xxx_keyword` methods exist.')


def end_keyword(data, result):
    print(f"Keyword '{result.full_name}' on line {data.lineno} {result.status}ED.")

You can use it, for example, with this data:

*** Test Cases ***
Library keyword
    Log    Hello!

User keyword
    User keyword
    User keyword

Non-existing keyword
    Non-existing keyword

*** Keywords ***
User keyword
    Log    Hillo!

[pekkaklarck]

Tests have been added. There are only two tests in the body_items_v3.robot file under atest/robot, but they are pretty thorough. The first test validates that all listener methods are called and the latter validates modifying data. You can see tests. listeners and a library these tests use in the body_items_v3 directory under atest/testdata. If you are interested in all interesting (and to some extend crazy) ways the data can be manipulated, see the the DataModifier listener.

I wrote earlier that it might be a good idea to add error codes or some other mechanism for differentiating various error situations to ease handling them. I still believe it could be useful, but I'm not sure how important it would be or how it should actually work. I believe this is something that's best left for the future.

This feature ought to now be ready except for documentation. That mainly means User Guide, but probably API docs can be enhanced as well. That's something we can do after beta 1 or even after we have a release candidate out.

[pekkaklarck]

There were still problems related to modifying results. Because information was already written to output.xml, modifications didn't have any real effect. That was fixed by the above commit, and more tests for modifying results will be committed shortly. Tests are modified quite a bit also otherwise (for example, there will be multiple tests under atest/robot), but they use the same files that were mentioned in the comment above. Hopefully this feature is now ready except for documentation.