StackStorm
keyvalue deleted by ttl but not triggered
SUMMARY
When a keyvalue is deleted by the ttl, no core.st2.key_value_pair.delete is generated
STACKSTORM VERSION
st2 version: st2 3.7.0, on Python 3.6.8
OS, environment, install method
Post what OS you are running this on, along with any other relevant information/
OS: CentOS Linux 3.10.0-1160.76.1.el7.x86_64
Installation: one-line install
Steps to reproduce the problem
(cluster)-[root@auto ~]$ st2 key set test_ttl toto -l 30
+------------------+-----------------------------+
| Property | Value |
+------------------+-----------------------------+
| name | test_ttl |
| value | toto |
| scope | st2kv.system |
| expire_timestamp | 2022-11-10T12:34:34.529983Z |
+------------------+-----------------------------+
(cluster)-[root@auto ~]$ st2 key list
+---------------------------+--------------------------------+--------+-----------+--------------+------+-------------------------------+
| name | value | secret | encrypted | scope | user | expire_timestamp |
+---------------------------+--------------------------------+--------+-----------+--------------+------+-------------------------------+
| test_ttl | toto | False | False | st2kv.system | | Thu, 10 Nov 2022 13:34:34 CET |
+---------------------------+--------------------------------+--------+-----------+--------------+------+-------------------------------+
(cluster)-[root@auto ~]$ st2 key get test_ttl
Key Value Pair "test_ttl" is not found.
ERROR: Resource test_ttl not found.
(cluster)-[root@auto ~]$st2 trigger-instance list
+--------------------------+--------------------------------+-------------------------------+-----------+
| id | trigger | occurrence_time | status |
+--------------------------+--------------------------------+-------------------------------+-----------+
| 636cefbc87c48a6504896824 | core.st2.key_value_pair.create | Thu, 10 Nov 2022 13:34:04 CET | processed |
+--------------------------+--------------------------------+-------------------------------+-----------+
Expected Results
Once the ttl expire the keyvalue is deleted by the system, it should create a trigger core.st2.key_value_pair.delete.
Actual Results
There is no trigger core.st2.key_value_pair.delete genereted but if you delete a key by hand, a trigger is generated
If someone is willing to contribute a fix, - this looks like a good first issue
I search in /opt/stackstorm/st2 tree, I find where the trigger are created but did not find how the instance called/generated. Do you have an idea? I will try to fix it :)
I see now, the TTL expiration in K/V is handled by the MongoDB itself. That explains why K/V delete by TTL doesn't trigger an event.
https://www.mongodb.com/docs/manual/tutorial/expire-data/#expire-documents-at-a-specific-clock-time
The MongoDB index is created with the expiration type: https://github.com/StackStorm/st2/blob/007beed8325b397d540d97905af8f2edb14b925f/st2common/st2common/models/db/keyvalue.py#L46
which is then populated via ttl:
https://github.com/StackStorm/st2/blob/007beed8325b397d540d97905af8f2edb14b925f/st2common/st2common/models/api/keyvalue.py#L65-L71
https://github.com/StackStorm/st2/blob/007beed8325b397d540d97905af8f2edb14b925f/st2common/st2common/models/api/keyvalue.py#L154-L157
The right way to make the K/V TTL work would be to move the expiration from the MongoDB side to the StackStorm service side. Not an easy task anymore.
This sounds like a limitation of the current implementation.
Hi @arm4b @cognifloyd
Do you have any information on this problem? It would be really interesting to have this option to fully automate certain workflows.
I don't have time to do much with new features (like this request) or even bug fixes (I've got a list of things I need to eventually fix for work). Instead, I'm focused on a lot of infrastructure for the project, trying to make it easier for new contributors to get into our codebase.
If you want to dive in and try to add this feature, you are welcome to. StackStorm is a community-driven project and new contributors are welcome!
