Influxdb query agent

[dansola]

Why are the changes needed?

Adding a sync agent to query from influxdb; a timeseries database.

What changes were proposed in this pull request?

Added an influxdb task and agent to pull timeseries data as a pandas dataframe.

How was this patch tested?

Setup process

from datetime import datetime, timedelta

from flytekit import workflow
from flytekitplugins.influxdb import InfluxDBTask, Aggregation, influx_json_to_df

url = "http://localhost:8086"
org = "dan"
influx_query_task = InfluxDBTask(name="influx_task", url=url, org=org)


@workflow
def influx_wf() -> str:
    bucket = "testing-bucket"
    measurement_name = "testing_measurement"
    start_time = datetime(2024, 4, 25, 0, 0, 0)
    end_time = start_time + timedelta(days=2)
    tag_filter_dict = {"tag_key": ["tag_value_1"]}
    fields = ["data_1"]
    aggregate_period_min = 30
    aggregate_function = Aggregation.LAST

    data = influx_query_task(
        bucket=bucket,
        measurement=measurement_name,
        start_time=start_time,
        end_time=end_time,
        tag_dict=tag_filter_dict,
        fields=fields,
        aggregation=aggregate_function,
        period_min=aggregate_period_min
    )

    return data

if __name__ == "__main__":
    data = influx_wf()
    df = influx_json_to_df(data)
    print(df.head())

Screenshots

Local

python influx_wf.py

Check all the applicable boxes

• [ ] I updated the documentation accordingly.
• [ ] All new and existing tests passed.
• [x] All commits are signed-off.

Related PRs

Docs link

[Future-Outlier]

If possible, can you also provide screenshots when running this agent locally and remotely?

Here's an example, thank you.

https://github.com/flyteorg/flytekit/pull/2226

[dansola]

If possible, can you also provide screenshots when running this agent locally and remotely?

Here's an example, thank you.

#2226

Thank you very much for the review! I got a local example going but I had a bit of trouble setting up a local cluster and getting the agent working there. I will put some more time into this and let you know once I have a good example working in a remote environment.

[Future-Outlier]

If possible, can you also provide screenshots when running this agent locally and remotely? Here's an example, thank you. #2226

Thank you very much for the review! I got a local example going but I had a bit of trouble setting up a local cluster and getting the agent working there. I will put some more time into this and let you know once I have a good example working in a remote environment.

No problem, thank you. These 2 docs might help you.

https://docs.flyte.org/en/latest/flyte_agents/testing_agents_in_a_local_development_cluster.html https://docs.flyte.org/en/latest/community/contribute.html#how-to-setup-dev-environment-for-flyteidl-flyteadmin-flyteplugins-flytepropeller-datacatalog-and-flytestdlib

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.60%. Comparing base (4dd4d22) to head (d709481). Report is 3 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2396       +/-   ##
===========================================
+ Coverage   75.81%   88.60%   +12.78%     
===========================================
  Files         181       69      -112     
  Lines       18327     3536    -14791     
  Branches     2580        0     -2580     
===========================================
- Hits        13895     3133    -10762     
+ Misses       3830      403     -3427     
+ Partials      602        0      -602     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.