feature: support for batch creating task through http_sd

[Calvin979]

What's changed?

Steps to batch create task through http_sd

1. Add http_sd params in template yml

  - field: http_sd
    name:
      zh-CN: http_sd
      en-US: http_sd
    type: text
    placeholder: Enter http_sd url
    required: false
    hide: true

2. Enter http_sd url when trying to create task(optional)

Checklist

• [x] I have read the Contributing Guide
• [ ] I have written the necessary doc or comment.
• [ ] I have added the necessary unit tests and all cases have passed.

Add or update API

• [ ] I have added the necessary e2e tests and all cases have passed.

[tomsun28]

👍 relate pr https://github.com/apache/hertzbeat/pull/1791

[Calvin979]

Relate issue #1625

[tomsun28]

hi thanks for contribution. there are some conflict files need be resolved if you are time.

[Calvin979]

hi thanks for contribution. there are some conflict files need be resolved if you are time.

@tomsun28 I've resolved conflict, please wait me to check the code and feature if is all set before merging.

[Calvin979]

@tomsun28 This branch can be merged now!

[tomsun28]

@tomsun28 This branch can be merged now!

thanks, we want to merge this after publish v1.6.0

[Calvin979]

@tomsun28 This branch can be merged now!

thanks, we want to merge this after publish v1.6.0

Got it.

[tomsun28]

hi, how about the design when update the monitoring.

[Calvin979]

hi, how about the design when update the monitoring.

Do you mean updating http_sd monitor or those monitors which are created by http_sd?

[tomsun28]

Do you mean updating http_sd monitor or those monitors which are created by http_sd?

hi, yes. Due the monitors automatic discovery by http_sd response data, when http_sd response data change, should we consider updating the automatically created monitors?

[tomsun28]

Suggest we consider these designs in the first version to facilitate future expansion, but do not have to implement it right away.

[Calvin979]

Do you mean updating http_sd monitor or those monitors which are created by http_sd?

hi, yes. Due the monitors automatic discovery by http_sd response data, when http_sd response data change, should we consider updating the automatically created monitors?

It should be discussed first. I think these steps are the key of updating monitors automatically:

1. The monitor which is configed http_sd(Maybe we can refer to it as the main monitor below) must have a relation in DB with monitors which are created by the main monitor. So that we can handle each monitor with the main monitor.
2. Create a cyclic task for the main monitor, which is used to collect service discovery data.
3. Isolate data in CommonDataQueue due to service discovery data is completely different from other data.
4. Manager poll service discovery data and update monitors.

So far this idea came into my mind. How about this design in general?

[tomsun28]

hi, look great!!

1. The monitor which is configed http_sd(Maybe we can refer to it as the main monitor below) must have a relation in DB with monitors which are created by the main monitor. So that we can handle each monitor with the main monitor.

It is recommended to create a new table to associate their relationships.
Not sure if we need to add an 'auto' field to 'hzb_monitor' to indicate if this monitor was created by auto discover or not, if it's not non-essential, we may choose not add it now.

The design makes full use of existing capabilities while minimizing modifications.👍

[Calvin979]

hi, look great!!

1. The monitor which is configed http_sd(Maybe we can refer to it as the main monitor below) must have a relation in DB with monitors which are created by the main monitor. So that we can handle each monitor with the main monitor.

It is recommended to create a new table to associate their relationships. Not sure if we need to add an 'auto' field to 'hzb_monitor' to indicate if this monitor was created by auto discover or not, if it's not non-essential, we may choose not add it now.

The design makes full use of existing capabilities while minimizing modifications.👍

We can add a tag named Auto created by http_sd to each monitor. I will try to implement this design this weekend.

[Calvin979]

@tomsun28 http_sd can detect target regularly and create monitor automatically now.

[Calvin979]

Two steps to support more sd if we need it:

1. Create a sd collect like any other collect
2. Add an enum element in org.apache.hertzbeat.common.entity.sd.ServiceDiscoveryProtocol

[Calvin979]

The entire process of sd: @tomsun28

[Calvin979]

hi, look great!!

1. The monitor which is configed http_sd(Maybe we can refer to it as the main monitor below) must have a relation in DB with monitors which are created by the main monitor. So that we can handle each monitor with the main monitor.

It is recommended to create a new table to associate their relationships. Not sure if we need to add an 'auto' field to 'hzb_monitor' to indicate if this monitor was created by auto discover or not, if it's not non-essential, we may choose not add it now.

The design makes full use of existing capabilities while minimizing modifications.👍

I create a table named hzb_monitor_bind to associate relationships. See also: org.apache.hertzbeat.common.entity.manager.MonitorBind

[Calvin979]

It's hard to merge after all these pr, I think I have to create another pr and this pr will be closed tomorrow. @tomsun28

[tomsun28]

It's hard to merge after all these pr, I think I have to create another pr and this pr will be closed tomorrow. @tomsun28

ok, we refactored some modules which caused so many conflicts.