Anyone tried to run Elastalert on AWS Lambda?

[jlory]

I was curious if anyone tried to run it on Lambda? Could be really nice to have Elasalert checking for ES AWS service.

[tomdavidson]

@jlory have you tried? With Elastalert's ability to start where it left of it should be able to work. Invoke the lambda with CloudWatch at the same frequency of your alert rule.

[jlory]

I might try that soon, let you know about that.

[amosshapira]

I'm also curios - reached this question looking for exactly the same thing.

[cgutshal]

@jlory Any lunk, so far I haven't been able to get it working.

[jlory]

@cgutshal Unfortunately no, I was pulled to do something else :) I still want to try that some day.

[cgutshal]

10/4 I'm pivoting to attempt to dockerize it 🐳 and run it within EBS for now. Thanks for the update.

[ms999]

I have an idea to try to run in in Lambda. I'll let you know guys

[ninapavlich]

@ms999 How did it go running Elastalert in Lamda? Any successes or roadblocks you ran into?

[beezz]

@jlory / @ninapavlich / @cgutshal I know it's old issue but maybe someone will find it helpfull. We are running elastalert in lambda using this piece of code https://github.com/beezz/elastalert-lambda

Would love to get feedback. Thanks!

[tomfotherby]

We run ElastAlert in aws lambda and are happy with it in general.

Having said that, we are having problems with duplicate alerts and I'm trying to investigate whether the realert directive is using internal memory (rather than elasticsearch) to store state. In a serverless environment like lambda, that would explain why realert doesn't seem to work for us.

realert: This option allows you to ignore repeating alerts for a period of time. (docs)

[Qmando]

@tomfotherby It actually does both. It stores it locally in memory and uses that as a priority but then if there's no entry for that alert, it searches elasticsearch. https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L1768

If you don't mind, which version of Elasticsearch are you using? I can try to see if there's a bug in that code because in most cases it's never used, except for when elastalert restarts or I guess when it's in lambda.

[tomfotherby]

@Qmando - Thanks for the info, that's very helpful. We use the AWS ElasticSearch service v6.7 and the latest ElastAlert code.

I am getting ElastAlerts every minute when the lambda runs with this rule that includes a realert directive:

name: "ElastAlert: PHP Fatal Errors [Production]"
type: any
index: logstash-*
filter:
- query:
    query_string:
      query: "message:\"PHP Fatal error\" AND stream:\"stderr\""
realert:
  minutes: 30

alert_text: "Please investigate these errors. The ElasticSearch query to trigger these alerts is: message:\"PHP Fatal error\" AND stream:\"stderr\" ."

alert:
- "sns"
sns_topic_arn: "arn:aws:sns:us-east-1:redacted:ElastAlert"

[briandefiant]

We're using the following method (extending the ElastAlerter class and overriding the start() method) to run Elastalert on Lambda: https://github.com/jertel/elastalert2/discussions/1169