Azure-Sentinel icon indicating copy to clipboard operation
Azure-Sentinel copied to clipboard

Transformkql samples request for standard tables

Open Ysuuuuuuuu opened this issue 1 year ago • 4 comments

My customer is planning to ingest the basic windows security event log via logstash into SecurityEvent table.

Per the logstash sample logs, it is found that many attributes are grouped together in a JSON format under one field. For example, one field named wineventlog is like: { "computer_name": "xxxxxx", "task": "Logon", "opcode": "Info", "provider_name": "Microsoft-Windows-Security-Auditing", "process": { "pid": 700, "thread": { "id": 740 } }, "channel": "Security", "event_data": xxxxxx, ......}

In order to ingest into the standard SecurityEvent table, a transformkql is needed here to map the fields in the standard table, which is not easy. So customer is asking whether MS could provide some transformkql samples in the documentations for their reference?

Thanks in advance.

Ysuuuuuuuu avatar Mar 20 '24 14:03 Ysuuuuuuuu

Hi @Ysuuuuuuuu ,Could you please share more details about the issue,What is solution/Connector and in that if parser share parser details with detailed screen shots

v-muuppugund avatar Mar 21 '24 03:03 v-muuppugund

Hello @v-muuppugund , the solution is similar to https://learn.microsoft.com/en-us/azure/sentinel/connect-logstash-data-connection-rules#create-dcr-resources-for-ingestion-into-a-standard-table.

Part of the logs that will be ingested via logstash are like: image

This schema is totally different from the schema of standard table SecurityEvent. Then a proper transformkql is needed in DCR to map the fields in the ingested logs to the fields of SecurityEvent.

Ysuuuuuuuu avatar Mar 21 '24 03:03 Ysuuuuuuuu

Hi @Ysuuuuuuuu , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 27Mar24. Thanks!

v-muuppugund avatar Mar 21 '24 11:03 v-muuppugund

Thanks in advance.

Ysuuuuuuuu avatar Mar 21 '24 11:03 Ysuuuuuuuu

Hello @v-muuppugund , may I get some updates for this ask? Thanks.

Ysuuuuuuuu avatar Mar 28 '24 01:03 Ysuuuuuuuu

Hello @v-muuppugund , may I get some updates for this ask? Thanks.

Hi @Ysuuuuuuuu ,Working on further analysis on this issue with Query,will update you

v-muuppugund avatar Mar 28 '24 02:03 v-muuppugund

Thanks. Look forward to your further updates.

Ysuuuuuuuu avatar Mar 28 '24 03:03 Ysuuuuuuuu

Hi @Ysuuuuuuuu , Could you please provide couple of time slots for teams meeting on the issue to [email protected] for further troubleshooting.

v-muuppugund avatar Apr 03 '24 01:04 v-muuppugund

Hi @Ysuuuuuuuu ,As discussed over call today got the data,will update you

v-muuppugund avatar Apr 03 '24 03:04 v-muuppugund

Hi @Ysuuuuuuuu ,We are working on it ,will update you ,as we are facing some permissions issue during standard conversion,will let you know if needed we can have a call with customer,will update you

v-muuppugund avatar Apr 10 '24 02:04 v-muuppugund

Hi @Ysuuuuuuuu ,As discussed over teams call today,as per your confirmation as there are no documentation available,we are closing your issue (https://github.com/Azure/Azure-Sentinel/issues/10185) as per our standard operating procedures. If you still need support for this issue, feel free to re-open at any time. Thank you for your co-operation

v-muuppugund avatar Apr 12 '24 06:04 v-muuppugund