detection-rules icon indicating copy to clipboard operation
detection-rules copied to clipboard

[FR] Add `source_updated_at` to Rule Schema as a Build Time Field

Open terrancedejesus opened this issue 2 years ago • 46 comments

Is your feature request related to a problem? Please describe. No.

Describe the solution you'd like Add creation_date and updated_date to rule objects when a release package is created.

Additional context When we build a rules release package, all rule objects should have a creation_date and updated_date field in them. This will be used by Kibana for the updates review workflow.

@jpdjere @approksiu

Dev branch: https://github.com/elastic/detection-rules/tree/fr-add-dates-to-rule-data

terrancedejesus avatar Jun 01 '23 15:06 terrancedejesus

Update

We have some considerations with adding this. These fields are currently in the rule meta since they do not matter for the SHA256 hash calculation. As a result, typically anything we add to the hash calculation should be moved into the rule data itself and have validation done on the values.

Options:

  1. Add creation_date and updated_date to the API formatted rule object, after it is built and hash has been calculated.
  2. Add creation_date and updated_date to strip_additional_fields and remove them during hash calculation.
  3. Move creation_date and updated_date into rule.contents.data and remove from rule.contents.meta.

Either way we need to add validation to these field value pairs to keep the date values consistent.

@Mikaayenson @eric-forte-elastic @brokensound77 - Any additional thoughts?

terrancedejesus avatar Jul 13 '23 14:07 terrancedejesus

I do not see an issue with this approach/solution :+1:

Just as a note, we will need to update unit tests to also have creation_data and updated_date in the rule.contents.data and update the following line from packaging.py

    def _package_kibana_index_file(self, save_dir):
        """Convert and save index file with package."""
        sorted_rules = sorted(self.rules, key=lambda k: (k.contents.metadata.creation_date, os.path.basename(k.path)))

None of these should be an issue as the functions/tests have access to the contents object of a rule allowing them access to both metadata and data.

eric-forte-elastic avatar Jul 13 '23 14:07 eric-forte-elastic

@terrancedejesus Can you explain why we want to make the change to move these fields at all (or even add to the build)? Was it requested upstream?

Mikaayenson avatar Jul 16 '23 15:07 Mikaayenson

@terrancedejesus Can you explain why we want to make the change to move these fields at all (or even add to the build)? Was it requested upstream?

Requested upstream from @jpdjere for UI regarding rule update workflow.

terrancedejesus avatar Jul 16 '23 15:07 terrancedejesus

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

botelastic[bot] avatar Sep 14 '23 16:09 botelastic[bot]

@jpdjere is this still a request from your team? If so, I'd like to get it correctly scoped for one of our upcoming sprints. Thank you!

terrancedejesus avatar Sep 14 '23 16:09 terrancedejesus

Hi @terrancedejesus , yes this is still a valid request. We won't be working on anything that needs this data for 8.11 but probably 8.12

jpdjere avatar Sep 14 '23 23:09 jpdjere

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

botelastic[bot] avatar Nov 13 '23 23:11 botelastic[bot]

Hi @terrancedejesus . Do you have bandwidth for this in any upcoming releases?

jpdjere avatar Nov 15 '23 07:11 jpdjere

@jpdjere - Thanks for the follow up! I added this to our teams next sprint cycle which starts Nov 27. With recent adjustments to our current sprint cycle, I will attempt to get started with this to determine if it is relatively straight forward and if so will have it in earlier.

terrancedejesus avatar Nov 16 '23 15:11 terrancedejesus

Hello @jpdjere 👋 ,

Can you provide more context for this request? We are just trying to understand the reasoning and whether this is the best representation of this information.

Right now, since we do not push this with the rules, the dates are pulled from kibana:

  • created when downloaded
  • updated when updates applied

This is reflective of the rules as they apply to a users stack, which seems accurate and informative.

Our dev cycle creates situations where rules may not be released for a few days or weeks after modification, so there is inconsistency that may cause confusion. More so, I think it may be more valuable understanding when a rule was created and modified within a stack vs when it was developed.

These fields under our metadata are currently used as a means to inform us on changes from a maintenance perspective.

Thoughts?

brokensound77 avatar Nov 16 '23 18:11 brokensound77

Hi @brokensound77 . Thanks for the follow up.

The idea behind this request is to give the users an idea of how "recent" the updates to a specific rule are, in order to know how long those specific updates for the rule have been pending. Here's a screenshot of the UI as proposed by our designers: image By sorting by the "Last updated" column, the user could have a quick understanding of which rules have been recently updated in the Fleet package and know which rule updates have been pending for a long time, i.e. should take their most immediate attention.

This becomes especially important when the user has neglected addressing updates from one (or many) package releases, and after some considerable amount of time sees in our Rule Update table a list of rule updates corresponding to more than one releases.

For example, a user seeing this table in October could see listed 4 rules that have updates coming from a package release made in March (so their updated_by timestamps would maybe be around January, February, March), and 5 more rules that have updates coming from a package released in August (and updated_by timestamps would maybe be around May, June, July). If a rule had updates in both releases, the latest would be displayed, since we always compare to the latest version.

(Sorry if this timelines don't make sense, I don't have in my mind right now what is your release cadence).

Having said that:

Our dev cycle creates situations where rules may not be released for a few days or weeks after modification, so there is inconsistency that may cause confusion.

I think that's a valid concern that can cause confusion to users, given the false impression that updates have been pending for a long time, when the rule updates have just been released. A couple questions:

  • How usual is this discrepancy between the actual update date of a rule on your side and the release of a package?
  • Do you have an approximate idea of how large can the date range be for rule updates within one package release? What I mean is: if a package release includes 10 updates, what is the earliest and the latest updated_at timestamp that we would see?

(This second question, I think, is not that important considering that the user might accumulate updates from many subsequent releases, but it's good to know).

jpdjere avatar Nov 21 '23 12:11 jpdjere

@jpdjere - Apologies for the questions going unanswered.

How usual is this discrepancy between the actual update date of a rule on your side and the release of a package?

We release OOB updates bi-weekly. Therefore, updated dated discrepancy could be 1-14 days apart as it would depend on the pull request merge and when the package reaches EPR. This could be expanded more if the release takes longer than expected, but is a rare occurrence.

Do you have an approximate idea of how large can the date range be for rule updates within one package release? What I mean is: if a package release includes 10 updates, what is the earliest and the latest updated_at timestamp that we would see?

Any time a rule is updated, the updated date value is updated. Therefore it could be any date between when the last package was available in EPR to the date when the next package is available in EPR. Again, we release bi-weekly so there is an approximate range of ~14 different dates that could apply.

terrancedejesus avatar Dec 20 '23 18:12 terrancedejesus

Update 01/08/2023

We are moving forward with this as it is a requirement upstream for customizing prebuilt detection rules, milestone 3. Below are considerations:

  • Do we add these fields to the Kibana API formatted JSON object after schema loading and validation?
  • Do we need to standardize date formats or what is required upstream?
  • Do we move created and updated dates to BaseRuleData?
  • Do we have current date unit tests or do we need to add these?
  • If we do update the schemas, these will backport and what is the resolution for schema failures on backported branches

@jpdjere or @banderror - Can you provide insight to the following for us. Thank you in advance!

  • What is the target minor release for this and will it backport to previous versions?
  • Is there any specific date format that is required?
  • Is this only for created_date and updated_date or are the keys named differently.

terrancedejesus avatar Jan 08 '24 16:01 terrancedejesus

Option 1 - In this option, we rely on post_dict_conversion to add a new method _convert_add_date_fields(). This method takes the rule metadata and assigns it to the same keys, except inside of the obj dictionary which is already converted to Kibana API format from to_api_format(). We also add a validate_date_format() method with validates_schema marshmallow decorator. This will ensure that the date formats are ISO 8601, if we choose to follow this standard. Notes below:

  • This option removes the need to move the date field:values from rule metadata into rule contents data. Thus no massive overhaul has to be done for ALL rules.
  • The API formatted rule will not fail schema validation as these date fields are added AFTER loading through the rule schema
  • Rule version logic is still applied through these changes so dates are now taken into consideration for SHA256
  • If we export a custom rule and attempt to load it through the schema and it has creation_date and updated_date, it would fail because these fields are not recognized in BaseRuleData. Thus these fields would need to be part of the metadata in the export as they typically are. However, I noticed that there are several fields in the first array object of an exported rule that differ from the rule objects we ship.

Commit Reference: https://github.com/elastic/detection-rules/commit/3bc8df6e8db0da0eab483ab26633e391fab18219

terrancedejesus avatar Jan 08 '24 17:01 terrancedejesus

Option 2 - We only add meta to the package release files. Prebuilt rule packaging and artifact building rely mainly on to_api_format() method in rule.py. We already have an option include_metadata that is by default False. If True, it will add the RuleMeta as a dictionary to the rule object that will become the rule asset in the prebuilt rules package. Therefore we can avoid altering any data schema's, backport concerns, etc. Instead upstream on the Kibana side, they would handle accessing whatever metadata is shipped with the rule. This also allows us to avoid version bumps as well since we are not altering the rule contents, yet adding the metadata to the dropped files instead.

Commit Reference: https://github.com/elastic/detection-rules/commit/0402dc2ea99f187b3a842a08d222d7a30f4a164a

terrancedejesus avatar Jan 08 '24 21:01 terrancedejesus

Option 3 - We move the date fields from RuleMeta and include them in BaseRuleData as requirements. We then need to adjust every rule back to 8.3 branch, automatically through backporting and manually. The outcome is that date fields will now be in the rule contents of the API formatted rule and be available explicitly upstream by Kibana. Dates would then affect the rule version as well as any changes to rule contents marks the rule as dirty. This option has the biggest amount of changes and correction across branches that would need to be addressed.

terrancedejesus avatar Jan 08 '24 21:01 terrancedejesus

Option 4 - Similar to option 2, only instead of using metadata, is there any reason why we can't use the date of the release (almost like a build time field)? From the description, it doesn't seem necessary to have the exact date the rule was modified by our rule authors.

Mikaayenson avatar Jan 08 '24 22:01 Mikaayenson

Option 4 - Similar to option 2, only instead of using metadata, is there any reason why we can't use the date of the release (almost like a build time field)? From the description, it doesn't seem necessary to have the exact date the rule was modified by our rule authors.

@Mikaayenson Great alternative with a few caveats. Our source-of-truth is typically the repository since that is where we lock versions. Let's say we lock versions and a rule has changes that cause the SHA256 to change. This "state" of the rule is only noticed during the lock versions, which we also release our commits from. Technically, up until this version lock, our rule could go through several changes and updates, but it is only when we lock versions do we track the current state of the rule. The last updated_date would be inline with this SHA256 change as the exact date when the version change was noticed.

We also have to take into consideration release timing. Releases could take 1-2 days, thus the potential for a divergence of dynamic dates based on building the package could occur not only from the version lock, but also between each package. All packages would have to be released GA on the same day for them to accurately reflect the same updated date.

terrancedejesus avatar Jan 08 '24 22:01 terrancedejesus

Option 4 - Similar to option 2, only instead of using metadata, is there any reason why we can't use the date of the release (almost like a build time field)? From the description, it doesn't seem necessary to have the exact date the rule was modified by our rule authors.

I concur. If approach 4 is not a heavy lift, I would prefer this as well. While I do not think this is an immediate concern, I think there may be a case where we would not want to include all of the metadata in the release.

Option 4 - Similar to option 2, only instead of using metadata, is there any reason why we can't use the date of the release (almost like a build time field)? From the description, it doesn't seem necessary to have the exact date the rule was modified by our rule authors.

@Mikaayenson Great alternative with a few caveats. Our source-of-truth is typically the repository since that is where we lock versions. Let's say we lock versions and a rule has changes that cause the SHA256 to change. This "state" of the rule is only noticed during the lock versions, which we also release our commits from. Technically, up until this version lock, our rule could go through several changes and updates, but it is only when we lock versions do we track the current state of the rule. The last updated_date would be inline with this SHA256 change as the exact date when the version change was noticed.

We also have to take into consideration release timing. Releases could take 1-2 days, thus the potential for a divergence of dynamic dates based on building the package could occur not only from the version lock, but also between each package. All packages would have to be released GA on the same day for them to accurately reflect the same updated date.

Not sure if it would be adding too much overhead, but could we add a release tag when we lock versions and then pull the tagged SHAs of the rules and compare that way?

That being said, I also like option 2 and I do not see any immediate issues with it :+1:

eric-forte-elastic avatar Jan 08 '24 22:01 eric-forte-elastic

so have to take into consideration release timing. R

IINM the date is to provide people with a general timeline:

By sorting by the "Last updated" column, the user could have a quick understanding of which rules have been recently updated in the Fleet package and know which rule updates have been pending for a long time, i.e. should take their most immediate attention.

So im not sure if we need it to align with rule updates or locked versions etc. It sounded like they just need to know when the package was last updated.

@eric-forte-elastic brought up another good idea in a slack thread about using release tags for the date information which is another interesting idea. Recording the idea here for posterity.

Mikaayenson avatar Jan 08 '24 22:01 Mikaayenson

@jpdjere or @banderror - Can you provide insight to the following for us. Thank you in advance!

@terrancedejesus, thanks for checking this with us. @jpdjere can keep me honest, but:

  • What is the target minor release for this and will it backport to previous versions?

No concrete target minor release is defined at the moment. The Milestone 3 for customizing prebuilt rules is currently at the stage of technical design, the development hasn't started yet. I think you guys can safely expect 2 release cycles from now until we can release anything. It could be more.

When rule customization is ready for release in Kibana, we will not backport this to prior minor versions. We don't backport new features in general.

The earliest version to which we could backport these new fields in the package would be 8.12, because only in 8.12.0 we made prebuilt rule schema forward compatible with new package updates on our side (https://github.com/elastic/security-team/issues/6888).

  • Is there any specific date format that is required?

The standard one: ISO 8601 date-time string in UTC. Example:

2023-01-29T14:48:00.000Z
  • Is this only for created_date and updated_date or are the keys named differently.

Not sure I understand the question.

Naming of the keys hasn't been defined yet on our side and they are not in the schema yet. @jpdjere Could you please create a ticket for us, describe the requirements, and prioritize work on it?

banderror avatar Jan 09 '24 12:01 banderror

@banderror thank you for taking the time to provide details.

The standard one: ISO 8601 date-time string in UTC. Example:

Our dates are typically YYYY\MM\DD. We can update all rules to use YYYY-MM-DD as it shouldn't affect their version or SHA256 since it is metadata.

Not sure I understand the question.

Right now we have creation_date and updated_date as the key names in each rule metadata. Do these need to be adjusted to match your key names upstream? We would prefer they stay the same downstream in our repository to reduce mass changes across different branches when we backport.

As you may have seen by TRaDe's discussion, the simplest approach for this would be to include the metadata of the rule as a root key in the JSON rule object. Thus any metadata from the rules can be accessed and used as your team pleases. Are there any objections to this or other preferences?

creation_date - Located in metadata of rule. Describes when the rule was first merged into main of our repository. updated_date - Located in metadata of rule. Describes the date of the latest update based on changes merged into main of our repository.

availability_date - This is not captured anywhere but would track when the rule update was made available (EPR GA) via typical Elastic rule update workflow. This could be deterministic on Fleet pulls from EPR. Not sure if we want to pursue this but it has come up in previous discussions and amongst TRaDE as we determine what we are attempting to convey to customers with the rule dates.

This is an example of the would be rule object:

{
  "author": [
    "Elastic"
  ],
  "building_block_type": "default",
  "description": "Identifies the execution of a command via Microsoft Visual Studio Pre or Post build events. Adversaries may backdoor a trusted visual studio project to execute a malicious command during the project build process.",
  "from": "now-119m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "interval": "60m",
  "language": "eql",
  "license": "Elastic License v2",
  "meta": {
    "creation_date": "2023/09/26",
    "integration": [
      "endpoint"
    ],
    "maturity": "production",
    "min_stack_comments": "New fields added: required_fields, related_integrations, setup",
    "min_stack_version": "8.3.0",
    "updated_date": "2023/09/26"
  },
  "name": "Execution via MS VisualStudio Pre/Post Build Events",
  "query": "sequence with maxspan=1m\n  [process where host.os.type == \"windows\" and event.action == \"start\" and\n   process.name : \"cmd.exe\" and process.parent.name : \"MSBuild.exe\" and\n   process.args : \"?:\\\\Users\\\\*\\\\AppData\\\\Local\\\\Temp\\\\tmp*.exec.cmd\"] by process.entity_id\n  [process where host.os.type == \"windows\" and event.action == \"start\" and\n    process.name : (\n      \"cmd.exe\", \"powershell.exe\",\n      \"MSHTA.EXE\", \"CertUtil.exe\",\n      \"CertReq.exe\", \"rundll32.exe\",\n      \"regsvr32.exe\", \"MSbuild.exe\",\n      \"cscript.exe\", \"wscript.exe\",\n      \"installutil.exe\"\n    ) and\n    not \n    (\n      process.name : (\"cmd.exe\", \"powershell.exe\") and\n      process.args : (\n        \"*\\\\vcpkg\\\\scripts\\\\buildsystems\\\\msbuild\\\\applocal.ps1\",\n        \"HKLM\\\\SOFTWARE\\\\Microsoft\\\\VisualStudio\\\\SxS\\\\VS?\",\n        \"process.versions.node*\",\n        \"?:\\\\Program Files\\\\nodejs\\\\node.exe\",\n        \"HKEY_LOCAL_MACHINE\\\\SOFTWARE\\\\Microsoft\\\\MSBuild\\\\ToolsVersions\\\\*\",\n        \"*Get-ChildItem*Tipasplus.css*\",\n        \"Build\\\\GenerateResourceScripts.ps1\",\n        \"Shared\\\\Common\\\\..\\\\..\\\\BuildTools\\\\ConfigBuilder.ps1\\\"\",\n        \"?:\\\\Projets\\\\*\\\\PostBuild\\\\MediaCache.ps1\"\n      )\n    ) and\n    not process.executable : \"?:\\\\Program Files*\\\\Microsoft Visual Studio\\\\*\\\\MSBuild.exe\" and\n    not (process.name : \"cmd.exe\" and\n         process.command_line :\n                  (\"*vswhere.exe -property catalog_productSemanticVersion*\",\n                   \"*git log --pretty=format*\", \"*\\\\.nuget\\\\packages\\\\vswhere\\\\*\",\n                   \"*Common\\\\..\\\\..\\\\BuildTools\\\\*\"))\n  ] by process.parent.entity_id\n",
  "references": [
    "https://docs.microsoft.com/en-us/visualstudio/ide/reference/pre-build-event-post-build-event-command-line-dialog-box?view=vs-2022",
    "https://www.pwc.com/gx/en/issues/cybersecurity/cyber-threat-intelligence/threat-actor-of-in-tur-est.html",
    "https://blog.google/threat-analysis-group/new-campaign-targeting-security-researchers/",
    "https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES/blob/master/Execution/execution_evasion_visual_studio_prebuild_event.evtx"
  ],
  "related_integrations": [
    {
      "package": "endpoint",
      "version": "^8.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "host.os.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.args",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.command_line",
      "type": "wildcard"
    },
    {
      "ecs": true,
      "name": "process.entity_id",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.executable",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.parent.entity_id",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.parent.name",
      "type": "keyword"
    }
  ],
  "risk_score": 21,
  "rule_id": "fec7ccb7-6ed9-4f98-93ab-d6b366b063a0",
  "severity": "low",
  "tags": [
    "Domain: Endpoint",
    "OS: Windows",
    "Use Case: Threat Detection",
    "Tactic: Defense Evasion",
    "Tactic: Execution",
    "Rule Type: BBR",
    "Data Source: Elastic Defend"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0005",
        "name": "Defense Evasion",
        "reference": "https://attack.mitre.org/tactics/TA0005/"
      },
      "technique": [
        {
          "id": "T1127",
          "name": "Trusted Developer Utilities Proxy Execution",
          "reference": "https://attack.mitre.org/techniques/T1127/",
          "subtechnique": [
            {
              "id": "T1127.001",
              "name": "MSBuild",
              "reference": "https://attack.mitre.org/techniques/T1127/001/"
            }
          ]
        }
      ]
    },
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0002",
        "name": "Execution",
        "reference": "https://attack.mitre.org/tactics/TA0002/"
      },
      "technique": []
    }
  ],
  "type": "eql",
  "version": 1
}

terrancedejesus avatar Jan 09 '24 15:01 terrancedejesus

@terrancedejesus Thanks for following up on this.

As you may have seen by TRaDe's discussion, the simplest approach for this would be to include the metadata of the rule as a root key in the JSON rule object. Thus any metadata from the rules can be accessed and used as your team pleases. Are there any objections to this or other preferences?

We assessed you proposal of adding the meta property to the rule object, and -since we only need the updated_date right now, we would strongly prefer not to "pollute" the object with all the other data included in the meta property. We currently have no use for any other metadata - and probably won't have in the future. Is adding this meta property something that you need for internal tooling/processes on your side? Otherwise, we would prefer to simply add an update_date field as a top level field.

Right now we have creation_date and updated_date as the key names in each rule metadata. Do these need to be adjusted to match your key names upstream?

updated_date is OK for our use. We also have similar updated_at keys in our internal rule object schemas but that has a different semantic meaning and want to avoid collisions in key naming.

creation_date - Located in metadata of rule. Describes when the rule was first merged into main of our repository. updated_date - Located in metadata of rule. Describes the date of the latest update based on changes merged into main of our repository. availability_date - This is not captured anywhere [...]

Based on your explanation of the biweekly releases, we decided that we don't need a precise date for the update, but a "ballpark" date. If the difference between updated_date and availability_date is maximum 14 days, we have no issues in using whatever value is easier to calculate or pass down to the rule object.

Our dates are typically YYYY\MM\DD. We can update all rules to use YYYY-MM-DD as it shouldn't affect their version or SHA256 since it is metadata.

We would strongly prefer if you could format the date of the top-level update_date field in ISO 8601 date-time string in UTC, such as: 2023-01-29T14:48:00.000Z. Since Kibana parses dates based on the server's locale, we cannot guarantee that 2021\01\02 will always be parsed as January 2nd, 2021 and not February 1st, 2021 in some other locale. Is this possible on your side?

Also, I'll create a ticket on the Kibana repo for adding this task and link it here.

jpdjere avatar Jan 11 '24 15:01 jpdjere

During the simplified protections sync, Juan mentioned they only need a general availability date, ideally called elastic_last_updated (@jpdjere please let us know if I recalled this name incorrectly). This would be similar to a build time field and placed at the root of the object. The two week window if that is how often we release is general enough for them.

Mikaayenson avatar Jan 11 '24 19:01 Mikaayenson

Thanks for the follow-up here @Mikaayenson

Yes, I was originally going for update_date but I think with elastic_last_updated we can have a more clear meaning that this field only refers to updates done by the Elastic team and not by the user, and is thus only valid for Prebuilt Elastic rules. Also, it avoids collisions and confusion with other similar fields that we have in out internal rule objects.

So 👍 from my side for naming the property like that, and being it at the root level of the rule object, and with a ISO 8601 format date.

jpdjere avatar Jan 12 '24 10:01 jpdjere

@Mikaayenson @jpdjere - Thanks for the insight and update.

Still some lingering questions that are unclear to me:

  1. Are we no longer wanting a date that represents when the rule was created?
  2. If we are targeting an "availability" date, is this logic not possible by Kibana when the rule update is first identified when Fleet pulls the package from EPR? The "availability" date of a rule update or new rule, if not represented by the metadata, is only determined based on when package is released.
  3. If we go down the route of "availability" date, then this date will be redundant across any rule that is new or updated. Is this what we are attempting to achieve.

terrancedejesus avatar Jan 12 '24 16:01 terrancedejesus

@terrancedejesus

Are we no longer wanting a date that represents when the rule was created?

No. As long as the update date matches the creation date when the rule is created, we don't need a separate field for the creation date, as we never need to show the creation date specifically to the user. What I mean is:

// Rule is created
creation_date: 2024-05-21
update_date: 2024-05-21

// Rule is updated the first time
creation_date: 2024-05-21
update_date: 2024-08-12

// Rule is updated the second time
creation_date: 2024-05-21
update_date: 2024-12-26

... and so on

If that's the behaviour of the update_date, then we only need that update_date.

If we are targeting an "availability" date, is this logic not possible by Kibana when the rule update is first identified when Fleet pulls the package from EPR? The "availability" date of a rule update or new rule, if not represented by the metadata, is only determined based on when package is released.

Prebuilt rule assets are installed to the Kibana kibana_security_solution index by Fleet's API; we don't have any additional logic for this installation that we could modify to track when a new rule is first identified. The prebuilt rule assets documents that are indexed into that index by Fleet do have a updated_at and created_at field, but both of them are always the date in which the latest installation or update happened, for all versions of a rule. This is why we need the elastic_update_date to be part of the rule's data.

If we go down the route of "availability" date, then this date will be redundant across any rule that is new or updated. Is this what we are attempting to achieve.

Not sure I completely understand this point, but I think my previous answer addresses this, we still need that update_date or availability_data as parte of the prebuilt rule asset data.

jpdjere avatar Jan 16 '24 10:01 jpdjere

If we go down the route of "availability" date, then this date will be redundant across any rule that is new or updated. Is this what we are attempting to achieve.

Not sure I completely understand this point, but I think my previous answer addresses this, we still need that update_date or availability_data as parte of the prebuilt rule asset data.

The main point is that every rule object updated will have the same elastic_update_date since we will be using the date the package was built. If the package has 10 rules updated, all ten rules will have the same date.

It's not necessarily a problem, just a note of redundant-looking information across all updated rules. Now if we have another package later with another different 10 rules updates, of course the date will be different from the first 10.

Mikaayenson avatar Jan 16 '24 20:01 Mikaayenson

Thanks for the explanation.

Now if we have another package later with another different 10 rules updates, of course the date will be different from the first 10.

This is good enough for us. We don't strictly need different dates within one package release - within that 14 day window. As long as we can distinguish between updates coming from different packages releases we are fine.

jpdjere avatar Jan 17 '24 12:01 jpdjere