brimdata
Additional external search options for IPs/Domains/Ports (e.g. VirusTotal)
In my use case I use a good bit of non-zeek ndjson data (such as flow and reverse/forward dns database's) as such I lack additional context to certain information surrounding the nature of IPs and Domains in that data. Some of that I would perhaps get with full network packet capture analyzed with Zeek, at least as it related to that particular session and I do mostly leverage Brim for the purpose of processing large PCAPs into analyzed Zeek logs. For all of these data sources it would be nice to query a service like Shodan, Binary Edge, or Censys to determine additional internet facing ports and services other than what's captured in a session/flow. Or perhaps query Urlscan.io to pivot on urls/domains to glean additional information. Another great resource would be the Internet Storm Center where they track a lot of daily variations in network activity and have port specific information. All of these ideally would function the same as the pivot to VirusTotal currently, whereas they just open a web page to that respective query. There are more than just these three but I would assume the process would be user specific. Another fantastic option would be user control of what to add if there was some flexibility to doing so, like a paid subscription to a threat intel platform or a paid API.
Links:
Shodan https://developer.shodan.io/api/clients
Urlscan.io https://urlscan.io/about-api/
Internet Storm Center (ISC)/DShield API - SANS https://isc.sans.edu/api/
I'd love to see a plugin architecture like this exposed so we could create/add private lookup tools. (If it's not doable already! If so, got some docs?!)
@cyamal1b4: Thanks for the pointers and use case information. You've correctly deduced that the approach we started with the right-click callout to VirusTotal is something we've often talked about wanting to expand to other tools, and community-driven tools like the ones you've referenced are certainly in the areas we intend to research. The potential of adding ways to leverage paid platforms/APIs has also been discussed. The core Brim development team is busy with other priorities at the moment, but we'll definitely hold onto this issue in case other community members see it and chime in with more suggestions, or maybe someone will even try writing some code. When the item bubbles to the top of our to-do list, we can circle back for a little more Q&A, maybe bounce some early designs off people like yourself if you're up for it, etc.
(To that end, anyone who's watching this issue and is interested in this topic, if you're already on the Brim public Slack or happen to join it, please point yourself out on #brim or to @phil so I can circle back to you in future discussions. I expect some topics might be easier to discuss in that forum and then summarized back here in GitHub, since it's easier to yank other people into the thread, attract helpful eavesdroppers, etc.)
@geekscrapy: There's not yet a generic plugin approach I can point you at, but I agree that (much like @cyamal1b4 implied) in addition to curated free/community options that might be in there out of the box, having a generic approach (e.g. right-click callout to a user's arbitrary script that calls any custom/paid logic or service) would make a lot of sense. We'll definitely keep that in mind when we design for this topic.
We've had another inquiry in this area on Slack, so I'm capturing it here for consideration when we get to designing this.
The original inquiry:
I filtered on dns queries by doing _path=dns | count () by query | sort -r count , I want to exclude the top visited, export to csv and query virus total is there a way to do that or do bulk VT queries from brim natively?
In a thread, I explained that one of the reasons we've not pursued this yet is because of the limitations of VirusTotal's public API that would be the likely way to build this integration. The areas of specific concern:
The Public API is limited to 500 requests per day and a rate of 4 requests per minute. The Public API must not be used in commercial products or services. The Public API must not be used in business workflows that do not contribute new files.
Since the Brim/zq projects are open source, the second point seems to not apply, and maybe not the third either since a lot of our community users are not in "business workflows", though some are. The first one has always been the main sticking point, since it feels like we could spend a bunch of time building a nice integration and find that users are constantly hitting that wall. OTOH if a lot of our users are already paying customers of VT and wouldn't be subject to these limitations, then maybe it'd be worth it.
In response to that last point, the user explained:
The client I support does have a premium API sub so I was interested in tapping in to that. I think it would be good to build a bulk VT query through use of the public API and build it to the documented limitations like ok if you want to query VT for different things based off system time and a queryCounter variable you can query up to 500 items and the query counter either gets decremented from 500 or goes up to 500 and caps it that way, if that makes sense
Indeed, this all seems to make sense. To be good citizens that comply with VT's third point, it seems like we'd probably also want to include some ways in the app to make it easy to make contributions.
We've had the opportunity to revisit this topic recently when a community user opened up #2475 to inquire about adding abuse.ch lookups similar to what's already there for VirusTotal. In discussing the implementation with @jameskerr, we both imagined shipping with these as a couple examples that use a "templated URL" to hit some kind of external API while plugging in the value from a right-clicked field, retrieve the response, and present it in the app. By following these examples, users could see how to add additional callouts of their own. For more sophisticated callouts that need something beyond a simple URL (e.g., complex auth, or responses that need to be formatted before presenting to the user) we discussed one day allowing users to plug in whole chunks of JavaScript to handle these callouts.