SURF

Free Text Search across your infrastructure platforms via CLI.

It's like ... | grep but for your entire infrastructure!

S.U.R.F is an acronym for: Search-Unified-Recursive-Fast

Supported Platforms

• [X] AWS Route53 and Targets
• [X] AWS ACM
• [X] AWS S3
• [X] AWS DynamoDB
• [x] Hashicorp Vault
• [X] Hashicorp Consul KV
• [X] ElasticSearch / AWS OpenSearch
• [X] Logz.io
• [ ] Kubernetes - TODO

Table of Contents

• Overview
• Examples
  • AWS Route53 Usage
  • AWS ACM Usage
  • AWS DynamoDB Usage
  • AWS S3 Usage
  • Hashicorp Vault Usage
  • Hashicorp Consul Usage
  • ElasticSearch / OpenSearch Usage
  • Logz.io Usage
• Install
  • Brew
  • Download Binary
  • Install from Source
• Authentication
  • Supported Authentication Methods
• Version check
• How it Works

Overview

SURF is built for Infrastructure Engineers as a CLI tool that enables searching any pattern across different platforms. Usually, the results are returned with a direct web URL.

The search process depends on the context, for example: if you're searching in Vault it'll pattern match against keys. Instead, if you're searching in Route53 AWS a DNS address it'll return links to the targets behind it (e.g Load balancer).

Usage Examples

AWS Route53 Usage

Based on AWS Route53: Search what's behind domain api.my-corp.com:

surf r53 -q api.my-corp.com

AWS ACM Usage

Search inside ACM Certificates in AWS.

Example search: containing a domain:

surf acm -q my-domain.com

Example search: certificate attached to a loab balancer:

surf acm -q 's:elasticloadbalancing:us-west-2:123:loadbalancer/app/alb' --filter-used-by

AWS DynamoDB Usage

Search free text data in DynamoDB

Supported Formats: protobuf, base64, json, binary, bytes.

Example: list existing tables

surf ddb --list-tables

Example: use -p for aws profile, -r for region

surf ddb -q val -t table -p my-aws-profile -r us-east-1

Example: search all tables with production in their name, where the data containing the pattern val

surf ddb -q val --all-tables -t production

Example: search all tables data containing the word val, output as JSON

surf ddb -q val --all-tables -o json

Example: stop on first match, search all tables data containing the word val

surf ddb -q val -t my-prefix-table --stop-first-match

AWS S3 Usage

Search inside S3 Buckets and Keys in AWS.

Example: Find all keys containing logs in all buckets containing the name prod-bucket:

surf s3  -q 'logs' -b 'prod-bucket'

Example: find all keys containng house with prefix key my

surf s3 --prefix my -q house -b my-bucket

Example: find all keys ending with .json and use non-default AWS profile:

surf s3 -q '\.json$' -b bucket-prefix -p my-aws-profile 

Optional: Configure a default bucket name (same as --bucket flag) to start search from (any regex pattern):

export SURF_S3_DEFAULT_MOUNT=<my-bucket-pattern>

Hashicorp Vault Usage

Search the query aws in Vault:

surf vault -q aws 

Configure a default mount to start search from in Vault:

export SURF_VAULT_DEFAULT_MOUNT=<my-default-mount>

Store LDAP auth on your OS keychain:

surf config

Hashicorp Consul Usage

Search all keys containing the substring server

surf consul -q "server"

Search under the scripts path for keys ending with .sh

surf consul --prefix scripts --query "\.sh$"

ElasticSearch and OpenSearch Usage

Search free text and/or KQL.

Supports Standard Elasticsearch and AWS Opensearch.

• Example: Search docs containing the term api return limit 40 results.

  surf es -q 'api' -l 40
  

• Example: Search in indexes prod-* and api-*

  surf es -q 'api' -i 'prod-*d' -i 'api-*'
  

• Example: Search docs containing the term api with client field and xyz* pattern and NOT containing the term staging

  surf es -q 'api AND client:xyz*' --nq staging
  

• Example: List Indexes

  surf es --list-indexes
  

• Example: Configure Elasticsearch default settings (use surf es --help for more):

  SURF_ELASTICSEARCH_URL
  SURF_ELASTICSEARCH_USERNAME='token-or-username-and-password'
  SURF_ELASTICSEARCH_PASSWORD='token-or-username-and-password'
  SURF_ELASTICSEARCH_TOKEN='token-or-username-and-password'
  SURF_ELASTICSEARCH_INDEXES='prod-*,api-*'
  

• Example: Store Username/Password or Token auth on your OS keychain:

  surf config
  

Logz.io Usage

Free text and KQL Search in logz.io, use the api website to get the address.

• Example: Search docs containing the word 'exception' with limit size 200

  surf logz -q 'exception' -l 200
  

• Example: Search docs containing the word 'something' across ALL sub-accounts matching production/automation

  surf logz -q 'something' --acc production --acc automation 
  

• Example: Search docs containing the word 'production', field errorCode with value Access* and are not containing 'dummy'

  surf logz -q 'production AND errorCode:Access*' --nq 'dummy'
  

• Example: Search docs across 10 day window with 2 days offset (e.g all matches between 12 days ago until 2 days ago)

  surf logz -o 2 -w 10d -q 'some pattern'
  

• Example: Configure Logz.io default settings (use surf logz --help for more):

  SURF_LOGZ_IO_TOKEN
  SURF_LOGZ_IO_URL
  SURF_LOGZ_IO_ACCOUNT_IDS
  

• Example: Store Username/Password or Token auth on your OS keychain:

  surf config
  

Install

Brew

MacOS (and ubuntu supported) installation via Brew:

brew tap isan-rivkin/toolbox
brew install surf

Download Binary

1. from releases

2. Move the binary to global dir and change name to surf:

cd <downloaded zip dir>
mv surf /usr/local/bin

Install from Source

git clone [email protected]:Isan-Rivkin/surf.git
cd surf
go run main.go

Authentication

Please open a PR and request additional methods if you need.

Supported Authentication Methods

• [x] Vault - LDAP (run $surf config )
• [x] AWS - via profile on ~/.aws/credentials file
• [x] Consul - None
• [X] Elasticsearch / Opensearch - User/Pass or Token (run $surf config or surf es --help)
• [X] Logz.io - Token (run $surf config or surf logz --help)

Version check

The CLI will query github.com to check if there is a newer version and print out a message to the terminal.

If you wish to opt out set the environment variable SURF_VERSION_CHECK=false.

No Data is collected it is purely github.com query.

How it Works

Contributors

• @AliRamberg
• @Isan-Rivkin