searchkit-schema: expose transport to es client to enable request signing to aws managed elasticsearch

[dmccaffery]

The SearchKitConfig should include support for configuring an instance of a transport round tripper that can be injected into the es client:

https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/advanced-config.html#_transport

The round tripper would be used here:

https://github.com/searchkit/searchkit/blob/7e75a6aeb16808123ff8e4a0383f27817140b5bf/packages/searchkit-schema/src/core/SearchkitRequest.ts#L68

The primary use case we have is to support a custom transport that performs request signing required by AWS Managed ElasticSearch. Other use cases I can think of is custom metrics collection as well as other forms of custom authentication.

I am happy to implement this and submit a PR -- wanted to get the thoughts of the community on a few things:

1. Does the community want this?
2. Should the transport be a singleton instance on the SearchKitConfig or a delegate used to create an instance?
   • I'm in favour of a singleton, personally -- I can't think of reasons that the tripper would need to retain state that is ephemeral based on the request

[joemcelroy]

Actually this was mentioned before and had some notes on how this could work https://github.com/searchkit/searchkit/issues/853 and having a new option connectionOptions which passes down the connectionOptions to elasticsearch JS SDK

In fact you raise a good point maybe the better approach here is giving the developer the ability to pass in an instance of ES Client with all the configuration rather than the host so we dont need to keep adding more options to the SearchkitConfig interface.

Happy to review a PR with that approach and should be relatively easy to do.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leogcba]

Hi folks! I also need to connect to an AWS Managed ElasticSearch instance using an API key. IMO, this will be a very common scenario. I can also create a PR with the approach suggested by @dmccaffery.

Do you think it is worth creating a new issue, or can this one be reopened?

[dmccaffery]

I meant to do this but got side tracked on other things. I think the maintainers want to support the ability to pass in an entire es client instance that is pre-configured instead of augmenting the config options and creating an instance internally. Given the fork for OpenSearch, this is likely the only solution going forward as I'm sure the two will diverge over time. Hopefully the underlying API will remain useable for some time to come.

[leogcba]

Mmmm... I see your point. What i did was to allow to send ClientOptions as an alternative to the node URL... this was enough for me. Any feedback is more than appreciated.

Tonight I'll try to extend it to also allow the client, as you suggest.

[joemcelroy]

thanks! happy to review any PRs :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[moltar]

I think this is possible with the use of a custom transporter.

You can implement a class following the SearchkitTransporter interface, which pretty much just defines one method: performRequest.

Passing an instance of OpenSearch (that is already pre-configured to sign requests using AWS4) works fine.

import type { Client } from '@opensearch-project/opensearch';
import type { SearchkitTransporter, SearchkitTransporterOverrides } from '@searchkit/sdk';

class OpenSearchSignedSearchkitTransporter implements SearchkitTransporter {
  constructor (
    private readonly client: Client, // OS Client instance
    private readonly index: string // index name
  ) {}

  async performRequest(requestBody: any, overrides?: SearchkitTransporterOverrides): Promise<any> {
    const response = await this.client.search({
      method: 'GET',
      index: this.index,
      body: requestBody,
    });

    return response.body;
  }
}

I am torn on passing the original Searchkit config to OpenSearchSignedSearchkitTransporter, so that the definition of host and index is not duplicated. But then it's also an overkill, as only a single prop is used. 🤷🏼

Because, otherwise, since OS client instance is already including the host value, and Transporter already including the index value, the values passed to Searchkit are redundant:

  const sk = Searchkit({
    // not needed, as they are set in the transport
    // but can also refactor to make transport accept SearchkitConfig, which has these + the rest
    host: '',
    index: '',

---

How to pre-configure OS client:

import { defaultProvider } from '@aws-sdk/credential-provider-node';
import type { Credentials } from '@aws-sdk/types';
import { Client, Connection } from '@opensearch-project/opensearch';
import aws4 from 'aws4';

const createAwsConnector = (credentials: Credentials, region: string) => {
  class AmazonConnection extends Connection {
    buildRequestObject(params: unknown) {
      const request = super.buildRequestObject(params);

      Object.assign(request, {
        service: 'es',
        region,
        headers: {
          ...request.headers,
          host: request.hostname,
        },
      });

      // @ts-ignore
      return aws4.sign(request, credentials);
    }
  }
  return {
    Connection: AmazonConnection,
  };
};

export const getClient = async (host: string, region: string) => {
  const credentials = await defaultProvider()();
  return new Client({
    ...createAwsConnector(credentials, region),
    node: `https://${host}`,
  });
};

Wish there was a way to avoid @ts-ignore and Object.assign hacks. If anyone has better solutions, please share. This is almost as-is from AWS docs.

[moltar]

Another option is to proxy all requests via Lambda, which signs them and passes them on to OS.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[moltar]

Not stale.

GitHub stale bot considered harmful

[joemcelroy]

Im closing this ticket as documented on how to connect to Elasticsearch / Opensearch with V4 using a custom transporter https://www.searchkit.co/docs/guides/setup-elasticsearch#custom-transport