[JS] Adding vertexAI Plugin increases memory consumption on Firebase Cloud Function Trigger

[dario-digregorio]

Describe the bug When I add the vertexAI plugin to genkit I am unable to deploy my cloud function triggers (Firestore document trigger). I get the error Memory limit of 256 MiB exceeded with 280 MiB used. Only increasing the memory or removing the vertexAI plugin fixes the problem. So I am wondering way the memory increases so high.

To Reproduce add this configuration:

configureGenkit({
  plugins: [
    firebase(),
    vertexAI(),
    googleAI(),
    dotprompt(),
  ],
  traceStore: "firebase",
  logLevel: "debug",
  defaultModel: gemini15Pro,
  enableTracingAndMetrics: true,
});

add this cloud function trigger:

export const generateChatAnswer = onDocumentCreated({
  document: "rooms/{roomId}/events/{eventId}",
  memory: "512MiB",
},
async (e) => {
  logger.info("Event triggered");
});

run a deploy:

firebase deploy --only functions

Expected behavior I should be able deploy the functions because I just use a different model API

Screenshots If applicable, add screenshots to help explain your problem.

Runtime (please complete the following information):

• OS: [e.g. Linux, MacOS]
• Version [e.g. 22]

** Node version

• v20.15.1 package.json dependencies

  "dependencies": {
    "@genkit-ai/ai": "^0.5.8",
    "@genkit-ai/core": "^0.5.8",
    "@genkit-ai/dotprompt": "^0.5.8",
    "@genkit-ai/firebase": "^0.5.8",
    "@genkit-ai/flow": "^0.5.8",
    "@genkit-ai/googleai": "^0.5.8",
    "@genkit-ai/vertexai": "^0.5.8",
    "@google-cloud/functions-framework": "^3.4.2",
    "express": "^4.19.2",
    "firebase-admin": "^12.2.0",
    "firebase-functions": "^5.0.1",
    "zod": "^3.23.8"
  },
  "devDependencies": {
    "@typescript-eslint/eslint-plugin": "^5.12.0",
    "@typescript-eslint/parser": "^5.12.0",
    "eslint": "^8.9.0",
    "eslint-config-google": "^0.14.0",
    "eslint-plugin-import": "^2.25.4",
    "firebase-functions-test": "^3.1.0",
    "typescript": "^4.9.5"
  },

[marcelpinto]

I was struggling with the same issue today. Not only this, but it affects all the other functions, they all started to fail because of memory limit.

For now, the solution was to configureGenKit inside the trigger function and increase that function memory.

[pavelgj]

Vertexai plugin has grown in size recently, so it's not too surprising that it's memory hungry. That said, I have some ideas how to improve this -- we can break up the vertexAI plugin into multiple plugins. Specifically the features with non-core deps (like anthropic and llama3.1 model support).

[cabljac]

I guess this needs some planning on what exactly we need to do, before i start? @pavelgj

[pavelgj]

Here's my thinking -- we split up the plugin into multiple sub-plugins with separate imports:

1. VertexAI prediction API (Gemini, iamgen and embedders, etc...)

import { vertexAI } from '@genkit-ai/vertexai'

2. Model Garden

import { vertexAIModelGarden } from '@genkit-ai/vertexai/modelgarden'

3. Evaluation

import { vertexAIEvaluation } from '@genkit-ai/vertexai/evaluation'

4. Vector Search

import { vertexAIVectorSearch } from '@genkit-ai/vertexai/vectorsearch'

5. Reranker

import { vertexAIReranker } from '@genkit-ai/vertexai/reranker'