continuedev
Embedding provider other than transformer.js creates invalid index
Before submitting your bug report
- [x] I believe this is a bug. I'll try to join the Continue Discord for questions
- [x] I'm not able to find an open issue that reports the same bug
- [x] I've seen the troubleshooting guide on the Continue Docs
Relevant environment info
- OS: Windows 11 Pro (Build 22631.3155, Windows Feature Experience Pack 1000.22684.1000.0)
- Continue: v0.8.2 - 2024-01-23 (Current stable version from VS Extensions)
- IDE: VS Code 1.87.0
- Model: Ollama any
Description
When im using a different embeddingsProvider other than the standard transformer.js, I get various bugs and an unexplained behavior. This is my setup:
Starting VS Code as a fresh install, adding the continue extension. Navigating into the config.js in users/USER/.continue/config.js and adding a custom embedding provider:
"embeddingsProvider": {
"model": "codellama:7b",
"apiBase": "http://localhost:11434",
"provider": "ollama"
}
Adding another Ollama "model" to be used for chat:
"models": [
{
"model": "deepseek-coder:33b-instruct-q4_K_M",
"title": "Ollama",
"apiBase": "http://localhost:11434",
"provider": "ollama"
}
],
Closing VS Code, deleting the .continue/index directory. Downloading the necessary models with
ollama pull codellama:7b
ollama pull deepseek-coder:33b-instruct-q4_K_M
Starting ollama in a seperate cmd window with "ollama serve".
Now when im starting VS-Code, i can see that the index is created even before I see any logs of my ollama. This would tell me, that it is somehow creating the embeddings without using the specified embeddings provider.
However, after a while I see that ollama is deploying the codellama model, and answer some api/embedding requests.
The first few seem to be sucessfull:
However, after the first few it gives me errors:
(It looks like that continue makes requests for the embeddings, but just uses "something" else to create a broken index. Nonetheless, if i now write a message into the chat window, after all the requests are finished, ollama switches to the deepseek-coder model to generate a text response. However, if im now including the index via @CodeBase inside my message, I get an Error:
Error getting context items from codebase: TypeError: failed to downcast any to Array
this is the vs code dev tools, for some reason can't copy the console content:
To reproduce
- Install all dependencies (ollama, required models)
- Have ollama running in the background with ollama serve
- Start VS Code with the continue extensions, adapt config file to use ollama as embeddingsProvider.
- Restart VS Code (delete index manually, reload window, ...) --> A "broken" index is created, that probably doesen't even use the specified embeddingProvider, ollama embedding requests are sent but response timeout, index is created even before ollama embedding api is called.
Log output
The extension host log of vs code:
2024-02-29 20:07:43.038 [info] Extension host with pid 22320 started
2024-02-29 20:07:43.039 [info] Skipping acquiring lock for c:\Users\Paul\AppData\Roaming\Code\User\workspaceStorage\cd89fda7b8384bf3ea99dc178bb04111.
2024-02-29 20:07:43.110 [info] ExtensionService#_doActivateExtension Continue.continue, startup: false, activationEvent: 'onView:continue.continueGUIView'
2024-02-29 20:07:43.185 [info] ExtensionService#_doActivateExtension vscode.emmet, startup: false, activationEvent: 'onLanguage'
2024-02-29 20:07:43.355 [info] ExtensionService#_doActivateExtension vscode.git-base, startup: true, activationEvent: '*', root cause: vscode.git
2024-02-29 20:07:43.360 [info] ExtensionService#_doActivateExtension vscode.git, startup: true, activationEvent: '*'
2024-02-29 20:07:43.409 [info] ExtensionService#_doActivateExtension vscode.github, startup: true, activationEvent: '*'
2024-02-29 20:07:43.471 [info] Eager extensions activated
2024-02-29 20:07:43.507 [info] ExtensionService#_doActivateExtension vscode.debug-auto-launch, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:43.511 [info] ExtensionService#_doActivateExtension vscode.merge-conflict, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:43.517 [info] ExtensionService#_doActivateExtension ms-vscode-remote.remote-wsl-recommender, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:43.526 [info] ExtensionService#_doActivateExtension eamodio.gitlens, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:56.468 [error] Error: `neon::types::Deferred` was dropped without being settled
multiple models have been tested, this is the current config.json I use which does create the same behaviour as before:
{
"models": [
{
"model": "codellama:7b",
"title": "Ollama",
"provider": "ollama"
}
],
"slashCommands": [
{
"name": "edit",
"description": "Edit selected code"
},
{
"name": "comment",
"description": "Write comments for the selected code"
},
{
"name": "share",
"description": "Download and share this session"
},
{
"name": "cmd",
"description": "Generate a shell command"
}
],
"customCommands": [
{
"name": "test",
"prompt": "Write a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
"description": "Write unit tests for highlighted code"
}
],
"contextProviders": [
{
"name": "diff",
"params": {}
},
{
"name": "open",
"params": {}
},
{
"name": "terminal",
"params": {}
},
{
"name": "problems",
"params": {}
},
{
"name": "codebase",
"params": {}
}
],
"embeddingsProvider": {
"model": "codellama:7b",
"provider": "ollama"
}
}
multiple models have been tested, this is the current config.json I use which does create the same behaviour as before:
Be sure to restart the extension once the new provider is added — Command Palette > Restart Extension Host
Here's a functioning embeddings provider using Voyage:
"embeddingsProviders": {
"provider": "openai",
"model": "voyage-code-2",
"apiBase": "https://api.voyageai.com/v1/",
"apiKey": "REMOVED"
},
I'm not getting embeddings to work.
I'm using LM Studio. Chat and tabAutocomplete are working as expected.
config.json file has:
{
"title": "LM studio",
"provider": "lmstudio",
"model": "AUTODETECT",
"apiBase": "http://localhost:1234/v1/",
"apiKey": "lm-studio",
"completionOptions": {}
}
…
"tabAutocompleteModel": {
"provider": "lmstudio",
"title": "Codestral autocomplete",
"model": "lmstudio-community/Codestral-22B-v0.1-Q8_0.gguf"
},
"embeddingsProvider": {
"provider": "ollama",
"model": "nomic-ai/nomic-embed-text-v1.5.Q8_0.gguf",
"apiBase": "http://localhost:1234/v1"
}
Albeit very slow, auto complete is working ok. but embeddings is showing this error:
models list is showing:
I also tried adding "apiKey": ""(with the actual key) changing the url to end with /v1/embeddings but also didn't work.
Any hints on what's going on?
Actually I think I found the problem:
Continue is looking for Ollama's http://127.0.0.1:1234/api/embeddings instead of http://127.0.0.1:1234/v1/embeddings used by LM Studio
I think this is another bug, probably should add it as issue. not sure @TyDunn ?
LM Studio uses OpenAI API format, try
"embeddingsProvider": {
"provider": "openai",
"model": "nomic-ai/nomic-embed-text-v1.5.Q8_0.gguf",
"apiBase": "http://localhost:1234/v1/"
},
"model": "nomic-ai/nomic-embed-text-v1.5.Q8_0.gguf", "apiBase": "http://localhost:1234/v1/"
It works :)
the only thing I changed was "provider": "openai". I had "provider": "ollama" didn't know the provider part was so important.
Thanks a lot @Henri-J-Norden
Thanks for helping out @Henri-J-Norden! Looks like both this and the original problem are resolved, so I'll close the issue
