Autocomplete on vscode stops working

[longle255]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:macOs
- Continue version: 0.9.215
- IDE version: 1.94.0
- Model: any (qwen2.5-coder:7b-base, deepseek-coder:6.7b-base)
- config.json:
  
{
  "models": [
    {
      "title": "Ollama",
      "provider": "ollama",
      "model": "AUTODETECT"
    },
    {
      "title": "qwen2.5-coder:7b-instruct",
      "model": "qwen2.5-coder:7b-instruct",
      "contextLength": 16384,
      "apiBase": "http://localhost:11434",
      "provider": "ollama"
    }
  ],
  "customCommands": [
    {
      "name": "test",
      "prompt": "{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
      "description": "Write unit tests for highlighted code"
    }
  ],
  "tabAutocompleteModel": {
    "title": "qwen2.5-coder:7b-base",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b-base"
  },
  "tabAutocompleteOptions": {
    "disableInFiles": ["*.md", "*.txt"]
  },
  "completionOptions": {
    "maxTokens": 8192
  },

  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "edit",
      "description": "Edit selected code"
    },
    {
      "name": "comment",
      "description": "Write comments for the selected code"
    },
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ],
  "embeddingsProvider": {
    "provider": "ollama",
    "model": "nomic-embed-text"
  }
}

Description

Autocomplete worked for me before, until recently (yesterday or today, I'm not sure when), and then it stopped working. The chat feature is still working fine with any models. Only autocomplete has the problem.

Output data of autocomplete shows it did not generate the prompt. The output of ollama shows no prompt was provided.

I tried a couple of versions of Continue (release, pre-release), but the problem persists.

To reproduce

No response

Log output

Continue log



==========================================================================
==========================================================================
##### Completion options #####
{
  "contextLength": 8096,
  "maxTokens": 8192,
  "model": "qwen2.5-coder:7b-base",
  "temperature": 0.01,
  "stop": [
    "<|endoftext|>",
    "<|fim_prefix|>",
    "<|fim_middle|>",
    "<|fim_suffix|>",
    "<|fim_pad|>",
    "<|repo_name|>",
    "<|file_sep|>",
    "<|im_start|>",
    "<|im_end|>",
    "\n\n",
    "\r\n\r\n",
    "/src/",
    "#- coding: utf-8",
    "```",
    "\nfunction",
    "\nclass",
    "\nmodule",
    "\nexport",
    "\nimport"
  ],
  "raw": true
}

##### Prompt #####
==========================================================================
==========================================================================
Completion:

output from ollama showing no prompt entered

DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=1 tid="0x1ebeefac0" timestamp=1728402861
[GIN] 2024/10/08 - 17:54:21 | 200 |   26.967541ms |       127.0.0.1 | POST     "/api/generate"
time=2024-10-08T17:54:21.898+02:00 level=DEBUG source=sched.go:407 msg="context for request finished"
time=2024-10-08T17:54:21.898+02:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/Users/test/.ollama/models/blobs/sha256-485c567a3dcfc5403695bf0d70b3b70c0df36034426f046a35eabf9d24676878 duration=30m0s
time=2024-10-08T17:54:21.898+02:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/Users/test/.ollama/models/blobs/sha256-485c567a3dcfc5403695bf0d70b3b70c0df36034426f046a35eabf9d24676878 refCount=0
time=2024-10-08T17:54:23.281+02:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=/Users/test/.ollama/models/blobs/sha256-485c567a3dcfc5403695bf0d70b3b70c0df36034426f046a35eabf9d24676878

[Patrick-Erichsen]

Hi @longle255 , any chance you are seeing the same error reported here? https://github.com/continuedev/continue/issues/2457#issuecomment-2400013843

[longle255]

@Patrick-Erichsen I can't tell for sure if the causes of those errors was the same. I don't see similar error message in vscode debug console. in fact I don't see any error at all, in any output

[alanxmay]

@Patrick-Erichsen I'm facing same issue, the reason why the completion was a space character is because the stop list contains "```".

And Qwen2.5-Coder-7B-Instruct completion result is

 ```python\n    # Uncomment the following line to set a custom chat template\n     # runtime.endpoint...

Modify stop list and it works. But I didn't figure out how to config continue to make this works.

[facelezzzz]

Same with Intellij IDEA

##### Completion options #####
{
  "contextLength": 8192,
  "maxTokens": 8192,
  "temperature": 0,
  "topP": 1,
  "presencePenalty": 0,
  "frequencyPenalty": 0,
  "model": "deepseek-chat",
  "stop": [
    "<｜fim▁begin｜>",
    "<｜fim▁hole｜>",
    "<｜fim▁end｜>",
    "//",
    "<｜end▁of▁sentence｜>",
    "\n\n",
    "\r\n\r\n",
    "/src/",
    "#- coding: utf-8",
    "```",
    "\nclass",
    "\nfunction"
  ],
  "raw": true
}

##### Prompt #####

[longle255]

After some debugging into continue code, I think I found the cause of my issue, it's the contextLength of tabAutocompleteModel.

Looks like by default the contextLength for tabAutocompleteModel is 8096, while my option for "completionOptions": { "maxTokens": 8192 }. That make the prompt to be pruned completely.

So my fix was adding contextLength option to tabAutocompleteModel

"tabAutocompleteModel": {
    "title": "qwen2.5-coder:7b-base",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b-base",
    "contextLength": 16384
  },
  "completionOptions": {
    "maxTokens": 8192
  }

this fixes the problem for me.

@facelezzzz I think it would also fix the problem for you, as your context length and maxToken were the same.

[facelezzzz]

@longle255 It works again! Thank you for your efforts.

[Patrick-Erichsen]

Thanks for the writeup here @longle255 ! We should definitely provide a better warning in this case.

Curious why both of you are explicitly setting your completionOptions to 8192?

[longle255]

@Patrick-Erichsen As far as I remember, I had a problem with Continue where the the generated answer was cut off in the middle for quite a few times for me. After digging a while I found an issue here talking about the same problem https://github.com/continuedev/continue/issues/2319 and they mentioned about the option in one of the comment. Adding the option did fix the problem for me.

[codejet]

For me, adding maxTokens also fixed the issue of the autocomplete not working. But the docs actually say that the param should be called maxPromptTokens (https://docs.continue.dev/customize/deep-dives/autocomplete) 🤔

[github-actions[bot]]

This issue hasn't been updated in 90 days and will be closed after an additional 10 days without activity. If it's still important, please leave a comment and share any new information that would help us address the issue.

[github-actions[bot]]

This issue was closed because it wasn't updated for 10 days after being marked stale. If it's still important, please reopen + comment and we'll gladly take another look!