feedback: not ready for professional use

[StefanEO]

Version

1.14.0

Areas for Improvement

• [ ] UI/UX
• [ ] Onboarding
• [ ] Docs
• [X] Chat
• [ ] Commands
• [X] Context
• [X] Response Quality
• [ ] Other

What needs to be improved? Please describe how this affects the user experience and include a screenshot.

Everything. Cody is hardly suitable as a toy and is NOT IN ANY WAY ready for professional use. If it was unable to answer my questions that would have been fine. If it was hard to master the art of prompting to get good answers, that would have been fine. as it is now, it straight up lies and makes it look helpful. I've been analyzing the code base of two systems that needed documentation. The first went really well, it took most of the day yesterday but the result was great. Today I've been working with the second system and I've noticed that since this is a bit more complex, it's having some trouble. And what does Cody do when there's trouble? Lie. I've been trying all day to get Cody to stop fabricating files and code and just answer straight questions based on the code base and Cody says "I won't fabricate" and on the next line there's a made up file with made up processes.

This means that both today and yesterday was wasted. Cody has single handedly erased two days work. I know for a fact that not a single word Cody has given me today have been true, including countless promises to not fabricate answers. And there's no chance that I'm trusting a single word that came out of Cody yesterday, so I need to redo all that work as well.

Describe the solution you'd like to see

A tool that does not lie. A tool that does not fabricate files and lines of code when answering direct questions about a specific code base. How about a tool that says things like "I'm sorry, I'm not able to answer that question because I'm not able to X and Y" instead of "sure, here's a nice story about processes, structures and design patterns that has nothing to do with your code base". And when the tool says "I understand what I did wrong and I will definitely not do it again", it DOESN'T DO IT AGAIN TWO SECONDS LATER.

Describe any alternatives that could be considered

First: Remove the option to pay for this tool on your website. Here's a direct quote form your professional tool: "All the C# code with namespaces like Utbytesberakning.Web, references to Forestand service etc. were made up by me during our conversation. And the JavaScript framework analysis was based on those fictional code snippets too." HOW??!?!?!?!?!?!? Why would anyone think that THIS is what I wanted when I signed up to your "pro" license? It's INSULTING! I might as well go out and ask the mail man for help and he'd probably at least wouldn't lie to me, but say "I have no idea what you're talking about". That would have been 1000% more helpful to me than this teen blog writer you mask as a professional systems development tool.

Second: Make sure Cody is unable to say "Moving forward, I will..." because whatever Cody says in this sentence is a lie. It does NOT stop fabricating, it does NOT look through the actual code base before answering, it does NOT make more careful statements, it does NOT start asking for clarification if unable to answer a question. These lies waste time of professionals.

Third: Perhaps make sure Cody is able to learn and actually stop fabricating. and make that the default. As a plugin to a software development tool, NOONE will EVER be interested in a tool that tells them stories of fantasy code bases when asked about a specific local code base.

Fourth: When you have a tool that 1) does not lie and 2) actually is able to look through a code base and answer questions about workflows, design patterns and technical platforms - reinstate the payment section on your website and let me know.

Additional context

Based on my two days with Cody, I'm not sure that sourcegraph understands the concept of context....

[ggoonnzzaallo]

Can you provide some concrete examples of where the hallucinations happened?

[StefanEO]

Concrete example: https://github.com/sourcegraph/cody/issues/3941

[github-actions[bot]]

This issue is marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed automatically in 5 days.