cody icon indicating copy to clipboard operation
cody copied to clipboard

[WIP ]Adding the feature flag with tests

Open arafatkatze opened this issue 1 year ago • 0 comments

This PR introduces several key changes to improve the autocomplete functionality in Cody’s inline completions, primarily focused on user experience improvements, testing, and analysis.

Changes:

1.	Feature Flag Implementation:
•	Introduced a new feature flag to disable specific logic related to inline completions. This will allow controlled experimentation and toggling of the logic during development and A/B testing.
2.	Autocomplete Behavior Improvements:
•	Tested the current behavior of the autocomplete system when completing in the middle of a line.
•	Addressed the issue where the model ignores the current line suffix and suggests conflicting completions.
3.	UX Enhancements:
•	Observed and documented cases where the model does not take the existing suffix into account.
•	Considered implementing behavior from Cursor, where a different suffix shows the full line completion and replaces the line when accepted. This change might be added in a follow-up if necessary.
4.	A/B Testing:
•	Prepared for A/B testing of the changes to assess the user experience and determine the impact of the improvements.
5.	Metric Analysis:
•	Introduced methods to analyze key autocomplete metrics:
•	CAR (Completion Acceptance Rate)
•	wCAR (Weighted Completion Acceptance Rate)
•	Number of accepted characters per user
•	Latency metrics

Test plan

Changelog

arafatkatze avatar Sep 04 '24 23:09 arafatkatze