Crashing with Diacritics

[JranZu]

It took me forever to track this down, but submitting a prompt with a diacritic (é, ç, etc.), it throws a corrupted memory exception. I was able to fix this by filtering my prompts when they come in. Just wanted to let you know:

private static string ReplaceDiacritics(string text)
{
	if (string.IsNullOrWhiteSpace(text))
	{
		return text;
	}

	// Normalize the text to decompose accents into separate characters
	var normalizedString = text.Normalize(NormalizationForm.FormD);
	int length = normalizedString.Length;

	// Use string.Create for optimized memory allocation
	return string.Create(length, normalizedString, (span, value) =>
	{
		int index = 0;
		foreach (var c in value)
		{
			// Directly check for non-spacing marks and only copy valid characters
			if (CharUnicodeInfo.GetUnicodeCategory(c) != UnicodeCategory.NonSpacingMark)
			{
				span[index++] = c;
			}
		}

		// Truncate the span to the valid length and normalize back to Form C
		var cleanedSpan = span.Slice(0, index);
		var cleanedString = new string(cleanedSpan);
		cleanedString.Normalize(NormalizationForm.FormC).CopyTo(span);
	});
}

[DarthAffe]

This is weird. It seems to be either model or backend dependant. I tried different things, but for me these characters work fine (like it does not crash, they don't produce good results).

[JranZu]

Weird - mine crash everytime on every model I have tried; using Cuda12 Windows. If you can't repro it feel free to close this.