Arabic isn't properly rendered in Neovim until a newline is inserted

[pro-anon]

The letters here aren't properly rendered Only after inserting a new line they get fixed This issue doesn't happen when typing directly into the terminal

[arakiken]

Sorry for replying late. Thanks for your report, but I couldn't reproduce this problem in my environment. (neovim 0.8.2 with empty ~/.config/nvim) Will you tell me your neovim version and configuration?

[pro-anon]

v0.8.2 I've tried it without any config, the issue is still the same

[pro-anon]

I'm running it on arch. Installed mlterm from here https://aur.archlinux.org/packages/mlterm

[arakiken]

I tested plain mlterm(3.9.2) and plain neovim(0.8.2) without any configuration, but I haven't reproduced it yet. I'm not sure, but does ./configure without --enable-optimize-redrawing option in building mlterm improve it?

[pro-anon]

Nope, still the same. Maybe it's an issue with the font? What font do you use?

[pro-anon]

I've noticed that, in addition to inserting a new line, inserting ﻻ re-renders the line properly

[arakiken]

I use "Not Sans Arabic" font to show arabic characters, but I think it doesn't caused by font configuration. Sorry but I haven't found out the cause yet.

[holytrousers]

I don't think the bug is in mlterm. Try to run nvim with the --clean option. Looks like it's happening when you have set cursorline and the cursor is highlighting the arabic text. This can happen if you are using variable width fonts. Make sure you have monospace fonts with Arabic glyphs installed and you have configured mlterm to use them (Try Dejavu Sans Mono) if you are using xft, you have to specify which fonts to use for Arabic glyphs. I have tried to set all Arabic Unicode ranges to use Deja Vu Sans Mono and managed to make it work :

DEFAULT = Fantasque Sans Mono, 12
ISO10646_UCS4_1_FULLWIDTH = Noto Sans Mono CJK JP, 10
U+10a60-10a7f=DejaVu Sans Mono, 10
U+10a80-10a9f=DejaVu Sans Mono, 10
U+600-6ff=    DejaVu Sans Mono, 10
U+750-77f=    DejaVu Sans Mono, 10
U+8a0-8ff=    DejaVu Sans Mono, 10
U+1ee00-1eeff=DejaVu Sans Mono, 10 
U+10e60-10e7f=DejaVu Sans Mono, 10
U+fe70-feff=  DejaVu Sans Mono, 10
U+fb50-fdff=  DejaVu Sans Mono, 10

[pro-anon]

@holytrousers I've tried the --clean flag with neovim and DejaVu Sans Mono, but the issue got worse. Now I have to insert a new line, then delete it so that the line before it would get rendered correctly.

[holytrousers]

have you tried the mlterm-git package ? the one you mentioned has been flagged as out of date

[holytrousers]

@pro-anon I have reproduced your problem by using the :set arabic option and actually writing the text. Previously i pasted arabic text from yamli.com into vim that's why i wasn't able to reproduce the problem. Writing in arabic (using the "ara" keymap) inside the shell produces the same behaviour though. لا fixes it. The same problem happens when writing in arabic inside the shell in other terminals xterm, alacritty. Looks like Kitty implements bicon properly out of the box. Running bicon in xterm and alacritty and writing in arabic inside the shell fixes the problem. Writing arabic with the :set arabic option in vim works fine in xterm and alacritty without running bicon. Running bicon in mlterm doesn't fix the problem though.

As a temporary workaround try pressing Control+L in normal mode inside vim, that will refresh the screen and render the ligatures properly (inoremap <Esc> <Esc><C-L> to refresh every time you switch to normal mode ) Another trick i've found fixes ligatures : try to insert a letter, then move backward one letter, so that the cursor is inserting text before that letter. ligatures seem to work then. Looks like it's a problem with the bicon implementation in mlterm. @arakiken might give an insight on this

[holytrousers]

ps. add inoremap <Esc> <Esc><C-L> to your vim configuration

[holytrousers]

NB. The trick with prepending a letter before the inserted text works only partially though. Baseline Ligatures work then but the letters at the end of the word don't change their shape.

[holytrousers]

I think we have narrowed down the problem, try :

• Start vim
• :set arabic
• Begin typing in latin by pressing ctrl-^ (from left-to-right)
• Switch back to arabic by pressing again ctrl-^ and write in arabic

Ligatures and end of word letters render properly. Which means that the bug occurs only when the whole line is written from right-to-left, ie. the beginning of the line is on the right side of the window.

[holytrousers]

@pro-anon : your prompt is styled to align the text to the left side of the screen, that's why when you type arabic text in the shell it displays properly. This confirms the previous conclusion : the problem occurs only when the whole line is RTL.

[holytrousers]

@pro-anon : i think enabling the Variable Width Column option solves the issue completely. Can you confirm that ?

[holytrousers]

Another issue i have found with arabic script : when using :set arabic in vim, beginnings of lines always stay on the left side, and text is switched to the right side only after some text is written. On a new line, the cursor remains on the right side. Other terminals don't seem to have this problem.

[pro-anon]

@holytrousers

I think we have narrowed down the problem, try :

* Start vim

* `:set arabic`

* Begin typing in latin by pressing `ctrl-^` (from left-to-right)

* Switch back to arabic by pressing again `ctrl-^` and write in arabic

Ligatures and end of word letters render properly. Which means that the bug occurs only when the whole line is written from right-to-left, ie. the beginning of the line is on the right side of the window.

Yes, that is exactly what I do set arabic. Sorry for not mentioning it before. And you were correct! Starting with Latin fixes it, <C-l> too but only in normal not insert mode.

[pro-anon]

I'm not sure what you mean here. If you mean the mlterm flag --varwidth then yes, it does fix it, but it looks very disorienting (I have used only one white space between every word, and no white space at the beginning of each line):

[holytrousers]

I'm not sure what you mean here. If you mean the mlterm flag --varwidth then yes, it does fix it, but it looks very disorienting (I have used only one white space between every word, and no white space at the beginning of each line): Yes, either

• --varwidth
• use_variable_column_width = true to your ~/.mlterm/main
• use the configuration dialog by pressing Control + RMB and go to the font tab.

I think your issue is because you have set the varwidth option but forgot to set the font to monospace. set it in the ~/.mlterm/vaafont use DejaVu Sans Mono for the moment, and set it for the unicode blocks i've mentioned earlier.

[arakiken]

Thanks very much. I can reproduce this problem. (enbugged over 5 years ago) I'll fix it next weekend. Please wait.

[pro-anon]

Yes, either

* `--varwidth`

* `use_variable_column_width = true` to your ~/.mlterm/main

* use the configuration dialog by pressing `Control + RMB` and go to the `font` tab.

I think your issue is because you have set the varwidth option but forgot to set the font to monospace. set it in the ~/.mlterm/vaafont use DejaVu Sans Mono for the moment, and set it for the unicode blocks i've mentioned earlier.

Can you show an example of a vaafont config file? I've added this DEFAULT=DejaVu Sans Mono to my ~/.mlterm/font which fixed the Arabic font issue but messed up the rest: @holytrousers

[holytrousers]

I believe that arabic text is not in DejaVu Sans Mono, it looks like some variable width font. I guess you are using cairo which switches automatically to the nearest font with necessary glyphs. I don't use cairo because its rendering quality is ugly and xft looks much crispier, but that's not related to this issue. Anyway i've set both aafont and vaafont to use these fonts

DEFAULT = Fantasque Sans Mono, 12
ISO10646_UCS4_1_FULLWIDTH = Noto Sans Mono CJK JP, 10
U+10a60-10a7f=DejaVu Sans Mono, 10
U+10a80-10a9f=DejaVu Sans Mono, 10
U+600-6ff=    DejaVu Sans Mono, 10
U+750-77f=    DejaVu Sans Mono, 10
U+8a0-8ff=    DejaVu Sans Mono, 10
U+1ee00-1eeff=DejaVu Sans Mono, 10 
U+10e60-10e7f=DejaVu Sans Mono, 10
U+fe70-feff=  DejaVu Sans Mono, 10
U+fb50-fdff=  DejaVu Sans Mono, 10

First line is for latin scripts, second line is for japanese and chinese, the rest are for arabic. It works for me, i'm sure not all blocks are necessary, but i've copied all of those i've found in the unicode table. Please try to set both files or at least the vaafont and report back if it works.

[pro-anon]

Much better now. Thanks @holytrousers

[arakiken]

Sorry for replying late. I think that this following commit improves it. https://github.com/arakiken/mlterm/commit/f8c6b88cb5a14d6896cb23f5c051290ec8034ffb

[pro-anon]

Works very well! Thanks!

Note though, it's better to remove use_variable_column_width = true from the user config because it had some issues like this:

https://github.com/arakiken/mlterm/assets/97809837/a4d25b42-428e-43d5-8511-4c85d82475e1

[holytrousers]

Sorry for replying late. I think that this following commit improves it. f8c6b88

Thank you for fixing on this, glyphs render properly now ! The only thing that is still conflicting with vim's arabic input is mlterm's right to left implementation : in vim, :set arabic moves the cursor to the right side on an new empty line, but it looks like mlterm moves it again to the opposite side thinking it's on the left side, then moves back to the right side upon entering arabic text.

[holytrousers]

reporting back from the front with some screenshots: See how the cursor is positioned on a new line, it should be on the right side (as in other terminals)

[holytrousers]

To reproduce it, just type some arabic text and erase it with backspace and the erased letters will leave those traces. This happens on letters that extend down the baseline like س ي ن

To make this glitch disappear set line_space = 1 or greater.

[holytrousers]

The problem with refreshing positional letters drawing (that has been solved in f8c6b88 ) still persists when using backspace. See this ب that draws as if stil connected to the next letter after it has been deleted: