Invalid UTF-16 on Linux if the system language isn’t English

[moxie-coder]

For some reason, and it might correlate with #1906 but Linux seems to have issues with starting games made with Lime if the system language isn’t English, I’m currently away from home and I don’t have Linux installed on my new computer yet, but I’ve seen two examples of this issue happening (which I’ll link here)

https://github.com/CodenameCrew/CodenameEngine/issues/573 (Tested on Arch Linux) https://github.com/JordanSantiagoYT/FNF-JS-Engine/issues/975 (Tested on an Debian/Ubuntu based Distribution)

Seems to happen on any Linux distribution as far as I know, this also has similar issues when running from a folder with special characters, but this was already fixed I believe, I haven’t tested on Windows/Mac yet, and I don’t have an Mac

[tobil4sk]

Which version of lime was being used in these cases?

[moxie-coder]

Which version of lime was being used in these cases?

In our case 8.1.3 and 8.1.2, this hasn't been tested yet on Lime 8.2

[tobil4sk]

We discovered that the cause of the issue isn't the system language itself, but the path from which the program is started. If that contains non-ascii characters (~/Documents, ~/Downloads, etc maybe be non-ascii with other languages), then this issue can be triggered when loading assets.

I tested CodenameEngine with git lime and was able to reproduce the utf-16 crash.

It looks like there is a similar issue when running on hashlink: #1757.

[moxie-coder]

We discovered that the cause of the issue isn't the system language itself, but the path from which the program is started. If that contains non-ascii characters (~/Documents, ~/Downloads, etc maybe be non-ascii with other languages), then this issue can be triggered when loading assets.

I tested CodenameEngine with git lime and was able to reproduce the utf-16 crash.

It looks like there is a similar issue when running on hashlink: #1757.

Ah okay, I had a feeling that’s what it was since the same sort of issue happens on Windows too, and I thought it’d be happening on Linux too due to how paths are handled here

[tobil4sk]

The invalid characters come from concatenating the relative font path to the root path of the assets, which is set here:

https://github.com/openfl/lime/blob/99e986ce39cf438c64768c20f20d8161566c77de/templates/haxe/ManifestResources.hx#L63

And the incorrect conversion occurs here: https://github.com/openfl/lime/blob/99e986ce39cf438c64768c20f20d8161566c77de/project/src/backend/sdl/SDLSystem.cpp#L115

This is somewhat similar to the windows issue: #1906

On Linux and some other systems, HOME is used for these directories, so if we override the environment variable to contain unicode, we can test if they are valid strings. Here is a minimal sample to demonstrate the issue when getting system directories:

// Source/Main.hx
class Main extends lime.app.Application {
	public function new() {
		super();
		trace(lime.system.System.applicationDirectory);
		Sys.putEnv("HOME", "/TÉST");
		trace(lime.system.System.desktopDirectory);
		trace(lime.system.System.documentsDirectory);
		trace(lime.system.System.userDirectory);
	}
}

<!-- project.xml -->
<?xml version="1.0" encoding="utf-8"?>
<project>

	<meta title="HelloWorld" package="com.sample.helloworld" version="1.0.0" company="Cømpany Name" />
	<app main="Main" path="Éxport" file="HelloWorld" />

	<source path="Source" />

	<haxelib name="lime" />

	<assets path="Assets" rename="assets" />

</project>

Running this currently gives the output:

Source/Main.hx:4: /.../HelloWorld/ힿ�ힿ�xport/linux/bin/؂ႀ읠꒜嗌
Source/Main.hx:6: /Tힿ�ힿ�ST/Desktop/
Source/Main.hx:7: /Tힿ�ힿ�ST/Documents/
Source/Main.hx:8: /Tힿ�ힿ�ST/
Failed to create /TÉST for shader cache (Permission denied)---disabling.
Critical Error: Allocating from a GC-free thread

Here is a PR to handle the string encoding conversions correctly: #1974.

Also, the reason why this issue only started happening after 8.1.0 is because we didn't use the applicationDirectory as the asset root before that version: #1619.

[joshtynjala]

Closing with merge of #1974.