Error running Ast::eliminate_conditions

[gibbed]

Thanks for your efforts on this project! It decompiles all but one script in my target without any visible errors so far.

Error running Ast::eliminate_conditions
Source: ast\ast.cpp:2367

Failed to eliminate all test and copy conditions

failing_script.zip

[marsinator358]

This is because of an edge case caused by a conditional assignment to a local that remains unused.
I already knew about this bug, but the fix is going to be part of a rewrite that won't happen for some time.

I've decompiled the file for you by manually fixing it with custom code, the error happens on line 100.
decompiled.zip

[gibbed]

Sounds good, thanks!

[marsinator358]

@Ezriral That would require quite a rework, it would be easier to just skip the entire current block.

But even that isn't necessary because once this gets fixed the decompiler
should be able to handle all unmodified luajit bytecode without any problems.

[marsinator358]

but there is still this nullptr error

That's because the repo is not up to date, please use the current executable instead.

I'm currently taking a break from this project (except bugfixes) so I won't implement anything right now, sorry.
If you have a file that fails to decompile you can send it to me though.

[gibbed]

That's because the repo is not up to date, please use the current executable instead.

If you could push your changes, that would be nice. No rush though.

My local changes are mostly for automation in my local setup that wouldn't make sense for the project in general.

I did figure out that the UTF8 detection for strings was bugged and fixed that in my local copy; which I assume is one of your non-pushed changes.

[marsinator358]

I did figure out that the UTF8 detection for strings was bugged

I didn't change anything since the original commit, could you please tell me what's wrong/show me an example?

[gibbed]

I didn't change anything since the original commit, could you please tell me what's wrong/show me an example?

Oh interesting. When I built luajit-decompiler-v2 locally, UTF8 strings were still getting escaped. I found char was getting sign extended, completely throwing off the building of the UTF value that's validated by your range checks.

My quick&dirty local fix:

diff --git a/lua/lua.cpp b/lua/lua.cpp
index e013f31..4ff5697 100644
--- a/lua/lua.cpp
+++ b/lua/lua.cpp
@@ -837,7 +837,7 @@ void Lua::write_string(const std::string& string) {
 	uint8_t digit;
 
 	for (uint32_t i = 0; i < string.size(); i++) {
-		value = string[i];
+		value = (uint8_t)string[i];
 
 		if (!(value & 0x80)) {
 			if (string[i] >= ' ' && string[i] <= '~') {
@@ -877,7 +877,7 @@ void Lua::write_string(const std::string& string) {
 		} else if ((value & 0xE0) == 0xC0) {
 			if (i <= string.size() - 2) {
 				value <<= 8;
-				value |= string[i + 1];
+				value |= (uint8_t)string[i + 1];
 
 				if ((value & 0xC0) == 0x80
 					&& value >= 0xC280
@@ -891,8 +891,8 @@ void Lua::write_string(const std::string& string) {
 		} else if ((value & 0xF0) == 0xE0) {
 			if (i <= string.size() - 3) {
 				value <<= 16;
-				value |= (uint16_t)string[i + 1] << 8;
-				value |= string[i + 2];
+				value |= (uint16_t)(uint8_t)string[i + 1] << 8;
+				value |= (uint8_t)string[i + 2];
 
 				if ((value & 0xC0C0) == 0x8080
 					&& ((value >= 0xE0A080
@@ -909,9 +909,9 @@ void Lua::write_string(const std::string& string) {
 		} else if ((value & 0xF8) == 0xF0) {
 			if (i <= string.size() - 4) {
 				value <<= 24;
-				value |= (uint32_t)string[i + 1] << 16;
-				value |= (uint16_t)string[i + 2] << 8;
-				value |= string[i + 3];
+				value |= (uint32_t)(uint8_t)string[i + 1] << 16;
+				value |= (uint16_t)(uint8_t)string[i + 2] << 8;
+				value |= (uint8_t)string[i + 3];
 
 				if ((value & 0xC0C0C0) == 0x808080
 					&& value >= 0xF0908080

[marsinator358]

Oh that's because I'm using the /J option (default char is unsigned).

[Aussiemon]

@marsinator358 Here are 4 additional examples of "Error running Ast::eliminate_conditions at ast.cpp:2367", and 1 example with a different failure point and error message "Source: ast\ast.cpp:2271 (Failed to build condition in function 29)":

eliminate_conditions_compiled.zip eliminate_conditions_ljd_decompiled.zip

These files were tested with the latest binary in releases.

LuaJIT Decompiler v2
Compiled on Nov 21 2023
--------------------
Input file: \eliminate_conditions\bot_group.lua

Error running Ast::eliminate_conditions
Source: ast\ast.cpp:2367

Failed to eliminate all test and copy conditions
--------------------
Input file: \eliminate_conditions\item_stats_blueprints.lua

Error running Ast::eliminate_conditions
Source: ast\ast.cpp:2367

Failed to eliminate all test and copy conditions
--------------------
Input file: \eliminate_conditions\minion_attack.lua

Error running Ast::eliminate_conditions
Source: ast\ast.cpp:2367

Failed to eliminate all test and copy conditions
--------------------
Input file: \eliminate_conditions\qrencode.lua

Error running Ast::eliminate_conditions
Source: ast\ast.cpp:2271 (Failed to build condition in function 29)
--------------------
Input file: \eliminate_conditions\weapon_trait_templates.lua

Error running Ast::eliminate_conditions
Source: ast\ast.cpp:2367

Failed to eliminate all test and copy conditions
--------------------

[marsinator358]

@Aussiemon Thanks, only weapon_trait_templates.lua
has the same problem (ljd decompiled them as slot16 in the _test_template function). The others had different bugs.

[Aussiemon]

@marsinator358 I have a new file that fails on this exception, but unlike previous failures, putting a try-catch-early-return on the assertion eventually leads to an unhandled exception that the other files don't have.

Specifically, this try-catch works for weapon_trait_templates, but not this file:

for (uint32_t j = index; j <= i; j++) {
	try {
		assert(!block[j]->assignment.variables.size(), "Failed to eliminate all test and copy conditions", bytecode.filePath, DEBUG_INFO);
		conditionBuilder.add_node(conditionBuilder.get_node_type(block[j]->instruction.type, block[j]->condition.swapped),
			block[j]->instruction.label, function.get_label_from_id(block[j]->instruction.target), &block[j]->assignment.expressions);
	}
	catch (...) {
		return;
	}
}

Might be a new issue or edge case? room.zip

[marsinator358]

@Aussiemon It's the same issue, here is the decompiled file: room.txt
The error is caused by the unused local color at the end of function 13.

putting a try-catch-early-return on the assertion eventually leads to an unhandled exception that the other files don't have

Returning early will leave the ast in an incomplete state, which may cause the lua writer to throw.
If you want to force the decompilation, try to use continue instead or remove this guard in lua.cpp: https://github.com/marsinator358/luajit-decompiler-v2/blob/b89b76b893e9b4c3231a5a917f47bfa8bfaccd7d/lua/lua.cpp#L317

[Aussiemon]

@Aussiemon It's the same issue, here is the decompiled file: room.txt The error is caused by the unused local color at the end of function 13.

putting a try-catch-early-return on the assertion eventually leads to an unhandled exception that the other files don't have

Returning early will leave the ast in an incomplete state, which may cause the lua writer to throw. If you want to force the decompilation, try to use continue instead or remove this guard in lua.cpp:

https://github.com/marsinator358/luajit-decompiler-v2/blob/b89b76b893e9b4c3231a5a917f47bfa8bfaccd7d/lua/lua.cpp#L317

~~Works like a charm, thanks!~~

~~Here's my temp fix, for posterity: https://github.com/Aussiemon/luajit-decompiler-v2/commit/5583b2832f17e895271a7bcdae4f1b38e55bacc2~~

EDIT: Spoke too soon. continue works for files like room.lua, but breaks decompilation for files like weapon_trait_templates.lua. return works for these instead, but breaks decompilation for files like room.lua.

Switching between the two should be enough to get workable output at least.

[marsinator358]

@Aussiemon I forgot that the test-copy assertion is not inside the main loop. You can simply comment the assert for that one out and use continue for the if-statement one and it should work.

Also should you have any other files besides these two that fail with the same error I would appreciate if you could provide those as well as I have started working on the rewrite that will include a fix for this and I don't have enough files for testing.

[Aussiemon]

I have 3 examples total; including one that hasn't been posted here: examples.zip

[Ruslan16022]

Hello, what you can say about this file? Is it luajit, cuz decompiler doesn't work with this base64.zip

[COBRAG2G]

R-Server.zip

[marsinator358]

@Ruslan16022
This is Lua bytecode, not LuaJIT, though I've noticed that your problem already got solved on SourceForge.

@COBRAG2G
The file contains obfuscated lua code and not bytecode. I can't help you with deobfuscating though.

[COBRAG2G]

@Ruslan16022 If you can decrypt it I can pay you