[windows] stdout does not contain the complete output

[phryneas]

Hi, I'm a user of https://github.com/jvenant/passff which uses subprocess to execute the tool password-store.

The default output of calling pass is the output of tree -C and looks like this:

Password Store
├── Business
│   ├── some-silly-business-site.com
│   └── another-business-site.net
├── Email
│   ├── donenfeld.com
│   └── zx2c4.com
└── France
    ├── bank
    ├── freebox
    └── mobilephone

Now, in my case it is a longer listing with 170 lines, making up roughly 6300 characters.

The interesting thing is now that subprocess returns only a subset of these 170 lines:

Line 1-71 and line 170, concatenated together in the stdout property.

Is there any known limit to output length/a maximum line count?

[phryneas]

I did some more testing by adding a manual stdout callback function and logging it - it is called 6 times.

• the first call passes the header line (16 chars)
• second call contains my sites from the beginning to "humb" from "humblebundle.com" (1055 chars)
• the third call continues as expected with "lebundle.com" and ends again mid-line with "├─" (1100 chars)
• calls four to six are empty strings.

I'm trying if I can find more information here.

[phryneas]

Mh :/

I did some more poking around but could not find any more info I believe would be relevant - I'm just not familiar enough with the win32 api :(

Only thing that seems way off to me is the fact that those strings get so long at all - shouldn't the callback be called At a minimum at each occurance of \n or \r. according to the documentation?

[phryneas]

PS: the char counts above may be off - I guess copy-pasting the strings out to measure them will have expanded some unprintable characters to readable multi-character counterparts :/

[phryneas]

Okay, sorry again for this spam, but I just found the logging function :grin:

It looks like a charset problem..

23:48:03.818 "got 15 bytes from stdout
" pass.js:252
23:48:03.826 "got 1024 bytes from stdout
" pass.js:252
23:48:03.827 "String conversion failed: [Exception... "Component returned failure code: 0x8050000e (NS_ERROR_ILLEGAL_INPUT) [nsIScriptableUnicodeConverter.ConvertToUnicode]"  nsresult: "0x8050000e (NS_ERROR_ILLEGAL_INPUT)"  location: "JS frame :: chrome://passff/content/subprocess.jsm :: convertBytes :: line 359"  data: no]
" pass.js:255
23:48:03.827 "got 1024 bytes from stdout
" pass.js:252
23:48:03.827 "String conversion failed: [Exception... "Component returned failure code: 0x8050000e (NS_ERROR_ILLEGAL_INPUT) [nsIScriptableUnicodeConverter.ConvertToUnicode]"  nsresult: "0x8050000e (NS_ERROR_ILLEGAL_INPUT)"  location: "JS frame :: chrome://passff/content/subprocess.jsm :: convertBytes :: line 359"  data: no]
" pass.js:255
23:48:03.827 "got 995 bytes from stdout
" pass.js:252
23:48:03.827 "String conversion failed: [Exception... "Component returned failure code: 0x8050000e (NS_ERROR_ILLEGAL_INPUT) [nsIScriptableUnicodeConverter.ConvertToUnicode]"  nsresult: "0x8050000e (NS_ERROR_ILLEGAL_INPUT)"  location: "JS frame :: chrome://passff/content/subprocess.jsm :: convertBytes :: line 359"  data: no]
" pass.js:255
23:48:03.828 "Got msg from stdout: EOF reached
" pass.js:252
23:48:03.828 "Pipe stdout closed
" pass.js:252

I've got it working by setting the environment variable TREE_CHARSET="windows-1252".

What IS interesting is that setting TREE_CHARSET to UTF8, tree seems to produce something that Firefox does NOT see as UTF-8.

So maybe it would be a good idea and add windows-1252 as a fallback encoding, as Mozilla itself suggests (although not exactly in this context) in https://developer.mozilla.org/en-US/docs/Web/Guide/Localizations_and_character_encodings .

In my tests, that has proven itself as somehow ugly, but at least it produced some content instead of failing silently which is the current behaviour if registerDebugHandler is not set.

[phryneas]

Just as a sidenote, if someone wants to dig further into this:

it looks like it was one of these characters that could not be converted:

(taken from colors.c of the tree source code )

 { utf8,        "\342\224\202\302\240\302\240",
"\342\224\234\342\224\200\342\224\200", "\342\224\224\342\224\200\342\224\200", "\302\251" }