emilsvennesson
Playback fails using newer Widevine CDM 4.10.2252.0 on ARM
It seems our users updating Widevine CDM on ARM are downloading a different kind of libwidevinecdm.so.
The error shown in the Kodi log is:
ERROR: AddOnLog: InputStream Adaptive: Unable to load widevine shared library (/storage/.kodi/cdm/libwidevinecdm.so)
ERROR: AddOnLog: InputStream Adaptive: OpenDRMSystem failed
The older Widevine CDM has the following dependencies:
kodi04:~ # cd ~/.kodi/userdata/addon_data/script.module.inputstreamhelper/backup
kodi04:backup # ls -l 13505.111.0/libwidevinecdm.so
-rwxr--r-- 1 root root 7297324 Feb 4 19:03 13505.111.0/libwidevinecdm.so
kodi04:backup # sha1sum 13505.111.0/libwidevinecdm.so
d5ee489d5c409ff6cb423f318a23dba71ff2f2b0 13505.111.0/libwidevinecdm.so
kodi04:backup # ldd 13505.111.0/libwidevinecdm.so
linux-vdso.so.1 (0xbeca5000)
/usr/lib/libarmmem-v7l.so (0xb67ac000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0xb6783000)
libm.so.6 => /usr/lib/libm.so.6 (0xb6719000)
libdl.so.2 => /usr/lib/libdl.so.2 (0xb6706000)
librt.so.1 => /usr/lib/librt.so.1 (0xb66ef000)
libnss3.so => /usr/lib/libnss3.so (0xb65fa000)
libnssutil3.so => /usr/lib/libnssutil3.so (0xb65c8000)
libnspr4.so => /usr/lib/libnspr4.so (0xb6594000)
libc.so.6 => /usr/lib/libc.so.6 (0xb6459000)
/usr/lib/ld-linux-armhf.so.3 (0xb6f0a000)
libplc4.so => /usr/lib/libplc4.so (0xb6f30000)
libplds4.so => /usr/lib/libplds4.so (0xb6f2c000)
The newer (broken) Widevine CDM has fewer dependencies:
kodi04:~ # ls -la .kodi/cdm/libwidevinecdm.so
-rwxr--r-- 2 root root 8887360 May 10 14:51 .kodi/cdm/libwidevinecdm.so
kodi04:~ # sha1sum .kodi/cdm/libwidevinecdm.so
4d129768d3753328a837a940e6ee903830c791c4 .kodi/cdm/libwidevinecdm.so
kodi04:~ # ldd .kodi/cdm/libwidevinecdm.so
linux-vdso.so.1 (0xbee96000)
/usr/lib/libarmmem-v7l.so (0xb5dcb000)
libdl.so.2 => /usr/lib/libdl.so.2 (0xb5db8000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0xb5d8f000)
librt.so.1 => /usr/lib/librt.so.1 (0xb5d78000)
libm.so.6 => /usr/lib/libm.so.6 (0xb5d0e000)
libc.so.6 => /usr/lib/libc.so.6 (0xb5bd3000)
/usr/lib/ld-linux-armhf.so.3 (0xb6f6c000)
This newer Widevine CDM no longer depends on libnss3, libnssutil3, libnspr4, libplc4 and libplds4.
Reported at CastagnaIT/plugin.video.netflix#1154 and xbmc/inputstream.adaptive#678
The immediate workaround is to rollback to a previously downloaded version by doing the following in Kodi:
Add-ons » Program add-ons » InputStream Helper » Restore Widevine CDM library...
Select a version prior to the broken Widevine CDM version 4.10.2252.0.
Obviously this will impact anyone setting up a new ARM Kodi system, with no access to a backup Widevine CDM.
If I interpret the latest news from Widevine correctly (https://www.widevine.com/news), older Widevine CDM versions will no longer be working as soon as 2021-05-31. So if we do not fix this new Widevine CDM, most of our ARM users would be affected in 3 weeks time.
It was also already reported at inputstream.adaptive. https://github.com/xbmc/inputstream.adaptive/issues/678 Let's hope they find a solution to get it working again.
A temporary solution is adding SCARLET to config.py and removing all other CHROMEOS_RECOVERY_ARM_HWIDS. This will work until June 1st 2021.
Raspberry Pi OS is still using the browser version of the CDM, 4.10.1679. Not sure if that will go through the same changes or not.
We will know for sure on June 1st 2021.
A new v0.5.3 version of InputStreamHelper is released which temporary fixes this issue by extracting the older 4.10.1679.0 Widevine CDM from the Chrome OS 13729.56.0 Scarlet image.
https://github.com/emilsvennesson/script.module.inputstreamhelper/releases/tag/v0.5.3
If I understand the issue correctly, the error cannot allocate memory in static TLS block is a result of a previously loaded library deplating the preallocated TLS area (https://bugzilla.redhat.com/show_bug.cgi?id=1722181). And a workaround is to preload that library.
For this new Widevine CDM, that would mean any of the below libraries could have caused this. And running it with PRELOAD=libwhatever.so could do the trick.
kodi04:~/.kodi/cdm # ldd libwidevinecdm.so
linux-vdso.so.1 (0xbecad000)
/usr/lib/libarmmem-v7l.so (0xb5d70000)
libdl.so.2 => /usr/lib/libdl.so.2 (0xb5d5d000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0xb5d34000)
librt.so.1 => /usr/lib/librt.so.1 (0xb5d1d000)
libm.so.6 => /usr/lib/libm.so.6 (0xb5cb3000)
libc.so.6 => /usr/lib/libc.so.6 (0xb5b78000)
/usr/lib/ld-linux-armhf.so.3 (0xb6f11000)
@wagnerch Since you have a working ARM environment incl. debug tools, can you run the one-liner using readelf from that link to see which library is using a larger TLS space? -> readelf -Wl libwhatever.so | grep TLS
@dagwieers All of those dependencies would already be loaded from the main kodi.bin. Personally, I think the issue might be because tcmalloc is statically linked in libwidevinecdm, and tcmalloc replaces malloc. So that may not play nice with any runtime allocation of TLS since it requires malloc. Not sure. I saw some similar issues with jemalloc which is another malloc replacement. TLS is deep in the weeds of compiler/linker stuff. The error itself is coming from ld.so (dynamic linker): https://github.com/bminor/glibc/blob/5633541d3b9a78fc5283af3a2f3e824126ef785a/elf/dl-reloc.c#L140
I am just using cross-compiler tools on Ubuntu to run readelf, objdump, etc.
BTW, the error is specific about static TLS block, so I am guessing there is no runtime allocation. Which I believe that is allocated by kodi.bin.
@wagnerch IANAKD, there is a preallocated block, but it is being used at runtime by libraries. The error occurs when that preallocated/static block is full used (allocated). That is what the error is about IMO, when loading the library it cannot "reserve" or "allocate" the memory it requires.
On my x64 laptop, I can see libwidevinecdm.so itself is using a much larger TLS area now (96 bytes instead of 24 bytes).
[dag@moria ~]$ readelf -Wl libwidevinecdm.so.x64.old.working | grep TLS TLS 0x321020 0x0000000000322020 0x0000000000322020 0x000010 0x000018 R 0x8 [dag@moria ~]$ readelf -Wl libwidevinecdm.so.x64.new.working | grep TLS TLS 0x9d8080 0x00000000009da080 0x00000000009da080 0x000014 0x000060 R 0x40
Would be nice to see the above for ARM.
there is a preallocated block, but it is being used at runtime by libraries. The error occurs when that preallocated/static block is full used (allocated). That is what the error is about IMO, when loading the library it cannot "reserve" or "allocate" the memory it requires.
I am aware of that, however a simple program that uses no TLS that also fails seems to suggest a different cause. This was demonstrated on ISA issue. Also noteworthy is 1679 was using a static TLS model, where as 2252 is using a dynamic TLS model. This is evident in "readelf -d" by inspecting the "FLAGS".
I think this is a red herring.
Would be nice to see the above for ARM.
I have Raspberry Pi OS Lite running in QEMU (qemu-system-arm), this is the output:
@wagnerch Are you saying that on ChromeOS they changed the behaviour of ld compared to Linux. Maybe fixed the implementation? Then we are a long way from home 😟
@dagwieers There are some other things bothering me:
$ readelf -d libwidevinecdm-4.10.2252.0.so
Dynamic section at offset 0x816c58 contains 34 entries:
Tag Type Name/Value
...
...
0x00000024 (<unknown>: 24) 0x1cf4
0x00000023 (<unknown>: 23) 0x2b8
0x00000025 (<unknown>: 25) 0x4
What are these unknown attributes in the dynamic section? Based on some research this is an Android-specific ELF extension called relative relocations (RELR), see: https://groups.google.com/g/generic-abi/c/bX460iggiKg https://chromium.googlesource.com/chromium/src.git/+/master/third_party/android_crazy_linker/src/src/crazy_linker_relr_relocations.h#12 https://android.googlesource.com/platform/bionic/+/master/libc/include/elf.h#246
There is also a ".relr.dyn" section in the library. Binutils has no idea what it is, one of the tools actually complains about it (I think objcopy, when I tried to strip the .init and .fini sections).
Also, when runninng: LD_PRELOAD=libwidevinecdm.so /bin/ls
And analyzing the core dump:
(gdb) bt
#0 0x003fd6a4 in ?? ()
#1 0xb6f50a48 in call_init (l=<optimized out>, argc=argc@entry=1,
argv=argv@entry=0xbef3fa14, env=env@entry=0xbef3fa1c) at dl-init.c:74
#2 0xb6f50b20 in call_init (env=<optimized out>, argv=<optimized out>,
argc=<optimized out>, l=<optimized out>) at dl-init.c:37
#3 _dl_init (main_map=0xb6f739a0, argc=1, argv=0xbef3fa14, env=0xbef3fa1c)
at dl-init.c:121
#4 0xb6f409a4 in _dl_start_user () from /lib/ld-linux-armhf.so.3
0x3fd6a4 is the virtual address for InitializeCdmModule_4, which is called by the ".init" section.
$ arm-linux-gnueabihf-nm -D libwidevinecdm-4.10.2252.0.so |grep InitializeCdmModule_4
arm-linux-gnueabihf-nm: libwidevinecdm-4.10.2252.0.so: unknown type [0x13] section `.relr.dyn'
003fd760 T InitializeCdmModule_4
Oh, and hey there is nm complaining about an unknown section ".relr.dyn". Interestingly, gdb sees this symbol at: (gdb) print InitializeCdmModule_4 $1 = {<text variable, no debug info>} 0xb61b1760 <InitializeCdmModule_4>
So to me that suggests maybe the relocation was not done, because glibc doesn't understand it? Not sure.
Hi All: How to extract libwidevinecdm-4.10.2252.0.so from ROOT-A.img which is get from chromeos_13816.64.0_veyron-fievel_recovery_stable-channel_fievel-mp.bin??
I get some error: $sudo mount -t ext2 -o loop ROOT-A.img ~/project/data mount: /home/dennis/project/data: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error.
Can you share me libwidevinecdm-4.10.2252.0.so? I can try it on chromium 90....
@cchsu23 I quickly created a script based on the code from inputstreamhelper. Just put it in the same directory as the ChromeOS recovery .bin or .zip and run it. https://gist.github.com/horstle/f3680902708084a11abb6428fb5fcff4
Hello together, thank you for all the work you invested (and still invest) into KODI & inputstreamhelper!!
I thought I interpreted the merged pull request #5376 from kszaq/widevine_support in into the libreELEC master right, when I thought the LibreELEC nightlies later than May 23rd and inputstreamhelper >= 0.5.4 should be able to work with Widevine >=4.10.2252.0 on ARM devices (in my case: RPi3B + libreELEC-RPi2.arm-10.0-nightly-20210527-63cf0d4)...
But I still get the error "OpenDRM system failed" because inputstream adaptive was unable to load the widevine.so lib. Within inputstreamhelper 0.5.4 there is no option to switch back to an older widevine, also removing and reinstalling widevine always installs 4.10.2252.0, taken from the chromeOS Image Tiger 13816.82.0.
I did not test this on my OrangePi PC H3 because it still works fine with the "old" widevine...
Did I misinterpret the merge on LibreELEC or can I help by testing in any way? (I am still not sure if this is the right place to ask... maybe LibreELEC fits better..)? Thank you & best regards, Chris
Can you post the content of /run/libreelec/kodi.conf here? I don't own a RPi myself, so it's very difficult to test the libreELEC nightly images.
...there are only these two lines in /run/libreelec/kodi.conf :
KODI_ARGS=""
MALLOC_MMAP_THRESHOLD_=8192
Thanks for the info. You need to add LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so to make the nightly image work with the newest Widevine.
Okay, it took a while for me as beginner to figure out that changes to /run/libreelec/kodi.conf are never permanent.
But adding a new kodi.conf with LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so inside in the path /storage/.config finally survived the reboot and the LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so is visible in the /run environment.
...and finally: That solved the DRM issue with the newest widevine!
Cool. That made my day. Now widewine works also on the RPI2. Thank you for the quick response and the good work! 👍
... one additional question: Was this an "issue" for me due to my nightly build or is there an additional script or something similar required for the headless installation with KODI and most of the users that do not want to dive into the config files?
Your issue is related to the nightly build, LibreELEC will release a new version 9.2.7 where the preloading is permanently added: https://github.com/LibreELEC/LibreELEC.tv/pull/5399
*** respect to people who are committed to this CeES-atd it works for me too now . Cool
LibreELEC users who are experiencing this problem should use the new 9.2.7 version: https://libreelec.tv/2021/05/le-9-2-7-10b4-fix-widevine/
I guess we should close this issue, or at least modify the title, to not confuse our users...
I would agree to modify the title: ISH 0.5.5 runs fine for me with the LibreElec 10 nightly f547d15 on my RPi3B+ and OrangePi PC but without update on CDM > 4.10.2252.0 we just know it works fine with CDM == 4.10.2252.0.... Maybe "Playback failed using Widevine CDM 4.10.2252.0 on ARM with ISH <= 0.5.2" is lesser indicating persistent trouble with Widevine and the actual ISH & KODI versions. The issue can be closed with the next trouble-free Widevine-update