[snips-asr-google][deprecated] snips-asr-google does not work (crashes and hangs)
Google ASR requests are sometimes hanging.
https://forum.snips.ai/t/google-asr-does-not-work-properly-after-update-to-0-61-1/1808
Version Snips Flow 1.1.0 (0.61.1)
How to reproduce Install and set snips-asr-google. Chains the request until the problem occurs
Expected All the requests are correctly taken into account and produce the correct text output (behavior observed on Snips Flow 1.0.3 (0.60.12)
Observed Depending on the platform, 10% to 60% of the requests hangs when google asr starts
Workaround
[EDIT cpoisson 27/09/2019]
Due to some changes on protocol sides, snips-asr-google does not work anymore.
This feature will be deprecated soon as it does not fit with our roadmap. We are working on a plan to remove it totally from the console.
The current recommended workaround is to develop an alternative asr service using google python API.
Fixed on version 1.1.2 (0.62.2)
I have upgraded my Snips platform to 0.62.3 and unfortunately Google ASR still does not work :( and as previously - it does not recognises anything. I can't hear "end of input" sound also.
This one is tricky :/, we could not see the problem after fixing the dependencies. Hopefully the workaround still work.
The way we tested it on our side is by comparing side by side the output of google asr for version 0.62.2 with the version 0.60.12. The behavior of snips-asr-google seemed back to normal to us. Meaning every requests out of 20 were correctly handled with the same output.
Here's come my questions
- Is snips-asr correctly uninstalled (or not running)?
- Does the issue appears immediately or after a while?
- you are running snips-asr-google on a Pi3 right?
- Can you start
snips-asr-google -vmanually on your terminal and provide the output of for few phrase you utter. (just make sure tosudo systemctl stop snips-asr-googlebeforehand)
Meanwhile we will retest the version 0.62.3 of the package.
Thanks for reopening ticket! Unfortunatelly snips-asr-google cant be downgraded to 0.60.12 with snips-platform-common in new version (package requirements), so I had to downgrade all back to 0.61.1.
Ok got it, too bad you had to downgrade :/. I have edited my comments just above with few questions.
Is snips-asr correctly uninstalled (or not running)?
It's uninstalled.
Does the issue appears immediately or after a while?
Immediately
you are running snips-asr-google on a Pi3 right?
Yes. I have RPI3B+ as a master (without a mic/speaker) and 3 satellites (2xRPI0 and RPI3B) with https://www.x-kom.pl/p/217636-glosniki-creative-sound-blaster-axx-sbx-8.html.
Can you start snips-asr-google -v manually on your terminal and provide the output of for few phrase you utter. (just make sure to sudo systemctl stop snips-asr-google beforehand)
This is from downgraded version:
[22:06:57.325518] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.assistant -> Ok(None)
[22:06:57.325827] DEBUG:snips_common_cli::cli: conf lookup: snips-common.assistant -> Ok(None)
[22:06:57.326373] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.credentials -> Ok(None)
[22:06:57.326480] DEBUG:snips_common_cli::cli: conf lookup: snips-common.credentials -> Ok(None)
[22:06:57.326550] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.partial -> Ok(None)
[22:06:57.326615] DEBUG:snips_common_cli::cli: conf lookup: snips-common.partial -> Ok(None)
[22:06:57.326687] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.partial_period_ms -> Ok(None)
[22:06:57.326753] DEBUG:snips_common_cli::cli: conf lookup: snips-common.partial_period_ms -> Ok(None)
[22:06:57.326858] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.bus -> Ok(None)
[22:06:57.326951] DEBUG:snips_common_cli::cli: conf lookup: snips-common.bus -> Ok(None)
[22:06:57.327125] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt -> Ok(None)
[22:06:57.327190] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt -> Ok(None)
[22:06:57.327414] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_username -> Ok(None)
[22:06:57.327486] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_username -> Ok(None)
[22:06:57.327555] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_password -> Ok(None)
[22:06:57.327621] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_password -> Ok(None)
[22:06:57.327692] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_tls_hostname -> Ok(None)
[22:06:57.327758] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_tls_hostname -> Ok(None)
[22:06:57.330328] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.audio -> Ok(Some(["raspberrypi@mqtt", "room@mqtt", "kitchen@mqtt", "bedroom@mqtt"]))
[22:06:57.330578] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.bus -> Ok(None)
[22:06:57.330655] DEBUG:snips_common_cli::cli: conf lookup: snips-common.bus -> Ok(None)
[22:06:57.330727] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt -> Ok(None)
[22:06:57.330860] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt -> Ok(None)
[22:06:57.331044] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_username -> Ok(None)
[22:06:57.331250] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_username -> Ok(None)
[22:06:57.331319] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_password -> Ok(None)
[22:06:57.331416] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_password -> Ok(None)
[22:06:57.331502] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_tls_hostname -> Ok(None)
[22:06:57.331629] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_tls_hostname -> Ok(None)
[22:06:57.334081] INFO :snips_asr_google : Connecting using MQTT site-id raspberrypi
[22:06:57.336262] INFO :snips_asr_google : Connecting using MQTT site-id room
[22:06:57.340500] INFO :snips_asr_google : Connecting using MQTT site-id kitchen
[22:06:57.348972] INFO :snips_asr_google : Connecting using MQTT site-id bedroom
[22:07:00.466005] INFO :snips_asr_google_lib::google_auth: Connecting using service account file "/usr/share/snips/googlecredentials.json"
[22:07:00.467383] DEBUG:snips_asr_google_lib::stream_ext : ["backpressure-thread"] Thread spawned
[22:07:00.466628] DEBUG:snips_asr_google_lib::google_auth: credentials: Credentials {
t: "service_account",
project_id: "assistant-217514",
private_key_id: "",
private_key: "",
client_email: "[email protected]",
client_id: "",
auth_uri: "https://accounts.google.com/o/oauth2/auth",
token_uri: "https://oauth2.googleapis.com/token",
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs",
client_x509_cert_url: ""
}
[22:07:00.467886] DEBUG:snips_asr_google_lib::google_auth: claims: JwtClaims {
iss: "[email protected]",
scope: "https://www.googleapis.com/auth/cloud-platform",
aud: "https://www.googleapis.com/oauth2/v4/token",
exp: 1553206020,
iat: 1553202420
}
[22:07:01.532667] DEBUG:snips_asr_google_lib::google_asr : Recognized results {alternatives {transcript: "kt\303\263ra"} stability: 0.01}
[22:07:01.754484] DEBUG:snips_asr_google_lib::google_asr : Recognized results {alternatives {transcript: "kt\303\263ra godz"} stability: 0.01}
[22:07:01.853624] DEBUG:snips_asr_google_lib::google_asr : Recognized results {alternatives {transcript: "kt\303\263ra godzina"} stability: 0.01}
[22:07:02.036163] DEBUG:snips_asr_google_lib::google_asr : Recognized results {alternatives {transcript: "kt\303\263ra"} stability: 0.9} results {alternatives {transcript: " godzina"} stability: 0.01}
[22:07:02.456658] DEBUG:snips_asr_google_lib::google_asr : Recognized results {alternatives {transcript: "kt\303\263ra godzina"} stability: 0.9}
[22:07:03.160432] DEBUG:snips_asr_google_lib::google_asr : Recognized results {alternatives {transcript: "kt\303\263ra godzina" confidence: 0.9631079} is_final: true}
[22:07:03.160581] DEBUG:snips_asr_google_hermes : Text captured: TextCapturedMessage { text: "która godzina", likelihood: 0.9631079, tokens: None, seconds: 2.0, site_id: "room", session_id: Some("ef66529d-7ac8-4d9d-be68-31975d3c156c") }, is_final: true
[22:07:05.468231] DEBUG:snips_asr_google_lib::stream_ext : ["backpressure-thread"] Thread stop
And this from new one:
[22:12:07.837055] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.assistant -> Ok(None)
[22:12:07.838082] DEBUG:snips_common_cli::cli: conf lookup: snips-common.assistant -> Ok(None)
[22:12:07.839375] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.credentials -> Ok(None)
[22:12:07.839910] DEBUG:snips_common_cli::cli: conf lookup: snips-common.credentials -> Ok(None)
[22:12:07.840273] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.partial -> Ok(None)
[22:12:07.840599] DEBUG:snips_common_cli::cli: conf lookup: snips-common.partial -> Ok(None)
[22:12:07.840940] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.partial_period_ms -> Ok(None)
[22:12:07.841265] DEBUG:snips_common_cli::cli: conf lookup: snips-common.partial_period_ms -> Ok(None)
[22:12:07.841593] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.bus -> Ok(None)
[22:12:07.841908] DEBUG:snips_common_cli::cli: conf lookup: snips-common.bus -> Ok(None)
[22:12:07.842231] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt -> Ok(None)
[22:12:07.842544] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt -> Ok(None)
[22:12:07.843067] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_username -> Ok(None)
[22:12:07.843411] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_username -> Ok(None)
[22:12:07.843801] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_password -> Ok(None)
[22:12:07.844126] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_password -> Ok(None)
[22:12:07.844466] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_tls_hostname -> Ok(None)
[22:12:07.844809] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_tls_hostname -> Ok(None)
[22:12:07.849822] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.audio -> Ok(Some(["raspberrypi@mqtt", "room@mqtt", "kitchen@mqtt", "bedroom@mqtt"]))
[22:12:07.850637] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.bus -> Ok(None)
[22:12:07.851100] DEBUG:snips_common_cli::cli: conf lookup: snips-common.bus -> Ok(None)
[22:12:07.851448] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt -> Ok(None)
[22:12:07.851818] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt -> Ok(None)
[22:12:07.852293] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_username -> Ok(None)
[22:12:07.852639] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_username -> Ok(None)
[22:12:07.853043] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_password -> Ok(None)
[22:12:07.853368] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_password -> Ok(None)
[22:12:07.853706] DEBUG:snips_common_cli::cli: conf lookup: snips-asr-google.mqtt_tls_hostname -> Ok(None)
[22:12:07.854086] DEBUG:snips_common_cli::cli: conf lookup: snips-common.mqtt_tls_hostname -> Ok(None)
[22:12:07.858853] INFO :snips_asr_google : Connecting using MQTT site-id raspberrypi
[22:12:07.860312] INFO :snips_asr_google : Connecting using MQTT site-id room
[22:12:07.862946] INFO :snips_asr_google : Connecting using MQTT site-id kitchen
[22:12:07.866149] INFO :snips_asr_google : Connecting using MQTT site-id bedroom
[22:12:11.677862] INFO :snips_asr_google_lib::google_auth: Connecting using service account file "/usr/share/snips/googlecredentials.json"
[22:12:11.678681] DEBUG:snips_asr_google_lib::stream_ext : ["backpressure-thread"] Thread spawned
[22:12:11.679083] DEBUG:snips_asr_google_lib::google_auth: credentials: Credentials {
t: "service_account",
project_id: "assistant-217514",
private_key_id: "",
private_key: "",
client_email: "[email protected]",
client_id: "",
auth_uri: "https://accounts.google.com/o/oauth2/auth",
token_uri: "https://oauth2.googleapis.com/token",
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs",
client_x509_cert_url: ""
}
[22:12:11.679973] DEBUG:snips_asr_google_lib::google_auth: claims: JwtClaims {
iss: "[email protected]",
scope: "https://www.googleapis.com/auth/cloud-platform",
aud: "https://www.googleapis.com/oauth2/v4/token",
exp: 1553206331,
iat: 1553202731
}
[22:12:17.277633] DEBUG:snips_asr_google_lib::stream_ext : ["backpressure-thread"] Thread stop
I can also observe the issue on MacOS, it's back.
Keep you posted on our investigations.
@cpoisson Since the latest update did not fix google asr and the update updated snips-platform-common to 0.62.3, we can't downgrade google asr component anymore as it requires < 0.61.12. Resulting in a nice: nothing to do if you made the mistake to update snips to latest..... as @ChrisB85 reported... Just saw his post a few answers above. Nice, have to fully downgrade 8 devices....
@Psychokiller1888 @ChrisB85
I found a workaround to re install the package without having to downgrade the whole thing.
The method comes with some caveats:
- It may break your apt configuration, before apt update you may need to uninstall the snips-asr-google and reinstall it afterward
- I have tested it on an assistant, seems ok, cannot guarantee 100% it may not have unwanted side effects.
apt download snips-asr-google=0.60.12
sudo dpkg -i --force-all snips-asr-google_0.60.12_armhf.deb
I hope it will help, sorry for the troubles.
@cpoisson thanks for info. Currently all my satellites are running snips-audio-server in latest version, and master is downgraded to 0.61.1/0.60.12. Everything works fine so I will just wait for next release.
@cpoisson currently using your method of downgrading snips-asr-google to 0.60.12, thanks for that. Are you working on a proper fix of this, and when is this scheduled to be solved?
@cpoisson How come this is not yet fixed??
@cpoisson Hallo? It's been 2 versions released that this is bugged. It's not "hangs randomly" anymore, it's "does not work at all" now
@Psychokiller1888
It is scoped for the next release. It is not an easy one.
Meanwhile the workaround should work right?
Hello.
My 2 cents : in my case, it's not working at all (no hanging or anything, snips-asr-google just stops right after a wake word.
I know you are working on this @cpoisson, I just wanted to share the fact it just shut down immediatly.
[23:54:16.566544] INFO :snips_asr_google : Connecting using MQTT site-id default
[23:54:19.697990] INFO :snips_asr_google_lib::google_auth: Connecting using service account file "/usr/share/snips/googlecredentials.json"
[23:54:19.698058] DEBUG:snips_asr_google_lib::stream_ext : ["backpressure-thread"] Thread spawned
[23:54:19.702402] DEBUG:snips_asr_google_lib::google_auth: credentials: Credentials {
t: "service_account",
project_id: "",
private_key_id: "",
private_key: "",
client_email: "",
client_id: "",
auth_uri: "https://accounts.google.com/o/oauth2/auth",
token_uri: "https://accounts.google.com/o/oauth2/token",
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs",
client_x509_cert_url: ""
}
[23:54:19.703410] DEBUG:snips_asr_google_lib::google_auth: claims: JwtClaims {
iss: "",
scope: "https://www.googleapis.com/auth/cloud-platform",
aud: "https://www.googleapis.com/oauth2/v4/token",
exp: 1559170459,
iat: 1559166859
}
[23:54:20.064290] WARN :httpbis::common::pump_stream_to_write_loop: stream 1 dead
[23:54:20.064383] ERROR:rumqtt::client : snips-asr-google|787-raspberrypi-3 Disconnected: (Error(SyncMpsc(Disconnected), State { next_error: None, backtrace: None }))
[23:54:20.064503] ERROR:snips_common_cli::cli : Unable to reconnect to the mqtt broker, ending the program.
$ sam status
Connected to device 192.168.0.2
OS version ................... Raspbian GNU/Linux 9 (stretch)
Installed assistant .......... Assistant1
Language ..................... fr
Hotword ...................... hey_snips
ASR engine ................... google
Status ....................... Live
Service status:
snips-analytics .............. (not running)
snips-asr .................... 0.63.2 (not running)
snips-audio-server ........... 0.63.2 (running)
snips-dialogue ............... 0.63.2 (running)
snips-hotword ................ 0.63.2 (running)
snips-nlu .................... 0.63.2 (running)
snips-skill-server ........... 0.63.2 (running)
snips-tts .................... 0.63.2 (running)
BTW, I don't see snips-asr-google in the list and no way to find proper documentation about how to properly install this (and make it work). I just assumed I had to apt i the package so I did, but is there some doc ? Sorry if I missed it somewhere.
Edit : obviously found it just after posting , but still, snips-asr-google does not appear in sam status. Normal ?
When is this release planned?
When is this release planned?
The question is not when the release is planned, but when this will be fixed? This ticket is still opened and I'm already thinking that Snips team has no interest to fix it becouse they are working on Snips ASR.
Well, if snips-asr would be able to also cover free texts so that finally use-cases like shopping lists, todo lists, wikipedia (.... you name it ....) are feasible I would also be fine. That would be the real use cases for me. But currently it's stuck as snips-asr doesn't support it and google doesn't work.
@gjdass Thanks for your feedbacks
Yes it is normal that snips-asr-google does not appear in SAM, it is a check list statically defined in which snips-asr-google is not included.
To install it sudo apt install snips-asr-google is the way to go (when working correctly)
A workaround is to install manually the debian package in version 0.60.12 as stated in the workaround
The cons is that it break a bit the dependencies of apt.
A more isolated way to go is to have another Pi (maybe a pi2 could work or a docker could work) and install only snips-asr-google 0.60.12 on it and configure it to connect to the mqtt broker of your other snips services.
@cburghardt @ChrisB85
Yes, to be transparent with you guys, it is not planned at the moment, we have other subject on our hands at the moment. There is a blind fix linked to a side issues tough. I'll test that and keep you posted if it fix snips-asr-google.
Free text is ongoing (no release date to provide yet)
We have made an asr service that is capable of deciding to stay offline and online on demand, using different asr services. The downside it's only Project Alice compatible. My problem is that we also had to make our own hotword service to overcome the inability to train custom hotwords, and I have finished a week ago to create our own dialog manager that holds dialog info instead of loosing them at each dialog part and that is able to respawn sessions. I totally understand things are not on your top list @cpoisson but what you seem to fail understanding is that everything started with hackers like me. Waiting on a simple fix for month for an asr problem is not very acceptable to say the least and you end up losing people or having alternatives spawning to overcome the shortage.
I fully agree with @Psychokiller1888 . I have meanwhile created a skill which queries Google for my todo list intent. That works but is of course again a workaround. It is awesome that Snips has an architecture that allows for such hacks. But what I can't understand is how on earth you can accept to have a broken service for several releases and simply stating it is not a priority. Either fix it or drop Google completely.
Or update the generic asr package...
Hello guys,
I need a bit of help here,
I'm currently testing snips-asr-google (again) comparing the execution of the current version in production (1.2.2 (0.63.3)) against an internal beta version.
The thing is, I cannot reproduce the bug anymore 🤔, The problem usually occurs after 20 tries, but here after more than 100 tries, I don't see the component becoming totally unreliable.
There is few events tough were the asr seems to not pickup that I stopped talking to it, but it is does not get in the flow so often.
I am running it on a Pi3 using raspbian stretch and freshly updated packages.
Can someone confirm me this behavior on their side? I'd like to confirm that I'm not the only one to experience this.
FYI @Psychokiller1888 @cburghardt @ChrisB85
For me it wasn't after 20 tries, but almost instantly, after the first or second. I cannot test unfortunately for the moment, we have built our own Google ASR. Will flash an SD a bit later and try the snips component
You built your Google asr module 👍
(I saw recently the comments above)
About the backlog priorities, I think there is a misunderstanding, I m not the one in charge deciding that by myself 😅.
But yeah, this half broken situation is clearly not something I'm comfortable with :/, we should deprecate this feature properly.
Recently I have build docker image with Snips (https://github.com/ChrisB85/snips_docker) because I moved from Raspberry Pi to Ubuntu server and forgot to hold packages on 0.61.1. The result was the same - Google ASR wasn’t working properly.
I have meanwhile build my own integration as I don't want to sit and wait until this issue might be fixed
Yeah, this is dragging since too long and feels like a let down. Remove the support fully or fix it, I mean it takes literally no time to fix, just update the api and no time to get rid of the whole thing and force the user base to use only snips asr or create alternatives
I have retested it yesterday. It does not work anymore. At least now it crashes with more useful logs.
We may have figured out why (bad HTTP2 implementation in a rust 3rd party library)
For you guys relying on google python library, you have a more reliable solution at hand as those low level libraries are directly maintained by google and save you the burden to deal with HTTP/2, gRPC and so on.
@cpoisson The error you have mentioned started showing from couple of days also in 0.60.12 which i'm using and was working fine:
Sep 18 18:46:29 0f0d8371995a snips-asr-google[16852]: INFO:snips_asr_google_lib::google_auth: Connecting using service account file "/usr/share/snips/googlecredentials.json" Sep 18 18:46:29 0f0d8371995a snips-asr-google[16852]: ERROR:snips_asr_google_lib::google_asr : Authentication error: Error: HTTP/2 Error: Encountered HTTP named error Sep 18 18:46:29 0f0d8371995a snips-asr-google[16852]: ERROR:rumqtt::client : snips-asr-google|16852-0f0d8371995a-3 Disconnected: (Error(SyncMpsc(Disconnected), State { next_error: None, backtrace: None })) Sep 18 18:46:29 0f0d8371995a snips-asr-google[16852]: ERROR:snips_asr_google : Something went wrong in Google ASR! Sep 18 18:46:29 0f0d8371995a snips-asr-google[16852]: ERROR:rumqtt::client : snips-asr-google|16852-0f0d8371995a-1 Disconnected: (Error(SyncMpsc(Disconnected), State { next_error: None, backtrace: None })) Sep 18 18:46:29 0f0d8371995a systemd[1]: snips-asr-google.service: Main process exited, code=exited, status=1/FAILURE Sep 18 18:46:29 0f0d8371995a systemd[1]: snips-asr-google.service: Failed with result 'exit-code'. Sep 18 18:46:34 0f0d8371995a systemd[1]: snips-asr-google.service: Service hold-off time over, scheduling restart.
It didn't occured earlier, so unfortunately it's not the same bug that issue is about.
At the moment my Snips is deaf :(