yapapi
yapapi copied to clipboard
golemfactory
Internal Server Error printed on the console, all tasks computed nevertheless
Found during testing session
Name: AZ linux + python 3.8 + 0.6.0-dev:
blender-yapapi-2021-05-25_12.19.09.log.gz
[2021-05-25 12:19:09,797 INFO yapapi] Using log file `blender-yapapi-2021-05-25_12.19.09.log`; in case of errors look for additional information there
yapapi version: 0.6.0-dev
Using subnet: devnet-beta.1, payment driver: zksync, and network: rinkeby
[2021-05-25 12:19:13,226 INFO yapapi.summary] Received proposals from 1 provider so far
[2021-05-25 12:19:13,253 INFO yapapi.summary] Received proposals from 2 providers so far
[2021-05-25 12:19:13,254 INFO yapapi.summary] Received proposals from 3 providers so far
[2021-05-25 12:19:13,256 INFO yapapi.summary] Received proposals from 4 providers so far
[2021-05-25 12:19:13,258 INFO yapapi.summary] Received proposals from 5 providers so far
[2021-05-25 12:19:13,262 INFO yapapi.summary] Received proposals from 6 providers so far
[2021-05-25 12:19:13,263 INFO yapapi.summary] Received proposals from 7 providers so far
[2021-05-25 12:19:13,264 INFO yapapi.summary] Received proposals from 8 providers so far
[2021-05-25 12:19:13,264 INFO yapapi.summary] Received proposals from 9 providers so far
[2021-05-25 12:19:13,264 INFO yapapi.summary] Received proposals from 10 providers so far
[2021-05-25 12:19:13,265 INFO yapapi.summary] Received proposals from 11 providers so far
[2021-05-25 12:19:13,277 INFO yapapi.summary] Received proposals from 12 providers so far
[2021-05-25 12:19:13,303 INFO yapapi.summary] Received proposals from 13 providers so far
[2021-05-25 12:19:13,374 INFO yapapi.summary] Received proposals from 14 providers so far
[2021-05-25 12:19:13,508 INFO yapapi.summary] Received proposals from 15 providers so far
[2021-05-25 12:19:13,620 INFO yapapi.summary] Received proposals from 16 providers so far
[2021-05-25 12:19:14,441 INFO yapapi.summary] Agreement proposed to provider 'demonic-care-aarch64-host'
[2021-05-25 12:19:14,784 INFO yapapi.summary] Agreement confirmed by provider 'demonic-care-aarch64-host'
[2021-05-25 12:19:15,697 INFO yapapi.summary] Task started on provider 'demonic-care-aarch64-host', task data: 0
[2021-05-25 12:19:16,793 INFO yapapi.summary] Agreement proposed to provider 'omx-testnode1'
[2021-05-25 12:19:17,583 INFO yapapi.summary] Agreement confirmed by provider 'omx-testnode1'
[2021-05-25 12:19:18,577 INFO yapapi.summary] Received proposals from 17 providers so far
[2021-05-25 12:19:18,751 INFO yapapi.summary] Task started on provider 'omx-testnode1', task data: 10
[2021-05-25 12:19:19,593 INFO yapapi.summary] Agreement proposed to provider 'lucas-shuttle'
[2021-05-25 12:19:20,071 INFO yapapi.summary] Agreement confirmed by provider 'lucas-shuttle'
[2021-05-25 12:19:20,983 INFO yapapi.summary] Task started on provider 'lucas-shuttle', task data: 20
Task computed: Task(id=2, data=10), result: output_10.png, time: 0:00:12.986184
[2021-05-25 12:19:31,740 INFO yapapi.summary] Task finished by provider 'omx-testnode1', task data: 10
[2021-05-25 12:19:31,740 INFO yapapi.summary] Task started on provider 'omx-testnode1', task data: 30
Task computed: Task(id=4, data=30), result: output_30.png, time: 0:00:08.751757
[2021-05-25 12:19:40,494 INFO yapapi.summary] Task finished by provider 'omx-testnode1', task data: 30
[2021-05-25 12:19:40,494 INFO yapapi.summary] Task started on provider 'omx-testnode1', task data: 40
Task computed: Task(id=3, data=20), result: output_20.png, time: 0:00:19.985733
[2021-05-25 12:19:40,971 INFO yapapi.summary] Task finished by provider 'lucas-shuttle', task data: 20
[2021-05-25 12:19:40,971 INFO yapapi.summary] Task started on provider 'lucas-shuttle', task data: 50
Task computed: Task(id=6, data=50), result: output_50.png, time: 0:00:08.006346
[2021-05-25 12:19:48,982 INFO yapapi.summary] Task finished by provider 'lucas-shuttle', task data: 50
Task computed: Task(id=5, data=40), result: output_40.png, time: 0:00:08.915189
[2021-05-25 12:19:49,412 INFO yapapi.summary] Task finished by provider 'omx-testnode1', task data: 40
[2021-05-25 12:19:51,463 WARNING yapapi.summary] Worker for provider 'demonic-care-aarch64-host' failed; reason: (500)
Reason: Internal Server Error
HTTP response headers: <CIMultiDictProxy('Content-Length': '311', 'Content-Type': 'application/json', 'Date': 'Tue, 25 May 2021 10:19:51 GMT')>
HTTP response body: {"message":"GSB error: Remote service at `/net/0xfc1bd14392e90b1b8dba0ff58a573ea222f406a1/exeunit/1dccc9e4275b43b192d9e28b054cdadd/GetExecBatchResults` error: GSB failure: GSB failure: Service /public/exeunit/1dccc9e4275b43b192d9e28b054cdadd/GetExecBatchResults call aborted due to its provider disconnection."}
[2021-05-25 12:19:51,464 INFO yapapi.summary] Task started on provider 'lucas-shuttle', task data: 0
[2021-05-25 12:19:52,187 INFO yapapi.summary] Terminated agreement with demonic-care-aarch64-host
[2021-05-25 12:19:52,611 INFO yapapi.summary] Accepted invoice from 'demonic-care-aarch64-host', amount: 0
[2021-05-25 12:19:57,874 INFO yapapi.summary] Received proposals from 17 providers so far
Task computed: Task(id=1, data=0), result: output_0.png, time: 0:00:07.766744
[2021-05-25 12:19:59,235 INFO yapapi.summary] Task finished by provider 'lucas-shuttle', task data: 0
[2021-05-25 12:19:59,235 INFO yapapi.summary] Computation finished in 46.8s
[2021-05-25 12:19:59,235 INFO yapapi.summary] Negotiated 3 agreements with 3 providers
[2021-05-25 12:19:59,235 INFO yapapi.summary] Provider 'omx-testnode1' computed 3 tasks
[2021-05-25 12:19:59,235 INFO yapapi.summary] Provider 'lucas-shuttle' computed 3 tasks
[2021-05-25 12:19:59,236 INFO yapapi.summary] Provider 'demonic-care-aarch64-host' did not compute any tasks
[2021-05-25 12:19:59,236 INFO yapapi.summary] Activity failed 1 time on provider 'demonic-care-aarch64-host'
[2021-05-25 12:20:00,237 INFO yapapi.executor] Waiting for 2 workers to finish...
[2021-05-25 12:20:01,655 INFO yapapi.executor] Waiting for Executor services to finish...
6 tasks computed, total time: 0:00:49.227094
[2021-05-25 12:20:01,656 INFO yapapi.executor] Waiting for Golem services to finish...
[2021-05-25 12:20:01,919 INFO yapapi.summary] Accepted invoice from 'lucas-shuttle', amount: 0.001223194510922222
[2021-05-25 12:20:02,788 INFO yapapi.summary] Accepted invoice from 'omx-testnode1', amount: 0.003459454871433333
[2021-05-25 12:20:02,797 INFO yapapi.summary] Total cost: 0.004682649382355555
[2021-05-25 12:20:02,797 INFO yapapi.summary] Executor has shut downPaste short logs here
demonic-care-aarch64-host is frequently failing, because it's not a x86 machine (other architectures are not officially supported). Its task was later computed by lucas-shuttle, that's why all tasks were computed in the end. I don't think it's a bug.
There is an issue reported after initial Beta.2 provider testing session (2021-05-27) golemfactory/yagna-triage#115
There is GetExecBatchResults is unavailable`
@tworec I don't understand how your comment impacts this issue :)
@filipgolem I thought that yapapi should wrap 500 a little bit nicer not to cause worry on the person looking into the logs. It is displayed in a way which draws attention (multiline, empty line), maybe it should look less frightenning? Maybe the actual reason should be a debug, not a warning? Still - decide how you want to handle this in yapapi and either ammend the code or close this issue :) it is SDK teams call.
