spread icon indicating copy to clipboard operation
spread copied to clipboard

task failure due to reboot request time out should print off all available logs up to that point

Open anonymouse64 opened this issue 5 years ago • 0 comments

Currently when debugging tests where the system is rebooted and it fails to come back after a reboot, we receive no information about the text execution like we would if the execute section had failed, etc. See for example this failure:


sergio@cachiomachine:~/workspace/snapcore/snapd$ spread external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:04:32 Project content is packed for delivery (4.66MB).
2020-10-02 16:04:32 If killed, discard servers with: spread -reuse-pid=233681 -discard
2020-10-02 16:04:32 Allocating external:ubuntu-core-20-64...
2020-10-02 16:04:32 Waiting for external:ubuntu-core-20-64 to make SSH available at localhost:8022...
2020-10-02 16:04:32 Allocated external:ubuntu-core-20-64.
2020-10-02 16:04:32 Connecting to external:ubuntu-core-20-64...
2020-10-02 16:04:32 Connected to external:ubuntu-core-20-64 at localhost:8022.
2020-10-02 16:04:32 Sending project content to external:ubuntu-core-20-64...
2020-10-02 16:04:33 Preparing external:ubuntu-core-20-64 (external:ubuntu-core-20-64)...
2020-10-02 16:04:33 Preparing external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64)...
2020-10-02 16:07:20 Preparing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64)...
2020-10-02 16:07:22 Executing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) (1/1)...
2020-10-02 16:08:56 Rebooting on external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) as requested...
2020-10-02 16:13:12 Rebooting on external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) as requested...
2020-10-02 16:18:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Current output:
-----
mkdir: cannot create directory ‘/home/gopath’: File exists
sudo: unable to resolve host ubuntu: Temporary failure in name resolution
-----
.
2020-10-02 16:23:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:28:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:33:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:38:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:43:11 Error executing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : kill-timeout reached after external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) reboot request
2020-10-02 16:43:11 Error debugging external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : ssh: unexpected packet in response to channel open: <nil>
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64 (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64 (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Discarding external:ubuntu-core-20-64...
2020-10-02 16:43:11 Successful tasks: 0
2020-10-02 16:43:11 Aborted tasks: 0
2020-10-02 16:43:11 Failed tasks: 1
    - external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:43:11 Failed task restore: 1
    - external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:43:11 Failed suite restore: 1
    - external:ubuntu-core-20-64:tests/core/
2020-10-02 16:43:11 Failed project restore: 1
    - external:ubuntu-core-20-64:project
error: unsuccessful run

The only information we get is the "is taking too long Current Output" message which is not enough information, and additionally the debug section is not executed.

anonymouse64 avatar Oct 05 '20 13:10 anonymouse64