op-test
op-test copied to clipboard
open-power
Facing ipmi login issue for Baremetal OpenBMC system
ipmi login failing for OpenBMC machine
Error Log:
# ./op-test --run testcases.RunHostTest.RunHostTest -c ./machine_test.conf
Logs in: /var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/test-reports/test-run-20240610021931
2024-06-10 02:19:31,178:op-test:setUpLoggerFile:INFO:Preparing to set location of Log File to /var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/test-reports/test-run-20240610021931/20240610071931178050.main.log
2024-06-10 02:19:31,178:op-test:setUpLoggerFile:INFO:Log file: /var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/test-reports/test-run-20240610021931/20240610071931178050.main.log
2024-06-10 02:19:31,178:op-test:setUpLoggerDebugFile:INFO:Preparing to set location of Debug Log File to /var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/test-reports/test-run-20240610021931/20240610071931178541.debug.log
2024-06-10 02:19:31,178:op-test:setUpLoggerDebugFile:INFO:Debug Log file: /var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/test-reports/test-run-20240610021931/20240610071931178541.debug.log
2024-06-10 02:19:31,179:op-test:do_testing_setup:INFO:TestCase Log files: /var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/test-reports/test-run-20240610021931/*
2024-06-10 02:19:31,179:op-test:do_testing_setup:INFO:StreamHandler setup normal
2024-06-10 02:19:37,782:op-test:__init__:INFO:Running Local HMC tests
runTest (testcases.OpTestFlash.FSPFWImageFLASH) ... skipped 'FSP In-band firmware Update test'
runTest (testcases.OpTestFlash.BmcImageFlash) ... skipped 'BMC image not provided, so skipping test'
runTest (testcases.OpTestFlash.PNORFLASH) ... skipped 'PNOR image not provided, so skipping test'
runTest (testcases.OpTestFlash.OpalLidsFLASH) ... skipped 'No custom skiboot/kernel to flash'
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK (skipped=4)
runTest (testcases.RunHostTest.RunHostTest) ... command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
2024-06-10 02:19:39,999:op-test.common.OpTestUtil:check_nvram_options:INFO:No NVRAM utility available to check options
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
2024-06-10 02:19:42,207:op-test.common.OpTestUtil:check_nvram_options:INFO:No NVRAM utility available to check options
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
ERROR
======================================================================
ERROR: runTest (testcases.RunHostTest.RunHostTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/testcases/RunHostTest.py", line 52, in setUp
self.console_thread = OpSOLMonitorThread(1, "console")
File "/var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/common/OpTestSOL.py", line 63, in __init__
self.c = self.system.console.get_console(logger=self.logger)
File "/var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/common/OpTestSSH.py", line 205, in get_console
self.connect(logger)
File "/var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/common/OpTestSSH.py", line 185, in connect
self.check_set_term()
File "/var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/common/OpTestSSH.py", line 196, in check_set_term
self.util.setup_term(self.system, self.pty,
File "/var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/common/OpTestUtil.py", line 1680, in setup_term
raise ConsoleSettings(before=pty.before, after=pty.after,
common.Exceptions.ConsoleSettings: Setting the prompt or logging in for the console was not successful, check credentials and review the following for more details
Expect Before Buffer="
~."
Expect After Buffer="<class 'pexpect.exceptions.EOF'>"
Message="Getting login and sudo not successful, probably connection or credential issue, retry"
----------------------------------------------------------------------
Ran 1 test in 6.629s
FAILED (errors=1)
2024-06-10 02:19:46,419:op-test:<module>:INFO:Exit with Result errors="1" and failures="0"
2024-06-10 02:19:50,445:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Starting to Gather BMC logs
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
2024-06-10 02:19:54,862:op-test.common.OpTestUtil:cleanup:WARNING:OpTestSystem encountered a problem cleaning up BMC /tmp files, you may want to check.
2024-06-10 02:19:54,862:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Completed Gathering BMC logs, find them in the Log Location
2024-06-10 02:19:54,862:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Starting to Gather ESEL's
2024-06-10 02:19:55,686:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Completed Gathering ESEL's, find them in the Log Location
2024-06-10 02:19:55,686:op-test.common.OpTestUtil:dump_versions:INFO:Log Location: /var/www/html/ltckvmci/nodes/ltc-wspoon14/op-test/test-reports/test-run-20240610021931/*debug*
2024-06-10 02:19:55,686:op-test.common.OpTestUtil:dump_versions:INFO:
----------------------------------------------------------
OpTestSystem Firmware Versions Tested
(if flashed things like skiboot.lid, may not be accurate)
----------------------------------------------------------
None
----------------------------------------------------------
----------------------------------------------------------
Manual Login through ipmi is sucessful
# ipmitool -I lanplus -H <bmc-ip> -U <bmc-user> -P <bmc-password> sol activate
Unable to Get Channel Cipher Suites
[SOL Session operational. Use ~? for help]
ltc-wspoon14 login: <host-user>
Password: <host-password>
Last login: Thu Jun 6 03:54:42 from <ip>
machine_test.conf file for the system
[op-test]
bmc_type=OpenBMC
bmc_ip=<bmc-ip>
bmc_username=<bmc-user>
bmc_usernameipmi=<bmc-user>
bmc_password=<bmc-password>
bmc_passwordipmi=<bmc-password>
host_ip=<host-ip>
host_user=<host-user>
host_password=<host-password>
host_cmd_timeout=360000
host_cmd_file=./test.conf
machine_state=OS
@misanjumn @harihare , seems the system has disabled rest API, but when in conf file OpenBMC given as BMC type so it will go and try to login using rest API , so if system connecting with IPMI please use FSP as BMC Type and let me know result
@PraveenPenguin
I tried using FSP as bmc_type, and it did not work. But using SMC as bmc_type worked!
Now I'm able to run the following optest command with no issues
# ./op-test --run testcases.RunHostTest.RunHostTest -c ./machine_test.conf
However, th optest command for Boot to OS is still throwing the error even with SMC as the bmc_type
./op-test --run testcases.BasicIPL.BootToOS -c ./machine_upstreamkernelbuild.conf
runTest (testcases.BasicIPL.BootToOS) ... 2024-07-21 05:05:09,453:op-test.common.OpTestSystem:run_POWERING_OFF:INFO:System is in standby/Soft-off state
Unable to Get Channel Cipher Suites
> Error: no response from RAKP 1 message
Error: Received an Unexpected RAKP 2 message
> Error: no response from RAKP 1 message
Error: Received an Unexpected RAKP 2 message
> Error: no response from RAKP 1 message
Error: Received an Unexpected RAKP 2 message
> Error: no response from RAKP 1 message
Get HPM.x Capabilities request failed, compcode = d4
ERROR
======================================================================
ERROR: runTest (testcases.BasicIPL.BootToOS)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/jenkins/workspace/kvmci/common/OpTestSystem.py", line 721, in run_IPLing
petit_r, petit_reconnect = self.wait_for_it(expect_dict=self.petitboot_expect_table, reconnect=self.petitboot_reconnect,
File "/data/jenkins/workspace/kvmci/common/OpTestSystem.py", line 540, in wait_for_it
sys_pty = self.console.connect()
File "/data/jenkins/workspace/kvmci/common/OpTestIPMI.py", line 310, in connect
raise CommandFailed('sol activate',
common.Exceptions.CommandFailed: Command 'sol activate' exited with '-1'.
Output
IPMI: pexpect.TIMEOUT while trying to establish connection, command was 'ipmitool -H ltc-wspoon14-bmc.aus.stglabs.ibm.com -I lanplus -U root -P S0undCraftUI24r sol activate'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/jenkins/workspace/kvmci/testcases/BasicIPL.py", line 170, in runTest
self.cv_SYSTEM.goto_state(OpSystemState.OS)
File "/data/jenkins/workspace/kvmci/common/OpTestSystem.py", line 389, in goto_state
self.state = self.stateHandlers[self.state](state)
File "/data/jenkins/workspace/kvmci/common/OpTestSystem.py", line 745, in run_IPLing
raise my_exception
common.Exceptions.UnknownStateTransition: Something happened system state="2" and we transitioned to UNKNOWN state. Review the following for more details
Message="OpTestSystem in run_IPLing and the Exception=
"Command 'sol activate' exited with '-1'.
Output
IPMI: pexpect.TIMEOUT while trying to establish connection, command was 'ipmitool -H ltc-wspoon14-bmc.aus.stglabs.ibm.com -I lanplus -U root -P S0undCraftUI24r sol activate'"
caused the system to go to UNKNOWN_BAD and the system will be stopping."
ah ok it turn as Super micro system @misanjumn thanks for trying suggestion , please close this issue
@misanjumn can you paste what test you are running here. test.conf info host_cmd_file=./test.conf
Latest Logs
OS: Fedora40
(Note: Manual ipmi login command working in the jenkins system)
machine_test.conf
# cat machine_test.conf
[op-test]
bmc_type=OpenBMC
bmc_ip=<bmc-ip>
bmc_username=<bmc-user>
bmc_usernameipmi=<bmc-user>
bmc_password=<bmc-password>
bmc_passwordipmi=<bmc-password>
host_ip=<host-ip>
host_user=<host-user>
host_password=<host-password>
host_cmd_timeout=360000
host_cmd_file=./test.conf
machine_state=OS
test.conf file
# cat test.conf
echo 'Testing the machine using the inputs provided...'
uname -a
date
echo 'Working fine ...'
Using OpenBMC as machine_type, this is the following error
# ./op-test --run testcases.RunHostTest.RunHostTest -c ./machine_test.conf
Logs in: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826021227
2024-08-26 02:12:27,260:op-test:setUpLoggerFile:INFO:Preparing to set location of Log File to /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826021227/20240826071227260063.main.log
2024-08-26 02:12:27,260:op-test:setUpLoggerFile:INFO:Log file: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826021227/20240826071227260063.main.log
2024-08-26 02:12:27,260:op-test:setUpLoggerDebugFile:INFO:Preparing to set location of Debug Log File to /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826021227/20240826071227260523.debug.log
2024-08-26 02:12:27,261:op-test:setUpLoggerDebugFile:INFO:Debug Log file: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826021227/20240826071227260523.debug.log
2024-08-26 02:12:27,261:op-test:do_testing_setup:INFO:TestCase Log files: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826021227/*
2024-08-26 02:12:27,261:op-test:do_testing_setup:INFO:StreamHandler setup normal
2024-08-26 02:12:33,758:op-test:__init__:INFO:Running Local HMC tests
runTest (testcases.OpTestFlash.FSPFWImageFLASH) ... skipped 'FSP In-band firmware Update test'
runTest (testcases.OpTestFlash.BmcImageFlash) ... skipped 'BMC image not provided, so skipping test'
runTest (testcases.OpTestFlash.PNORFLASH) ... skipped 'PNOR image not provided, so skipping test'
runTest (testcases.OpTestFlash.OpalLidsFLASH) ... skipped 'No custom skiboot/kernel to flash'
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK (skipped=4)
runTest (testcases.RunHostTest.RunHostTest) ... command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
2024-08-26 02:12:35,975:op-test.common.OpTestUtil:check_nvram_options:INFO:No NVRAM utility available to check options
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
2024-08-26 02:12:38,183:op-test.common.OpTestUtil:check_nvram_options:INFO:No NVRAM utility available to check options
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
ERROR
======================================================================
ERROR: runTest (testcases.RunHostTest.RunHostTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/testcases/RunHostTest.py", line 52, in setUp
self.console_thread = OpSOLMonitorThread(1, "console")
File "/var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/common/OpTestSOL.py", line 63, in __init__
self.c = self.system.console.get_console(logger=self.logger)
File "/var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/common/OpTestSSH.py", line 205, in get_console
self.connect(logger)
File "/var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/common/OpTestSSH.py", line 185, in connect
self.check_set_term()
File "/var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/common/OpTestSSH.py", line 196, in check_set_term
self.util.setup_term(self.system, self.pty,
File "/var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/common/OpTestUtil.py", line 1680, in setup_term
raise ConsoleSettings(before=pty.before, after=pty.after,
common.Exceptions.ConsoleSettings: Setting the prompt or logging in for the console was not successful, check credentials and review the following for more details
Expect Before Buffer="
~."
Expect After Buffer="<class 'pexpect.exceptions.EOF'>"
Message="Getting login and sudo not successful, probably connection or credential issue, retry"
----------------------------------------------------------------------
Ran 1 test in 6.630s
FAILED (errors=1)
2024-08-26 02:12:42,395:op-test:<module>:INFO:Exit with Result errors="1" and failures="0"
2024-08-26 02:12:46,438:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Starting to Gather BMC logs
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
~.command-line line 0: Unsupported option "afstokenpassing"^M
^M~.
2024-08-26 02:12:50,855:op-test.common.OpTestUtil:cleanup:WARNING:OpTestSystem encountered a problem cleaning up BMC /tmp files, you may want to check.
2024-08-26 02:12:50,855:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Completed Gathering BMC logs, find them in the Log Location
2024-08-26 02:12:50,856:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Starting to Gather ESEL's
2024-08-26 02:12:51,704:op-test.common.OpTestUtil:cleanup:INFO:OpTestSystem Completed Gathering ESEL's, find them in the Log Location
2024-08-26 02:12:51,705:op-test.common.OpTestUtil:dump_versions:INFO:Log Location: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826021227/*debug*
2024-08-26 02:12:51,705:op-test.common.OpTestUtil:dump_versions:INFO:
----------------------------------------------------------
OpTestSystem Firmware Versions Tested
(if flashed things like skiboot.lid, may not be accurate)
----------------------------------------------------------
None
----------------------------------------------------------
----------------------------------------------------------
@PraveenPenguin @abdhaleegit Thank you for the solution.
Login successful after adding the following changes in the ssh config file
# cat .ssh/config
Host *bmc*
HostKeyAlgorithms=ssh-rsa
PubkeyAcceptedKeyTypes +ssh-rsa
Test Run
# ./op-test --run testcases.RunHostTest.RunHostTest -c ./machine_test.conf
Logs in: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826040312
2024-08-26 04:03:12,601:op-test:setUpLoggerFile:INFO:Preparing to set location of Log File to /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826040312/20240826090312601292.main.log
2024-08-26 04:03:12,601:op-test:setUpLoggerFile:INFO:Log file: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826040312/20240826090312601292.main.log
2024-08-26 04:03:12,601:op-test:setUpLoggerDebugFile:INFO:Preparing to set location of Debug Log File to /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826040312/20240826090312601758.debug.log
2024-08-26 04:03:12,602:op-test:setUpLoggerDebugFile:INFO:Debug Log file: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826040312/20240826090312601758.debug.log
2024-08-26 04:03:12,602:op-test:do_testing_setup:INFO:TestCase Log files: /var/www/html/ltckvmci/nodes/ltcwspoon14/op-test/test-reports/test-run-20240826040312/*
2024-08-26 04:03:12,602:op-test:do_testing_setup:INFO:StreamHandler setup normal
2024-08-26 04:03:19,182:op-test:__init__:INFO:Running Local HMC tests
runTest (testcases.OpTestFlash.FSPFWImageFLASH) ... skipped 'FSP In-band firmware Update test'
runTest (testcases.OpTestFlash.BmcImageFlash) ... skipped 'BMC image not provided, so skipping test'
runTest (testcases.OpTestFlash.PNORFLASH) ... skipped 'PNOR image not provided, so skipping test'
runTest (testcases.OpTestFlash.OpalLidsFLASH) ... skipped 'No custom skiboot/kernel to flash'
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK (skipped=4)
runTest (testcases.RunHostTest.RunHostTest) ... command-line line 0: Unsupported option "afstokenpassing"^M
[console-expect]#which bash && exec bash --norc --noprofile
PS1=\[console-expect\]#
which bash && exec bash --norc --noprofile
/usr/bin/bash
bash-5.2# PS1=\[console-expect\]#
[console-expect]#which stty && stty cols 300;which stty && stty rows 30
which stty && stty cols 300;which stty && stty rows 30
/usr/bin/stty
/usr/bin/stty
[console-expect]#export LANG=C
export LANG=C
[console-expect]#
[console-expect]#date
date
Mon Aug 26 04:03:52 CDT 2024
Closing this issue now.