VirtualClient
VirtualClient copied to clipboard
microsoft
Various changes to enable FIO and PostgreSQL tests in (Docker/ACI) container
Added a DummyDiskManager that just returns one root disk. However this is a TODO item - we want to make it use df to determine any additional volumes mounted to the container, and report correct filesystem size (makes more sense in ACI where it's the sandbox VHD's size), as well as any mounted volumes (need to experiment to see how volumes actually turn up in containers / ACI)
Test command:
./VirtualClient --profile=PERF-IO-FIO.json --profile=MONITORS-NONE.json --iterations=1 --ltf --log-level=Trace --pm FileSize=1G --scenarios -DiskFill --packages=https://virtualclient.blob.core.windows.net/packages
./VirtualClient --profile=PERF-POSTGRESQL-HAMMERDB-TPCC.json --profile=MONITORS-NONE.json --iterations=1 --ltf --log-level=Trace --pm DiskFilter=osdisk:true --packages=https://virtualclient.blob.core.windows.net/packages --pm WarehouseCount=40 --pm SharedMemoryBuffer=2000
For some reason this currently still fails in Docker (testing TODO in ACI):
[02/11/2025 21:35:55] Executing: python3 /VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py --dbName hammerdb_tpcc --serverIp 127.0.0.1 --password ... --port 5432 --inMemory 2000
[02/11/2025 21:35:55] PostgreSQLServerConfiguration.PostgreSQLServerConfiguration.ProcessDetails
[02/11/2025 21:35:55] could not change directory to "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64": Permission denied
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
Traceback (most recent call last):
File "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py", line 49, in <module>
subprocess.run(f'sudo -u postgres psql -c "ALTER USER postgres PASSWORD \'{password}\';"', shell=True, check=True)
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'p5P4ospdxIF4LU7P+fWIWFvrnUeoJ8paCAXlZ7O1TgY=';"' returned non-zero exit status 2 (error/exit code=1, command=python3 /VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py --dbName hammerdb_tpcc --serverIp 127.0.0.1 --password p5P4ospdxIF4LU7P+fWIWFvrnUeoJ8paCAXlZ7O1TgY= --port 5432 --inMemory 2000).
StandardOutput: Copying pg_hba.conf file to change authentication to peer to allow user authentication
Restarting the PostgreSQL services on the system.
StandardError: could not change directory to "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64": Permission denied
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
Traceback (most recent call last):
File "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py", line 49, in <module>
subprocess.run(f'sudo -u postgres psql -c "ALTER USER postgres PASSWORD \'{password}\';"', shell=True, check=True)
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'p5P4ospdxIF4LU7P+fWIWFvrnUeoJ8paCAXlZ7O1TgY=';"' returned non-zero exit status 2.
at VirtualClient.ProcessExtensions.ThrowErrored[TError](IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 385
at VirtualClient.ProcessExtensions.ThrowIfErrored[TError](IProcessProxy process, IEnumerable`1 successCodes, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 228
at VirtualClient.ProcessExtensions.ThrowIfDependencyInstallationFailed(IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 180
at VirtualClient.Dependencies.PostgreSQLServerConfiguration.ConfigurePostgreSQLServerAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Dependencies/PostgreSQLServer/PostgreSQLServerConfiguration.cs:line 196
at VirtualClient.Dependencies.PostgreSQLServerConfiguration.ExecuteAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Dependencies/PostgreSQLServer/PostgreSQLServerConfiguration.cs:line 160
at VirtualClient.Contracts.VirtualClientComponent.<>c__DisplayClass130_1.<<ExecuteAsync>b__4>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientComponent.cs:line 659
--- End of stack trace from previous location ---
at VirtualClient.Contracts.VirtualClientLoggingExtensions.<>c__DisplayClass17_0.<<LogMessageAsync>b__0>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 603
--- End of stack trace from previous location ---
at VirtualClient.Contracts.VirtualClientLoggingExtensions.LogMessageAsync[TResult](ILogger logger, String message, LogLevel level, EventContext eventContext, Func`1 body) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 641
But works on WSL using DOTNET_RUNNING_IN_CONTAINER=true
Not written any unit tests at this point, nor have I tested if the profile still works in a VM (I think it should)
Note that postgresql still doesn't work "out of the box", and we need a mock systemctl in the container image. However I decided to not bake that into VirtualClient - maybe the configure scripts in the package need to be updated.
#!/usr/bin/bash
if [ "$#" -ne 2 ]; then
echo "Usage: $0 {start|restart|stop} postgresql"
exit 1
fi
action="$1"
service="$2"
if [ "$service" != "postgresql" ]; then
echo "Error: Unsupported service '$service'. Only 'postgresql' is supported."
exit 1
fi
case "$action" in
start)
/usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main start
exit $?
;;
restart)
/usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main stop || true
/usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main start
exit $?
;;
stop)
/usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main stop
exit $?
;;
*)
echo "Error: Unsupported action '$action'. Use start, restart, or stop."
exit 1
;;
esac
Actually there is one more thing that need to be fixed - calculate({SystemMemoryMegabytes}) doesn't work because it uses dmidecode. Maybe it can use something like free?
Note: the SuperuserPassword parameter change is not actually required, and since it causes CodeQL alert I can drop it
Looks like unfortunately the sysbench postgresql tests still doesn't quite work:
[02/13/2025 17:39:21] Profile: Action = SysbenchConfiguration (scenario=PopulatePostgreSQLDatabase)
[02/13/2025 17:39:21] SysbenchConfiguration.ExecuteStart
[02/13/2025 17:39:21] PackageManager.GetPackageStart
[02/13/2025 17:39:21] PackageManager.GetPackageStop
[02/13/2025 17:39:21] PackageManager.GetPackageStart
[02/13/2025 17:39:21] PackageManager.GetPackageStop
[02/13/2025 17:39:21] Executing: python3 /opt/virtualclient/packages/sysbench-1.0.20.rev1/configure-workload-generator.py --distro Ubuntu --databaseSystem PostgreSQL --packagePath /opt/virtualclient/packages/sysbench-1.0.20.rev1
[02/13/2025 17:39:21] SysbenchConfiguration.SysbenchExecutor.ProcessDetails
[02/13/2025 17:39:21] usage: configure-workload-generator.py [-h] -d DISTRO [-y SERVERTYPE] -p
PACKAGEPATH
configure-workload-generator.py: error: unrecognized arguments: --databaseSystem PostgreSQL (error/exit code=2, command=python3 /opt/virtualclient/packages/sysbench-1.0.20.rev1/configure-workload-generator.py --distro Ubuntu --databaseSystem PostgreSQL --packagePath /opt/virtualclient/packages/sysbench-1.0.20.rev1).
StandardError: usage: configure-workload-generator.py [-h] -d DISTRO [-y SERVERTYPE] -p
PACKAGEPATH
configure-workload-generator.py: error: unrecognized arguments: --databaseSystem PostgreSQL
at VirtualClient.ProcessExtensions.ThrowErrored[TError](IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 385
at VirtualClient.ProcessExtensions.ThrowIfErrored[TError](IProcessProxy process, IEnumerable`1 successCodes, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 228
at VirtualClient.ProcessExtensions.ThrowIfErrored[TError](IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 209
at VirtualClient.Actions.SysbenchExecutor.InitializeExecutablesAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Actions/Sysbench/SysbenchExecutor.cs:line 388
at VirtualClient.Actions.SysbenchExecutor.InitializeAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Actions/Sysbench/SysbenchExecutor.cs:line 319
at VirtualClient.Contracts.VirtualClientComponent.<>c__DisplayClass130_1.<<ExecuteAsync>b__4>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientComponent.cs:line 656
--- End of stack trace from previous location ---
at VirtualClient.Contracts.VirtualClientLoggingExtensions.<>c__DisplayClass17_0.<<LogMessageAsync>b__0>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 603
--- End of stack trace from previous location ---
at VirtualClient.Contracts.VirtualClientLoggingExtensions.LogMessageAsync[TResult](ILogger logger, String message, LogLevel level, EventContext eventContext, Func`1 body) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 641
[02/13/2025 17:39:21] SysbenchConfiguration.ExecuteStop