[Feature:System] Revamp worker setup and fix QEMU

[skara9]

Please check if the PR fulfills these requirements:

• [ ] Tests for the changes have been added/updated (if possible)
• [ ] Documentation has been updated/added if relevant
• [ ] Screenshots are attached to Github PR if visual/UI changes were made

What is the current behavior?

Worker configuration currently fails on QEMU due to lack of support for private networking.

What is the new behavior?

The QEMU plugin has been modified to add support for socket networking. The worker management configuration has been revamped to better accomodate the networking needs of the new QEMU plugin setup.

Testing instructions: See Submitty/submitty.github.io#621

[GO-viper7]

Please request a review after this PR gets out of draft.

[codecov[bot]]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 20.67%. Comparing base (676c9d5) to head (40242dd). :warning: Report is 115 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10530      +/-   ##
============================================
- Coverage     20.82%   20.67%   -0.15%     
- Complexity     9115     9187      +72     
============================================
  Files           261      264       +3     
  Lines         34966    35226     +260     
  Branches        460      460              
============================================
+ Hits           7280     7284       +4     
- Misses        27230    27486     +256     
  Partials        456      456              

Flag	Coverage Δ
autograder	21.34% <ø> (ø)
js	2.13% <ø> (ø)
migrator	100.00% <ø> (ø)
php	19.35% <ø> (-0.16%)	:arrow_down:
python_submitty_utils	80.08% <ø> (ø)
submitty_daemon_jobs	88.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[skara9]

@ziesski When you install homebrew, it gives you a command to update your PATH. You may have missed this step.

[ziesski]

It has been long while since I installed homebrew.

This is my current HOMEBREW_PREFIX What do you want me to update it to.

[ziesski]

@skara9 I am doing this time on my M3 MacOS 14, qemu, and getting below error. this is after I did vagrant destroy Could you tell me what is wrong? I tried searching online, but can't seem to find recent issues.

[ziesski]

Tested on intel Mac OS 12, works. Will test on M3 once @skara9 provides the fix.

[ziesski]

This is what I get after vagrant workers up, is this expected? @skara9

[bmcutler]

@skara

Notes on instructions above

• Please start with "vagrant destroy" (if you have an existing vagrant VM) ... that is, assuming you must destroy the main VM first? But maybe you don't? I was getting errors, but I'm just starting to go through these steps.

• BASE=1 is the old version. We switched to FROM_SCRATCH=1

• Where do I specify the # of workers? These instructions are replacing the old instructions which had that as a clear option https://submitty.org/developer/getting_started/worker_vm

[skara9]

@bmcutler You don't need to destroy the main VM for setting up workers, but you will have to reprovision it once workers are up because the installation scripts on the main still send the SSH key to the workers.

For multiple workers, it is vagrant workers generate -n X, this is documented in the new instructions (not merged yet).

[bmcutler]

@skara9 Please resolve conflicts

[GO-viper7]

@skara9 I followed the testing instructions and here's what I got in windows Also, I made sure to set FROM_SCRATCH=1, as you have mentioned current prepackaged VM does not support workers. It still didn't work as intended.

[skara9]

@Chriun

Works on Windows 10 with 1 and 3 workers. I did run into problems initially with the setup since I was building the pre-packaged vm instead of from scratch. I think this should be highlighted in the documentation as well.

Ideally once this is merged we can package the worker setup as well.

[bmcutler]

NOTE: This PR works on MacOS 14 and below. We will open an issue describing the problems with MacOS 15.