[BUG] onedir - vendor system python packages

[ITJamie]

Description note: follow on from https://saltstackcommunity.slack.com/archives/CNZKJMQ1E/p1691686473917269?thread_ts=1691617721.100279&cid=CNZKJMQ1E

some vendors supply custom os's with built in custom python packages (cisco, sonic (edgecore, broadcom), nimble, f5) which are not available from pypi etc AND have specific customizations in them for that specific build of the custom os/image.

those python packages make adjusting config/settings of these switches/sans/loadbalancers easier. with classic packaging it was possible to import and use these python packages, with onedir it is not.

Setup An example with sonic:

• install qemu on your system.
• download a sonic-vs.img (eg https://sonic-build.azurewebsites.net/api/sonic/artifacts?branchName=master&definitionId=1&artifactName=sonic-buildimage.vs&buildId=334542&target=target%2Fsonic-vs.img.gz)
• start the qemu image with an e1000 bridged nic (see https://github.com/sonic-net/SONiC/wiki/SONiC-on-virtual-machine-for-Windows#3-start-sonic-image for windows example qemu command)
• install onedir salt 3006.x inside the os (debian based)
• run salt's python interpreter, attempt to import any of the images python libs eg:

    from sonic_py_common.multi_asic import is_multi_asic
    from sonic_py_common.interface import get_interface_table_name, get_port_table_name
    from swsscommon import swsscommon
    from swsssdk import ConfigDBConnector
    from swsscommon.swsscommon import SonicV2Connector
    import utilities_common.cli as clicommon
    from config.config_mgmt import sonic_cfggen
    import click
    import config.main as sonic_config_main

this will fail as onedir cant access these on-system libs

[OrangeDog]

@dwoz this hasn't been triaged

[dwoz]

@OrangeDog The description of the issue is sufficient and it's a valid issue. Triage is only needed when we need to confirm the issue is indeed an issue.

[OrangeDog]

@dwoz that's not what triage is. Triage is the process of assigning severity and priority to an issue. Barely a single non-Documentation issue has been actually triaged since I was asked to leave it for the core team meetings to do it. You've been basically ignoring your own process specifications since Chunga took over.

These days, you're supposed to at least assign a milestone or project, which @anilsil has now done.

[ITJamie]

Ive been thinking about potential workarounds for this.

Maybe some alternative from/import functions which could allow an alternative python library path to search in?

Now that 3005 is leaving active support its becoming more pressing

[dwoz]

Ive been thinking about potential workarounds for this.

Maybe some alternative from/import functions which could allow an alternative python library path to search in?

Now that 3005 is leaving active support its becoming more pressing

I like this approach better.

[dwoz]

@ITJamie What is the best salt state/module to use for a POC of your suggested approach?

[ITJamie]

Theoretically it could be any pip module that only exists on the system level python and not in salt-pip's python. I guess the hadest part would be handling sub-imports. Eg when a library trys to import another library how would it know to use system level python paths vs salt-pip's paths.

From an integration testing pov im happy to test it on actual sonic/edgecore switches once we have some working code/builds

[bdrx312]

Ive been thinking about potential workarounds for this. Maybe some alternative from/import functions which could allow an alternative python library path to search in? Now that 3005 is leaving active support its becoming more pressing

I like this approach better.

I don't think it is as simple as this since salt possibly (likely in the cases of os like rhel) running on a newer version of python than the system python.

[dwoz]

Ive been thinking about potential workarounds for this. Maybe some alternative from/import functions which could allow an alternative python library path to search in? Now that 3005 is leaving active support its becoming more pressing

I like this approach better.

I don't think it is as simple as this since salt possibly (likely in the cases of os like rhel) running on a newer version of python than the system python.

True, though I think we'd get pretty far with this approach on most OSes. There can be cases of things being used in the system libraries that have been deprecated in salt's version of python. When in doubt, shell out.