Add support for detecting the number of available Neuron cores

[aws-taylor]

Before or while filing an issue please feel free to join our slack channel to get in touch with development team, ask questions, find out what's cooking and more!

Issue #, if available:

Description of changes:

Similar to GPUs, when running on AWS Inf1 instance types, automatically detect the number of available Neuron cores and set the number of workers accordingly.

Testing done:

Ran tests on inf1, logs correctly show 16 neuron cores on an inf1.6xlarge.

...
   Temp directory: /tmp
    Number of GPUs: 0
    Number of Neuron Cores: 16
    Number of CPUs: 24
    Max heap size: 10457 M
    Python executable: python
...

To run CI tests on your changes refer README.md

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[aws-taylor]

I believe the following block in getDefaultWorkers() will cause it to have the desired effect:

if (workers == 0) {



workers = getNumberOfNeuronCores();


}

From: lxning @.> Reply-To: awslabs/multi-model-server @.> Date: Wednesday, August 24, 2022 at 12:15 PM To: awslabs/multi-model-server @.> Cc: "Goodhart, Taylor" @.>, Mention @.***> Subject: Re: [awslabs/multi-model-server] Add support for detecting the number of available Neuron cores (PR #1002)

@aws-taylorhttps://github.com/aws-taylor This PR make MMS get the number of Neuron Cores, but don't see it is used in MMS internally. Will you have another PR to address it?

— Reply to this email directly, view it on GitHubhttps://github.com/awslabs/multi-model-server/pull/1002#issuecomment-1226135196, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANYNIBQKSB272JDEDFIIR7DV2ZYBRANCNFSM57MQNFLA. You are receiving this because you were mentioned.Message ID: @.***>