aws
Sagemaker Local Mode with Local Code setting and HuggingFaceModel still requires S3 access
Describe the bug
When deploying a HuggingFace model with model data on disk, sagemaker SDK still tries to access the AWS API to determine the Sagemaker default bucket. I don't currently have access to my AWS credentials, so this is failing with access denied errors. I would like to be able to run locally without needing S3 access, and this is blocking me from doing so.
To reproduce
In an environment without AWS Access (no credentials setup).
from sagemaker.huggingface import HuggingFaceModel
from sagemaker import Session, get_execution_role
from sagemaker.local import LocalSession
# sagemaker_session = LocalSession()
# sagemaker_session.config = {'local': {'local_code': True}}
huggingface_model = HuggingFaceModel(
model_data="file:///path/to/model.tar.gz", # path to your trained SageMaker model
role='SageMakerRole', # IAM role with permissions to create an endpoint
transformers_version="4.26", # Transformers version used
pytorch_version="1.13", # PyTorch version used
py_version='py39',
)
huggingface_model.deploy(
initial_instance_count=1,
instance_type='local'
)
Maybe I still need access to AWS via the HF inference toolkit pulling the correct ECR image? However, that doesn't seem to be where it's failing.
Expected behavior Local mode to deploy an endpoint into a Docker container.
System information A description of your system. Please provide:
[tool.poetry.dependencies] python = "^3.12" sagemaker = {extras = ["local"], version = "^2.217.0"} jupyterlab = "^4.1.8" setuptools = "^69.5.1"
- CPU or GPU: CPU
- Custom Docker image (Y/N): N
Additional context
It looks like regardless of the local code setting in ~/.sagemaker/config.yaml, the SDK will still try to access the S3 API to determine the default bucket name for sagemaker.
From https://github.com/aws/sagemaker-python-sdk/blob/b17d332a5e4542d57d2039d08b124edc6042f9fb/src/sagemaker/model.py#L694.
def _upload_code(self, key_prefix: str, repack: bool = False) -> None:
"""Uploads code to S3 to be used with script mode with SageMaker inference.
Args:
key_prefix (str): The S3 key associated with the ``code_location`` parameter of the
``Model`` class.
repack (bool): Optional. Set to ``True`` to indicate that the source code and model
artifact should be repackaged into a new S3 object. (default: False).
"""
local_code = utils.get_config_value("local.local_code", self.sagemaker_session.config)
bucket, key_prefix = s3.determine_bucket_and_prefix(
bucket=self.bucket,
key_prefix=key_prefix,
sagemaker_session=self.sagemaker_session,
)
if (self.sagemaker_session.local_mode and local_code) or self.entry_point is None:
self.uploaded_code = None
elif not repack:
self.uploaded_code = fw_utils.tar_and_upload_dir(
session=self.sagemaker_session.boto_session,
bucket=bucket,
s3_key_prefix=key_prefix,
script=self.entry_point,
directory=self.source_dir,
dependencies=self.dependencies,
kms_key=self.model_kms_key,
settings=self.sagemaker_session.settings,
)
# ...
Within determine_bucket_and_prefix , sagemaker_session.default_bucket() is called. In my environment, this still fails even with a LocalSession object.
from sagemaker.huggingface import HuggingFaceModel
from sagemaker import Session, get_execution_role
from sagemaker.local import LocalSession
sagemaker_session = LocalSession()
sagemaker_session.config = {'local': {'local_code': True}}
sagemaker_session.default_bucket()
---------------------------------------------------------------------------
ClientError Traceback (most recent call last)
[/Users/jacob.windle/Projects/sagemaker_local_testing/TestM2MLocalMode.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/TestM2MLocalMode.ipynb) Cell 2 line 8
[6](vscode-notebook-cell:/Users/jacob.windle/Projects/sagemaker_local_testing/TestM2MLocalMode.ipynb#W1sZmlsZQ%3D%3D?line=5) sagemaker_session = LocalSession()
[7](vscode-notebook-cell:/Users/jacob.windle/Projects/sagemaker_local_testing/TestM2MLocalMode.ipynb#W1sZmlsZQ%3D%3D?line=6) sagemaker_session.config = {'local': {'local_code': True}}
----> [8](vscode-notebook-cell:/Users/jacob.windle/Projects/sagemaker_local_testing/TestM2MLocalMode.ipynb#W1sZmlsZQ%3D%3D?line=7) sagemaker_session.default_bucket()
File [~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:586](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:586), in Session.default_bucket(self)
[584](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:584) default_bucket = self._default_bucket_name_override
[585](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:585) if not default_bucket:
--> [586](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:586) default_bucket = generate_default_sagemaker_bucket_name(self.boto_session)
[587](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:587) self._default_bucket_set_by_sdk = True
[589](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:589) self._create_s3_bucket_if_it_does_not_exist(
[590](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:590) bucket_name=default_bucket,
[591](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:591) region=region,
[592](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:592) )
File [~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7359](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7359), in generate_default_sagemaker_bucket_name(boto_session)
[7351](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7351) """Generates a name for the default sagemaker S3 bucket.
[7352](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7352)
[7353](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7353) Args:
[7354](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7354) boto_session (boto3.session.Session): The underlying Boto3 session which AWS service
[7355](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7355) """
[7356](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7356) region = boto_session.region_name
[7357](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/sagemaker/session.py:7357) account = boto_session.client(
...
-> [1021](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/botocore/client.py:1021) raise error_class(parsed_response, operation_name)
[1022](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/botocore/client.py:1022) else:
[1023](https://file+.vscode-resource.vscode-cdn.net/Users/jacob.windle/Projects/sagemaker_local_testing/~/Projects/sagemaker_local_testing/.venv/lib/python3.12/site-packages/botocore/client.py:1023) return parsed_response
ClientError: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.
Why is a LocalSession still trying to access S3?
If this is actually a problem, and not just my own ignorance, I'm happy to contribute.
I'm experiencing the same issue using XGBOOST, and sagemaker[local]==2.222.0.
A code snippet extracted from this example: https://github.com/aws-samples/amazon-sagemaker-local-mode/blob/main/xgboost_script_mode_local_training_and_serving/xgboost_script_mode_local_training_and_serving.py
from sagemaker import TrainingInput
from sagemaker.xgboost import XGBoost, XGBoostModel
from sagemaker.local import LocalSession
DUMMY_IAM_ROLE = 'arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001'
LOCAL_SESSION = LocalSession()
LOCAL_SESSION.config = {'local': {'local_code': True}} # Ensure full code locality, see: https://sagemaker.readthedocs.io/en/stable/overview.html#local-mode
FRAMEWORK_VERSION = "1.7-1"
def main():
xgb_inference_model = XGBoostModel(
model_data="./tests/resources/models/fake/model.json",
role=DUMMY_IAM_ROLE,
entry_point="inference.py",
source_dir="./src",
framework_version=FRAMEWORK_VERSION,
sagemaker_session=LOCAL_SESSION
)
print('Deploying endpoint in local mode')
predictor = xgb_inference_model.deploy(
initial_instance_count=1,
instance_type="local",
)
def test_inference_endpoint():
main()
Expected behaviour: Model repacked locally. The docs imply you can " keep everything local, and not use Amazon S3 either, you can enable “local code”"
Actual behaviour: Failure when creating an S3 bucket during the deploy job.
The error:
operation_name = 'CreateBucket'
api_params = {'Bucket': 'sagemaker-eu-west-1-xxxxxxxxxxxx', 'CreateBucketConfiguration': {'LocationConstraint': 'eu-west-1'}}
My next steps: I'll give it the details of a bucket I have secure access to, and see what it actually tries to do with said bucket.
@fahran-wallace I'm getting the same error. Did you end up doing the test bucket in the end?
