MLFlowClient.jl icon indicating copy to clipboard operation
MLFlowClient.jl copied to clipboard

Adding S3 Support for Artifacts

Open usmcamp0811 opened this issue 2 years ago • 3 comments

I have enhanced the logartifact function to accommodate MLFlow instances configured with S3 storage for artifact management. This enhancement includes an added verification step to determine if the specified URI corresponds to an S3 bucket. In cases where the URI is not an S3 bucket, the function retains its original behavior of saving artifacts to the disk. Conversely, when an S3 bucket is identified, the function leverages AWS and Minio services for artifact storage in the bucket. This implementation supports both AWS and Minio, although its compatibility with AWS S3 buckets has not been verified due to the lack of an AWS S3 bucket for testing. However, extensive testing has been conducted with an MLFlow instance utilizing Minio S3 buckets, and the functionality aligns with expected outcomes. It's important to note that when using Minio or similar S3 services, the MLFLOW_S3_ENDPOINT_URL environment variable must be set to ensure proper operation.

usmcamp0811 avatar Dec 26 '23 20:12 usmcamp0811

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (5a4237f) 77.64% compared to head (290ac2a) 74.71%. Report is 3 commits behind head on main.

Files Patch % Lines
src/loggers.jl 42.30% 15 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #34      +/-   ##
==========================================
- Coverage   77.64%   74.71%   -2.94%     
==========================================
  Files           9        9              
  Lines         331      348      +17     
==========================================
+ Hits          257      260       +3     
- Misses         74       88      +14     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Dec 29 '23 10:12 codecov[bot]

@pebeto all the docs I looked at for AWS said to use both the packages, but I can try to leave one off.

@deyandyankov yea I can make those changes here soon. I'm pretty busy right now, catching up from the holidays but should be able to get to it this weekend 🤞

usmcamp0811 avatar Jan 03 '24 15:01 usmcamp0811

You can find Mocking.jl useful to write tests for this specific implementation.

pebeto avatar Mar 07 '24 18:03 pebeto