QCAlgorithm.Download incompatible with S3 Authentication Headers

[Grosner300]

Expected Behavior

QCAlgorithm.Download permits AWS S3 Authentication Headers. The code for Download appears to have changed in #7976 from WebClient to HttpClient which adds strict header validation when using .Add to add a header

Actual Behavior

Exception: Unhandled exception. System.FormatException: The format of value 'AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-date,Signature=EXAMPLE_SIGNATURE' is invalid.

Potential Solution

In Api.cs DownloadBytes, use client.Value.TryAddWithoutValidation(header.Key, header.Value); instead of client.Value.DefaultRequestHeaders.Add(header.Key, header.Value);

Reproducing the Problem

In QuantConnect

var headers = new Dictionary<string, string>
{
	{"Authorization", "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-date,Signature=EXAMPLE_SIGNATURE"}
};
Download("https://www.example.com", headers.ToList());

Unit Test

[Test, Parallelizable(ParallelScope.Self)]
public void Download_With_S3_Authentication_Header_Successfully()
{
    var algo = new QCAlgorithm();
    algo.SetApi(new Api.Api());

    var headers = new List<KeyValuePair<string, string>>
    {
        new KeyValuePair<string, string>("Authorization","AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-date,Signature=EXAMPLE_SIGNATURE")
    };

    var content = string.Empty;
    Assert.DoesNotThrow(() => content = algo.Download("https://www.quantconnect.com/", headers));
    Assert.IsNotEmpty(content);
}

Outside QuantConnect

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Clear();
httpClient.DefaultRequestHeaders.Add("Authorization", "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-date,Signature=EXAMPLE_SIGNATURE");
httpClient.DefaultRequestHeaders.Dump();

System Information

Windows 11

Checklist

• [X] I have completely filled out this template
• [X] I have confirmed that this issue exists on the current master branch
• [X] I have confirmed that this is not a duplicate issue by searching issues

• [X] I have provided detailed steps to reproduce the issue

[Martin-Molinero]

Thank you for the report @Grosner300!