dev-sec
Compliance Report too big to be compatible with AWS SSM
Describe the bug Hi!
We configured a workload on AWS EC2s and configured AWS SSM to run a baseline check of the EC2 AMIs using this repo. SSM failed to run the command due to its inability to download the baseline.
Expected behavior Successful run of baseline which returns results.
Actual behavior
* Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/placement/availability-zone HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
> X-aws-ec2-metadata-token: xxx
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Content-Length: 10
< Content-Type: text/plain
< Date: Tue, 12 Apr 2022 06:48:42 GMT
< Last-Modified: Tue, 12 Apr 2022 05:41:15 GMT
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Connection: close
< Server: EC2ws
<
{ [10 bytes data]
* Closing connection 0
/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Compliance item can have up to 800 KB in total. (Aws::SSM::Errors::ItemSizeLimitExceededException)
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.130.0/lib/seahorse/client/request.rb:72:in `send_request'
from /root/.chefdk/gem/ruby/3.0.0/gems/aws-sdk-ssm-1.134.0/lib/aws-sdk-ssm/client.rb:8065:in `put_compliance_items'
from ./Report-Compliance-20200225:122:in `<main>'
failed to run commands: exit status 1
Example code Follow the example provided by AWS. It was a manual setup, so no code to paste 😞
OS / Environment Not relevant for this issue (see below).
Inspec Version Not sure how to get this, but not relevant for this issue (see below).
Baseline Version
18fd9203d64e67f04d64fff9c5b60cd2b4065953
Additional context
AWS throws a Compliance item can have up to 800 KB in total. error which is outlined in their documentation as a service quota.
Per AWS Service Quota documentation, adjustments can be made. However, this particular item is not yet available in their automated service quota requests (through console) nor in the aws request-service-quota-increase (through the cli) and there is no guarantee that there is software in place to handle a request if one is made to customer service.
This issue was not hit by the linux-baseline because the total repo size is less than the 800KB requirement.
There is an option to set a path in the configuration of the AWS RunCommand,
{
"owner":"dev-sec",
"repository":"linux-baseline",
"path": "",
"getOptions" : "branch:master",
"tokenInfo":"{{ssm-secure:github-personal-token}}"
}
However, it seems like there are a few directories and files that are relevant. Can consolidating the code into a single directory and then specifying that work for AWS SSM?
The problem is that the .git directory of the repo takes up more than half of the disk quota, so the idea is that by specifying the consolidated directory, only the relevant files will be procured by the download and SSM will only hit this issue again if the compliance rules themselves grow to be more than 800KB,
Upon testing from using a smaller repo, the error still persisted. So we have determined that there is a control that is getting tested or a set of controls that is getting grouped that generates a Compliance Report that is too big to be exported by AWS SSM (for the same service quota limit).
Thanks to @mogul for helping to clarify the issue!
With the new results, it seems like it is also an issue that is irrespective of the operating environment. Does anyone know which report may be greater than 800KB?
Hi @nickumia-reisys,
I think I know what the problem is, but I literally haven't touched a keyboard in the last couple months so my memory could be off:
The problem is that the .git directory of the repo takes up more than half of the disk quota, so the idea is that by specifying the consolidated directory, only the relevant files will be procured by the download and SSM will only hit this issue again if the compliance rules themselves grow to be more than 800KB
It's not the size of the .git directory that's causing the error, but the size of the payload sent to the SSM Compliance API endpoint ( downloaded by the AWS-RunInspecChecks document)
SSM Doc:
https://us-east-1.console.aws.amazon.com/systems-manager/documents/AWS-RunInspecChecks/content?region=us-east-1
Ruby Script to format InSpec output and post to SSM:
curl -sS https://s3.us-east-1.amazonaws.com/aws-ssm-us-east-1/statemanagerdocumentspayload/AWS-RunInspecChecks/Report-Compliance-20200225
# trimmed for brevity
...
execution_id = pwd.parent.basename.to_s;
ssm = Aws::SSM::Client.new(region: region);
results = JSON.parse(STDIN.read);
# initialize compliance object
comp = {
resource_id: instance_id,
resource_type: 'ManagedInstance',
compliance_type: 'Custom:InSpec',
execution_summary: {
execution_time: Time.now,
execution_id: execution_id,
execution_type: 'Command'
},
items: Array.new()
};
...
results['profiles'].each do |profile|
profile['controls'].each do |control|
if(control.has_key?('results'))
control['results'].each do |result|
severity = impact_to_severity(control['impact']);
item = {
id: "#{control['id']}-#{comp[:items].length}",
severity: severity,
title: "#{control['title']} : #{result['code_desc']}"
};
status = result['status'];
if(status == 'passed')
item[:status] = 'COMPLIANT';
compliant += 1;
compliant_by_sev[severity] += 1;
elsif(status == "failed")
item[:status] = 'NON_COMPLIANT';
non_compliant += 1;
non_compliant_by_sev[severity] += 1;
else
next;
end
comp[:items].push(item);
end
end
end
end
resp = ssm.put_compliance_items(comp); ### <- CULPRIT
...
This issue was not hit by the linux-baseline because the total repo size is less than the 800KB requirement.
The reason the linux-baseline profile works is because it has significantly fewer controls than the cis-dil-benchmark profile (dozens vs hundreds).
There's definitely more than a few warts when using the Custom:Inspec compliance types and the massive size from the json reporter output doesn't make it any easier 🙃.
The quickest/dirtiest thing would prob be to make a clone of the AWS-RunInspecChecks document and do something to minimize the report output. Other options would be similar to what's discussed in https://github.com/dev-sec/cis-dil-benchmark/issues/112#issuecomment-817071258 as possible work arounds.
All that said, I don't see an easy/simple fix that wouldn't break workflows for existing consumers of the profile.
I think the only option in the near term is to open an AWS support ticket requesting better support for InSpec configuration/options.
@schurzi @micheelengronne thoughts?
cc: @amlodzianowsk
I agree with everything you said @deric4 😄
Sorry for not updating this ticket earlier, but we actually did end up submitting an AWS Support Ticket to try and increase the Quota Limit,
Reported upstream to AWS in the ssb-production account and asked for the service quota to be increased if possible, since the forms don't allow for it... Case ID 9924596491.
We were able to manually run the baseline by ssh'ing and running inspec in docker. So we got around the immediate hurdle. The primary concern would be a way to support the smaller output requirements in case AWS can't increase their quota for whatever reason haha.. And the current solution is either separating the controls or waiving some controls and running the baseline multiple times.
I suppose at the minimal, may I ask that this just be documented for other people who try to run the baseline and hit this issue?
Hmm.. so it took some time; however, AWS emailed us back saying that they were able to increase our quota to 1200KB. Upon trying to run it again, it seems AWS SSM gave this error.. any insights? @deric4 I'm still able to run it manually by ssh'ing. This only happens when SSM tries to RunCommand.
* Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/placement/availability-zone HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
> X-aws-ec2-metadata-token: xxx
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Content-Type: text/plain
< Accept-Ranges: none
< Last-Modified: Mon, 09 May 2022 17:41:32 GMT
< Content-Length: 10
< Date: Mon, 09 May 2022 18:26:01 GMT
< Server: EC2ws
< Connection: close
<
{ [10 bytes data]
* Closing connection 0
Don't understand inspec profile in ., it doesn't look like a supported profile structure.
/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/json-2.6.1/lib/json/common.rb:216:in `parse': 859: unexpected token at '' (JSON::ParserError)
from /opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/json-2.6.1/lib/json/common.rb:216:in `parse'
from ./Report-Compliance-20200225:39:in `<main>'
failed to run commands: exit status 1