kics Process in CodeBuild hangs

Expected Behavior

it runs

Actual Behavior

it hangs for more than 10 mins, no error messages

Steps to Reproduce the Problem

Run the following commands in CodeBuild:

[Container] 2022/03/25 18:39:00 Running command echo "kics_img_name $kics_img_name"
kics_img_name checkmarx/kics:v1.5.4

[Container] 2022/03/25 18:39:17 Running command docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation
Unable to find image 'checkmarx/kics:v1.5.4' locally
v1.5.4: Pulling from checkmarx/kics
Digest: sha256:628ed3b084e6ace14838b2772bf261187a606066d7bb31ed77087beb66ed0847
Status: Downloaded newer image for checkmarx/kics:v1.5.4
6:39PM DBG Could not find string flag ci
6:39PM DBG console.scan()
6:39PM DBG console.scan()

                   .0MO.                                    
                   OMMMx                                    
                   ;NMX;                                    
                    ...           ...              ....     
WMMMd     cWMMM0.  KMMMO      ;xKWMMMMNOc.     ,xXMMMMMWXkc.
WMMMd   .0MMMN:    KMMMO    :XMMMMMMMMMMMWl   xMMMMMWMMMMMMl
WMMMd  lWMMMO.     KMMMO   xMMMMKc...'lXMk   ,MMMMx   .;dXx 
WMMMd.0MMMX;       KMMMO  cMMMMd        '    'MMMMNl'       
WMMMNWMMMMl        KMMMO  0MMMN               oMMMMMMMXkl.  
WMMMMMMMMMMo       KMMMO  0MMMX                .ckKWMMMMMM0.
WMMMMWokMMMMk      KMMMO  oMMMMc              .     .:OMMMM0
WMMMK.  dMMMM0.    KMMMO   KMMMMx'    ,kNc   :WOc.    .NMMMX
WMMMd    cWMMMX.   KMMMO    kMMMMMWXNMMMMMd .WMMMMWKO0NMMMMl
WMMMd     ,NMMMN,  KMMMO     'xNMMMMMMMNx,   .l0WMMMMMMMWk, 
xkkk:      ,kkkkx  okkkl        ;xKXKx;          ;dOKKkc    


Scanning with Keeping Infrastructure as Code Secure v1.5.4


6:39PM INF Scanning with Keeping Infrastructure as Code Secure v1.5.4
6:39PM DBG storage.NewMemoryStorage()
6:39PM DBG Looking for queries in executable path and in current work directory
6:39PM DBG helpers.GetDefaultQueryPath()
6:39PM DBG helpers.GetExecutableDirectory()
6:39PM DBG Queries found in /app/bin/assets/queries
6:39PM DBG source.NewFilesystemSource()
6:39PM DBG engine.NewInspector()
6:39PM DBG Custom library not provided. Loading embedded library instead
6:39PM DBG Custom library not provided. Loading embedded library instead
6:39PM DBG Could not open embedded library data for cloudFormation platform
6:39PM INF Inspector initialized, number of queries=496
6:39PM INF Query execution timeout=1m0s
6:39PM DBG provider.NewFileSystemSourceProvider()
6:39PM DBG parser.NewBuilder()
6:39PM DBG resolver.Add()
6:39PM DBG resolver.Build()

here it hangs, you have to stop CodeBuild manually. Could you help, please?

Mar 25 '22 18:03 ALutchko

ping :( If I can get more verbose debug please let me know.

Apr 04 '22 07:04 ALutchko

Hello, @ALutchko!

Thank you so much for reaching us 😊

Can you give us details about your project, please?

Apr 11 '22 17:04 rafaela-soares

Hello @rafaela-soares, What kind of information do you need? We're building fully on AWS/CloudFormation.

Apr 12 '22 08:04 ALutchko

Hi, @ALutchko! Can you provide us with the project that you are running against KICS, please?

Apr 12 '22 08:04 rafaela-soares

I am really sorry but no, it is not OpenSource. If you provide me with more details what kind of information is needed then most likely I'll be able to provide it.

Buildspec for CodeBuild looks like:

      - ecr_domain="$CURR_ACC_ID.dkr.ecr.$REGION.amazonaws.com"
      - echo "ecr_domain $ecr_domain"
      - kics_img_name=$(grep checkmarx/kics ../ecr_images.txt)
      - aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $CURR_ACC_ID.dkr.ecr.$REGION.amazonaws.com/checkmarx
      - echo "kics_img_name $kics_img_name"
      - docker pull $ecr_domain/$kics_img_name
      - cd ../..  # repo root
      - pwd  # check we're in the correct place
      - docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation

The repo root has a few directories which contain CloudFormation yamls. Above "path" is literally the word "path", no substitution in this message.

Apr 12 '22 08:04 ALutchko

I understand 😊 Let's try to figure out what is happening.

Can you provide us with the size of the project?
Can you use the flag --profiling MEM and let us know the value of Total MEM usage for prepare_sources, please?
Is it stop at resolver.Build()? If not, please send us the entire log.

Apr 12 '22 08:04 rafaela-soares

@rafaela-soares , thank you very much for the quick response.

total size is ~23M but of course there are not only CF files. The largest CF template is ~46k
should I just add the flag to the end of docker run line or in any other way?
correct, it hangs there for more than 10 mins until I stop the build manually.

Apr 12 '22 08:04 ALutchko

Thank you. I will try to simulate a scan in CodeBuild with an approximate size project
Yes, for example: docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation --profiling MEM
For how long did you wait until you stopped?

Apr 12 '22 09:04 rafaela-soares

The result (I've added mem just before):


[Container] 2022/04/12 09:57:32 Running command free -h
              total        used        free      shared  buff/cache   available
Mem:          3.6Gi       709Mi       403Mi       0.0Ki       2.5Gi       2.7Gi
Swap:            0B          0B          0B
[Container] 2022/04/12 09:57:32 Running command docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation --profiling MEM
Unable to find image 'checkmarx/kics:v1.5.4' locally
v1.5.4: Pulling from checkmarx/kics
Digest: sha256:628ed3b084e6ace14838b2772bf261187a606066d7bb31ed77087beb66ed0847
Status: Downloaded newer image for checkmarx/kics:v1.5.4
9:57AM DBG Could not find string flag ci
9:57AM DBG console.scan()
9:57AM DBG console.scan()

                   .0MO.                                    
                   OMMMx                                    
                   ;NMX;                                    
                    ...           ...              ....     
WMMMd     cWMMM0.  KMMMO      ;xKWMMMMNOc.     ,xXMMMMMWXkc.
WMMMd   .0MMMN:    KMMMO    :XMMMMMMMMMMMWl   xMMMMMWMMMMMMl
WMMMd  lWMMMO.     KMMMO   xMMMMKc...'lXMk   ,MMMMx   .;dXx 
WMMMd.0MMMX;       KMMMO  cMMMMd        '    'MMMMNl'       
WMMMNWMMMMl        KMMMO  0MMMN               oMMMMMMMXkl.  
WMMMMMMMMMMo       KMMMO  0MMMX                .ckKWMMMMMM0.
WMMMMWokMMMMk      KMMMO  oMMMMc              .     .:OMMMM0
WMMMK.  dMMMM0.    KMMMO   KMMMMx'    ,kNc   :WOc.    .NMMMX
WMMMd    cWMMMX.   KMMMO    kMMMMMWXNMMMMMd .WMMMMWKO0NMMMMl
WMMMd     ,NMMMN,  KMMMO     'xNMMMMMMMNx,   .l0WMMMMMMMWk, 
xkkk:      ,kkkkx  okkkl        ;xKXKx;          ;dOKKkc    

Scanning with Keeping Infrastructure as Code Secure v1.5.4

9:57AM INF Scanning with Keeping Infrastructure as Code Secure v1.5.4
9:57AM DBG storage.NewMemoryStorage()
9:57AM DBG Looking for queries in executable path and in current work directory
9:57AM DBG helpers.GetDefaultQueryPath()
9:57AM DBG helpers.GetExecutableDirectory()
9:57AM DBG Queries found in /app/bin/assets/queries
9:57AM DBG source.NewFilesystemSource()
9:57AM DBG engine.NewInspector()
9:57AM DBG Started MEM profiling for get_queries
9:57AM DBG Custom library not provided. Loading embedded library instead
9:57AM DBG Custom library not provided. Loading embedded library instead
9:57AM DBG Could not open embedded library data for cloudFormation platform
9:58AM DBG Stopped MEM profiling for get_queries
9:58AM INF Total MEM usage for get_queries: 226.13MB
9:58AM INF Inspector initialized, number of queries=496
9:58AM INF Query execution timeout=1m0s
9:58AM DBG provider.NewFileSystemSourceProvider()
9:58AM DBG parser.NewBuilder()
9:58AM DBG resolver.Add()
9:58AM DBG resolver.Build()
9:58AM DBG Started MEM profiling for prepare_sources

it hanged in this state for 15 mins then I stopped the build.

Apr 12 '22 10:04 ALutchko

Hello again, @ALutchko!

I think I finally reproduced the issue!

With 3 GB memory, 2 vCPUs:

Scan time without --type flag: It fails
Scan time with --type flag: It fails

With 7 GB memory, 4 vCPUs:

Scan time without --type flag: ~3/4 min
Scan time with --type flag: ~9/10/11 min

P.S. I used this "project": https://github.com/rafaela-soares/codebuild-kics-scan

We will try to understand why there is a difference in the scan time and fix it. There is a strange behaviour when using the flag --type. We apologize for it.

Apr 12 '22 17:04 rafaela-soares

@ALutchko, can you try with 7 GB memory, 4 vCPUs and let us know if the build succeeded, please?

Apr 12 '22 17:04 rafaela-soares

Have just tried, waited 10 mins, the same result: Scanning with Keeping Infrastructure as Code Secure v1.5.4

7:39PM INF Scanning with Keeping Infrastructure as Code Secure v1.5.4
7:39PM DBG storage.NewMemoryStorage()
7:39PM DBG Looking for queries in executable path and in current work directory
7:39PM DBG helpers.GetDefaultQueryPath()
7:39PM DBG helpers.GetExecutableDirectory()
7:39PM DBG Queries found in /app/bin/assets/queries
7:39PM DBG source.NewFilesystemSource()
7:39PM DBG engine.NewInspector()
7:39PM DBG Started MEM profiling for get_queries
7:39PM DBG Custom library not provided. Loading embedded library instead
7:39PM DBG Custom library not provided. Loading embedded library instead
7:39PM DBG Could not open embedded library data for cloudFormation platform
7:40PM DBG Stopped MEM profiling for get_queries
7:40PM INF Total MEM usage for get_queries: 252.36MB
7:40PM INF Inspector initialized, number of queries=496
7:40PM INF Query execution timeout=1m0s
7:40PM DBG provider.NewFileSystemSourceProvider()
7:40PM DBG parser.NewBuilder()
7:40PM DBG resolver.Add()
7:40PM DBG resolver.Build()
7:40PM DBG Started MEM profiling for prepare_sources

What will be the application behavior if I just remove --type CloudFormation? It will scan for all possible types and try to guess?

Apr 12 '22 19:04 ALutchko

Yes. Can you try without the flag --type, please?

Apr 13 '22 08:04 rafaela-soares

The same. Switched off after 27 min from start. Also, I've updated to 1.5.5

9:25AM INF Scanning with Keeping Infrastructure as Code Secure v1.5.5
9:25AM DBG storage.NewMemoryStorage()
9:25AM DBG Looking for queries in executable path and in current work directory
9:25AM DBG helpers.GetDefaultQueryPath()
9:25AM DBG helpers.GetExecutableDirectory()
9:25AM DBG Queries found in /app/bin/assets/queries
9:25AM DBG source.NewFilesystemSource()
9:25AM DBG engine.NewInspector()
9:25AM DBG Started MEM profiling for get_queries
9:25AM DBG Custom library not provided. Loading embedded library instead
9:25AM DBG Custom library not provided. Loading embedded library instead
9:25AM DBG Could not open embedded library data for cloudFormation platform
9:25AM DBG Stopped MEM profiling for get_queries
9:25AM INF Total MEM usage for get_queries: 260.02MB
9:25AM INF Inspector initialized, number of queries=496
9:25AM INF Query execution timeout=1m0s
9:25AM DBG provider.NewFileSystemSourceProvider()
9:25AM DBG parser.NewBuilder()
9:25AM DBG resolver.Add()
9:25AM DBG resolver.Build()
9:25AM DBG Started MEM profiling for prepare_sources
9:35AM DBG Stopped MEM profiling for prepare_sources
9:35AM INF Total MEM usage for prepare_sources: 1.48GB
9:35AM DBG Started MEM profiling for start_scan
9:35AM DBG service.StartScan()
9:35AM DBG service.StartScan()
9:35AM DBG engine.Inspect()

Apr 13 '22 09:04 ALutchko

So it is not related to the behaviour of the flag --type.

Can you try to use 15 GB memory, 8 vCPUs, please?

Apr 13 '22 10:04 rafaela-soares

@ALutchko, can you send us only the IaC code to [email protected]? Only for debugging purposes to understand what is happening.

Apr 13 '22 10:04 rafaela-soares

Large: just for test purpose, because using such size in real workflow won't make sense due to price. Tested, switched off after 20 min from start

10:21AM INF Scanning with Keeping Infrastructure as Code Secure v1.5.5
10:21AM DBG storage.NewMemoryStorage()
10:21AM DBG Looking for queries in executable path and in current work directory
10:21AM DBG helpers.GetDefaultQueryPath()
10:21AM DBG helpers.GetExecutableDirectory()
10:21AM DBG Queries found in /app/bin/assets/queries
10:21AM DBG source.NewFilesystemSource()
10:21AM DBG engine.NewInspector()
10:21AM DBG Started MEM profiling for get_queries
10:21AM DBG Custom library not provided. Loading embedded library instead
10:21AM DBG Custom library not provided. Loading embedded library instead
10:21AM DBG Could not open embedded library data for cloudFormation platform
10:21AM DBG Stopped MEM profiling for get_queries
10:21AM INF Total MEM usage for get_queries: 225.68MB
10:21AM INF Inspector initialized, number of queries=496
10:21AM INF Query execution timeout=1m0s
10:21AM DBG provider.NewFileSystemSourceProvider()
10:21AM DBG parser.NewBuilder()
10:21AM DBG resolver.Add()
10:21AM DBG resolver.Build()
10:21AM DBG Started MEM profiling for prepare_sources
10:30AM DBG Stopped MEM profiling for prepare_sources
10:30AM INF Total MEM usage for prepare_sources: 1.48GB
10:30AM DBG Started MEM profiling for start_scan
10:30AM DBG service.StartScan()
10:30AM DBG service.StartScan()
10:30AM DBG engine.Inspect()

I'll try to send something.

Apr 13 '22 10:04 ALutchko

Sent.

Apr 13 '22 12:04 ALutchko

Hello again, @ALutchko! Thank you so much for sending it.

We ran KICS against your folder and the scan was quick.

Did you try to run KICS locally against your (entire) project?
Are your CF IaC samples in the same folder inside your project?

Apr 13 '22 14:04 rafaela-soares

Locally it works, even on VM having 1GB ram so memory isn't the issue here. Looks strange but timestamps are in GMT, not my local time.

r@l $ docker run -v /home/ans/clsm-ci-cd:/path checkmarx/kics scan  --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --profiling MEM
2:19PM DBG Could not find string flag ci
2:19PM DBG console.scan()
2:19PM DBG console.scan()

   [ logo ]

Scanning with Keeping Infrastructure as Code Secure v1.5.5


2:19PM INF Scanning with Keeping Infrastructure as Code Secure v1.5.5
2:19PM DBG storage.NewMemoryStorage()
2:19PM DBG Looking for queries in executable path and in current work directory
2:19PM DBG helpers.GetDefaultQueryPath()
2:19PM DBG helpers.GetExecutableDirectory()
2:19PM DBG Queries found in /app/bin/assets/queries
2:19PM DBG source.NewFilesystemSource()
2:19PM DBG engine.NewInspector()
2:19PM DBG Started MEM profiling for get_queries
2:19PM DBG Custom library not provided. Loading embedded library instead
2:19PM DBG Custom library not provided. Loading embedded library instead
2:19PM DBG Could not open embedded library data for cloudFormation platform
2:20PM DBG Stopped MEM profiling for get_queries
2:20PM INF Total MEM usage for get_queries: 273.14MB
2:20PM INF Inspector initialized, number of queries=496
2:20PM INF Query execution timeout=1m0s
2:20PM DBG provider.NewFileSystemSourceProvider()
2:20PM DBG parser.NewBuilder()
2:20PM DBG resolver.Add()
2:20PM DBG resolver.Build()
2:20PM DBG Started MEM profiling for prepare_sources
2:20PM DBG Stopped MEM profiling for prepare_sources
2:20PM INF Total MEM usage for prepare_sources: 273.14MB
2:20PM DBG Started MEM profiling for start_scan
2:20PM DBG service.StartScan()
2:20PM DBG service.StartScan()
2:20PM DBG engine.Inspect()
2:20PM DBG match: true :: 6.256660416485939
2:20PM DBG match: true :: 5.037401197654112
2:20PM DBG engine.Inspect()


2:22PM DBG Stopped MEM profiling for start_scan
2:22PM INF Total MEM usage for start_scan: 403.90MB
2:22PM DBG model.CreateSummary()
2:22PM DBG HTTP POST to descriptions endpoint
2:22PM DBG HTTP Status: 200 OK 253.227496ms
2:22PM DBG console.resolveOutputs()
2:22PM DBG helpers.PrintResult()
Files scanned: 38
Parsed files: 38
Queries loaded: 496
Queries failed to execute: 0

------------------------------------

EC2 Not EBS Optimized, Severity: INFO, Results: 1
Description: It's considered a best practice for an EC2 instance to use an EBS optimized instance. This provides the best 
....
Results Summary:
HIGH: 67
MEDIUM: 38
LOW: 76
INFO: 1
TOTAL: 182

2:22PM INF Files scanned: 38
2:22PM INF Lines scanned: 8594
2:22PM INF Parsed files: 38
2:22PM INF Lines parsed: 8594
2:22PM INF Queries loaded: 496
2:22PM INF Queries failed to execute: 0
2:22PM INF Inspector stopped
2:22PM DBG console.printOutput()
2:22PM DBG Output formats provided [junit]
2:22PM DBG helpers.GenerateReport()
2:22PM DBG Started MEM profiling for generate_report
2:22PM INF Results saved to file /path/junit-kics-report.xml fileName=junit-kics-report.xml
Results saved to file /path/junit-kics-report.xml
2:22PM DBG Stopped MEM profiling for generate_report
2:22PM INF Total MEM usage for generate_report: 403.90MB
Scan duration: 3m46.054786187s
2:22PM INF Scan duration: 3m46.054786187s

The repo root contains files like below:

aws-ci-cd # (the dir in the archive)
...
kics.config
README.md

Just in case:

$ cat kics.config

log-file: true
log-level: DEBUG
#log-path: .
silent: false
type:
  - CloudFormation
# Ansible, AzureResourceManager, Buildah, CloudFormation, DockerCompose, Dockerfile,
# GRPC, GoogleDeploymentManager, Kubernetes, OpenAPI, Terraform
verbose: true

But the same config exists locally.

Looks like the issue is in relations with CodeBuild.

Apr 13 '22 14:04 ALutchko

I've double-checked, "PrivilegedMode" set to "true" for this CodeBuild

Apr 13 '22 14:04 ALutchko

Yes, it seems that is some CodeBuild issue...

You can specify with directories/files you want to scan. Maybe it will help in the CodeBuild.

For example: docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path/<IaCFolderOrFileName>,/path/<IaCFolderOrFileName2> -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation

Let me know if it helps 🤞

Apr 13 '22 14:04 rafaela-soares

This scan completed in 1.5 mins:

[Container] 2022/04/13 14:59:58 Running command path_to_check="/path/aws-ci-cd/infra-auto/infra-auto.v2.yml"
[Container] 2022/04/13 14:59:58 Running command docker run -v $(pwd):/path $kics_img_name scan --ignore-on-exit all -p "$path_to_check" -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --profiling MEM

So, imho it makes sense to add debug to kics before and after each file and/or dir to check which one breaks test. What do you think?

Apr 13 '22 15:04 ALutchko

You can try. But as you showed, the scan is quick locally. I do not know why CodeBuild is crashing 😢

Apr 13 '22 16:04 rafaela-soares

I cannot try, it should be done from inside of KICS :) You have that loop inside which iterated over dirs/files :)

Apr 13 '22 17:04 ALutchko

Unfortunately, we can not do much if we do not have access to the entire project.

Regarding the part of the project that you sent us, we did not find anything that breaks or crashes. We tested it locally, and it was quick. And we tested it in the CodeBuild (in a private repo, of course), and it was fast too (even with 3 GB memory, 2 vCPUs).

Apr 13 '22 17:04 rafaela-soares

Furthermore, as you showed, your entire project took ~3/4 min locally. So, it seems that it is not a problem from the KICS side.

Apr 13 '22 17:04 rafaela-soares

I don't blame "it's on your side" but this is the only way I can think about to debug. It's impossible to try files one by one in CodeBuild env :(

Apr 13 '22 18:04 ALutchko

Maybe the issue can be escalated to the main team?

Apr 13 '22 18:04 ALutchko

The main team is already aware of this issue.

Apr 13 '22 18:04 rafaela-soares