Minimal gitlab configuration is failing to cp to default $OUT

[loganfred]

Probably a duplicate of #100.

I'm stuck integrating with gitlab. Here is a snippet showing the failure mode where a copy to $OUT is failing.

Successfully built 1e6e6b3e9a9c
Successfully tagged external-cfl-project-664774e26cc14adcb17b756aa191a6ca:latest
2023-04-26 18:30:07,330 - root - INFO - repo_dir: /builds/software/<PROJECT>.
2023-04-26 18:30:07,341 - root - INFO - Docker container: .
2023-04-26 18:30:07,341 - root - INFO - Building with address sanitizer.
2023-04-26 18:30:07,341 - root - INFO - Running: docker run --rm --privileged --shm-size=2g --platform linux/amd64 -e FUZZING_ENGINE=libfuzzer -e CIFUZZ=True -e SANITIZER=address -e ARCHITECTURE=x86_64 -e FUZZING_LANGUAGE=c++ -e OUT=/builds/33190/build-out -v /builds/33190:/builds/33190 -v /builds/software/<PROJECT>:/builds/software/<PROJECT> external-cfl-project-664774e26cc14adcb17b756aa191a6ca /bin/bash -c compile.
---------------------------------------------------------------
Compiling libFuzzer to /usr/lib/libFuzzingEngine.a...  done.
cp: cannot create regular file '/builds/33190/build-out/': Not a directory

Following the documentation, I have confirmed that the fuzzers can be built and run on my local machine. I've also tried an explicit mkdir -p $OUT line in the Dockerfile or the build.sh with no luck.

My configuration file is pretty minimal.

variables:
  SANITIZER: address
  CFL_PLATFORM: gitlab
  # DOCKER_HOST: "tcp://docker:2375" # may be removed in self-managed Gitlab instances
  DOCKER_IN_DOCKER: "true" # may be removed in self-managed Gitlab instances
  GIT_SUBMODULE_STRATEGY: recursive
  GIT_SUBMODULE_FORCE_HTTPS: "true"
  MODE: "code-change"

clusterfuzzlite:
  image:
    name: gcr.io/oss-fuzz-base/clusterfuzzlite-run-fuzzers:v1
    entrypoint: [""]
  services:
    - docker:dind # may be removed in self-managed Gitlab instances
    
  stage: test
  parallel:
    matrix:
      - SANITIZER: [address, undefined]
  #rules:
  #  # Default code change.
  #  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  #    variables:
  #      MODE: "code-change"
  before_script:
    # Get gitlab's container id.
    - export CFL_CONTAINER_ID=`cut -c9- < /proc/1/cpuset`
  script:
    # Will build and run the fuzzers.
    - python3 "/opt/oss-fuzz/infra/cifuzz/cifuzz_combined_entrypoint.py"
  artifacts:
    # Upload artifacts when a crash makes the job fail.
    when: always
    paths:
      - artifacts/

What should I try to keep troubleshooting this?

[loganfred]

hmm. Echoing $OUT in the dockerfile returns /out but the docker run command is OUT=/builds/33190/build-out Is this expected? Maybe I can force create the latter somehow.

[jonathanmetzman]

I think this is expected. /out is something inherited from OSS-Fuzz. Is the repo you are doing this on public? What version of gitlab's CI platform are you using? @catenacyber do you have any ideas?

[jonathanmetzman]

DO you know why /builds/33190 might be a file instead of a dir?

[loganfred]

Gitlab 15.11.0-ee with 15.11 runners Private repo unfortunately. I'm evaluating clusterfuzzlite for my company. I'm not sure at what layer that folder gets created, but I don't think it is getting made. At least based on my attempts to put ls -l in the dockerfile to see what's going on

[loganfred]

Also, this env var here is empty if I echo it

# Get gitlab's container id
- export CFL_CONTAINER_ID=`cut -c9- < /proc/1/cpuset`

Seems like a good clue..?

I'll pair with our dev ops person when they're available to get some more information

[catenacyber]

What kind of Gitlab runner do you have ?

This looks like cp $(which llvm-symbolizer) $OUT/ in compile is failing...

[loganfred]

These are self-hosted runners using kubernetes and helm.

I'm not sure what the root issue is, but I think I just need to figure out how to pass $CI_JOB_ID down from .gitlab-ci.yml into the Dockerfile so I can explicitly create that directory

[securitykernel]

2023-04-26 18:30:07,341 - root - INFO - Docker container: .

This is the cause. The OUT dir being not set properly is a subsequent error. Maybe we could add some sanity check. The issue is described and a fix provided in #126.

[loganfred]

Looks like the docs are already updated. Is there anything I need to do to update or will the image automatically pull cfl changes?

[jonathanmetzman]

I just updated the docs, I think you still need to recopy the example in the docs.

[loganfred]

The suggested .gitlab-ci.yaml change wasn't the silver bullet I was hoping for. What does CFL_CONTAINER_ID do? It still isn't populated for me with this syntax.

[jonathanmetzman]

It depends if you are using docker in docker, if you aren't, it's not used for anything. If you are: It's supposed to give a unique id to each run so they can write to a directory that persists between containers. If you have a substitute for this, use that.

[securitykernel]

So you are trying to run ClusterFuzzLite on a Kubernetes executor? This is currently not supported, though I'd love to see this. I currently use a private runner on a VM or bare metal that runs a Docker executor which has docker.sock mounted as a volume inside the containers.