Support Pipeline Parameters in Config Processing

[philnielsen]

• [x] I have read Contribution Guidelines.
• [x] I have checked for similar issues and haven't found anything relevant.
• [x] This is not a security issue (which should be reported here: https://circleci.com/security/)

Do you want to request a feature or report a bug? Feature

What is the current behavior? Currently when you run something like circleci config process .circleci/config.yml > process.yml it runs with the default pipeline parameters given in the config. It would be nice if you could supply pipeline parameters in the CLI call so you don't have to change your source config every time to test pipeline parameter changes.

EXAMPLE config.yml:

version: 2.1
parameters:
  image-tag:
    type: string
    default: "latest"
  workingdir:
    type: string
    default: "~/main"

jobs:
  build:
    docker:
      - image: circleci/node:<< pipeline.parameters.image-tag >>
    environment:
      IMAGETAG: << pipeline.parameters.image-tag >>
    working_directory: << pipeline.parameters.workingdir >>
    steps:
      - run: echo "Image tag used was ${IMAGETAG}"
      - run: echo "$(pwd) == << pipeline.parameters.workingdir >>"

EXAMPLE process.yml

version: 2
jobs:
  build:
    docker:
    - image: circleci/node:latest
    environment:
    - IMAGETAG: latest
    working_directory: ~/main
    steps:
    - run:
        command: echo "Image tag used was ${IMAGETAG}"
    - run:
        command: echo "$(pwd) == ~/main"
workflows:
  version: 2
  workflow:
    jobs:
    - build

# Original config.yml file:
# version: 2.1
# parameters:
#   image-tag:
#     type: string
#     default: \"latest\"
#   workingdir:
#     type: string
#     default: \"~/main\"
# 
# jobs:
#   build:
#     docker:
#       - image: circleci/node:<< pipeline.parameters.image-tag >>
#     environment:
#       IMAGETAG: << pipeline.parameters.image-tag >>
#     working_directory: << pipeline.parameters.workingdir >>
#     steps:
#       - run: echo \"Image tag used was ${IMAGETAG}\"
#       - run: echo \"$(pwd) == << pipeline.parameters.workingdir >>\"

What is the expected behavior?

I'd like something like circleci config process -p workingdir="~/home" .circleci/config.yml > process.yml to end up like:

DESIRED process.yml

version: 2
jobs:
  build:
    docker:
    - image: circleci/node:latest
    environment:
    - IMAGETAG: latest
    working_directory: ~/home
    steps:
    - run:
        command: echo "Image tag used was ${IMAGETAG}"
    - run:
        command: echo "$(pwd) == ~/home"
workflows:
  version: 2
  workflow:
    jobs:
    - build

# Original config.yml file:
# version: 2.1
# parameters:
#   image-tag:
#     type: string
#     default: \"latest\"
#   workingdir:
#     type: string
#     default: \"~/main\"
# 
# jobs:
#   build:
#     docker:
#       - image: circleci/node:<< pipeline.parameters.image-tag >>
#     environment:
#       IMAGETAG: << pipeline.parameters.image-tag >>
#     working_directory: << pipeline.parameters.workingdir >>
#     steps:
#       - run: echo \"Image tag used was ${IMAGETAG}\"
#       - run: echo \"$(pwd) == << pipeline.parameters.workingdir >>\"

It would also be nice to be able to supply a file of parameters to simplify adding a lot at once.

Which version of the CLI and OS are you using? Did this work in previous versions? Please provide the output of circleci version and circleci diagnostic.

New Feature so not working in any version as far as i know. Version: 0.1.5879+416032d

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: <PATH>/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, <MY NAME>

If you have any questions, feel free to ping us at @CircleCI-Public/x-team.

Thanks for your time!

[marcomorain]

Hi @philnielsen , I'm going to close this issue in favour of #363 – we have opened a JIRA internally to track this work.

[aengelberg]

We fixed #363, but that fix didn't incorporate this feature request yet, so I'm re-opening this ticket.

[furushchev]

@marcomorain Any update on this issue?

[zumby]

Workflows still cannot access << pipeline.xxx >> parameters defined outside of workflows during config validation @aengelberg

commands:
  setup-env:
    steps:
      - run:  echo "export BUILD_NUMBER=<< pipeline.number >>" >> $BASH_ENV
...
workflows:
  SOME-WF:
    jobs:
      - some-job:
          pre-steps:
            - checkout
            - setup-env

Results in:

$ circleci config validate
Error: Error calling workflow: 'SOME-WF'
Error calling job: 'some-job'
Error calling command: 'setup-env
Unknown variable(s): number

[Xavientois]

It looks like the default values added in https://github.com/CircleCI-Public/circleci-cli/pull/405 were removed in https://github.com/CircleCI-Public/circleci-cli/pull/408 and have not been added back.