tektoncd
[TEP-0086] Update PVC/workspace result passing POC
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign alangreene after the PR has been reviewed.
You can assign the PR to them by writing /assign @alangreene in a comment when ready.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
@Tomcli brain-storming ideas here.
An alternative to field storage.
// TaskResult used to describe the results of a task
type TaskResult struct {
// Name the given name
Name string `json:"name"`
// Type is the user-specified type of the result. The possible type
// is currently "string" and will support "array" in following work.
// +optional
Type ResultsType `json:"type,omitempty"`
// Is this a referenced result.
IsRef ResultRefType `json:"isRef,omitempty"` // <<---new field
// Properties is the JSON Schema properties to support key-value pairs results.
// +optional
Properties map[string]PropertySpec `json:"properties,omitempty"`
// Description is a human-readable description of the result
// +optional
Description string `json:"description,omitempty"`
}
// TaskRunResult used to describe the results of a task
type TaskRunResult struct {
// Name the given name
Name string `json:"name"`
// Type is the user-specified type of the result. The possible type
// is currently "string" and will support "array" in following work.
// +optional
Type ResultsType `json:"type,omitempty"`
IsRef bool `json:"isRef,omitempty"` // <<---new field
// Value the given value of the result
Value ResultValue `json:"value"`
}
Reference can be: yes/no/auto, where auto is autodetect whether result should be a reference incase the size exceeds certain threshold. If the result ends up being a reference corresponding taskRun status will contain field IsRef as true.
// Valid ResultsRefType:
const (
ResultsRefTypeYes ResultRefType = "yes"
ResultsRefTypeNo ResultRefType = "no"
ResultsRefTypeAuto ResultRefType = "auto"
)
// AllResultsTypes can be used for ResultsTypes validation.
var AllResultRefTypes = []ResultRefType{ResultsRefTypeYes, ResultsRefTypeNo, ResultsRefTypeAuto}
May be IsRef can also be evaluated by Tasks as tasks.<taskName>.results.<resultName>.isRef and even WhenExpressions can evaluate it.
An example Yaml will look like:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-task-storage
spec:
resources:
requests:
storage: 32M
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: gen-random-text
spec:
workspaces:
- name: result-storages
results:
- name: text
# The result is reference, so it will use workspace to store its contents.
isRef: "yes"
description: large randomly generated string
steps:
- name: add
image: alpine
command: ["/bin/sh", "-c"]
# Generate a very large amount of data(~6MB) to result named text of type reference.
args:
- cat /dev/urandom | base64 | head -c 6000000 | tee $(results.text.path);
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: concat-text
spec:
workspaces:
- name: result-storages
params:
- name: first
description: the first operand
- name: second
description: the second operand
results:
- name: concatenated-text
description: concatenate strings
isRef: "yes"
steps:
- name: concat
image: alpine
command: ["/bin/sh", "-c"]
args:
- cat $(params.first) $(params.second) | tee $(results.concatenated-text.path) ;
---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: concat-text-pipeline
spec:
tasks:
- name: first-task
taskRef:
name: gen-random-text
workspaces:
- name: result-storages
workspace: shared-data
- name: second-task
runAfter:
- first-task
taskRef:
name: gen-random-text
workspaces:
- name: result-storages
workspace: shared-data
- name: third-task
runAfter:
- second-task
params:
- name: first
value: $(tasks.first-task.results.text)
- name: second
value: $(tasks.second-task.results.text)
taskRef:
name: concat-text
workspaces:
- name: result-storages
workspace: shared-data
results:
- name: sum
isRef: "yes" # <--- new field.
description: the concat of all texts
value: $(tasks.third-task.results.concatenated-text)
workspaces:
- name: shared-data
# Specify which workspace should be used for storing results.
resultWorkspace:
name: shared-data
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: concat-text-pipeline-run
spec:
pipelineRef:
name: concat-text-pipeline
workspaces:
- name: shared-data
persistentVolumeClaim:
claimName: shared-task-storage
POC for this approach can be checked out at: https://github.com/ScrapCodes/pipeline/tree/tep-86-alternative
Generated PipelineRun
$ kubectl describe pr
Name: concat-text-pipeline-run
Namespace: default
Labels: tekton.dev/pipeline=concat-text-pipeline
Annotations: <none>
API Version: tekton.dev/v1beta1
Kind: PipelineRun
Metadata:
Creation Timestamp: 2022-08-29T15:23:40Z
Generation: 1
Managed Fields:
API Version: tekton.dev/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:tekton.dev/pipeline:
Manager: controller
Operation: Update
Time: 2022-08-29T15:23:40Z
API Version: tekton.dev/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:pipelineRef:
.:
f:name:
f:workspaces:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-08-29T15:23:40Z
API Version: tekton.dev/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:childReferences:
f:completionTime:
f:conditions:
f:pipelineResults:
f:pipelineSpec:
.:
f:resultWorkspace:
.:
f:name:
f:results:
f:tasks:
f:workspaces:
f:startTime:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-08-29T15:24:02Z
Resource Version: 342728
UID: 35a2a94b-4aa4-4102-9e81-e36b678d64db
Spec:
Pipeline Ref:
Name: concat-text-pipeline
Service Account Name: default
Timeout: 1h0m0s
Workspaces:
Name: shared-data
Persistent Volume Claim:
Claim Name: shared-task-storage
Status:
Child References:
API Version: tekton.dev/v1beta1
Kind: TaskRun
Name: concat-text-pipeline-run-first-task
Pipeline Task Name: first-task
API Version: tekton.dev/v1beta1
Kind: TaskRun
Name: concat-text-pipeline-run-second-task
Pipeline Task Name: second-task
API Version: tekton.dev/v1beta1
Kind: TaskRun
Name: concat-text-pipeline-run-third-task
Pipeline Task Name: third-task
Completion Time: 2022-08-29T15:24:02Z
Conditions:
Last Transition Time: 2022-08-29T15:24:02Z
Message: Tasks Completed: 3 (Failed: 0, Cancelled 0), Skipped: 0
Reason: Succeeded
Status: True
Type: Succeeded
Pipeline Results:
Name: sum
Value: /workspace/result-storages/concat-text-pipeline-run-third-task/concatenated-text # <--- Result value is a reference.
Pipeline Spec:
Result Workspace:
Name: shared-data
Results:
Description: the concat of all texts
Is Ref: yes # <--- new field.
Name: sum
Value: $(tasks.third-task.results.concatenated-text)
Tasks:
Name: first-task
Task Ref:
Kind: Task
Name: gen-random-text
Workspaces:
Name: result-storages
Workspace: shared-data
Name: second-task
Run After:
first-task
Task Ref:
Kind: Task
Name: gen-random-text
Workspaces:
Name: result-storages
Workspace: shared-data
Name: third-task
Params:
Name: first
Value: $(tasks.first-task.results.text)
Name: second
Value: $(tasks.second-task.results.text)
Run After:
second-task
Task Ref:
Kind: Task
Name: concat-text
Workspaces:
Name: result-storages
Workspace: shared-data
Workspaces:
Name: shared-data
Start Time: 2022-08-29T15:23:40Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 66s (x2 over 66s) PipelineRun
Normal Running 66s (x2 over 66s) PipelineRun Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 3, Skipped: 0
Normal Running 54s (x2 over 54s) PipelineRun Tasks Completed: 1 (Failed: 0, Cancelled 0), Incomplete: 2, Skipped: 0
Normal Running 49s (x2 over 49s) PipelineRun Tasks Completed: 2 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0
Normal Succeeded 44s PipelineRun Tasks Completed: 3 (Failed: 0, Cancelled 0), Skipped: 0
@Tomcli @ScrapCodes I'm not sure I understand why a user would want to use a resultworkspace here instead of just using a normal workspace. as far as I can tell, the task still needs to be modified (to add the storage: reference field). Is the idea that the result will be stored in the workspace only if it is too large to be a terminationmessage?
@lbernick , Thank you for taking a look.
ResultWorkspace, is a workspace used by tekton to store the results, when they are large. Since this is an opt-in feature, i.e. workspace needs to be configured and user has to indicate storage:reference then tekton will use the provided workspace to store the value of the results.
Is the idea that the result will be stored in the workspace only if it is too large to be a terminationmessage?
Yes, that is correct.
I rebased this. Since this is an update to one of the alternatives, I think it should be safe to merge, so we don't lose track of any of the proposals.
@jerop @lbernick what do you think about merging the updates to the PVC alternative?
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: afrittoli, jerop
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~teps/OWNERS~~ [afrittoli,jerop]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment