multi-environment pipelines

[dlpzx]

We want isolate our CICD resources, our non-production workloads and our production workloads in different AWS accounts. Why? because we want our developers to develop without interfering in the test infrastructure tested and/or validated externally (this might be a legal requirement) and we want a complete different account for the production that cannot be accessible by any of the previous personas. This way we avoid human intervention and subsequent manual errors. Pipelines are linked with datasets that are used in environments. Ideally, we would be able to have a clear view of where our data pipelines are deployed and what datasets they are using.

Describe alternatives you've considered

Alternatives	Description	User Experience	Coding complexity
Projects	We introduce a new construct, Projects, projects correspond with a real project, a use-case. A use-case involves multiple Environments, datasets and pipelinesLike in a real use-case, It can also include documentation about the use-case.	A user with Project permissions creates the Project and invites environment-teams to the Project. All users invited can see the metadata of the project, but they will have access only to the environments that they belong to. The invited environments are assigned a type of environment (DEV,TEST,PROD). We can tag datasets with a Project name and allow to be searched in the catalog by Project. When users create a pipelinethey can select a Project and multiple environments and datasets.	- new UI views for Construct- new methods- new RDS(many can be copied/pasted from other objects, not that difficult as it seems)- difficulty in the part of selecting multiple environments as input or multiple datasets as input from frontend side
Organizations	We re-use the concept of Organizations, and allow environments to be linked together in a logical way (tag the environment with dev, test, prod..). When we create a pipeline we select the Organization, not the environment.	A user creates a pipeline and selects the Organization, the organization becomes more like a "domain"	- modify UI and calls
Pipelines	We allow the pipelines to select multiple environments, at least as metadata.	When creating the pipeline, the user can select multiple environments	- modify UI and RDS