Add multiple paths property under packageDirectories in sfdx-project.json for MPD and packages over 10,000 files

[jmccotter]

Is your feature request related to a problem? Please describe. I would like to use packageDirectories with pushPackageDirectoriesSequentially to deploy our source in our core package which is over the 10,000 file limit in 2-3 batches. Our current package directory, which predates SFDX, MPD, etc, is broken up into dozens of modules. Because there are many directories and we only need to break up the directories deployed into two or three batches, the packageDirectories command does not work well for our case. In order to use it in its current state, we have to create a new layer of directories purely for representing the batches which is creating a directory structure based on only the push command rather than our business domain.

What are you trying to do I would instead like to be able to specify multiple paths under one package directory like you can do with force:source:deploy -p

Describe the solution you'd like Instead of an sfdx-project.json packageDirectories like this where we create a new folder layer:

{
  ...
  "packageDirectories": [
    {
      "path": "src/Namespace/folder-just-for-SFDX-1",
      "default": true
    },
    {
      "path": "src/Namespace/folder-just-for-SFDX-2",
    },
    {
      "path": "src/Namespace/folder-just-for-SFDX-3",
    }
  ],
  "pushPackageDirectoriesSequentially": true
}

We need to do something like this to replace our current scripts. You could use the existing path property with some logic to differentiate between a single path and list or create a new paths or batchedPaths property or something to indicate the difference:

{
  ...
  "packageDirectories": [
    {
      "paths": ["src/Namespace/domain1", "src/Namespace/domain2", "src/Namespace/domain3"]
      "default": true
    },
    {
      "paths": ["src/Namespace/domain4", "src/Namespace/domain5", "src/Namespace/domain6"]
    },
    {
      "paths": ["src/Namespace/domain7", "src/Namespace/domain8", "src/Namespace/domain9"]
    }
  ],
  "pushPackageDirectoriesSequentially": true
}

Describe alternatives you've considered

• We currently have to do a batch push script using forceignore with the legacy push command to get source tracking to work for development. We use a batch deploy script for CI and cases where we don't need source tracking. The push script isn't ideal since the legacy commands will eventually be discontinued, but the script does not work with the new source tracking and our metadata. I've been working on trying to isolate which files are behaving differently with the new push vs the legacy push causing the issue before submitting an issue for that.
• We could try to use the new deploy command with new source tracking but that would still require creating a script to run the deploy command with the appropriate paths and our own JSON of the paths in each batch. Ideally, that is why we'd like this change to work with the functionality already created without rearranging our directory structure so devs can just run push. Then SFDX via the sfdx-project.json file handles that config and logic.
• We are also in the process of breaking up the monolithic repo to likely move some code out to another package at some point. However it was monolithic because it has existed for longer than SFDX. That work is likely a year or more out to finish and possibly partially dependent on figuring out the issues with 2GP so it could be even further out. For now, we still need a way to address deploying packages with over 10,000 files.

Additional context N/A

[github-actions[bot]]

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.

[uip-robot-zz]

This issue has been linked to a new work item: W-11139620

[mshanemc]

have you tried using the rest metadata api? (sfdx config:set restDeploy=true) ? It had its own issues when it came out, but it does have the advantage of supporting larger deployments.

I'd try that without pushPackageDirectoriesSequentially just to see if it is possible to do it all in one go. If I understand correctly, that might make the whole question moot.

If all that doesn't work, I think introducing a new folder level that represents "how deployments are grouped" would be the best option--pkgDirs affect both sfdx push/pull and 2GP uses those folders so we probably can't easily change the behavior to do groupings inside sfdx-project.json for push without making a mess for 2GP users.

[jmccotter]

@mshanemc We're over the 10,000 files limit rather than the 39 MB limit so I don't think the REST deploy would make a difference. However, there seems to be a bug getting it to actually do REST deploy. It seems to completely disregard the config and environment variable.

% SFDX_REST_DEPLOY=true sfdx force:source:push -f
*** Pushing with SOAP API v52.0 ***
ERROR running force:source:push:  INVALID_OPERATION: Too many files in zip

% sfdx config:set restDeploy=true
=== Set Config

 Name       Value Success 
 ────────── ───── ─────── 
 restDeploy true  true    
% sfdx force:source:push -f 
*** Pushing with SOAP API v52.0 ***
ERROR running force:source:push:  INVALID_OPERATION: Too many files in zip

That's with this SFDX version:

 CLI Version : 
	sfdx-cli/7.156.1

 Architecture: 
	darwin-x64

 Node Version : 
	node-v14.19.3

 Plugin Version: 
	@oclif/plugin-autocomplete 0.3.0 (core)
	@oclif/plugin-commands 1.3.0 (core)
	@oclif/plugin-help 3.3.1 (core)
	@oclif/plugin-not-found 1.2.6 (core)
	@oclif/plugin-plugins 1.10.11 (core)
	@oclif/plugin-update 1.5.0 (core)
	@oclif/plugin-warn-if-update-available 1.7.3 (core)
	@oclif/plugin-which 1.0.4 (core)
	@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
	alias 2.1.0 (core)
	apex 0.13.0 (core)
	auth 2.2.0 (core)
	community 2.0.0 (core)
	config 1.4.12 (core)
	custom-metadata 2.0.0 (core)
	data 2.0.4 (core)
	generator 2.0.1 (core)
	info 2.0.1 (core)
	isvte-sfdx-plugin 1.1.15 (1.1.15)
	limits 2.0.1 (core)
	org 1.13.2 (core)
	salesforce-alm 54.6.0 (core)
	schema 2.1.1 (core)
	sfdx-cli 7.156.1 (core)
	signups 1.2.0 (core)
	source 2.0.2 (core)
	telemetry 2.0.0 (core)
	templates 55.0.0 (core)
	trust 2.0.1 (core)
	user 2.1.0 (core)

 OS and Version: 
	Darwin 20.6.0

I went down to SFDX v 7.155 and it looks like REST deploy is broken or it's related to too many files. I get this same error in lower versions.

% sfdx force:source:push -f
*** Pushing v52.0 metadata with REST API v54.0 connection ***
ERROR running force:source:push:  Metadata API request failed: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.

It doesn't have to be that specific property. It could be a rearrangement of the brackets around the packageDirectories or a property on each packageDirectory. I'm less fond of the latter because it would mean you could put them in a different order than they deploy in using the group number or something like that. With that in mind something like this to maintain backwards compatibility? Then you would just need to flatten the list of lists in all cases except force:source:push when pushPackageDirectoriesSequentially is true.

{
  ...
  "packageDirectories": [
    [
      {
        "path": "src/Namespace/domain1",
        "default": true
      },
      {
        "path": "src/Namespace/domain2"
      },
      {
        "path": "src/Namespace/domain3"
      }
    ],
    [
      {
        "path": "src/Namespace/domain4"
      },
      {
        "path": "src/Namespace/domain5"
      },
      {
        "path": "src/Namespace/domain6"
      }
    ],
  ],
  "pushPackageDirectoriesSequentially": true
}

Alternatively, the latter option which seems a little more fragile to me, but doesn't require rewriting any of the path code except for push when pushPackageDirectoriesSequentially is true:

{
  ...
  "packageDirectories": [
    {
      "path": "src/Namespace/domain1",
      "default": true,
      "pushGroup": 1
    },
    {
      "path": "src/Namespace/domain2",
      "pushGroup": 1
    },
    {
      "path": "src/Namespace/domain3",
      "pushGroup": 1
    },
    {
      "path": "src/Namespace/domain1",
      "default": true,
      "pushGroup": 2
    },
    {
      "path": "src/Namespace/domain2",
      "pushGroup": 2
    },
    {
      "path": "src/Namespace/domain3",
      "pushGroup": 2
    }
  ],
  "pushPackageDirectoriesSequentially": true
}

[github-actions[bot]]

This issue has not received a response in 60 days. It will auto-close in 7 days unless a response is posted.