forcedotcom
Metadata is not generated correctly when pushing a big object index
Summary
When doing sfdx force:source:push to deploy a project that contains a big object index, the metadata that gets generated is not correct.
However this only seems to be happening in our CI instance, that is using Ubuntu 20.04.4 LTS (Focal Fossa).
To illustrate further, when I set up the SFDX_MDAPI_TEMP_DIR environment variable to debug the metadata that gets generated, this is how the package.xml looks like:
As you can see there is an entry created for an EmailServicesFunction with the name of the big object index, as well as the actual big object index. Just for additional context, we do not have any EmailServicesFunction defined in our project, so we wouldn't expect any to be defined in our definition file.
Navigating to the metadata component that is generated for the EmailServicesFunction:
circleci@08572840bdbb:~/metadataPackage_1652479702973$ ls -l
total 172
drwxr-xr-x 2 circleci circleci 74 May 13 22:08 applications
drwxr-xr-x 7 circleci circleci 187 May 13 22:08 aura
drwxr-xr-x 2 circleci circleci 71 May 13 22:08 cachePartitions
drwxr-xr-x 2 circleci circleci 32768 May 13 22:08 classes
drwxr-xr-x 2 circleci circleci 93 May 13 22:08 components
drwxr-xr-x 2 circleci circleci 112 May 13 22:08 contentassets
drwxr-xr-x 2 circleci circleci 4096 May 13 22:08 customMetadata
drwxr-xr-x 2 circleci circleci 208 May 13 22:08 customPermissions
drwxr-xr-x 2 circleci circleci 48 May 13 22:19 emailservices
drwxr-xr-x 2 circleci circleci 285 May 13 22:08 flexipages
drwxr-xr-x 2 circleci circleci 46 May 13 22:08 flows
drwxr-xr-x 2 circleci circleci 46 May 13 22:08 globalValueSets
drwxr-xr-x 2 circleci circleci 4096 May 13 22:08 layouts
drwxr-xr-x 107 circleci circleci 4096 May 13 22:08 lwc
drwxr-xr-x 2 circleci circleci 57 May 13 22:08 notificationtypes
-rw-r--r-- 1 circleci circleci 103760 May 13 22:08 package.xml
drwxr-xr-x 2 circleci circleci 4096 May 13 22:08 pages
drwxr-xr-x 2 circleci circleci 158 May 13 22:08 permissionsets
circleci@08572840bdbb:~/metadataPackage_1652479702973$ cd emailservices/
circleci@08572840bdbb:~/metadataPackage_1652479702973/emailservices$ ls -l
total 4
-rw-r--r-- 1 circleci circleci 746 May 13 22:08 CM_Document_Audit_Log_Index.indexe
circleci@08572840bdbb:~/metadataPackage_1652479702973/emailservices$ cat CM_Document_Audit_Log_Index.indexe
<?xml version="1.0" encoding="UTF-8"?>
<Index xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>CM_Document_Audit_Log_Index</fullName>
<fields>
<name>CM_Document__c</name>
<sortDirection>ASC</sortDirection>
</fields>
<fields>
<name>CM_Event_Type__c</name>
<sortDirection>ASC</sortDirection>
</fields>
<fields>
<name>CM_Date__c</name>
<sortDirection>DESC</sortDirection>
</fields>
<fields>
<name>CM_Related_Record_Id__c</name>
<sortDirection>ASC</sortDirection>
</fields>
<fields>
<name>CM_Performed_By_Id__c</name>
<sortDirection>ASC</sortDirection>
</fields>
<label>Document Audit Log Index</label>
</Index>
It looks like somewhere along the process sfdx force:source:push is mistakenly parsing the original index as an EmailServicesFunction.
Steps To Reproduce:
Repository to reproduce: N/A (private)
- Deploy the project with a big object index using
sfdx force:source:push.
Expected result
The entire project, including the big object index, gets deployed.
Actual result
The deployment fails since a EmailServicesFunction component that does not exist gets added to the package.xml.
System Information
-
sfdx
{
"cliVersion": "sfdx-cli/7.150.0",
"architecture": "linux-x64",
"nodeVersion": "node-v14.19.2",
"pluginVersions": [
"@oclif/plugin-autocomplete 0.3.0 (core)",
"@oclif/plugin-commands 1.3.0 (core)",
"@oclif/plugin-help 3.3.1 (core)",
"@oclif/plugin-not-found 1.2.6 (core)",
"@oclif/plugin-plugins 1.10.11 (core)",
"@oclif/plugin-update 1.5.0 (core)",
"@oclif/plugin-warn-if-update-available 1.7.3 (core)",
"@oclif/plugin-which 1.0.4 (core)",
"@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
"alias 2.0.0 (core)",
"apex 0.11.0 (core)",
"auth 2.0.2 (core)",
"community 1.1.4 (core)",
"config 1.4.6 (core)",
"custom-metadata 1.1.0 (core)",
"data 0.6.15 (core)",
"generator 2.0.0 (core)",
"info 2.0.0 (core)",
"limits 2.0.0 (core)",
"org 1.12.1 (core)",
"salesforce-alm 54.3.0 (core)",
"schema 2.1.0 (core)",
"sfdx-cli 7.150.0 (core)",
"signups 1.0.0 (core)",
"source 1.9.7 (core)",
"telemetry 1.4.0 (core)",
"templates 54.6.0 (core)",
"trust 1.1.0 (core)",
"user 1.7.1 (core)"
],
"osVersion": "Linux 5.13.0-1021-aws"
}
Additional information
Running sfdx force:source:convert and sfdx force:mdapi:deploy works as expected and does not generate the EmailServicesFunction.
Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I ran into the same issue and spend a significant number of hours to track this down. I hope the following details will help to identify and fix the root cause for this behaviour:
The error first occurred in my build server (CircleCI using Node LTS container).
Steps To Reproduce:
Repository to reproduce: https://github.com/j-fischer/rflib
- Check out the project
- Create new scratch org
- run
sfdx force:source:pushwith a version 7.149 or later
I was also able to reproduce the issue locally (Windows - sfdx v7.150.0) after I ran sfdx force:source:tracking:clear and then created a new scratch org and started pushing the source.
@cdelmoral, this might be helpful to you: I was able to resolve the issue by reverting the CLI version on my build server back to 7.142.0. Just making that switch and no other changes fixed the problem. Thanks for posting the detailed description of the problem here.
Thank you for the detail, helped us after many days of struggling with this.
Just thought I'd check on this item, as it is preventing our team from upgrading past 7.143 as well.
@j-fischer I have been trying to reproduce this error using 7.143.o, with no avail. May I ask a question regarding the MD file name of the big index. Is the .index portion of the name .index. or .indexe.? If it is the latter, could you try renaming it to .index. and the try the push again using 7.143.0?
@peternhale I was not able to reproduce the issues in 7.143.0, but I was able to reproduce it with the new push implementation in 7.149.0.
Checking the name of the index, it used the .indexe suffix. I renamed it as you suggested and tried again, which fixed the issue.
Last, I tried everything with version 7.161.0 and it was a success ass well. So, renaming the file name did the trick. Thanks for the suggestion.
@j-fischer thanks for getting back to me and I am glad the renaming workaround got you past the error.
@peternhale Renaming indexe to index also worked for me when doing force:source:push.
However this does not seem to work when creating a new package using force:package:version:create. Is the advice to keep renaming indexes depending on if we are pushing source or creating a package?
In any case, it seems like this issue should be reopened as there is a bug somewhere in the CLI, whether that is force:source:pull pulling the index with the wrong extension, or force:source:push not being able to push a valid extension.
Thanks for checking the package creation, @cdelmoral , I would certainly be running into this issue in the future. And thanks for continuing to look into this, @peternhale .
Hi @peternhale Same issue here. only renaming indexe to index in the BO metadata, allows a successful push to the scratch If i try to restore to original indexe file name, push error appears again. i need to generate a new version package, but not sure in which version to do, because of previous comments
monitor git:(feature/CHN4-9024-flushThroughApi) ✗ sfdx force:source:status
Source Status
IGNORED STATE FULL NAME TYPE PROJECT PATH
─────── ───────────── ─────────────────────────── ───────────────────── ────────────────────────────────────────────────────────────────────────────
false Local Add UUid EmailServicesFunction force-app/main/default/objects/AteneaEventBO__b/indexes/UUid.indexe-meta.xml
=== Component Failures [2]
Type Name Problem
───── ───────────────────── ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Error AteneaEventBO__b.UUid DELETE isn't supported for Big Objects indexes: UUid
Error UUid An object 'UUid' of type EmailServicesFunction was named in package.xml, but was not found in zipped directory
this is my current sfdx version:
➜ monitor git:(feature/CHN4-9024-flushThroughApi) ✗ sfdx version --verbose CLI Version: sfdx-cli/7.167.2
Architecture: darwin-x64
Node Version: node-v16.17.0
Plugin Version: @oclif/plugin-autocomplete 1.3.0 (core) @oclif/plugin-commands 2.2.0 (core) @oclif/plugin-help 5.1.12 (core) @oclif/plugin-not-found 2.3.1 (core) @oclif/plugin-plugins 2.1.0 (core) @oclif/plugin-update 3.0.0 (core) @oclif/plugin-version 1.1.2 (core) @oclif/plugin-warn-if-update-available 2.0.4 (core) @oclif/plugin-which 2.1.0 (core) alias 2.1.0 (core) apex 1.2.0 (core) auth 2.2.3 (core) community 2.0.1 (core) config 1.4.19 (core) custom-metadata 2.0.0 (core) data 2.1.2 (core) generator 2.0.2 (core) info 2.0.1 (core) limits 2.0.1 (core) org 2.2.0 (core) packaging 1.6.0 (core) schema 2.1.1 (core) signups 1.2.0 (core) source 2.0.13 (core) telemetry 2.0.0 (core) templates 55.1.0 (core) trust 2.0.3 (core) user 2.1.0 (core) @salesforce/sfdx-plugin-lwc-test 1.0.1 (core) salesforce-alm 54.8.1 (core) sfdx-mohanc-plugins 0.0.202 (user)
OS and Version: Darwin 21.6.0
Shell: zsh
Root Path: /Users/clemente.sanchez/.local/share/sfdx/client/7.167.2-93d663e
Best Regards Clemente
@peternhale any update on this? I would like to publish a new version of my package but the command fails due to this error.
ERROR running force:package:version:create: Unexpected file found in package directory: C:\Users\Johannes\Projects\Salesforce\rflib\rflib\main\default\objects\rflib_Logs_Archive__b\indexes\rflib_Log_Index.index-meta.xml
Warning: Command failed: sfdx force:package:version:create --path rflib --package RFLIB --installationkeybypass -c --wait 30
ERROR running force:package:version:create: Unexpected file found in package directory: C:\Users\Johannes\Projects\Salesforce\rflib\rflib\main\default\objects\rflib_Logs_Archive__b\indexes\rflib_Log_Index.index-meta.xml
Is there any ETA when we can expect this to be fixed?
Thanks, Johannes
hey @j-fischer sorry for the radio silence, can you try your packaging workflows using the beta packaging commands (force:package(1):beta:*)? The beta commands are using our standardized library and should be able to handle the MD edge cases much better
@WillieRuemmele Thanks, will give beta commands a try and report back during my next packaging cycle. Cheers
@WillieRuemmele I finally managed to find the time to work on another version of my library creating another package version. Using the beta commands (2nd gen packaging), I was able to publish the big object without having to rename the index file name.
Looks like the beta commands are the way to go. Thanks for the help and apologies that it took that long to confirm.
Cheers
Hey is there any update for the original issue? I am still getting the problem with push throwing the EmailServicesFunction error because of our big index objects.
Error Index An object 'Index' of type EmailServicesFunction was named in package.xml, but was not found in zipped directory
I have tried the workaround with renaming indexe to index and that has fixed the push command to our local scratch orgs but it has caused issues down the line into our sandbox orgs with a deploy.
ERROR running force:source:deploy: Unexpected file found in package directory: <directory>/<big-object>/indexes/Index.index-meta.xml.
Here is the output of sfdx doctor
{
"versionDetail": {
"cliVersion": "sfdx-cli/7.182.1",
"architecture": "darwin-x64",
"nodeVersion": "node-v18.12.1",
"pluginVersions": [
"@oclif/plugin-autocomplete 1.3.6 (core)",
"@oclif/plugin-commands 2.2.1 (core)",
"@oclif/plugin-help 5.1.19 (core)",
"@oclif/plugin-not-found 2.3.9 (core)",
"@oclif/plugin-plugins 2.1.8 (core)",
"@oclif/plugin-search 0.0.4 (core)",
"@oclif/plugin-update 3.0.7 (core)",
"@oclif/plugin-version 1.1.3 (core)",
"@oclif/plugin-warn-if-update-available 2.0.15 (core)",
"@oclif/plugin-which 2.2.2 (core)",
"alias 2.1.12 (core)",
"apex 1.3.0 (core)",
"auth 2.3.3 (core)",
"community 2.0.18 (core)",
"config 1.4.22 (core)",
"custom-metadata 2.0.5 (core)",
"data 2.1.17 (core)",
"generator 2.0.13 (core)",
"info 2.2.13 (core)",
"limits 2.1.3 (core)",
"org 2.2.19 (core)",
"packaging 1.10.10 (core)",
"schema 2.1.19 (core)",
"signups 1.2.12 (core)",
"source 2.3.8 (core)",
"telemetry 2.0.5 (core)",
"templates 55.1.0 (core)",
"trust 2.2.2 (core)",
"user 2.1.22 (core)",
"@salesforce/sfdx-plugin-lwc-test 1.0.1 (core)",
"@salesforce/sfdx-scanner 3.7.1 (user)",
"isvte-sfdx-plugin 1.1.16 (user)",
"salesforce-alm 54.8.4 (core)",
"texei-sfdx-plugin 1.17.1 (user)"
],
"osVersion": "Darwin 22.2.0",
"shell": "zsh",
"rootPath": "/Users/greg/.local/share/sfdx/client/7.182.1-dad7ae3"
},
"sfdxEnvVars": [
[
"SFDX_REDIRECTED",
"1"
],
[
"SFDX_BINPATH",
"/Users/greg/.local/share/sfdx/client/bin/sfdx"
],
[
"SFDX_AUTOUPDATE_DISABLE",
"true"
],
[
"SFDX_DISABLE_AUTOUPDATE",
"true"
],
[
"SFDX_UPDATE_INSTRUCTIONS",
"Use \"npm update --global sfdx-cli\" to update npm-based installations."
]
],
"sfEnvVars": [],
"cliConfig": {
"_base": "@oclif/[email protected]",
"debug": 0,
"topicSeparator": " ",
"warned": false,
"commandPermutations": {
"validPermutations": {}
},
"topicPermutations": {
"validPermutations": {}
},
"_commands": {},
"_topics": {},
"root": "/Users/greg/.local/share/sfdx/client/7.182.1-dad7ae3",
"name": "sfdx-cli",
"version": "7.182.1",
"channel": "stable",
"valid": true,
"arch": "x64",
"platform": "darwin",
"windows": false,
"bin": "sfdx",
"dirname": "sfdx",
"flexibleTaxonomy": true,
"userAgent": "sfdx-cli/7.182.1 darwin-x64 node-v18.12.1",
"shell": "zsh",
"home": "/Users/greg",
"cacheDir": "/Users/greg/Library/Caches/sfdx",
"configDir": "/Users/greg/.config/sfdx",
"dataDir": "/Users/greg/.local/share/sfdx",
"errlog": "/Users/greg/Library/Caches/sfdx/error.log",
"binPath": "/Users/greg/.local/share/sfdx/client/bin/sfdx",
"nodeEngine": ">=14.0.0"
},
"pluginSpecificData": {
"@salesforce/plugin-source": [
{
"sourceApiVersion": "56.0",
"targetDevHubApiVersion": "56.0",
"targetOrgApiVersion": "56.0"
}
]
},
"diagnosticResults": [
{
"testName": "salesforcedx plugin not installed",
"status": "pass"
},
{
"testName": "no linked plugins",
"status": "pass"
},
{
"testName": "[@salesforce/plugin-source] sourceApiVersion matches apiVersion",
"status": "pass"
},
{
"testName": "[@salesforce/plugin-source] default target DevHub max apiVersion matches default target org max apiVersion",
"status": "pass"
},
{
"testName": "[@salesforce/plugin-source] sourceApiVersion matches default target org max apiVersion",
"status": "pass"
},
{
"testName": "using latest or latest-rc CLI version",
"status": "pass"
}
],
"suggestions": [
"Check https://github.com/forcedotcom/cli/issues for CLI issues posted by the community.",
"Check http://status.salesforce.com for general Salesforce availability and performance."
],
"logFilePaths": []
}
Thanks, Greg
i'm having the exact same error as gstephens123
Error Index An object 'Index' of type EmailServicesFunction was named in package.xml, but was not found in zipped directory
i'm able to workaround using sfdx force:source:legacy:push, but not ideal.
cheers
Hi @gilgrencho @gstephens123 and others... I was able to partially recreate the issue, but not identically
➜ oc
Successfully created scratch org: 00D8G000000HbwyUAC, username: [email protected].
➜ RGD hub:(GLOBAL - DevHub) scratch:([email protected])
➜ status
Source Status
STATE FULL NAME TYPE PROJECT PATH
───────── ───────── ─────────────────────────── ──────────────────────────────────────────────────────────────────────────────────────────────────
Local Add test__b CustomObject force-app/main/default/objects/test__b/fields/field__c.field-meta.xml
Local Add test__b CustomObject force-app/main/default/objects/test__b/indexes/BOI.index-meta.xml
Local Add test__b CustomObject force-app/main/default/objects/test__b/test__b.object-meta.xml
Local Add Admin Profile force-app/main/default/profiles/Admin.profile-meta.xml
Local Add New_Graph RelationshipGraphDefinition force-app/main/default/relationshipGraphDefinitions/New_Graph.relationshipGraphDefinition-meta.xml
➜ RGD hub:(GLOBAL - DevHub) scratch:([email protected])
➜ push
Pushing v55.0 metadata to [email protected] using the v56.0 SOAP API
DEPLOY PROGRESS | ████████████████████████████████████████ | 5/5 Components
Updating source tracking... done
=== Pushed Source
STATE FULL NAME TYPE PROJECT PATH
─────── ──────────────── ─────────────────────────── ──────────────────────────────────────────────────────────────────────────────────────────────────
Created test__b.field__c CustomField force-app/main/default/objects/test__b/fields/field__c.field-meta.xml
Created test__b.field__c CustomField force-app/main/default/objects/test__b/fields/field__c.field-meta.xml
Created test__b CustomObject force-app/main/default/objects/test__b/test__b.object-meta.xml
Created New_Graph RelationshipGraphDefinition force-app/main/default/relationshipGraphDefinitions/New_Graph.relationshipGraphDefinition-meta.xml
=== Component Failures [1]
Type Name Problem
───── ─────────── ─────────────────────────────────────────────────────────────────────────────────────────────────────
Error test__b.BOI An object 'test__b.BOI' of type Index was named in package.xml, but was not found in zipped directory
I created a scratch org, and created the big object, added a custom field, and added an index on that object, I pulled that down and everything succeeded. I then created a new scratch org, and when I pushed, I saw the above error, but it's not matching your's. What CLI versions are you experiencing this on? I'm on 7.184.2
@WillieRuemmele, i guess we are running the same cli version: sfdx-cli/7.184.2 darwin-x64 node-v18.12.1
let-me know what details you might need, my big object has 10 fields and 1 index
Thanks for helping @gilgrencho - I've seen multiple people report that changing .indexe to .index seems to fix the problem, I'm just stumped why when you pull it it down it's written as .indexe, mine was written as .index...
Maybe try moving that Big Object to a temporary directory outside of the project, and running the pull again and seeing if it regenerates it in a new format, if it doesn't work you can always restore the BO from that saved location
EDIT: I was working with another type, and created an org and was able to succesfully push the BO and Index to the org without any problems :hmm:
Created test__b.index Index force-app/main/default/objects/test__b/indexes/index.index-meta.xml
Created test__b.index Index force-app/main/default/objects/test__b/indexes/index.index-meta.xml
@WillieRuemmele sorry for late reply, i wasn't able to sort the metadata with push/pull. Manually remove the extra e ended up working.
Thank you so much!
The code that was producing the extra e no longer exists. If you have a very old project with indexe use the workarounds above to be able to deploy.
