cli icon indicating copy to clipboard operation
cli copied to clipboard
verified publisher icon forcedotcom

sf agent generate agent-spec always produces empty topics array regardless of configuration

Open hangingman opened this issue 7 months ago • 8 comments

Summary

The sf agent generate agent-spec command consistently fails to generate topics, resulting in topics: [] in the output YAML file, regardless of agent properties provided. This occurs even with simple, standard agent configurations and affects both interactive and flag-based command execution.

Steps To Reproduce

Environment Setup:

  1. Create SFDX project with sf project generate --name agentforce-test
  2. Authenticate to Developer Edition org with Einstein/Agentforce enabled
  3. Ensure CLI is updated to latest version
✗ sf version
@salesforce/cli/2.89.8 darwin-x64 node-v23.11.1

Reproduction Steps:

Test 1: Basic Interactive Execution

sf agent generate agent-spec --target-org DevOrg

Interactive inputs:

  • Type of agent: internal
  • Company name: Test Company
  • Company description: Salesforce consulting company
  • Role of agent: Customer support agent

result

agentType: internal
companyName: Test Company
companyDescription: Salesforce consulting company
role: Customer support agent
maxNumOfTopics: 5
enrichLogs: false
tone: casual
topics: []

Test 2: Flag-based Execution (Simple)

sf agent generate agent-spec --type internal \
  --company-name "Test Company" \
  --company-description "Salesforce consulting company" \
  --role "Customer support agent" \
  --max-topics 1 --tone formal \
  --target-org DevOrg

result

agentType: internal
companyName: Test Company
companyDescription: Salesforce consulting company
role: Customer support agent
maxNumOfTopics: 1
enrichLogs: false
tone: formal
topics: []

Expected result

According to the documentation at https://developer.salesforce.com/docs/einstein/genai/guide/agent-dx-generate-agent-spec.html:

The spec file also contains an LLM-generated list of topics based on this information... The bottom part contains the LLM-generated list of topics.

The generated specs/agentSpec.yaml should contain topics like:

topics:
  - name: Guest Check-In Automation
    description: Streamline the guest check-in process using AI.
  - name: Personalized Guest Services
    description: Provide tailored recommendations and services.
  - name: Predictive Maintenance
    description: Use AI to forecast and prevent equipment failures.

Actual result

All executions consistently produce the same result with empty topics:

Generated file content:

agentType: internal
companyName: Test Company
companyDescription: Salesforce consulting company
role: Customer support agent
maxNumOfTopics: 1
enrichLogs: false
tone: formal
topics: []  # ← Always empty regardless of configuration

Additional information

Plugin Version Warning: Every execution shows:

Warning: Plugin @salesforce/plugin-agent (1.22.2) differs from the version specified by sf (1.21.1)

Attempted Variations:

  • Different --max-topics values (1, 2, 3)
  • Various agent types and descriptions
  • Both new spec creation and existing spec refinement

Consistent Behavior:

  • Command completes without errors
  • All agent properties are correctly saved
  • Only topics: [] array remains consistently empty
  • Issue persists across multiple days and CLI sessions

System Information

Shell: zsh Platform: macOS

✗ sf version --verbose --json
{
  "architecture": "darwin-x64",
  "cliVersion": "@salesforce/cli/2.89.8",
  "nodeVersion": "node-v23.11.1",
  "osVersion": "Darwin 22.6.0",
  "rootPath": "/usr/local/Cellar/sf/2.89.8/libexec/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.2.28 (core)",
    "@oclif/plugin-commands 4.1.25 (core)",
    "@oclif/plugin-help 6.2.28 (core)",
    "@oclif/plugin-not-found 3.2.51 (core)",
    "@oclif/plugin-plugins 5.4.37 (core)",
    "@oclif/plugin-search 1.2.24 (core)",
    "@oclif/plugin-update 4.6.39 (core)",
    "@oclif/plugin-version 2.2.28 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.39 (core)",
    "@oclif/plugin-which 3.2.35 (core)",
    "@salesforce/cli 2.89.8 (core)",
    "agent 1.22.2 (user) published 6 days ago (Wed May 21 2025)",
    "apex 3.6.17 (core)",
    "api 1.3.3 (core)",
    "auth 3.6.120 (core)",
    "data 4.0.32 (core)",
    "deploy-retrieve 3.22.10 (core)",
    "info 3.4.60 (core)",
    "limits 3.3.54 (core)",
    "marketplace 1.3.8 (core)",
    "org 5.7.8 (core)",
    "packaging 2.13.3 (core)",
    "schema 3.3.62 (core)",
    "settings 2.4.27 (core)",
    "sobject 1.4.56 (core)",
    "telemetry 3.6.41 (core)",
    "templates 56.3.47 (core)",
    "trust 3.7.89 (core)",
    "user 3.6.21 (core)"
  ]
}

Development Org Configuration:

  • Type: Developer Edition
  • Einstein: Enabled
  • Agentforce: Enabled
  • Data Cloud: Created and active

Impact: This issue blocks the entire Agentforce CLI workflow as the subsequent sf agent create command fails with "No topics are associated with agent. Skip saving." when attempting to create an agent from the spec file.

hangingman avatar May 27 '25 09:05 hangingman

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

github-actions[bot] avatar May 27 '25 09:05 github-actions[bot]

Thank you for the thorough issue report. Unfortunately, using the same plugin version and the exact repro steps I don't see the same behavior. My agent spec has this content:

agentType: internal
companyName: Test Company
companyDescription: Salesforce consulting company
role: Customer support agent
maxNumOfTopics: 5
enrichLogs: false
tone: casual
topics:
  - name: Customer Issue Resolution
    description: Assist in resolving customer issues using Salesforce.
  - name: Knowledge Base Management
    description: Update and maintain the internal knowledge base.
  - name: Case Routing Optimization
    description: Automate case routing for efficient resolution.
  - name: Customer Interaction History
    description: Track and analyze customer interactions.
  - name: Performance Metrics Reporting
    description: Generate reports on support team performance.

I suspect that something might not be configured in the org or with user permissions. Typically, that would result in an error reported by the server side API that the CLI is calling. Are you able to create an agent with generated topics using the org UI in that same org? If you use the above as your agent spec are you able to create an agent in the org (or preview agent creation) using the CLI command? If you append --dev-debug there could be some debug output that provides a clue as well.

shetzel avatar May 27 '25 21:05 shetzel

@shetzel

Thank you for your reply. I tried it today again, but no succeed.

I suspect that something might not be configured in the org or with user permissions. Typically, that would result in an error reported by the server side API that the CLI is calling. Are you able to create an agent with generated topics using the org UI in that same org?

I tried creating an agent with using org UI in the Developer edition organization. It succeeded. From UI, topics were easily generated by any keywords. (It seems the salesforce UI is using aura API /aura for generating topics. I saw it from browser tool.)

If you use the above as your agent spec are you able to create an agent in the org (or preview agent creation) using the CLI command? If you append --dev-debug there could be some debug output that provides a clue as well.

I tried the option --dev-debug, then it shows calling Connect API endpoint. This indicates the API return "isSuccess: true" while it contains empty topics. But I guess this is unpublished API, so no way to debug. I wish the API should return error if no topics generated.

[11:52:04.532] DEBUG (sf:MaybeMock): Making POST request to /connect/ai-assist/draft-agent-topics
[11:52:04.532] DEBUG (sf:connection): request
    method: "POST"
    url: "/connect/ai-assist/draft-agent-topics"
    body: "{\"agentType\":\"internal\",\"generationInfo\":{\"defaultInfo\":{\"role\":\"Customer support agent\",\"companyName\":\"Test Company\",\"companyDescription\":\"Salesforce consulting company\"}},\"generationSettings\":{\"maxNumOfTopics\":1}}"
    headers: {
      "content-type": "application/json",
      "user-agent": "sfdx toolbelt:"
    }
{
  "status": 0,
  "result": {
    "isSuccess": true,
    "specPath": "/path/to/specs/agentSpec.yaml",
    "agentType": "internal",
    "companyName": "Test Company",
    "companyDescription": "Salesforce consulting company",
    "role": "Customer support agent",
    "maxNumOfTopics": 1,
    "topics": [],
    "enrichLogs": false,
    "tone": "casual"
  },
  "warnings": []
}

hangingman avatar May 29 '25 05:05 hangingman

A few more things to try:

  1. If you haven't already, can you try --max-topics 5
  2. To see the raw API response can you modify this file: /Users/<your_username>/.local/share/sf/node_modules/@salesforce/plugin-agent/node_modules/@salesforce/agents/lib/agent.js on line 250 to be: console.dir(response, { depth: 10 };

If there is no agent.js file in that location you can run sf plugins inspect agent and look at the location property to modify the path above.

shetzel avatar May 29 '25 17:05 shetzel

@shetzel

  1. If you haven't already, can you try --max-topics 5

I've tried it. But, still not working.

  1. To see the raw API response can you modify this file: /Users/<your_username>/.local/share/sf/node_modules/@salesforce/plugin-agent/node_modules/@salesforce/agents/lib/agent.js on line 250 to be: console.dir(response, { depth: 10 };

If there is no agent.js file in that location you can run sf plugins inspect agent and look at the location property to modify the path above.

Added debug log.

[17:51:04.072] DEBUG (sf:connection): API version cache last checked on 5/30/2025, 5:42:48 PM (now is 5/30/2025, 5:51:04 PM)
[17:51:04.072] DEBUG (sf:connection): Using cached API version: 63.0
[17:51:04.072] DEBUG (sf:connection): Connection created with apiVersion 63.0
...
[17:51:41.142] DEBUG (sf:MaybeMock): Making POST request to /connect/ai-assist/draft-agent-topics
[17:51:41.142] DEBUG (sf:connection): request
    method: "POST"
    url: "/connect/ai-assist/draft-agent-topics"
    body: "{\"agentType\":\"customer\",\"generationInfo\":{\"defaultInfo\":{\"role\":\"Manage the resort.\",\"companyName\":\"Coral Cloud Resorts\",\"companyDescription\":\"Provide a luxury experience.\"}},\"generationSettings\":{\"maxNumOfTopics\":5}}"
    headers: {
      "content-type": "application/json",
      "user-agent": "sfdx toolbelt:"
{ isSuccess: true, topicDrafts: [] }
Creating agent spec... done

Response format is changed. Maybe it caused by I updated salesforce cli because CLI is not latest version.

✗ sf version
@salesforce/cli/2.90.4 darwin-x64 node-v22.15.0

hangingman avatar May 30 '25 09:05 hangingman

At this point I suggest contacting Customer Support and they can help troubleshoot at the org level, or maybe try with a completely different DE org. I don't see anything wrong with what the CLI code is doing but the API is not sending back a good response.

shetzel avatar May 30 '25 21:05 shetzel

We have determined that the issue you reported exists in code owned by another team that uses only the official support channels. To ensure that your issue is addressed, open an official Salesforce customer support ticket with a link to this issue. We encourage anyone experiencing this issue to do the same to increase the priority. We will keep this issue open for the community to collaborate on.

github-actions[bot] avatar May 30 '25 21:05 github-actions[bot]

@shetzel Thank you, I'll try contacting Customer Support

hangingman avatar Jun 02 '25 00:06 hangingman