[Support] MongoDB output generates error with update-one + upsert

[willwolfram18]

Description

When using MongoDB as an output, and configuring it with operation: update-one and upsert: true, the benthos application produces an error with the following details:

level=error msg="Failed to send message to mongodb: update document must contain key beginning with '$'" @service=benthos label="" path=root.output

This prevents taking messages from a given benthos input and upserting them into MongoDB as an update operation to only overwrite specific fields.

Alternative considered

• Use operation: replace-one - this isn't ideal, as I may want to only update a specific portion of the document from a given input, and replace would overwrite the existing document

Steps to Reproduce

I've created a repository with a minimal example that recreates the issue: https://github.com/willwolfram18/benthos-mongodb-update-error The example uses docker-compose with a Kafka input and MongoDB output.

1. Open a command prompt, navigate to the example and start the applications using docker-compose -d --build
2. Execute docker-compose logs --follow consumer and keep the command prompt open.
3. Navigate to Control Center (http://localhost:9021/) > controlcenter.cluster > topics > example-topic
4. Select messages and click "Produce message" providing a message with the following schema for a value and publish the message:

{
  "message_id": "id-1",
  "message_name": "example name"
}

5. Once the message is published, you should start seeing the error message in the command prompt appear.

[Jeffail]

Hey @willwolfram18, I'm not too familiar with the mongodb components but this sounds like something we could and ideally should support. It looks as though this issue comes back from the official mongodb client library, but I'm assuming we simply need to specify a different type of operation rather than contribute upstream.

[willwolfram18]

Tbf, I'm not super familiar with Golang but I would imagine you're right: this doesn't seem like it's an upstream change but rather an issue in how the MongoDB query is being generated. Based on the error message, it seems like it's a small syntax error where the update query that's produced is something like { set: { name: "<name value>" } }, but MongoDB syntax is expecting { $set: { name: "<name value>" } }.

But since I'm not super familiar with the MongoDB internals of benthos I couldn't really dive in and figure out exactly how it I could debug and investigate this issue further ☹️

Docs for reference: https://www.mongodb.com/docs/manual/reference/operator/update/set/

[mihaitodor]

I should've looked at this sooner, but it fell through the cracks... Hope you're not blocked by this @willwolfram18. I did find this issue here which explains why this $set is needed... This prompted me to play around with the existing implementation and it looks like the test here does cover this scenario, but the docs don't... The reason it's done this way is to let the user specify the operator they want to use. Here's a working example:

input:
  generate:
    count: 1
    mapping: |
      root = {
        "id": "id-1",
        "name": "example name"
      }
    interval: 1s

output:
  mongodb:
    url: mongodb://localhost:27017
    database: food
    collection: fruits
    operation: update-one
    upsert: true
    write_concern:
      w_timeout: 5000ms
    filter_map: root._id = this.id
    document_map: |
      {"$set": {"name": this.name}}

As a test, you can replace $set with $setOnInsert and observe that it doesn't change the record if it already exists.

[akshay-pramod]

Thank you @mihaitodor!

I had a similar issue and you're solution has resolved it.

[willwolfram18]

Thanks for the explanation @mihaitodor ! My team finally revisited this and was able to confirm your suggestion works as expected.

Thanks!