Support for batching programatic migrations

[mbossenbroek]

Is there support for batching transactions where you specify a norm as a function that returns a set of transaction data? In a lot of cases for us, the migration data is far too large to transact as a single transaction.

This is what I'm using today to partition it into many smaller transactions:

(defn run-update
  "Run an update query; likely related to a schema migration"
  [connection q]
  (let [db (d/db connection)]
    (->>
      (q db)
      (partition-all 100)
      (reduce (fn [running-total tx-data]
                (let [total (+ (count tx-data) running-total)]
                  (log/infof "Running update tx %d" total)
                  (d/transact connection tx-data)
                  total)) 0))))

[qtrfeast]

I don't believe there is currently. There's good documentation on pipelining transactions. I'm not able to take this on right now, but I'm open to others trying.