Q: is there a way to invalidate lazy-loaded entity bean field?

[Incanus3]

Hi guys, I have a question regarding lazy-loaded bean fields. Let's say we have these two entities with 1-N relationship:

@Entity
class EaObject(
    @Id
    var id: Long = 0,
) {
    @OneToMany(mappedBy = "eaObject", cascade = [CascadeType.ALL])
    var properties: List<EaObjectProperty> = listOf()
}

@Entity
data class EaObjectProperty(
    @Id
    var id: Long = 0,

    @ManyToOne
    @JoinColumn(name = "OBJECT_ID", nullable = false)
    var eaObject: EaObject,
) {
    @Column
    var name: String? = null,

    @Column
    var value: String? = null,
}

BTW there is a definite ownership here - EaObjectProperty makes no sense without the EaObject, should die with it and will never be transferred to another EaObject.

And then we have a code like this:

    override fun setPropertyValue(eaObject: EaObject, name: String, value: String) {
        val query = QEaObjectProperty(database).where().eaObject.eq(eqObject).name.eq(name)

        if (query.exists()) {
            query.asUpdate().set(QEaObjectProperty._alias.value, value).update()
        } else {
            database.insert(EaObjectProperty(eaObject = eaObject, name = name, value = value))
        }
    }

The problem here is that if eaObject.properties have been access (and thus loaded) before calling this function, after calling it, the attribute is out-of-date, because the eaObject instance has no way of knowing that the value in database has changed. So my question is 1) is there a way to invalidate the eaObject.properties value on the instance so that it will be reloaded on next access and 2) if not, is there some better way to implement the update function which is reasonably efficient and will make the passed-in eaObject instance aware of the change? To be honest, we're currently only using @OneToMany attributes for reading, because 1) we've hit some problems with them not being correctly (from our perspective, not a bug, just different expectations) persisted on change in the past and 2) it's not very efficient to load all related obect if we just want to update one.

[rbygrave]

What database are you using?

If it is Postgres then you could use the InsertOptions.ON_CONFLICT_UPDATE which does this atomically. For non-Postgres you could ponder if you want to use a SQL MERGE.

1. is there a way to invalidate the eaObject.properties value on the instance so that it will be reloaded on next access

You could try using BeanState.setPropertyLoaded("properties", false) which was technically there for another reason (stateless update) but should do what you are looking for.

2. if not, is there some better way to implement the update function

Well Postgres users would look to use InsertOptions.ON_CONFLICT_UPDATE for this case I'd say.

[Incanus3]

hi @rbygrave and thanks for your answer. sadly our clients use different dbs for this, mostly SQL server and oracle. we do use postgres for our internal tables where we can choose, but not for this.

yeah, pg upsert would be great for this, but I wasn't really trying to solve the conditional insert/update case here, we can live with that. our problem was with the already loaded bean having stale state afterwards. I'm not sure how I should get to the BeanState for the bean instance, but will definitely investigate and give it a try.

thanks again, your help is greatly appreciated.

[rob-bygrave]

I'm not sure how I should get to the BeanState for the bean instance,

Something like:

BeanState beanState = database.beanState(myEntityBean);
beanState.setPropertyLoaded("properties", false);

[Incanus3]

Hmm, this doesn't seem to work when db is not default. When I access the property after calling database.beanState(eaObject).setPropertyLoaded("properties", false), I get PersistenceException: No registered default server with this stack trace:

jakarta.persistence.PersistenceException: No registered default server
	at io.ebean.bean.InterceptReadWrite.loadBean(InterceptReadWrite.java:709)
	at io.ebean.bean.InterceptReadWrite.preGetter(InterceptReadWrite.java:836)
	at cz.sentica.qwazar.ea.core.entities.EaObject._ebean_get_properties(EaObject.kt:1)
	at cz.sentica.qwazar.ea.core.entities.EaObject.getProperties(EaObject.kt:262)

even though the entity has the @DbName("ea") annotation and the DatabaseConfig does have the name = "ea" set.

[Incanus3]

This is kinda strange, when I add a break point at the start of InterceptReadWrite.setBeanLoader() and debug the test, this method is correctly called with a LoadBuffer bean loader when fetching the property before the setPropertyLoaded call, after it though, the method is never called again, which is probably why both beanLoader and ebeanServerName are null inside the InterceptReadWrite.loadBean() call.

[Incanus3]

created a minimal test for this, just to be on the same page:

import cz.sentica.qwazar.ea.core.entities.EaObject
import cz.sentica.qwazar.ea.core.entities.EaObjectProperty
import cz.sentica.qwazar.ea.core.entities.query.QEaObject
import cz.sentica.qwazar.ea.core.entities.query.QEaObjectProperty
import io.ebean.DatabaseFactory
import io.ebean.config.CurrentUserProvider
import io.ebean.config.DatabaseConfig
import io.ebean.datasource.DataSourceConfig
import io.kotest.matchers.shouldBe
import org.junit.jupiter.api.Test

class OneToManyPropertyInvalidationTest {
    private val dsConfig = DataSourceConfig().also {
        it.username = "sa"
        it.password = "sa"
        it.url = "jdbc:h2:mem:eadb"
        it.driver = "org.h2.Driver"

        it.addProperty("quoteReturningIdentifiers", false)
        it.addProperty("NON_KEYWORDS", "VALUE")
    }

    private val dbConfig = DatabaseConfig().also {
        it.name = "ea"
        it.isDefaultServer = false
        it.currentUserProvider = CurrentUserProvider { "test" }

        it.isDdlRun = true
        it.isDdlGenerate = true

        it.setDataSourceConfig(dsConfig)
        it.setClasses(
            listOf(
                EaObject::class.java,
                EaObjectProperty::class.java,
            ),
        )
    }

    private val database = DatabaseFactory.create(dbConfig)

    @Test
    fun itWorks() {
        val inserted = EaObject(packageId = 0, name = "test")

        database.save(inserted)
        database.save(EaObjectProperty(eaObject = inserted, property = "test", value = "initial"))

        // this works np
        QEaObjectProperty(database)
            .where().eaObject.eq(inserted).property.eq("test")
            .findOne()!!.value shouldBe "initial"

        val found = QEaObject(database).where().id.eq(inserted.id).findOne()!!

        // these work fine too
        found.name shouldBe "test"
        found.properties.first().property shouldBe "test"
        found.properties.first().value shouldBe "initial"

        QEaObjectProperty(database)
            .where().eaObject.eq(inserted).property.eq("test")
            .asUpdate().set(QEaObjectProperty._alias.value, "updated").update()

        // correctly updated in db
        QEaObjectProperty(database)
            .where().eaObject.eq(inserted).property.eq("test")
            .findOne()!!.value shouldBe "updated"

        // this is out-of-date now, which is logical
        found.properties.first().value shouldBe "initial"

        database.beanState(found).setPropertyLoaded(EaObject::properties.name, false)

        // this fails with
        // jakarta.persistence.PersistenceException: No registered default server
        //     at io.ebean.bean.InterceptReadWrite.loadBean(InterceptReadWrite.java:709)
        //     at io.ebean.bean.InterceptReadWrite.preGetter(InterceptReadWrite.java:836)
        //     at cz.sentica.qwazar.ea.core.entities.EaObject._ebean_get_properties(EaObject.kt:1)
        //     at cz.sentica.qwazar.ea.core.entities.EaObject.getProperties(EaObject.kt:262)
        //     at cz.sentica.qwazar.ea.db.services.OneToManyPropertyInvalidationTest.itWorks(OneToManyPropertyInvalidationTest.kt:90)
        found.properties.first().value shouldBe "updated"
    }
}

[Incanus3]

after I put breakpoints in both InterceptReadWrite.loadBean and .setBeanLoader and found out that

• for reads before the setPropertyLoaded call, setBeanLoader is called inside the first() call, after the found.properties getter returns, loadBean is never called for these
• for reads after the setPropertyLoaded call, loadBean is called inside the found.properties getter call, before it even gets to the first() call, setBeanLoader is never called (since first() is never evaluated)

[Incanus3]

ok, so the difference here is that for those found.properties accesses before the setPropertyLoaded call, including the first one, the isLoadedProperty(propertyIndex) test in InterceptReadWrite.preGetter returns true, so loadBean(propertyIndex) isn't called. this seems surprising to me since we're not specifying the fetch param for the OneToMany annotation on EaObject.properties, the default should be LAZY and if I print the the found.properties before calling first() on it, it does print BeanList<deferred>, so I'd expect isLoadedProperty to return false in the first access.

in the last found.properties access isLoadedProperty(propertyIndex) returns false (as expected), leading to the loadBean() call, which fails because setBeanLoader() hasn't been called.

[Incanus3]

I'm not sure how to continue debugging this - I though I'd try to see where this InterceptReadWrite is actually instantiated and how is it injected into the bean. I found exactly one place where this is instantiated - inside ElementEntityBean constructor, but if I put a breakpoint there, it is never hit throughout the whole test run, so I guess the instantiation must be in some generated code IDEA doesn't know about so it won't show the reference? BTW, is there some simple way to see the generated _ebean methods added to the entities?

[Incanus3]

Hmm, so I found a reference to InterceptReadWrite in the ebean-agent EnhanceConstants file, the C_INTERCEPT_RW is then used by ClassMeta.interceptNew(), which is then called from ConstructorAdapter and InterceptField classes, but I can't even start to decipher the code in those, it seems like some really dark magic I haven't been initiated into.

Anyway, is there something more I can do about this at this point? Should I create a minimal repo with the above test?

[Incanus3]

Now I found a really strange behavior. While reading through BeanList, I noticed there's a reset() method, which sets the list property back to null, which should be the initial, not loaded state (as given by isPopulated()), so instead of the database.beanState(found).setPropertyLoaded(EaObject::properties.name, false) call above, I tried to do

        val beanList = (found.properties as BeanList)
        beanList.reset(beanList.owner(), beanList.propertyName())

this actually seems to be working when debugging and if I pass all the way to the end, the test actually passes, but if I just run the test (without debug), it fails because the last found.properties.first().value still returns "initial".

[Incanus3]

further investigation shows that

• calling database.refresh(found) (which eventually calls BeanDescriptor.resetManyProperties, which calls BeanList.reset()) does work - the next found.properties access returns BeanList<deferred> and when called first() upon, it returns up-to-date value, but this seems a bit heavy for our use-case
• calling database.refreshMany(found, EaObject::properties.name) (which also calls BeanDescriptor.resetManyProperties) doesn't work - found.properties.first() returns "initial", same as with the direct beanList.reset() call above

[Incanus3]

so, as a final note on my investigation of this path (using refresh instead of BeanState.setPropertyLoaded() to make this work), to see why refresh works while BeanList.reset or BeanDescriptor.resetManyProperties or Database.refreshMany does not, I actually copied DefaultBeanLoader.refreshBeanInternal into my code, tried to call it as refreshBeanInternal(bean, SpiQuery.Mode.REFRESH_BEAN, -1) same way as it is called from DefaultServer.refresh (which works) and then tried to prune everything that seemed unneeded while keeping it working (found.properties should return BeanList<deferred> afterwards and found.properties.first().value should be "udpated"). The result is this function:

    private fun DefaultServer.refreshBeanInternal(bean: EntityBean) {
        val desc = this.descriptor(bean.javaClass)
        val pc = DefaultPersistenceContext()
        val id = desc.getId(bean)

        desc.contextPut(pc, id, bean)

        val query = this.createQuery(desc.type())

        // don't collect AutoTune usage profiling information
        // as we just copy the data out of these fetched beans
        // and put the data into the original bean
        query.isUsageProfiling = false
        query.setPersistenceContext(pc)
        query.setMode(SpiQuery.Mode.REFRESH_BEAN)
        query.setId(id)

        // make sure the query doesn't use the cache
        query.setBeanCacheMode(CacheMode.OFF)

        val dbBean = query.findOne() ?: throw EntityNotFoundException(
            "Bean not found during lazy load or refresh. Id:" + id + " type:" + desc.type(),
        )

        desc.resetManyProperties(dbBean)
    }

this does indeed work, but

• I don't really understand how it works - the resetManyProperties isn't called on the original bean, instead the bean is only used to create a context for the query which then retrieves a new instance of the bean, which is then reset, but magically, this also somehow resets the original bean
• sadly, this is still too heavy because
  1. it will probably reload more than just the one property
  2. what I was looking for was a solution that doesn't fire any query immediately, instead only making a query if and when the lazy property is accessed afterwards (and only loading that one)

yes, I know this isn't a solution that you proposed and there's nothing wrong with this not behaving as I hoped, I was just trying to find a different solution since I wasn't successful with the one you proposed (hopefully we can still work on that). on the other hand, I'm still not sure why BeanList.reset or Database.refreshMany(bean, propertyName) doesn't work for this case - it should reset the internal BeanList.list to null which should be the "not-loaded" state.

[Incanus3]

so, going back to the original BeanState.setPropertyLoaded() suggestion, I tried to set the bean loader for the EntityBeanIntercept explicitly, using either SingleBeanLoader.Ref(database) or SingleBeanLoader.Dflt(database) (inspired by the setBeanLoader calls in BeanDescriptor). this does solve the PersistenceException: No registered default server, but then the property behaves similarly as after calling reset on the BeanList - the BeanList does revert to "deferred", but when iterated, it returns the stale results again. I presume in both of these cases this is because those properties are cached by ebean, so I guess for this to work, I would also need to invalidate these records in the cache. is there some accessible API to do this granularly?

[Incanus3]

so, it seems I finally found a workable solution - after resetting the lazy-loaded property I can do

@Suppress("CAST_NEVER_SUCCEEDS")
val intercept = (found as EntityBean)._ebean_getIntercept()
val context = intercept.persistenceContext()

context.clear(EaObjectProperty::class.java, changedPropId)

after doing this, when I iterate over found.properties I finally get up-to-date values. BTW this works for resetting both using BeanState.setPropertyLoaded() and using BeanList.reset(), so I'll use the latter since then I don't need to call EntityBeanIntercept.setBeanLoader() manually to make the property work again. do you think there's a simpler way to do all this or is this the "right way"?

[Incanus3]

just for completeness, the complete current solution is:

foundObject.properties.first().value shouldBe "initial"

val beanList = (foundObject.properties as BeanList)
beanList.reset(beanList.owner(), beanList.propertyName())

@Suppress("CAST_NEVER_SUCCEEDS")
val intercept = (foundObject as EntityBean)._ebean_getIntercept()
val context = intercept.persistenceContext()
context.clear(EaObjectProperty::class.java, changedPropId)

foundObject.properties.first().value shouldBe "updated"

(both assertions pass)

[Incanus3]

hmm, I encountered one (hopefully last) problem with this solution. this works well with a bean instance loaded from database, but if I create a new instance (using the bean class constructor), then save it in the database, then (after inserting/updating some properties related to it using direct query) I reset the property (as above), on the next access I get

Cannot invoke "io.ebean.bean.BeanCollectionLoader.loadMany(io.ebean.bean.BeanCollection, boolean)" because "this.loader" is null
java.lang.NullPointerException: Cannot invoke "io.ebean.bean.BeanCollectionLoader.loadMany(io.ebean.bean.BeanCollection, boolean)" because "this.loader" is null
	at io.ebean.common.AbstractBeanCollection.lazyLoadCollection(AbstractBeanCollection.java:90)
	at io.ebean.common.BeanList.init(BeanList.java:136)
	at io.ebean.common.BeanList.iterator(BeanList.java:322)

I would expect loader to be the server that was used to save the instance, but this is not the case.

BTW, I also tried to call (eaObject.properties as BeanList).setActualList(properties) instead of reset in this case, because I know exactly which properties should be present (and I also skipped the cache invalidation, because there are no "old" properties), but even then, although the list should now be populated and I don't see a reason it should be reloaded, lazyLoadCollection is called resulting in the same error.