adlnet
Extension refinements
In § 5.3 Extensions, the spec declares:
An LRS MUST NOT reject a Statement based on the values of the extensions map.
However, it seems to me that if extensions are to be useful, beyond the required:
The controller of an IRL extension key SHOULD make a human-readable description of the intended meaning of the extension supported by the IRL accessible at the IRL.
Extensions should provide a schema (JSON-Schema or what have you) at the IRL so that LRSs can learn to validate them. Iff an extension provides a schema for itself, and a malformed extension pointing to this IRL violates the schema, the LRS SHOULD be able to reject the Statement for a malformed nested value like any other.
This may be a 2.0 thing, but it's something I'm actively pursuing and think needs to be discussed if extensions and/or xAPI are to be useful (e.g., the only way to provide indirect objects {actor: foo :verb gave :object {:id //book {:extensions [{://indirect-object {:preposition :to :objectType Agent :name :bar}}]}}}
That's messy and incorrect but I think illustrative... if the "indirect-object" extension schema requires a String value for a "preposition" key, and it's absent, the Statement should be rejectable iff the LRS knows it's wrong.
If the LRS is extension-ignorant, it can accept what it wants, but then an extension-aware LRS might not accept some of its stored statements.
I think we're on the cusp of something great, but extensions are kludgy to begin with, and we need a way of either reigning them in or expanding the semantic scope of the Statement object itself.
I can see why rejecting statements based on incorrectly formed extensions is useful in the same way that rejection incorrectly formed statement properties is useful. It gives the activity provider instant feedback that something is wrong so the problem can be dealt with rather than waiting for the error to appear at report-time.
On the other hand, there's two problems that would arise if some LRSs accept statements with badly populated extensions and some don't:
- Activity Providers that work with one LRS will break with another. This is the sort of thing that leads to clients telling their LRS vendor "Well, it works with such and such LRS, you need to fix it.". That leads to weird hacks, which lead to hate, anger and suffering; it's a dark path.
- An LRS might accept a statement and then find that another LRS rejects it. By this point though, it has no way to feed that error back to the original creator of the statement.
These problems could occur if:
- We allow this as an optional behaviour and some LRSs implement and some don't.
- The extension schema is unavailable to the original LRS accepting the statement and becomes available later when the statement is forwarded. This means that making the behaviour required is still not a good solution.
Thinking about how we can avoid those problems, I wonder if this could be an LRS behaviour that is only applied if specifically requested by the activity provider via a header parameter. That way activity providers following a particular profile with extensions could ensure their statements are validated by passing that parameter, but an LRS forwarding statements would not pass that parameter. It'd be backwards compatible with activity providers who don't pass that parameter too. Think of it a bit like the strict mode in javascript and html.
The process would be:
- Activity provider fetches the about resource to determine that the LRS supports this feature.
- If yes, it passes an additional parameter along with the statement(s).
- The LRS checks for that parameter and if present validates extensions.
The downside of that approach is that the LRS/report tool has to be able to handle the fact that it might still get extensions with bad data, but it's not a big deal to treat that data in the same way as statements that do not have the extensions at all.
The upside is that you could probably implement this today and still be conformant. I think it'd also be valid to have a permission that can be tied to credentials to determine whether or not the credentials are allowed to send non-strict statement requests. I imagine it'd be a struggle to get widespread adoption of "strict mode"; I'm thinking more that this is applicable within projects that are very much under your control.
I see keeping the MUST requirement as extremely problematic. cmi5 for example, would break (https://github.com/AICC/CMI-5_Spec_Current/blob/quartz/cmi5_spec.md) because it uses extensions that essentially have controlled vocabulary. Not being able to reject those Statements for violating such vocabulary and then not being able to query correctly (let's say two letters were transposed) creates a huge problem.
LRSs have been given latitude to reject Statements for plenty of reasons, and I've always felt that we define as few reasons as possible for the LRS to reject Statements and let them choose additional rules due to profiles, etc.
As far as action on this, I would say that an LRS SHOULD NOT reject a Statement on the basis of anything in extensions UNLESS it determines the extension is not following the rules it is intended to (leaving this vague as it could be an IRI in the extension or a profile which may or may not use the category property)
Extensions should provide a schema (JSON-Schema or what have you) at the IRL so that LRSs can learn to validate them. Iff an extension provides a schema for itself, and a malformed extension pointing to this IRL violates the schema, the LRS SHOULD be able to reject the Statement for a malformed nested value like any other.
My $.02... I want this and I want to set a timetable for when this happens, because this is going to likely break things on the LRS side (at least going forward) and that means, pragmatically, there needs to be a plan.
@garemoko's point is totally spot-on. I'm for a requirement of using a schema for validation, but that means the schema has to be publicly available, imho. And that's not a guarantee right now -- let alone that the schema will persist in the location that someone arbitrarily sets for it. Purl.org is a hedge, for sure. A federated registry for profiles, verbs and (maybe) activities would be really helpful in this regard.
It will take time to stand up a federated registry and the machinations of getting adopters on board to support this (and potentially breaking changes) is something that DISC will be better suited to take on once that's up and running.
I don't want to table everything (and I also recognize that there have been scant details on what the consortium plan is as of yet). In the meantime... I like where @canweriotnow is going with this. I agree with @garemoko that should/should-not is problematic (I mean, that's why the big focus in 1.0.3 is on getting things as tight in terms of must/must-not as possible).
I can also see @andyjohnson's point that
As far as action on this, I would say that an LRS SHOULD NOT reject a Statement on the basis of anything in extensions UNLESS it determines the extension is not following the rules it is intended to (leaving this vague as it could be an IRI in the extension or a profile which may or may not use the category property)
Given the potential for how messy data can get today as it's on the developer to figure out exactly which profile and verb set they're going to use and how they're to be identified... I want this identified this as a major issue and set extensions and profile cleanup to be a milestone for 2016.
If there are non-breaking things we can talk to now that would encourage better emergent practice (but not something we'll want to retract later... this is key), let's talk about inclusion now. For example...
The process would be:
- Activity provider fetches the about resource to determine that the LRS supports this feature.
- If yes, it passes an additional parameter along with the statement(s).
- The LRS checks for that parameter and if present validates extensions.
How might this be impacted with stricter requirements on the LRS (at some future date) on requiring schema?
@aaronesilvers - Totally agree we need a process for 2016 regarding profiles and extensions. For 1.0.3, (which I hope is done before 2016) the good news is that even though this is a MUST requirements, getting more lax does not invalidate any current LRS. A "MUST NOT reject" means that it allows any, which to @garemoko 's point, there isn't any active reason TO reject.
Since there isn't any rejecting going on anyway, and there isn't active conformance testing to smack the LRS that tries to reject an extension (which, by the way, an LRS would pass anyways as it could find no criteria to reject the extension), the cost of changing this requirement to be inclusive of rejection possibilities is minimal. I'd say adding the UNLESS clause in there does this. We could go so far as make the base requirement a SHOULD NOT. I think the UNLESS and leaving the "evidence" vague would do the trick.
@andyjohnson so the kind of thing we've said in the past is that an LRS can reject anything based on permissions associated with specific credentials. So taking your cmi5 example, the LRS would have to be capable of accepting statements with any content, but you could have permissions that are not allowed to send statements that are tagged as following cmi5 but are non-conformant with cmi5. In that case, it's rejecting the statement based on permissions, not the value of extensions.
@aaronesilvers What I was thinking was that it would remain optional for the LRS. I imagine having a hard time convincing the rest of the team at Watershed that we need this until there's a client project that calls for it. If we did move to demanding that the LRS requires a schema, that would make the process I outlined pointless, but as it would be a 2.0 change a lot of things would break at that point anyway!
@garemoko - Agreed- permissions is where most of the latitude in LRS gets in rejecting Statements. In the cmi5 example, wouldn't you have to adjust one's permissions on the fly based on the conformance of the Statements? Sounds odd.
One thing we don't explicitly get into is the notion of "the LRS cannot reject Statements unless this spec says it can", but this requirement seems to flirt with that notion quite a bit. I would say we need to push this one way or the other. I'd say it should be more of an LRS MUST NOT reject a Statement unless it a) violates a requirement in the spec or b) has sufficient evidence to determine the Statement is not what the AP intended (based on profiles, category tags, extension schemas, etc.)
We discussed this on the last call. I had to leave half way through the call, but my recollection of the outcome was that we'd not make any changes in this patch version as they could be breaking changes.