openid
Opportunities to improve the credential query/request syntax
Over a period of time through implementation feedback on OpenID4VP, several items around the current credential query syntax (based on P.E v2) have been raised. To the best of my ability I have attempted to summarise some of those that I have heard below
Usage of JSON Path/Pointer
Presentation exchange v2 which is the current target of OpenID4VP makes use of JSON Path, which is a highly expressive DSL for reading JSON documents. The problems identified with it are that because of its broad expressivity and feature set, it creates certain security and use-ability challenges.
JSON pointer is a narrower expression syntax then JSON path which alleviates most of the security challenges and has been considered before as an alternative. But to some it still represents significant use-ability challenges and is still arguably too flexible/complex for most of the applications we have within OpenID4VP credential queries/request syntax where the path expressions we desire to be able to define are usually just simple traversals through nest maps and or arrays.
Furthermore some of the credential formats defined by OpenID4VP are not JSON based instead use technologies like CBOR which gives rise to other possible challenges.
The feature set of P.E is massive
Presentation Exchange itself beyond its usages of JSON Path/Pointer is highly expressive enabling all sorts of incredibly complex queries to be represented. However this complexity burdens both developers who need to write robust implementations of P.E and know how to create valid credential request/queries using P.E. Certainly the feedback overtime that I have heard is many of the features of P.E aren't needed for most usecases or the implementation complexity they create outweighs their value.
P.E protocol agnostic design goal complexity
Presentation exchange from the outset aimed to define a query/request syntax that was protocol agnostic. While this design goal does lead to the potential for re-use of P.E across different protocols. It has created duplication of features that are already inherently present in OpenID4VP (and OAuth2 and OpenID Connect for that matter) such as capability negotiation between the protocol participants. For example OpenID and OAuth2 of which OpenID4VP is built atop has always largely handled capability negotiation through metadata exchanged/registered between the protocol participants rather then doing this negotiation during execution of the protocol itself. Whereas presentation exchange due to its protocol agnostic design goal defines how to negotiate things such as cryptographic suites in a credential request/query.
There are several options we have to solve these challenges and the purpose of this issue is to first better collectively understand these concerns and then discuss what solutions we could apply.
Notes from conversations from three open space sessions at OSW and IIW titled "What does Presentation Exchange do and what parts of it do we actually need?" can be found at https://self-issued.info/?p=2427 and https://self-issued.info/?p=2395. In all these conversations, many people expressed the viewpoint that Presentation Exchange is pretty complicated.
I think I understand the points outlined in this issue and agree we should discuss if/how we would address them. For the limitations of using JSON Path with CBOR-based mdoc format, this thread also might be useful: https://github.com/WICG/digital-identities/issues/80#issuecomment-1955444351.
Having said that, I also think it is important to discuss and understand what in PE has worked well. So I'll go on record here that PE does the job:
- its flexibility allowed our implementation to meet customers' requirements without the need to go back to the WG and ask to add a new parameter.
- as an implementer that need to support multiple credential formats, being able to reuse PE across multiple formats is helpful.
- ISO has done multiple test events where multiple vendors were able to interoperate using OID4VP with PE for mdocs.
- OID4VP has never mandated implementing the entire feature set of PE and so far we have taken the approach of limiting the feature set of PE i.e. as in HAIP and security considerations section in OID4VP.
Again, not saying PE is perfect, or the current approach is the best, just believe these points should be taken into account in this discussion.
You will see from comments that I made over a year ago that mandating the use of DIF PE was a bad design choice. Instead we should give flexibility to implementors to use whatever query language their community of users choose. By replacing DIF PE with a type and value extension mechanism, then those that want to use DIF PE can set the type to "DIF PE" and the value to the PE query. Those that want to use SQL, or the W3C presentation request (see https://w3c-ccg.github.io/vp-request-spec/) or any other query language can set the type to their chosen language and the value to the query in their language. The metadata of verifiers and wallets can say which query type(s) they support.
I also believe we need a better way than PE for expressing queries.
I have worked on and with several different credential formats, including fully JSON-based ones like IETF SD-JWT VC that do not have any special namespace considerations and have native support for selective disclosure, W3C VCDM 2.0 with a lot of optionality even in the proof format and the way claims use namespaces via @context/JSON-LD processing, and also ISO 18013-5 mdocs that are CBOR-native using COSE_Sign1 and have their own way to define namespaces. All of them are quite different, and I had troubles in applying PE because of differences in:
- typing (JSON/JSON-LD)
- structure (where in the credential claims are defined)
- encoding (JSON/CBOR),
- feature set (selective disclosure, policies, etc.).
I believe having a credential-format agnostic query language like PE is quite challenging for those reasons.
As cited at https://github.com/openid/OpenID4VCI/issues/266#issuecomment-1946999918, a few of us have been working on a format-specific query language proposal as a strawman to replace PE. This was motivated in part by @tlodderstedt's request during the IIW session to "show me a specific counter-proposal".
See https://docs.google.com/document/d/10JT--pXWsfwC4QVu3XJpXGwcO08M6tnpkZ4PKdtCEWo .
I'm not a fan of PE. However, it is implemented now with OID4VP in a couple of places. So it is not a show stopper per se and people have invested money in it. That's why I'm hesitant to replace it.
I think most concerns can be addressed by defining a subset of PE. That's what we did in the HAIP spec. Please have a look
Having said that. I would be ok to add another way to request credentials (and deprecated PE) if the existing implementers and the SDOs that build standards on top of OID4VP w/ PE support that step.
Chair hat on, would suggest to proceed as following:
- as a first step, agree on what are the problems with PE we want to address
- than discuss how to solve those problems
Notes from Feb-22-2024 WG call,
Some requirements
- need to work well with currently known credential formats, which include both JSON-based and CBOR-based formats
- could define credential format specific syntax, but sounded like there is some implementation feedback that ability to reuse the common syntax across credential formats is useful (@ve7jtb said it better)
- need to clarify how verifier metadata negotiation happens: using traditional oauth style mechanisms, or using the query language
- seems to be the feedback that oauth style mechanism might not be the best fit for issuer-holder-verifier model and using query language has worked
Options in the solution space (does not have to be only one of these - doing 1 and 4, for example, seems to be an option)
- define a (mandatory to implement?) minimum subset of PEv2.0
- work with DIF on PEv3.0
- define a credential format agnostic query syntax in OIDF. maybe something like the one proposed in https://github.com/openid/OpenID4VCI/pull/276
- define a credential format specific query syntax in OIDF.
- having the query language as an extension point
Chair hat on, would suggest to proceed as following:
- define a (mandatory to implement?) minimum subset of PEv2.0
- work with DIF on PEv3.0
- define a credential format agnostic query syntax in OIDF. maybe something like the one proposed in Define claims display description and claims path query OpenID4VCI#276
- define a credential format specific query syntax in OIDF.
In all cases, we need to define profiles for credential formats. Even if we had one query syntax that appears similar for all credential formats, it would have significantly different meanings per credential format, which is equivalent to defining a syntax per credential format. This applies to all four options.
Under this assumption:
Regarding option 1, I'm not convinced that having a profile for PE for each credential format is the best option. There is a risk that implementers will make mistakes, implement additional features not recommended, or become overwhelmed by the feature set that PEv2 offers. Note that they will need to refer to the PE spec anyway and find sections that apply.
Regarding option 2, it is similar to option 1.
Regarding option 3, I would be fine with that, but again, we would need to profile it anyway. I am uncertain if being credential format agnostic is a goal that can be achieved or one that we actually want to achieve. Reusing some components across credential formats does make sense, for example, how to encode constraints such as certain fields being required in the response. Note that https://github.com/openid/OpenID4VCI/pull/276 is actually more like option 4 since it uses the top-level namespace/credentialSubject object for mdocs/VCs, which is not common among the credential formats.
Regarding option 4, it is essentially the same as option 3.
Regarding options 3 and 4, I see them both as almost equivalent since you will need profiles anyway, and the interpretation of the query will differ depending on the format even it looks similar. I think the reuse of constraints (required, requiredIfPresent, intentToRetain, etc.) makes a lot of sense, but we should have the flexibility to cater to structural/encoding characteristics of formats, which has already been demonstrated in https://github.com/openid/OpenID4VCI/pull/276. I don't think the goal is to have an n-number of query syntaxes for one credential format. Ideally, it should be just one, defined aligned with the patterns the respective community has already used.
need to clarify how verifier metadata negotiation happens: using traditional oauth style mechanisms, or using the query language
- seems to be the feedback that oauth style mechanism might not be the best fit for issuer-holder-verifier model and using query language has worked
I don't understand why verifier metadata negotiation is part of this discussion. There is a mechanism in place that works well with OID4VP. We even added a feature, client identifier scheme, to facilitate this negotiation. The method by which PE populates some of the supported curves is redundant and unnecessary. We even have a paragraph that states, "The Wallet MUST ignore any format property inside a presentation_definition object if that format was not included in the vp_formats property of the metadata." Therefore, format in PE is yet another potential source of developer confusion. Negotiation by taking into account wallet capabilities is another topic we try to address in #59.
@Sakurann You appear to missed out the option of having the query language as an extension point, which brings a lot of advantages, a main one being future proofing, another one being innovation. Some in the WG like this approach whilst others do not. An argument made against it was that it kills interop. But this argument is fallacious for several reasons. a) we already have a number of extension points e.g. the crypto suite. Is anyone arguing for replacing this with fixed a crypto suite? b) having a fixed query language can kill interop when more than one standard query language exists. We originally mandated DIF PEv1, now DIF PEv2 and some are thinking of moving to DIF PEv3. There is also the W3C presentation request draft that I referred to above, which many have implemented. Why should OID4VP try to pick the winner? Let implementations and the market decide and let the protocol cater for all.
@awoie verifier metadata negotiation is part of the discussion because there is no agreement on the statement The method by which PE populates some of the supported curves is redundant and unnecessary.
@David-Chadwick thank you! updated my original comment to add "having the query language as an extension point" as an option :)
@awoie verifier metadata negotiation is part of the discussion because there is no agreement on the statement
The method by which PE populates some of the supported curves is redundant and unnecessary.
My point was that there is implicit agreement codified by normative language in the OID4VP spec: "The Wallet MUST ignore any format property inside a presentation_definition object if that format was not included in the vp_formats property of the metadata."
that language might be residual from before we negotiated with PE authors to add a top level format parameter in the PE. also that sentence only talks about the formats negotiation and not the rest of the metadata. so I think the large question is still in tact.
that language might be residual from before we negotiated with PE authors to add a top level format parameter in the PE. also that sentence only talks about the formats negotiation and not the rest of the metadata. so I think the large question is still in tact.
I don't understand what this means, according to github, format has been in PE for a long time (v2.0.0, > 2 year), even before the normative statement I was talking about was added (02/2023).
IMO, it is not a requirement of any query syntax to communicate any of these metadata. We have client metadata for this and changing this is would be a fundamental shift for OID4VP.
We have an issue to do real capability negotiation and using PE for this was never brought up nor included in any proposal. I believe nobody argued for putting other client metadata into the query syntax.
I'll go on record here as being not a fan of PE.
On the 22 Feb 24 call @Sakurann pointed out that this kind of "voting" alone on PE wasn't particularly helpful, which is fair. My concern with PE is somewhat more high-level. It promises to provide an awful lot of useful functionality but is extremely complex. This kind of standard often sounds/looks good on paper and even in narrowly constrained testing at first but the real cost of the complexity and sometimes inability to actually provide purported functionality doesn't show up until later and manifests in serious interoperability and security issues or lack of adoption/use. That's maybe not much more helpful that a "vote" but tried to convey some of my rationale.
Our experience is that PE supports a lot of functionality that on paper sounds useful, but that in the end leads to a lot of complexity when trying to build systems that support many different flavors of credentials for reasons many mentioned above (encoding matters, storage method matters, structure matters, typing matters, features matter...) Main reasons that lead to this complexity were:
- JSON Path richness of expression
- Regex filters
- and submission_requirements mixing and matching
We ignored Features for now
When only using absolute paths with non-regex filters we came to a more or less understandable implementation - but one that still involved having Intermediate Representation of credentials, and translating PE to another query language, and one that was still very complicated.
Chair hat on, would suggest to proceed as following:
1. define a (mandatory to implement?) minimum subset of PEv2.0
Reducing the set of PE would make sense at least to me, but I am not sure how that would be enforced and communicated. The feature set would need to be very clearly defined, and referenced in the base spec and not in the interop-profile.
2. work with DIF on PEv3.0
This could potentially give us an opportunity to define the base/core of the spec that we think is aligned with our requirements, so I think I prefer it more compared to suggestion #1.
3. define a credential format agnostic query syntax in OIDF. maybe something like the one proposed in [Define claims display description and claims path query OpenID4VCI#276](https://github.com/openid/OpenID4VCI/pull/276) 4. define a credential format specific query syntax in OIDF
Could make sense, but seems like a lot of work, and given that many have already implemented PE it might be easier to try to fix that one before designing yet another mini-language.
5. having the query language as an extension point
I think the spec should allow for multiple query syntax(or their versions), but that it should mandate one across all implementations. Example: VP 18 mandates PEv2, but it could allow you to use a specific one that might fit your system better. Mandated one(s) would ensure interoperability, while the optional ones would allow for experimentation, competition, and local optimizations. This could possibly also allow for easier switching from one version/flavor of query syntax to another. This would probably require some registry of known query syntaxes.
I'd like to add support for @Sakurann's suggestions, namely
define a (mandatory to implement?) minimum subset of PEv2.0 work with DIF on PEv3.0
I'd take this a step further and suggest PE v3 is adopted as a work item of this group.
To paraphrase @selfissued I do not believe we should allow for multiple query languages if one will do. Standards make choices. Optionality is the enemy of successful interoperability.
Because I believe PE v3 will take a while, my immediate suggestion is to profile a subset of PE v2 and require it for all implementers of OID4VP.
Separately, I'd like to make a comment on the complexity of PE. As one of the authors/editors of PE v1 and a contributor to v2, we tried to represent the complexity present in the credentials space, which is itself highly complex. I could succeed in making an argument that OAuth, OIDC, and OID4VC are all really complex specs. This is not me suggesting that we increase that complexity, rather, it's an acknowledgment that our subject area is inherently complex and the reasoning "too complex let's exclude it" is misleading at best.
Many engineering teams have implemented PE successfully. I have personally done so multiple times. It's not an insurmountable task, albeit annoying at times.
This represents an opportunity for us as editors to find opportunities for simplicity that works for our use cases.
does anyone have an example of how to turn a PE request into a meaningful user consent form that can fine on a smart phone screen?
does anyone have an example of how to turn a PE request into a meaningful user consent form that can fine on a smart phone screen?
@TomCJones My understanding of current models is that the user consents to what information can be released that satisfies the PE request, not what information was requested by PE, so there is never a need to turn PE into a user consent screen.
So for example if the PE requests a proof of name from either a driving license or a passport, it does not seem important for the user to know that this was the request - only for the user to be asked (in the case where they only have a driving license on their phone and not a passport) whether they would like to release the name in their driving license credential to the verifier.
Can you explain a situation where it would instead be important for the user to understand the request the verifier made? At least in my example, it would seem considerably less user friendly to try to explain the request to the user.
I think you are telling me that the verifier gets to create a sql injection attack against the user, the user gets a list of all of the credentials and all of the data fields in those credentials as a consent request from what? one aggregator wallet, each wallet in turn (perhaps three of them) will they are standing in a line to get on an airplane or into a concert and makes an informed consent decision? All of the ideas about informed consent that i have been involved with the question in their mind is what the verifier is going to do with the data. I don't see where informed consent is addressed in any of this! It sounds to me like a direct violation of nearly every data protection law in existence.
Hi Tom
To complete my example, if the user has a driving license and a passport stored on their phone, and for the purposes of this example they are stored in different wallets, then my understanding of the flow might work (as is being defined in the WICG group):
- Verifier tells the browser that it wants a verified name from a passport or driving license
- The browser asks the installed wallets what credentials they have that meet that requirement (this matcher is sandboxed, so no information can be leaked to the wallet providers in this step)
- Assuming more than one credential matches, the user is asked which credential they want to provide (or if they don't want to provide one)
- If the user does want to share a credential, the wallet that holds that credential is then invoked to provide the credential and asks for any necessary user consent before returning the credential to the verifier.
This is fully informed consent. No one here has any interest in designing flows that fail to comply with data protection laws, that would be a fruitless endeavour.
i guess i don't see an example of the screens the user sees. Who's job is it to display the name of verifier and what they want to do with the data? For what purpose do they want the data? Is the verifier trustworthy? the device or each wallet. I can see some simple use cases going to six screens depending on the design.
Hi Tom. I think your original question about PE has been answered and we've now veered significantly away from the subject of this ticket. I think some of the WICG folks have demos of the flow, it might be worth asking there.
To paraphrase @selfissued I do not believe we should allow for multiple query languages if one will do. Standards make choices. Optionality is the enemy of successful interoperability.
+1 here I dont believe we should allow multiple query languages, that would be a failure of the standard to foster interoperability.
Separately, I'd like to make a comment on the complexity of PE. As one of the authors/editors of PE v1 and a contributor to v2, we tried to represent the complexity present in the credentials space, which is itself highly complex. I could succeed in making an argument that OAuth, OIDC, and OID4VC are all really complex specs. This is not me suggesting that we increase that complexity, rather, it's an acknowledgment that our subject area is inherently complex and the reasoning "too complex let's exclude it" is misleading at best.
If you are referring to my last point, I don't believe that is the argument I'm making. I'm merely pointing out that there is a trade off, every feature a protocol or standard chooses to define has a cost associated to and that there needs to be a counter balance saying "do we really need this feature, is it worth the cost"? Having also participated somewhat in P.E I don't believe it got that right, the inclination was to say "yes" to every use case, rather then challenging which features were actually needed. I think its abundantly clear that especially in the context of OpenID4VP that P.E defines way more features then required for its application, evidence of which can been seen via the heavy profiling of P.E done in HAIP, ISO 18013-7 and OpenID4VP itself. That redundancy has a significant cost associated to it.
Valid points are raised, but I also want to make a few points from our side:
- Fully agreed on the complexity. It took us significant amount of engineering to support almost everything from the PE spec for v1 and v2. Having said that, PEv2 already introduced the concept of features. These are all optional, and thus if consensus is that a lot of these features are not used/needed for most use cases, creating a profile of PE makes sense; just leave out a lot of the features
- PEX v2.1 will move to JSONPointer and make JSONPath optional, where V3 will remove JSONPath altogether, alleviating the security concerns (which are described in both PE and OID4VP, HAIP)
- Unless I overlooked, nobody mentioned yet that PE is not only used in OID4VP, but also in other SSI contexts, like for instance Aries Present Proof v3. PE is agnostic to the transport protocol. Any party implementing support for other protocols next to OID4VP, thus would have to support another specification with very similar goals.
- A lot of parties are supporting PE in their solutions. The OID4VC specs (no judgement) have already caused a lot of other new related specs to emerge for things that have been around in a slightly different form for quite some time. We are okay with that, but it does mean that vendors do need to add yet another way of doing similar things. The OID4VC specs are seeing constant change, which is totally understandable, at the same time the EU is moving fast and a lot of vendors already implemented PE in their solutions. IMO changing to a new credential query/request specification and then implementation will cost significant time and money on the spec level as well as on the implementation/adoption level. Almost everyone I talk to is already feeling the pain of the constant changes and amount of optionality in the OID4VC specs. Adding this big of change certainly won't help in that department.
I agree and am in favor of others pointing out:
- it makes sense to profile to a subset of PE; the optional features would allow for that, as the PE core implementation would be considerably simpler.
- Work together on PE v3. The whole point of PE is credential query/requests and is used in different contexts, so focus attention there
is there any evidence that normal human users or verifiers will accept VPs? Especially in view of the likelihood of multiple wallets responding to one request? I guess it's a foregone conclusion that it will be the browser or device that will function as the metawallet?
In an effort to make some of my concerns more concrete, I've compiled 3 examples of P.E requests, I think they highlight very clearly why profiling P.E would be entirely insufficient to fix the issues here
Example 1
How should one process the following query?
{
"id":"mDL-sample-req",
"input_descriptors":[
{
"id":"org.iso.18013.5.1.mDL",
"format":{
"mso_mdoc":{
"alg":[
"EdDSA",
"ES256"
]
},
},
"constraints":{
"limit_disclosure":"required",
"fields":[
{
"path":[
"$['org.iso.18013.5.1']['driving_privileges']['codes']"
],
"intent_to_retain":false
}
]
}
}
]
}
The ambiguity around this query is that selective disclosure in an mDoc can only be done to the "$['org.iso.18013.5.1']['driving_privileges']" level. So what should a wallet respond with here, the whole driving_privileges structure OR nothing?
A similar issue also exists when applied to an SD-JWT that encodes a nested object as a single disclosure rather as a series of nested disclosures all the way down to the leafs.
This in my opinion quite clearly highlights the issue with using either JSON Path OR JSON Pointer as the basis for a query syntax, it can't express the nuanced constraints of a credential format like SD-JWT or mDoc reliably.
Example 2
Similar question how should one process this query?
{
"id":"mDL-sample-req",
"input_descriptors":[
{
"id":"org.iso.18013.5.1.mDL",
"format":{
"mso_mdoc":{
"alg":[
"EdDSA",
"ES256"
]
},
},
"constraints":{
// Note no limit disclosure
"fields":[
{
"path":[
"$['org.iso.18013.5.1']['driving_privileges']"
],
"intent_to_retain":false
}
]
}
}
]
}
Without limit_disclosure in the query how should a wallet receiving this interpret the query? Should they return all the claims or just the "fields" in the request? If they return only the fields in the request does this undermine the point of limit_disclosure in the first place? The point I'm making with this example is P.E has bunch of features that make sense in isolation but not when composed together, whether specific claims can be requested is a property of the credential format and shouldn't be independently communicated/negotiated in the request.
Example 3
As a verifier if I were trying to determine whether the holder has a US drivers license, e.g by determining whether it contains a real ID claim without them knowing, I could use the following query to disguise my request
{
"id":"mDL-sample-req",
"input_descriptors":[
{
"id":"org.iso.18013.5.1.mDL",
"format":{
"mso_mdoc":{
"alg":[
"EdDSA",
"ES256"
]
},
},
"constraints":{
// Note no limit disclosure
"fields":[
{
"path":[
"$['org.iso.18013.aamva']['real_id']",
"$['org.iso.18013.5.1']['family_name']"
],
"name": "Family name",
"purpose": "Requesting family name",
"intent_to_retain":false
}
]
}
}
]
}
The problem here is two fold
-
pathallowing multiple expressions is meant to be a feature to create equivalent expressions across formats, but there is nothing stopping one from using it to query different attributes within the same credential to disguise the intent of a request. -
namecreates a second source of truth for what is being requested meaning the verifier can use it to further disguise the intent of a request to a wallet.
Based on the logical processing of P.E for the above if the real_id attribute was present on the matched credential it would be returned and the holder would likely have given consent based on the "name" thinking they were releasing their "Family name". Secondly in the case the wallet actually didn't have the real_id attribute on the drivers license, the fact that the family name is returned here, proves the real_id attribute doesn't exist, leaking this to the relying party.
Both of these cases above are concerning from a data privacy perspective and can't simply be addressed by profiling P.E they are fundamentally designed into it.
@tplooker can you explain how these situations can be resolved please - e.g. are you suggesting defining a query language that simply doesn't allow the queries in your examples to be made?
@tplooker can you explain how these situations can be resolved please - e.g. are you suggesting defining a query language that simply doesn't allow the queries in your examples to be made?
In short yes I believe a proposal like what is given in https://docs.google.com/document/d/10JT--pXWsfwC4QVu3XJpXGwcO08M6tnpkZ4PKdtCEWo/edit#heading=h.7igj7m3na8ru avoids these issues by
- Not using a generalised DSL syntax like JSON path/pointer which has no awareness of the data structure it is referring to
- Assumes the queries are in the context of a format, so features that cut across formats in awkward ways like limit_disclosure are no longer an issue, because either the credential format supports selective disclosure or it does not.
- Doesn't allow multiple path expressions to request the same attribute because the query itself is already localised to the credential format and type.