aep-dev
160: include more examples and some explanation on how to use cel
Users are having difficulty adopting AEP-160. The primary issues include:
- lack of example queries
- some clause that explains that you don't need to use CEL in it's entirety.
- mapping CEL syntax to database queries.
- lack of explanation of how to adopt CEL into your stack (libraries avaiable, etc)
While we have the chance, we probably want to examine this AEP and filtering in general a bit more, and see how we can help resolve these challenges. It could result in friction to adopt the AEPs.
Bump!
Can we get some more examples on this? I'm new to CEL, and ChatGPT it's suggesting things like this:
# property == whatever
GET /properties?filter=property%20%3D%3D%20%22whatever%22
# "foo" in tags || "bar" in tags
GET /properties?filter=%22foo%22%20in%20tags%20%7C%7C%20%22bar%22%20in%20tags
This is kinda 🤮 to human eyes, and I'm used to more normal query params that are named directly, not a filter string, but I'm open to being convinced this is the way with some good reasoning and some good examples to help learn.
Query strings being ugly is not a new thing or unique to AEP.
I think there could also be some consideration for how much of CEL is recommended to be implemented, or if any of it is must or even should territory.
While we have the chance, we probably want to examine this AEP and filtering in general a bit more, and see how we can help resolve these challenges. It could result in friction to adopt the AEPs.
One thing I'd like to challenge in 160 is this statement:
It is tempting to define a structure to handle the precise filtering needs for each API. However, filtering requirements evolve frequently, and therefore it is prudent to use a string field with a well-defined syntax. This allows updates to be able to be made transparently, without waiting for UI or client updates.
Is it really the case that filtering requirements change so much that specialized filtering requests are not advised? Even when such requests could be described using (for example) Protobuf messages with their backwards/forward-compatibility features, and some sensible API versioning?
I get we (well, Google first) are striking the balance between usability and future-proofing, but this approach is heavily skewed against usability to the point of being not feasible...
...unless there is some easy-to-use library that makes the translation between this CEL-based filtering and your internal domain pain-free (which I'm sure Google has)
I know there is go.einride.tech/aip (limited to Go, so not a good approach for a general API standard) and @toumorokoshi very own cel2db (again, currently limited)
I think I agree with the above, but what alternative is there? Many APIs have very basic filtering needs that could be defined by simple query strings that match model fields, like books?category=fiction, while others may need more advanced queries.
My fear with CEL is that it's SO open ended that either you end up having some less-well-documented internal rules about what that particular API supports, or you end up possibly having to support a broad range of things, some of which may have bad database performance implications and maybe even security concerns. I could be wrong here, this is new to me.
I think we can make this easier on developers while still keeping a consistent standard by supporting two filtering options:
1. Basic Filtering
The simplest approach: normal query params that map directly to resource fields. Easy to read, easy to type, and no new syntax to learn.
GET /books?category=fiction&year=2024
GET /pets?type=dog&limit=10
Good for cases where filter needs are small, predictable, and not changing often. This is common in public APIs, but there is no formal standard.
2. Advanced Filtering (CEL)
For APIs that need more complex or evolving filters, we stick with AEP-160’s filter string but provide guidance for best practices and how to handle common challenges or questions that come up with this approach.
What we would need to define
For Basic:
- Parameter naming conventions
- Supported operators such as exact match and ranges
- Encoding rules
- Examples
For Advanced:
- Exact CEL subset including operators, functions, and disallowed features
- Which fields can be filtered on
- Query complexity limits
- How to handle errors for unsupported filters
- Example queries in both readable and URL-encoded form
- Recommended libraries for parsing and mapping to database queries
- Guardrails for performance and security
We may not know enough to really nail this down until we get some usage and feedback, but we can take a stab at it before AEP 2025 locks in.
Final thoughts
Rule of thumb: start with Basic unless there is a clear need for Advanced.
We would probably need to provide a way to inform systems and users of which filter type to use, possibly by extending x-aep-resource with a filter_type field, and some details in the description for the LIST endpoint.
Is there any reason NOT to provide the basic filtering option?
While we have the chance, we probably want to examine this AEP and filtering in general a bit more, and see how we can help resolve these challenges. It could result in friction to adopt the AEPs.
One thing I'd like to challenge in 160 is this statement:
It is tempting to define a structure to handle the precise filtering needs for each API. However, filtering requirements evolve frequently, and therefore it is prudent to use a string field with a well-defined syntax. This allows updates to be able to be made transparently, without waiting for UI or client updates.
Is it really the case that filtering requirements change so much that specialized filtering requests are not advised? Even when such requests could be described using (for example) Protobuf messages with their backwards/forward-compatibility features, and some sensible API versioning?
I get we (well, Google first) are striking the balance between usability and future-proofing, but this approach is heavily skewed against usability to the point of being not feasible...
...unless there is some easy-to-use library that makes the translation between this CEL-based filtering and your internal domain pain-free (which I'm sure Google has)
I know there is
go.einride.tech/aip(limited to Go, so not a good approach for a general API standard) and @toumorokoshi very own cel2db (again, currently limited)
@ar3s3ru @thegagne thanks for the feedback! I'm going to create a separate issue on this topic, as this issue is more about including examples on how to use cel - it is not intended to be a discussion on whether to use cel.
Let's continue this discussion on CEL alternatives in https://github.com/aep-dev/aeps/discussions/322.
After discussion in the weekly, we've agreed that we should try to tackle this further in the 2027 edition of AEPs, rather than rush an untested solution.
We will probably continue to work through a description format that allows one to express that they support a limited set of expressions, as proposed in https://github.com/aep-dev/aeps/pull/324.
I do think improving the doc with some simple examples could be helpful, but it would not block the release of 2025.