gotify
[Feature Request] API Hook for Date/Time Interval Retrieval of Messages
Hi!
I am one of the developers working on Winify, a graphical UI client for Windows and one of the features that we implement is the retrieval of past notifications between a certain time-interval.
The problem is that there is no direct API hook to do that directly via Gotify. What we do instead is use the /application/{Id}/message REST path and recursively retrieve all the messages by using the Gotify pagination. After that, we sort all the messages and only then apply the user-requested date-range filter. I am unaware of any correlation between the since parameter and the actual date when the notification was sent, is there a correlation?
As you can imagine, if the Gotify server has been running for a while, retrieving all the messages or going through all the messages just to find a date-range is a miserable experience for the user, let alone the fact that loading the messages just means more a/deallocations producing memory spikes unnecessarily (this is something that we can optimize on our end, but not the time it takes to retrieve them all).
Would it be possible for you to add a hook for the API that would allow the retrieval of messages from a given application by applying a date-time filter? Right now there is:
/application/{Id}/message
so maybe something like:
/application/{Id}/message/{date}
or part of the rest of the other parameters
/application/{Id}/message?date=...
By implementing this, time to the amount of linear time complexity O(a * m) where a would be the number of applications and m would be the number of messages could just be reduced to O(a). In case the application ID is known, which in many cases it is because the option in our software is usually provided from the UI to select an application, then the whole O(a *m) is reduced to just a single operation O(1) which is an amazing performance boost at no cost.
I have not looked into Gotify myself, so I do not know the internals and I haven't dabbled in Go yet but the actual date when a notification arrives is already recorded by Gotify internally as part of the message structure such that it should not be too costly. We also filter by other criteria when searching, but that criteria is not to be found in your Gotify message structure so that would be more difficult.
Feature request #376 seems somewhat related given that a notification typically has a very short lifespan because it is supposed to notify the user what happened "now" contrasted to say a database that can hold long-term data. However, at the same time, browsing through age-old notifications like "outdoor lights went on" that took place a month ago seems irrelevant. Pagination is great for an UI when a user can click inputs and browse pages of data arranged such that not too many appear at the same time, but when you are accessing Gotify programmatically like from Winify, pagination can take a huge time and makes a bunch of useless requests.
Could you please implement this REST API path?
Thank you!
Sounds good, I think it's a good feature and can help with some friction regarding #376 without a major work.
I will look into the details and reply with a plan tomorrow !
Will ?before=<unix_timestamp> ?after= work for your use case? This way we can potentially add other filters later.
@eternal-flame-AD thank you for your response: yes, definitely, it's perfect.
You are also right about adding other filters later because it occurred to me that you might be able to offer that data quickly, in terms of computational complexity, compared to a client that would have to first scrape the data and only then apply filters. For example, given the data structure of, say, messages, you can build dictionaries or maps to make the lookup-cost be an O(1) operation on Gotify itself with little cost to memory given that the table could just consist in references to data and not a copy of the data.
I dislike the use of dates for pagination, as for the reason described in here: https://github.com/gotify/server/issues/34#issuecomment-375875240. Tho, the use-case is valid, so it's probably okay, but we should document to prefer id based paging where possible.
@jmattheis as I understand it, the comment by Islah on #34 where he cites https://coderwall.com/p/lkcaag/pagination-you-re-probably-doing-it-wrong claiming that two IDs can have the same date would not apply for this usage case.
Islah describes an usage case where the date would be relied upon to perform pagination which evidently leads to issues with uniqueness but this feature request is more about filtering the results after the pagination takes place.
With that said, it would even be fine if the implementation is just a linear search with a conditional that would only return messages iff. they fall within a certain date regardless of the pagination:
/application/{Id}/message?before=...&after=...&since=...&limit=...
where before, after, since and limit would be predicates that would all have to match for a result to be returned when querying Gotify.
You could make it part of the specification that since is the required parameter such that before and after on their own cannot be used to paginate.
Similar filters could be implemented, for example:
/application/{Id}/message?before=...&after=...&since=...&limit=...&text=
where the text parameter would use SQLite FTS search to filter message content regardless if the pagination is performed by since and limit.
Another way would be to create a substructure for filters that will not have anything to do with pagination but would just be ANDed together and would probably be more extensible in the long run, similar to:
filter : {
date : {
before : "...",
after : "..."
},
text : hello
}
but that looks like much more work. The point here is that Gotify tracks the date so that information is already available which opens up opportunities for optimizations.
Otherwise there is no way to filter the messages without downloading them all with the client and only then process them which is a lot of overhead.
With the program we develop you wouldn't care too much because it is meant for a workstation (even though, that's why I'm here, because it takes a while due to the Gotify server we test with having ran for years).
On an Android phone you could end up with gigabytes of RAM in allocations and a whole lot of wasted CPU / Android time and network traffic, in particular if the Gotify server accessed is outside the LAN just to sort messages that are known to be false positives when they're sent with a notification history that can reach back a few years.