EFForg
[Updated October 2020] webRequestBlocking deprecation by Google (Manifest v3)
Updates
Latest update (September 4)
Mozilla made a statement on Manifest v3, see https://github.com/EFForg/https-everywhere/issues/17268#issuecomment-528034884
Previous updates
View
May 31
Blocking capabilities of webRequest are still being deprecated, see https://github.com/uBlockOrigin/uBlock-issues/issues/338#issuecomment-495936668 (original post deleted) and commentary by @gorhill: https://github.com/uBlockOrigin/uBlock-issues/issues/338#issuecomment-496009417.
Thanks @Giltyhub for update!
Statements made by other browser vendors:
Mozilla
Content blocking: We have no immediate plans to remove blocking webRequest and are working with add-on developers to gain a better understanding of how they use the APIs in question to help determine how to best support them.
Vivaldi
Once the change is introduced to Chromium, believe me when I say that there are many, many possible scenarios.
Restoring the API could be one of them. We’ve restored functionality before.
If the API is removed altogether and no decent alternative is implemented, we might look into creating a limited extensions store.
Brave
Brendan Eich's answer to ZDNet's inquiry:
To respond on the declarativeWebRequest change (restricting webRequest in full behind an enterprise policy screen), we will continue to support webRequest for all extensions in Brave.
[source: ZDNet (archive.is link)]
Microsoft
None yet.
If you know others, please post a comment in this issue or edit it if you are able to.
Original text by @Parasrah
Hi, I've recently read a string of reports about the chromium project moving to a new API called declarativeNetRequest, and removing the capabilities of the webRequest API to block or redirect requests. From what I understand declarativeNetRequest will still allow blocking and redirect rules, but it will not be as flexible as webRequest in that you will essentially be providing a fairly simple rule-set and will no longer be able to react to a request programatically. They claim this is to increase the privacy for their users and to make redirects/blocking more efficient, although it may also conveniently (for them) break the functionality of many ad-blockers (apparently according to their authors in the reports I've read, I could not find a source for this).
I was wondering how this will affect HTTPS Everywhere (I know you make extensive use of the webRequest API), and if the extension will still be able to function in Chromium. I've also seen speculation that the number of rule sets permitted with declarativeNetRequest would be limited, which would obviously be a problem for this extension if the limit was too low.
Draft of Manifest V3 for chrome extensions
PDF (for those who would like to avoid google docs)
Thanks for all the work the team has put in to enhance the security of their users, hopefully there is a way around these changes
Update: I see where the speculation on the maximum number of rules came from, the api docs mention it will be 30,000
@Parasrah Thank you for bringing this to attention. We had some discussion yesterday about how this impacts the extension as the Manifest proposal currently stands. The main concerns are around:
DNR's methods of handling rules are static
- Which means we can't use it to redirect http://example.com/foo to https://example.com/foo
- Rules handling would be statically handled and not programmatically as mentioned above
- The secure cookie feature would be interrupted
- With the static
rules.jsonfile, this could mean having to do a full extension update rather than just updating the rules separately like it is handled now. - Rules are capped at 30K in DNR, reducing our coverage significantly
Service Workers
Transference from background scripts to service workers could potentially allow sites to have more control over the extension's upgrade actions
Missing features in DNR
Redirect looping handling and other features are not necessarily addressed for replacement in declarativeNetRequest are also in question. Error handling is almost nonexistent.
Conclusion
Overall the move to be more static with declarativeNetRequest would take out most of our dynamic functionality - Ultimately blocking the extension's capabilities
There is already bug filed here with lengthy discussion: https://bugs.chromium.org/p/chromium/issues/detail?id=896897&desc=2
This conversation is also being brought up with other privacy oriented extensions: https://github.com/uBlockOrigin/uBlock-issues/issues/338
Thanks for the response, glad to hear this wasn't a surprise and there have been talks internally. Would it be possible to update this issue when a decision has been reached by the team as to next steps?
@Parasrah yea we will update this as a consensus is made on what to do programmatically. Right now the discussion is still young among us and other privacy extensions. There is a working group having discussion and discourse being had with the proper channels.
Since this is a draft the possibility for changes remain and we are hoping to extend the future of the extension's current capabilities if we clearly outline early on what issues this declarativeNetRequest API currently causes as it stands.
Thanks, I appreciate it, and will keep an eye on the various conversations
Linked is the document I attached to the Google Group discussing Manifest V3. The general thought process is to request functionality that is missing for the declarativeNetRequest API and breaks HTTPSE as it currently stands. Will be on stand by for further responses from the group and watching out for any changes in the dNR API.
https://docs.google.com/document/d/1vzSyzrrjOJrUavfOWC5FC2PNwcBOfN2WkK_HcTdTacw/edit?usp=sharing
As an update, set up general asks if declarativeNetRequest were to happen. So far some responses on what can and can't happen, but generally still watching out for written updates to the potential dNR API
Public Conversation here in the Google Group. https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/veJy9uAwS00/UfJdEm2eBgAJ
@zoracon Care to delete #17268 (comment), 5445384#r32812770, 5445384#r32812750?
@pipboy96 Currently in the process from another issue these comments were flagged in. I like to keep topics separate in issues
When was the last edit of this document? Last time I checked we were waiting for a second draft to track what has changed.
Right now we are still waiting for the second draft of Manifest V3. The last time the document was updated was November 18th, 2018.
Public conversation and any further questions can be posted to the public Google group. https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/veJy9uAwS00/UfJdEm2eBgAJ
But unfortunately we are stuck in the position of waiting on changes to the draft
Will Declarative Net Request API be able to modify request and response headers to bypass CORS and access remote content that extension has permission to access ?
We have developed and used various automation tools that heavily rely on modifying request and response headers to send automated web requests to web servers.
Therefore I am hoping that Declarative Net Request API will support
- modifying request and response headers.
- adding new request or response headers.
- removing request or response headers.
There are countless web automation tools deployed on the web that rely on modifying request and response headers.
Apart from automation tools, there are many other legitimate use cases for modifying request or response headers, for example: chrome extension to change user agent.
At the moment there are millions of users that make use of web automation tools.
If Declarative Net Request API fails to add support for modifying headers and webRequest API is phased out then many web automation tools will stop working.
@zoracon Any updates?
Looking at the docs for declarativeNetRequest, they seemed to have added some changes involving dynamic rules and redirect. The Google Group discussion so far came back with with statement:
That said, the extensions team is currently working on a "developer preview" of the MV3 changes. The intent is to implement most of the major changes to the platform so that extension developers can start transitioning and sharing feedback about the process. We don't have an exact date for this, but we're hoping to ship it in the next few months. - Simeon - @dotproto, Extensions Developer Advocate
Right now, the best action for a HTTPS Everywhere volunteer, user, enthusiast would be to look into jumping into the discussion in the public Google Group discussing these changes. On our end, we are discussing internally what next steps look like and doing our best to create a plan.
UPDATE: Blocking capabilities of webRequest are still being deprecated, see https://github.com/uBlockOrigin/uBlock-issues/issues/338#issuecomment-495936668 (original post deleted) and commentary by @gorhill: https://github.com/uBlockOrigin/uBlock-issues/issues/338#issuecomment-496009417.
Thanks @Giltyhub for update!
@andresbase If this warning is too strong, please remove it.
Looks like if webRequest is got removed in future, HTTPS Everywhere will come as a local proxy than an extension.
Here is a Chromium blog post on the impact of the breaking change (from their own perspective): https://blog.chromium.org/2019/06/web-request-and-declarative-net-request.html
Why Not Both? In addition to the performance concerns raised above, the Chrome team strongly believes that users should not have to expose their emails, photos, social media, or any other sensitive data to an extension if the extension doesn’t actually need that access to perform its function. And historically, when extension developers are given the choice between capability and security, the vast majority of developers choose capability. We've seen this repeatedly on the extensions platform with event pages, optional permissions, and activeTab.
It sound like that they are going to depreciate the Web Request APIs anyway...
@cschanaj It sounds like they are trying to justify the choice they made with arguments other than the ones that made them make the choice. They are trying to "sell" the changes to the people.
Update from Mozilla: https://blog.mozilla.org/addons/2019/09/03/mozillas-manifest-v3-faq/
The way it's formulated it seems some degree of rewriting will be necessary for extension to stay compatible even if Firefox will decide to keep webRequestBlocking permission (due to replacement of background pages with service workers).
@zoracon ~~Can EFF (and possibly Tor Project) officially contact Mozilla, explaining them why webRequestBlocking should be kept? I'm sure they will listen.~~
EFF already made such contact.
Removed chrome and firefox labels since they are the only browsers we support anyway.
Update: 12-17 After much back and forth between different privacy extensions and the Chrome team. They have rolled out Manifest V3 into Canary https://groups.google.com/a/chromium.org/forum/#!msg/chromium-extensions/hG6ymUx7NoQ/TiAe0gI5AQAJ
First steps:
- [x] Going to load up HTTPS Everywhere in Canary and see what breaks. (Likely everything :( )
~~- [ ] Next I'm going to move faster to clean up rulesets and run a fetch test as soon as possible~~ ~~- [ ] Once that is done, I am going to package trivialized rulesets for Chrome (because at this point, the more complicated rulesets will just slow the process down of migration to V3)~~
- [x] Internally we are working on messaging on how we will communicate this Chrome version
- [ ] Please allow patience on other features at this time and design implementation will work will be temporarily put on hold until this is sorted out
Unknowns
-
[x] Still not sure how Chrome plans on allocating rulesets, only the privacy extensions mainly use this feature. But this new cap could prove to be a very big challenge. Update 12-17: The 30K rulecap is still in effect. Will most likely convert over to an "EASE mode by default" functionality.
-
[x] Still not sure on the full impact of migrating from background pages to service workers. I'm very concerned on how debugging the extension will be handled now. Update 12-17: There is a bug in discussion here with sleep/wake cycles: https://bugs.chromium.org/p/chromium/issues/detail?id=1024211
@zoracon Current Canary can still load Manifest V2 extensions it seems.
@pipboy96 Right, they don't specify when V2 will hit EoL. To test V3, just have to change the manifest.json file to "manifest_version": 3. Of course, HTTPS Everywhere simply doesn't install in it's current state. I'll outline more tasks once I read through the migration guide.