'Access denied' from Microsoft.SharePoint.Client ExecuteQueryRetry - previously working queries are now coming back with error

[brett-src]

Target SharePoint environment

SharePoint Online

What SharePoint development model, framework, SDK or API is this about?

SharePoint CSOM

Developer environment

Windows

What browser(s) / client(s) have you tested

• [ ] 💥 Internet Explorer
• [ ] 💥 Microsoft Edge
• [ ] 💥 Google Chrome
• [ ] 💥 FireFox
• [ ] 💥 Safari
• [ ] mobile (iOS/iPadOS)
• [ ] mobile (Android)
• [X] not applicable
• [ ] other (enter in the "Additional environment details" area below)

Additional environment details

No response

Describe the bug / error

SharePoint CSOM scripts that are running in an Azure WebJob have been stable for a long time (years). Recently (about a week ago), these stable scripts started reporting some 'Access denied' failures. These failures occur upon using ExecuteQueryRetry from the ClientContext.

Keep in mind that:

• The service account that runs these scripts has up-to-date and valid credentials (it should not be denied in any of its actions)
• Many of the steps in the script work, but certain ones are only recently failing

Two operations that I've seen fail so far:

1. Add to a FileCollection using FileCreationInformation. In this step, we attempt to overwrite the master page in the master page catalog.
2. DeleteObject on a File. In this step, we attempt to delete a page from Site Pages.

I imagine that some downstream operations might fail but I cannot progress far enough into my scripts to determine what will work.

Last successful run: 6/18/2024 10:01:17 AM (US Central Time) First failure run: 6/20/2024 07:03:30 PM (US Central Time)

My suspicion is that some update happened within SharePoint which perhaps changed the way certain CSOM operations work. Is there any idea as to what could be going on?

Steps to reproduce

Below are the two scripts referenced in the error.

Overwrite master page

var web = clientContext.Site.OpenWeb("relative_site_url_here");
var masterPageCatalog = web.GetCatalog((int)ListTemplateType.MasterPageCatalog);
clientContext.Load(masterPageCatalog.RootFolder);
clientContext.ExecuteQueryRetry();

var masterPageBytes = MethodThatGetsBytesOfAnotherMasterPage();
var masterPageString = Encoding.UTF8.GetString(masterPageBytes);

var masterPageFileCreationInfo = new FileCreationInformation()
{
  Url = string.Format(@"{0}/{1}", masterPageCatalog.RootFolder.ServerRelativeUrl, "seattle.master"),
  Overwrite = true,
  Content = Encoding.UTF8.GetBytes(masterPageString)
};

var updatedMasterPage = masterPageCatalog.RootFolder.Files.Add(masterPageFileCreationInfo);
clientContext.Load(updatedMasterPage);
// Failure of 'Access denied' occurs here
clientContext.ExecuteQueryRetry();

Delete a page from Site Pages

var web = clientContext.Site.OpenWeb("relative_site_url_here");
var sitePages = web.Lists.GetByTitle("Site Pages");
clientContext.Load(sitePages.RootFolder);
clientContext.ExecuteQueryRetry();

// Delete 'How To Use This Library' file if it is found
var howToUsePage = default(Microsoft.SharePoint.Client.File);
try
{
  howToUsePage = sitePages.RootFolder.GetFile("How To Use This Library.aspx");
}
catch { howToUsePage = null; }
if (howToUsePage != null)
{
  howToUsePage.DeleteObject();
  // Failure of 'Access denied' occurs here
  clientContext.ExecuteQueryRetry();
}

Expected behavior

The stable CSOM scripts should not have any Access denied errors.

[brett-src]

Hi there - I am hoping I can get some support! My client's automated processes are broken.

Tagging @qianghuang1 as I've worked with you previously on an issue. Can you help me out? Or let me know who to contact?

[Tran-Kien]

Hi, we are also experiencing the same problem. We're using CSOM to create a sub site, set its masterpage, break permission inheritance. Executed with an account that is Site Col Admin.

It was working last week, now it is giving the following Access is denied response.

[ { "SchemaVersion": "15.0.0.0", "LibraryVersion": "16.0.25019.12006", "ErrorInfo": { "ErrorMessage": "Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))", "ErrorValue": null, "TraceCorrelationId": "[guid]", "ErrorCode": -2147024891, "ErrorTypeName": "System.UnauthorizedAccessException" }, "TraceCorrelationId": "[guid]" } ]

Here's the code snippet.

            var webCreateInfo = new SP.WebCreationInformation();

            webCreateInfo.set_title(siteTitle);
            webCreateInfo.set_language(templateLanguage);
            webCreateInfo.set_url(genUrl);
            webCreateInfo.set_useSamePermissionsAsParentSite(useSamePermissionsAsParentSite);
            webCreateInfo.set_webTemplate(customSiteTemplateId);

            // Add sub site using newWeb global object
            newWeb = web.get_webs().add(webCreateInfo);

            //set masterpage
            newWeb.set_masterUrl(masterpageUrl);

            // Set nav
            newWeb.get_navigation().set_useShared(useShared);

            // BREAK INHERITING GROUPS
            if( useSamePermissionsAsParentSite == "false")
            {
                newWeb.breakRoleInheritance(copyRoleAssignments, clearSubscopes);
            }
            
            // Disable members sharing if (newWeb.get_membersCanShare())
            newWeb.set_membersCanShare(false);

            if (accessRequestEmail != null)
            {
            	newWeb.set_requestAccessEmail(accessRequestEmail);
            }

            newWeb.update();

            context.executeQueryAsync()

[brett-src]

@Tran-Kien sorry to hear, but it's reassuring to know that we're not alone. Is seems more than likely that some SharePoint update caused this type of behavior. Please note from my original post the timeframe in which we saw the last success vs the first failure.

Last successful run: 6/18/2024 10:01:17 AM (US Central Time) First failure run: 6/20/2024 07:03:30 PM (US Central Time)

Do you have any knowledge of your last success or first failure that can help narrow this window?

@qianghuang1 again tagging you in the hopes that someone can start triaging this breaking issue.

[qianghuang1]

@brett-src Thanks for your patience. The better way in my mind to get quick support is to open a ticket by the tenant that run into the issue via Microsoft Support System instead of asking in GitHub. For the issue itself, i didn't own the service, but looks like it lacks permission to do the sensitive operations. I would suggest open a ticket, list the permission that your application identity had (used by the azure webjob), and provide a correlation id of the failed csom request so that supports/devs can better investigate on the issue. Meanwhile, you can try to grant the app with higher privilege to see if it can mitigate this issue (like files.readwrite.all & sites.readwrite.all). (BTW, it's holiday season in US due to the Independence Day, so the response in github may be slower)

[brett-src]

@Tran-Kien I am continuing to triage this issue. It seems that I can replicate the problem within SharePoint Online as a logged-in administrative user. So it is certainly not a problem with our scripting code and it also does not seem to be a true Access Denied problem.

There is some of SharePoint issue going on. I am trying to resolve with Microsoft Support but it's taking forever and they aren't responsive. Any tips @qianghuang1?

[qianghuang1]

@brett-src last week, i internally report this issue to corresponding team, but lacks correlation id + networktrace. Can you detail you reproduce steps or provide the networktrace? You can email it to me.

[qianghuang1]

@brett-src or if you have a ticket created, you can tell me the ticket number, i can forward to corresponding team

[brett-src]

@qianghuang1 thanks very much for your persistence to help!

Action: upload & overwrite file ‘seattle.master’ in Master Page Gallery of site. Result: Access Denied against user who has ‘Full Control’ permissions in library.

Correlation ID: 57af3aa1-b0d6-0000-1a2e-111a27bb2442

If you need more information, let me know and I can email it to you. 👍

[qianghuang1]

@brett-src I personally checked the log, looks like the identity you used to upload the seattle.master lacks one permission. Please make sure the identity have web full control + ACP permission. Follow the doc: https://learn.microsoft.com/en-us/sharepoint/allow-or-prevent-custom-script. You can check the permission in the advanced check view to see if it contain all permission https://www.netwrix.com/how_to_check_sharepoint_permissions.html

[brett-src]

Hey @qianghuang1 - I was just able to verify that the DenyAddAndCustomizePages setting is what's causing the problem. I used the SharePoint admin center to allow custom scripts for a site and then was able to update it's master page.

Why was this changed? I can see that the article you linked has a date of '07/10/2024' on it. What was the old default setting on this? I am trying to find a way to update my scripts to work, but they don't have tenant administrative rights it seems. So trying to create a Tenant and GetSitePropertiesByUrl to load / change the DenyAddAndCustomizePages is not working.

The irony in the last section of the article is pretty hilarious.

[qianghuang1]

@brett-src Sorry that i don't have answer to the question, i am not owning the service. My guess is recent security enhancement may affect this, and i am not sure if this is an expected change. You can keep pushing the ticket to ask the owner team to confirm. For me, the scenario requires ACP permission is a pretty solid permission ask for safety.

[brett-src]

@qianghuang1 unfortunately, I am getting zero help from Microsoft Support. You've been the most helpful, so I really appreciate that you've taken some time to help me investigate.

[qianghuang1]

@brett-src glad to hear it works for you. The owner team replied to me that it is a designed security enhancement. Documented in here: https://learn.microsoft.com/en-us/sharepoint/allow-or-prevent-custom-script#features-affected-when-custom-script-is-blocked We are internally discuss to have a better way to expose this change more explicitly.

[brett-src]

@Tran-Kien you might be interested in this code which can update the DenyAddAndCustomizePages. I created a function that turns the DenyAddAndCustomizePages to DenyAddAndCustomizePagesStatus.Disabled. Then I do my various scripting. At the end, I change DenyAddAndCustomizePages back to DenyAddAndCustomizePagesStatus.Enabled.

var tenant = new Tenant(context);

var siteProperties = tenant.GetSitePropertiesByUrl(url, true);
context.Load(siteProperties);
context.ExecuteQueryRetry();

siteProperties.DenyAddAndCustomizePages = allow ? DenyAddAndCustomizePagesStatus.Disabled : DenyAddAndCustomizePagesStatus.Enabled;

siteProperties.Update();
context.ExecuteQueryRetry();