psf
Cookies returned in Set-Cookie header not added to RequestsCookieJar
I am making a post request where the response object has a 'Set-Cookie' with cookie data when inspecting response.headers, but where response.cookies has an empty cookie jar.
My request contains a number of parameters which I am reproducing here, but I doubt the issue has to do with the parameters passed:
response = requests.post(url, headers=headers, cookies=cookies, data=data, allow_redirects=False)
I suspect that the cookies returned in the header might contain non-standard syntax or are too long. I have tried logging in to the site in question using both Firefox and Chrome with Javascript off. Both browsers store the cookies for the response in question and submit the cookies for subsequent requests to the site in question.
Expected Result
My response object contains the following headers. I have edited a few alphanumeric characters in the values for safety reasons, but in terms of length and syntax nothing has been touched:
response.headers
>> {'Cache-Control': 'no-cache,no-store', 'Pragma': 'no-cache', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'Location': '/', 'Set-Cookie': '.AspNetCore.Correlation.jF47v6lTv_fiF9L6IoIRm_NyIwGokyQGecjNK4J6Qyc=; expires=Thu, 01 Jan 1970 00:00:00 GMT; path=/signin-oidc; secure; samesite=none; httponly, .AspNetCore.OpenIdConnect.Nonce.CfDJ8N0wv-Ekr1hBgNlqS3YOmQRnE85Vk8rRFKjF8vB02JU-Q7_OtSCXPp_ukzPNIjKavJA13Ndwb9b7dDzR82gSyrHRa6edkcAUq4WxoL8KIz7ZkSN0BY0Fi6BPPDjZT-tog2HzormhD982Lioo2hYSu1oh6UHBHoVMwrEQfLxN54ySXY45aE2p-TmjBbSI4_LpLpodpubI46HtsrBX_h0x9Qys7bRitwhjX9aH4zHd9hxAWwZ9NQbyfKl7uAxEr8rs5dsz5y410uiMrlwEfPJKnCM=; expires=Thu, 01 Jan 1970 00:00:00 GMT; path=/signin-oidc; secure; samesite=none; httponly, .AspNetCore.Cookies.V2=chunks-2; domain=; path=/; secure; samesite=lax; httponly, .AspNetCore.Cookies.V2C1=CfDJ8N0wv-Ekr1hBgNlqS3YOmQRfOKeY0UmP1Odn0-2suN0E5YV2JDz7dlW5ZIRp2WEOVpjLDknFkmvAxZrV7zy8Mz7wJerKsjKXvlm37iIoM8CmkvXXug4_BFXiPJtPPLa4GDaMlh7MdIf_i2wavk9OrDxlyQi0LVFNURifPuL1-AQne1MpbiOshDP7Ap71SKh9FiNiY4bzGe8LJfPBCMGU-k-CXNz3WlRuKM6I6WuuPZ-F_6mbL1s9E4ecdvqLNhthnOgHzyNZtlfZoYaJSSGQQonQXKo0cI1Png7Kh6RRaGBa43b39XgilTDHPqwsUgYzSuaUBhJ2vW0MfYckMKKsrVK77jVg5oEKqpCCBOvQ4mkYWgIQ6v4TdtLT3IwLp_V_NcovbybhDlWmWfZIGqvPN7jqKcmrYD00blVCKtmUSQh1qF4cuy458Tk64gAp7rcwQQ6m0ye84iGJDuzXBe_qqoUOLiVqVTmZiSENuH-9_9ltF5gnp9IdPHsLFbQqtr8ahlKnyvmRjQSEbpNBSf1N0xga83ZfLeQTRKop-qLiQW4wTNHtJlTIJAuYGF-yuUmHApo-45yOh5qN7naiXejfG0oFP-adnRS9mUu4hErwzM3_QhORBvXjPsSlK-PNoNvNpsqwFq69Z321x4qwSrdBnUd0W7SphPh0tMCurcg4AHSY2uMRX9QXwL8dEGvtvBhF3Omt_g14rhYMoeNDKegTvRFGG_q6DUdf_t9Jn4rhw3uxfC9UUIUfYKsbLlWvr6_Bz4cSu1H287nRCeIONQFjTCnR2C1DFwFaeOwip5GYKeJGJbVlu5nCx4X1rdIhuTpko83IvQaAYP5okESCGm215bST0higylflnG0uChRgTyI6ABusBKLyHYVeBTuqZkF-GebWSoD2iLp8SoNZ8GVfvCQMbEwotPTfiycMG3dfI_03h9X3rJD8AHruOFwJsaly4xxorXwKlbALJ_Ym4jLISaJJEjJ3CpIE_6pch32cxQ-t3GvnduCdfkc3oFRB2cakuW_W5-Bqj3wyF6mqNzfy9rwslGNOrkPiLCcCqbq4KjvzXIYPOw-eP7iPU8OO_a7TJsTOX_QIgMEWuufdxNlA8okiNL-9dqGee3rOrwQpCxr2vWUu29dSisVZwQtQF0Bbn3gAyPtHS3lL0pSeRjupVpC4BMZZzYUdy3oy1TYASBT2uIp9SbvnYd2j65xNutTtgONLY_Zf9p4KewFRRfnZzttX02YFt3yjRsa6IGqUaDMtpy9KaKN9nk_IhrWbTpqwkYYAtrsKpwNoJfyIT2WAJAE6NbrqGbLxNqJBHYDnJRY2yCon0dmHFzzKp5YWPVabopHEtHV6OcpX_0zH0Dt_1Lz61FDHJiQcPg0yCr63q8lzku5MDt8mgrTJoWF4MYr9Bg4R-dzTOBVCaCBytfH2k18IdImmWRjthQR1sBsgyQjh6LeeY4lDClfqmgGEs_dOofsQAc6QcQN2VdD7o3TZR5drOPk2zLsVv25mGs9f9rKM5rMWcHAdGmZ5kzucxPN9S7WbuxR3Iw_BXVx0lgR1aw88jLcWDiC6EmOD07l43EDLxt8Jq3ZARfXxcvVOnaY5Rg9pu1U7ex2WSkowzIRo6XXhe1I-NuFVjZq5wE-GP3NoQznhYbZEF2sMYwvKm7nDArt8TcFbxXy_ZUudiiRQ8zU89m4rzlaSQdyz6qDmx2sYUU61RirfSZDsdhZKtC4_nXRVb-GdfJJDAIpwYwqcoEETodgmrjBa1Vw1uxbdA0F5GnU_pMRrep1hywy9SidcMmeqvCTCkThqaiZ-3DV2RVaLn8vugggtZtAxpxE0POUfEVxmTfZQKZsQ9Ec57AEsJhMsfN7gJJyocjm9j1-GKzrplor76zy3DcxoaQSCZ7pfUG7cIxEVFhQD_cKdSldHnHfAK6S3TYmi7Whn-Y10aANTLlZDzgKkpYugMjpq1RqPE7NfOT-mChe9Wda0lpWh44cxgHA6FUAu1KGfTl-EGg6frYHuYSgoMl3cmDaLJPme2SlrYdjr0rKoM3kWlWFEf9YxstbGgoGoLqSG6A2BJuiRaq5nxT3lLkUinaeqFs42YzeRAouTRrdJhSvVwIU6EEZeDeC6GXl7lC2U_EDdejgBMnANf-uSntMENJF8N4_p4YInxhG-AKLYJRK15MTMVUjg-Qir43SHB02sBbKyDPltr4Ripm5KVnRORahPfVMH8HsYwRc4CSyyebl-Nyl6RajQdVBP2OzT6iFAhCyNbWckkoIhzGtP9HF2v_ilyh4iqEakJnuKzDdicNi0f0LE2-GPo9td4NrFQSePz9wRkvqbsePq6nLlVeljYwtjagPlS6T-y0xAKOceQH-wOmehidWsx__yLytKL9QMXWykdDGrPyB6EarA-BqySWDWIKr0VgQiy8XcAxdJgWltvA3X7qGLYnWO9DmyWovAYt60H7V0pXDESkr8BKnklrVk4ALzEnCWvY1jY-39pxqiAsFKjKDoxFRUSp7zThjRTvPZ5LoNE3BAZU0zHv5ZTNca-qFgwEaMNgD0OmCdPE3VflBxb4_No4Md-LsezU_m4nZvwv3QbmQ4wgMSmIBFJ_JJzjVSo6bJcIdrrNiIQsYbCVIDGb2ZlTSqnBtBlze3MEYhCdBIJWsOyWSTOMygtH5aQ5D4yP4W1k_9eNNP-s9RdwRGvFfuFNO5T8FDZgWKYGUnhtRuhAIALbzIlZuSjd11qNxVtXtEtHMbWoYGHQqjVlvodTEAXVyA5-G-MjSBZzBO5b2mrDAvlKrHT0s8qAii9GBDM4Hrjhts6Q4zuCqV-34GrovkEKVokNDuSotRPycjCdeOLQDHtWsBy9VK_3jrxlSqTq2RHHpKhgeURd6--cjIjX9gMkUc3NRj1zCRkF4_LsGLQcaIkybZjUP4Fh1G9viOqJYlZGlkLQMV1vIIIPN6uFRAVz_QwKbK00R1OAS2HiqTP8dMD_6S4jYNKSbyFBku6bZG5GkG2NxefJx5Y9fFnX7AkzKOK9GNbcYqO9g5uv_yqFEtH0tYpjtumVRcU8lTUQoVZJdC1I12qDfGZ5KOIWOZmnS_IA-rDUYFp2IYtOyLuNvFVIyiOMRKGlTo5i1nAZXK-dF_kAoiIz9SmJa5q_uBliyo_G1iyNcCECtK_B3C4YmFYAOKDiUcStIiPo5DQjRe_94dr2Q-D4RPnxHrE1jKh9SbZpDLHserbTUGXwyHnO0wmoZRrrLleV8bzG2x2KJJyDBKwfXn5MKN51_DesTY95Ek7o-PEJTtAwTFchweH1DbEogvTn7uPJZwkU30ez3XsIJLEFvmXI8Vz0S7tvstpzg0SdtOEcFSkiRT1ONyC1k2fq8CTVH8quVXEmVXbwrSh1K_AH_kc9gMtrPmcIxjnekvDS1-O8HzufgAz8N6fWg4JyLSAn-MlnMfBYGiFNRcJxic2yr6ypGwALcDCHPRsjj-1Hm50LiTSi-FUpTVSuKpnmkt_ECmWVbt86OdfR88j76Pj0f_y3Usa9kq8AL0d8gRzXnxq-73_3pTa71KlGrwPdKbMU04XX6xBsMyMPXDX_Ahh29TlUTLkBDQW_YgTbLL16mOt4s8mVl79zb9yM6zDuSZQ6DL4LuPRSvOushld390_85f2RND0z_yDAB44QXtghPHOM1oyUI3obpnhKp-1canAdsFR6truMIjDceN1IImqXgWzjMzN106wJjadsFyWc5qbZfvpHUSiwNLJDYgwHQI-dUs2mbdNcyotCWmvzUxRoDvJwWXNX4b7FBmfIsZEeqkt3vOs1F5Xz6bXM5-XwQUELdK5m18hfCkhZ2CEhaLd7Nwn-1bQpvjbHg4f0ftMPYxGnSFsePWOFmaUQgUOmE9qr2EgHmWzRe837KWHsqAW3N89TKmlrxecbf3ekxsuqKm7GHHQyzGOz2eHkI6M_k5Ix0eGLN6-E4Gywpbvxfgqgtkv_2cMKuYJtp; domain=; path=/; secure; samesite=lax; httponly, .AspNetCore.Cookies.V2C2=wQS0WKB4ajMPd7QaJarvKMBSR6cvj4d80LSzHJEe1GXtS49K9t2HHcEYjBOG4rtkA4tzE5ZRzuY0hbxxi9jLF1CDzsE7_jG6eCWPVpkNITF-vteDH52CYywJi-RHZ68Yq7UFNCYlf8dw8U9irplfv33IDTt_-1Uy3JcB4DPYYp0wPZAuZAg3GzvnJB6T2LXcxJ-P7a8gwV2Tz0w2BAmbUaD1gL4Jq2abwgkE3NqGDyKsfISega78J9jLgaZvyBFQ86yetjuaPuCZ-ZiM3CPyiIg3Bvv15FK0sGi6blVKxfSsf_RMYxNptMgn39Xh5idQYxW4GwMIvojANiufWmAhNr959CiRm-dVaYjs4FBPG7CwyYf5RSFFoR1bMYv_moTbmct0ttAAY2zv6IifJyEDt7bkrL0dQkI8vAH868hG06huqd8ihFpi0GUyinx1Qgp1LakCesausD8B_cHSNmzYXReQn1Ttu8vCD8T4mLXtBW-J535ybMlKnv49LiIzUG41HLzhkRqqpgst-7hGFIeZUcbM3Gs6J1MkSsEJx8StFxSddn3apbZx2nT9429j4Rq1jk-4SBgOt1y7fTcAknLPMl9aoSbTL7mfIG8yqZuvzJMKWxy0jhtkpfa0lpHnk8bf6ZToQlJInvksRKRmbKTWGEKhSEobN3ONmM_zT1KUf64O6JHj95uh6_xooguHRuQhVnV4N1uIZodzBm-_N_SvAPt3U0qvcWJ7WJykc5B8kKSNz9Dk3nWzSEfXygd88r-aV_3bLIZf1kM6qzB8wd9zEZasb9ud6lIosoQx_riquJxkh4PjRkxKjOvZp9ESaPTISeNBc1tHC-UEy5UglJ95sQ6os7bO7YBz5lXcvoH4kFTj4D-h_DFq3wdQ-hFFiUSuNafF0Vs0vwQmq1er9MVeLmyrEs8mO1M6MsExJ0UIB2jG4qcEc0YVtOaayEXAsQU_02zHJyV96IuXOzh3bYx6K6ztHvECl0XfMjgRFLtAiyByf0zRGWxxf_lfKJMKW2CY1HWcieaaaUsnMQbXSXlCPuEVnnisRTH3eVgjHsP5Vz3uuVJ_e5ZBDVpvuwBuVcinxLKIfaCJjooQKk3Hxwu04cC0OmUOf9NXWzRuWEX-UkXxEE8SiaLzzDFlawHBTRDWD2WNC_4zP_oGI6y9AgM-h3iY8MWApimSUxKClBl9mhmgABM0q7UTApAYr5gkaGxl7nqcEj0wjSk6q0pyV78_OAU4E6c2w0LEOPvlCvdVulMe8T9-x3P37zWzTwNv0s-kJ62KfNqBPuLmzNti9qQObw9TQtOYVGEG1jnemgHSjPggcE5YEN4SnOsdgoI7IKNogzE4_3_zVTsv0pZ-Uhj56guc5G9gOEjDITTigm-6sFvsAIgf4Id3lRdoUmtyW1LhpIh2r87XLPaAPNvBpSHIapWRr7R3uPfqunMvcXagRbOtfHoTCdKq8JClybnonwR6nYTK6tnVXRTPfvLzC2jyWP2YbsNvV0zKLcB3iOhgf; domain=; path=/; secure; samesite=lax; httponly', 'X-Content-Type-Options': 'nosniff', 'X-Robots-Tag': 'noindex, nofollow', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Content-Security-Policy': "report-uri https://csp-report.browser-intake-datadoghq.eu/api/v2/logs?dd-api-key=pubd9330f91e9b3979f9a62f277730740cb&dd-evp-origin=content-security-policy&ddsource=csp-report&ddtags=env%3aproduction%2cversion%3a58610%2cservice%3aDinero.Frontend;default-src 'none';script-src *.chargebee.com *.chargebeestatic.com *.adyen.com connect.facebook.net www.google.com googleads.g.doubleclick.net www.googleadservices.com www.google-analytics.com www.googletagmanager.com ajax.googleapis.com maps.googleapis.com *.linkedin.com snap.licdn.com bat.bing.com 'self' *.dinero.dk *.workbox.dk 'unsafe-eval' 'unsafe-inline' trackcmp.net https://cdneu.net/app.js https://storage.googleapis.com/snowplow-cto-office-tracker-bucket/2.15.0/sp.js analytics.twitter.com static.ads-twitter.com js.userflow.com disutgh7q0ncc.cloudfront.net *.zopim.com *.zendesk.com *.zdassets.com;style-src *.chargebee.com 'self' *.dinero.dk *.workbox.dk data: 'unsafe-inline' js.userflow.com;img-src www.facebook.com www.google.dk www.google.com stats.g.doubleclick.net www.google-analytics.com www.googletagmanager.com *.googleapis.com maps.googleapis.com *.ggpht.com maps.gstatic.com *.linkedin.com bat.bing.com 'self' *.dinero.dk *.workbox.dk data: blob: *.visma.com cb-prod-eu-c1-invoice-logos.s3.eu-central-1.amazonaws.com gallery.mailchimp.com www.partner-ads.com analytics.twitter.com t.co js.userflow.com cdn.nordicapigateway.com nagpublic.blob.core.windows.net *.zopim.com *.zopim.io;font-src 'self' *.dinero.dk *.workbox.dk data: *.zopim.com;child-src 'none';connect-src *.algolia.net https://*.browser-intake-datadoghq.eu dawa.aws.dk www.facebook.com stats.g.doubleclick.net www.google-analytics.com *.oribi.io bat.bing.com 'self' dinero.dk *.dinero.dk *.workbox.dk ws: wss: *.visma.com https://cdneu.net/app.js https://capture-api.eu.autopilotapp.com/ js.userflow.com *.wootric.eu wootric-eligibility.herokuapp.com *.zopim.com *.zopim.io *.zdassets.com wss://*.zopim.com wss://dinero.zendesk.com;worker-src blob:;media-src *.zopim.com;frame-src *.chargebee.com *.adyen.com *.dinero.dk *.workbox.dk player.vimeo.com;frame-ancestors https://connect.visma.com;form-action adfs.mylogbuy.com 'self' https://connect.visma.com;object-src 'none';", 'Date': 'Sat, 28 Jan 2023 17:34:56 GMT', 'Content-Length': '0', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'}
I would expect my response.cookies to contain the cookie data in response.headers['Set-Cookie'].
Actual Result
Instead of containing the cookies from the response headers, the response cookie jar is empty:
response.cookies
>> <RequestsCookieJar[]>
I have tried logging in to the site in question using both Firefox and Chrome with Javascript off. Both browsers store the cookies for the response in question and submit the cookies for subsequent requests to the site in question.
Reproduction Steps
I am unable to provide reproduction steps in the bug report, since a complete reproduction includes user credentials for the site in question. I hope that someone with more in-depth knowledge of Requests is able to reproduce the issue by parsing the Set-Cookie part of the header above with the relevant class or functions in Requests.
I would be happy to provide a bug fixer with test credentials and code to reproduce the issue on the site returning the response. Just reach out to me.
System Information
{
"chardet": {
"version": null
},
"charset_normalizer": {
"version": "2.0.9"
},
"cryptography": {
"version": "36.0.0"
},
"idna": {
"version": "3.3"
},
"implementation": {
"name": "CPython",
"version": "3.10.1"
},
"platform": {
"release": "10",
"system": "Windows"
},
"pyOpenSSL": {
"openssl_version": "101010cf",
"version": "22.0.0"
},
"requests": {
"version": "2.28.2"
},
"system_ssl": {
"version": "101010cf"
},
"urllib3": {
"version": "1.26.7"
},
"using_charset_normalizer": true,
"using_pyopenssl": true
}
Many of those cookies (I didn't check them all while on my phone) were set to expire in the year 1970. Thus they will never end up in the Jar. The server is misconfigured
I can see that some of the cookies are set to expire. The ones I would expect to be added to the cookie jar, and which do not have an expiry date, as far as I can see, are:
.AspNetCore.Cookies.V2 .AspNetCore.Cookies.V2C1 .AspNetCore.Cookies.V2C2
These are the ones that Firefox and Chrome save and use for subsequent requests.
I have created a test account on the site in question and here is code that should be able to reproduce the issue.:
# Script to login to dinero.dk
import requests
from bs4 import BeautifulSoup
username = '[email protected]'
password = '52GeGAMZeN7NwNsUfoE8!'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/109.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'en',
'DNT': '1',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'none',
'Sec-Fetch-User': '?1',
'Pragma': 'no-cache',
'Cache-Control': 'no-cache',
}
headers['Host'] = 'app.dinero.dk'
login_page_url = 'https://app.dinero.dk/'
visit_login_page = requests.get(login_page_url, allow_redirects=False, headers=headers)
dinero_dk_cookies = dict(visit_login_page.cookies)
if visit_login_page.status_code == 302:
headers['Host'] = 'connect.visma.com'
second_page_url = visit_login_page.headers['Location']
visit_second_page = requests.get(second_page_url, allow_redirects=False, headers=headers)
if visit_second_page.status_code == 302:
third_page_url = visit_second_page.headers['Location']
visit_third_page = requests.get(third_page_url, allow_redirects=False, headers=headers)
if visit_third_page.status_code == 200:
connect_visma_cookies = dict(visit_third_page.cookies)
login_soup = BeautifulSoup(visit_third_page.text, 'lxml')
username_form = login_soup.find('form', {'class': 'form-connect-login'})
username_form_data = {}
for input in username_form.find_all('input'):
try:
username_form_data[input['name']] = input['value']
except:
pass
username_form_data['Username'] = username
connect_url = 'https://connect.visma.com/'
headers['Referer'] = third_page_url
headers['Content-Type'] = 'application/x-www-form-urlencoded'
post_username = requests.post(connect_url, allow_redirects=False, headers=headers, cookies=connect_visma_cookies, data=username_form_data)
if post_username.status_code == 302:
post_username_cookies = dict(post_username.cookies)
connect_visma_cookies['rememberUsername'] = post_username_cookies['rememberUsername']
del headers['Content-Type']
password_page_url = 'https://connect.visma.com' + post_username.headers['Location']
get_password_page = requests.get(password_page_url, allow_redirects=False, headers=headers, cookies=connect_visma_cookies)
if get_password_page.status_code == 200:
username_submitted_soup = BeautifulSoup(get_password_page.text, 'lxml')
password_form = username_submitted_soup.find('form', {'class': 'form-connect-login'})
password_form_data = {}
for input in password_form.find_all('input'):
try:
password_form_data[input['name']] = input['value']
except:
pass
password_form_data['password'] = password
password_submit_url = 'https://connect.visma.com/password'
headers['Referer'] = password_page_url
headers['Content-Type'] = 'application/x-www-form-urlencoded'
post_password = requests.post(password_submit_url, allow_redirects=False, headers=headers, cookies=connect_visma_cookies, data=password_form_data)
if post_password.status_code == 302:
post_password_cookies = dict(post_password.cookies)
for key, value in connect_visma_cookies.items():
if '.AspNetCore.Antiforgery' in key:
post_password_cookies[key] = value
del headers['Content-Type']
callback_page_url = 'https://connect.visma.com' + post_password.headers['Location']
get_callback_page = requests.get(callback_page_url, allow_redirects=False, headers=headers, cookies=post_password_cookies)
get_callback_page_cookies = dict(get_callback_page.cookies)
password_submitted_soup = BeautifulSoup(get_callback_page.text, 'lxml')
redirect_form = password_submitted_soup.find('form')
redirect_form_data = {}
for input in redirect_form.find_all('input'):
try:
redirect_form_data[input['name']] = input['value']
except:
pass
redirect_form_data['scope'] = redirect_form_data['scope'].replace(" ","+")
redirect_url = redirect_form['action']
signin_url = redirect_url
headers['Content-Type'] = 'application/x-www-form-urlencoded'
headers['Referer'] = 'https://connect.visma.com/'
headers['Origin'] = 'https://connect.visma.com/'
headers['Host'] = 'app.dinero.dk'
post_signin = requests.post(signin_url, allow_redirects=False, headers=headers, cookies=dinero_dk_cookies, data=redirect_form_data)
post_signin_header_cookies = post_signin.headers['Set-Cookie']
print('Cookie jar has no cookies:')
print(post_signin.cookies)
print('---')
print('Set-Cookie header contains cookies:')
print(post_signin_header_cookies)
I'm able to reproduce this behavior. If I pull the Set-Cookie header out the response and pass it into http.cookies.SimpleCookie, I can fetch the value perfectly fine.
The code I use if fairly simple:
res = requests.post(
URL,
data=JSON,
headers={"Host": "localhost"},
)
res.cookies["testcookie"] # throws a `KeyError`
jar = http.cookies.SimpleCookie(res.headers["Set-Cookie"])
jar["testcookie"].value # works fine
the Set-Cookie header in question looks like this:
testcookie="hello"; Domain=localhost; expires=Fri, 07 Apr 2023 19:57:22 GMT; HttpOnly; Path=/
This only seems to happen when the domain of the cookie is set to localhost, though. Providing other values for the domain and Host, everything works as I would expect.
Thanks for exploring the issue. I notice that the cookies from my response are missing a domain value. Here is one copied from headers['Set-Cookie'] (my reproduction code above still works):
.AspNetCore.Cookies.V2C2=JCRw_3JmNIVpBQ4l4uz951pzrOzWzN9wnsV8f16Dr50QLz7pg55IY01Msaw31FZWMfkxHXqgMgygan99Lsj84YYEYMKguAfAEmCcNsNewq3wYkm14mmaZTQXi44Yw-nWHA0HZwxzmMX13dN5DDEdEjq9Lc-rtURh1d3nhhKlGUst8r7CFtrnRhLSOfJYFTrzZSBYpRypuHcvvPnOxQYvyQXl0vNTbb-GaZB5i1ZPtH05a1xDfhtmUWSfyBONgpdDlBN7_hoElvq-nftkuglgfhN4MSwSfnINxwNm1jAKEJMv3UwC0uqJ9JNMo6CHEmxYHJUfke0w0tN4xtZt7WhUXXl_vPZR7B98fcPtUsa1aa_KzhGjl4584qGH30czopfrpBMTWcH20gFi0t1qVY6fYZxxHTNSG1JnFarGGUuyYY2pr5xq5Vw5WYL7KXUWIZx5GudbSKP8ZdNmNaj-JLIxsOH483zUs2XPbaDERn8NCtJRTEFXcXfDcK3q9Ifw_9rfKXSCShQotwK6fRuRjoryeUiN_I4JQ8Qz6IMOeP3pEHG9e1faHM0Lh0PQQBX30YScU3payh6ImbIr000b1moeI-O0YI9cTQIICL5WPVE1mCc-NShq7eICWMF5fH2XVBAvaD5cw7SA6PeyNwxMvfkLsoeaSeidA6NpRBAWGCHRxAKPFvhVsIPUPrgCNql3v1GavvwIkJSEW8eBXiMWPw2cn2HjmxRaDBUgMUoKAXCcSKvdgVndixtn7TbP2bKmJI_xado6sPbx8mpTiAu5ks7B_26j9rVYfmqWv31J1ZwOSG7Y9z_meTSQZSim74H26u5mwSFGTZEsF_aR5zEzNragUJAO-Yix854zWuva_Czg2g-iTTYuvrPFX-VcDQdfLRcSw-KMZTmydUebV1nQfCmfpOVamlmTzuSkB29wzbPUb0G21hGP2MdEfblfhD0CQN-thWRceOmcgOMll5WW2aZ8jxIK87iyHZuRvfccAhnXjFT6P0rx_q1CKhx1-MO6CJ0lTaR5DY3w_Bz1E3bUxMUjzWkhVuQRExQJHPPNi4zSzR3rspXxI0mCAixWag_GIdBQ2nX2O_MM6Z1RYY-ymouf7aSBeIzOTyJAbJ1u4IsNFJcXl0_i_jNe1ynFfIqqkkM2eHVeCJO8JH3EmxaT76CeggpwM4uiT5Hx_fnpxPVUY3v1IDc6Mly8e6CUdzQp9XvwtK6QRI4Gr5LaYndM5OX0-ElkH7QMgmXw4u1zZO_cOc5aLyDPGeGW6HokGBBSCAk-KBzl7P8Mz5kPTB176qxjs7AsrK3a6rUhqD7HinhwoEX_7CntzkQYu7agS5LxreJ_CiDr52Y27KwSNQHM6NKc9mYuCOBX2DvXkiTfm8SrDEjsEYyXK_65E9xuYh3Yj1xPtsX9Ay1qONVdeZ_gO5XYtXLTFhZgugAxosNsrYbGRFGerfTXTzEIloEqLspLmIjKIlkbTwoXXmfsicDuZRIVRBprbq5z0ZhPlgzLoVoQ7; domain=; path=/; secure; samesite=lax; httponly
Likely this issue is the same as https://github.com/psf/requests/issues/6245
Likely this issue is the same as #6245
Perhaps it extends beyond missing domains and has something to do with not FQ domains? Because in my case, it doesn't work if it is set to localhost either.
I can also reproduce this issue with a localhost cookie. I found this:
Fixed in https://github.com/python/cpython/pull/30108 for 3.11.
from https://github.com/python/cpython/issues/90233
So I would try 3.11 and see if that helps. For me, it did get the cookie, but decided to set it with a domain of .localhost instead of localhost, which wasn't accepted by the server. So I ended up with something like this to fake it:
if s.cookies.get(cookie_name, domain="localhost") is None:
cookie_value = s.cookies.get(cookie_name)
s.cookies.clear(name=cookie_name)
s.cookies.set(cookie_name, cookie_value, domain="")
It seems like allowing localhost cookies at all may be a nonstandard extension, it just happens to be one that is supported by any browser I've tested with as well as other http client libraries.
...domains must have at least two (2) or three (3) periods in them to prevent domains of the form: ".com", ".edu", and "va.us". Any domain that fails within one of the seven special top level domains listed below only require two periods. Any other domain requires at least three. The seven special top level domains are: "COM", "EDU", "NET", "ORG", "GOV", "MIL", and "INT".
Note that the number of periods above probably assumes that a leading period is required. This period is however ignored in modern browsers and it should probably read...
at least one (1) or two (2) periods
