microsoftgraph
Exception: string index out of range when fetching site lists with expand:fields
Describe the bug
I am trying to fetch the list of lists for a given site, with an expand on the fields. This crashes during deserialisation with a "string index out of range". The full stack trace is the following
File /usr/local/lib/python3.10/site-packages/msgraph/generated/sites/item/lists/item/items/items_request_builder.py:81, in ItemsRequestBuilder.get(self, request_configuration)
78 raise Exception("Http core is null")
79 from ......models.list_item_collection_response import ListItemCollectionResponse
---> 81 return await self.request_adapter.send_async(request_info, ListItemCollectionResponse, error_mapping)
File /usr/local/lib/python3.10/site-packages/kiota_http/httpx_request_adapter.py:196, in HttpxRequestAdapter.send_async(self, request_info, parsable_factory, error_map)
194 return None
195 _deserialized_span = self._start_local_tracing_span("get_object_value", parent_span)
--> 196 value = root_node.get_object_value(parsable_factory)
197 parent_span.set_attribute(DESERIALIZED_MODEL_NAME_KEY, value.__class__.__name__)
198 _deserialized_span.end()
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:222, in JsonParseNode.get_object_value(self, factory)
220 if on_before := self.on_before_assign_field_values:
221 on_before(result)
--> 222 self._assign_field_values(result)
223 if on_after := self.on_after_assign_field_values:
224 on_after(result)
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:291, in JsonParseNode._assign_field_values(self, item)
289 continue
290 field_deserializer = field_deserializers[field_name]
--> 291 field_deserializer(JsonParseNode(field_value))
292 elif item_additional_data is not None:
293 item_additional_data[field_name] = self.try_get_anything(field_value)
File /usr/local/lib/python3.10/site-packages/msgraph/generated/models/list_item_collection_response.py:40, in ListItemCollectionResponse.get_field_deserializers.<locals>.<lambda>(n)
36 from .base_collection_pagination_count_response import BaseCollectionPaginationCountResponse
37 from .list_item import ListItem
39 fields: Dict[str, Callable[[Any], None]] = {
---> 40 "value": lambda n : setattr(self, 'value', n.get_collection_of_object_values(ListItem)),
41 }
42 super_fields = super().get_field_deserializers()
43 fields.update(super_fields)
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:170, in JsonParseNode.get_collection_of_object_values(self, factory)
165 """Gets the collection of type U values from the json node
166 Returns:
167 List[U]: The collection of model object values of the node
168 """
169 if isinstance(self._json_node, list):
--> 170 return list(
171 map(
172 lambda x: self._create_new_node(x).get_object_value(factory), # type: ignore
173 self._json_node,
174 )
175 )
176 return []
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:172, in JsonParseNode.get_collection_of_object_values.<locals>.<lambda>(x)
165 """Gets the collection of type U values from the json node
166 Returns:
167 List[U]: The collection of model object values of the node
168 """
169 if isinstance(self._json_node, list):
170 return list(
171 map(
--> 172 lambda x: self._create_new_node(x).get_object_value(factory), # type: ignore
173 self._json_node,
174 )
175 )
176 return []
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:222, in JsonParseNode.get_object_value(self, factory)
220 if on_before := self.on_before_assign_field_values:
221 on_before(result)
--> 222 self._assign_field_values(result)
223 if on_after := self.on_after_assign_field_values:
224 on_after(result)
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:291, in JsonParseNode._assign_field_values(self, item)
289 continue
290 field_deserializer = field_deserializers[field_name]
--> 291 field_deserializer(JsonParseNode(field_value))
292 elif item_additional_data is not None:
293 item_additional_data[field_name] = self.try_get_anything(field_value)
File /usr/local/lib/python3.10/site-packages/msgraph/generated/models/list_item.py:76, in ListItem.get_field_deserializers.<locals>.<lambda>(n)
68 from .list_item_version import ListItemVersion
69 from .sharepoint_ids import SharepointIds
71 fields: Dict[str, Callable[[Any], None]] = {
72 "analytics": lambda n : setattr(self, 'analytics', n.get_object_value(ItemAnalytics)),
73 "contentType": lambda n : setattr(self, 'content_type', n.get_object_value(ContentTypeInfo)),
74 "documentSetVersions": lambda n : setattr(self, 'document_set_versions', n.get_collection_of_object_values(DocumentSetVersion)),
75 "driveItem": lambda n : setattr(self, 'drive_item', n.get_object_value(DriveItem)),
---> 76 "fields": lambda n : setattr(self, 'fields', n.get_object_value(FieldValueSet)),
77 "sharepointIds": lambda n : setattr(self, 'sharepoint_ids', n.get_object_value(SharepointIds)),
78 "versions": lambda n : setattr(self, 'versions', n.get_collection_of_object_values(ListItemVersion)),
79 }
80 super_fields = super().get_field_deserializers()
81 fields.update(super_fields)
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:222, in JsonParseNode.get_object_value(self, factory)
220 if on_before := self.on_before_assign_field_values:
221 on_before(result)
--> 222 self._assign_field_values(result)
223 if on_after := self.on_after_assign_field_values:
224 on_after(result)
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:293, in JsonParseNode._assign_field_values(self, item)
291 field_deserializer(JsonParseNode(field_value))
292 elif item_additional_data is not None:
--> 293 item_additional_data[field_name] = self.try_get_anything(field_value)
294 else:
295 warnings.warn(
296 f"Found additional property {field_name} to \
297 deserialize but the model doesn't support additional data"
298 )
File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:316, in JsonParseNode.try_get_anything(self, value)
313 if self.__is_four_digit_number(value):
314 return value
--> 316 datetime_obj = pendulum.parse(value)
317 if isinstance(datetime_obj, pendulum.Duration):
318 return datetime_obj.as_timedelta()
File /usr/local/lib/python3.10/site-packages/pendulum/parser.py:30, in parse(text, **options)
26 def parse(text: str, **options: t.Any) -> Date | Time | DateTime | Duration:
27 # Use the mock now value if it exists
28 options["now"] = options.get("now")
---> 30 return _parse(text, **options)
File /usr/local/lib/python3.10/site-packages/pendulum/parser.py:43, in _parse(text, **options)
40 if text == "now":
41 return pendulum.now()
---> 43 parsed = base_parse(text, **options)
45 if isinstance(parsed, datetime.datetime):
46 return pendulum.datetime(
47 parsed.year,
48 parsed.month,
(...)
54 tz=parsed.tzinfo or options.get("tz", UTC),
55 )
File /usr/local/lib/python3.10/site-packages/pendulum/parsing/__init__.py:78, in parse(text, **options)
75 _options: dict[str, Any] = copy.copy(DEFAULT_OPTIONS)
76 _options.update(options)
---> 78 return _normalize(_parse(text, **_options), **_options)
File /usr/local/lib/python3.10/site-packages/pendulum/parsing/__init__.py:116, in _parse(text, **options)
113 return parse_iso8601(text)
115 with contextlib.suppress(ValueError):
--> 116 return _parse_iso8601_interval(text)
118 with contextlib.suppress(ParserError):
119 return _parse_common(text, **options)
File /usr/local/lib/python3.10/site-packages/pendulum/parsing/__init__.py:217, in _parse_iso8601_interval(text)
214 first, last = text.split("/")
215 start = end = duration = None
--> 217 if first[0] == "P":
218 # duration/end
219 duration = parse_iso8601(first)
220 end = parse_iso8601(last)
IndexError: string index out of range
The field that is being deserialised is 'WorkPhone' with a value of '/'. That is clearly not a date even though it is trying to decode it as a date.
Expected behavior
I expect to retrieve the decoded object
How to reproduce
from azure.identity import ClientSecretCredential
from msgraph import GraphServiceClient
import asyncio
from msgraph.generated.users.item.messages.messages_request_builder import MessagesRequestBuilder
credentials = ClientSecretCredential(
tenant_id=SHAREPOINT_TENANT_ID,
client_id=SHAREPOINT_CLIENT_ID,
client_secret=SHAREPOINT_SECRET_VALUE
)
client = GraphServiceClient(credentials, scopes=["https://graph.microsoft.com/.default"])
loop = asyncio.get_event_loop()
site_id = <the_site_id>
query_params = MessagesRequestBuilder.MessagesRequestBuilderGetQueryParameters(expand=['fields'])
configuration = MessagesRequestBuilder.MessagesRequestBuilderGetRequestConfiguration(query_parameters=query_params)
fn = client.sites.by_site_id(site_id)lists.by_list_id("User Information List").items.get(request_configuration=configuration)
a = asyncio.run_coroutine_threadsafe(fn, loop)
a.result()
This obviously triggers the issue only when the data coming from the request contains the mentioned field with the specific value.
SDK Version
1.11.0
Latest version known to work for scenario above?
None
Known Workarounds
No response
Debug output
No response
Configuration
No response
Other information
No response
After several tests, the last configuration that works is the following:
msgraph-core==1.1.6
msgraph-sdk==1.5.3
microsoft-kiota-abstractions==1.5.0
microsoft-kiota-authentication-azure==1.5.0
microsoft-kiota-http==1.5.0
microsoft-kiota-serialization-form==1.5.0
microsoft-kiota-serialization-json==1.2.0
microsoft-kiota-serialization-multipart==1.5.0
microsoft-kiota-serialization-text==1.5.0
with microsoft-kiota-serialization-json==1.2.0 pinned to 1.2.0
Hello @nicolas-lurkin-b12 thanks for using the SDK and for raising this.
This looks like an issue we have been experiencing with our JSON serialization: If I get you right, you mean the latest versions do not work for your scenario and only works for 1.2.0?
Hello @shemogumbe Thanks for your answer. That is correct. As far as I could tell, using microsoft-kiota-serialization-json above 1.2.0 leads to the issue.
I encountered possibly the same issue here.
Main Logic
async def list_outlook_events(self, start_date: datetime, end_date: datetime):
"""
Lists events created or participated in by the user between start_date and end_date.
Ref: https://learn.microsoft.com/en-us/graph/api/event-get?view=graph-rest-1.0
"""
events = []
request_configuration = RequestConfiguration()
request_configuration.headers.add("Prefer", 'outlook.timezone="Asia/Taipei"')
# Instead of RequestConfiguration, manually set headers in the query
result = await self.graph_client.me.events.get(
request_configuration=request_configuration
)
old_event_count = 0
stop_processing = False
while True:
for event in result.value:
try:
# transformation logic
except Exception as e:
print(f"Error processing event: {e}")
traceback.print_exc()
pyperclip.copy(str(event))
sys.exit(0)
if stop_processing:
break
# Handle pagination if there are more events to retrieve
if result.odata_next_link:
result = await self.graph_client.teams.with_url(result.odata_next_link).get()
.
.
.
return events
I have 2 MacBook Pros, that said, MacBook 1 and MacBook 2. The Python script on MacBook 1 works pretty much well and MacBook 2 the same UNTIL TODAY, i.e., MacBook 1 good job and MacBook 2 wanted to play a game with me, a.k.a, failed.
Error Message from Traceback
Traceback (most recent call last):
File "/Users/shu-jenghsieh/Desktop/Automation Center/journal/jira/metrics/collect_stats.py", line 506, in <module>
asyncio.run(collect_calendar_events(input_year, all=True))
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 194, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 721, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "/Users/shu-jenghsieh/Desktop/Automation Center/journal/jira/metrics/collect_stats.py", line 477, in collect_calendar_events
events = await lister.list_outlook_events(start_date_taipei, end_date_taipei)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shu-jenghsieh/Desktop/Automation Center/journal/jira/metrics/teams/list_outlook_events.py", line 546, in list_outlook_events
result = await self.graph_client.teams.with_url(result.odata_next_link).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/msgraph/generated/teams/teams_request_builder.py", line 69, in get
return await self.request_adapter.send_async(request_info, TeamCollectionResponse, error_mapping)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/kiota_http/httpx_request_adapter.py", line 196, in send_async
value = root_node.get_object_value(parsable_factory)
File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 222, in get_object_value
self._assign_field_values(result)
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 291, in _assign_field_values
field_deserializer(JsonParseNode(field_value))
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/msgraph/generated/models/team_collection_response.py", line 40, in <lambda>
"value": lambda n : setattr(self, 'value', n.get_collection_of_object_values(Team)),
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 170, in get_collection_of_object_values
return list(
map(
...<2 lines>...
)
)
File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 172, in <lambda>
lambda x: self._create_new_node(x).get_object_value(factory), # type: ignore
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 222, in get_object_value
self._assign_field_values(result)
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 293, in _assign_field_values
item_additional_data[field_name] = self.try_get_anything(field_value)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 316, in try_get_anything
datetime_obj = pendulum.parse(value)
File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parser.py", line 30, in parse
return _parse(text, **options)
File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parser.py", line 43, in _parse
parsed = base_parse(text, **options)
File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parsing/__init__.py", line 78, in parse
return _normalize(_parse(text, **_options), **_options)
~~~~~~^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parsing/__init__.py", line 116, in _parse
return _parse_iso8601_interval(text)
File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parsing/__init__.py", line 221, in _parse_iso8601_interval
elif last[0] == "P":
~~~~^^^
IndexError: string index out of range
In the beginning, I tweaked try_get_anything in the module of json_parse_node to see what field cannot be parsed and there are quite a lot and NOT every (Outlook) fetching from odata_next_link failed yet still quite some link couldn't return results correctly. By conjecturing some issue in JSON parsing and asking Teach Goo (Google), I saw this discussion. Thanks to @nicolas-lurkin-b12 , my issue is resolved by adding microsoft-kiota-serialization-json==1.3.0 in requirements.txt in my project for automation.
I have tried the following versions for microsoft-kiota-serialization-json
microsoft-kiota-serialization-json==1.3.1
microsoft-kiota-serialization-json==1.3.2
microsoft-kiota-serialization-json==1.3.3
microsoft-kiota-serialization-json==1.5.0
microsoft-kiota-serialization-json==1.6.0
And all led me to the same error message.
Machine Info
ProductName: macOS
ProductVersion: 14.6.1
BuildVersion: 23G93
Python Version
Python 3.13.0
It's installed by brew on the laptop where I encountered the issue.
Hopefully, my experience can help more developers. cc @shemogumbe
I'm experiencing the same problem with my Airflow DAG that connects to OneDrive file with Graph SDK.
What is confusing in my case is that I have multiple similar DAGs connecting to OneDrive, but only one of them has this error. Also if I run the same pipeline manually on Windows, it works.
Our setup:
Ubuntu 22.04.5 LTS
microsoft-kiota-abstractions==1.6.0
microsoft-kiota-authentication-azure==1.0.0
microsoft-kiota-http==1.3.1
microsoft-kiota-serialization-form==1.6.0
microsoft-kiota-serialization-json==1.6.0
microsoft-kiota-serialization-multipart==1.6.0
microsoft-kiota-serialization-text==1.6.0
Python version:
Python 3.12.4
On Windows:
microsoft-kiota-serialization-json==1.3.0
After upgrading from microsoft_kiota_serialization_json-1.4.5 to microsoft_kiota_serialization_json-1.9.2, I could confirm that the issue is resolved.
This issue stems from pendulum library accessing out of range string indices. (Fixed in this MR: https://github.com/python-pendulum/pendulum/pull/843 , yet to be released by upstream)
The library, pendulum, has been removed in this MR (https://github.com/microsoft/kiota-python/commit/8a66dc486eccc10d370a2f0e6459b1c91757ddfa , released).
Testing with SharePoint ListItem API, it was found that the test case of / value no longer produce error, and the issue should therefore be considered fixed.
@lifegivesyoulemons https://touching-notch53.github.io/blQllthpM4j3XJyTCieZgeqF6d44nyowGX96USMznqd_q7WwhRJJ3wfDqg/
