Azure
TypeError: argument of type 'azureml.dataprep.rslex.StreamInfo' is not iterable
Hi, I encountered below error, I have azureml-opendatasets==1.55.0
Calling to_spark_dataframe()
Traceback (most recent call last):
File "/mnt/c/users/cruiseli/OneDrive - Microsoft/Desktop/workspace/SynapseML-Utils/test_aml.py", line 30, in
Code to reproduce:
import azureml.core
from azureml.core import Datastore, Dataset
from azureml.core.workspace import Workspace
from azureml.core.experiment import Experiment
from azureml.core.authentication import InteractiveLoginAuthentication
import logging
import pandas as pd
import time
import sys
print("Testing opendatasets -- start")
from azureml.opendatasets import NycTlcYellow
from datetime import datetime
from dateutil import parser
end_date = parser.parse('2018-05-30')
start_date = parser.parse('2018-05-28')
nyc_tlc = NycTlcYellow(start_date=start_date, end_date=end_date)
print("Calling to_pandas_dataframe()")
ts = time.time()
nyc_tlc_df = nyc_tlc.to_pandas_dataframe()
te = time.time()
print("Time taken to perform to_pandas_dataframe():" + str(te-ts))
print("Calling to_spark_dataframe()")
ts2 = time.time()
nyc_tlc_df2 = nyc_tlc.to_spark_dataframe()
te2 = time.time()
nyc_tlc_df2.show(2, truncate = False)
print("Time taken to perform to_spark_dataframe():" + str(te2-ts2))
print("Testing opendatasets -- end")
Thank you for reporting this issue, I have investigated it and found the underlying bug. This is now fixed and will be released in the next update to open-datasets package.
@lhrotk The patch for opendatasets==1.55.0.post1 is now released and this should work now
Hi, @lhrotk has the issue been resolved for you? Asking because I'm receiving the same error using 1.55.0.post1...
Hey @anliakho2, I am also getting this same error even after upgrading the azureml.opendatasets to 1.55.0.post1...
