from hone import Hone

def process_mapper(): h = Hone() schema = h.get_schema('curlim__table__mapping.csv') # returns nested JSON schema for input.csv result = h.convert('curlim__table__mapping.csv') # returns converted JSON as Python dictionary

def main(): process_mapper()

if name == 'main': main()

ERROR - ImportError: cannot import name 'Hone' from 'hone' (/Users/user1/code-repo/test/datamapper/hone.py)

Jun 01 '20 23:06 vagarwal77

Correct me if I'm wrong, but it looks like you copied the source code into /Users/user1/code-repo/test/datamapper/. Can you try installing the module via pip (pip3 install hone) instead?

Jun 01 '20 23:06 chamkank

Thanks for pointing...I had not downloaded the code but gave my test file name as hone.py.

Changing it had solved the issue.

Now, i am seeing another issue while parsing attached csv - Vivek-Agarwals-MacBook-Pro:datamapper vivekagarwal$ python3 hone_test.py Traceback (most recent call last): File "hone_test.py", line 15, in main() File "hone_test.py", line 12, in main process_mapper() File "hone_test.py", line 9, in process_mapper result = h.convert('countries.csv') # returns converted JSON as Python dictionary File "/usr/local/lib/python3.7/site-packages/hone/hone.py", line 18, in convert json_struct = self.populate_structure_with_data(column_schema, column_names, data) File "/usr/local/lib/python3.7/site-packages/hone/hone.py", line 36, in populate_structure_with_data exec("json_row"+key_path+"="+"'"+cell+"'") File "", line 1 json_row['euname']='COTE D'IVOIRE' ^ SyntaxError: invalid syntax

Jun 01 '20 23:06 vagarwal77

Thanks for pointing...I had not downloaded the code but gave my test file name as hone.py. Changing it had solved the issue. Now, i am seeing another issue while parsing attached csv - Vivek-Agarwals-MacBook-Pro:datamapper vivekagarwal$ python3 hone_test.py Traceback (most recent call last): File "hone_test.py", line 15, in main() File "hone_test.py", line 12, in main process_mapper() File "hone_test.py", line 9, in process_mapper result = h.convert('countries.csv') # returns converted JSON as Python dictionary File "/usr/local/lib/python3.7/site-packages/hone/hone.py", line 18, in convert json_struct = self.populate_structure_with_data(column_schema, column_names, data) File "/usr/local/lib/python3.7/site-packages/hone/hone.py", line 36, in populate_structure_with_data exec("json_row"+key_path+"="+"'"+cell+"'") File "", line 1 json_row['euname']='COTE D'IVOIRE' ^ SyntaxError: invalid syntax

From: Cham [email protected] Reply-To: chamkank/hone [email protected] Date: Monday, June 1, 2020 at 7:43 PM To: chamkank/hone [email protected] Cc: Vivek Agarwal [email protected], Author [email protected] Subject: Re: [chamkank/hone] cannot import name 'Hone' from 'hone' (#16)

This message originated outside of Concerto Health AI , please use caution before opening attachments, clicking links and replying to this message.

Correct me if I'm wrong, but it looks like you copied the source code into /Users/user1/code-repo/test/datamapper/. Can you try installing the module via pip (pip3 install hone) instead?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fchamkank%2Fhone%2Fissues%2F16%23issuecomment-637183451&data=02%7C01%7Cvagarwal%40concertohealthai.com%7C952169e6d58d42a97c6e08d806859c4f%7C42831aea6fc242368bfdcf4136c882fa%7C1%7C0%7C637266518217504393&sdata=7%2Fovbvzw%2F0V9K0GsnV%2BcBxelmhG%2FsI8NzWLMez0eVhc%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALCPUMFKWAE5WTN7NFJGIK3RUQ4KXANCNFSM4NQHCEVQ&data=02%7C01%7Cvagarwal%40concertohealthai.com%7C952169e6d58d42a97c6e08d806859c4f%7C42831aea6fc242368bfdcf4136c882fa%7C1%7C0%7C637266518217504393&sdata=9kSV1dPE%2BacjJP%2FGHrGXbxPNMSAtVqu8GLR4Q4xg58M%3D&reserved=0.

Jun 01 '20 23:06 vagarwal77

That's definitely a bug. Looks like the issue is that the single quote in 'COTE D'IVOIRE' isn't escaped properly. I'll get to fixing it later today, thanks for catching this!

Jun 02 '20 00:06 chamkank

Thanks!!! I have removed single quotes in CSV and it is working like a charm.

I have a another question (and hopefully last) I have a specific requirement like -

I have a CSV in-hand which user can import.. format of this CSV is not fixed/define.
I have a nested destination json schema which have dict, array, object, string act (sample as https://clinicaltrials.gov/api/query/full_studies?expr=NCT03322540&fmt=json)
I would like to have a mapper file (may be dictionary) which can be used to map CSV column to the destination json attributes... (This mapper file is required as each client will have its own CSV whose format (column names and all) is client specific). With this mapper, it can server as a generic solution for all the clients by just change the mapper file.

Can hone repo code be modified to accommodate this use case? or any other pointers for me to make progress?

Please suggest as it will be very helpful.

Jun 02 '20 00:06 vagarwal77

I have a nested destination json schema which have dict, array, object, string act

Hone currently only converts values to strings, but type casting to JSON data types is a planned feature (see #11).

I would like to have a mapper file (may be dictionary) which can be used to map CSV column to the destination json attributes...

Hone generates the JSON structure automatically based on delimiters in the column names. ~so there's no real way to specify a strict schema. I think the best way to do this is to just read the CSV into a Python list, convert it to a dictionary, write code to make certain fields in the dictionary nested based on your mapper file, and then finally serialize the dict to JSON using json.dumps~ You could just swap the auto-generated schema with the schema from your user-defined mapper file. In the Hone.convert method, you just need to replace column_schema with your own schema.
- ```
def convert(self, csv_filepath):
    self.set_csv_filepath(csv_filepath)
    column_names = self.csv.get_column_names()
    data = self.csv.get_data_rows()
    column_schema = self.generate_full_structure(column_names)
    json_struct = self.populate_structure_with_data(column_schema, column_names, data)
    return json_struct
```

I'm not sure if that was helpful, but please let me know if you have any other questions. I'd be more than happy to help where I can!

Jun 02 '20 01:06 chamkank

Thanks .. very helpful... It is what i have created a mapper file mapper = {

csv_column_name json_attribute attribute_type json_attribute_parent json_attribute_parent_type

'euname':       {'Keyword', 'Array', 'KeywordList', 'Object'},
'linked_country': {'Condition', 'Array', 'ConditionList', 'Object'},
'country': {'BriefSummary', 'String', 'DescriptionModule', 'Object'}

}

And have converted csv to list of dictionary def csv_tolistofDict(): with open('countries.csv', 'r') as read_obj: # pass the file object to DictReader() to get the DictReader object dict_reader = DictReader(read_obj) # get a list of dictionaries from dct_reader list_of_dict = list(dict_reader) # print list of dict i.e. rows print(list_of_dict) eg - OrderedDict([('euname', 'DOMINIQUE'), ('linked_country', ''), ('country', 'Dominica')])

TODO - Some how, i need to iterate each key in this list of dict and find a match in the mapper with key to key comparison....... than fetch the json_attribute name and replace its value with matching list of dict value.... I am not sure how i will handle the json datatype and above condition....

Jun 02 '20 03:06 vagarwal77

I am not sure how i will handle the json datatype and above condition....

#11 will add support for JSON arrays, numbers, objects, etc.
Could you clarify what "above condition" is referring to?

Jun 02 '20 23:06 chamkank