csv-stream icon indicating copy to clipboard operation
csv-stream copied to clipboard

Clean characters in columns

Open sbehrends opened this issue 6 years ago • 1 comments

While implementing csv-streamify I found out that there was a strange character (or wrongly encoded file) in the CSV that returned on an dirty column name.

example.csv

weird_char,ok_col,some_emoj🤙
1,1,1

The object returning included the exact same string as in the keys.

{
  'weird_char': 1,
  ok_col: 1,
  'some_emoj🤙': 1
}

My request is to add the ability to map or clean this column names to get a cleaner object, like

{
  weird_char: 1,
  ok_col: 1,
  some_emoj: 1
}

One solution would be to add a regex replace in https://github.com/klaemo/csv-stream/blob/master/csv-streamify.js#L56, example

if (state.lineNo === 0) {
  state._columns = state._line.map(col => col.replace(/[^a-zA-Z0-9_]/g,''))
  state.lineNo += 1
  reset()
  return
}

A nicer alternative would be to allow to pass a function as value for columns

const csv = require('csv-streamify')
const parser = csv({
  columns: (cols) => cols.map(col => col.replace(/[^a-zA-Z0-9_]/g,'')),
  objectMode: true,
})

sbehrends avatar Feb 12 '19 23:02 sbehrends

I see that this feature request looks alike to https://github.com/klaemo/csv-stream/pull/40

sbehrends avatar Feb 12 '19 23:02 sbehrends