dsq

dsq copied to clipboard

Add support for dumping result as CSV (and eventually other formats)

8

I'm trying to put together a simple data migration pipeline that reads from Excel sheets into a PostgreSQL database, and I'd like to avoid writing new code if possible. My...

infojunkie

dsq --schema missing array in 11GB file

2

**Describe the bug and expected behavior** In my testing with large datasets, there is at least one array of objects that is not being reported with --schema when the array...

mccorkle

**Describe the bug and expected behavior** I have a large number of customer Paquet files which have the timestamp in INT96 format. Timestamp is being interpreted as STR rather than...

andyEllixson

eatonphil

suggestion to use v prefix for tags

5

Hi! Based on this approach https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module to track tool dependencies for a module, as [an example](https://github.com/kamilsk/dotfiles/blob/b7c282d8b5ca726aae7c751a4bc0bcf5e90a9058/toolset/tools.go#L18), I found a slight inconvenience when working with `dsq`, because `dsq` uses a different...

kamilsk

Newline in xlsx header causes crash

5

Using the [attached xlsx sheet](https://github.com/multiprocessio/dsq/files/8985955/newline.xlsx), running `dsq newline.xlsx --schema` generates the following error: ``` json: error calling MarshalJSON for type *runner.Shape: invalid character '\n' in string literal ``` Running `dsq...

infojunkie

test.py fails on caching duration

1

I have installed dsq on my manjaro (arch linux like) raspberry pi 4 with this file: [PKGBUILD](https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=dsq) Everything has worked fine until the check part which runs the test.py script....

dulfox

Adding Miller / mlr to the comparison

1

Could you also include [miller](https://miller.readthedocs.io/)/mlr to the list of tools? Here is the command line to get the same result with pretty formatting: `mlr --icsv --opprint --barred stats1 -a count,mean...

adren

Catchall for now for potential improvements to datastation/dsq. * SQL pre-processing * Import only used fields (see #71) * Do pre-filtering of data in SQLiteWriter, only insert things that match...

eatonphil

Ingest limited columns in basic SQL queries

4

There may not be an existing SQLite parser we can use from Go but for simple queries we can use a PostgreSQL parser, see [here for a good one and...

eatonphil