columnify

columnify copied to clipboard

td-agent throwing error - 2023-05-22 14:08:44 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'columnify' utility must be in PATH for -h compression"

3

2023-05-22 14:08:44 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'columnify' utility must be in PATH for -h compression" # which columnify /root/go/bin/columnify # columnify -h Usage of columnify: columnify [-flags] [input...

DivyaYash

Inefficient Parquet Conversion with columnify compared to pyarrow

3

I am working on converting JSONL log files to Parquet format to improve log search capabilities. To achieve this, I've been exploring tools compatible with Fluentd, and I came across...

harishoss

Error while reading json records: Failed to write: bufio.Scanner: token too long

1

vishal7695

Bumps [github.com/xitongsys/parquet-go](https://github.com/xitongsys/parquet-go) from 1.5.3 to 1.6.2. Release notes Sourced from github.com/xitongsys/parquet-go's releases. v1.6.2 No release notes provided. v1.6.1 fix some bugs v1.6.0 Big changes in the type. Not compatiable with...

dependabot[bot]

Using FluentD with Columnify. Running on Kubernetes to push logs to S3 in parquet. Issue's arising when trying to use avro schema with a nested map, or list of strings....

i5okie

hi, i'm using columnify with avro input record. and found that records of logical types(around datetime: date, timemillis, timemicros, timestampmillis, timestampmicros) are broken. for example, the sample data gets result...

ryota717

#45 Implement arrow -> json writer

15

https://github.com/reproio/columnify/issues/45 TODO - [x] prototyping for PoC: (various inputs) -> map's -> arrow -> map's -> json -> parquet - [x] remove parquet writing side Go intermediates: (various inputs) ->...

syucream

retry to implement Arrow record typed intermediate representation, once more! I think we can gradually switch to that by below steps: - prototyping for PoC: (various inputs) -> map's ->...

syucream

Consider whither using Apache Arrow intermediate representation

2

Columnify uses Apache Arrow Schema/Record as an intermediate representation between various input formant and output ( currently only parquet ). It's powerful, fast memory accesses, supports columnar like representation. But...

syucream

How to estimate memory usage?

4

We are running columnify as a part of fluent-plugin-s3 compressor (msgpack to parquet) for these days. But columnify caused no memory error in some environments. So I want to estimate...

okkez