Allow write precision of TimeUnit other than Nanoseconds

[tony-aq]

Would like to make use of the compression gains based on using an appropriate time precision.

[viragkothari]

+1

[har-ki]

+1

[andyfeller]

+1

One of InfluxDB best practices is to lower point precision to improve service performance and disk usage. This is much more serious of a bug than a suggestion / feature request.

[andyfeller]

It seems there are 2 places where this issue take place:

1. org.influxdb.dto.Point.formattedTime() automatically converts timestamps to nanoseconds; should take precision as parameter passed in whenever point is being converted and written.

    private StringBuilder formatedTime() {
        final StringBuilder sb = new StringBuilder();
        if (null == this.time) {
            this.time = this.precision.convert(System.currentTimeMillis(),TimeUnit.MILLISECONDS);
        }
        sb.append(" ").append(TimeUnit.NANOSECONDS.convert(this.time, this.precision));
        return sb;
    }

2. org.influxdb.impl.InfluxDBImpl.write(BatchPoints) and org.influxdb.impl.InfluxDBImpl.write(String, String, ConsistencyLevel, String) force nanosecond precision.

    @Override
    public void write(final BatchPoints batchPoints) {
        this.batchedCount.addAndGet(batchPoints.getPoints().size());
        RequestBody lineProtocol = RequestBody.create(MEDIA_TYPE_STRING, batchPoints.lineProtocol());
        execute(this.influxDBService.writePoints(
                this.username,
                this.password,
                batchPoints.getDatabase(),
                batchPoints.getRetentionPolicy(),
                TimeUtil.toTimePrecision(TimeUnit.NANOSECONDS),
                batchPoints.getConsistency().value(),
                lineProtocol));
    }

    @Override
    public void write(final String database, final String retentionPolicy, final ConsistencyLevel consistency,
            final String records)   {
        execute(this.influxDBService.writePoints(
                this.username,
                this.password,
                database,
                retentionPolicy,
                TimeUtil.toTimePrecision(TimeUnit.NANOSECONDS),
                consistency.value(),
                RequestBody.create(MEDIA_TYPE_STRING, records)));
    }

The desired precision should be passed down to whatever is formatting points to be written defaulting to nanosecond if/when not specified.

[majst01]

Hi,

thanks for the analysis, can you prepare a PR to show howto solve this issue ?

[joelmarty]

I am working on a PR for this but I am unsure how to test it. Since the tests are directly hitting on an influxdb instance, is there any way to know the time precision of inserted points using queries ?

[andyfeller]

@joelmarty so for testing this I imagine there would be a mock InfluxDB service that would examine if the points' timestamps were of nanosecond precision vs an expected precision.

could you post a link to your fork with the fix as I'm finally being freed to do more time series work and would like to contribute.

[joelmarty]

I created a pull request there: https://github.com/influxdata/influxdb-java/pull/321

It's based on an older version of the client and I don't have the time to work on it atm so there's conflicts but I hope you'll be able to work with that.

[PavelSarek]

+1 We would really appreciate this feature. Thanks for the pull request.

[asashour]

I guess this should be closed now.

[agolovenko]

I guess this should be closed now.

In batching writes all the batch still defaults to nanos. Regardless of points precision.

[andyfeller]

Unless there has been a PR elsewhere to fix this, this is still a legit bug. The fact this has sat here for 2+ years doesn’t reflect well on Influx as supporting its customers, which is probably why everyone moving to Prometheus.

[majst01]

@andyfeller this is not a defect of influxdb-java which only can write what influxdb supports. influxdb-java is not a influx.com product, its a community maintained open source project.

[fmachado]

@andyfeller thanks for pointing out this is still a relevant issue that should be fixed. I'm reopening this ticket and I'll work on it when possible (obviously we would be happy to have your support here, in case you could submit a PR :) ).