st icon indicating copy to clipboard operation
st copied to clipboard

Validate fixes

Open Stantheman opened this issue 7 years ago • 0 comments

This change makes st a bit quicker for large inputs. The main boost comes from de-duplicating the validate check for every number. The validation check is also replaced with Scalar::Util's looks_like_number, which passes the existing test suite, is part of core, and is implemented in C.

This change, as it is now, would require a version bump on the library since previously, process would die on invalid input. This change assumes the caller has validated the input. It should be straightforward to avoid that if necessary.

I've included some simple runs on a 40mb file showing the before and after below. The run time for this file was reduced to 14s, originally taking 21s. Both times were taken after multiple runs of tests to account for page caches being warm etc.

Current st:

-bash-4.2$ time st --complete /tmp/input.txt
N	min	q1	median	q3	max	sum	mean	stddev	stderrvariance
4.45998e+06	0.002401	0.00371	0.004048	0.004653	0.069798	28189.6	0.00632058	0.00852295	4.03574e-06	7.26406e-05

real	0m21.341s
user	0m20.551s
sys	0m0.790s

Proposed:

-bash-4.2$ time st --complete /tmp/input.txt
N	min	q1	median	q3	max	sum	mean	stddev	stderrvariance
4.45998e+06	0.002401	0.00371	0.004048	0.004653	0.069798	28189.6	0.00632058	0.00852295	4.03574e-06	7.26406e-05

real	0m14.079s
user	0m13.255s
sys	0m0.823s

Stantheman avatar Dec 03 '18 17:12 Stantheman