problem after interrupting program with Control-c and then resuming with e

[ocyzl]

I'm using Chez Scheme Version 9.5.8 64-bit nonthreaded on Windows 7.

The program writes 600 MB of data to a file, then reads it back in and checks that what it read is the same as what it wrote. The writing and reading are done 6 MB at a time from/to a bytevector. I haven't tried to interrupt the writing, but if I interrupt and then resume the reading and checking, sometimes the program finds differences where there are none. For example:

Exception in check-parts: at hex offset 19396FFE, found hex bytes (A3 BD BC 5A 87 BD), but was expecting (A3 BD A0 FB 4B 72)

The first two bytes, A3 and BD, at offsets 19396FFE and 19396FFF from the beginning of the file, match, but the next four bytes, starting at offset 19397000, don't. Actually, the bytes in the file at offset 19397000 are what it was expecting (A0 FB 4B 72), but somehow the bytes that it read (BC 5A 87 BD) came incorrectly from offset 19398000, i.e, 4 KB later.

The problem doesn't always happen, but when it does, it's always like this. At some offset that's a multiple of 4 KB, the program gets data from 4 KB later in the file than it should.

Here's the whole program:

; global variables
(define progress #f)
(define file-port #f)  ; so it can be closed manually on error

; constants
(define value-size 6)  ; in bytes
(define part-size (* value-size 1024 1024))  ; in bytes
(define n-parts 100)  ; number of parts
(define first-value #x123456789abc)

(define shift-left bitwise-arithmetic-shift-left)

(define next-value
  (let* ((n-bits (* 8 value-size))
         (mask (- (shift-left 1 n-bits) 1))
         (d 173961102589771))  ; golden ratio times 2^48
    (lambda (v)
      (bitwise-and mask (+ v d)))))

; fill bytevector bv with somewhat-random values,
; 6 bytes at a time, starting with v (0 <= v < 2^48).
; the length of bv should be a multiple of 6.
; return value is suitable as v in the next call to fill-bv!
(define (fill-bv! bv v)
  (let ((len (bytevector-length bv)))
    (do ((i 0 (+ i value-size))
         (k v (next-value k)))
        ((>= i len) k)
        (bytevector-uint-set! bv i k (endianness little) value-size))))

; check that bytevector bv has the same contents
; as (fill-bv! bv v) would have put there.
(define (check-bv bv v)
  (let ((len (bytevector-length bv)))
    (let recur ((i 0)
                (k v))
      (if (>= i len)
          k  ; check succeeded, next value is k
          (let ((val (bytevector-uint-ref bv i (endianness little) value-size)))
            (if (= val k)
                (recur (+ i value-size) (next-value k))
                (list i val k)))))))  ; check failed, found val instead of k at offset i

; lst is one of:
;   ('write file-n file-val part-n part-val)
;   ('check file-n part-n part-val)
; Files and parts are numbered from 0.
; file-val is the first value in the file.
; part-val is the first value in the part.
(define (set-progress! . lst)
  (set! progress (cons (current-date) lst)))

; write to port all remaining parts of the file, starting
; with part number part-n and value part-val.  (file-n and
; file-val are used only for recording progress.)
(define (write-parts port file-n file-val part-n part-val)
  (set-progress! 'write file-n file-val part-n part-val)
  (when (< part-n n-parts)
    (let* ((bv (make-bytevector part-size))
           (next-val (fill-bv! bv part-val)))
      (put-bytevector port bv)
      (write-parts port file-n file-val (+ 1 part-n) next-val))))

(define (value->bytes v)
  (let ((bv (make-bytevector 6)))
    (bytevector-uint-set! bv 0 v (endianness little) 6)
    (map (lambda (n) (number->string n 16))
         (bytevector->u8-list bv))))

(define (make-error-msg part-n lst)
  (let ((offset (car lst))
        (found-val (cadr lst))
        (expected-val (caddr lst)))
    (format "at hex offset ~a, found hex bytes ~a, but was expecting ~a"
      (number->string (+ (* part-n part-size) offset) 16)
      (value->bytes found-val)
      (value->bytes expected-val))))

(define (check-parts port file-n part-n part-val)
  (set-progress! 'check file-n part-n part-val)
  (let ((bv (get-bytevector-n port part-size)))
    (if (>= part-n n-parts)
        (if (eof-object? bv)
            part-val  ; success, return first value of next file
            (error 'check-parts "file is too long"))
        (if (eof-object? bv)
            (error 'check-parts "file is too short")
            (let ((next-val (check-bv bv part-val)))
              (if (pair? next-val)
                  (error 'check-parts (make-error-msg part-n next-val))
                  (check-parts port file-n (+ 1 part-n) next-val)))))))

(define (filename file-n)
  (number->string file-n))

(define (write-file file-n file-val)
  (set! file-port (open-file-output-port (filename file-n)))
  (write-parts file-port file-n file-val 0 file-val)
  (close-port file-port))

(define (check-file file-n file-val)
  (set! file-port (open-file-input-port (filename file-n)))
  (let ((next-val (check-parts file-port file-n 0 file-val)))
    (close-port file-port)
    next-val))

To create the file, do this once:

(write-file 0 first-value)

It will be named "0".

To check the file:

(check-file 0 first-value)

If interrupting the check with Control-c and then resuming it with e doesn't give an error right away, just try again. I haven't had to try more than seven times before the problem happened.

[ocyzl]

I just tried running the program under Linux (Ubuntu 20.04.4 LTS). The same problem occurs, except that now the incorrect bytes come from 8 KB later in the file than they should, instead of 4 KB later.

For the problem to occur on Linux, I had to make n-parts large enough for the file not to fit in memory. Initially, when it did fit in memory and the program just read it back from the cache without accessing the disk, everything worked fine.