micropython icon indicating copy to clipboard operation
micropython copied to clipboard

py/objarray: Raise error on out-of-bound memoryview slice start.

Open andrewleech opened this issue 3 years ago • 2 comments

32 bit platforms only support a slice offset start of 24 bit max due to the limited size of https://github.com/micropython/micropython/blob/9dfabcd6d3d080aced888e8474e921f11dc979bb/py/objarray.h#L47

As it stands, if a memoryview object is sliced with a start index of 16MB or more then it's aliased back to the start of the array providing incorrect results.

I initially tried to increase the size of size_t free in the link above when used on a 32bit platform, however this breaks a lot more. Not sure why, presuably the total size of that structure is hardcoded / assumed elsewhere perhaps?

For now, this PR just raises an exception if the start index is too high to prevent accidental use of the wrong data.

andrewleech avatar Aug 07 '22 06:08 andrewleech

Codecov Report

Merging #9028 (5c4153e) into master (d6bc34a) will increase coverage by 0.00%. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #9028   +/-   ##
=======================================
  Coverage   98.50%   98.50%           
=======================================
  Files         155      155           
  Lines       20537    20539    +2     
=======================================
+ Hits        20229    20231    +2     
  Misses        308      308           
Impacted Files Coverage Δ
py/objarray.h 100.00% <ø> (ø)
py/objarray.c 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov-commenter avatar Aug 08 '22 04:08 codecov-commenter

It's rather rare that 32bit platforms will need to essentially memory map a >16MB file. For my own use case I've got a workaround using uctypes that's also quite performant. As such I think a section about this on the https://docs.micropython.org/en/latest/genrst/modules.html#array page is probably enough, possibly with my workaround snippet as example.

Instead of:

 memoryview(buff)[start:end]

you can use

uctypes.bytearray_at(uctypes.address_of(buff)+start, end-start)

andrewleech avatar Aug 11 '22 09:08 andrewleech

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 

github-actions[bot] avatar Jan 20 '23 05:01 github-actions[bot]

I've simplified the error message and updated the test so that it works on 32-bit and 64-bit archs.

dpgeorge avatar Jan 20 '23 05:01 dpgeorge

Thanks for the fix, this was definitely a bad gotcha.

dpgeorge avatar Jan 20 '23 06:01 dpgeorge