Not taking mod size - `rol`, `ror`

[mfaerevaag]

Description

When executing instructions rol and ror, the count is sometimes calculated without taking the correct mod size, where size is the operand size.

According to manual, the correct semantics for calculating count: tempCOUNT ← (COUNT & COUNTMASK) MOD SIZE

Reference: Ref. Intel 64 and IA-32 Architecture Software Developer's Manual Vol. 2B 4-519

Affected instructions:

0xc000ff     # rol
0xd200
0x66c100ff
0xd208       # ror
0x66d200
0x66d300

NOTE: All combinations of prefixes and operands are omitted.

Reproduction guide

Instruction:

00000000  C000FF            rol byte [eax],byte 0xff

Input:

bap-mc "c000ff" --show-bil --arch=X86

Observed output:

{
  orig_count1 := 0xFF:8 & 0x1F:8
  mem32 := mem32
             with [pad:32[low:32[EAX]], el]:u8 <- ((mem32[pad:32[low:32[EAX]], el]:u8) << orig_count1) | ((mem32[pad:32[low:32[EAX]], el]:u8) >> (0x8:8 - orig_count1))
  CF := if orig_count1 = 0x0:8 then CF
          else low:1[mem32[pad:32[low:32[EAX]], el]:u8]
  OF := if orig_count1 = 0x0:8 then OF
          else if orig_count1 = 0x1:8
                 then CF ^ (high:1[mem32[pad:32[low:32[EAX]], el]:u8])
                 else unknown[OF undefined after rotate of more then 1 bit]:u1
}

Expected output: orig_count1 is calculated with taking mod of operator size (8).

Rappel

A more specific example done in Rappel using the same instruction as above, with [EAX] set to 0xfd.

~/repos/rappel/bin(master) » ./rappel
eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
esi: 0x00000000 edi: 0x00000000
eip: 0x00400001 esp: 0xffe64510 ebp: 0x00000000
flags: 0x00000202 [cf:0, zf:0, of:0, sf:0, pf:0, af:0, df:0]
> mov byte [esp], 0xfd
eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
esi: 0x00000000 edi: 0x00000000
eip: 0x00400005 esp: 0xffe64510 ebp: 0x00000000
flags: 0x00000202 [cf:0, zf:0, of:0, sf:0, pf:0, af:0, df:0]
> mov eax, esp
eax: 0xffe64510 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
esi: 0x00000000 edi: 0x00000000
eip: 0x00400003 esp: 0xffe64510 ebp: 0x00000000
flags: 0x00000202 [cf:0, zf:0, of:0, sf:0, pf:0, af:0, df:0]
> rol byte [eax], byte 0xff
eax: 0xffe64510 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
esi: 0x00000000 edi: 0x00000000
eip: 0x00400004 esp: 0xffe64510 ebp: 0x00000000
flags: 0x00000202 [cf:0, zf:0, of:0, sf:0, pf:0, af:0, df:0]
> mov ebx, [esp]
eax: 0xffe64510 ebx: 0x000000fe ecx: 0x00000000 edx: 0x00000000
esi: 0x00000000 edi: 0x00000000
eip: 0x00400004 esp: 0xffe64510 ebp: 0x00000000
flags: 0x00000202 [cf:0, zf:0, of:0, sf:0, pf:0, af:0, df:0]
>

According to manual, the correct calculation of tempCount should be ((0xff & 0x1f) % 8), which equals 7.

In the BIL output showed above, count is calculated to 0xff & 0x1f, which is incorrect.

System Info

OS:

# uname -a
Linux ubuntu 4.10.0-28-generic #32-Ubuntu SMP Fri Jun 30 05:32:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=zesty
DISTRIB_DESCRIPTION="Ubuntu 17.04"

BAP:

# bap-mc --version
1.0.0
# bap --version
1.2.0

[ivg]

Yep, in bap.1.3 it is even more visible:

$ bap-mc "c000ff" --show-bil --arch=X86 --show-insn=asm --x86-lifter=legacy
rolb $0xff, (%eax)
{ 
  orig_count1 := 0x1F
  mem := mem 
             with [EAX] <- mem[EAX] << orig_count1 | mem[EAX] >> 8 - orig_count1
  if (orig_count1 = 0) {
    CF := CF
  }
  else {
    CF := low:1[mem[EAX]]
  }
  if (orig_count1 = 0) {
    OF := OF
  }
  else {
    if (orig_count1 = 1) {
      OF := CF ^ high:1[mem[EAX]]
    }
    else {
      OF := unknown[OF undefined after rotate of more then 1 bit]:u1
    }
  }
}