0x66 0xF2 instruction prefixes

[ZaiRoN]

Is "66 F2 0F 38 29 1E" a valid instruction?

Some tools identify it as "repne pcmpeqq xmm3, xmmword ptr [esi]" and others mark it as invalid instruction.

[Kayaker]

Just taking a bit of a stab at it, the Intel SSE4 Instruction Set doc gives

66 0F 38 29 /r - PCMPEQQ xmm1, xmm2/m128 - Compare packed qwords in xmm2/m128 and xmm1 for equality.

F2 is the REPNE/REPNZ prefix

so "repne pcmpeqq xmm3, xmm word ptr [esi]" seems to be feasible. Which tool gave you that?

I tried the opcodes in the new ODA online disassembler, but it too didn't recognize it. Didn't test with the usual disassemblers.

[blabberer]

neither windbg nor ollydbg recognizes these opcodes iirc windbg never disassembled 0F series of opcodes correctly (virtual pc uses these opcodes)

0:000> cdb: Reading initial command 'u eip;eb eip 66 F2 0F 38 29 1E 66 F2 0F 38
29 1E ; u eip ; q'
ntdll!KiFastSystemCallRet:
7c90e514 c3              ret
7c90e515 8da42400000000  lea     esp,[esp]
7c90e51c 8d642400        lea     esp,[esp]
ntdll!KiIntSystemCall:
7c90e520 8d542408        lea     edx,[esp+8]
7c90e524 cd2e            int     2Eh
7c90e526 c3              ret
7c90e527 90              nop
ntdll!RtlRaiseException:
7c90e528 55              push    ebp
ntdll!KiFastSystemCallRet:
7c90e514 66              ???
7c90e515 f2              ???
7c90e516 0f              ???
7c90e517 3829            cmp     byte ptr [ecx],ch
7c90e519 1e              push    ds
7c90e51a 66              ???
7c90e51b f2              ???
7c90e51c 0f              ???
quit:

C:\>

which tool disassembled it with repne ?

edit odbg2.01 assembles and disassembles it without the rep/repne (f2/f3) prefix
sandpile.org indicates availabilty of 66 and f3 in 0f 38 2X opcode group

beaEngine seems to disassemble it again without repne prefix though it indicates the opcode length as 6

visual studio 2010 seems to compile the asm block and emit correct opcodes it seems

i get an illegal instruction error when i try tp windbg on it in an old machine (might not have sse 4.1 avx )

c:\>type disop.cpp
#include <stdio.h>
#include <windows.h>
#include "beaengine.h"
int (__stdcall *disme) (LPDISASM);
int main (void)
{
    HMODULE hMod = LoadLibrary("BeaEngine.dll");
    *(FARPROC *)&disme = GetProcAddress(hMod,"_Disasm@4");
    BYTE buff[] = { 0x66,0xf2,0x0f,0x38,0x29,0x1e,0xf2,0x66,0x0f,0x38,0x29,0x1e,0x90,0x90,0xCC,0x90,0x00,0x00,0x00,0x00,0x00 };
    DISASM mydis;    
    memset(&mydis,0,sizeof(DISASM));
    mydis.EIP = (UIntPtr)&buff;
    int i = 0;
    while (i < 7)
    {
        int len = disme(&mydis);
        printf("disasm = %s length = %x\n",mydis.CompleteInstr,len);
        mydis.EIP = mydis.EIP + len;
        i++;
    }
__asm
{
repne pcmpeqq xmm3,xmmword ptr [esi]
repe pcmpeqq xmm3,xmmword ptr [esi]
rep pcmpeqq xmm3,xmmword ptr [esi]
pcmpeqq xmm3,xmmword ptr [esi]

}
    return 0;
}
c:\>cl disop.cpp
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.30319.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

disop.cpp
Microsoft (R) Incremental Linker Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:disop.exe
disop.obj

c:\>disop.exe
disasm = pcmpeqq xmm3, dqword ptr [esi] length = 6
disasm = pcmpeqq xmm3, dqword ptr [esi] length = 6
disasm = nop  length = 1
disasm = nop  length = 1
disasm = int3  length = 1
disasm = nop  length = 1
disasm = add byte ptr [eax], al length = 2

c:\>

004010b3 7d4b            jge     image00400000+0x1100 (00401100)         [br=1]
0:000> 
eax=0013ff6a ebx=7ffda000 ecx=00000007 edx=0040c9a8 esi=00000000 edi=009bf6ee
eip=00401100 esp=0013fcbc ebp=0013ff78 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
image00400000+0x1100:
00401100 f2              ???
0:000> u 
image00400000+0x1100:
00401100 f2              ???
00401101 660f38291e      pcmpeqq xmm3,xmmword ptr [esi]
00401106 f3              ???
00401107 660f38291e      pcmpeqq xmm3,xmmword ptr [esi]
0040110c f3              ???
0040110d 660f38291e      pcmpeqq xmm3,xmmword ptr [esi]
00401112 660f38291e      pcmpeqq xmm3,xmmword ptr [esi]
00401117 33c0            xor     eax,eax
0:000> p
(fdc.f8c): Illegal instruction - code c000001d (first chance)

[ZaiRoN]

Ollydbg, Windbg and ODA are not able to recognize the opcode series, IDA and my disasm engine reveals the pcmpeqq instruction.
According to Intel manual and Sandpile the opcodes sequence should be valid and the correct disasmed instruction is "repne pcmpeqq xmm3, xmmword ptr [esi]".


The problem arises when you combine 0x66 and 0xF2 prefixes into a 3 byte opcodes instruction. There are two instructions defined by this sequence:

66 F2 0F 38 F0 .. : CRC32 Gd, Eb
66 F2 0F 38 F1 .. : CRC32 Gd, Ew


So, the question is: is there a specific 3 byte opcode table (256 entries defined by the 3° opcode) where "66F20F38" initial sequence is always invalid except for 0xF0 and 0xF1 cases?