it’s rather a dirty hack than anti-IDA trick, but… it’s quite interesting to be described. as far as we all know, IDA-Pro traces ESP register to reconstruct local variables layout. of course, it’s easy to abuse that mechanism. the tracer emulates very restricted set of CPU commands, however, we want to abuse not only IDA-Pro, but hackers as well. how we can do that?
consider the following example:

int a = 0xC0DE;
mov eax, a
add ebp, 4 ; // change EBP
mov eax, a ; // read memory outside var a
sub bp, 4 ; // change BP (not EBP!)
mov eax, a ; // read memory from… couldn’t really say

mov eax, [esp] ; // read some memory cell (B)
add esp, 4 ; // change ESP
mov eax, [esp] ; // read another memory cell (D)
sub sp, 4 ; // change SP (not ESP)
mov eax, [esp] ; // read memory from (B) or somewhere else
the point is: if LOWORD(ESP/EBP) < 0xFFC, the instruction "sub sp/bp, 4" restores the original value of the register. otherwise we accessing another memory cell. thus, behavior of mov eax, [esp/ebp] depends on ESP/EPB value. interesting!

what’s about IDA-Pro? let’s get IDA-Pro 5.3 and see it! download the compiled example and disassemble it:

mov [ebp+var_4], 0C0DEh
mov eax, [ebp+var_4]
add ebp, 4
mov eax, [ebp+var_4]
sub bp, 4
mov eax, [ebp+var_4]
mov eax, [esp+10h+var_10]
add esp, 4
mov eax, [esp+0Ch+var_C]
sub sp, 4
mov eax, [esp+10h+var_10]
IDA-Pro does not trace EBP at all! um, is she supposed to do it? IDA-Pro flattery tells us [ebp+var_4] and it means: trace EBP by yourself! thanks, IDA-Pro! we will do it! but what about ESP?

as we can see IDA-Pro traces ESP and knows that EP is part of ESP. wow! how clever! I mean how stupid! IDA-Pro totally forgets what’s part. in general case “(WORD) ( ((DWORD) a) + 4) - 4″ is not equal a. it might be the same, but might be different. depends on a. I would prefer to see honest “sp-analysis failed” than unreliable result.

it’s a bug of IDA-Pro. if 32-bit function changes SP she has to stop tracing if she is unable predict the result (and she is unable to do that)

do you want to make reversing harder? just change EBP somewhere in EBP-based function. reverses probably will not find out it fast, especially if it’s a huge function.