Ollydbg not disassembling optimized executable correctly.

Greatwolf · Post by **Greatwolf** » Wed May 11, 2011 6:47 pm

I'm checking the produced assembly of an optimized executable I compiled with gcc. However one of the functions is just showing up as a bunch of 'db' bytes at a code address I know is a function(I had a printf printing out it's function address).

Any idea why this is happening? There's a 'nop' 1 byte above the start of the function and 1 byte below it and I also have the base pointer omitted. Could these things cause ollydgb to not properly recognize a function? How can this problem be fixed? Is there any option(s) I can use in olly to make it properly recognize the function?

Thanks

Post by **dELTA** » Wed May 11, 2011 7:08 pm

What happens if you "reanalyze" while selecting the start address of it?

naides · Post by **naides** » Thu May 12, 2011 5:53 am

[quote="Greatwolf"]Any idea why this is happening? There's a 'nop' 1 byte above the start of the function and 1 byte below it and I also have the base pointer omitted. Could these things cause ollydgb to not properly recognize a function?[/QUOTE]

Olly disassembler is actually pretty good for its very compact size, but it was written and conceived for more or less classical assemblers. Once you go into optimized code with the "modern" function prolog (No EBP concept, directly referring to the slippery ESP pointer, so no one, except a fucking computer can keep the fields in the stack straight), Olly and IDA get off base. The inserted nops may have to do with code alignment to optimize code feeding into the pipe and syncronization with other processors, virtual, real or coprocessors. In summary, legibility and elegance in the code is sacrificed to efficiency in the machine. . .

[quote="Greatwolf"]
How can this problem be fixed? Is there any option(s) I can use in olly to make it properly recognize the function?

Thanks[/QUOTE]

IDA with emphasis in the I(nteractive) is the correct approach and even IDA requires painful analysis. imagine when you are going through unkown code, where you have NO idea where functions are or where they start. . .

![Ry4n.4pr1l]! · Post by **![Ry4n.4pr1l]!** » Sat May 14, 2011 9:58 am

I am pretty sure some comic by sapheads from the past solved this question..
What I do is I select the area that is byte code in Olly and click binary edit and then just click OK.. This usually reanalyzes the code into asm instructions, if this doesn't work,I try locating that piece of memory in the dump and viewing it differently..If this doesn't work as well I set a break point on the code and just single step each instruction..

http://hackerschool.org/DefconCTF/17/B300.html

Hope that helps..

Greatwolf · Post by **Greatwolf** » Sat May 14, 2011 4:32 pm

Thanks Ry4n. The link was helpful and the comic is rather cute. I tried the suggestions but no luck so far.

@delta As far as trying to reanalyze it, it didn't seem to make any difference, the db bytes are still there.

Is there no way to tell ollydbg to interpret a specified region of memory address as a bunch of instructions? It seems like something like this should be possible.

Greatwolf · Post by **Greatwolf** » Sat May 14, 2011 5:05 pm

Ok, I just tried it again and I got it to decode correctly! The trick was I had to actually modify one of the bytes or olly won't re-decode the instructions. So I just changed the first push ebp instruction at the beginning since that's pretty safe.

Thanks for the tip everyone.