Welcome to the new Woodmann RCE Messageboards Regroupment
Please be patient while the rest of the site is restored.

To all Members of the old RCE Forums:
In order to log in, it will be necessary to reset your forum login password ("I forgot my password") using the original email address you registered with. You will be sent an email with a link to reset your password for that member account.

The old vBulletin forum was converted to phpBB format, requiring the passwords to be reset. If this is a problem for some because of a forgotten email address, please feel free to re-register with a new username. We are happy to welcome old and new members back to the forums! Thanks.

All new accounts are manually activated before you can post. Any questions can be PM'ed to Kayaker.

Ollydbg not disassembling optimized executable correctly.

Support forums for OllyDbg 32-bit Assembler-Level Debugger.
Developed by Oleh Yuschuk (http://www.ollydbg.de)
Locked
Greatwolf

Ollydbg not disassembling optimized executable correctly.

Post by Greatwolf »

I'm checking the produced assembly of an optimized executable I compiled with gcc. However one of the functions is just showing up as a bunch of 'db' bytes at a code address I know is a function(I had a printf printing out it's function address).

Any idea why this is happening? There's a 'nop' 1 byte above the start of the function and 1 byte below it and I also have the base pointer omitted. Could these things cause ollydgb to not properly recognize a function? How can this problem be fixed? Is there any option(s) I can use in olly to make it properly recognize the function?

Thanks
User avatar
dELTA
Posts: 4209
Joined: Mon Oct 30, 2000 7:00 am
Location: Ring -1

Post by dELTA »

What happens if you "reanalyze" while selecting the start address of it?
"Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."
naides
Posts: 1655
Joined: Sat Jan 12, 2002 12:00 pm
Location: Planet Earth

Post by naides »

[quote="Greatwolf"]Any idea why this is happening? There's a 'nop' 1 byte above the start of the function and 1 byte below it and I also have the base pointer omitted. Could these things cause ollydgb to not properly recognize a function?[/QUOTE]


Olly disassembler is actually pretty good for its very compact size, but it was written and conceived for more or less classical assemblers. Once you go into optimized code with the "modern" function prolog (No EBP concept, directly referring to the slippery ESP pointer, so no one, except a fucking computer can keep the fields in the stack straight), Olly and IDA get off base. The inserted nops may have to do with code alignment to optimize code feeding into the pipe and syncronization with other processors, virtual, real or coprocessors. In summary, legibility and elegance in the code is sacrificed to efficiency in the machine. . .

[quote="Greatwolf"]
How can this problem be fixed? Is there any option(s) I can use in olly to make it properly recognize the function?

Thanks[/QUOTE]

IDA with emphasis in the I(nteractive) is the correct approach and even IDA requires painful analysis. imagine when you are going through unkown code, where you have NO idea where functions are or where they start. . .
![Ry4n.4pr1l]!
Posts: 22
Joined: Sat Apr 23, 2011 5:38 pm
Location: lost in translation

Post by ![Ry4n.4pr1l]! »

I am pretty sure some comic by sapheads from the past solved this question..
What I do is I select the area that is byte code in Olly and click binary edit and then just click OK.. This usually reanalyzes the code into asm instructions, if this doesn't work,I try locating that piece of memory in the dump and viewing it differently..If this doesn't work as well I set a break point on the code and just single step each instruction..

http://hackerschool.org/DefconCTF/17/B300.html


Hope that helps.. :D
So much is lost and I guess should never be found..
Greatwolf

Post by Greatwolf »

Thanks Ry4n. The link was helpful and the comic is rather cute. I tried the suggestions but no luck so far.

@delta As far as trying to reanalyze it, it didn't seem to make any difference, the db bytes are still there.

Is there no way to tell ollydbg to interpret a specified region of memory address as a bunch of instructions? It seems like something like this should be possible.
Greatwolf

Post by Greatwolf »

Ok, I just tried it again and I got it to decode correctly! The trick was I had to actually modify one of the bytes or olly won't re-decode the instructions. So I just changed the first push ebp instruction at the beginning since that's pretty safe.

Thanks for the tip everyone.
Locked