Page 1 of 5 12345 LastLast
Results 1 to 15 of 61

Thread: Good binary code profilers?

  1. #1
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5

    Good binary code profilers?

    When the subject of profilers came up briefly in some discussion here on the board a while ago, I remember catching myself feeling surprised that they are practically never mentioned in reversing contexts. Coverage tools like Paimei/pStalker are sometimes (but rarely) mentioned in reversing contexts, and I guess that compared to the more complex profilers, these code coverage tools are also the most natural (and quite efficient too, check out Paimei/pStalker if you haven't already: http://www.woodmann.com/forum/showthread.php?t=10851), but for some purposes, a more profiler centric tool would be more efficient, e.g. in the event of pin-pointing some code that consumes lots of CPU power (e.g. a bug or other suspect piece of code like this one: http://www.woodmann.com/forum/showthread.php?t=11302) or when you want to efficiently pinpoint e.g. some encryption/decryption, checksum code or similar, where the same code blocks are hit a very high number of times during a short period of time. And of course, the target will be an executable for which we don't have the source code.

    My Google searches about this subject have been hard to make good results of. Partly because of the ambiguous "profiler" word, but mostly because most profiler software seems to be primarily aimed and centered around analyzing programs that you already have the source code for. Also, the area of code profiling (let alone binary, source code-less, code profiling) is so small in relation to other areas of interest, that it is easily drowned even more in irrelevant search results, and this also makes it very hard to find out which, if any, products are popular or good within this field.

    So, this is an excellent time to consult the vast experience in the areas of debugging, programming and analyzing code that is present on this board, by asking: Which tools do you use and/or recommend for binary profiling as described above?

    To clarify: What I'm primarily looking for is logging of code execution hits on the basic block level, with hit counters and sorting in decreasing order of the most frequently hit code blocks (possibly of the approximate kind, i.e. it's not necessary that the hits are counted exactly by means of breakpoints, many profilers use sampling techniques too, to speed up the process at the cost of more approximative results).

    Any good tips or ideas, anyone?
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

  2. #2
    I entirely agree that low-level profiling is something of an undiscovered gem in the RCE trade, but I think that this also explains the absence of any particularly strong tools for the job. I have been using AMD CodeAnalyst for a little while and it's pretty good at what it does, though its results are obviously at a loss compared to the equivalent generated with accompanying source-code. Nevertheless, the profiling method successfully performed a very tricky task in almost no time when I needed to isolate QuickTime's decryption routine, though admittedly this particular example is a large block of code that was quite well isolated from the remainder, making for easy identification using EIP samples alone.

    It shouldn't be too difficult to produce a heuristic block-based profiler using a little static analysis, so I'd be disappointed, though not surprised, if none exists. If this is the case, it'd certainly a project to consider home-brewing. So many ideas, but so little time .

    For the record, AMD CodeAnalyst only works with AMD processors. The equivalent for Intel is vTune. Each is available on the respective website for free limited use. It should also be pointed out that these two profilers differentiate themselves from the pack as they sample in kernel-mode, rather than using the Windows debugging API, which makes them indispensable for inter-process and driver-based work.

  3. #3
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5
    Thanks for the tip Admiral! VTune was already in the profiler category of the CRCETL, and I now also added CodeAnalyst:

    http://www.woodmann.com/collaborative/tools/index.php/Category:Profiler_Tools

    I guess that should cover the initial needs for sampling-based profilers. Just like you though, I'd be very interested in getting my hands on some breakpoint-based (of INT3, or even better, memory access breakpoint type) basic block level profilers too, most likely available in the form of OllyDbg or IDA plugins. Hasn't anyone around here ever heard of such a tool? In that case please speak up!

    Hmm, the Conditional Branch Logger OllyDbg plugin (http://www.woodmann.com/collaborative/tools/index.php/Conditional_Branch_Logger) should be relatively easy to turn into such a thing in the worst case...

    Also Admiral, very cool to hear that you had already used the CodeAnalyst profiler for one of the first things that came to mind when I thought about possible areas of use for profilers in the reverse engineering field, i.e. pinpointing crypto code.
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

  4. #4
    When you say profiler, are you talking about TIMING or about COVERAGE?

    I gather for most RCE purposes, COVERAGE is what we are talking about. If so, IDA -> Debug -> Trace works as a "crude" profiler.

    Have Phun
    Blame Microsoft, get l337 !!

  5. #5
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5
    As mentioned above, I'm talking about coverage + hit counters per basic code block, which in some ways I guess could be considered as, well... timing.

    The idea is to pinpoint:

    1. First the exact code that was hit to begin with (coverage).
    2. Then how many times each code block was hit, in order to be able to differentiate the interesting code even more, e.g. in order to be able to pinpoint intensely used code blocks like crypto code inside loops, or the other way around, code that you know will only have been called once during a certain period of time).


    In the coverage step, filters like the ones in Paimei/pStalker are really great and useful, I'd just like to combine this coverage functionality with code block counters/timers too, see what I mean?

    Or the really short answer I guess: Timing
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

  6. #6
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5
    The following is a pure code coverage (i.e. one break/logging per code basic block) plugin for IDA, written by Ilfak:

    http://www.woodmann.com/collaborative/tools/index.php/CoverIt

    For the reasons mentioned in the the accompanying article (http://www.hexblog.com/2006/03/coverage_analyzer.html), it might not be very easy to convert into the counter-logging kind though:

    Quote Originally Posted by Ilfak
    Since we do not have 'real' breakpoints that have to be kept intact after firing, the logic becomes very simple (note that the most difficult part of breakpoint handling is resuming the program execution after it: you have to remove the breakpoint, single step, put the breakpoint back and resume the execution - and the debugged program can return something unexpected at any time, like an event from another thread or another exception).
    So please keep the tips coming for such a plugin/tool!
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

  7. #7
    If you're interested in non-free production tools, there's AutomatedQA'a AQTime, IBM's ProfilerPlus and a few more that aren't so mainstream.

  8. #8
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5
    Thanks for the tips roocoon! I've actually looked at AQtime before, but didn't find any information at that point that indicated it would be good, or even useful, for programs that you don't have the source code for? Do you (or anyone else) know if it's useful for such a situation at all (which is, as stated above, the main objective for my inquiry here). While looking at it again now, I did find that it had some kind of "disassembler" feature, but nothing in that information really indicated that it was useful for profiling purposes, mostly looks to be for viewing assembly code at first glance (http://automatedqa.com/products/aqtime/images/aqtime5/disassembler_panel.gif). But it is of course very possible that is can be useful for this too, just that they don't push it in the marketing material because most people aren't primarily interested in that.

    So, does anyone have experience with AQtime and can say if it's useful at all for non-source code situations?

    About the IBM profiler, after starting to doubt my Google skills there for a moment (and being quite annoyed at Adobe for apparently choosing the same name for their color calibration whatever thingy ), I finally concluded that it was rather you who had mistaken its name, which should be "PurifyPlus" rather than "ProfilerPlus", and that in order to make things even worse, it was not created by IBM, but rather by Rational, and just included in the deal when Rational was acquired by IBM a while ago.

    Anyway, I did at least find a well-hidden comment in the PurifyPlus marketing material saying that it "does not require access to source code and can thus be used with third-party libraries in addition to home-grown code", which bodes well, even though I could not find anything more about this feature or any related aspects at the moment.

    So, here too, does anyone have experience with IBM/Rational PurifyPlus, and can say if it's useful at all for non-source code situations like the ones I inquire?

    Also roocoon, I'd be glad to hear about all those other "not so mainstream" profilers that you are referring to! When it comes to our area of interest, such tools can often very well be the best ones.

    Everyone else are of course still welcome to post any additional tips or suggestions related to my initial inquiry too!
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

  9. #9
    AQtime DOES NOT profile binary. At best, it takes the API dependencies... Ask someone who has tried it from its old v3.

    Intel VTune v5 USED to have PURE ASSEMBLER profiling (Yeah!). However, since v6, they mysteriously discarded that feature. Don't know why.

    Rational PurifyPlus NEEDS source code, OR an executable compiled in Visual Studio with the profiling information on.

    Numega Truetime & Truecoverage (same - need source code or just view the APIs called)...

    Have Phun.
    Blame Microsoft, get l337 !!

  10. #10
    Sorry for the mistake. PurifyPlus it is.

    I had a look at it and it needs debug information (pdb, dbg, or map). It run against a file with no debug info in it but didn't produce any output. (Wasn't there something that produced a map file out of a program or my memory starts failing?)

    Aimless is right about AQTime. It needs source code.

    The others I had in mind were a couple of older programs.
    Turbopower's Sleuth QA (now extinct but some libraries are available for free - check at www.turbopower.com) also needed debug info.
    One of Parasoft's programs that I had come across 15+ years ago. Their newer batches have some similar products like C++ Test but none of them is the one I remember. It had a distinct ugly gray screen with a couple of buttons that used to crash too often to be pleasant (then again, it could have been my patch ). But that used to run with plain binaries.
    Borland too has Gauntlet but I'm sure this will have the same requirements.

    I'll keep my eyes open.

    Take care all.

  11. #11
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5
    Thanks for all the useful info roocoon and Aimless!

    And Aimless, do you think it's possible that Intel VTune or the Numega tools (nowadays rather being included in the group name "Compuware DevPartner Studio") also can handle things as long as they have available debug information (which can often be arranged after the fact with e.g. IDA Pro), contrary to the need for real source code, just like in the case of PurifyPlus?

    So far, it would seem like AMD CodeAnalyst Performance Analyzer (http://www.woodmann.com/collaborative/tools/index.php/CodeAnalyst_Performance_Analyzer) is our prime candidate for a reverser's profiler tool anyway!?

    Admiral also confirmed above that CodeAnalyst works fine, by letting us know he used it successfully to find crypto code in a bloated/complex target for which he definitely didn't have any source code or debug symbols.

    Admiral, what version of CodeAnalyst was that? Is it possible that they also removed the binary profiling features in their most recent releases, in this seemingly widespread conspiracy to exterminate binary profiling? Luckily, the statements on their website indicate that they might not have caught on to that trend just yet, but it would be great with some confirmation anyway Admiral!
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

  12. #12
    I remember I've wrote something similar to that, just as a proof of concept for something I'd like to see in the RCE world. But, later on...
    The only reason for time is so that everything doesn't happen at once. [A. Einstein]

  13. #13
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5
    I'm not exactly sure what you mean by "but later on", but if there was ever a runnable result for that project, I'm sure many people here would be interested in seeing it, if possible?
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

  14. #14
    Well, the problem is that this POC is part of a paper I wrote 7 months ago which is not published yet (It's about proposing a new MDL for Asm in RCE)... Please note, this POC is super dump but statefull and nothing special about it except It would be a breakthrough if I find something like that in the RCE community...
    Last edited by tHE mUTABLE; February 14th, 2008 at 12:24.
    The only reason for time is so that everything doesn't happen at once. [A. Einstein]

  15. #15
    (PART 1)
    Sounds like exactly along the same lines of ideas I've had too.
    And thanks for info here, some more avenues to investigate.

    I'll share my history and research on it.

    I thought one day about five years ago when playing with the memory hack utility "TSearch" (now of days it would be Cheat Engine, MHS, etc.)
    that: "..wouldn't it be nice to have something for code as TSearch is to data/memory?"

    Originally I was interested in just getting call hits and doing deleta operations
    on them (again like TSearch did using it's various filters).
    An example: Messing around with MORPG games to make private bots for them.
    I found my self always trying to find particular functions in the client.
    Like the loot/pick function. If I could take before and after snapshots of call hits, and applying delta filters, I thought I might be able to pinpoint these functions with less work..

    See my blog post here for sort of an introduction of my research:
    http://www.openrce.org/blog/view/838/Real_Time_Tracing


    The first iteration I did a few years ago was to do the sort of the "break point on every function" approach like Pedram Amini's "Process Stalker", "CoverIt", etc.

    FIRST TRY:
    Meant to work in conjunction IDA, you had to first run a script that would go through the DB and create a simple list of every single function entry point.
    The working components consisted of a GUI front end, that injected a service DLL into the target. I put either a JMP5 hook, or int3 hook on every function from the IDA list. This requires the creation of automatic stubs/detours for every function.
    I would analyze the entry points and put a JMP5 where possible (5 bytes open), or a single byte int3 if there wasn't enough room . The int3 hooks had stubs too, to avoid the restore-single step-replace cycle. As long as the hook stubs were align16 and the majority of the hooks were JMP then it's very fast and makes for a real time tool.

    If you try to do this thing in a debugger, and, or using the OS debug API's only the most simplest processes would work. Those functions (do to the OS IPC, and all the layers) are just to slow to use real time. With a DLL in the same process space I could maximize the speed since it's sharing the process'es space. The real acid test to try a multi-threaded resource hogging process like a video game, etc.

    This BP on every approach works fairly well, but IMHO has too many problems to make it generically useful (as a tool). Mainly there are too many errors in trying to find every single function entry point. While IDA may typically find correctly 90% of them, it will either miss many entirely, or create functions at the wrong boundaries.
    I'm not knocking IDA, I'm pretty sure the problem exists with any disassembler trying to find function boundaries. I considered doing runtime analysis (like Olly does) but I'm sure one will run into the same issue. Depends on the executable it's self, what language, what compiler used, etc. And after all, while there might be conventions in higher level languages on an actual function definition, there isn't one for binary code (AFAIK). In particular if you turn on full function link compile in compilers, take a look at the disassembly. You might find all kinds of disjointed functions. Half of something here, the other half in another place.
    At best you could find most of them, but not all no matter how good the analyzer IMHO.

    To get this method to work, I had to spend a lot of time hand fixing either the functions in IDA, and or, editing the function list to get it to work and not crash all the time (mostly because of bad/wrong entry points).
    Also another downside of this is that it requires code modification. Most of the time not a problem, you could could shadow the code space to hide this, etc., but not ideal.

    (continued in PART 2)
    Last edited by Sirmabus; February 15th, 2008 at 12:23.

Similar Threads

  1. Can't Dump a w32 binary (malware)?
    By digdugg in forum Malware Analysis and Unpacking Forum
    Replies: 5
    Last Post: January 17th, 2011, 15:14
  2. Good Laugh
    By NoLoader in forum Off Topic
    Replies: 3
    Last Post: August 30th, 2007, 05:12
  3. REQ: binary calculator
    By yaa in forum Tools of Our Trade (TOT) Messageboard
    Replies: 10
    Last Post: May 3rd, 2004, 04:33
  4. reversing the binary code of .exe and .dll
    By Alawi in forum Advanced Reversing and Programming
    Replies: 4
    Last Post: December 21st, 2001, 14:35
  5. Good time to get into cracking?
    By Unregistered in forum Malware Analysis and Unpacking Forum
    Replies: 3
    Last Post: November 11th, 2001, 13:16

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •