Results 1 to 8 of 8

Thread: DirectX debugging: modified PIX shows callstacks

  1. #1

    DirectX debugging: modified PIX shows callstacks

    Quote Originally Posted by http://msdn.microsoft.com/en-us/library/bb173085(VS.85).aspx
    PIX is a debugging and analysis tool that captures detailed information from a Direct3D application as it executes. PIX can be configured to gather data, such as the list of Direct3D APIs called, timing information, mesh vertices before and after transformations, screenshots, and select statistics. PIX can also be used for debugging vertex and pixel shaders, including setting breakpoints and stepping through shader code.
    Thus, a highly useful tool right from the MS DirectX SDK for e.g. finding the cause of a rendering problem: for any captured frame, you can click through the executed DX API functions and see how the frame is being built up, eventually finding out what part is to blame.



    But what about reversing a closed source application's renderer? PIX does not store a call stack; it merely logs *what* DX functions are called, but not from *where*. Therefore it is not very useful for reversing by default.

    I didn't want to let such a great tool go to waste. After some reversing work I ended up patching PIX to log and show (part of) the call stack for each DirectX call that the target program makes. Each call stack entry has both the virtual address and the module name.



    Example usage of the resulting modified tool is finding out about and messing with a game's renderer, or more simply locating the HUD rendering code and quickly finding the data that it represents (e.g. health, money) rather than having to resort to memory scanning.

    Fullsize screenshot: http://files.uploadffs.com/0/54880c80/callstack.png
    Download: http://www.mediafire.com/file/nhtzmjdi0z2/pix-callstack.rar
    Last edited by arc_; May 25th, 2009 at 19:58.

  2. #2
    Quote Originally Posted by arc_ View Post
    But what about others' applications? PIX does not store a call stack; it merely logs *what* DX functions are called, but not from *where*. Therefore it is not very useful for reversing by default.
    I'm not sure what you're saying. Are you looking for tools to reverse DX apps? Softice works if you move the DX app into a box from fullscreen, so it behaves. Silver is the expert on that. Look up some of his stuff on DX.

    You probably know that a DX app is like any other app up to the point where the DirectX function is called which initializes the directx process. From there on the mouse cursor is controlled by the DX process, so you can't use any normal windows functions or messaging. Besides that, DX apps take over full screen and don't like relinquishing control (BSODs). I got around that, with judicious amounts of help from Silver, by coming in the back door through the mouse driver in ring 0, using softice. Once you get into the DX code it's like any other reversing, with access to all your DX processes.

    Silver has at least one DX crackme. Although DX takes over the mouse cursor, it can't take over the mouse per se. The actual hardware is connected through Windows and that is controlled by the mouse driver. If you can tap into that, which requires some ring 0 work, you can get access to any DX stuff.

  3. #3
    First of all, I'm not looking for anything (anymore) - if you had read the rest of my post, you would have seen that I answered my own question by creating this patch. And now I'm offering it to the public. This is not a question thread, but a tool offering thread

    The point is that having a list of DX calls that the program does to render a frame, and being able to see what each one changes on screen, is easier than taking guesses and placing breakpoints that might be hit hundreds of times in a frame. Take the healthbar example: this could be a polygon rendered using DrawIndexedPrimitive, but that function is called for pretty much every single object in the scene - so with a regular debugger and just a breakpoint on this DX function, you'll spend quite some time to find the game's code that draws the bar (and subsequently, find the memory address/class offset where the health value is stored).

    Concerning full-screen and mouse cursor stuff:
    - PIX works by injecting a .dll into the target process and capturing a frame into a log file when a hotkey is pressed. Later, you exit the target and view the log.
    - Generally it's possible to make DirectX programs run in window mode anyway


    Looking at it again my original post was a bit confusing though. I edited it and added inline screenshots.
    Last edited by arc_; May 25th, 2009 at 19:59.

  4. #4
    Great tool. Better then adding a bunch of DirectX hooks manually for suspect interfaces, etc.

    Had to comment on the IMHO non-sequitur argument by "WaxfordSqueers" detracting your thread.

    When thinking of understanding how a process is using DirectX, honestly who thinks of SoftIce?
    If you use SoftIce as your main debugger, or for some reason wanted to do stuff in kernel mode
    (to hide from anti-hack systems?) then understandable, but otherwise..
    The processes interface to the DX API is userland.
    IMHO it's much easier to look at this stuff from there. Having the return addresses like he's added really
    helps with RE'ing stuff. You can pinpoint where things are happening in the code.

    And for that matter, why couldn't you use a userland debugger like OllyDBG?

    What you, or Sifu "Silver", needs to learn is to force DX to start in windowed mode.
    Use a loader w/DLL and hook the interfaces "->CreateDevice()" (or what ever depending on the DX version).
    In theory you can't do this with any app (depending on how it restores lost devices/surfaces, etc.) but I've been able to force
    every application I've ever tried (to go windowed mode).
    Probably because a lot of developers cut and paste DX samples, and, or, have dev code in that still supports windowed mode et al.
    Some times your stuck with a bare window in the corner at "0,0", but I've hooked CreateWindow/Ex() some times to make the applications main window look better, add a minimize button, etc.

    This applies to DirectInput too (since you mention mouse problem).
    You can hook the "Create", polling, and or event (to inject inputs), etc., to do what you want there.
    I recently wanted an application to respond to fake/spoofed keyboard inputs while it was minimized or in background.
    I hooked the interface "SetCooperativeLevel()" to it set "DISCL_BACKGROUND | DISCL_NONEXCLUSIVE" for the keyboard device.
    (Incidentally using a tool like this you could find the return address to the code and patch it there instead of hooking).

    If you can't or just don't want to do this, you can probably use a dual monitor setup instead.
    I've used this in a dev setup (I work/ed in game dev), should work in a hack/RE/debug setup.
    Game window on the primary, your tools (this, debuggers, etc.) on a second monitor.
    This way you should be able to work on a full screen DirectX window in real time.

    Consider such a tool has usage beyond (gasp) games. Other applications Poker Clients, Educational tools, monitor/control programs, etc.

    And don't be confused by anti-hack systems and reversing.
    Meaning, think of the ideal of reversing then think of what you might have to do to either circumvent, bypass, or otherwise hide behind anti-hack protections and detection systems. The lengths of which that you might have to do, that don't necessarily apply to normal reversing, etc.
    Last edited by Sirmabus; May 26th, 2009 at 00:21.

  5. #5
    Quote Originally Posted by arc_ View Post
    The point is that having a list of DX calls that the program does to render a frame, and being able to see what each one changes on screen, is easier than taking guesses and placing breakpoints that might be hit hundreds of times in a frame.
    Like I said, I did not understand your question. I did not know if you were asking about other apps like softice with relationship to DX. If your tool does the job, all the better.

    From what I recall about DX, which isn't a lot, the code is written like a script. The code also flows regularly from one frame mechanism to another and there isn't a lot the programmer can do to affect that. The thing to remember is that DX came from Microsoft and it's a special engine on its own. As far as I know, the programmer can only affect it in limited ways.

    If you stepped into the code with softice, you could watch one frame being formed. I found ways to step through repetative code by letting certain sections run themselves to completion. I have never been into just setting breakpoints and hoping.

  6. #6
    Quote Originally Posted by Sirmabus View Post
    Great tool. Better then adding a bunch of DirectX hooks manually for suspect interfaces, etc. Had to comment on the IMHO non-sequitur argument by "WaxfordSqueers" detracting your thread.
    The way the post was worded it sounded like the poster might have been asking a question rather than offering advice on a tool. He admitted in a later post that his wording was ambiguous. I took the post as a question asking if other tools worked on DX, so I offered my experience with softice.

    Quote Originally Posted by Sirmabus View Post
    When thinking of understanding how a process is using DirectX, honestly who thinks of SoftIce?
    Maybe someone who's into reverse engineering?

    Quote Originally Posted by Sirmabus View Post
    If you use SoftIce as your main debugger, or for some reason wanted to do stuff in kernel mode (to hide from anti-hack systems?) then understandable, but otherwise..
    I use only softice and it's all I have ever used. I have not turned to Olly because it can't go into ring 0. I have nothing against anyone using Olly and I can't think of any reason why someone would be opposed to using an excellent app like softice. It's becoming an exclusive club, most people can't even get it going in XP.

    Quote Originally Posted by Sirmabus View Post
    The processes interface to the DX API is real mode. IMHO it's much easier to look at this stuff from there. Having the return addresses like he's added really helps with RE'ing stuff. You can pinpoint where things are happening in the code.
    Real mode is DOS and DX does not work in DOS mode. Softice will not only get you directly to the code in question, you can follow the DX engine through kernel mode. In fact, with the DX nms files loaded, I can get the same information presented on the softice screen as the PIX app gives. I'm all for using whatever tools are available. If PIX helps in making RE more intuitive, great. My reply had nothing to do with the validity of the app.

    Quote Originally Posted by Sirmabus View Post
    And for that matter, why couldn't you use a real mode debugger like OllyDBG?
    When I did Silver's DX crackme, I traced the app from the code entry point using softice. It went through the typical windows registration up to UpdateWindow, but that only brought up a windowed frame. Then the code diverted into the DX code, where the engine was initialized. To get through the first part of that code, you had to changed the code to put the DX app in a window, but then the DX switched cursors to DX mode and none of the windows messages or functions could do anything. You could set BPs on anything you wanted, but none of them would take.

    That's why Olly is useless for DX. The only way in was to BP on a mouse driver function and follow it back through ring 0 code to the DX code. Once in the DX code, you could set BP's on the DX functions, and with softice and the nms files, they were all there in plain English.

    Quote Originally Posted by Sirmabus View Post
    What you, or Sifu "Silver", needs to learn is to force DX to start in windowed mode.
    no...what you need to do is learn some respect. Silver is a good member on this board and an expert in DX. He's a decent person who is very helpful with people. Before you go insulting people, maybe you should find out who you're insulting.

  7. #7
    Correction (and edit) I meant "user space" aka "userland", or R3..

    I intend not to start a flame war, but rather enlighten.
    Anything else PM please, we already noised up the guy's poor thread enough.
    Last edited by Sirmabus; May 26th, 2009 at 01:05.

  8. #8
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,204
    Blog Entries
    5
    "Give a man a quote from the FAQ, and he'll ignore it. Print the FAQ, shove it up his ass, kick him in the balls, DDoS his ass and kick/ban him, and the point usually gets through eventually."

Similar Threads

  1. Replies: 10
    Last Post: February 1st, 2013, 09:13
  2. Problem debugging DirectX application
    By LOPAN in forum The Newbie Forum
    Replies: 4
    Last Post: March 1st, 2010, 19:16
  3. DirectX 5
    By Cenobyte in forum OllyDbg Support Forums
    Replies: 2
    Last Post: June 30th, 2005, 08:57
  4. Replies: 3
    Last Post: March 18th, 2003, 16:19
  5. W32DASM shows n0thinq
    By ON'error in forum Malware Analysis and Unpacking Forum
    Replies: 1
    Last Post: December 27th, 2001, 08:31

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •