Results 1 to 10 of 10

Thread: Windbg remote break command line process ($MFT related)

  1. #1
    Teach, Not Flame Kayaker's Avatar
    Join Date
    Oct 2000
    Posts
    4,124
    Blog Entries
    5

    Windbg remote break command line process ($MFT related)

    I'd like to debug remotely a command line app and am not quite sure how to get it done. Specifically I'm trying to trace into API's used by Strings64. I'm using Windbg and VirtualKD to connect to Win7x64 running in VMWare. The usual steps I use that seems to work for most usermode apps is like this

    Code:
    !gflag +ksl
    sxe ld app.exe
    g
    
    r $proc
    .process
    
    bp /p @$proc nt!NtMapViewOfSection
    g
    
    bp0 /p @$proc ntdll!RtlUserThreadStart
    g
    At this point I usually used a little script to parse the PE header and find the OEP of the app to break at program start. Blabberer mentioned the easier trick that the OEP can be obtained from the x64 register rcx at the break at RtlUserThreadStart.

    Either way I can normally break at the start of a usermode process remotely without too much problem, including cmd.exe. However I'm not sure how to extend that to a command line process such as strings, findstr, dir, etc. In Win7 a command window by itself runs as the process cmd.exe. In Win10 there is also a child process conhost.exe involved.

    ******************************************

    As to why I'm interested in tracing Strings64, in another thread we discussed how if you pipe the output to a file it will pick up the logfile as a search file and can lead to a recursive unending search if any results have been written to the logfile. The solution I found was that the logfile MUST be named alphabetically so it is the very first file found in the search directory, that way it will be empty when Strings opens it for searching, then closed for good and not lead to recursive results.

    Using Procmon I determined that Strings uses FindFirstFile / FindNextFile, a standard routine for listing files in a directory. This led to wondering what character is "first" in a sorted list of filenames. It turns out that the alphabetical sort order you see in Explorer is different from the sort order yielded by FindFirstFile. The definition for the API states that the order in which the function returns the file names is dependent on the file system type, NTFS vs FAT for example.

    This is further explained in a blog post by Raymond Chen where it is revealed that NTFS captures the case mapping table at the time the drive is formatted

    Why do NTFS and Explorer disagree on filename sorting?
    https://devblogs.microsoft.com/oldnewthing/20050617-10/?p=35293

    I suspected that this case mapping table might be what is defined in the NTFS Master File Table ($MFT) as the $UpCase metadata. Using 7-Zip, of all things, I was able to dump the $UpCase metadata and sure enough it seems to match the exact alphabetical order of filenames returned by FindFirstFile / FindNextFile.

    Here is the method of accessing $MFT using 7-Zip
    https://thestarman.pcministry.com/asm/mbr/IntNTFSfs.htm

    Attached is a screenshot of $UpCase opened in HxD. The first usable alphabetical character for filenames is "!", the last (after regular letters) is "~". This matches all the tests I did using Procmon to monitor Strings Query Directory results of filenames.

    Finally, Procmon points to FindFirstFile eventually leading to FLTMGR.SYS being involved where I was wondering if it was possible to actually see the $MFT being accessed through code, that being the bottom line of all this.

    I could of course just make a gui app that uses FindFirstFile / FindNextFile and trace into it that way, but I'm still wondering how to remotely debug a command line process with my usual Windbg setup.

    Kayaker
    Attached Images Attached Images  

  2. #2
    Quote Originally Posted by Kayaker View Post
    I'd like to debug remotely a command line app and am not quite sure how to get it done
    Don't fully understand what you're doing but how different is it from the example Blabbs gave in my DISM thread? I realize I am using a debugger locally but I am debugging a command line app. Why could you not attach to a command line app remotely and follow the commands given by Blabbs? to find the OEP, winmain, etc.?

    I get it that you are trying to see how the app loads re alphanumeric protocol. I managed to trace through k-code to ntfs.sys once using softice. Not sure I can do the same using windbg although ntfs.sys may not be k-code.

    Also, there's a good app for viewing $MFT...Active@ Disk Editor. It makes it easy to find your way around $MFT.

  3. #3
    Teach, Not Flame Kayaker's Avatar
    Join Date
    Oct 2000
    Posts
    4,124
    Blog Entries
    5
    That's one thing I wanted to try, if I can attach to the command line window remotely and see if I could begin debugging any process it starts (Strings64.exe for example). An sxe type exception break on the secondary process directly doesn't seem to catch, or I'm not setting it up right, which is where the problem lies.

    I'm not familiar with all the various remote debugging options such as given here, some might be more suitable for working with a command line process.

    Remote Debugging Review
    http://www.nynaeve.net/?p=38

    However I'm trying to make it work with my existing setup where Windbg is on my host desktop and the target is in a VM. I can catch and debug any gui app I start in the VM, including cmd.exe itself when it starts up, but not once the command window has opened up and you later enter something on the command line. Unfortunately I don't think you can start the target program remotely from Windbg on the host with my configuration.

  4. #4
    Quote Originally Posted by Kayaker View Post
    Unfortunately I don't think you can start the target program remotely from Windbg on the host with my configuration.
    I made notes at the time I was doing remote debugging but can't find them at the moment. I'll keep looking and get back to you if I find them. Of course, I was using a laptop as host and running the target on a desktop.

    I recall there are other ways to do remote debugging but I'm fuzzy on that right now.

  5. #5
    Quote Originally Posted by Kayaker View Post
    I'd like to debug remotely ...
    Don't know if this will help. It's a question I asked in:

    http://www.woodmann.com/forum/showthread.php?15764-USB-drivers-for-Win-7-on-8th-generation-Intel-chipset/page7

    "Is there no way to attach to an application on the target, or start it remotely, with full kernel debugging power, from a host computer?

    If I run !process 0 0, I can see the target app but it seems I cannot attach to it remotely".

    ****

    blabberer replied:

    host
    sxe ld foo.exe
    g

    target double click foo.exe

    windbg will break on loading the exe

    reload symbols

    set process specific breaks as reqd bp /p {eproc } [module!symbol]

    btw keep in mind sxe ld will work only once per boot for one specific binary

    *************more**************

    Another thought that I'd forgotten about:

    Under the key: HKLM\Software\Microsoft\WindowsNT\CurrentVersion\Image File Execution Options

    You can create a new key and give it the name of your application.

    Under the values for the key, you name it Debugger and give it a value of the debugger you want to use. When the app is opened it comes up in the debugger.

    I used this trick to get Windows to open a command prompt at the logon screen. There is an icon there for Accessibility and it leads to a file in the Windows\System32 directory called utilman.exe. I did it because I wanted to use a command window app for loading drivers at that stage before Windows could intercede. Did not help but it was nice to know I could do that.

    The trick is to make a copy of cmd.exe, rename utilman.exe to utilman.bak, then rename the copy of cmd.exe to utilman exe. Second part of trick is to go to the HKLM key above and enter Utilman.exe as a new key name. For value, enter Debugger, and for it's value enter cmd.exe.

    Voila! Next time you boot, hit the Accessibility key and a command window opens.

    Something to ponder. If you used utilman.exe as the key name, as described, and instead of cmd.exe under Debugger, you entered a debugger name, would the accessibility icon at the logon screen open a debugger at the logon screen? And would it be of any use? Apparently ntsd can be used in this manner.

    I did not know this, but apparently hitting the Windows key + U before the logon screen will start utilman.exe hence any app renamed in its place. If you wanted to play a game of solitaire as you logon, you might try that.
    Last edited by WaxfordSqueers; February 3rd, 2020 at 22:28.

  6. #6
    Teach, Not Flame Kayaker's Avatar
    Join Date
    Oct 2000
    Posts
    4,124
    Blog Entries
    5
    Well this is interesting, and unusual. My concern was that I couldn't break on ANY command line process using the above method, but that's not the case. I made my own basic console app and had no problem. The same with Windows findstr, I could break and debug with no issue.

    However with Sysinternals Strings or Strings64 the method will repeatedly not work. I haven't checked with other random console programs or with other Sysinternal apps, but it makes me wonder what the reason is.

    Could there be some anti-debugging code or other unusual reason that 'sxe ld:' does not allow the debugger exception to take hold with Strings? I haven't tried debugging Strings as a local process yet, only under the remote conditions.

    More digging might be in order.

  7. #7
    Quote Originally Posted by Kayaker View Post
    Could there be some anti-debugging code or other unusual reason that 'sxe ld:' does not allow the debugger exception to take hold with Strings?
    Just a thought. Can you remotely break on cmd.exe using sxe ld? If so, I wonder if you could then set a BP in cmd.exe then run strings in the command window to see what is going on?

  8. #8
    Super Moderator
    Join Date
    Dec 2004
    Posts
    1,511
    Blog Entries
    15
    oops was it a long break

    From vista+ Neither NtCreateProcess nor NtCreateProcessEx are used

    kernel32 transfers control to ntdll!NtUserCreateProcess

    which has its name matching counterpart in ntkrxxx

    i think nt!DbgLoadModuleSymbols isnt also called which is why sxe ld probably doesnt work

    for strings (i assume strings is probably compiled with minimum osversion )

    any way lets set a break and go run strings * in target


    Code:
    kd> bp nt!NtCreateUserProcess
    kd> g
    Breakpoint 0 hit
    
    eax=0000005d ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
    eip=82a55a76 esp=80fe8d04 ebp=80fe8d34 iopl=0         nv up ei pl zr na pe nc
    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000246
    nt!NtCreateUserProcess:
    82a55a76 68b8060000      push    6B8h
    The stack wont be correct until SEH_prolog4_GS() is executed
    (eip will be one off when entered through sysenter )
    so lets step over until then (3 steps in win 7 x86)

    Code:
    kd> p
    eax=0000005d ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
    eip=82a55a7b esp=80fe8d00 ebp=80fe8d34 iopl=0         nv up ei pl zr na pe nc
    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000246
    nt!NtCreateUserProcess+0x5:
    82a55a7b 6898438882      push    offset nt! ?? ::FNODOBFM::`string'+0x82d8 (82884398)
    kd> 
    eax=0000005d ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
    eip=82a55a80 esp=80fe8cfc ebp=80fe8d34 iopl=0         nv up ei pl zr na pe nc
    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000246
    nt!NtCreateUserProcess+0xa:
    82a55a80 e883cfe0ff      call    nt!_SEH_prolog4_GS (82862a08)
    kd> 
    eax=80fe8cf0 ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
    eip=82a55a85 esp=80fe8628 ebp=80fe8d00 iopl=0         nv up ei pl nz na pe nc
    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000206
    nt!NtCreateUserProcess+0xf:
    82a55a85 8b4508          mov     eax,dword ptr [ebp+8] ss:0010:80fe8d08=0029ecf0 <<<
    stack will be correct here but symbols aren't loaded so we need to use .reload /f


    Code:
    kd> kb
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    05 0029f048 4ab43f94 000a0d18 000b0e88 00000000 kernel32!CreateProcessW+0x2c
    WARNING: Frame IP not in any known module. Following frames may be wrong.
    06 0029f20c 4ab43cb5 000b56a8 00000000 00000000 0x4ab43f94
    07 0029f46c 4ab43d48 000b56a8 00000000 00000000 0x4ab43cb5
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    0f 0029fa1c 00000000 4ab4829a 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0x1b
    
    kd> .reload /f
    
    kd> kb
     # ChildEBP RetAddr  Args to Child              
    00 80fe8d00 8286687a 0029ecf0 0029eccc 02000000 nt!NtCreateUserProcess+0xf
    01 80fe8d00 773a70b4 0029ecf0 0029eccc 02000000 nt!KiFastCallEntry+0x12a
    02 0029e9b0 773a5784 76dfe5d5 0029ecf0 0029eccc ntdll!KiFastSystemCallRet
    03 0029e9b4 76dfe5d5 0029ecf0 0029eccc 02000000 ntdll!NtCreateUserProcess+0xc
    04 0029f010 76db2079 00000000 000a0d18 000b0e88 kernel32!CreateProcessInternalW+0xe75
    05 0029f048 4ab43f94 000a0d18 000b0e88 00000000 kernel32!CreateProcessW+0x2c
    06 0029f20c 4ab43cb5 000b56a8 00000000 00000000 cmd!ExecPgm+0x20c   <<<<< see  we have correct symbols
    07 0029f46c 4ab43d48 000b56a8 00000000 00000000 cmd!ECWork+0x7f
    08 0029f484 4ab415c5 000b56a8 55cc3b23 00000001 cmd!ExtCom+0x47
    09 0029f8e0 4ab422c0 000b56a8 00000002 76e01e2e cmd!FindFixAndRun+0x1f7
    0a 0029f930 4ab576f0 00000000 000b56a8 4ab64204 cmd!Dispatch+0x14b
    0b 0029f974 4ab4835e 00000005 003d1098 003d1510 cmd!main+0x21a
    0c 0029f9b8 76e03c45 7ffdf000 0029fa04 773c37f5 cmd!_initterm_e+0x163
    0d 0029f9c4 773c37f5 7ffdf000 77637989 00000000 kernel32!BaseThreadInitThunk+0xe
    0e 0029fa04 773c37c8 4ab4829a 7ffdf000 00000000 ntdll!__RtlUserThreadStart+0x70
    0f 0029fa1c 00000000 4ab4829a 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0x1b
    lets run until cmd!ExecPgm()


    Code:
    kd> g 4ab43f94
    
    eax=00000001 ebx=00000000 ecx=76dfebf7 edx=00010180 esi=4ab65260 edi=00000001
    eip=4ab43f94 esp=0029f078 ebp=0029f20c iopl=0         nv up ei pl zr na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
    cmd!ExecPgm+0x20c:
    001b:4ab43f94 89459c          mov     dword ptr [ebp-64h],eax ss:0023:0029f1a8=0029f45c
    we have an EXECUTIVE PROCESS OBJECT but entry point has not been called yet

    which will be called when cmd executes cmd!Wait() with this process objects user mode Handle

    Code:
    kd> !process 0 0 strings.exe
    PROCESS 844c3420  SessionId: 1  Cid: 0220    Peb: 7ffdf000  ParentCid: 0740
        DirBase: 0d14b000  ObjectTable: 968a9598  HandleCount:   0.
        Image: strings.exe
        
    
    kd> dt nt!_EPROCESS -y sec 844c3420
       +0x128 SectionObject : 0x96ad8918 Void
       +0x12c SectionBaseAddress : 0x013a0000 Void  <<<<< this is the Mapped Address of strings.exe
       +0x190 SecurityPort : (null)
    lets set the process context check the ms dos header use !dh to find the AddressOfEntryPoint

    and set a process specific breakpoint on this Address clear all other breakpoints and continue


    Code:
    kd> .process /p /r 844c3420
    Implicit process is now 844c3420
    .cache forcedecodeuser done
    Loading User Symbols
    PEB is paged out (Peb.Ldr = 7ffdf00c).  Type ".hh dbgerr001" for details
    
    kd> db 13a0000 l20
    013a0000  4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00  MZ..............
    013a0010  b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00  ........@.......
    
    kd> bp /p @$proc 13a41C6
    kd> bl
         0 e Disable Clear  82a55a76     0001 (0001) nt!NtCreateUserProcess
         1 e Disable Clear  013a41c6     0001 (0001) 
         Match process data 844c3420
    
    kd> bd 0
    
    kd> g
    we hit the entrypoint


    Code:
    Breakpoint 1 hit
    eax=76e03c33 ebx=7ffdf000 ecx=00000000 edx=013a41c6 esi=00000000 edi=00000000
    eip=013a41c6 esp=0013fe1c ebp=0013fe24 iopl=0         nv up ei pl zr na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
    001b:013a41c6 e8524a0000      call    013a8c1d
    reload symbols and check

    Code:
    kd> .reload /f
    
    kd> r
    eax=76e03c33 ebx=7ffdf000 ecx=00000000 edx=013a41c6 esi=00000000 edi=00000000
    eip=013a41c6 esp=0013fe1c ebp=0013fe24 iopl=0         nv up ei pl zr na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
    strings+0x41c6:
    001b:013a41c6 e8524a0000      call    strings+0x8c1d (013a8c1d)
    
    kd> kb
     # ChildEBP RetAddr  Args to Child              
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00 0013fe24 773c37f5 7ffdf000 760b4914 00000000 strings+0x41c6
    01 0013fe64 773c37c8 013a41c6 7ffdf000 00000000 ntdll!__RtlUserThreadStart+0x70
    02 0013fe7c 00000000 013a41c6 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0x1b
    see the commandline used to start the process

    Code:
    ansi_string->Buffer
    
    kd> dpa KERNELBASE!BaseAnsiCommandLine+4 l1
    7557478c  00231de0 "strings.exe   *"
    
    or ansi_string display
    
    kd> ds KERNELBASE!BaseAnsiCommandLine
    00231de0  "strings.exe   *"

  9. #9
    Brilliant stuff, Blabbs. Worth waiting for.

  10. #10
    Teach, Not Flame Kayaker's Avatar
    Join Date
    Oct 2000
    Posts
    4,124
    Blog Entries
    5
    Hey B, thanks for stopping execution and handling the break

    Nice, a break on nt!NtCreateUserProcess works really well as a replacement for sxe ld:[] if a process won't break on the latter exception. The rest of the procedure is pretty much the same.

    This time I used Sysinternals hex2dec64, which also showed the problem of not breaking for sxe ld:[]. What is it with these apps?

    Here is what the stack looks like in Win7x64 at the first break:

    Code:
    kd> !gflag +ksl
    kd> bp nt!NtCreateUserProcess
    kd> g
    
    Breakpoint 0 hit
    nt!NtCreateUserProcess:
    fffff800`02b784a0 fff3            push    rbx
    
    kd> kb
     # RetAddr           : Args to Child                                                           : Call Site
    00 fffff800`028cc8d3 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!NtCreateUserProcess
    01 00000000`77661dea : 00000000`7750efcd 00000000`00431620 00000000`00000000 00000000`00080000 : nt!KiSystemServiceCopyEnd+0x13
    02 00000000`7750efcd : 00000000`00431620 00000000`00000000 00000000`00080000 00000000`0029e828 : 0x77661dea
    03 00000000`00431620 : 00000000`00000000 00000000`00080000 00000000`0029e828 00000000`00000000 : 0x7750efcd
    04 00000000`00000000 : 00000000`00080000 00000000`0029e828 00000000`00000000 00000000`00000000 : 0x431620
    All I did from there is execute 'gu' to go back one step to KiSystemServiceCopyEnd

    Code:
    kd> gu
    
    nt!KiSystemServiceCopyEnd+0x13:
    fffff800`028cc8d3 65ff042538220000 inc     dword ptr gs:[2238h]
    
    kd> kb
     # RetAddr           : Args to Child                                                           : Call Site
    00 00000000`77661dea : 00000000`7750efcd 00000000`00431620 00000000`00000000 00000000`00080000 : nt!KiSystemServiceCopyEnd+0x13
    01 00000000`7750efcd : 00000000`00431620 00000000`00000000 00000000`00080000 00000000`0029e828 : 0x77661dea
    02 00000000`00431620 : 00000000`00000000 00000000`00080000 00000000`0029e828 00000000`00000000 : 0x7750efcd
    03 00000000`00000000 : 00000000`00080000 00000000`0029e828 00000000`00000000 00000000`00000000 : 0x431620
    At this point the process was loaded and I could continue with the previous steps to get to RtlUserThreadStart

    Code:
    kd> !process 0 0
    
    PROCESS fffffa8002a8a060
        Image: hex2dec64.exe
    
    kd> .process fffffa8002a8a060
    Implicit process is now fffffa80`02a8a060
    
    kd> bp0 /p @$proc nt!NtMapViewOfSection
    kd> g
    
    Breakpoint 0 hit
    nt!NtMapViewOfSection:
    fffff800`02be69e0 48895c2410      mov     qword ptr [rsp+10h],rbx
    
    kd> bp0 /p @$proc ntdll!RtlUserThreadStart
    kd> g
    
    Breakpoint 0 hit
    ntdll!RtlUserThreadStart:
    0033:00000000`7763c500 4883ec48        sub     rsp,48h
    
    Single step from here or find Original Entry Point to get to process start
    For comparison, here's what the stack looks like with a process where sxe ld:[] DOES break

    Code:
    kb
     # RetAddr           : Args to Child                                                           : Call Site
    00 fffff800`02961eda : fffff880`03884d20 fffff880`03884a20 fffffa80`02092aa0 fffff800`0292b8b1 : nt!DebugService2+0x5
    01 fffff800`02c8e34b : fffffa80`02cfce20 fffffa80`02092aa0 fffffa80`02cfce20 00000000`000007ff : nt!DbgLoadUserImageSymbols+0x2a
    02 fffff800`02c1b668 : 00000000`0001b000 fffffa80`02a77278 fffffa80`02a77060 fffffa80`02e743f0 : nt!MiLoadUserSymbols+0x21b
    03 fffff800`02bf6997 : fffffa80`02e743f0 fffffa80`02a77060 fffff880`03884d00 fffff880`03884d30 : nt! ?? ::NNGAKEGL::`string'+0x2ae17
    04 fffff800`02b85ff4 : 00000000`00000004 fffffa80`02a77060 fffff880`03884d00 00000000`00000000 : nt!MiMapViewOfSection+0x367
    05 fffff800`02b87aa3 : fffffa80`01bdf4b0 fffffa80`01bdf4b0 fffff8a0`01bd3060 00000000`000000a0 : nt!MmInitializeProcessAddressSpace+0x40c
    06 fffff800`02b88944 : 00000000`00000000 00000000`003d0100 fffff880`038856f0 fffffa80`021307fe : nt!PspAllocateProcess+0x6b3
    07 fffff800`028dc8d3 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!NtCreateUserProcess+0x4a3
    08 00000000`77331dea : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
    09 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwCreateUserProcess+0xa

Similar Threads

  1. Ida pro 6.1 remote debugging Windbg
    By dbenchtk in forum Advanced Reversing and Programming
    Replies: 0
    Last Post: October 20th, 2013, 18:35
  2. FSG unpacker with command line support?
    By going in forum Malware Analysis and Unpacking Forum
    Replies: 2
    Last Post: January 7th, 2010, 23:00
  3. Ollydbg => command line arguments Ax200
    By totofino in forum OllyDbg Support Forums
    Replies: 8
    Last Post: November 21st, 2006, 20:04
  4. ollydbg command line parameter??
    By pillii in forum OllyDbg Support Forums
    Replies: 9
    Last Post: October 7th, 2005, 04:32
  5. Opening an installer with command line
    By andresiraola in forum OllyDbg Support Forums
    Replies: 2
    Last Post: February 9th, 2005, 08:07

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •