Welcome to the new Woodmann RCE Messageboards Regroupment
Please be patient while the rest of the site is restored.

To all Members of the old RCE Forums:
In order to log in, it will be necessary to reset your forum login password ("I forgot my password") using the original email address you registered with. You will be sent an email with a link to reset your password for that member account.

The old vBulletin forum was converted to phpBB format, requiring the passwords to be reset. If this is a problem for some because of a forgotten email address, please feel free to re-register with a new username. We are happy to welcome old and new members back to the forums! Thanks.

All new accounts are manually activated before you can post. Any questions can be PM'ed to Kayaker.

Windbg remote break command line process ($MFT related)

All-in-one reversing related discussions
Post Reply
User avatar
Kayaker
Posts: 4169
Joined: Thu Oct 26, 2000 11:00 am

Windbg remote break command line process ($MFT related)

Post by Kayaker »

I'd like to debug remotely a command line app and am not quite sure how to get it done. Specifically I'm trying to trace into API's used by Strings64. I'm using Windbg and VirtualKD to connect to Win7x64 running in VMWare. The usual steps I use that seems to work for most usermode apps is like this

Code: Select all

!gflag +ksl
sxe ld app.exe
g

r $proc
.process

bp /p @$proc nt!NtMapViewOfSection
g

bp0 /p @$proc ntdll!RtlUserThreadStart
g
At this point I usually used a little script to parse the PE header and find the OEP of the app to break at program start. Blabberer mentioned the easier trick that the OEP can be obtained from the x64 register rcx at the break at RtlUserThreadStart.

Either way I can normally break at the start of a usermode process remotely without too much problem, including cmd.exe. However I'm not sure how to extend that to a command line process such as strings, findstr, dir, etc. In Win7 a command window by itself runs as the process cmd.exe. In Win10 there is also a child process conhost.exe involved.

******************************************

As to why I'm interested in tracing Strings64, in another thread we discussed how if you pipe the output to a file it will pick up the logfile as a search file and can lead to a recursive unending search if any results have been written to the logfile. The solution I found was that the logfile MUST be named alphabetically so it is the very first file found in the search directory, that way it will be empty when Strings opens it for searching, then closed for good and not lead to recursive results.

Using Procmon I determined that Strings uses FindFirstFile / FindNextFile, a standard routine for listing files in a directory. This led to wondering what character is "first" in a sorted list of filenames. It turns out that the alphabetical sort order you see in Explorer is different from the sort order yielded by FindFirstFile. The definition for the API states that the order in which the function returns the file names is dependent on the file system type, NTFS vs FAT for example.

This is further explained in a blog post by Raymond Chen where it is revealed that NTFS captures the case mapping table at the time the drive is formatted

Why do NTFS and Explorer disagree on filename sorting?
https://devblogs.microsoft.com/oldnewth ... 0/?p=35293

I suspected that this case mapping table might be what is defined in the NTFS Master File Table ($MFT) as the $UpCase metadata. Using 7-Zip, of all things, I was able to dump the $UpCase metadata and sure enough it seems to match the exact alphabetical order of filenames returned by FindFirstFile / FindNextFile.

Here is the method of accessing $MFT using 7-Zip
https://thestarman.pcministry.com/asm/mbr/IntNTFSfs.htm

Attached is a screenshot of $UpCase opened in HxD. The first usable alphabetical character for filenames is "!", the last (after regular letters) is "~". This matches all the tests I did using Procmon to monitor Strings Query Directory results of filenames.

Finally, Procmon points to FindFirstFile eventually leading to FLTMGR.SYS being involved where I was wondering if it was possible to actually see the $MFT being accessed through code, that being the bottom line of all this.

I could of course just make a gui app that uses FindFirstFile / FindNextFile and trace into it that way, but I'm still wondering how to remotely debug a command line process with my usual Windbg setup.

Kayaker
Attachments
$UpCase.jpg
WaxfordSqueers
Senior Member
Posts: 1000
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

Kayaker wrote:I'd like to debug remotely a command line app and am not quite sure how to get it done
Don't fully understand what you're doing but how different is it from the example Blabbs gave in my DISM thread? I realize I am using a debugger locally but I am debugging a command line app. Why could you not attach to a command line app remotely and follow the commands given by Blabbs? to find the OEP, winmain, etc.?

I get it that you are trying to see how the app loads re alphanumeric protocol. I managed to trace through k-code to ntfs.sys once using softice. Not sure I can do the same using windbg although ntfs.sys may not be k-code.

Also, there's a good app for viewing [email protected] Disk Editor. It makes it easy to find your way around $MFT.
User avatar
Kayaker
Posts: 4169
Joined: Thu Oct 26, 2000 11:00 am

Post by Kayaker »

That's one thing I wanted to try, if I can attach to the command line window remotely and see if I could begin debugging any process it starts (Strings64.exe for example). An sxe type exception break on the secondary process directly doesn't seem to catch, or I'm not setting it up right, which is where the problem lies.

I'm not familiar with all the various remote debugging options such as given here, some might be more suitable for working with a command line process.

Remote Debugging Review
http://www.nynaeve.net/?p=38

However I'm trying to make it work with my existing setup where Windbg is on my host desktop and the target is in a VM. I can catch and debug any gui app I start in the VM, including cmd.exe itself when it starts up, but not once the command window has opened up and you later enter something on the command line. Unfortunately I don't think you can start the target program remotely from Windbg on the host with my configuration.
WaxfordSqueers
Senior Member
Posts: 1000
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

Kayaker wrote:Unfortunately I don't think you can start the target program remotely from Windbg on the host with my configuration.
I made notes at the time I was doing remote debugging but can't find them at the moment. I'll keep looking and get back to you if I find them. Of course, I was using a laptop as host and running the target on a desktop.

I recall there are other ways to do remote debugging but I'm fuzzy on that right now.
WaxfordSqueers
Senior Member
Posts: 1000
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

Kayaker wrote:I'd like to debug remotely ...
Don't know if this will help. It's a question I asked in:

http://www.woodmann.com/forum/showthread.php?15764-USB-drivers-for-Win-7-on-8th-generation-Intel-chipset/page7

"Is there no way to attach to an application on the target, or start it remotely, with full kernel debugging power, from a host computer?

If I run !process 0 0, I can see the target app but it seems I cannot attach to it remotely".

****

blabberer replied:

host
sxe ld foo.exe
g

target double click foo.exe

windbg will break on loading the exe

reload symbols

set process specific breaks as reqd bp /p {eproc } [module!symbol]

btw keep in mind sxe ld will work only once per boot for one specific binary

*************more**************

Another thought that I'd forgotten about:

Under the key: HKLM\Software\Microsoft\WindowsNT\CurrentVersion\Image File Execution Options

You can create a new key and give it the name of your application.

Under the values for the key, you name it Debugger and give it a value of the debugger you want to use. When the app is opened it comes up in the debugger.

I used this trick to get Windows to open a command prompt at the logon screen. There is an icon there for Accessibility and it leads to a file in the Windows\System32 directory called utilman.exe. I did it because I wanted to use a command window app for loading drivers at that stage before Windows could intercede. Did not help but it was nice to know I could do that.

The trick is to make a copy of cmd.exe, rename utilman.exe to utilman.bak, then rename the copy of cmd.exe to utilman exe. Second part of trick is to go to the HKLM key above and enter Utilman.exe as a new key name. For value, enter Debugger, and for it's value enter cmd.exe.

Voila! Next time you boot, hit the Accessibility key and a command window opens.

Something to ponder. If you used utilman.exe as the key name, as described, and instead of cmd.exe under Debugger, you entered a debugger name, would the accessibility icon at the logon screen open a debugger at the logon screen? And would it be of any use? Apparently ntsd can be used in this manner.

I did not know this, but apparently hitting the Windows key + U before the logon screen will start utilman.exe hence any app renamed in its place. If you wanted to play a game of solitaire as you logon, you might try that. :devil:
User avatar
Kayaker
Posts: 4169
Joined: Thu Oct 26, 2000 11:00 am

Post by Kayaker »

Well this is interesting, and unusual. My concern was that I couldn't break on ANY command line process using the above method, but that's not the case. I made my own basic console app and had no problem. The same with Windows findstr, I could break and debug with no issue.

However with Sysinternals Strings or Strings64 the method will repeatedly not work. I haven't checked with other random console programs or with other Sysinternal apps, but it makes me wonder what the reason is.

Could there be some anti-debugging code or other unusual reason that 'sxe ld:' does not allow the debugger exception to take hold with Strings? I haven't tried debugging Strings as a local process yet, only under the remote conditions.

More digging might be in order.
WaxfordSqueers
Senior Member
Posts: 1000
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

Kayaker wrote:Could there be some anti-debugging code or other unusual reason that 'sxe ld:' does not allow the debugger exception to take hold with Strings?
Just a thought. Can you remotely break on cmd.exe using sxe ld? If so, I wonder if you could then set a BP in cmd.exe then run strings in the command window to see what is going on?
blabberer
Senior Member
Posts: 1535
Joined: Wed Dec 08, 2004 11:12 am

Post by blabberer »

oops was it a long break :)

From vista+ Neither NtCreateProcess nor NtCreateProcessEx are used

kernel32 transfers control to ntdll!NtUserCreateProcess

which has its name matching counterpart in ntkrxxx

i think nt!DbgLoadModuleSymbols isnt also called which is why sxe ld probably doesnt work

for strings (i assume strings is probably compiled with minimum osversion )

any way lets set a break and go run strings * in target

Code: Select all

kd> bp nt!NtCreateUserProcess
kd> g
Breakpoint 0 hit

eax=0000005d ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
eip=82a55a76 esp=80fe8d04 ebp=80fe8d34 iopl=0         nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000246
nt!NtCreateUserProcess:
82a55a76 68b8060000      push    6B8h
The stack wont be correct until SEH_prolog4_GS() is executed
(eip will be one off when entered through sysenter )
so lets step over until then (3 steps in win 7 x86)

Code: Select all

kd> p
eax=0000005d ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
eip=82a55a7b esp=80fe8d00 ebp=80fe8d34 iopl=0         nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000246
nt!NtCreateUserProcess+0x5:
82a55a7b 6898438882      push    offset nt! ?? ::FNODOBFM::`string'+0x82d8 (82884398)
kd> 
eax=0000005d ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
eip=82a55a80 esp=80fe8cfc ebp=80fe8d34 iopl=0         nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000246
nt!NtCreateUserProcess+0xa:
82a55a80 e883cfe0ff      call    nt!_SEH_prolog4_GS (82862a08)
kd> 
eax=80fe8cf0 ebx=82a55a76 ecx=84363d48 edx=82a55a76 esi=0029e9e8 edi=0029ecf0
eip=82a55a85 esp=80fe8628 ebp=80fe8d00 iopl=0         nv up ei pl nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00000206
nt!NtCreateUserProcess+0xf:
82a55a85 8b4508          mov     eax,dword ptr [ebp+8] ss:0010:80fe8d08=0029ecf0 <<<
stack will be correct here but symbols aren't loaded so we need to use .reload /f

Code: Select all


kd> kb
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
05 0029f048 4ab43f94 000a0d18 000b0e88 00000000 kernel32!CreateProcessW+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
06 0029f20c 4ab43cb5 000b56a8 00000000 00000000 0x4ab43f94
07 0029f46c 4ab43d48 000b56a8 00000000 00000000 0x4ab43cb5
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
0f 0029fa1c 00000000 4ab4829a 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0x1b

kd> .reload /f

kd> kb
 # ChildEBP RetAddr  Args to Child              
00 80fe8d00 8286687a 0029ecf0 0029eccc 02000000 nt!NtCreateUserProcess+0xf
01 80fe8d00 773a70b4 0029ecf0 0029eccc 02000000 nt!KiFastCallEntry+0x12a
02 0029e9b0 773a5784 76dfe5d5 0029ecf0 0029eccc ntdll!KiFastSystemCallRet
03 0029e9b4 76dfe5d5 0029ecf0 0029eccc 02000000 ntdll!NtCreateUserProcess+0xc
04 0029f010 76db2079 00000000 000a0d18 000b0e88 kernel32!CreateProcessInternalW+0xe75
05 0029f048 4ab43f94 000a0d18 000b0e88 00000000 kernel32!CreateProcessW+0x2c
06 0029f20c 4ab43cb5 000b56a8 00000000 00000000 cmd!ExecPgm+0x20c   <<<<< see  we have correct symbols
07 0029f46c 4ab43d48 000b56a8 00000000 00000000 cmd!ECWork+0x7f
08 0029f484 4ab415c5 000b56a8 55cc3b23 00000001 cmd!ExtCom+0x47
09 0029f8e0 4ab422c0 000b56a8 00000002 76e01e2e cmd!FindFixAndRun+0x1f7
0a 0029f930 4ab576f0 00000000 000b56a8 4ab64204 cmd!Dispatch+0x14b
0b 0029f974 4ab4835e 00000005 003d1098 003d1510 cmd!main+0x21a
0c 0029f9b8 76e03c45 7ffdf000 0029fa04 773c37f5 cmd!_initterm_e+0x163
0d 0029f9c4 773c37f5 7ffdf000 77637989 00000000 kernel32!BaseThreadInitThunk+0xe
0e 0029fa04 773c37c8 4ab4829a 7ffdf000 00000000 ntdll!__RtlUserThreadStart+0x70
0f 0029fa1c 00000000 4ab4829a 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0x1b

lets run until cmd!ExecPgm()

Code: Select all


kd> g 4ab43f94

eax=00000001 ebx=00000000 ecx=76dfebf7 edx=00010180 esi=4ab65260 edi=00000001
eip=4ab43f94 esp=0029f078 ebp=0029f20c iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
cmd!ExecPgm+0x20c:
001b:4ab43f94 89459c          mov     dword ptr [ebp-64h],eax ss:0023:0029f1a8=0029f45c

we have an EXECUTIVE PROCESS OBJECT but entry point has not been called yet

which will be called when cmd executes cmd!Wait() with this process objects user mode Handle

Code: Select all

kd> !process 0 0 strings.exe
PROCESS 844c3420  SessionId: 1  Cid: 0220    Peb: 7ffdf000  ParentCid: 0740
    DirBase: 0d14b000  ObjectTable: 968a9598  HandleCount:   0.
    Image: strings.exe
    

kd> dt nt!_EPROCESS -y sec 844c3420
   +0x128 SectionObject : 0x96ad8918 Void
   +0x12c SectionBaseAddress : 0x013a0000 Void  <<<<< this is the Mapped Address of strings.exe
   +0x190 SecurityPort : (null)

 

lets set the process context check the ms dos header use !dh to find the AddressOfEntryPoint

and set a process specific breakpoint on this Address clear all other breakpoints and continue

Code: Select all


kd> .process /p /r 844c3420
Implicit process is now 844c3420
.cache forcedecodeuser done
Loading User Symbols
PEB is paged out (Peb.Ldr = 7ffdf00c).  Type ".hh dbgerr001" for details

kd> db 13a0000 l20
013a0000  4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00  MZ..............
013a0010  b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00  [email protected]

kd> bp /p @$proc 13a41C6
kd> bl
     0 e Disable Clear  82a55a76     0001 (0001) nt!NtCreateUserProcess
     1 e Disable Clear  013a41c6     0001 (0001) 
     Match process data 844c3420

kd> bd 0

kd> g
we hit the entrypoint

Code: Select all

Breakpoint 1 hit
eax=76e03c33 ebx=7ffdf000 ecx=00000000 edx=013a41c6 esi=00000000 edi=00000000
eip=013a41c6 esp=0013fe1c ebp=0013fe24 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
001b:013a41c6 e8524a0000      call    013a8c1d
reload symbols and check

Code: Select all

kd> .reload /f

kd> r
eax=76e03c33 ebx=7ffdf000 ecx=00000000 edx=013a41c6 esi=00000000 edi=00000000
eip=013a41c6 esp=0013fe1c ebp=0013fe24 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
strings+0x41c6:
001b:013a41c6 e8524a0000      call    strings+0x8c1d (013a8c1d)

kd> kb
 # ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0013fe24 773c37f5 7ffdf000 760b4914 00000000 strings+0x41c6
01 0013fe64 773c37c8 013a41c6 7ffdf000 00000000 ntdll!__RtlUserThreadStart+0x70
02 0013fe7c 00000000 013a41c6 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0x1b

see the commandline used to start the process

Code: Select all


ansi_string->Buffer

kd> dpa KERNELBASE!BaseAnsiCommandLine+4 l1
7557478c  00231de0 "strings.exe   *"

or ansi_string display

kd> ds KERNELBASE!BaseAnsiCommandLine
00231de0  "strings.exe   *"
WaxfordSqueers
Senior Member
Posts: 1000
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

Brilliant stuff, Blabbs. :p Worth waiting for.
User avatar
Kayaker
Posts: 4169
Joined: Thu Oct 26, 2000 11:00 am

Post by Kayaker »

Hey B, thanks for stopping execution and handling the break :p

Nice, a break on nt!NtCreateUserProcess works really well as a replacement for sxe ld:[] if a process won't break on the latter exception. The rest of the procedure is pretty much the same.

This time I used Sysinternals hex2dec64, which also showed the problem of not breaking for sxe ld:[]. What is it with these apps?

Here is what the stack looks like in Win7x64 at the first break:

Code: Select all

kd> !gflag +ksl
kd> bp nt!NtCreateUserProcess
kd> g

Breakpoint 0 hit
nt!NtCreateUserProcess:
fffff800`02b784a0 fff3            push    rbx

kd> kb
 # RetAddr           : Args to Child                                                           : Call Site
00 fffff800`028cc8d3 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!NtCreateUserProcess
01 00000000`77661dea : 00000000`7750efcd 00000000`00431620 00000000`00000000 00000000`00080000 : nt!KiSystemServiceCopyEnd+0x13
02 00000000`7750efcd : 00000000`00431620 00000000`00000000 00000000`00080000 00000000`0029e828 : 0x77661dea
03 00000000`00431620 : 00000000`00000000 00000000`00080000 00000000`0029e828 00000000`00000000 : 0x7750efcd
04 00000000`00000000 : 00000000`00080000 00000000`0029e828 00000000`00000000 00000000`00000000 : 0x431620
All I did from there is execute 'gu' to go back one step to KiSystemServiceCopyEnd

Code: Select all

kd> gu

nt!KiSystemServiceCopyEnd+0x13:
fffff800`028cc8d3 65ff042538220000 inc     dword ptr gs:[2238h]

kd> kb
 # RetAddr           : Args to Child                                                           : Call Site
00 00000000`77661dea : 00000000`7750efcd 00000000`00431620 00000000`00000000 00000000`00080000 : nt!KiSystemServiceCopyEnd+0x13
01 00000000`7750efcd : 00000000`00431620 00000000`00000000 00000000`00080000 00000000`0029e828 : 0x77661dea
02 00000000`00431620 : 00000000`00000000 00000000`00080000 00000000`0029e828 00000000`00000000 : 0x7750efcd
03 00000000`00000000 : 00000000`00080000 00000000`0029e828 00000000`00000000 00000000`00000000 : 0x431620
At this point the process was loaded and I could continue with the previous steps to get to RtlUserThreadStart

Code: Select all

kd> !process 0 0

PROCESS fffffa8002a8a060
    Image: hex2dec64.exe

kd> .process fffffa8002a8a060
Implicit process is now fffffa80`02a8a060

kd> bp0 /p @$proc nt!NtMapViewOfSection
kd> g

Breakpoint 0 hit
nt!NtMapViewOfSection:
fffff800`02be69e0 48895c2410      mov     qword ptr [rsp+10h],rbx

kd> bp0 /p @$proc ntdll!RtlUserThreadStart
kd> g

Breakpoint 0 hit
ntdll!RtlUserThreadStart:
0033:00000000`7763c500 4883ec48        sub     rsp,48h

Single step from here or find Original Entry Point to get to process start
For comparison, here's what the stack looks like with a process where sxe ld:[] DOES break

Code: Select all

kb
 # RetAddr           : Args to Child                                                           : Call Site
00 fffff800`02961eda : fffff880`03884d20 fffff880`03884a20 fffffa80`02092aa0 fffff800`0292b8b1 : nt!DebugService2+0x5
01 fffff800`02c8e34b : fffffa80`02cfce20 fffffa80`02092aa0 fffffa80`02cfce20 00000000`000007ff : nt!DbgLoadUserImageSymbols+0x2a
02 fffff800`02c1b668 : 00000000`0001b000 fffffa80`02a77278 fffffa80`02a77060 fffffa80`02e743f0 : nt!MiLoadUserSymbols+0x21b
03 fffff800`02bf6997 : fffffa80`02e743f0 fffffa80`02a77060 fffff880`03884d00 fffff880`03884d30 : nt! ?? ::NNGAKEGL::`string'+0x2ae17
04 fffff800`02b85ff4 : 00000000`00000004 fffffa80`02a77060 fffff880`03884d00 00000000`00000000 : nt!MiMapViewOfSection+0x367
05 fffff800`02b87aa3 : fffffa80`01bdf4b0 fffffa80`01bdf4b0 fffff8a0`01bd3060 00000000`000000a0 : nt!MmInitializeProcessAddressSpace+0x40c
06 fffff800`02b88944 : 00000000`00000000 00000000`003d0100 fffff880`038856f0 fffffa80`021307fe : nt!PspAllocateProcess+0x6b3
07 fffff800`028dc8d3 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!NtCreateUserProcess+0x4a3
08 00000000`77331dea : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
09 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!ZwCreateUserProcess+0xa
Post Reply