Results 1 to 13 of 13

Thread: Remote kernel mode debugging in non-VM environment

  1. #1

    Remote kernel mode debugging in non-VM environment

    I thought remote kernel debugging warranted it's own thread in a non-VM environment. At the moment, I am trying to get a remote k-mode session established with XP but I am having issues...again....establishing the session with W7 and W10.

    Came across this old thread with kayaker, blabberer, naides, evaluator, deroko, Delta, etc., in which blabbs got a remote session going with XP. I'm wondering if he recalls the method. Not sure if a VM was available in 2008. In message #12 on November 15, 2008 @ 9:22, blabbs posted a log of his XP session:

    http://www.woodmann.com/forum/showthread.php?12193-Driver-PE-Header-ImageBase-modified-by-OS-Loader&highlight=ImageBase

    Found the entire thread interesting although I'm not clear on the overall meaning. The thread covers things I want to cover, however.

  2. #2
    @blabberer ...I was going through some older threads to see if I could get some hints about what I did last time to get serial debugging going. Some of it was useful.

    You mentioned in one thread that you need to turn on boot debugging to debug the boot stage. I had never heard of that but noticed it the other day as an option in W10. Can you elaborate on that? Is it available in W7? Were you talking about local k-mode debugging via windbg or remote debugging in k-mode?

    I am also getting a bit turned around with the reference to hypervisor serial debugging in W10. Don't know whether hypervisor should be turned on in W10. When it's on, it grabs COM1 but I don't know if it is interfering with my current inability to get a serial connection going.

    ps. it's OK to be a pr**k. I know you hate hardware debugging, but, hey...I forgive you.

  3. #3
    Just noticed this disclaimer offered by microsoft:

    "The 1394 transport is available for use in Windows 10, version 1607 and earlier. It is not available in later versions of Windows. You should transition your projects to other transports, such as KDNET using Ethernet".

    They have probably discontinued old-style serial debugging as well since 1394 is a form of serial comm. That's the only reasonable explanation as to why serial debugging suddenly stopped working. They do offer serial debugging via hypervisor but W7 in particular is probably not capable of using a hypervisor. I'm sure XP is not capable...unless it's modded.

    I'm wondering if hypervisor serial debugging is aimed at VMs only.

    Seems I may have to break my dual boot with W7/W10 and orphan W10 since I think it's a piece of crap anyway. I much prefer working in the W7 environment. I was in the process of doing so by rewriting the BCD store in both W7 and W10 separately.

    W10 uses a separate system partition to hold the BCD store but also the EFI store if you're into EFI boot. The difference is that BCD uses the old MBR booting method whereas EFI boots from the BIOS. As a result, W10 in EFI mode is not backward compatible with boot methods using the MBR boot. I just reset W10 to use both BCD and EFI and I have blocked the EFI in the past by changing the name to $FI.

    If you have W10 and W7 dual-booted, W10 takes over and stores the W7 BCD store in the system partition. I am reasoning therefore that if my version of W10, which is 1909, I think, it likely no longer supports serial debugging of the old style, having converted to hypervisor serial debugging.

    Anyway, W10 is turning into a pain in the butt and I am looking into dual booting W7 with XP, a more natural fit. First, I have to rewrite the W7 BCD store which is normally located on the %windir% partition. XP doesn't care, it stores debug info in boot.ini, and it is read by ntoskrnl to determine the type of debugging. If the /debugport entry says COMx, it knows it is serial.

  4. #4
    Super Moderator
    Join Date
    Dec 2004
    Posts
    1,529
    Blog Entries
    15
    1) Serial Debugging is Available right from win3.1 to w10 infinity
    2) hypervisor in w10 is an inbuilt vm (avl only in pro not home you need to enable windows features to use hype)
    3) 1394 has been discontinued from some xy version of w10
    4) if you have a compatible nic ethernet works better than serial in w10
    5) find and run the kdnet.exe in whichever target you want to debug save the key and start a session
    5) get your self comfortable in a vm before cutting cables
    6) VMware player (the free for non commenrcial use ) works nice
    7) virtual box also works ok
    8) to learn a new thing you need to unlearn first
    9) windbg is not sice and wont ever be sice
    a) sice was probably best in its time the clock has turned decades now
    Last edited by blabberer; April 15th, 2020 at 04:13.

  5. #5
    Quote Originally Posted by blabberer View Post
    4) if you have a compatible nic ethernet works better than serial in w10
    Thanks for reply, Blabbs. I am currently off on a tangent trying to reverse a Visual Basic app. Any ideas? I am checking this site to see if there's any tools. Seems to me there was one along the lines of Boundschecker. Kayaker might recall the name. In fact, I think I still have it.

    I did have a net connection going before I had my serial connection going but suddenly my serial connection worked and I abandoned the other. Can't think of the reason right now.

    Quote Originally Posted by blabberer View Post
    5) get your self comfortable in a vm before cutting cables
    I have no problem using a VM and I do use them for debugging. My current stubbornness is based on the fact that I have a newer mobo and I'd like to get into its hardware features. Someone suggested loading an app in a VM and debugging the entire VM with the app loaded. Don't know anything about that but it might be feasible via W10 and a hypervisor connection.

    BTW...I am using VMWare player to run various OSes for debugging. It does work nice. I also need to take a look at kayaker's advice to try visualKD. Too many things to do in life, not enough time.

    Quote Originally Posted by blabberer View Post
    8) to learn a new thing you need to unlearn first
    In Zen they call that emptying one's cup. I am familiar with the technique. It's actually good in life, to empty one's mind of the past to leave room for insight in the present. The easiest way to empty one's mind is to say, "I don't know".

    Quote Originally Posted by blabberer View Post
    a) sice was probably best in its time the clock has turned decades now
    Oooooooh....that's a bit blaspemous.

  6. #6
    Wracking my brain trying to find a scientific way to detect what is wrong with my k-mode connection between my W7 host and my XP target. It's the same with W7 and W10 even though I did have a good debug session going at one time.

    I have tested the serial com port both ways. I can detect control/data signals coming from the host and they are correct. In hex, I get 69 69 69 69 06 00 00 00 00 00. I can send replies using a serial com app and the host acknowledges the data. Hex data transmits correctly both way.

    Seems I have a problem with kdcom.dll, dbgeng.dll or a related process. The only good way I can see is to trace the code to see where it goes and where it is getting hung up.

    I propose using the comm app I have because I can enter data in it and send it to the host. That might help me trace the path but what I really need is to find the process in the target the host is trying to reach. I need to see where the data listed above enters the host and how it is processed.

    I am stumped as to how I can do that using windbg. I can't attach to ntoskrnl or kdcom.dll and I can't think of a running process to which I can attach. I can't run an executable that I know of that is related to serial com except serial.sys and serenum.sys.

    Some info on the two:

    https://docs.microsoft.com/en-ca/windows-hardware/drivers/serports/features-of-serial-and-serenum

  7. #7
    Teach, Not Flame Kayaker's Avatar
    Join Date
    Oct 2000
    Posts
    4,157
    Blog Entries
    5
    Hey Wax. You seem to have had this problem for a long time. I've never set up anything like that, debugging over 2 computers, but the docs seem to make it straightforward enough, heh, heh.

    Presumably you've tried 'everything', but I guess the "scientific method" would be to go through every possible combination that you can regarding TCP, NPIPE, SPIPE, SSL, and COM protocols, cabling (com/usb/firewire/network), debugger (kd, cdb, ntsd, Windbg, gdb, Visual Studio), baud rates, other settings, blah blah.

    I know you know all that, I'm just trying to get a handle on the complexities of it myself. I think at one point you had things set up but were waiting on the dreaded "waiting for remote connection" message.

    Can you use hyperterminal, putty or something to intercept the communication packets, which I guess is what you're trying to do?

  8. #8
    Quote Originally Posted by Kayaker View Post
    I know you know all that, I'm just trying to get a handle on the complexities of it myself. I think at one point you had things set up but were waiting on the dreaded "waiting for remote connection" message.

    Can you use hyperterminal, putty or something to intercept the communication packets, which I guess is what you're trying to do?
    Thanks Kayaker. I have tried all the obvious things, now I need to do some reversing and find out where things are getting hung up. I have sent a binary file between target and host (using z-modem ...remember BBSes?) and back on XP using hyperterminal. There is nothing I have seen so far on the Net that addresses this problem.

    BTW...I can't use a Net debug session since I don't yet have a Net connection on XP. Seeing as it's not working on W7 or W10, I could use a Net connection on either to check things out. However, I need a serial connection for XP right now unless I buy an add-on LAN card, if they have drivers for XP that will work with a 300-series chipset. I was hoping a k-mode debug session between host and target would help me figure a lot of those issues out.

    I don't have a clue how the kernel mode debugging system works. Microsoft may have a low-level description of it somewhere, maybe in the DDK, but I don't have a clue where to look. I have read briefly that dbgeng.dll handles the mechanics on the target end but kdcom.dll, for a serial connection, is the extension that works with ntoskrnl, hal, etc., to enable the data stream from the host.

    I have already verified with hyperterminal and a serial comm app that hex data can be sent both ways. The host has acknowledged the connection to COM2 on it's end and it is waiting for acknowledgement from the target. I have sent it an ACK (0x06) from the target but I don't know the complete string required for a response. The host responds to the ACK but claims it can't read the data. No kidding!!

    Apparently, it's getting nothing from dbgeng.dll and there could be several reasons for that. I have considered tracing the data stream from hyperterminal or the serial app but that would reveal nothing about the debug environment. I really need to attach to something that is involved with the debug process on the target end.

    That's where I'm stumped. I don't know enough yet about windbg to set up a session so I can look at modules like kdcom.dll, or whatever is involved with the serial chain to the comm port. Apparently I can issue IN and OUT instructions with windbg but how can I set up a session to use either?

    It just occurred to me that a local kd or windbg session on the target might help me in that respect. I would ideally like to send the target some data from windbg on the host, or the hex sequence that the host sends, which is ASCII iiii or hex 69 69 69 69 06 00 00 00 00 00, etc. If I could set up a BP on the IN port at COM2 on the target, I might be able to trace it through the target to see where it's getting hung up. I could even trace the ctrl-break = 0x62. Oddly enough, 0x62 = b, seemingly for break, but I don't know how the ctrl fits in.

    We're getting into an area where not many people seem to know what's going on with a debug session at this level. Someone told me that the designers of the debug system tested it using printf, but they could not elaborate.

  9. #9
    Teach, Not Flame Kayaker's Avatar
    Join Date
    Oct 2000
    Posts
    4,157
    Blog Entries
    5
    I remember doing some parallel port programming with QBasic, but I don't think that would help here

    OK, crazy reversing idea, would running an API monitor (i.e. Rohitab) and/or Procmon on Windbg, on either end, be of any help? It might stop logging where Windbg hangs, waiting for continuation.

  10. #10
    Quote Originally Posted by Kayaker View Post
    I remember doing some parallel port programming with QBasic, but I don't think that would help here
    Hey...back in the bad old days, I used to program Basic straight into a computer to set up the heads on a disk drive. I actually wrote little programs on screen that were interpreted immediately without compilation, Dims and everything, to make the heads jump out to a pattern on a repair disk at a certain track. The pattern was read on an oscilloscope and as you adjusted the heads they would move back and forth over the pattern so you could align them. We would also write loops so we could test the heads as they jumped back and forth.

    Today, that's not possible due to the extremely high track density. Alignment is done in the factory through a hole in the base of the drive and that's it. I'm sure a Russian has figured a way to do it since they seem to be experts in that field.

    That kind of dates me, eh?

    Quote Originally Posted by Kayaker View Post
    OK, crazy reversing idea, would running an API monitor (i.e. Rohitab) and/or Procmon on Windbg, on either end, be of any help? It might stop logging where Windbg hangs, waiting for continuation.
    Not crazy, good idea. Never tried procmon on a serial device before, might be interesting to see what it reveals. I'll look into the API monitor as well. Thanks.

    ps. I should have mentioned that I found issues with my Debugging Tools for Windows setup. I have stripped everything related to DT4W off my computer and re-installed. Tried re-installing from SDK but it complains that it cannot find Net Framework of any version. I have received this complaint from some other apps but all versions from 1.1. to 4+ are on my XP installation. I am working on that issue right now.

    Also, I was playing with windbg with the -kl command line. Tried !lmi nt and it recognized ntkrnlmp (not using ntoskrnl apparently). Then I tried !lmi kdcom.dll and it found that OK. However, when I tried !lmi dbgeng.dll, it could not find it even though there is a copy in %windir%\system32 and in the backup directory.

    I was in debug mode when I ran windbg.

    There was an oddity as well. I found some sym commands like symenumtypes and windbg kept removing the 's' from symenumtypes and claiming it did not understand ymenumtypes. Why would it remove the 's'? One of the sym command did work but with the rest, windbg removed the leading 's'. I don't have DT4W loaded right now so I can't give you the exact command I used.
    Last edited by WaxfordSqueers; May 22nd, 2020 at 17:13.

  11. #11
    Super Moderator
    Join Date
    Dec 2004
    Posts
    1,529
    Blog Entries
    15
    i don't recall any command named synenumtype so obviously it errs because s is command for Searchmemory

    and it cant resolve ymen or symen as a module

    Code:
    0:000> symen
    Couldn't resolve error at 'ymen'
    0:000> s 00007ff7`e9d60000 l100 'M'
    00007ff7`e9d60000  4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00  MZ..............
    0:000>

    below is a cinematographic writeup so keep a pinch of salt handy

    as to kdcom,kdnet,kdusb2,kdusb3,kd1394 all of them are polled
    that is client keeps on running a ReadPacket at a certain DeadDropBox
    it retrieves what is dropped in the DeadDropBox
    acts upon it and Writes A packet to DeadDropBox

    this happens on both side

    so both simply use some predetermined signature to inform each other

    what is dropped and what is supposed to be Redropped

    like one packet from one end may say I am sending 600 bytes
    post back when you picked all the bytes and

    the other end send an ack yeah boss I am waiting

    so as long as the retriever doesn't signal back saying I got all the bytes
    the sender will keep on dumping 600 bytes periodically in the drop box
    it will stop dumping only when it receives an ack or a timeout occurred

    if a timeout occurred it will wait for reconnection



    if you read thrillers and spy novels you will be having a sense of DeadDropBoxes both side doesn't know who drops and who picks
    both side spy upon some signals reach the box untailed / untagged / retrieves / drops and still be unknown to each other
    and all this hits the wire as simple

    IN OUT of the olden days at 0x2f8


    Code:
    C:\>cd "Program Files (x86)\Windows Kits\10\Debuggers\x64\
    
    C:\Program Files (x86)\Windows Kits\10\Debuggers\x64>dbh dbgeng.dll
    
    dbgeng [1000000]: x *comport*
    
     index            address     name
         1            12f297c :   SetComPortName
         2            12f2bac :   OpenComPort
         3            12f2758 :   ComPortWrite
         4            12f2474 :   ComPortRead
         5            12f2a8c :   SetComPortBaud
    
    dbgeng [1000000]:


    a stack where it reads a byte in uart (beware this bp can bsod coz you are choking the transport)

    Code:
    1: kd> bp kdcom!KdReceivePacket
    1: kd> g
    SetContext failed, 0xD0000001  <<<<<<<<<<<<<<<<<  unstable breakpoint
    MachineInfo::SetContext failed - Thread: 0000016A46111F10  Handle: 2  Id: 2 - Error == 0xD0000001 <<<<<<<<<<<<<<<< unstable breakpoint
    Break instruction exception - code 80000003 (first chance)
    kdcom!READ_PORT_UCHAR+0x4:
    fffff807`63a01094 c3              ret
    1: kd> kb
     # RetAddr           : Args to Child                                                           : Call Site
    00 fffff807`63a033a1 : ffffb680`b1631ba0 fffff807`630c8e01 fffff807`63241810 fffff807`638af216 : kdcom!READ_PORT_UCHAR+0x4
    01 fffff807`63a02aa4 : 00000000`00000b3f 0000027f`192a8164 ffffb680`b1631c20 00000000`ab9a0dc1 : kdcom!ReadPortWithIndex8+0x21
    02 fffff807`63a02239 : ffffb680`b1631c24 00000000`00000001 00000000`00000b3f 00000000`00000001 : kdcom!Uart16550GetByte+0x54
    03 fffff807`63a01689 : 00000000`00000000 ffffb680`b1631c60 00000000`00000000 00000000`00000002 : kdcom!KdCompGetByte+0x169
    04 fffff807`63a01c03 : ffffb680`b1631df0 00000000`00000000 00000000`00000000 ffffb680`b1631da0 : kdcom!KdReceivePacket+0xb9
    05 fffff807`6372785c : ffffb680`b163e340 ffffb680`b1631e20 ffffb680`b1631f50 ffffb680`b1631f50 : kdcom!KdSendPacket+0x1b3
    06 fffff807`637269ef : ffffb680`b163e340 ffffb680`b163e340 00000000`00000000 ffffb680`b1632938 : nt!KdpSendWaitContinue+0x75c
    07 fffff807`6309b2b4 : ffffb680`b1632180 ffffb680`b1632680 00000000`0010001f ffffb680`b1670000 : nt!KdpReportExceptionStateChange+0x9b
    08 fffff807`63729665 : ffffb680`b1632180 ffffb680`b1632680 ffffb680`b1632180 ffffb680`b1670080 : nt!KdpReport+0xb4
    09 fffff807`62e7c1f8 : ffffb680`b1632938 ffffb680`b1632100 ffffb680`b1632180 ffffb680`b1632680 : nt!KdpTrap+0x14d
    0a fffff807`62e7be65 : ffffb680`b1632938 ffffb680`b1632680 ffffb680`b1632180 ffffb680`b1670080 : nt!KdTrap+0x2c
    0b fffff807`62fd4e42 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiDispatchException+0x135
    0c fffff807`62fceb7b : 00000230`00000000 fffff807`63a033a1 ffff9007`78f39000 00000000`00000039 : nt!KiExceptionDispatch+0xc2
    0d fffff807`62fcb241 : fffff807`62fdff72 ffffb680`b1660180 00000000`00000001 ffffb680`b1670080 : nt!KiBreakpointTrap+0x2fb
    0e fffff807`62fdff72 : ffffb680`b1660180 00000000`00000001 ffffb680`b1670080 00000000`00000d91 : nt!DbgBreakPointWithStatus+0x1
    0f fffff807`63065fd8 : 00000000`00000001 00000000`00000000 00000000`00000d90 ffffb680`b20c04c0 : nt!KdCheckForDebugBreak+0x90bda
    10 fffff807`62f33f4f : ffffb680`b1bc0020 ffffb680`b1660180 00000000`00000286 00000000`00000d91 : nt!KeAccumulateTicks+0x12ea18
    11 fffff807`6388247c : 00000000`00000000 ffff9007`78e9e300 fffffb08`23e29690 ffff9007`78e9e3b0 : nt!KeClockInterruptNotify+0xcf
    12 fffff807`62e528b5 : ffff9007`78e9e300 fffff807`62e93ff7 00000000`00000000 ffff9c88`73a07332 : hal!HalpTimerClockIpiRoutine+0x1c
    13 fffff807`62fc4fda : fffffb08`23e29690 ffff9007`78e9e300 ffff9007`7a6df100 ffff9007`78e9e300 : nt!KiCallInterruptServiceRoutine+0xa5
    14 fffff807`62fc5527 : 00000000`00000000 fffffb08`23e29690 ffff9007`78e9e300 00000000`00000000 : nt!KiInterruptSubDispatchNoLockNoEtw+0xfa
    15 fffff807`6389754f : fffff807`62f91d86 00000000`00000000 ffff9007`7a6df100 ffffb680`b1660180 : nt!KiInterruptDispatchNoLockNoEtw+0x37
    16 fffff807`62f91d86 : 00000000`00000000 ffff9007`7a6df100 ffffb680`b1660180 ffff9007`7a6df010 : hal!HalProcessorIdle+0xf
    17 fffff807`62f3624b : 00000000`00000000 00000000`00989680 00000000`00000000 00000000`00000021 : nt!PpmIdleDefaultExecute+0x16
    18 fffff807`62f359ff : 00000048`0011017e 00000000`00000004 00003407`00570002 00000000`00000000 : nt!PpmIdleExecuteTransition+0x6bb
    19 fffff807`62fc6fcc : 00000000`00000000 ffffb680`b1660180 ffffb680`b1670080 ffff9007`7a010040 : nt!PoIdle+0x33f
    1a 00000000`00000000 : fffffb08`23e2a000 fffffb08`23e24000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x2c

    your IN instruction here

    Code:
    1: kd> ub .
    kdcom!WRITE_REGISTER_ULONG64+0xa:
    fffff807`63a0108a cc              int     3
    fffff807`63a0108b cc              int     3
    fffff807`63a0108c cc              int     3
    fffff807`63a0108d cc              int     3
    fffff807`63a0108e cc              int     3
    fffff807`63a0108f cc              int     3
    kdcom!READ_PORT_UCHAR:
    fffff807`63a01090 0fb7d1          movzx   edx,cx
    fffff807`63a01093 ec              in      al,dx   <<<<<<<<<<<<<<<<<<<<<
    th IRQ old style comport
    Code:
    1: kd> r @rdx
    rdx=00000000000002f8

  12. #12
    Quote Originally Posted by blabberer View Post
    i don't recall any command named synenumtype so obviously it errs because s is command for Searchmemory
    Thanks for info Blabbs, very helpful.

    Re symenumtypes, I got it from the debug help library in dbghelp.chm. Maybe I am using the command incorrectly, they may be for writing debug drivers/apps. If you look under DbgHelp References under DbgHelp Functions you can see several starting with 'sym'. At least one of them did work, the 's' was accepted, but just as many failed with windbg omitting the 's'. I noted that you are using dbh, which I had forgotten.

    I have made some headway with the comm stream protocol. I am able to get the host to recognize more serial traffic by sending it commands from a serial comm app. Would be nice to find the actual codes used and to trace those from the host into the debugging engine on the target. I'll play more with what you provided.

    I am playing with other mods in DT4W, like kdbgctrl.exe on the target. If I enter kdbgctrl.exe -c it says kernel mode debugger is enabled. If I enter kdbgctrl.exe -e it says kernel debugger is already enabled, as expected. However, if I try kdbgctrl.exe -ea, for auto-enable, it returns 'Unable to set kernel debugger auto-enable, NTSTATUS 0XC0000003. I am trying to determine if kernel debugging mode has somehow been turned off.

    There is some good info on kdcom.dll and the structure of its related headers at the following link. I'd like to find a way to check it out from the target end but i don't know as yet whether I can make windbg -kl work in that way. I notice you did something with the IN port using the serial address but have not had a chance to check it yet.

    http://sysprogs.com/legacy/articles/kdvmware/kdcom.shtml

  13. #13
    @kayaker... procmon on the target did not react to the host machine running windbg in k-mode. I'll have to check out apimon or maybe logger.exe in theDT4W directory if I ever figure out how to make it work.

Similar Threads

  1. Replies: 0
    Last Post: January 12th, 2008, 00:08
  2. Replies: 0
    Last Post: January 12th, 2008, 00:08
  3. Replies: 0
    Last Post: January 12th, 2008, 00:08
  4. Replies: 0
    Last Post: January 12th, 2008, 00:08
  5. Replies: 0
    Last Post: January 12th, 2008, 00:08

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •