Results 1 to 12 of 12

Thread: VMWare & Softice - Experiences, problems and solutions

  1. #1
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,206
    Blog Entries
    5

    VMWare & Softice - Experiences, problems and solutions

    I recently installed Softice (DriverStudio 3.1) inside a VMWare (4.0.0.4460) machine. While doing this, I experienced some problems, and I thought it would be a good idea to document these problems and solutions here, since I found that they had not earlier been discussed either on this board or the exetools board.

    First of all, when installing Softice to a standard configured VMWare machine, it will hang the system as soon as you try to start Softice from the start menu. This problem is described in the following article in the VMWare support knowledge base:

    http://www.vmware.com/support/kb/enduser/std_adp.php?p_faqid=965&p_created=1049834747&p_sid=l6hHm4Mg

    and also in the following article in the Compuware support knowledge base (which in turn refers to the VMWare article):

    http://frontline.compuware.com/nashua/kb/doc/1702.asp

    For documentation reasons, I also quote it here:

    Q: When I using the SoftICE debugger in a VMWare virtual machine, the guest operating system hangs. How can I work around this problem?

    A: If you plan to use SoftICE in a virtual machine, open the configuration file for the virtual machine in a text editor and add the following two lines:

    vmmouse.present = FALSE
    svga.maxFullscreenRefreshTick = 5

    Otherwise, your mouse and screen may become unresponsive when breaking into the SoftICE debugger, making it appear as if your guest operating system has hung. svga.maxFullscreenRefreshTick can be set to any value greater than 0. Lowering the value of svga.maxFullscreenRefreshTick increases the responsiveness of SoftICE, but it also has a negative impact on the performance of the guest operating system when SoftICE is not running. We recommend removing or commenting out these configuration options when you are not using SoftICE.

    Sometimes, even with these configuration options, pressing Ctrl-D does not display the SoftICE window and your mouse and screen appear unresponsive. Exit SoftICE by pressing Ctrl-D again, then use SoftICE in VGA mode as explained below.


    Open a command prompt (for instance, choose Start > Run, type cmd and click OK).

    Press Alt-Enter to enter VGA mode.
    Press Ctrl-D to enter SoftICE.
    The first time I tried this solution I had two problems:

    First, I added those two lines to the end of the VMWare virtual machine config file (.vmx), but VMWare just kept deleting them! This happened also when VMWare was completely deactivated (and also did not have any active snapshots or suspension states) while I performed the edit of the file. As soon as I started the virtual machine (i.e. not directly when starting the VMWare program) those lines were deleted from the the config file and consequently ignored in the virtual machine.

    For some reason, I found the following method to work to get those lines to stay:

    1.
    Install Softice in the virtual machine.
    2.
    Reboot the virtual machine.
    3.
    Pause the virtual machine after logging into it and being at the desktop of it.
    4.
    Perform a "save snapshot" in VMWare for the virtual machine.
    5.
    Add the two patch lines to the end of the virtual machine config file (.vmx):
    ----------------
    vmmouse.present = FALSE
    svga.maxFullscreenRefreshTick = 5
    ----------------
    6.
    Start the virtual machine again (with the resume/play button).
    7.
    The mouse should now act weird inside the virtual machine (you cannot click anything with it, although it moves, and it will not escape the virtual machine screen without pressing the ctrl+alt hotkey).
    8.
    Press the ctrl+alt hotkey to get the mouse out of the VMWare window, and click the "Reset" button for the virtual machine, it will now reboot.
    9.
    The lines should now still remain in the config file, and be used in the booted virtual machine.
    10.
    Start Softice from the start menu of the virtual machine. It should not hang the system anymore (nothing will display on the screen except a DOS-windows appearing and disappearing though).


    Ok, now comes the second strange part. As described in the quoted VMWare support article text above, "Sometimes, even with these configuration options, pressing Ctrl-D does not display the SoftICE window and your mouse and screen appear unresponsive". At this point it is only the Softice window that is invisible for some reason (i.e. you can press ctrl-d again to get back to the system, or you can type "X" and then press enter too to get back, the Softice window is there and you can interact with it, but for some reason it's just invisible).

    I have no idea why this happens to some people, and even more strange, it first happened to me too. I then tried the solution described above with the DOS window, and it worked, but it is not very desirable I guess, since when a breakpoint hits when you are not in DOS VGA mode you will be in trouble with the invisible window I would think (I have not confirmed this though).

    Even funnier was that after almost giving up trying to get it working without the DOS window (I tried both the default universal video driver mode, and other video modes too, without any luck), I accidently reverted my virtual machine to a snapshot I made just before installing Softice (i.e. DriverStudio 3.1) into the virtual machine (this feature in VMWare kinda sucks btw, the revert button is just beside the tabs from which you choose your virtual machines, and there is not warning when you accidently press that button, it just reverts your virtual machine right away to the last snapshot, possibly causing you to lose tons of work, so be very careful with your aiming when clicking on those tabs!). Anyway, I then re-installed Softice and now all of a sudden it worked perfectly, the Softice window shows up nicely on the screen as soon as I press ctrl-d, and can be used successfully (using all default settings in Softice, both regarding video mode and otherwise, nothing was altered from the defaults). This means that this problem does not have anything directly to do with the virtual machine operating system itself, and neither with the VMWare version or something like that (since it was the exact same virtual machine used, just reverted to the point just before Softice was installed), but rather something else, seemingly random. I guess the VMWare/Compuware guys don't really now either, since they don't give any more info about it in their help either, except for that quite unsatisfactory workaround with the DOS VGA mode. Btw, I'm running my virtual machine in 800*600 resolution, and have VMWare Tools installed and running.

    I hope this will be a help to at least someone, and it would be great if people could collect their own experiences, problems and solutions with VMWare + Softice in this thread too, to make it a really good resource for archival reasons.


    dELTA

  2. #2
    Here is an tut posted by WhoCares on the Exetools Forum. He advised:

    "This is not a cracking tutorial, but a short guide to set up VMWare to support remote debugging via serial port and named pipe. Credits go to my friends from CCF."

    I'd post a link to the thread, but the rest of it was just "thank you" statements. I haven't tried it out, but it does have pictures.

    Regards,
    Attached Files Attached Files
    JMI

  3. #3
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,206
    Blog Entries
    5
    Thanks, strange that this one didn't show up in my search over at exetools, and even stranger that it didn't show up on the search I just tested based on your quote (search string: "VMWare remote debugging")? I've always found the search functionality over at exe tools a little unstable for some reason, any idea why this is?

    Even a search for only "VMWare" returns only fours threads, out of which the one you're referencing is not one.

  4. #4
    Hey dELTA:

    In case you didn't find it, its in the Windows Crack Tutorials Forum, at:

    http://www.exetools.com/forum/showthread.php?t=3499

    It doesn't show up for my search there either. But knowing vBull, a search is only as good as the last time the search index was updated. This thread was from February 24, 2004 and maybe Aaron hasn't updated the indexs since then. As I know from personal experience on this Forum, it does take a good long while on the version 3 series software to update them all. I can't think of anything else that might cause it. They are actually running the "final" 3.0.0 version. I have also noticed that vBull has released a 3.0.1 "bug fix" edition, which I'll download soon for installation here.

    Regards,
    JMI

  5. #5
    jsynesio
    Guest

    Quotes missing

    I had the same problem, but when I added these two lines, I included quotes and this solved the problem for me.

    ----------------
    vmmouse.present = "FALSE"
    svga.maxFullscreenRefreshTick = "5"
    ----------------
    I promise that I have read the FAQ and tried to use the Search to answer my question.

  6. #6
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,206
    Blog Entries
    5
    I noted that those lines in the VMWare/Compuware information lacked quotes too, but it did no difference for me when adding them. I'm currently running it fully working without the quotes, just like in the VMWare/Compuware examples.

  7. #7
    least
    Guest
    Hi,
    I also had a hard times playing with sice and vmware. I've found out that
    when you DON'T install vmware tools and configure sice using universal
    video driver (not windowed - fullscreen) it runs quite fine (W98,W2K).
    The only drawback is that you can use just 640x480, 16 colors.
    BTW DS 3.1 should cooperate better with vmware
    Regards,
    least
    I promise that I have read the FAQ and tried to use the Search to answer my question.

  8. #8
    Naides is Nobody
    Join Date
    Jan 2002
    Location
    Planet Earth
    Posts
    1,647
    Sorry to bother everyone by re-surrecting this thread, but the remote.zip that JMI kindly posted here is corrupted now. Could anyone upload a working copy?
    Last edited by naides; March 14th, 2005 at 16:37.

  9. #9
    Timbo
    Guest
    VOILA
    I promise that I have read the FAQ and tried to use the Search to answer my question.

  10. #10
    Naides is Nobody
    Join Date
    Jan 2002
    Location
    Planet Earth
    Posts
    1,647
    Got it. Thankyou much!

  11. #11
    dmx
    Guest
    2least for use windowed mode try this truck (at VMWare), try edit in config file *.vmx :
    vmmouse.present = "FALSE"
    svga.maxFullscreenRefreshTick = "5"

    here you can found some add-on's for VMWare like Virtual Floppy Driver or something else http://chitchat.at.infoseek.co.jp/vmware/
    I promise that I have read the FAQ and tried to use the Search to answer my question.

  12. #12
    eminemence
    Guest
    Cool tip man.
    Was having a tough time testing my drivers on the actual PC.
    Now I can test my drivers in VMWare.
    Bye.
    --eminemence.
    I promise that I have read the FAQ and tried to use the Search to answer my question.

Similar Threads

  1. Common WinDbg problems and solutions
    By Nynaeve in forum Blogs Forum
    Replies: 0
    Last Post: January 12th, 2008, 00:08
  2. Softice, VMWare and INT3
    By asr in forum Tools of Our Trade (TOT) Messageboard
    Replies: 5
    Last Post: May 26th, 2004, 19:53
  3. Softice itstallation problems on W98
    By IcyDee in forum Tools of Our Trade (TOT) Messageboard
    Replies: 1
    Last Post: April 15th, 2002, 04:55
  4. Softice's ? (evaluate) command problems...
    By DGR in forum Tools of Our Trade (TOT) Messageboard
    Replies: 4
    Last Post: June 8th, 2001, 04:13
  5. Softice problems
    By skiabox in forum Malware Analysis and Unpacking Forum
    Replies: 1
    Last Post: February 15th, 2001, 05:03

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •