Information Security & US & Life & Love & Fashion & Hacking & Passion

Anyone who uses SoftICE on a regular basis will tell you that it's the most powerful debugger on the planet. SoftICE is a kernel debugger that sits in between the operating system and the CPU so when SoftICE is active, the entire operating system stops dead. The fact that SoftICE can gain control of the operating system like that means that it can hunt anything it wants to down in memory and tell you about it. The real power of SoftICE is that its informational commands make it a piece of cake to find out what just about anything is in the system. For example, you can look at any event in the system and see what its name is and what its state is. The following list shows you just a taste at the Windows internals SoftICE can tell you about:

• Complete thread and process information
• All events, mutex, and semaphore information
• All object handle information
• Complete process and driver memory information
• All Win32 heap information
• Structured Exception Handling (SEH) frames
• DLL exports

SoftICE is also a regular debugger as well that offers full source and symbolic debugging just like any GUI debugger. The vast majority of SoftICE users are using it to debug device drivers. However, because of SoftICE's extreme power, I find that SoftICE is the only tool that helps in three key scenarios for us user mode developers. The first is in heavily multithreaded applications. Since graphical debuggers using system resources along with the debuggee, they can interfere with the debuggee and mask subtle timing problems. The second area where SoftICE really shines is when debugging larger multi-process applications and NT services such as cross process COM systems, MTS, and IIS applications. Some of the big web sites that people are developing might have five to fifteen different processes associated with them. If you have 4GB of RAM on the machine, you might be able to get all processes running under different copies of the always memory hungry VC debugger. With SoftICE, you just load the symbols for all the processes and set breakpoints in the processes where you want to stop. If you want to single step from a client application to a server application going through the RPC channel, it's a piece of cake with SoftICE. The final scenario where SoftICE is invaluable is reverse engineering. At times, you need to see how something works or is implemented and with SoftICE's ability to break on anything anywhere and with all those informational commands, the whole operating system is much easier to figure out.

While there are many positive things about SoftICE, it's only fair if I mention some of the problems associated with it. The first is that SoftICE is not cheap if you are only interested in user mode debugging. The device driver developer is SoftICE's main audience so it's only sold as part of NuMega's DriverStudio or the SoftICE Driver Suite. There are wonderful tools, such as BoundsChecker, TrueTime, and TrueCoverage for drivers, in those packages but if you are only looking for SoftICE, you have to purchase some tools you might not use. The biggest problem with SoftICE is that it is just hard to use. On one hand, we should commend NuMega that it has kept a consistent SoftICE user interface from the original DOS version through to today's Windows 2000 version. On the other hand, the fact that that user interface is character based with none of the ease of use features we are used to in Windows applications means you must make a solid time investment to get up speed with SoftICE because the majority of the interaction with the tool is through typing commands. Additionally, you must also make the time investment to understand what SoftICE is trying to tell you from an informational standpoint. The SoftICE programmers assume that you already know what the items they display are so you need to know what's going on from a system-programming standpoint. Even with all these issues, the pros certainly outweigh the cons and if you need SoftICE, you need it bad.

Before Staring

Once you have the latest and greatest version, it's time to check your video card. SoftICE video support used to be the biggest pain in the neck for all SoftICE users. Fortunately, with the advent of DirectX, SoftICE can now grab the video memory in a much cleaner manner so each video card doesn't need a specific SoftICE video driver. During SoftICE installation, you will have to pick the video mode. In the Display Adapter setup, check the "Universal Video Driver" check box, set the card selection to Standard VGA, and press the "Test" button. If your video card supports the Universal Video Driver, you should be in good shape. Even if the Display Adapter setup appears to work, there are odd cases where SoftICE still might not properly display. If that's the case, check with NuMega Technical Support and try using a different video card.

Finally, before you ever use SoftICE, take an afternoon and read the manuals! The SoftICE Technical Writers have done an excellent job and put in all sorts of excellent advice on how to get the most out of this very complicated product. An excellent chapter in the Using SoftICE manual is Chapter 11, Exploring Windows NT. That chapter will introduce you to using SoftICE to reverse engineer a few things in Windows NT. For the rest of the column, I will assume that you've read the manuals and have at least debugged "Hello World!" with SoftICE.

Getting Up To Speed

Before I talk about various tips and tricks, I want to briefly discuss a few screen positioning commands, a small bit about the mouse, and what to do when the mouse goes nuts. The default screen position for SoftICE is on the upper left corner of the screen, which is not the best place since you mostly look at the center of the screen. The magic key command of "CTRL+ALT+C" will automatically center the SoftICE window on the screen. If you want to move the SoftICE window around, your can use "CTRL+ALT+" to move in any direction you want. SoftICE also has three different fonts, numbered one, two, and three, for your viewing pleasure. Font 1 is the smallest and font 3 is the largest. The default is font 1 and unless you are running at 800x600, it will probably be too small to read. You can use the "set font #" command to switch between the fonts. I personally find font 2 to be the most readable on all the computers I use. Two SoftICE macros I have found very useful are as follows:

MACRO BIG="set font 2;lines 52"
MACRO SMALL="set font 1;lines 94"

I add these two to WINICE.DAT, which you can add either manually or through LOADER32.EXE's SoftICE Initialization Settings dialog Macro tab, so I can always use them. The BIG macro forces the font to the medium size version and increases the lines shown in the window. Depending on your screen resolution, you might need more or less. The SMALL macro, as you can guess, shrinks the font and increases the number of lines you can see.

One thing you will want to get familiar with very quickly is SoftICE's mouse support as it is very helpful to speeding up your debugging. First off, double clicking on a source line or an address will set a breakpoint on that line. Single clicking on a register value in the Register window or a data location in the Data window allows you to edit the values in place. Right clicking will bring up the context menu. If you right click on a valid address or symbol, right clicking will also highlight that value. One thing that's a little different about the SoftICE context menu is that it includes the "Copy & Paste" command. When you are looking to add parameters to normal commands, the "Copy & Paste" command can make your usage much faster. For example, if you want to set a breakpoint on a function in the code window disassembly, type "BPX " (notice the space!), right click on the function and select "Copy & Paste" from the right click context menu. Another really cool command on the right click menu is the "What" command. If you ever wanted to know what any value in a register, the disassembly, or in the Data window was, right click and select "What." SoftICE is quite good at figuring out most items so just ask away!

Once of the biggest frustrations with SoftICE is when the mouse goes berserk. If you happen to be moving the mouse just right when SoftICE pops up because of an exception or a breakpoint, SoftICE interrupts the mouse packets in the middle so the mouse can be out of synchronization. The trick is to carefully tap the mouse so that the right click menu appears. When the menu appears, press "CTRL+M" a few times to get the mouse back in the groove.

Helpful Tricks

Getting Back to the Current Instruction Pointer

Setting the Address Context

Breakpoints

Unassemble Command

Minimizing Extraneous Noise

Getting Out of Infinite Loops

Wrap Up

Before we get started, I have to discuss a bit about the legal ramifications to reverse engineering. Most of your software licenses have clauses in them where you are not supposed to do any reverse engineering. What I discuss in this column might cause you to break those licenses. Therefore, caveat emptor, buyer beware. If a software manufacturer does sue you, you cannot hold me responsible as I am giving you plenty of warning. I am not a lawyer so check with your own legal council before you proceed. The final note about reverse engineering is that it can make your life very difficult in future releases of your product. While you might figure out how something works internally, if you rely on that internal knowledge in your product, you can easily break the next time the operating system or third party product you integrate changes. You should never rely on hacks you figured out through reverse engineering unless you are prepared to spend a considerable amount of time re-reverse engineering each time a new release comes out. Operating system writers and third party vendors spend a considerable amount of time working on documented interfaces for you to use. If you circumvent them, you can pay an exorbitant price down the road. They are not called undocumented interfaces or techniques for nothing!

The Big Mistakes

The best way to show the first mistake is to start out with the first two lines of an email I received recently: "I need to figure out how Word does word wrapping with variable pitched fonts. How do I start?" The mistake is that people think they can reverse engineer their way to an algorithmic design for their product. While I'm sure if you were given enough resources and infinite time, you could probably figure it out. However, you would take the remainder of your 30+-year career looking at the same four billion assembly language instructions. Reverse engineering will never take the place of designing your application.

The second, and most common, mistake is that people try to reverse engineer far more than they should. It's the age-old case of biting off more than you can chew. To successfully reverse engineer, you need to have a clear and concise goal. My rule of thumb is to never embark on a reverse engineering task unless I feel it's solvable in less than a day or two. It's just not worth the effort to reverse engineer something for several weeks when you could spend a couple of days designing around the problem or issue right up front.

What You Have To Know

While most people think being an assembly language programming god is the first step to reverse engineering, it really isn't. It helps quite a bit, but I've figured out how many things work without ever cracking a disassembler. The most important thing when reverse engineering is to step back and figure out how you would implement the functionality you are reverse engineering. By writing out the algorithm you would use to solve a problem, you can many times "see" very quickly how something works.

An excellent example is when I needed to figure out how compiled VB binaries and p-code VB binaries called into the VB run time, MSVBVM60.DLL. My first thought was that if I were responsible for designing the VB run time, I would want the interfaces to be the same no matter how the VB code was compiled. That way I would have only one way of testing interface calling. I had heard that p-code executes directly and not run through a Just-In-Time (JIT) compilation process. Therefore, the p-code calls would have to go through some "thunk" to call the run time. In scripting languages, thunks allow the scripting language to call into actual CPU code. The interesting thing with thunks is that they are allocated memory that the programmer has the CPU instruction pointer jump to. With this thought, I figured that if I were writing the compiled VB portions, I would use the same technique.

When I was going through this thought process, I never once used the debugger or looked at a disassembly. In essence, I was making a hypothesis. The good old scientific method proves itself yet again. Armed with my hypothesis on how VB made the calls, loaded up one native compiled application and one p-code compiled application into two debuggers. I set a breakpoint on rtcBeep exported from MSVBVM60.DLL because I guessed that the VB intrinsic function, Beep, must call down into rtcBeep. When each compiled program stopped on rtcBeep, I looked up the call stack at the calling function. The Call stack window showed that the address for the caller did not have symbols. I then checked the address of the memory against the Modules dialog and noticed the address of the memory did not appear in any of the loaded modules. I then when through the same process with the p-code compiled application, so I could verify my hypothesis again. Therefore, memory containing the thunk callers came from allocated memory and both native compiled and p-code compiled VB both called through thunks the same way. It didn't take any knowledge of assembly language to figure out the solution, just a hypothesis on how I would have implemented the functionality if I were to write it, and a way to verify that hypothesis.

As you can see from the previous discussion, it also helps to have an idea how different problems can be solved using the facilities provided by the operating system. In the Windows world, that means knowing about how Windows itself works. The first book you need to read cover to cover is Charles Petzold's Programming Windows. Charles covers how the basics of Windows and shows you how it all fits together. Fundamentally, Windows is a simple messaging based system and if you know messaging like the back of your hand, you will have a much better chance at figuring out how to consider solving various reverse engineering challenges. You will learn more about Windows if you sit down and write Notepad in straight C programming than almost anything else. The second book you need to read from cover to cover is Jeffrey Richter's Programming Applications for Windows. Once you understand the fundamentals of Windows, Jeffrey's book will get you up to speed on things like memory management and DLLs. Once you have a good grasp of those two technologies, you will be able to see how many problems in Windows get solved. Depending on what you are doing, a few other books might be useful as well. David Solomon and Mark Russinovich's Inside Windows 2000 can give you insight as to how Windows 2000 works at the kernel level. If you want to learn how to take advantage of the debugger, my own Debugging Applications can show you how to do advanced things with the Visual C++ debugger.

As much as you would like to avoid it, you do need to know assembly language in order to do the most advanced reverse engineering. There are still a few books floating around on how to program Intel x86 assembly language. The one I used to learn with was Mastering Turbo Assembler by Tom Swan, which I am sure is out of print. Assembly language is still taught at the college level so there are good learning books out there. In order to learn assembly language you should look at using the Microsoft Assembler (MASM), which is available with your Universal MSDN subscription, to write either a few simple programs or a DLL with some routines in them. You don't have to get super proficient at assembly language, you just need to be able to read it.

What You Have To Use

After reading the books, you need to start developing your toolkit. There are many tools you can use, but I thought I would list the tools that I have purchased or acquired and I move from machine to machine when reverse engineering. I'll start out with the free products and work my way to the commercial products.

PEDUMP

Matt Pietrek wrote PEDUMP and it's available on the MSDN CD or MSDN Online. PEDUMP dumps all the information about a Portable Executable (PE) binary. You can get the same output with DUMPBIN from Visual Studio, but I like the format of PEDUMP better. When looking for imported and exported functions, you need PEDUMP.

REGMON and FILEMON

Mark Russinovich wrote both REGMON and FILEMON, which are free and downloadable from www.sysinternals.com. REGMON monitors and completely reports all registry access on your computer. FILEMON monitors all disk and file accesses on you computer. Both of these tools allow you to easily see who's doing what to whom. One time I purchased a product that was downloadable and as a challenge, I wanted to see if I could break their registration scheme before I entered my valid, purchased ID. A total of two minutes with REGMON and I broke the scheme.

DEPENDS

The DEPENDS program from the Platform SDK reports all imported functions used by a program. You can even run an application under depends and see what functions it acquires through GetProcAddress. DEPENDS is the tool for monitoring what exports are used out of a DLL.

BoundsChecker

BoundsChecker is a commercial error detection tool from Compuware/NuMega. You can get more information about BoundsChecker by visiting www.numega.com. What many people don't realize about BoundsChecker is that it will monitor and record each and every API call a program makes and show them in the wonderful Event view. What makes it even more interesting is that BoundsChecker will record the complete parameter information and function return values as well. While you can't see into the APIs, BoundsChecker makes it quite easy to see API functions an algorithm called to get the work done. When I worked at NuMega, one of the demos we had was to show how the Solitaire game did the card magic at the end of the game.

SoftICE

SoftICE is also a commercial product from Compuware/NuMega. When you think of reverse engineering in Windows, SoftICE is right there because it's used by more people to reverse engineer things than anything else. I described how to get started with SoftICE in a previous column so you can turn there to get an idea how to use it. What I've always found amusing is that SoftICE is one of the most heavily pirated pieces of software around today. The beauty of SoftICE is that it allows you to see anywhere and everywhere, as well as get more information about the operating system than anything else.

A Disassembler

The final tool you need for larger reverse engineering chores is a disassembler. You already have one with the -DISASM switch to DUMPBIN. What makes DUMPBIN a little more useable is that it will use any symbols it can find so you can get more information. What you will probably want to do is to write a Perl script to process the output to make it more readable. While you can always use the debugger's Disassembly window, you sometimes need the disassembly in a text file.

Wrap Up