Lab: Kernel Mode

Objectives After completing this lab, you will be able to: Configure a Windows-based computer to create a complete memory dump Enable CrashOnCtrlScroll and use it to blue screen your computer Use basic debugger commands like !analyze -v and lm v m [module] to get information about a blue screen from a kernel mode memory dump Set up a live kernel mode debugging session Set breakpoints in a live kernel debug Note This lab focuses on the concepts in this module and as a result may not comply with Microsoft security recommendations. Prerequisites Before working on this lab, you must have: Completed the Windows Architecture lab exercises Installed the Debugging Tools for Windows Completed the Symbols and Debugging Tools lab exercises

Estimated time to complete this lab: 90 minutes

Lab: Kernel Mode

Exercise 0 Lab Setup

The Lab Setup section lists the tasks that you must perform before you begin the lab.

Tasks
1.

Detailed steps
a.

Determine if you have a PS/2 keyboard or a USB keyboard. Find out from the Instructor if you will be doing Exercise 4a or 4b

Exercise 2 is configuring CrashOnCtrlScroll. If you have a USB keyboard, you can still set the registry key, but it will not work at the end of the lab. Exercise 4a involves connecting the live kernel debugger via serial cable. Virtual PC image via named pipe.

2.

a.

b. Exercise 4b involves connecting the live kernel debugger to a guest c.

It is unlikely that your classroom will be equipped to support both of these exercises. Verify with your instructor which variation of this exercise you should complete.

Lab: Kernel Mode

Exercise 1 Configuring your Computer for Complete Memory Dump

In this exercise, you will configure your Windows-based computer to create a complete memory dump when a blue screen occurs.

Scenario
In all versions of Windows server operating systems, creating a complete memory dump when a blue screen occurs is the default option on computers with two gigabytes of RAM or less. Windows Workstations default to creating only a small memory dump (mini-dump).

Tasks
1.

Detailed steps
a.

Open System Properties.

Click on the Start button. Double click on the icon for System.

b. Navigate to the Control Panel. c. 2.

Navigate to the Startup and Recovery Options

a. In the System Properties window, select the Advanced tab. b. Under Startup and Recovery Options, click the Settings button. c. In the Startup and Recovery Options window, review the current

settings for System Failure.

3.

Configure the Startup and Recovery Options

a.

Verify that the boxes are checked for Write an event to the system event log and Automatically restart. information box.

b. Verify that Complete memory dump is selected in the Write debugging c.

Verify that the Dump file path is %SystemRoot%\MEMORY.DMP Click Ok to close the Startup and Recovery window Still on the Advanced tab of the System Properties window, click the Settings button in the Performance section. Find the Virtual Memory section and click the Change button. Write down the initial page file size on the C: drive. It should be larger than the current amount of RAM. Click Ok to close the Virtual Memory window. Click Ok to close the Performance Options window. Navigate to My Computer Verify that the free space is greater that the amount of RAM

d. Verify that the box is checked for Overwrite any existing file. e. 4.

Check the pagefile configuration

a.

b. In the Performance Options window, select the Advanced tab. c. d. e. f. 5.

Check the free space on C:

a.

b. Right click on the C: drive and choose Properties. c.

Lab: Kernel Mode

Exercise 2 Enabling CrashOnCtrlScroll

In this exercise, you will enable CrashOnCtrlScroll and use it to blue screen your computer.

Scenario
When you need to test changes made to the Windows recovery settings, CrashOnCtrlScroll allows you to generate a blue screen and test the recovery settings without waiting for a problem to occur. CrashOnCtrlScroll is also useful for creating memory dumps of servers that are not responsive. Because you are able to cause a blue screen, you can use this feature to capture a dump file with information about the state of your computer when it is not responding.

Tasks
1.

Detailed steps
a.

Open regedit.exe.

Click on the Start button and click Run. NOTE: On Windows 2000 use regedt32.exe instead.

b. In the Run window, type regedit.exe.

c. 2. Create the value a.

Click Ok. In Registry Editor, expand HKEY_LOCAL_MACHINE HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ i8042prt \ Parameters In Parameters you should already see values for PollingIterations, PollingIterationsMaximum, and ResendIterations. DWORD value.

CrashOnCtrlScroll and set it equal to 1.

b. Keep expanding to navigate to:

c.

d. Click on the Edit drop-down menu, navigate to New, and choose e. f. g.

For the name of the new value, type CrashOnCtrlScroll Double click on the value you just created and set it to 1 Click Ok.

h. Close Registry Editor 3. Reboot. 4. Cause a blue screen a. a.

Shut down and restart your computer. While holding down the Control key on the RIGHT side of your keyboard, press the Scroll Lock key twice. This will cause a blue screen crash. After your computer restarts, log in to Windows.

5. Log into Windows

a.

Lab: Kernel Mode

Exercise 3 Analyzing Blue Screen Memory Dumps

In this exercise, you will open a kernel mode memory dump from three different blue screens, set your symbol path, reload your symbols, and then use !analyze -v and lm v m [module name] to extract information from the memory dump.

Scenario
Memory1.dmp, Memory2.dmp, and Memory3.dmp are located in the folder C:\LabFiles\Module 04 Kernel Mode Troubleshooting\Blue Screen examples. You will perform initial analysis on these memory dumps. Based on what you have learned this far in the workshop, try to form an action plan to continue troubleshooting the issue. For your convenience, each dump has an accompanying solution.txt file which contains a log of the debugger commands used to diagnose these crashes, and a brief description of root cause at the end of the file.

Tasks
1.

Detailed steps
a.

Open WinDbg.

Click on the Start button. Debugging Tools for Windows and click WinDbg to launch the Windows Debugger. From the File drop-down menu, select Open Crash Dump. 04 Kernel Mode Troubleshooting\Blue Screen examples

b. Navigate to All Programs

2. Open Memory1.dmp.

a.

b. In the Open Crash Dump dialog box, browse to C:\ LabFiles\Module c. a. 6.

Choose Memory1.dmp and click Open. When prompted to save the Workspace Information, click No. View your current symbol path with the command: .sympath Symbol search path is:

.sympath

a.

b. You will probably receive the following output: 7.

.symfix c:\symbols

a.

Set your symbol path to the Microsoft Internet Symbol Server with the command: .symfix c:\symbols

b. This command will not return a response

NOTE: If your lab computer does not have access to the Internet, substitute the following command instead to manually set your symbol path to a local share configured for this workshop: .sympath srv*c:\symbols*c:\LabFiles\Symbols
8.

.sympath

a.

View your new symbol path with the command: .sympath Symbol search path is: SRV*C:\Symbols*http://msdl.microsoft.com/downlo

b. You should receive the following output:

9.

.reload

a. a.

Now that your symbol path is set correctly, reload your symbols with the command: .reload Review the output you receive from the command: !analyze -v

10. !analyze -v

Lab: Kernel Mode

b. What was the stop code? c.

What does this stop code mean?

d. What modules were on the stack that faulted? 11. lm v m [module name] 12. Research a. a.

Get as much information as you can about the drivers on the stack that faulted with the command: lm v m [module name] Given the information you now have from the memory dump, consider the following information about each of the memory dumps as you are working on them. MEMORY1.DMP Problem Description: The user says someone from IT made some changes to his computers registry and was still working on his computer when the blue screen occurred. The IT staff member did not explain what she was doing to the users computer. The user wants to be sure his computer will not blue screen again while he is working. MEMORY2.DMP Problem Description: The user reports that she recently updated drivers for several pieces of hardware in her computer. Her video card, her sound card, her network card, and a TV tuner card. She suspects that one of these updated drivers caused the problem but wants to know which one. MEMORY3.DMP Problem Description: The user reports that he has not changed anything since he installed a new USB device two weeks ago, and he has not had any problems since then. Yesterday he unplugged the USB device and his computer blue screened. Because the computer blue screened almost immediately after he unplugged the USB device, he suspects that the device may have caused the crash, but he would like you to analyze the memory dump and see if you can tell if the driver was involved. The user is skeptical because this device was installed two weeks ago and this is the first time he has seen this problem.

b. Search the Internet for more information. 13. Create an Action Plan a.

Based on your analysis of the memory dump and your research, why do you believe the computer blue screened? MEMORY1: ______________________________________________ _________________________________________________________ _________________________________________________________ MEMORY2: ______________________________________________ _________________________________________________________ _________________________________________________________ MEMORY3: ______________________________________________ _________________________________________________________ _________________________________________________________

b. What is your action plan to continue troubleshooting the problem?

Lab: Kernel Mode

Exercise 4a Connecting the Kernel Debugger via Null Modem Cable

In this exercise, you will connect the kernel mode debugger to a live physical computer via null modem cable.

Scenario
As we discussed in the kernel mode module, it is sometimes necessary to connect the kernel debugger do debug a live system instead of a memory dump. This lab requires that you pair up with a partner at the computer next to you, and you must both have at least one physical, 9-pin serial port on each machine and a null modem cable.

Tasks
1.

Detailed steps
a.

Designate one computer as the TARGET and the other will be the HOST.

All of the steps in this lab will specify that they are to be performed on the TARGET, the HOST, or on BOTH computers. which of you will be the TARGET computer and which will be the HOST computer.

b. Between yourself and the person at the computer next to you, decide

(The TARGET is the computer on which the problem is occuring,)

2. Connecting the two a.

computers.
3. Configuring boot.ini on the a.

On BOTH computers: connect the null modem cable to COM1 on each computer. Perform the following tasks on the TARGET computer only. If you see the file boot.ini in the root of the C: drive, skip ahead to step j. Tools drop-down menu and select Folder Options.

TARGET computer

b. Open My Computer and navigate to the C: drive c.

d. If you do not see the boot.ini file in the root of the C: drive, go to the e. f. g.

Click on the View tab. Select the radio button for Show hidden files and folders. Uncheck the box for Hide extensions of known file types files (Recommended). After unchecking this you will receive a warning, click Yes.

h. Scroll down and uncheck the box for Hide protected operating system

i. j.

Click Ok to apply your changes and close the Folder Options window. In the root of the C: drive, find the file boot.ini. Right click on it and select Properties. Click Ok to close the boot.ini Properties window. in Notepad.

k. On the General tab, clear the check box for Attributes: Read-only. l.

m. In the root of the C: drive, double click on boot.ini. This file should open n. In the Format drop-down menu, be sure Word Wrap is not checked. o.

You should only have one ARC path at the end of your boot.ini that starts with multi(0)disk(0) If you have more than one ARC path, ask

Lab: Kernel Mode the instructor for further instructions.

p. Copy the whole last line of boot.ini (the ARC path). This will be similar

to: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=Microsoft Windows XP Professional /noexecute=optin /fastdetect

q. Paste the same ARC path below the original path. You will edit the

original path and the new last line will be a backup option.
r. s. t.

Edit the original ARC path. Add the following switches to the end of the line: /debug /debugport=COM1 Save the boot.ini file and close it. When the computer starts back up you should receive a boot menu. Be sure to boot to the option that says [Debugger Enabled]. Do not continue until after the TARGET computer finishes rebooting. Perform the following tasks on the HOST computer only. Navigate to All Programs Debugging Tools for Windows and click WinDbg to launch the Windows Debugger. On the COM tab, enter the Baud Rate: 57600 On the COM tab, enter the Port: COM1 Click Ok to start kernel debugging. WinDbg will return the following output: Opened \\.\COM1 Waiting to reconnect In the Debug drop-down menu, choose Break. information about the TARGET computer should start to be displayed in the windbg command window.

u. /baudrate=57600 v.

w. Reboot the TARGET computer. x. y. 4. Start WinDbg on the HOST a.

computer and connect.

b. Click on the Start button. c.

d. From the File drop-down menu, select Kernel Debug. e. f. g.

h. When prompted to save the workspace, click No. i.

j.

k. Breaking in with the Debugger should cause it to connect and

5. The TARGET computer

a.

should freeze completely.

Once you are broken in TARGET computer should freeze completely. You will not be able to do anything on the TARGET while the debugger is broken in. This might take a few minutes. On the HOST computer, hit the Enter key. NOTE:Do not close windbg ! Closing windbg on the HOST computer will not free the TARGET computer if the debugger is broken in.

b. In windbg on the HOST computer type the command: g c.

d. The TARGET computer should immediately start to respond again.

Lab: Kernel Mode

Exercise 4b Kernel Debugging a Virtual Machine via Named Pipe

In this exercise, you will connect the kernel debugger to a live virtual machine via a named pipe between a virtual COM port on the guest operating system and windbg on the host computer.

Scenario
As we discussed in the kernel mode module, it is sometimes necessary to connect the kernel debugger do debug a live system instead of a memory dump. This lab requires that you have either Microsoft Virtual PC 2004, Microsoft Virtual Server 2005, or Microsoft Virtual Server 2005 R2 with a guest operating system that is either Windows 2000, Windows XP, or Windows Server 2003.

Tasks
1.

Detailed steps
a.

Identifying the TARGET and the HOST.

All of the steps in this lab will specify that they are to be performed on the TARGET, the HOST, or on BOTH computers. and the Virtual Server Host computer is also the debugger HOST computer.

b. For the purposes of this exercise, the Virtual Machine is the TARGET

2. Configuring boot.ini on the

a.

Perform the following tasks on the TARGET computer only. If you see the file boot.ini in the root of the C: drive, skip ahead to step j. Tools drop-down menu and select Folder Options.

TARGET computer

b. Open My Computer and navigate to the C: drive c.

d. If you do not see the boot.ini file in the root of the C: drive, go to the e. f. g.

Click on the View tab. Select the radio button for Show hidden files and folders. Uncheck the box for Hide extensions of known file types files (Recommended). After unchecking this you will receive a warning, click Yes.

h. Scroll down and uncheck the box for Hide protected operating system

i. j.

Click Ok to apply your changes and close the Folder Options window. In the root of the C: drive, find the file boot.ini. Right click on it and select Properties. Click Ok to close the boot.ini Properties window. in Notepad.

k. On the General tab, clear the check box for Attributes: Read-only. l.

m. In the root of the C: drive, double click on boot.ini. This file should open n. In the Format drop-down menu, be sure Word Wrap is not checked. o.

You should only have one ARC path at the end of your boot.ini that starts with multi(0)disk(0) If you have more than one ARC path, stop ask the instructor for further instructions.

p. Copy the whole last line of boot.ini (the ARC path). This will be similar

10

Lab: Kernel Mode to: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=Microsoft Windows XP Professional /noexecute=optin /fastdetect

q. Paste the same ARC path below the original path. You will edit the

original path and the new last line will be a backup option.
r. s. t.

Edit the original ARC path. Add the following switches to the end of the line: /debug /debugport=COM1 Save the boot.ini file and close it. Turn off the TARGET Virtual Machine and leave it off for now. Use these steps ONLY if you are using Microsoft Virtual PC 2004. Click the Settings button. Select the radio button for Named pipe: In the box type: \\.\pipe\WindowsCPM Click Ok to close the Settings window. When the computer boots you should receive a boot menu. Be sure to boot to the option that says [Debugger Enabled]. Use these steps ONLY if you are using Microsoft Virtual Server 2005. Navigate to All Programs Administration Website Microsoft Virtual Server Virtual Server

u. /baudrate=57600 v.

w. Shut down the TARGET Virtual Machine. x. 3. Mapping the Virtual COM a.

port (Virtual PC 2004)

b. In the Virtual PC Console, select the TARGET Virtual Machine. c.

d. In the Settings for the Virtual Machine click on COM1. e. f. g.

h. Start (turn on) the TARGET Virtual Machine. i. 4. Mapping the Virtual COM a.

port (Virtual Server 2005)

b. Click on the Start button. c.

d. Locate the TARGET Virtual Machine you will debug in the list of

machines and float the mouse over its machine name so the menu appears. Click the link to Edit Configuration.
e. f. g.

In the Virtual Machines Configuration, click the like for COM ports. Under COM port 1, click the radio button for Named pipe:. In the text box for Named pipe:, type the pipe name: \\.\pipe\WindowsCPM On the Virtual Machine Status page, start (turn on) the TARGET Virtual Machine. When the computer boots you should receive a boot menu. Be sure to boot to the option that says [Debugger Enabled]. Perform the following tasks on the HOST computer only. Navigate to All Programs Accessories Command Prompt

h. Click Ok to save your changes to COM 1. i. j. 5. Start WinDbg on the HOST a.

computer and connect with command-line switches.

b. Click on the Start button. c.

d. Change directories to C:\Program Files\Debugging Tools for Windows

with the command: cd\Program Files\Debugging Tools for Windows

e.

Run the command: windbg.exe -k com:pipe,port=\\.\pipe\WindowsCPM

Lab: Kernel Mode WinDbg will return the following output: Opened \\.\COM1 Waiting to reconnect In the Debug drop-down menu, choose Break.

11

f.

g.

h. Breaking in with the Debugger should cause it to connect and

information about the TARGET computer should start to be displayed in the windbg command window.
6. The TARGET Virtual a.

Machine should freeze completely

Once you are broken in TARGET Virtual Machine should freeze completely. You will not be able to do anything on the TARGET while the debugger is broken in.This can take a few minutes. On the HOST computer, hit the Enter key. NOTE: Do not close windbg! Closing windbg on the HOST computer will not free the TARGET computer if the debugger is broken in.

b. In windbg on the HOST computer type the command: g c.

d. The TARGET computer should immediately start to respond again. e.

12

Lab: Kernel Mode

Exercise 5 Setting Kernel Mode Breakpoints

In this exercise, you will set breakpoints on a kernel mode function in the driver TCPIP.SYS to simulate the beginnings of like kernel mode debugging.

Scenario
In Exercise 4 you connected the kernel debugger from a HOST computer to a live TARGET computer or virtual machine. You will now use that live debugging session to set a breakpoint on a function on the TARGET. The breakpoint you set will be on tcpip!ICMPEchoRequest, which is the function the TCPIP.SYS driver uses to send ICMP Echo requests (aka. to send pings). Once this breakpoint is set on the TARGET computer, you will log into it and send a ping to another computer on the network. The debugger will break on this function. NOTE: This lab requires access to the Microsoft Internet Symbol Server to work correctly.

Tasks
1.

Detailed steps
a.

Break in from the HOST computer.

In WinDbg on the kernel mode debugging HOST computer from Exercise 4, go to the Debug drop-down menu and select Break.

b. This action will cause the TARGET computer to freeze completely. 2.

.sympath

a.

View your current symbol path with the command: .sympath Symbol search path is:

b. You will probably receive the following output: 3.

.symfix c:\symbols

a.

Set your symbol path to the Microsoft Internet Symbol Server with the command: .symfix c:\symbols NOTE: For this exercise to work correctly, you must have access to the Microsoft Internet Symbol Server.

b. This command will not return a response

4.

.sympath

a.

View your new symbol path with the command: .sympath Symbol search path is: SRV*C:\Symbols*http://msdl.microsoft.com/downlo

b. You should receive the following output:

5. 6.

.reload !process 0 0

a. a.

Now that your symbol path is set correctly, reload your symbols with the command: .reload Dump the list of running processes with the debugger command: !process 0 0 System process: **** NT ACTIVE PROCESS DUMP **** PROCESS 812719e8 SessionId: none Cid:0004 Peb:00000000 DirBase: 00039000 ObjectTable: e1000b58 HandleCount: 306

b. Look at the very top of the list for the PROCESS address for the

Lab: Kernel Mode Image: System

c. 7.

13

The address you want to copy is the one right after the word PROCESS. In the example above, you would use 812719e8. Use the .process command with the address you copied from the previous step.

.process [ADDRESS]

a.

b. For example: .process 812719e8 8.

.reload

a.

The previous command changed out process context. Without getting too deep into the meaning of the context, it does mean we need to do a .reload. Use the x command to examine the symbols for tcpip.sys, checking for all functions that have ICMP in their name. tcpip!ICMPEchoRequest tcpip!SendICMPIPSecErr tcpip!SendICMPErr tcpip!SendICMPMsg

9.

x tcpip!*ICMP*

a.

b. The debugger should output >10 functions similar to these:

10. bp tcpip!ICMPEchoRequest 11. bl

a. a.

Use the bp [module!function] command to set the breakpoint: bp tcpip!ICMPEchoRequest List all breakpoints to confirm your breakpoint was set with the command: bl 0 e fc3b7108 0001 (0001) tcpip!ICMPEchoRequest

b. The debugger should respond with output like: 12. g a.

Start the TARGET machine running again with the command: g a PING command.

b. The TARGET machine will now run normally until a user tries to send 13. Send a ping from the a.

On the running TARGET computer or Virtual Machine, log in locally. In the Run command, type cmd.exe and click Ok. 127.0.0.1.

TARGET computer Ping 127.0.0.1

b. Click on the Start button and select Run. c.

d. From the command prompt, use the ping command to ping the IP e. 14. .reload 15. k 200 16. Review the stack 17. bc * 18. g 19. Close WinDbg a. a. a. a. a. a.

The TARGET computer should immediately freeze and the HOST debugger should indicate that Breakpoint 0 was hit. After hitting the breakpoint, reload symbols again with the .reload command. View the thread stack trace that hit the breakpoint with the command: k200 You should be able to follow the stack all the way down past ping!main. Clear the breakpoint with the command: bc * Let the TARGET computer start running again with the command: g Close windbg on the HOST computer.