Archive
Save dump: The computer has rebooted from a bugcheck
How to analyse a server which has crash after a blue screen of dead (BSOD). An example: error from rebooting DC’s after BSOD in the eventlog:
The computer has rebooted from a bugcheck. The bugcheck was: 0x0000001e (0xc0000005, 0x8046e3b2, 0x00000000, 0x00000000). Microsoft Windows 2000 [v15.2195]. A dump was saved in: C:\WINNT\MEMORY.DMP.
To troubleshoot the Memory.DMP, follow Microsoft knowledgebase article
1) First, download the Windows XP support tools at http://www.microsoft.com/downloads/details.aspx?FamilyId=49AE8576-9BB9-4126-9761-BA8011FABF38&displaylang=en
2) Run “dumpchk” against the memorydump file. (I copied the file 2gb to my laptop for fast processing). This will result in a dumpfile:
----- 32 bit Kernel Full Dump Analysis
DUMP_HEADER32:
MajorVersion 0000000f
MinorVersion 00000893
DirectoryTableBase 30f71000
PfnDataBase 89092000
PsLoadedModuleList 80485b80
PsActiveProcessHead 80487608
MachineImageType 0000014c
NumberProcessors 00000002
BugCheckCode 0000001e
BugCheckParameter1 c0000005
BugCheckParameter2 8046e3b2
BugCheckParameter3 00000000
BugCheckParameter4 00000000
PaeEnabled 00000000
KdDebuggerDataBlock 80471a70
Physical Memory Description:
Number of runs: 5
FileOffset Start Address Length
00001000 00001000 0001f000
00020000 00022000 0001e000
0003e000 00050000 0004f000
0008d000 00100000 00eff000
00f8c000 01000000 7eff3000
Last Page: 7ff7e000 7fff2000
KiProcessorBlock at 80484f20
2 KiProcessorBlock entries:
ffdff120 8905f120
Windows 2000 Kernel Version 2195 (Service Pack 4) MP (2 procs) Free x86 compatible
Kernel base = 0x80400000 PsLoadedModuleList = 0x80485b80
Debug session time: Thu Jul 16 06:45:30 2009
System Uptime: 0 days 0:20:27
start end module name
80400000 805a2940 nt Checksum: 001AB628 Timestamp: Mon Mar 05 16:51:43 2007 (45EC3C8F)
Unloaded modules:
f6580000 f6589000 redbook.sys Timestamp: Thu Jul 16 06:25:29 2009 (4A5EABB9)
f66d0000 f66d5000 Cdaudio.SYS Timestamp: Thu Jul 16 06:25:29 2009 (4A5EABB9)
bfdaf000 bfdb2000 Sfloppy.SYS Timestamp: Thu Jul 16 06:25:29 2009 (4A5EABB9)
Finished dump check
3) If this doesn’t give any hint about the reason for the crash you need to install the microsoft debugging tools. (Windbg). The debugging tool can be find at:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
4) WinDBG uses symbol files to determine which component (or which reference) caused the problem. They come in two forms: a http link to the Microsoft directory and installable symbol files.
http://www.microsoft.com/whdc/DevTools/Debugging/symbolpkg.mspx
6) Extract the symbol files for the OS to c:\windows\symbols
Just click on the installer(s) to install the Symbol files. This will take a while and will take about 750 mb of memoryspace. Note: make sure that you’ll install them all in the same directory.
7) Then add the Microsoft symbol filepath:
File -> Symbol file Path ...
Add the following link:
c:\windows\symbols; http://msdl.microsoft.com/download/symbols
Open the memory.dmp file:
File -> Open Crash DUmp ...
Select the MEMORY.DMP
After loading the Windbg will load the symbol files and check the MEMORY.DMP
9) Click on the prompt and type the following text:
!analyze -v
The following (example) output shows:
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
KMODE_EXCEPTION_NOT_HANDLED (1e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: dd46e3b2, The address that the exception occurred at
Arg3: 00000000, Parameter 0 of the exception
Arg4: 00000000, Parameter 1 of the exception
Debugging Details:
------------------
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".
FAULTING_IP:
nt!ExFreePoolWithTag+162
dd46e3b2 f60701 test byte ptr [edi],1
EXCEPTION_PARAMETER1: 00000000
EXCEPTION_PARAMETER2: 00000000
READ_ADDRESS: 00000000
DEFAULT_BUCKET_ID: INTEL_CPU_MICROCODE_ZERO
BUGCHECK_STR: 0x1E
PROCESS_NAME: ntfrs.exe
LAST_CONTROL_TRANSFER: from 00000000 to dd4308e6
STACK_TEXT:
f55b3b28 00000000 00000000 00000000 00000000 nt!KiDispatchException+0x30e
STACK_COMMAND: .bugcheck ; kb
FOLLOWUP_IP:
nt!ExFreePoolWithTag+162
dd46e3b2 f60701 test byte ptr [edi],1
SYMBOL_NAME: nt!ExFreePoolWithTag+162
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: nt
IMAGE_NAME: ntkrnlmp.exe
DEBUG_FLR_IMAGE_TIMESTAMP: 45ec3c8f
FAILURE_BUCKET_ID: 0x1E_nt!ExFreePoolWithTag+162
BUCKET_ID: 0x1E_nt!ExFreePoolWithTag+162
Followup: MachineOwner
---------
It will (hopefully) show the right process (in this case ntfrs.exe that caused the problem) for further troubleshooting…
Unmute soundmixer of remote system and set volume to maximum ;)
Nice little vbs-script to unmute a computer. Use psexec tools to fire on remote system. We use this to play wave file’s and MP3 on computers of colleagues
set oshell=createobject("wscript.shell")
oshell.run "sndvol32"
oshell.appactivate "volume control"
WScript.sleep 600
oshell.appactivate "volume control"
oshell.sendkeys "{tab}"
wscript.sleep 600
For X = 1 To 500
oshell.sendkeys "^{UP}"
oshell.appactivate "volume control"
Next
wscript.sleep 600
oshell.appactivate "volume control"
wscript.sleep 600
oshell.sendkeys "{tab} "
wscript.sleep 600
oshell.sendkeys "%{f4}"
Linux performance monitoring with TOP
Top has some usefull settings for troubleshooting performanceproblems on Linux settings. For Example, try the current process-thread mode:
top -i -d 0.05 -c
Monitoring QoS with TCPDUMP
To monitor the AF31 class you could use the following filter:
tcpdump -i eth0 -nn ip[1]=0x68
This command will show all AF31 traffic. For other classes:
AF11 = '001010', AF12 = '001100', AF13 = '001110', AF21 = '010010', AF22 = '010100', AF23 = '010110', AF31 = '011010', AF32 = '011100', AF33 = '011110', AF41 = '100010', AF42 = '100100', AF43 = '100110'.
PIX Generate new RSA key for SSH
If you change the hostname or domain-name on your pix the RSA key will become invalid. You will need to generate a new key, like in the following example:
hostname FW12345 domain network4all.local ca zeroize ca save all ca gen rsa key 1024 ca save all wr m (wr n)
Decimal system
Now you know what the next harddisksizes will be: pentabytes
Decimal system Prefix's Factor Exponent Prefix --------------------------------------------------- 1 000 000 000 000 000 000 000 000...10^24....yotta 1 000 000 000 000 000 000 000...10^21....zetta 1 000 000 000 000 000 000...10^18....exa 1 000 000 000 000 000...10^15....peta 1 000 000 000 000...10^12....tera 1 000 000 000...10^9.....giga 1 000 000...10^6.....mega 1 000...10^3.....kilo 100...10^2.....hecto 10...10^1.....deka 0.1...10^-1....deci 0.01...10^-2....centi 0.001...10^-3....milli 0.000 001...10^-6....micro 0.000 000 001...10^-9....nano 0.000 000 000 001...10^-12...pico 0.000 000 000 000 001...10^-15...femto 0.000 000 000 000 000 001...10^-18...atto 0.000 000 000 000 000 000 001...10^-21...zepto 0.000 000 000 000 000 000 000 001...10^-24...yocto ---------------------------------------------------
ICMP type codes
ICMP(Types/Codes) Testing Destination Reachability & Status (0/0) Echo-Reply (8/0) Echo Unreachable Destinations (3/0) Network Unreachable (3/1) Host Unreachable (3/2) Protocol Unreachable (3/3) Port Unreachable (3/4) Fragmentaion Needed and DF set (Pkt too big) (3/5) Source Route Failed (3/6) Network Unknown (3/7) Host Unknown (3/9) DOD Net Prohibited (3/10) DOD Host Prohibited (3/11) Net TOS Unreachable (3/12) Host TOS Unreachable (3/13) Administratively Prohibited (3/14) Host Precedence Unreachable (3/15) Precedence Unreachable Flow Control (4/0) Source-Quench [RFC 1016] Route Change Requests from Gateways (5/0) Redirect Datagrams for the Net (5/1) Redirect Datagrams for the Host (5/2) Redirect Datagrams for the TOS and Net (5/3) Redirect Datagrams for the TOS and Host Router (6/-) Alternate-Address (9/0) Router-Advertisement (10/0) Router-Solicitation Detecting Circular or Excessively Long Routes (11/0) Time to Live Count Exceeded (11/1) Fragment Reassembly Time Exceeded Reporting Incorrect Datagram Headers (12/0) Parameter-Problem (12/1) Option Missing (12/2) No Room for Option Clock Synchronization and Transit Time Estimation (13/0) Timestamp-Request (14/0) Timestamp-Reply Obtaining a Network Address (RARP Alternative) (15/0) Information-Request (16/0) Information-Reply Obtaining a Subnet Mask [RFC 950] (17/0) Address Mask-Request (18/0) Address Mask-Reply Other (30/0) Traceroute (31/0) Conversion-Error (32/0) Mobile-Redirect
RFC1918 Reserved namespaces
0.0.0.0/8 ! broadcast 10.0.0.0/8 ! RFC 1918 private 127.0.0.0/8 ! loopback 169.254.0/16 ! link local 172.16.0.0/12 ! RFC 1918 private 192.0.2.0/24 ! TEST-NET 192.168.0/16 ! RFC 1918 private 224.0.0.0/4 ! class D multicast 240.0.0.0/5 ! class E reserved 248.0.0.0/5 ! reserved 255.255.255.255/32 ! broadcast
Network Classes
Net Host Total Net Addr Addr Addr Number Class Range NetMask Bits Bits of hosts ---------------------------------------------------------- A 0-127 255.0.0.0 8 24 16777216 (i.e. 114.0.0.0) B 128-191 255.255.0.0 16 16 65536 (i.e. 150.0.0.0) C 192-254 255.255.255.0 24 8 256 (i.e. 199.0.0.0) D 224-239 (multicast) E 240-255 (reserved) F 208-215 255.255.255.240 28 4 16 G 216/8 ARIN - North America G 217/8 RIPE NCC - Europe G 218-219/8 APNIC H 220-221 255.255.255.248 29 3 8 (reserved) K 222-223 255.255.255.254 31 1 2 (reserved) (ref: RFC1375 & http://www.iana.org/assignments/ipv4-address-space ) ( http://www.iana.org/numbers.htm ) ----------------------------------------------------------