"Production debugging ASP.NET applications - PowerPoint"
Production debugging ASP.NET web applications Alex Thissen Trainer/coach Agenda • Why production debugging is hard • Debugging process • Tools • Common scenarios and ASP.NET specifics • Crashes • Memory leaks • Hangs • Questions and answers Introducing… • Class-A – Kennisprovider – Microsoft development – Training & Coaching – http://www.class-a.nl • Alex Thissen – Trainer/ coach – http://blog.alexthissen.nl Challenges of production environment Errors you Limited No Visual never High stakes access to Studio to experienced and pressure environment debug during testing Challenges Production debugging process Prepare Reproduce Observe Analyze Fix If necessary, reproduce again • Focus is on ASP.NET applications – But, most applies to other apps as well Getting started Prepare • Create debug build if possible – Have symbols ready for debugging • Get access to production machine • Deploy tools on machine • Learn tooling • Know how ASP.NET and CLR work Essential tooling Prepare • Debugging Tools for Windows 6.11 – Debuggers: WinDbg, KD, CDB – Scripts: ADPlus.vbs • Debug Diagnostics 1.1 • Stress and load testing – TinyGet (IIS 6.0 Resource Kit) – Visual Studio 2008/2010 Test Edition Reproducing the problem Reproduce • Reduce problem to its essence – Eliminate differences and unknowns – Find simplest path that leads to bug • Repeat bug situation – Put it under load – Make it crash again • Make it easy to reproduce situation Gathering information Observe • Event log • Application trace and log information – Health monitoring • Reliability and performance monitor • Make a memory dump Scenario 1: Unexpected restarts • Symptoms • Slow response time • Error in browser • Loss of session state • Determine • Why caused application restart? • Was it our fault or someone else's? Unexpected application restarts External shutdown Unhandled exception • IISReset • Exception on non-request thread • Preemptive recycle • Kill Possible causes Process exit trigger Application changes • Out of memory (OOM) • Machine/Web.config • Stack overflow • Assemblies • Heap corruption 1st chance, 2nd chance exceptions Worker process (w3wp.exe or aspnet_wp.exe) Common Language Runtime Incoming Handling Background requests CLR Worker threads requests and GC threads 1st chance exception 2nd chance exception Global exception handler CLR Shutdown Two strikes, YOU'RE OUT! Finding shutdown reason • Health monitoring <healthMonitoring> web.config </rules> <add name="Lifetime" eventName="Application Lifetime Events“ provider="SqlWebEventProvider” profile="Default« minInstances="1" maxLimit="Infinite" minInterval="00:01:00" /> </rules> </healthMonitoring> • Investigate _shutdownMessage and _shutdownStack fields of HttpContext Tracking down exceptions • Strategies – Attach debugger and stop on exceptions – Look at recent exceptions in hang dump – Log exceptions with debugger running • Tip: disable pool recycling – Recycled apps get in the way of finding real reason for shutdown Demo: crash application pool • Caused by stack overflow • Do some live debugging CDB and WinDbg quick overview • Debuggers for unmanaged code only • Still very useful for managed code • Load extension Son of Strike (sos.dll) – Make sure it’s for correct .NET framework – .loadby sos mscorwks Scenario 2: Memory leak • Symptoms • Increasing memory usage • Determine • Managed or native memory leak • Possible causes • Static event handler • Finalizer thread blocked • Forgotten GC handles Memory dumps • .dmp files with complete process state • Useful when debugging is not possible – Unable to attach debugger – Post mortem analysis of crashes • 64-bit specifics – Dumps get big – Different GC algorithm (less collections) Taking a dump • Tools to create a dump 1. ADPlus.vbs script: adplus –crash –pn w3wp.exe adplus –hang –p 1337 2. Debug Diagnostics 1.1 3. Task Manager in Vista and Server 2008 • Make dump at the right time – Hang: catch it in the act – Crash: JIT or post mortem About classes, methods and objects Some part of managed memory Method tables (MT) Managed heap Method descriptors (MD) 03ef00cd: Customer 03ef015d 0840a2ac Message: "…" InnerException: 03ef0264 0842001a 0841fda3 Name: "Jansen" Message: "…" 04ab0712: Exception City: Hedel InnerException: null 04ab0720 084200fd 084200fd Name: "Stelt" Name: "Stelt" 04ab085f City: Ridderkerk City: Ridderkerk Loaded managed code Objects on heap Memory in Windows Virtual address space (Max: 2GB on 32 bit, 4GB on 64 bit) Working set CLR Managed Native code memory memory (recently used memory pages) Private bytes (not shared) Demo: memory leak • Caused by rooted references Scenario 3: "Hanging" application • Symptoms – Slow response times – Near 100% CPU usage • Determine – Who's using up CPU cycles? Reasons for hangs High CPU hangs Low CPU hangs • Infinite loop • Deadlocks caused by • High % time in GC Lock, WaitOne or • Busy Server critical Section • Waiting for external resources Find out what's causing hang • .time for time busy and !runaway • Dump native and .NET stacks of threads – ~* kb 2000 and ~* e !clrstack – What are threads doing? • Find any deadlocks with !syncblk – SosEx might help with other blocking kinds Heaps of garbage N-core processor Large Object Heap 0 Heap #N LOH #N Heap (> 85k) … 1 2 3 5 6 4 12 7 11 8 9 10 13 14 15 16 Gen 2 Sweep Compact Mark Gen 0 Gen 1 • Generational garbage collection • Unreachable vs. rooted reference Demo: High CPU hang • Caused by spending a lot of time on garbage collection • Load test application • If I feel lucky: remote debugging Additional tools • SOSex WinDbg extensions: • MDbg (C# managed debugger) • SOSAssist • CLR Profiler • Global Flags (part of Debugging Tools) Top tips • If you do have Visual Studio 2005/2008 on production server: – You can load SOS during debugging! – Managed and unmanaged debugging • Make sure you have symbols – No hassle demystifying call stacks • Open dump files on same architecture machine Next steps… • Practice debugging • Do labs on Tess Ferrandez' blog • Don't hesitate to look at production machines • Go and see my colleagues as well • Frits Ankersmit: DSDM voor kleine projecten • Anko Duizer: Windows Azure for Architects 1 & 2 • Astrid Hackenberg: Working with Web Services in BizTalk Server • Dennis van der Stelt: If you build it, you'll ship it • Alex Thissen: Oslo overview & M language Review • Be prepared: learn your debugging stuff • Let debugging process guide you • Symbols for effective debugging • Use your instinct now and again • Don't be afraid to debug production apps • Get the debug tooling Resources • Blogs: Tess Ferrandez, Doug Stewart, Tom Christian, Carlo Cardella, Johan Straarup, Jim Robbins, Volker von Einem • WinDbg Help file • Presentations – Common ASP.NET production issues and how to troubleshoot them with WinDbg – ASP.NET Hang analysis with VSTS Test part 1 and 2 – Hardcore production debugging of .NET applications Questions? Answers! Discussion!!! Thanks for attending Happy debugging!