Software Generalist

Resuming reading notebook and plan to fill it at least 3-4 times per week

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Implicit or hidden function calls in high level languages, step-into command, GDB step (pp. 133 - 135) - t command in WinDbg

step-out command to bypass stepping through implicit functions, GDB finish command (p. 136) - gu command in WinDbg

temporary breakpoint in GDB, tbreak (p. 136) - also .step_filter in WinDbg to create the list of function to skip

conditional breakpoints and breakpoint commands (pp. 138 -139) - bp and ba commands in WinDbg have an additional parameter: command string; Also bp “J” and bp “.if” conditional variants in WinDbg

print strcmp(…) and $$0 in GDB (p. 139) - I think in WinDbg this can be done via .call and $callret

static constructors and initializers (pp. 140 - 143)

technique: slowing the program to have time to attach the debugger (p. 142)

Advanced Windows Debugging by M. Hewardt and D. Pravat:

.dbgdbg command (p. 368)

technique: using local kernel debugger when user debugger extension fails (p, 375)

Concurrent Programming on Windows by J. Duffy:

A thread may already begin or even finish by the time CreateThread returns (p. 89)

thread pseudo-handle from GetCurrentThread == -2 (p. 94)

thread handle from thread id - OpenThread (p. 95)

use CRT function only in threads created by _beginthread(ex) (p. 96)

_beginthread closes handle automatically by return of start routine (p. 97)

Software Factories by J. Greenfield, et. al.:

EAI to solve the problem of data flow and transformation between applications on different servers (p. 21)

Building applications from business process perspective (p. 22)

DCOM and CORBA are tightly coupled to component implementations (p. 23)

service-oriented architecture (SOA): loosely coupled coarse grained components, message interaction, protocol sequencing defined by contracts and negotiated constraints (SLA, service level agreements) (p. 25)

BPMS (business process management system) as logical analog to DBMS (pp. 25 - 26)

pi-calculus as theoretical platform for BPMS (p. 25)

CRM subprocesses: campaigns, sales force and leads, customer management, customer service and self-service (pp. 28 - 29)

Portals as encapsulation of user interaction (p. 30) - an idea of tool portals for troubleshooting; Also each troubleshooting domain is implemented as a troubleshooting process or service that encapsulates troubleshooting logic and required data access; troubleshooting facades working with older troubleshooting tools

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

Didn’t have time today - resume reading tomorrow

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

An application as a set of processes (p. 103)

strace (Linux) and truss (Solaris) to trace OS calls (pp. 104 - 106) - On Windows we can trace API while debugging using WinDbg extensions: http://www.dumpanalysis.org/blog/index.php/2007/01/03/tracing-win32-api-while-debugging-a-process/. Process Monitor can also be used to trace API subsets like File and Registry I/O.

Compiler bugs (pp. 106 - 107)

Debugger and compiler incompatibilities (p. 107)

Link-time bugs (Chapter 9)

Missing symbols (pp. 112 - 113) - On Windows search strategies like Explorer Search and dumpbin can be useful for dynamic linking errors

Advanced Windows Debugging by M. Hewardt and D. Pravat:

The importance of lazy initialization and associated impersonation bugs (pp. 347 - 354)

Concurrent Programming on Windows by J. Duffy:

.NET programs are multithreaded from the start (gc runs on a separate thread) (p. 79) - We can see this in WinDbg when attaching it to a .NET app container

Thread as a virtual processor metaphor (p. 80)

Thread as an execution context (p. 81)

Non-local transfer of control; context switch, exception handling, hardware interrupt (borrows kernel stack), DPC and APC (pp. 84 - 85)

DPC and APC run in the context of the current thread (p. 85)

The differences of CLR (managed) threads: just additional CLR information is stored on per OS thread basis (p. 86)

Explicit threading: explicit thread creation (p. 87) with thread pool alternative (p.88)

Software Factories by J. Greenfield, et. al.:

Failure of enterprise data modeling during terminal era of 70s - 80s (p. 13)

Bifurcation of of software into personal and enterprise with the advent of PCs (pp. 15 - 16)

The notion of the phenomenon of cheaper products eventually displacing market leaders after improvement over time (p. 16)

Thick client: business rules on both tiers (p. 17)

The growth of packaged application industry and outsourcing after client-server era (pp. 18 - 21)

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

SE principles to counter SE constraints (cognitive, organizational and resource) (p. 2)

Software as a unique abstract structure (p. 7)

The notion of the intelligent behavioural metaphor (p. 7)

Inductive math-based methodology of theoretical software problems vs. deductive experiment-driven for empirical problems: both must be used for SE problems (1st Law of SE)  (p.

The need for mathematical modeling of software system architecture and software behaviours (p. 9)

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Performance bugs include slow and fast program runs (p. 63) - Do I need to include them in my collection of patterns?

Upfront performance analysis before debugging performance problems (p. 64) - can be delegated to troubleshooting engineer

Measuring time (pp. 64 - 65) - Complex products sometimes write time for every trace statement like Citrix Common Diagnostic Facility (CDF) technology based on Event Tracing for Windows (ETW). Also memory dumps usually have various times saved inside. 

Varying CPU clock speed (p. 67)

Disconnect computer from network for clean testing to remove possible network influence on execution time (p. 68)

Concurrent Programming on Windows by J. Duffy:

Spin waiting causes CPU spikes and blocks threads (pp. 63 - 64) - See my real life patterns Dispatch Level Spin and Affine Thread http://www.dumpanalysis.org/blog/index.php/2008/01/25/crash-dump-analysis-patterns-part-44/ and http://www.dumpanalysis.org/blog/index.php/2008/06/27/crash-dump-analysis-patterns-part-68/

Busy-waiting for fine-grained concurrency to avoid scheduling costs (p. 65) 

Continuation passing style (CPS) as alternative to waiting - packaging the rest of computation into some code that we pass to an interface that runs the continuation when the wait condition is satisfied (p. 66)

(Event, predicate) -> race condition -> monitor (critical region, conditional variable) (p. 68)

Structured parallelism - coordination abstraction (cobegin, forall, futures) (p. 70)

Message passing (encapsulation of inter-thread states) (p. 71)

Shared memory as an optimization to message passing (p. 71)

Coordination and Concurrency Runtime - first-class message passing (MS Robotics SDK) (p. 73)

Software Factories by J. Greenfield, et. al.:

Didn’t have time to read today

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

Didn’t have time to read today

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Data structure instrumentation (pp. 54-55) - I devised a visual method called Colometric Computer Memory Dating http://www.dumpanalysis.org/blog/index.php/2008/04/16/computer-colometric-memory-dating-part-1/

Too much detailed data from memory profiler (p. 55) - This is true indeed in the case of UMDH or user mode stack trace db on Windows

Memory usage increase might also be because of not cleared buffers or containers after functional iterations (p. 61) - I might need to investigate this on Windows to add to my pattern collection

Concurrent Programming on Windows by J. Duffy:

Peterson’s algorithm simplifies DD’s (pp. 53 - 54)

Lamport’s Bakery algorithm, thread failure in critical region  doesn’t destroy liveness (pp. 54 - 55)

Most modern synchronization primitives are built using CAS, compare and swap or interlocked instructions  (p. 57)

Software concurrency algorithms may not work because of compiler optimizations and hardware read/write reordering (pp. 58 - 60)

Causal thread dependence and state dependence (p. 62)

Software Factories by J. Greenfield, et. al.:

Tools lag platforms (p. 4) - True also for T&D tools

Increased stakeholder expectations (pp. 4 - 5) - Does it affect customer expectations?  Same limits of troubleshooters pool. They are overwhelmed too, like developers

Agile development doesn’t scale (p. 5)

The need to encapsulate knowledge as reusable assets (p. 6) - The same need for T&D. Encapsulation of T&D patterns in T&D tools

Web services as progress in packaging and interfacing (p. 6) - Are Web services the nexte step for T&D tools? There are some companies that do live debugging remotely

Aspect-oriented methods to incorporate contextual into functional (p. 6) - Can we use this for T&D tools?  I need to elaborate on this: eliminate the need to rewrite T&D tools 

Software development challenges (p. 7) - Great parallels (word-to-word) with software troubleshooting challenges 

An idea of a fictitious company to show adoption of methods (p. 7) - This might be useful idea to borrow for a fictitious software support centre

Batch era (pp. 9 - 11) - Are we still in batch era of software troubleshooting?

Multiple islands of data (p. 10) - Good metaphor

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

Didn’t have time to read today

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Strategy of memory profiling (p. 45) - good summary of a general approach broken down in several steps.

The need to have an estimated and / or expected memory usage value (p. 45) - customers always ask me the question whether the certain amount of VM consuption is normal

Phases of execution and measurements at phase boundaries (p. 47)

Memory growth as a function of input data (linear, logarithmic, quadratic, …, exponential) (p. 48)

Stop points (pp. 48 - 49) - I also recommend to dump process memory to a memory dump file at regular intervals.

top tool from Cygwin (p. 49)

Using Task Manager (pp. 50 - 51) - Actually there are several memory columns for processes and it is important to choose the most appropriate. For example, Commit Size also includes memory allocated in page file as well. The following picture shows various columns in Vista: 

Deallocation not releasing memory to OS (pp. 51 - 52) - On Windows it might not decrease the size of VM but might decrease the size of committed memory and reduce page file usage. I need to test this: writing this down in my hardback Idea: Blogger’s notebook. Also on Windows there can be several separate process heaps with their own lifetime semantics. 

Memory profilers: Massif, AQtime and mpatrol (pp. 53 - 54) - On Windows you can use Gflags and select user mode stack trace database and then use WinDbg: http://www.dumpanalysis.org/blog/index.php/2007/08/06/crash-dump-analysis-patterns-part-20a/

Concurrent Programming on Windows by J. Duffy:

Association of a critical region with static lexical scope (p. 43)

Orphaned lock (p. 45) - I have seen it many times. Looks like good candidate to be included in my crash dump analysis patterns.

Coarse- vs. fine-grained locks, false sharing (pp. 46 - 47)

CAS instructions (compare and swap) (p. 48)

Strict alternation algorithm - critical region implementation (pp. 49 - 50)

Dekker’s and Dijkstra’s algorithm - critical region implementation (pp. 50 - 53)

Out-of-order execution breaks DD’s algorithm (p. 53)

Software Factories by J. Greenfield, et. al.:

Problem: small market for vertical focus and domain knowledge concentration in end-user organizations (p. xviii)

Ideal product built by an individual is an exception (pp. xviii - xix) - I think this depends on the project size an complexity.

Tools help with mechanistic tasks (p. xix)

Generative Programming book (p.  xxiii) - I need to finish reading it, I started 5 years ago…

Tools automate and help to focus on creativity (p. xxvii) - T&D tools help in focusing on creative part of T&D  

Architecturally driven and contextually aware tools for specific domains (p. xxix)

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

I bought this book last year and started reading now to get the most possible broad overview of software and related disciplines.

Historical programming-language-centered approach to SE (p. xxxvi) - I propose a memory-centered approach in my forthcoming books about Memoretics and Memiotics including a foundational philosophy called Memoidealism where Software is an emergent and predictable artifact.

Architecture of the book (p. xxxix) - concept and example of a book architecture (especially for a foundational book where SE can be templetized to a different discipline)

Broad multidisciplinary foundations of software rather than just computer science (p. xlix)

The author plans to apply metaphors from other disciplines - very much like I started to do with philosophy and physics. I also coined the term metaphorical bijection to describe two-way application of metaphors, for example, from physics to software and from software to physics. 

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Memory debugger vs. general debugger (p. 33)

The missing code as a bug (p. 34) 

Valgrind examples (pp. 36 - 39) - Gflags, Application and Driver Verifiers are analogs of memory debuggers in Windows world. Also Visual Studio and MFC runtime have capabilities to detect various classes of memory errors

Increased resource consumption when using memory debuggers (p. 42) - this is true when full page is enabled on Windows or user mode stack trace DB

Notion of non-standard memory handlers to curcumvent bugs in dynamic memory usage (pp. 42 - 43) - skip free/delete and release all heap at once at the end - I used this

Concurrent Programming on Windows by J. Duffy:

Data publication / privatization via transfer of ownership (p. 33)

Assume that received references via parameters or function arguments are shared by default (p. 34)

Shallow vs. deep immutability (p. 34)

SSA - Static Single Assignment (p. 34) - uses of C++ const for SSA enforcement (p. 38)

Dynamic Single Assignment Verification (p. 38)

Distinction between data vs. control synchronization (pp. 38 - 39)

Critical region as serialized sequence of operations (p. 40)

Code- vs. data-centric critical regions (p. 42)

Semaphore as a generalization of a critical region (p. 42) that allows N threads to enter that region (counting semaphore)

Mutex as critical region with N == 1 - binary semaphore (p. 42) 

Software Factories by J. Greenfield, et. al.:

This book I bought a few years ago and just started reading mainly because of my current work on DebugWare: The Art and Craft of Writing Troubleshooting and Debugging Tools book where I advocate a Software Tool Factory approach.

The book is written by architects of Visual Studio Team System (from backcover)

Root causes of software problems: monolithic construction, excessive generality, on-off development and process immaturity (p. xvii)

Solution: integration of languages, patterns, frameworks and tools  (p. xvii)

Broad horizontal domains (p. xvii) - troubleshooting and debugging is one of them

SF provides unique variants of an archetypical product (p. xviii) - unique special debugging tools as variants of an ultimate debugging tool

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Seems the book doesn’t mention Debugging Tools for Windows and WinDbg at all but this is circumvented by the authors’  blog featuring a link and related books. The book uses GDB (primary) and Visual Studio debugger (secondary) for examples.

Larger object files with debug information - Visual C++ can store debugging symbols in separate PDB files

Basic GDB commands (pp. 25-30)

Debugger expression evaluation feature (p. 30) - caught my attention and got a vision of using it for tracking pre-, post-conditions and method invariants

Concurrent Programming on Windows by J. Duffy:

Serialized threads (p. 25)

Data race (p. 26)

Stale read and unrepeatable read (p. 28). The latter happens when a second thread reads too early.

Serializability is not sufficient for correctness. Serialization orders must be legal - need for critical regions (p. 30)

Linearization points - when a batch atomic updates are visible to other threads (p. 30)

CLR AppDomains as intra-process isolation (p. 32)

Cache as an isolated state of master copy (p. 32)

Isolation by convention (p. 32) - commonplace, reminds me guidelines for ownership of COM interfaces 

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Test-driving debugging tools (p. 14) - I’ve seen many times indeed when engineers advised on the phone to collect information using certain tools but didn’t try those tools themselves…

Golden output (p. 16) - an output to compare to, reference output

The notion of common bugs that are easily and readily reproducible with test cases (p. 18)

Idealist heisenbug (p. 19) - a bug stemming from invalid optimization code

Secret bugs (p. 20) - I’ve seen that many times when customers do not send crash dumps due to security reasons. If this is because they cannot inspect binary contents for potential information leaks I devised a way to instruct a debugger to output textual log files with information relevant for at least problem identification. 

Concurrent Programming on Windows by J. Duffy:

Handcrafted semi transactional system, importance of state in-the-small and shared memory dependance (p. 13) 

State management: isolation | immutability | synchronization (p. 14)  

Control vs. data synchronization (p. 14)

What is a state? (p. 15) - I define a state as an memory snapshot at a given time where memory is taken in a broad sense including processor(s) state as well

The transitive nature of state sharing, data privatization and publication (p. 15) - From Windows kernel perspective everything is shared

A reference to escape analysis techniques and ownership types (p. 19)

Critical region as a simulation of atomic r/w not possible in hardware (p. 21)

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

The notion of debugging opportunities - the flow of opportunities to debug from earlier macro instrumentation using source code preprocessing techniques to run-time debugging tools (p. 6)

The notion of debugging breaking personal relationships (p. v, foreword by A. Zeller)

Solution-oriented way

13 rules extend 9 rules of Agans

Rule 3: Simplify the Test Case (p. 9) - Very important in postmortem debugging when someone collects data for you

Rule 4: Read the Right Error Message (p. 9) - Good rule for postmortem debugging too

Rule 6: Separate Facts from Interpretation (p. 10) - Good one

Side-by-side debugging (pp. 11-12) - very important in real life: ETW-based CDF traces from normal and problem scenarious

Rule 8: Match the Tool to the Bug (p. 12) - psychology of debugging

Rule 10: Keep an Audit Trail (pp. 12-13) - for example, logging in WinDbg

Concurrent Programming on Windows by J. Duffy:

Why concurrency? (pp. 3-4) - one reason is missed: to understand memory dumps, traces and logs better - the view from software technical support

Natural scalability: latent -> actual concurrencly during the course of software evolution (p. 5). Some degree of inherent latent concurrencly in programming languages constrained by stylistic habits (p. 5)

“Concurrency begins with architecture.” (p. 6) - merits a new quotation

Decomposition of programs into agents (roughly: entry points interacting asycnronously with envorinment), tasks and data (pp. 6-7)

Architecturing to ease the synchronization burden (p. 7)

Natural domain parallelism (pp. 8-9)

- Dmitry Vostokov @ SoftwareGeneralist.com -