Software Generalist

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

VTune profiling (pp. 82 - 83) - there are also Visual Studio profiler http://msdn.microsoft.com/en-us/magazine/cc337887.aspx, Visual Studio Team System profiling tools and Windows kernel profiler kernrate

TBB, OpenMP, MPI and Google MapReduce (p. 87) - I have a book about Intel TBB *Threading Building Blocks”, should read or browse it soon

Mimic the scheduling (p. 90) - One technique I found useful is when you suspect two separate racing functions we can patch function prolog with a return to see if the problem is gone

Using log file to localize race conditions (p. 91 - 93) - In Citrix we use CDF tracing based on ETW (http://msdn.microsoft.com/en-us/library/aa468736.aspx) as application-, service- and system-wide logging mechanism. It records TID and PID for every trace statement. All debug statements are actually written in a production environment if we choose to record certain or all categories of them:
http://support.citrix.com/article/ctx117426

stderr should be unbuffered (p. 91)

debugger tracepoints (p. 93) - stop, record data, resume - similar techniques exist in WinDbg/CDB/KD family of debuggers

Advanced Windows Debugging by M. Hewardt and D. Pravat:

Concurrent Programming on Windows by J. Duffy:

Software Factories by J. Greenfield, et. al.:

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

Didn’t have time today to read these 3 books because of the work on my next book ”Windows Debugging: Practical Foundations”

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

How Profilers Work section (pp. 73 - 74) - good addition to How Debugger Work books. I think detouring can also used for profiling in real-time, like famous MOV EDI, EDI in Windows

2 core profiler areas: collecting and presenting data (p. 73)

profiler sampling (interrupt and collect stack traces) and instrumentation data collection approaches (p. 73) - The problem with sampling is that most threads wait during their life time, so it is more useful only for CPU-intensive apps

flat profile and call graph presentation techniques (p. 74)

Callgrind as part of Valgrind (pp. 81 - 82)

Advanced Windows Debugging by M. Hewardt and D. Pravat:

I resume this book reading from page 328 after very long pause.

!token -n (pp. 329 - 330) - command shows no impersonation if it fails to get one

SD pointer in an object header is a pseudo-pointer with the first 3 bits used for a different purpose (p. 331)

ISC/ASC client impersonation sequence for remote authentication (p. 335) - possible UML sequence diagram for my book Windows Internals Distilled

The importance of impersonation level in a token (pp. 337 - 338)

Concurrent Programming on Windows by J. Duffy:

Software Factories by J. Greenfield, et. al.:

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

Didn’t have time today to read these 3 books because of the work on my next book “Windows Debugging: Practical Foundations”

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Performance bugs include slow and fast program runs (p. 63) - Do I need to include them in my collection of patterns?

Upfront performance analysis before debugging performance problems (p. 64) - can be delegated to troubleshooting engineer

Measuring time (pp. 64 - 65) - Complex products sometimes write time for every trace statement like Citrix Common Diagnostic Facility (CDF) technology based on Event Tracing for Windows (ETW). Also memory dumps usually have various times saved inside. 

Varying CPU clock speed (p. 67)

Disconnect computer from network for clean testing to remove possible network influence on execution time (p. 68)

Concurrent Programming on Windows by J. Duffy:

Spin waiting causes CPU spikes and blocks threads (pp. 63 - 64) - See my real life patterns Dispatch Level Spin and Affine Thread http://www.dumpanalysis.org/blog/index.php/2008/01/25/crash-dump-analysis-patterns-part-44/ and http://www.dumpanalysis.org/blog/index.php/2008/06/27/crash-dump-analysis-patterns-part-68/

Busy-waiting for fine-grained concurrency to avoid scheduling costs (p. 65) 

Continuation passing style (CPS) as alternative to waiting - packaging the rest of computation into some code that we pass to an interface that runs the continuation when the wait condition is satisfied (p. 66)

(Event, predicate) -> race condition -> monitor (critical region, conditional variable) (p. 68)

Structured parallelism - coordination abstraction (cobegin, forall, futures) (p. 70)

Message passing (encapsulation of inter-thread states) (p. 71)

Shared memory as an optimization to message passing (p. 71)

Coordination and Concurrency Runtime - first-class message passing (MS Robotics SDK) (p. 73)

Software Factories by J. Greenfield, et. al.:

Didn’t have time to read today

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

Didn’t have time to read today

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Data structure instrumentation (pp. 54-55) - I devised a visual method called Colometric Computer Memory Dating http://www.dumpanalysis.org/blog/index.php/2008/04/16/computer-colometric-memory-dating-part-1/

Too much detailed data from memory profiler (p. 55) - This is true indeed in the case of UMDH or user mode stack trace db on Windows

Memory usage increase might also be because of not cleared buffers or containers after functional iterations (p. 61) - I might need to investigate this on Windows to add to my pattern collection

Concurrent Programming on Windows by J. Duffy:

Peterson’s algorithm simplifies DD’s (pp. 53 - 54)

Lamport’s Bakery algorithm, thread failure in critical region  doesn’t destroy liveness (pp. 54 - 55)

Most modern synchronization primitives are built using CAS, compare and swap or interlocked instructions  (p. 57)

Software concurrency algorithms may not work because of compiler optimizations and hardware read/write reordering (pp. 58 - 60)

Causal thread dependence and state dependence (p. 62)

Software Factories by J. Greenfield, et. al.:

Tools lag platforms (p. 4) - True also for T&D tools

Increased stakeholder expectations (pp. 4 - 5) - Does it affect customer expectations?  Same limits of troubleshooters pool. They are overwhelmed too, like developers

Agile development doesn’t scale (p. 5)

The need to encapsulate knowledge as reusable assets (p. 6) - The same need for T&D. Encapsulation of T&D patterns in T&D tools

Web services as progress in packaging and interfacing (p. 6) - Are Web services the nexte step for T&D tools? There are some companies that do live debugging remotely

Aspect-oriented methods to incorporate contextual into functional (p. 6) - Can we use this for T&D tools?  I need to elaborate on this: eliminate the need to rewrite T&D tools 

Software development challenges (p. 7) - Great parallels (word-to-word) with software troubleshooting challenges 

An idea of a fictitious company to show adoption of methods (p. 7) - This might be useful idea to borrow for a fictitious software support centre

Batch era (pp. 9 - 11) - Are we still in batch era of software troubleshooting?

Multiple islands of data (p. 10) - Good metaphor

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

Didn’t have time to read today

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Strategy of memory profiling (p. 45) - good summary of a general approach broken down in several steps.

The need to have an estimated and / or expected memory usage value (p. 45) - customers always ask me the question whether the certain amount of VM consuption is normal

Phases of execution and measurements at phase boundaries (p. 47)

Memory growth as a function of input data (linear, logarithmic, quadratic, …, exponential) (p. 48)

Stop points (pp. 48 - 49) - I also recommend to dump process memory to a memory dump file at regular intervals.

top tool from Cygwin (p. 49)

Using Task Manager (pp. 50 - 51) - Actually there are several memory columns for processes and it is important to choose the most appropriate. For example, Commit Size also includes memory allocated in page file as well. The following picture shows various columns in Vista: 

Deallocation not releasing memory to OS (pp. 51 - 52) - On Windows it might not decrease the size of VM but might decrease the size of committed memory and reduce page file usage. I need to test this: writing this down in my hardback Idea: Blogger’s notebook. Also on Windows there can be several separate process heaps with their own lifetime semantics. 

Memory profilers: Massif, AQtime and mpatrol (pp. 53 - 54) - On Windows you can use Gflags and select user mode stack trace database and then use WinDbg: http://www.dumpanalysis.org/blog/index.php/2007/08/06/crash-dump-analysis-patterns-part-20a/

Concurrent Programming on Windows by J. Duffy:

Association of a critical region with static lexical scope (p. 43)

Orphaned lock (p. 45) - I have seen it many times. Looks like good candidate to be included in my crash dump analysis patterns.

Coarse- vs. fine-grained locks, false sharing (pp. 46 - 47)

CAS instructions (compare and swap) (p. 48)

Strict alternation algorithm - critical region implementation (pp. 49 - 50)

Dekker’s and Dijkstra’s algorithm - critical region implementation (pp. 50 - 53)

Out-of-order execution breaks DD’s algorithm (p. 53)

Software Factories by J. Greenfield, et. al.:

Problem: small market for vertical focus and domain knowledge concentration in end-user organizations (p. xviii)

Ideal product built by an individual is an exception (pp. xviii - xix) - I think this depends on the project size an complexity.

Tools help with mechanistic tasks (p. xix)

Generative Programming book (p.  xxiii) - I need to finish reading it, I started 5 years ago…

Tools automate and help to focus on creativity (p. xxvii) - T&D tools help in focusing on creative part of T&D  

Architecturally driven and contextually aware tools for specific domains (p. xxix)

Software Engineering Foundations: A Software Science Perspective, by Y. Wang:

I bought this book last year and started reading now to get the most possible broad overview of software and related disciplines.

Historical programming-language-centered approach to SE (p. xxxvi) - I propose a memory-centered approach in my forthcoming books about Memoretics and Memiotics including a foundational philosophy called Memoidealism where Software is an emergent and predictable artifact.

Architecture of the book (p. xxxix) - concept and example of a book architecture (especially for a foundational book where SE can be templetized to a different discipline)

Broad multidisciplinary foundations of software rather than just computer science (p. xlix)

The author plans to apply metaphors from other disciplines - very much like I started to do with philosophy and physics. I also coined the term metaphorical bijection to describe two-way application of metaphors, for example, from physics to software and from software to physics. 

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Memory debugger vs. general debugger (p. 33)

The missing code as a bug (p. 34) 

Valgrind examples (pp. 36 - 39) - Gflags, Application and Driver Verifiers are analogs of memory debuggers in Windows world. Also Visual Studio and MFC runtime have capabilities to detect various classes of memory errors

Increased resource consumption when using memory debuggers (p. 42) - this is true when full page is enabled on Windows or user mode stack trace DB

Notion of non-standard memory handlers to curcumvent bugs in dynamic memory usage (pp. 42 - 43) - skip free/delete and release all heap at once at the end - I used this

Concurrent Programming on Windows by J. Duffy:

Data publication / privatization via transfer of ownership (p. 33)

Assume that received references via parameters or function arguments are shared by default (p. 34)

Shallow vs. deep immutability (p. 34)

SSA - Static Single Assignment (p. 34) - uses of C++ const for SSA enforcement (p. 38)

Dynamic Single Assignment Verification (p. 38)

Distinction between data vs. control synchronization (pp. 38 - 39)

Critical region as serialized sequence of operations (p. 40)

Code- vs. data-centric critical regions (p. 42)

Semaphore as a generalization of a critical region (p. 42) that allows N threads to enter that region (counting semaphore)

Mutex as critical region with N == 1 - binary semaphore (p. 42) 

Software Factories by J. Greenfield, et. al.:

This book I bought a few years ago and just started reading mainly because of my current work on DebugWare: The Art and Craft of Writing Troubleshooting and Debugging Tools book where I advocate a Software Tool Factory approach.

The book is written by architects of Visual Studio Team System (from backcover)

Root causes of software problems: monolithic construction, excessive generality, on-off development and process immaturity (p. xvii)

Solution: integration of languages, patterns, frameworks and tools  (p. xvii)

Broad horizontal domains (p. xvii) - troubleshooting and debugging is one of them

SF provides unique variants of an archetypical product (p. xviii) - unique special debugging tools as variants of an ultimate debugging tool

- Dmitry Vostokov @ SoftwareGeneralist.com -

The subtitle of this blog has changed from “All you need to know to become a successful software engineer” to “Connecting Software with Engineering, Science, Philosophy and Religion” to reflect the shift in content towards the broader spectrum of topics since its foundation almost a year ago. The original idea to provide short independent survey articles about various aspects of software engineering now becomes one of categories of posts on this blog.

- Dmitry Vostokov @ SoftwareGeneralist.com -

This is a new book planned by OpenTask with the some preliminary details:

The Variety of Software: The Richness of Computation (ISBN: 978-1906717544)

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Seems the book doesn’t mention Debugging Tools for Windows and WinDbg at all but this is circumvented by the authors’  blog featuring a link and related books. The book uses GDB (primary) and Visual Studio debugger (secondary) for examples.

Larger object files with debug information - Visual C++ can store debugging symbols in separate PDB files

Basic GDB commands (pp. 25-30)

Debugger expression evaluation feature (p. 30) - caught my attention and got a vision of using it for tracking pre-, post-conditions and method invariants

Concurrent Programming on Windows by J. Duffy:

Serialized threads (p. 25)

Data race (p. 26)

Stale read and unrepeatable read (p. 28). The latter happens when a second thread reads too early.

Serializability is not sufficient for correctness. Serialization orders must be legal - need for critical regions (p. 30)

Linearization points - when a batch atomic updates are visible to other threads (p. 30)

CLR AppDomains as intra-process isolation (p. 32)

Cache as an isolated state of master copy (p. 32)

Isolation by convention (p. 32) - commonplace, reminds me guidelines for ownership of COM interfaces 

- Dmitry Vostokov @ SoftwareGeneralist.com -

Comments in italics are mine and express my own views, thoughts and opinions

Developer’s Guide to Debugging by M. Wloka, et al.:

Test-driving debugging tools (p. 14) - I’ve seen many times indeed when engineers advised on the phone to collect information using certain tools but didn’t try those tools themselves…

Golden output (p. 16) - an output to compare to, reference output

The notion of common bugs that are easily and readily reproducible with test cases (p. 18)

Idealist heisenbug (p. 19) - a bug stemming from invalid optimization code

Secret bugs (p. 20) - I’ve seen that many times when customers do not send crash dumps due to security reasons. If this is because they cannot inspect binary contents for potential information leaks I devised a way to instruct a debugger to output textual log files with information relevant for at least problem identification. 

Concurrent Programming on Windows by J. Duffy:

Handcrafted semi transactional system, importance of state in-the-small and shared memory dependance (p. 13) 

State management: isolation | immutability | synchronization (p. 14)  

Control vs. data synchronization (p. 14)

What is a state? (p. 15) - I define a state as an memory snapshot at a given time where memory is taken in a broad sense including processor(s) state as well

The transitive nature of state sharing, data privatization and publication (p. 15) - From Windows kernel perspective everything is shared

A reference to escape analysis techniques and ownership types (p. 19)

Critical region as a simulation of atomic r/w not possible in hardware (p. 21)

- Dmitry Vostokov @ SoftwareGeneralist.com -