Mod N Reading System

November 17th, 2009

ModerN Reading System

Gradually I perfected my cooperative multireading technique by combining modular arithmetic with software data structures and algorithms like sets, circular buffers, priority queues and round-robin scheduling. It sounds complicated but in reality the technique is very simple and suited well to everyone who wants to learn everything at once and doesn’t like traditional a book after a book method. All books we want to read are organized in sets (here I give my own arrangements as an example):

  • Commuting sets
  • Home reading set
  • Background office reading sets
  • Lunch reading set

Every set is organized as a circular buffer (mod N). Some buffers are optimized to avoid heavy load while commuting. For example, my commuting set is split into two buffers: one is at home and another is in the office. When I leave to the office I take 2 books from the top of the example queue I have currently at home:

When I arrive to the office I put them at the bottom of the corresponding office book set. When I leave for my home I take 2 books from the top of the office queue and when I arrive at home I put them at the bottom of the depicted queue above. Thus I manage to read 4 different books every day during commuting. Sometimes I don’t have a place to sit on the train or just stand waiting for its arrival. For such cases I have a separate queue of 16 Routledge books (The Basics series). They are small and I read only one of them every day. In total this amounts to 5 different books a day and I read 4 - 12 pages from each. For each commuting direction I have 3 books (2 + 1).

Next I have semi-fixed set of books for lunch reading, usually 5 or 6 of them. I read 6 - 12 pages from each. These books are organized as a priority queue where books with more pages have higher priority. If 2 or 3 books are on the same topic they are put into a circular buffer to read one per day. In addition, I put a few magazines I’m subscribed to in a cyclic buffer too.

In addition to this, I read only one book at the time at home from cover to cover (usually in Russian). At home I mostly write books (instead of reading).

In the office I have different sets for background reading (instead of cigarette breaks I had before I quit smoking). This set of sets is organized as a priority queue with every subset having a circular structure as well if it has more than one book. One long term set with higher priority is The CRC Encyclopedia of Mathematics. Other books I read in the office include software engineering titles and for them I publish notes on this blog.

It can be boring sometimes to read the same 1,000 page books for long periods of time so I also introduce an element of randomness by injecting some recently purchased book or a book from the pool of old unread books.

It is very scalable even if you have only a few hours to read per day. Most important, it also gives a certain satisfactory feeling of having started reading all books you accumulated and provides cross-book idea fertilization and better knowledge acquisition by repetition.

Now I apply the same reading system to my renewed study of foreign languages. Currently it is German where I have 10 basic language level books arranged in a circular buffer.

Another thing to keep in mind is that you need to have a goal: why you read all these books.

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 16-November-09

November 16th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

ChangeServiceConfig2 (p. 292) - http://msdn.microsoft.com/en-us/library/ms681988(VS.85).aspx

sc qprivs <service name> (p. 293) - example for Terminal Service:

C:\Users\Administrator>sc qprivs TermService
[SC] QueryServiceConfig2 SUCCESS

SERVICE_NAME: TermService
        PRIVILEGES       : SeAssignPrimaryTokenPrivilege
                         : SeAuditPrivilege
                         : SeChangeNotifyPrivilege
                         : SeCreateGlobalPrivilege
                         : SeImpersonatePrivilege
                         : SeIncreaseQuotaPrivilege

Union of privileges for svchost.exe (p. 294)

Service SID (restricted and unrestricted) (p. 295)

process - window station - desktop - windows (p. 297) - an entity relationship diagram on slide 14 (Intro: Windows) in my past Selected Citrix Tools presentation: http://www.dumpanalysis.org/CitrixTools/Selected%20Citrix%20Troubleshooting%20Tools.htm

Hung non-interactive services waiting for user input (p. 298) - this partially inspired Message Box crash dump analysis pattern: http://www.dumpanalysis.org/blog/index.php/2008/02/19/crash-dump-analysis-patterns-part-51/

SERVICE_INTERACTIVE_PROCESS Type modifier only for local system accounts (p. 298)

Shatter attacks by window messages (p. 299)

Interactive Services Detection (UI0Detect) service (p. 299)

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 09-November-09

November 9th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

SApp - SCP - SCM (p. 282)

HKLM\S\CCS\Ss\<>\ErrorControl = 3, SERVICE_ERROR_CRITICAL (p. 284) - can be used to force BSOD if service startup fails for postmortem memory dump analysis of the failure

HKLM\S\CCS\Ss\<>\PreshutdownTimeout (p. 286)

HKLM\S\CCS\Ss\<>\RequiredPrivileges (p. 286)

Service threads (p. 287) - some typical thread stack traces can be seen in this case study that also show that service main thread calls control handler functions: http://www.dumpanalysis.org/blog/index.php/2007/10/01/windows-service-crash-dumps-on-vista/

Service accounts (p. 288) - attached WinDbg will not download symbols from MS symbol server unless Run as Administrator

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 29-October-09

October 29th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

!reg hivelist -> viewlist -> db (pp. 274 - 275)

Registry cell index mapping as directory:table:offset (p. 275)

Registry key handle -> kcb (pp. 276 - 278)

Hive sync every 5 seconds, *.log{1|2} (pp. 278 - 279)

Registry filtering altitudes (p. 280)

Internal registry optimizations (pp. 280 - 281) - good implementation case study if you need to devise your own database. I perhaps borrow some ideas for the next version of PDBFinder.

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 28-October-09

October 28th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

TxR -> CLFS + \System32\Config\Txr (pp. 260 - 261)

Precedence of a non-transactional writer (p. 261)

Read-commit vs. predictable-read (p. 261)

Process Monitor internals (p. 262) - rather short section but inspired yet another DebugWare pattern: http://www.dumpanalysis.org/blog/index.php/2009/10/28/debugware-patterns-part-11-2/

Process Monitor troubleshooting techniques (pp. 264 - 265) - PM log is a kind of a software trace so the following growing list of patterns may be useful to keep in mind: http://www.dumpanalysis.org/blog/index.php/trace-analysis-patterns/

Surviving logoff (persistent processes) (pp. 265 - 266)

Service profiles are stored in \ServiceProfiles\{Local|Network}Service\Ntuser.dat (p. 267)

HKLM\S\CCS\C\hivelist shows hive to file name mapping (p. 267)

x64 paged pol for registry - x86 mapped views (p. 268)

System and Software hives support values of >1Mb (p. 269)

Bin - block - cell - cell index (pp. 270 - 272)

Disk Probe, dskprobe.exe (pp. 271 - 272)

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 22-October-09

October 22nd, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

REG_NONE and REG_QWORD (p. 251) - the former should have a purpose as a name switch

REG_LINK (pp. 251 - 252)

HKU\.DEFAULT as a local system profile (p. 253)

\Users location can be changed in HKLM\So\M\WNT\CV\ProfileList\ProfilesDirectory (p. 254)

BCDEdit is for HKLM\BCD, how to enable /DEBUG (pp. 255 - 257) - I also had to add more permissions to Administrators for Elements key to be able to add modifications. Before editing:

C:\Users\Administrator>bcdedit

Windows Boot Manager
--------------------
identifier              {bootmgr}
device                  partition=C:
description             Windows Boot Manager
locale                  en-US
inherit                 {globalsettings}
default                 {current}
displayorder            {current}
toolsdisplayorder       {memdiag}
timeout                 30
resume                  No

Windows Boot Loader
-------------------
identifier              {current}

After editing:

C:\Users\Administrator>bcdedit

Windows Boot Manager
--------------------
identifier              {bootmgr}
device                  partition=C:
description             Windows Boot Manager
locale                  en-US
inherit                 {globalsettings}
default                 {current}
displayorder            {current}
toolsdisplayorder       {memdiag}
timeout                 30
resume                  No

Windows Boot Loader
-------------------
identifier              {current}
device                  partition=C:
path                    \Windows\system32\winload.exe
description             Microsoft Windows Server 2008
locale                  en-US
inherit                 {bootloadersettings}
osdevice                partition=C:
systemroot              \Windows
resumeobject            {cc03280e-0762-11de-b63a-af7e963a0201}
nx                      OptOut
debug                   Yes

Pdh.dll (p. 260) - Typical stack trace in TS environments:

[...]
winsta!WinStationQueryInformationW+0x2a
perfts!CollectTSObjectData+0x12f
advapi32!QueryExtensibleData+0x617
advapi32!PerfRegQueryValue+0x536
advapi32!LocalBaseRegQueryValue+0x306
advapi32!RegQueryValueExW+0x96
pdh!GetSystemPerfData+0x83
pdh!GetQueryPerfData+0x7f
pdh!PdhiCollectQueryData+0x40
pdh!PdhCollectQueryData+0x42
[...]

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 15-October-09

October 15th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

TxF <-> KTM <-> TxR (pp. 240 - 241) - how to use files transactionally (CreateFileTransacted): http://msdn.microsoft.com/en-us/library/aa365008(VS.85).aspx

hotpatching is actually 7 bytes: 2 from mov edi, edi and 5 from the previous function (p. 243)  - impressive combination of near and far jumps

KPP (PatchGuard) (pp. 244 - 246)

DKOM, direct kernel object modification (p. 245) - looks like subclassing in kernel

Bugcheck 109 - CRITICAL_STRUCTURE_CORRUPTION (p. 245) - used to be not very frequent but increased in frequency since the time of this post: http://www.dumpanalysis.org/blog/index.php/2008/03/12/bug-check-frequencies/

Enhanced process notifications since Vista SP1 to block process launch (p. 246)

Two driver signing policies (KMCS and PnP) (p. 247)

Protected Media Path (p. 247) - more information is here: http://msdn.microsoft.com/en-us/library/aa376846(VS.85).aspx

Per-page image authentication (p. 247)

- Dmitry Vostokov @ SoftwareGeneralist.com -

Computational Collectives

October 8th, 2009

This is the notion of a collective applied to the realm of computation where a set of computation processes (and possibly hardware, people and other entities, forming a hybrid entity) share the common goal and use interface bonds (relations). Sometimes certain research needs to be done to identify hidden relations to draw or to rethink boundaries between a collective and its environment, as in a simple and low-level example of coupled processes or wait chains. Note that this is not the same and it is not used in the same sense as a computational collective intelligence although certain computational collectives can give that impression.

See also: interface relations

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 05-October-09

October 7th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

.local file (p. 226) - found this explanation: http://msdn.microsoft.com/en-us/library/aa375142(VS.85).aspx

bound import table, forwarder entries (pp. 226 - 227)

TLS initializers as possible causes of DLL load failures (p. 228)

Hyper-V architectural stack (p. 229) - running processes inside W2K8 Hyper-V host running additional instance of W2K8:

         063c vmwp.exe         13755 (     55020 Kb)
         0750 vmconnect.exe    13445 (     53780 Kb)
         0aa4 mmc.exe          13184 (     52736 Kb)
         0714 vmms.exe         12228 (     48912 Kb)
         011c svchost.exe       7930 (     31720 Kb)
         09c0 explorer.exe      4603 (     18412 Kb)
         0340 svchost.exe       4525 (     18100 Kb)
         0ba4 WmiPrvSE.exe      2485 (      9940 Kb)
         04b0 svchost.exe       2321 (      9284 Kb)
         03e4 svchost.exe       2222 (      8888 Kb)
         0158 SLsvc.exe         2220 (      8880 Kb)
         01ec svchost.exe       1858 (      7432 Kb)
         0238 svchost.exe       1753 (      7012 Kb)
         05a8 spoolsv.exe       1698 (      6792 Kb)
         029c lsass.exe         1630 (      6520 Kb)
         0530 taskeng.exe       1275 (      5100 Kb)
         0768 svchost.exe       1120 (      4480 Kb)
         06cc WmiPrvSE.exe      1018 (      4072 Kb)
         03cc msdtc.exe         1007 (      4028 Kb)
         0384 svchost.exe        979 (      3916 Kb)
         0944 taskeng.exe        953 (      3812 Kb)
         0104 svchost.exe        910 (      3640 Kb)
         02a8 lsm.exe            877 (      3508 Kb)
         0708 svchost.exe        820 (      3280 Kb)
         0290 services.exe       802 (      3208 Kb)
         0348 svchost.exe        696 (      2784 Kb)
         0628 svchost.exe        680 (      2720 Kb)
         0004 System             673 (      2692 Kb)
         0214 csrss.exe          579 (      2316 Kb)
         0240 csrss.exe          531 (      2124 Kb)
         0274 winlogon.exe       520 (      2080 Kb)
         0980 dwm.exe            482 (      1928 Kb)
         0248 wininit.exe        472 (      1888 Kb)
         0634 svchost.exe        347 (      1388 Kb)
         068c svchost.exe        306 (      1224 Kb)
         01d0 smss.exe           117 (       468 Kb)
         0b7c wlrmdr.exe           0 (         0 Kb)

optimization enlightments (p. 229)

VM Infrastructure driver vid.sys and hypervisor API winhv.sys (p. 232)

Vdev virtual devices (p. 234) - here is the list of loaded modules in the virtualization worker process vmwp.exe:

0: kd> lm1m
odbcint
kernel32
USER32
ntdll
PSAPI
vmwp
dssenh
vmbusvdev
rdp4vs
vmicshutdown
vmicvss
vmickvpexchange
synthnic
vmictimesync
vmicheartbeat
azroles
cryptnet
ODBC32
fastprox
wbemsvc
msxml3
SensApi
wbemprox
vsconfig
framedynos
wbemcomn
vmprox
vmwpctrl
vid_7fefb420000
vmbuspipe
COMCTL32_7fefbb80000
napinsp
winrnr
rasadhlp
WINTRUST
XmlLite
comctl32
NLAapi
wshtcpip
NTMARTA
GPAPI
rsaenh
schannel
mswsock
wship6
kerberos
bcrypt
ncrypt
dhcpcsvc6
dhcpcsvc
tspkg
wevtapi
slc
credssp
IPHLPAPI
CRYPT32
MPR
NTDSAPI
MSASN1
SAMLIB
DNSAPI
cryptdll
NETAPI32
WINNSI
AUTHZ
Secur32
USERENV
GDI32
MSCTF
SETUPAPI
ole32
SHLWAPI
msvcrt
ADVAPI32
CLBCatQ
USP10
RPCRT4
IMM32
LPK
imagehlp
WS2_32
OLEAUT32
COMDLG32
NSI
SHELL32
WLDAP32

0: kd> lmv m vmicheartbeat
start             end                 module name
000007fe`fa540000 000007fe`fa57c000   vmicheartbeat   (deferred)            
    Image path: C:\Windows\System32\vmicheartbeat.dll
    Image name: vmicheartbeat.dll
    Timestamp:        Wed Jun 11 15:27:36 2008 (484FE0D8)
    CheckSum:         00033B40
    ImageSize:        0003C000
    File version:     6.0.6001.18016
    Product version:  6.0.6001.18016
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     vmicheartbeat
    OriginalFilename: vmicheartbeat.dll
    ProductVersion:   6.0.6001.18016
    FileVersion:      6.0.6001.18016 (vistasp1_gdr_vm_rtm.080611-0040)
    FileDescription:  Virtual Machine Integration Service Heartbeat Vdev
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

emulated and synthetic (enlightened I/O) devices (pp. 234 - 237):

0: kd> lmv m synthnic
start             end                 module name
000007fe`fa340000 000007fe`fa38a000   synthnic   (deferred)            
    Image path: C:\Windows\System32\synthnic.dll
    Image name: synthnic.dll
    Timestamp:        Wed Jun 11 15:27:30 2008 (484FE0D2)
    CheckSum:         0003D7AA
    ImageSize:        0004A000
    File version:     6.0.6001.18016
    Product version:  6.0.6001.18016
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     SynthNic.dll
    OriginalFilename: SynthNic.dll
    ProductVersion:   6.0.6001.18016
    FileVersion:      6.0.6001.18016 (vistasp1_gdr_vm_rtm.080611-0040)
    FileDescription:  Microsoft Synthetic Network Card
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

VSC vs. VSP, VSC driver re-routes requests to VMBus (PnP enum for synthetic devices) (pp. 236 - 237)

Virtualized processors, virtual APIC, dynamic processor addition (pp. 237 - 238)

GVA, guest virtual address space -> GPA, guest physical address space -> SPA, system physical address space (pp. 238 - 239) - need to add this to my debugging dictionary: http://www.dumpanalysis.org/blog/index.php/category/dictionary-of-debugging/

SPT. shadow page tables, for direct GVA -> SPA

- Dmitry Vostokov @ SoftwareGeneralist.com -

Software Generalist Worldview

September 28th, 2009

I was recently revisiting my old post about model-based definition of software defects in relation to their forthcoming classification. When thinking I recalled a three worlds diagram in Roger Penrose’s The Road to Reality book depicting the Platonic mathematical, the physical and the mental and came up with Software Generalist three worlds: World, Models and Software:

  

- Dmitry Vostokov @ SoftwareGeneralist.com -