[K42-discussion] Jonathan's big commit
Orran Y Krieger
okrieg at us.ibm.com
Mon Dec 26 13:40:22 EST 2005
Here are some notes from Jonathan on his big commit on the scalability
work from his thesis.
-- Orran
----- Forwarded by Orran Y Krieger/Watson/IBM on 12/25/2005 09:39 PM -----
Jonathan Appavoo/Watson/IBM at IBMUS
12/23/2005 05:57 PM
To
k42-watson at kitch0.watson.ibm.com
cc
Subject
commit
Can someone forward this to the discussion list. I was having a hard
time posting. Sorry thanks.
Hi Everyone,
I have commited my merged changes from my phd and demo (percs initial
demo) trees.
I recommend backing up files so that you can back out the change and test
in case you find
something wrong.
Here is a copy of a README and commit log. Of which a copies along with a
diff ( changes.txt) can
be found in contrib/jonathan/phd.
-- README --
This documents the merge of changes in trees I had for
my Phd and the percs initial demo into the mainline repository.
This was a complex merge and there are chances that things got messed
up. I have run regress, sdet and ssh.
Please note there was a performance drop due to this commit.
Some drop might be justified as there are a few fixes that
may have a negative performance impact but are correct :-(.
Such as the use of CompareAndStoreSynced instead of
CompareAndStoreVolatile
See comment in commit.txt. However the drop should really be
investigated more throughly.
There are also various things that can be pursued to
tune performance: Turn on MultiRepRegions, use sane
initial sizes for DhashTable in FCMs, play with DHash Counting,
etc.
Changes.txt: is a diff of the changes
commit.txt: is a file that documents the changes as per what I put into
the commit log.
The directory memclone-hw is what I had in my kitchlinux/users directory.
It was a user I used to run the automated memclone tests for my phd and
the percs demo.
--- commit.txt ---
Files ADDED :
=================
contrib/jappavoo/phd/README
File that documents how these changes came into being and what the changes
to the tree are and what files in this tree of sub-dirs are:
---
contrib/jappavoo/phd/changes.txt
output of cvs -q diff prior to commit
---
contrib/jappavoo/phd/commit.txt
commit log (this file)
---
contrib/jappavoo/phd/newmemclone.C
The version of memclone used to gather my thesis and percs results
---
contrib/jappavoo/phd/memclone-hw/CommandFile
contrib/jappavoo/phd/memclone-hw/CommandFile.full
contrib/jappavoo/phd/memclone-hw/CommandFile.k0.perProcRegion
contrib/jappavoo/phd/memclone-hw/CommandFile.k0.perProcRegion.noPreTouch
contrib/jappavoo/phd/memclone-hw/CommandFile.k0.singleRegion
contrib/jappavoo/phd/memclone-hw/CommandFile.k0.singleRegion.noPreTouch
contrib/jappavoo/phd/memclone-hw/CommandFile.k0.singleRegion.noPreTouch.noResize
contrib/jappavoo/phd/memclone-hw/CommandFile.k0.singleRegion.noPreTouch.noResize.big
contrib/jappavoo/phd/memclone-hw/CommandFile.k0.singleRegion.noPreTouch.noResize.big.detailed
contrib/jappavoo/phd/memclone-hw/CommandFile.qdo
contrib/jappavoo/phd/memclone-hw/CommandFile.test
contrib/jappavoo/phd/memclone-hw/Makefile
contrib/jappavoo/phd/memclone-hw/sysinit
This is a copy of the user directory that was used to gather the automated
results. The sysinit reboot the machine and ran a single line ofcommands
in the CommandFile and then reboot to proceed to the next line.
The various CommandFile where for various runs.
---
os/kernel/mem/FRComputationDefault.C
os/kernel/mem/FRComputationDefault.H
Two new FR's (FRComputationPart.[Ch] already in the tree,
which were used for the PERCS demo to allow the user to pick a specific
fcm to back a test region of user memory.
---
os/kernel/mem/RegionDefaultMultiRep.C
os/kernel/mem/RegionDefaultMultiRep.H
The new multi-rep region objects discussed in my thesis.
---
Files MODIFIED:
=================
/u/kitchawa/cvsroot/kitch-core/Status
---
/u/kitchawa/cvsroot/kitch-linux/users/spec/k42.filelist.Img.4.cpio
add oisServer to k42 files... helpful for debugging a user crash
during sdet. After crash occurs you can login start oisServer and
then explore processes and address spaces.
---
/u/kitchawa/cvsroot/kitch-core/lib/libc/cobj/CObjRootMultiRep.C
/u/kitchawa/cvsroot/kitch-core/lib/libc/cobj/CObjRootMultiRep.H
/u/kitchawa/cvsroot/kitch-core/lib/libc/cobj/missHandlingtest.C
Updated misshandling code and destruction code to match what
was in my phd and demo tree. Think I got the merge right
but to be honest not completely sure. Objects with cluster
size greater than one need to be tested (particularly
destruction and resource reclaimation).
---
/u/kitchawa/cvsroot/kitch-core/lib/libc/misc/DHashTable.H
Commited what I think is a critical bug FIX which never made it
to the mainline tree.
Curcial syncronication operation was not using synced version but rather
volatile. This change can have a negative performance impact.
Status file gives a brief idea but requires more study.
But I believe it is right and was what was for my phd results.
But at some point more careful though should be given to this and
we should see if it is really necessary of if there is a better approach
(eg fine grain RCU).
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/init/KernelInit.C
Added printing of new control flag bit to turn on use of
multirep region for user regions.
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMCommonMultiRep.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMCommonMultiRep.H
Added defines to controll what type of storage is used for
local and global allocations (eg. LOCAL_STRICT vs GLOBAL_PADDED)
Also added support for turning of counting in the hash tables
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMCommonMultiRepRoot.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMCommonMultiRepRoot.H
Added defines to allow use of Dhash when it is configured to
use the RCU based request counter
Added defines to controll what type of storage is used for
local and global allocations (eg. LOCAL_STRICT vs GLOBAL_PADDED)
Also added support for turning of counting in the hash tables
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMDefaultMultiRep.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMDefaultMultiRep.H
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMDefaultMultiRepRoot.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMDefaultMultiRepRoot.H
Addes support for representatives being automatically sized at
the time of there creation rather than starting at zero.
This is probably turned off right now. But could be an important
turning factor. As is initial size of the master tables.
Also includes same changes as FCMCommonMultiRep*.[CH]
---
u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMFile.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMFile.H
Static FCMFile::CreateFixedSize(...)
Added a kludgy interface to create and FCM of a specific size.
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMPartitioned.H
Made compatible with new function interface to page descriptors
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMPrimitiveKernelMultiRep.H
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FCMPrimitiveKernelMultiRepRoot.H
Added defines to control use of LOCAL vs GLOBAL memory
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/FRComputation.C
Added construction of new Meta objects for FRComputationDefault
FRComputationPart. Might want to turn these off.
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/Makefile
Added new files
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/PageList.H
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/PageSet.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/PageSet.H
Added support for sizeing to an initial size. Used by
FCMFile::CreateFixedSize
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/RegionDefault.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/mem/RegionDefault.H
Losts of lines changesd to allow multi-rep to derive from
default. Mainly to make use of request count swingable.
This was a complex merge and I am not sure I got it all
right. And there may be performance impacts here.
Also added support for kludgy test interfaces for region
creation.
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/proc/ProcessReplicated.C
/u/kitchawa/cvsroot/kitch-core/os/kernel/proc/ProcessShared.C
Changed printf that occurs on a bad address fault... to include
vp, this, root, ref. All can be used with ois/kore to debug
the address space.
---
/u/kitchawa/cvsroot/kitch-core/os/kernel/sys/KernelInfo.H
Added bit to turn on DISTRIBUTED_USER_REGIONS,
---
/u/kitchawa/cvsroot/kitch-core/tools/misc/kore/lib/tcl/gdb.tcl
/u/kitchawa/cvsroot/kitch-core/tools/misc/kore/lib/tcl/kore.tcl
Misc changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://ozlabs.org/pipermail/k42-discussion/attachments/20051225/9ec92943/attachment.htm
More information about the K42-discussion
mailing list