[K42-discussion] Questions and comments about lazyReOpen + a patch for lazyReOpen
Patrick Bozeman
PEBozeman at lbl.gov
Fri Sep 15 05:41:02 EST 2006
I have a number of questions regarding the fork/exec lazyReOen functionality
in K42 and have attached a patch for a definite bug.
Those already familiar with these code paths may want to jump directly to
the Questions and Comments section at the end. I go into detail about
the background of storing state in xhandles and into detail about how
lazyGiveAccess and lazyReOpen work in order to provide people other than the
authors of these modules some context for my questions. (And to help
page in
some info in case the original authors forgot some of this :))
*******************
Background:
*******************
------------------
State:
------------------
The following is some background on K42 client and server objects for those
on the list that are not familiar with it. I'm including this both to help
other understand some of this and to make sure that I do myself (and
hopefully
someone will correct me if I get it wrong.)
When a client opens a file object (files, directories, streams, etc) state
is maintained in several places. The server side object (ServerFile,
DirLinuxFS, etc) maintains state for managing the server object itself, e.g.
ServerFiles know what DirLinuxFS objects point to them, DirLinuxFS objects
have a list of currently instantiated children, etc.
In addition, the server object can optionally associate client specific
data with an open request. As part of an open, the server object calls
giveAccessByServer to tell the COBJ layer that it wants to allow a new
client to access the server object. giveAccessByServer will in turn call
giveAccessSetClientData, which the server object has the option to override.
The server can then associate an 8byte token (an XHandle) with
the ObjectHandle returned to the client. The COBJ layer passes the
ObjectHandle (and related XHandle) to the ServerObject during all future
PPC calls made by clients against that ObjectHandle. Typically, the
ServerObject allocates a client specific data structure and stores a pointer
to it in the xhandle. (I assume, but haven't verified, that there are
mechanisms to keep clients from diddling with the xhandle values.)
Finally, state is stored in the client object, including the stub object
(and ObjectHandle) used to contact the corresponding server object. The
client object for regular files, FileLinuxFile, stores its open flags in
addition to the stub to the ServerFile.
------------------
File Descriptors:
------------------
Every user level process object contains a static FD class that manages the
state of file descriptors. It maps fds to their associated client side file
objects and manages several bitmaps representing fd state. It has a
bitmap for
free files, files that need to be re-opened post fork (see below) and files
that need to be closed on exec.
*******************
Pre-Fork
*******************
--------------------------------------------
Pre-fork Client Side
--------------------------------------------
When a process forks, new copies of the client state maintained in the
server need to be created. K42 does this lazily to avoid the case
where the new client (i.e. the child) doesn't access the ObjectHandles
granted to its parent.
Prior to forking, FD::ForkAll is called to prepare all open files for post
fork lazyReOpening. FD::ForkAll calls clientFileRef->lazyGiveAccess for
every
open file, disassociates the fd with the current client file reference and
marks the fd as being ready for lazyReOpening. This FD state is propegated
across the fork since it resides within the process itself. However, the
state associated with the ObjectHandle and client file reference works
differently.
The pattern for propagating the ObjectHandle and client file state seems to
be consistent across all the file objects and works as follows. The
associated
client and server share a common lazyState transfer data structure
defined by
the client. When the FD object calls clientFileRef->lazyGiveAccess, the
client
fills in part of the structure and then calls
serverFileRef->_lazyGiveAccess(fd, partiallyFilledXferData...).
--------------------------------------------
Pre-fork Server Side
--------------------------------------------
What the server object does is somewhat interesting. During
lazyGiveAccess it
creates a new ObjectHandle on behalf of the **kernel** via
giveAccessByServer.
The server allocates a new client state structure, populates it, and
stores a pointer to it in the ObjectHandle created for the kernel.
Server side
client state is then added to the partially filled transfer data buffer
passed
in by the client file object.
The server then calls
process->_lazyGiveAcecss(fd, xferData, kernelObjectHandleForTheFile).
The kernel process server then tucks away the xferData and the
ObjectHandle in
a data structure accessed by the fd, i.e. in an instance of the
LazyState class.
--------------------------------------------
Post-fork Client and Server
--------------------------------------------
When the child (or the parent for that matter) accesses any of the file
descriptors opened prior to the fork, FD detects the readyForLazy bit and
makes a call to its process to reopen the file. The resulting call graph
looks as follows. (Note, the call graph is highly abbreviated and doesn't
contain the cobj layer or many of the intermediate client/stub calls)
_FD::locked_lazyGetFD(fd)
char data[512];
process->_lazyReOpen(__in fd, __out &oh, __out &type,
__outbuf data, __out &DataLen)
// This call is routed to the kernel process object e.g.
// ProcessReplicated or ProcessShared
LazyState.lazyReOpen(pid, fd, &type, &oh, char *data, dataLen);
// Lookup stub and marshaled data by fd
// copy the data passed prefork via lazyGiveAccess into the
// data buf. XXX: there is a bug here, see the
questions/comments
memcpy(data, preForkData, preForkDataLen);
// Call the server object's lazy reopen
stub._lazyReOpen(oh, pid, match, nomatch, data, dataLen);
// The stub is to the server object being reopened,
// e.g. a ServerFile or a DirLinuxFS.
// server file based example...
ServerFile::_lazyReOpen(__out &oh,
__in toProcID,
__in AccessRights match,
__in AccessRights nomatch,
__inoutbuf(datalen:datalen:datalen)
char *data,
__inout datalen,
__XHANDLE xhandle)
// The data passed to this reopen call is the data that
// was stored in LazyState. The server object is
// supposed to create and return a new oh for forked
// client. Note however that the server state was
// passed in two different forms, i.e. it was passed
// in the marshaled data buffer, but it is also
// contained in the Xhandle for the stub's oh.
// Create a new oh for the forked client
giveAccessByServer(oh, toProcID, match, nomatch);
// cobj layer ends up calling....
// (I don't really need to show this part
// for the purpose of this discussion, but it may
// be useful for someone to help understand
// the entire process.
ServerFile::giveAccessSetClientData(&oh,
toProcID,
match,
nomatch,
type)
// Allocate client data state
ClientData *clientData = new ClientData();
// Tell the cobj layer to grant access
// to the forked client, i.e. create
// an ObjectHandle and associate the
// newly allocated clientData structure
// with its XHandle.
giveAccessInternal(oh, toProcID,
match, nomatch, type,
(uval)clientData);
// (back in SeverFile::_lazyReopen)
// We now need to set the client state on the oh
returned
// from giveAccessByServer
ClientData *clnt =
(XHandleTrans::GetClientData(oh.xhandle)
clnt->state = un-marshaled state from *data
// Unwind all the way back to FD, returning the new oh to the forked
// client and the data buf so that the forked client can recreate
// the client side state
// Recreate client object and state
switch (type) {
...
// case statement per object type, e.g.
case FileLinux_FILE:
rc = FileLinuxFile::LazyReOpen(fileRef, oh, buf, dataLen);
break;
}
// store fileref in fd table and clear its close on exec bit
return fileRef;
************************
Questions and comments
************************
1) Why pass around and store the marshaled data buffer for the lazy state?
In lazyGiveAccess(), the server object creates an object handle to store in
the kernel on behalf of the soon-to-be forked client. The server then
associates the client state with this oh in its un-marshaled form.
Passing the data in a marshaled format on top of this seems redundant.
2) Is there a race in the placement of the FD::ForkAll call? FD::ForkAll is
called inside kitch-linux/lib/emu/fork.C prior to calling
ProgExec::ForkProcess.
However, what happens if there is a thread switch between the ForkAll and
the spot inside ForkProcess where the process's local scheduler is disabled?
If another thread in the process performs file IO, it seems that the
lazy bits
inside FD will be lost. It seems that the same thing could occur even while
ForkAll is running. Should ForkAll be moved between Scheduler::Disable
and Scheduler::DisabledScheduleFunction(ForkWorker...)?
(Is this even possible? ForkAll is going to be making calls
like DREF(fileRef)->lazyGiveAccess(..). Would these calls be serviceable
while the local scheduler is disabled?)
3) Is the placement of FD::ForkAll really what was intended? The state for
FD's is lazily reopened in the child, but it also causes the parent to go
through the lazy reopen process as well. Was this the intent? Should the
call to FD::ForkAll be even farther along in the process than I was
asking in
question 2? i.e. should it be after the ForAddressSpace call in the
ForkWorker?
4) lazyReOpen can lead to memory corruption if the marshaled data passed
to LazyState during lazyGiveAccess exceeds 512 bytes. ProcessServer and
the base kernel Process object both declare their _lazyReOpen calls to have
a databuf of __outbuf(dataLen:512) and they don't pass the size of this
buffer to LazyState. LazyState then has no way to know if it is going to
overflow the buffer or not. I experienced this in one of my tests when
I forked while holding a directory open. (I have attached a patch for
this.)
5) Where is the marshaled client data deallocated?
It looks like it should be deallocated in the destructor for
StubFileLinuxServerHolder, but it isn't. Is the deallocation via a subtle
interaction that I don't see?
6) The reference counting of the lazy state objects has a race. When
decRefCount returns 0, the caller is supposed to delete it. Here is the
code:
uval LazyState::StubFileLinuxServerHolder::decRefCount() {
lock.acquire();
refcount--;
lock.release();
return refcount;
}
In practice, I don't think it will be triggered because a higher level lock
is being held on the LazyState object itself, but say the refcount = 2 and
two different threads enter this routine at the same time (which I don't
think
is possible just based on the semantics of the objects), then they can
be scheduled such that they will both return 0 and a double delete will be
performed. So, if there is indeed a real reason to be performing this lock,
the state of refcount needs to be saved in a local variable prior to
releasing the lock. If the lock isn't needed, it should probably just
be removed.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lazy-memcpy.patch
Type: text/x-patch
Size: 3387 bytes
Desc: not available
Url : http://ozlabs.org/pipermail/k42-discussion/attachments/20060914/a4b4d7cf/attachment.bin
More information about the K42-discussion
mailing list