<br><font size=2 face="sans-serif">Most of what you say about IPC call
and return is correct. The exception-level semantics of the two services
are almost identical, and our original implementation had just one system
call that was used for both paths. Calls and returns were distinguished
by values the senders placed in a register that the kernel passed along.</font>
<br>
<br><font size=2 face="sans-serif">We first split the calls just to save
a few cycles. The senders no longer had to set the "direction"
register, the receivers didn't have to do conditional branches, and the
register itself became available for data transfer. Splitting the
paths also enabled one tiny optimization in the kernel. "Wild-carding"
the destination VP is the default for calls, but makes no sense for returns.
Calls should go to a "local" dispatcher in the target address
space if possible, but a return has to go back to the calling dispatcher,
even if there's a local dispatcher in the calling address space. The
kernel now wild-cards the VP unconditionally on calls and insists on a
specific target on returns. With this change, the services are no
longer semantically identical.</font>
<br>
<br><font size=2 face="sans-serif">The implementations of the two paths
diverged further when we started worrying about resource accounting. As
much as possible, we want to treat a called server as executing in the
resource domain of the caller. We have a notion of the "current
resource domain", which is not necessarily the resource domain of
the currently executing dispatcher. On a call, we switch to a new
dispatcher but the current resource domain doesn't change. If the
callee makes a call of its own, the current resource domain still doesn't
change. If services complete without interruption, the returns transfer
control back to the calling dispatchers and the current resource domain
is charged for the whole chain of calls. But now we have to be careful
when a server, executing in client A's resource domain, returns not from
client A's request but from an earlier request from client B. We
can't have client B executing in client A's resource domain. We therefore
maintain a small stack of dispatcher id's at exception level. The
id's are those of the dispatchers involved in a chain of calls, all made
on behalf of the current resource domain. When we get a return, we
check the destination against the top of the stack. If it matches,
we "fast path" the return without switching resource domains.
If it doesn't match, we blow away the whole stack and return via
a "slow" path that switches resource domains and may context
switch to another dispatcher altogether if the new resource domain isn't
currently entitled to run. The exception-level dispatcher stack is
really just a cache or a set of hints, in that it can be blown away at
any time without affecting correct operation. We blow it away whenever
any sort of unusual event (page fault, quantum expiration, etc.) occurs.
Also, because it's just a cache it doesn't have to be large enough
to hold all possible call chains. If the stack overflows, we just
blow it away and start over. The only consequence is that earlier
calls in the current chain will be unnecessarily slow-pathed, but if the
call chain is that deep it probably doesn't matter.</font>
<br>
<br><font size=2 face="sans-serif">The call stack lets us avoid most hashtable
lookups in the IPC return path: we simply save the calling dispatcher's
pointer along with its id on the stack. We've essentially optimized
for what we hope is the typical case: a client executing in its own resource
domain calls a server, which may call other servers, and everybody returns
back to the client without any sort of interruption. In that case,
all the returns match the call stack, the stack itself tells us the destination
dispatchers for the returns without any hashtable lookups, and the control
transfers all happen without any scheduling decisions.</font>
<br>
<br><font size=2 face="sans-serif">To answer your final question, you could
implement a dispatcher that uses only calls to transfer control to other
dispatchers and never uses returns, but you'd be getting some fairly strange
resource accounting and you'd be missing out on the return-path optimizations.
It should work, but I wouldn't recommend it.</font>
<br>
<br><font size=2 face="sans-serif">- Bryan</font>
<br>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>Andrew Baumann <andrewb@inf.ethz.ch></b>
</font>
<br><font size=1 face="sans-serif">Sent by: k42-discussion-bounces+rosnbrg=us.ibm.com@ozlabs.org</font>
<p><font size=1 face="sans-serif">11/07/2007 05:29 AM</font>
<table border>
<tr valign=top>
<td bgcolor=white>
<div align=center><font size=1 face="sans-serif">Please respond to<br>
Discussion about K42 <k42-discussion@ozlabs.org></font></div></table>
<br>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">"k42discuss" <k42-discussion@ozlabs.org></font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">[K42-discussion] difference between
IPC_CALL and IPC_RETURN syscalls</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><tt><font size=2>Hi all,<br>
<br>
In the context of another OS design, I'm thinking about IPC in a split
<br>
user/kernel threading model rather like what K42 uses, and am trying to
<br>
understand K42's IPC mechanism (ie. the low-level synchronous call/return
<br>
path on top of which PPC is implemented).<br>
<br>
My question is: from the kernel's perspective (or in K42 terminology, perhaps,
<br>
from exception-level's perspective), what is the difference between an
<br>
IPC_CALL syscall and an IPC_RETURN syscall? IIRC, any dispatcher can do
a <br>
call or a return to any other dispatcher (the kernel just delivers the
<br>
sender's ID), and the dispatcher has to check that a call or return is
valid <br>
anyway. Furthermore, from the kernel's perspective the success or failure
of <br>
a call or a return depends on whether the target is disabled, and not whether
<br>
it is in a call phase waiting for a return or anything so high-level.<br>
<br>
Are these statements correct? If so, why have the two different primitives?
Is <br>
it just an optimisation for user code (I can see how it helps the dispatchers
<br>
to know what is a return), or is there an important difference to the kernel?
<br>
If I implemented my own dispatcher that only ever used IPC_CALL to transfer
<br>
control between dispatchers, and never did a return, would I confuse the
<br>
kernel?<br>
<br>
Thanks,<br>
Andrew<br>
_______________________________________________<br>
K42-discussion mailing list<br>
K42-discussion@ozlabs.org<br>
https://ozlabs.org/mailman/listinfo/k42-discussion<br>
</font></tt>
<br>