From andrewb at inf.ethz.ch Wed Nov 7 21:29:50 2007 From: andrewb at inf.ethz.ch (Andrew Baumann) Date: Wed, 7 Nov 2007 12:29:50 +0200 Subject: [K42-discussion] difference between IPC_CALL and IPC_RETURN syscalls Message-ID: <200711071129.50810.andrewb@inf.ethz.ch> Hi all, In the context of another OS design, I'm thinking about IPC in a split user/kernel threading model rather like what K42 uses, and am trying to understand K42's IPC mechanism (ie. the low-level synchronous call/return path on top of which PPC is implemented). My question is: from the kernel's perspective (or in K42 terminology, perhaps, from exception-level's perspective), what is the difference between an IPC_CALL syscall and an IPC_RETURN syscall? IIRC, any dispatcher can do a call or a return to any other dispatcher (the kernel just delivers the sender's ID), and the dispatcher has to check that a call or return is valid anyway. Furthermore, from the kernel's perspective the success or failure of a call or a return depends on whether the target is disabled, and not whether it is in a call phase waiting for a return or anything so high-level. Are these statements correct? If so, why have the two different primitives? Is it just an optimisation for user code (I can see how it helps the dispatchers to know what is a return), or is there an important difference to the kernel? If I implemented my own dispatcher that only ever used IPC_CALL to transfer control between dispatchers, and never did a return, would I confuse the kernel? Thanks, Andrew From rosnbrg at us.ibm.com Thu Nov 8 01:38:41 2007 From: rosnbrg at us.ibm.com (Bryan S Rosenburg) Date: Wed, 7 Nov 2007 09:38:41 -0500 Subject: [K42-discussion] difference between IPC_CALL and IPC_RETURN syscalls In-Reply-To: <200711071129.50810.andrewb@inf.ethz.ch> Message-ID: Most of what you say about IPC call and return is correct. The exception-level semantics of the two services are almost identical, and our original implementation had just one system call that was used for both paths. Calls and returns were distinguished by values the senders placed in a register that the kernel passed along. We first split the calls just to save a few cycles. The senders no longer had to set the "direction" register, the receivers didn't have to do conditional branches, and the register itself became available for data transfer. Splitting the paths also enabled one tiny optimization in the kernel. "Wild-carding" the destination VP is the default for calls, but makes no sense for returns. Calls should go to a "local" dispatcher in the target address space if possible, but a return has to go back to the calling dispatcher, even if there's a local dispatcher in the calling address space. The kernel now wild-cards the VP unconditionally on calls and insists on a specific target on returns. With this change, the services are no longer semantically identical. The implementations of the two paths diverged further when we started worrying about resource accounting. As much as possible, we want to treat a called server as executing in the resource domain of the caller. We have a notion of the "current resource domain", which is not necessarily the resource domain of the currently executing dispatcher. On a call, we switch to a new dispatcher but the current resource domain doesn't change. If the callee makes a call of its own, the current resource domain still doesn't change. If services complete without interruption, the returns transfer control back to the calling dispatchers and the current resource domain is charged for the whole chain of calls. But now we have to be careful when a server, executing in client A's resource domain, returns not from client A's request but from an earlier request from client B. We can't have client B executing in client A's resource domain. We therefore maintain a small stack of dispatcher id's at exception level. The id's are those of the dispatchers involved in a chain of calls, all made on behalf of the current resource domain. When we get a return, we check the destination against the top of the stack. If it matches, we "fast path" the return without switching resource domains. If it doesn't match, we blow away the whole stack and return via a "slow" path that switches resource domains and may context switch to another dispatcher altogether if the new resource domain isn't currently entitled to run. The exception-level dispatcher stack is really just a cache or a set of hints, in that it can be blown away at any time without affecting correct operation. We blow it away whenever any sort of unusual event (page fault, quantum expiration, etc.) occurs. Also, because it's just a cache it doesn't have to be large enough to hold all possible call chains. If the stack overflows, we just blow it away and start over. The only consequence is that earlier calls in the current chain will be unnecessarily slow-pathed, but if the call chain is that deep it probably doesn't matter. The call stack lets us avoid most hashtable lookups in the IPC return path: we simply save the calling dispatcher's pointer along with its id on the stack. We've essentially optimized for what we hope is the typical case: a client executing in its own resource domain calls a server, which may call other servers, and everybody returns back to the client without any sort of interruption. In that case, all the returns match the call stack, the stack itself tells us the destination dispatchers for the returns without any hashtable lookups, and the control transfers all happen without any scheduling decisions. To answer your final question, you could implement a dispatcher that uses only calls to transfer control to other dispatchers and never uses returns, but you'd be getting some fairly strange resource accounting and you'd be missing out on the return-path optimizations. It should work, but I wouldn't recommend it. - Bryan Andrew Baumann Sent by: k42-discussion-bounces+rosnbrg=us.ibm.com at ozlabs.org 11/07/2007 05:29 AM Please respond to Discussion about K42 To "k42discuss" cc Subject [K42-discussion] difference between IPC_CALL and IPC_RETURN syscalls Hi all, In the context of another OS design, I'm thinking about IPC in a split user/kernel threading model rather like what K42 uses, and am trying to understand K42's IPC mechanism (ie. the low-level synchronous call/return path on top of which PPC is implemented). My question is: from the kernel's perspective (or in K42 terminology, perhaps, from exception-level's perspective), what is the difference between an IPC_CALL syscall and an IPC_RETURN syscall? IIRC, any dispatcher can do a call or a return to any other dispatcher (the kernel just delivers the sender's ID), and the dispatcher has to check that a call or return is valid anyway. Furthermore, from the kernel's perspective the success or failure of a call or a return depends on whether the target is disabled, and not whether it is in a call phase waiting for a return or anything so high-level. Are these statements correct? If so, why have the two different primitives? Is it just an optimisation for user code (I can see how it helps the dispatchers to know what is a return), or is there an important difference to the kernel? If I implemented my own dispatcher that only ever used IPC_CALL to transfer control between dispatchers, and never did a return, would I confuse the kernel? Thanks, Andrew _______________________________________________ K42-discussion mailing list K42-discussion at ozlabs.org https://ozlabs.org/mailman/listinfo/k42-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: http://ozlabs.org/pipermail/k42-discussion/attachments/20071107/18a3be38/attachment.htm