What's Happening?

Rusty's GPG Key

Friday December 19 2003

Reading through the hotplug CPU patches in detail to document them fully, I came across four bugs. Ouch.

Monday December 15 2003

Last week I finished my revision (ie. rewrite) of Rusty's Unreliable Guide To Kernel Locking, in the hope of making it a little less misleading for 2.6 kernels, at least. Spent more time on it today: lots of feedback, much from people who clearly hadn't heard of it before. Oh well, they have now.

Thursday December 11 2003

Got some out-of-the-blue mail from Justin Mason who works on Bayesian filtering for SpamAssassin. I learnt a lot from his two mails, and some more from reading the spamassassin code. In summary, though I dislike spamassassin's speed, and think the hashing approach is interesting, it's getting very refined: I'll probably stop dicking around and switch back to spamassassin.

There is, however, more room for scalability improvement on spamsum. Tridge and David Gibson have some really interesting ideas.

Thursday December 4 2003

Spent last night poring over some partial traffic dumps from what looked like an rsync exploit. Nothing conclusive, but some targeted auditing revealed a problem, and my patch this morning has been released as 2.5.7 with minor mods.

Wednesday December 3 2003

Andrew Morton dropped by: he was home in Australia visiting his parents and he and his Dad drove down for the morning. As you can imagine, little work done today, long lunch.

Sounds like Andrew will release 2.6.0 pretty much "as is" and sort through -mm for 2.6.1. I think that makes sense. People have complained about fbdev being in a broken state, and certainly my laptop doesn't suspend under -test10.

Steve Youngs tracked down why his sound modules didn't autoload in 2.6: turns out that most char devices don't have their MODULE_ALIAS_CHARDEV declarations. He's fed me one patch, and hopefully will send more.

Thursday November 27 2003

Bugged Linus again about linux.conf.au. It's getting close. Still no response, so maybe he won't be coming.

Monday November 17 2003

Broome was really nice. Got up to 43 degrees, but relaxing and quiet.

Spent a few hours on the weekend writing a new spam filter: I thought about it in Broome. It sucks, but I'm sure some more time tweaking it will make it better. I'm pretty happy with the stats-gathering part, it's the analysis part that really needs work. I don't have the statistical background to know the best way of handling the wealth of information it produces, so I'm playing.

FHS arguing continues. Well, it's better than silence.

While I was away, more bugs were found (and fixed) in the hotplug CPU code, mainly by Vatsa (Srivatsa Vaddagiri). Impressive work.

Tuesday November 4 2003

Tested the per-cpu stuff: found two compile bugs, and then two bugs in testing. Disconnecting dialup and using modem to connect, since that gives me access to inside the IBM network (there's a way using Freeswan, but I don't have a key yet).

I'm hoping that in final testing tonight, it will pass stress tests. Then I can relax in Broome (Alli and I are away for 9 days).

Tridge tested my rzip mods against an 8GB file, and found some regressions, which turned out to be bugs in handling > 600MB files. Fixed now though: I found that my version ran much faster over 800M of test data on my laptop, probably because the hash size was 32MB (the older one was 64 or 128 IIRC).

Monday November 3 2003

Finished another audit of the uses of per-cpu variables in the kernel (for the hotplug CPU patch). There weren't many missing: found a bug in net/core/flow.c, and ended up simplifying a chunk of it.

Worked a little more on rzip. Tridge and I discussed it, and it's almost at the "1.0" stage. Perhaps some more polishing needs to be done first, but it's pretty good already. Its strength is really large files which contain some duplicate strings (it uses libbz2 as a backend), but it's slow to compress (thing bzipk, not gzip). Here's the 2.5.0 kernel source, as an example:

-rw-------    1 rusty    rusty    23748963 Feb 20  2002 linux-2.5.0.tar.bz2
-rw-rw-r--    1 rusty    rusty    22080728 Nov  3 22:10 linux-2.5.0.tar.rz

Friday October 31 2003

Had DSL installed at home today. Chose Pacific Internet since they were cheapest and have Debian and kernel.org mirrors internally (ie. don't count towards bandwidth use).

Started Alli off with Quanta which, despite generally preferring GTK programs, seems to be the front-runner for this kind of thing.

Her web page, when it's up, will appear on the Ozlabs server.

Thursday October 30 2003

rzip is coming along well. Introduced a 16% speedup (and 0.1% size improvement). Tests on compression of random files with small amounts of repeated data showed that it's pretty optimal already, but there might be some small gains to be had.

Thursday October 23 2003

Thanks to everyone who sent flowers.

Monday October 20 2003

(Sadly Rusty is incommunicado so this entry is written on he & Alli's behalf)

Rusty and Alison wish to advise that the funeral for their son, Will, is to be held at 10am, Wednesday October 22nd at the Norwood Park Crematorium, Sanford Street, Mitchell ACT. Following the memorial a morning tea will be held at their home in Queanbeyan. Rusty & Alli would welcome colourful flowers.

Monday October 6 2003

Up at 4:45am to take Alli to the airport (she's heading to Townsville to visit her brother while I'm in Japan. So I was in at work at 6am: it's a public holiday but I have bugs to chase down in the new Hotplug CPU code, and other stuff to finish before I leave.

Sunday October 5 2003

I know better than to vent on a mailing list. Really. Especially the FHS list. It's a PITA when someone throws a spanner in the works when you think you're close to completion, and it's damn tempting to shoot the messenger.

Monday September 29 2003

Spent a few hours today changing the module-init-tools testsuite to run everything under valgrind: I changed something which I knew introduced a bug, so it was a good time to do it. Also, it found an out-by-one bug in the rest of the code: bonus.

Closed a couple of minor module-init-tools issues. Interestingly, wrote some new test cases for the backwards compatibility code, and discovered that "modprobe -t net" without "-l" never worked. Since there were no bug reports, I know that noone does that, so I could get rid of that part of the code (which was really ugly anyway).

Friday September 26 2003

Linus finally took a trivial patch run. Good.

Thursday September 25 2003

Since Ingo changed the migration thread startup code (and broke the hotplug CPU patch) I thought it was about time that I write some decent kernel thread helpers: the hoops you have to jump through to start and stop kernel threads are a PITA, and it effects the hotplug code quite a bit.

I've seen an earlier version which handed back a structure: I wanted the code to work with simple task_structs, so they could be used like any other threads. The result is a kthread_create interface which I like quite alot.

Wednesday September 24 2003

Jeremy Andrews of kerneltrap finished his interview with me: I think it turned out pretty well, actually. Interesting to have an interview in which column space isn't an issue: it was kind of fun.

Thursday September 11 2003

Yay! Greg Kroah-Hartmann started the "module parameters in sysfs" stuff for me. This is good, since it will encourage more people to convert to the new module parameters, which is better for everyone.

Documentation cleanups coming through: I removed modules.txt, and then realized that there were about 1000 references throughout Kconfig files. Since that cut & paste sentence is to verbose anyway, I solicited help to clean it up, and it worked. If course, Nicolas Kaiser just may not realize how big the job he started is...

Monday September 2 2003

Bitsy day. Waded through FHS bugzilla . Now need to conglomerate them all into one big patch to send out to the list.

Also waded through the linux.conf.au paper submissions. They look excellent: more tutorials would be good though, as always. Looks like I won't be able to make it, so I'm hoping the audio recording is up to the superb standard of last year's conference CDROM .

Sunday August 31 2003

Jeremy Andrews of Kernel Trap asked to interview me, and I waded through the first set of questions today. Hard going (hmm, do I really not have any hobbies?).

Friday August 29 2003

Wanted to note The Real David Gibson Home Page .

Thursday August 28 2003

Spent the day reading brighter minds re: unpinned futexes, and implementing and checking a new patch. My test case found problems, which is always good.

Main thing I got from today is that I want to send Fabrice Bellard a big Thankyou: it made tweaking the futex stuff so painless compared with booting a real machine. He is my hero.

Friday August 16 2003

Linux Beer Hike was fun, but I arrived Tuesday night, so only really got the tail end of it. Then I got sick on Friday, but I got two good days in. Still lots of things on my TODO list unfortunately.

Some more module work: released 0.9.13, so I could get onto 0.9.14. Things have been picking up as more people are playing with 2.6 kernels. I put zlib support in, mainly because a few people wanted it and it was fairly simple. I have an idea for running the whole testsuite under valgrind to uncover extra bugs (either by turning each binary into a script which runs valgrind and checks the output, or simply running the whole damn thing under valgrind). So much to do, so little time.

I was hoping to get some different coding done during my time here, like playing with Keith Packard's Cairo library, which I got a glance of at his OLS presentation .

Saturday August 10 2003

Spent a week relaxing in Portland: a nice midpoint in this trip. Alli flew in for the week. Greg Kroah-Hartmann and his family are there, as is Pat Mochel, and it's a fairly nice corner of the US this time of year. Local knowledge really helps: they recommended some excellent restaurants, and we had dinner on the last night with Pat and Chris Wright and his wife.

Skimmed my mail, there are some important TODOs for next week at the Linux Beer Hike which from previous experience is actually quite a productive time. Particularly, finishing my MODULE_VERSION implementation and getting serious about out-of-tree module building with Kai.

Monday July 28 2003

Several times on Saturday, and even today at the airport the airport someone asked me when the slides were going to be up. Here they are in Open Office and HTML forms. The HTML is basically a bunch of JPEGs (yuk), so choose the Open Office if you can.

Sunday July 27 2003

Day of relaxation. Got kicked out of the bar around 3am, slept in until 2pm. That's better. Read my mail: Linus seems serious about the freeze which is good. Debates continue about the merit of module unloading, which I think marks my final attempt to do anything about it. On to other things...

Saturday July 26 2003

Got back to my hotel room before midnight for the first time last night, only to spend two and a half hours cleaning up my keynote. When I practiced it before I left, it was 90 minutes long (without questions) so I had to make some painful cuts. Oh well.

In the end I went for almost exactly an hour, and there were no questions. I didn't wait a long time for them, because it was the end of the conference and everyone wanted to go to the pub, but it was only when noone asked questions that I realized that questions are my favorite part. Fortunate, though, because giving out the O'Reilly books took a painfully long time (my attempts to speed up the process were only a partial success: James Bottomley hit Paul McKenney in the head with his throw of one small book).

Thursday July 24 2003

Experimental technique of regulating body clock using alcohol has proven less than successful. But I persisted to the end, in the name of science.

Tuesday July 22 2003

As expected, I didn't have a great deal to contribute at the Kernel Summit. A little disappointing that some people weren't there, but there wasn't really anything major in the formal part anyway. A couple of asides about the module stuff though, but that's to be expected 8)

Sunday July 6 2003

Fabrice says he'll take my Copy-On-Write block device patches for qemu , and fix the stupid option name. And he made IDE work on 2.5 kernels (I haven't tested yet, but I'm really looking forward to it). My hero!

Thursday July 3 2003

Sent more patches to Linus: per-cpuification. We're getting there: last I measured slightly more than half the uses of smp_processor_id() were simply to get to a per-cpu variable. Once it's about 80% I'll propose a switch.

David Mosberger-Tang seemed surprised (and delighted) that Linus is taking these patches, since ia64 is the immediate benificiary.

Marcelo asked me to re-xmit the trivial stuff once he releases -pre3, which I'll do. I've settled on fortnightly for sending them to Linus, and much less frequently for him.

More FHS work today. We're close to a 2.3 draft. I'm fairly happy with the /srv proposal, but I know that some others won't be.

Wednesday July 2 2003

Got my Hungarian visa, so I'm ready for the Netfilter Summit immediately following the Linux Beer Hike in Slovakia (no tourist visa needed, according to the embassy). It's going to be a long trip, but I'm looking forward to it.

Monday June 30 2003

OLS presentation going well. Writing this has taken me longer than any previous presentation, but that's partly because I usually sit down for a day and just write it. Tentatively I'm planning on spending a solid day on it next week: you really need to do that to make sure it has good flow, and isn't a random collection of disjointed slides. This is particularly tricky because I'm not simply talking about something technical like futexes or netfilter or the module rewrite: writing something that is general and yet useful is hard. Noone wants to hear me talk for an hour on "Debug your code and be good to your mother".

The conference get together went well: the committee, and particularly Michael Davies the head organizer, have the right priorities (ie. if it's not essential, and it can't be delegated, drop it). As each previous conference has raised the bar so much, it's scary to see how much work it is to make the conference work now. Even though the conference is incredibly casual, getting the t-shirts, cds, bags, tutorial notes, dinners, food, drinks, gifts, audio streams, rooms, security, nametags, sponsors, speakers, students, hoobyists, professionals, projectors, volunteers, (seven!) minicons, promo materials, sponsor's banners, etc. organised is a major piece of work in itself, and none of these people are being paid. Not to mention the new ideas they have...

Saturday June 28 2003

Did some work on per-cpu'ing the rest of the [NR_CPUS] arrays in the core kernel. If we get to the stage where the vast majority of these vars are per-cpu, we can make smp_processor_id() a per-cpu variable, and put the per-cpu offset in the thread_info struct.

Flying to Adelaide now to do Ghosts of Conference Past: the now-traditional hazing of the new linux.conf.au committee by previous committees. I'm not so sure it's as neccessary as it was in the early days, where most of the organizers hadn't been to one before, but OTOH, Linux Australia is feeling more confident financially after four successive profitable conferences, and it's worked well in the past.

The conference itself is looking stellar: I always fret about the whole thing becoming a victim of its own success, mainly that last year was the first time we had to turn people away. That's bad, but there's a fairly broad consensus that more than 500 people doesn't really work.

Monday June 23 2003

Linus took a record four of my patches at once. Leaving Transmeta must have effected him strangely.

Writing this damn OLS keynote. I've been taking notes and gathering thoughts for a while, but it's not easy. A keynote should be fairly general and entertaining, and yet have some fundamental point. Well, actually, all my talks have a fundamental point, even if it's only "this is fun, get involved, we could use your help!".

But Rule #1 of Usenet also applies to giving talks ("Noone gives a fuck about your opinion"). So you want to give something which is original, and useful. Only problem is, I'm a kernel programmer, and if I talk about that, it's boring, and the world doesn't actually need more kernel programmers as much as we need other programmers. So you try to extract general lessons from kernel hacking, and that's where I hit problems. Things often come out as too obvious, or just kinda preachy. I've got a few good things though.

I have a section on kernel politics: explaining why it is, really. But sorting my thoughts out on the matter really did make me decide that I want to work on something else: it often isn't a very pleasant atmosphere to work in. Don't want this to sound bitter and twisted, either 8)

Wednesday April 30 2003

Found my random crashes were caused by activating the NMI oopser on the boot command line. Forgotted I'd done that, and spent lots of time back-testing various kernels before figuring it out.

Now I have my SMP crashbox back, I've been debugging the kmalloc-percpu stuff, which should allow per-cpu vars in modules. I was concerned about possibly breaking sparc64: I've sent Dave the patch and we'll see.

If that's OK I need to talk with David Mosberger-Tang about merging the ia64 implementation back with the generic one: he wants per-cpu stuff setup earlier, so I've made the setup_per_cpu_areas() call idempotent, which might be enough.

Then all I have to do is get an architecture to put the per-cpu offset in a register: with these changes even kmalloc_percpu uses the same per-cpu offset, which means that almost every speed-sensitive smp_processor_id() or get_cpu() can basically be replaced with a get_per_cpu() or get_cpu_ptr() invocation. In addition, this kmalloc_percpu is closer to Dipankar's original patch, and hence far more space-efficient than the one which went in.

Tuesday April 29 2003

Fabrice used the qemu mailing list. Yay!

Finished first cut of per-cpu module support (well, first cut using the new kmalloc_percpu). Compile and test it tomorrow.

Also changed the module reference counts to use kmalloc_rcu, making them much smaller (currently they have to be cacheline aligned so they don't alias: kmalloc_rcu can share cachelines with others).

Wednesday April 16 2003

Oh, yeah. Qemu now has a Savannah Page and a developers' mailing list. Not much action yet, but I want to get back to it (netfilter non-linearization work has been interfering, and there'll probably be more work there).

I've been doing testing on my dual x86 crash box "mingo" (which is an insult to Ingo Molnar, since it's old and slow, but Paul names boxes ending in -go, and I couldn't resist). It kept crashing after ten minutes to a couple of hours (2.4.19 solid as a rock, 2.5 flaky), so I've been working back through kernel versions. I've discovered that "nmi_watchdog=2" causes it to lock up solid too (even on 2.4), so I have taken that out and am repeating my tests. It seems to be some APIC wierdness, since it also crashes on UP unless X86_UP_APIC=n. It's slow going (to be sure of a positive result, I need to leave it running overnight). Tonight: 2.5.14.

Tuesday April 15 2003

Continued the non-linear skb handling work for 2.5; this time NAT (did conntrack late last week). Uncovered a bug in the old code while I was there, so I've sent that to Marcelo for fixing.

The second netfilter summit is going to happen, just after the Linux Bier Wanderung; I'm really looking forward to both, and hope I can go (depends on the price difference between just going from here to OLS, and continuing around the world.

I'm particularly looking forward to having beers with Harald, Marc, James and Jozsef; they're all really nice guys and great fun to hang around with. The summit will no doubt be incredibly productive, too: hopefully LBW beforehand will give us some time to do more coding.

Friday April 11 2003

More Qemu stuff. Found a bug in the fstat64 translation which caused memory corruption; explains why dash and sash segfaulted. Also, sent Dave a patch which pushed the skb linarization in netfilter down to the hooks in 2.5. He and Alexey were getting impatient: and more needs to be done.

There are three areas remaining with Qemu. Firstly some code cleanups would be nice: wrestling with the debugging output convinced me that qemu's elf.h leaves something to be desired. Secondly is the obvious one: performance. Ratios range from 3.5:1 to about 12:1, depending on the platform and code. There are two obvious culprits: firstly that translated code is about 5 times longer than the code before translation, and secondly that there are no direct jumps from emulated code to other emulated code. Finally, there is coverage: I had to implement fcntl lock translation to get some programs to run, for example. I'll try running the POSIX test suite under it next week and see how it goes.

So, how good can performance be? I don't think a factor of 3:1 is impossible, but it's important for the portability and correctness of qemu that too many arch-specific optimizations don't occur, or at least are not required. I think that register renaming and simple common subexpression elimination can probably drop generated code size by 30% at least, and block coalescing can win more: at the moment, the average code block translated by qemu seems to be around 5 instructions long: 50% of runtime is probably spent in the code execution engine (more will be spent in the translation engine). Also, block coalescing should improve optimization opportunities significantly.

What gives me optimism is that these optimizations can (if neccessary) be done only once code has shown to be run several times. Even better, this reveals the runtime nature of the code. For example, there's little reason why blocks whose only relationship is at runtime cannot be coalesced, eg. through function calls (obviously, you might end up with two versions of the same function in this case, one inside the coalesced block and a standalone version for other callers).

But Fabrice has probably thought of all this and more, since he's done a good number of important optimizations already (such as minimizing the calculation of the flags word), and he's obviously fairly experienced with this kind of thing. It'll be fun to watch!

Thursday April 10 2003

After discussions with Fabrice, set up a Savannah qemu site. I'd have preferred he did it, but since he wanted a volunteer... I've had several people contact me about qemu, so I think that at least having a mailing list for developers is going to speed development significantly.

Friday April 4 2003

Played with a little: looks promising. Haven't tested speed yet, of course.

Tuesday April 1 2003

Back home for a week now; caught up with mail and am bombing Linus with patches (he's distacted at the moment apparently).

Damn automake: the man page patch for module-init-tools sent to me doesn't work for automake 1.4 (1.6 is fine), so Terje Rosten sent me a fix. Since I used automake 1.4, the man pages didn't install. Released 0.9.11a.

Sunday March 23 2003

I am so slack about updating my diary. Had a wonderful time in India, and got lots of work done in Beaverton and New York. It'll be nice to be home though.

I promised to put the talk I did in India: here it is in MagicPoint (7M) and in HTML (9M) .

Wednesday Feb 19 2003

Looks like I'm heading across to Oregon then New York after India. Can't say I'm really looking forward to it, either

Werner Almesberger has taken lead of the "registration races" discussion, and I'll be interested to see where he takes it. I don't really have an opinion, just an awareness that various forms of this problem has been around since 2.2 without resolution: 2.4 established a partial solution (ie. the module unload case), which is slowly becoming increasingly widespread, but it'll be interesting to see if he can come up with an answer which doesn't gut efficiency, or obfuscate code so badly that apathy wins again.

Either way, I think I'll owe him a beer.

Tuesday Feb 18 2003

Looks like I'll be giving a talk at CONCEPTS in Pune, India on 14th March. Don't know exactly what I'm talking about yet, though: looks like the audience will need something fairly general, probably a rallying call for everyone to become a kernel hacker is a good idea.

Tuesday Feb 11 2003

Alli and I are now godparents, of Tala Honor Olds. I plan on rewatching the Godfather I through III soon, as part of due diligence.

Wading through the SSA gcc code for constant propagation: between the textbook, the Value Range Propagation paper, and that code, I think I'm starting to understand it.

Friday Feb 7 2003

Jeff posted a story on linux.conf.au on slashdot: the responses were generally positive, but there was one message about an international KDE speaker who was invited then beaten up at customs. Of course, a little checking reveals that it's a troll: we didn't invite any KDE speakers from overseas, and we know everyone who didn't turn up. The person obviously didn't come to the conference.

It's weird: I guess someone has a problem with Australia and they thought that this was a way for them to do some damage. But maybe that's too logical. I've seen the classic "rapist almost caught me in a supermarket carpark" urban myth with local names inserted, doing the rounds on EMail. I don't understand.

Tuesday Feb 4 2003

Zwane Mwaikambo updated and tested the hotplug CPU patches: I wasn't planning on pushing them for 2.6, as there's a freeze on and all that. It's great to see someone running with it though, and wider exposure. Sometimes Eric Raymond is right about this Free Labor thing.

Linus has woken up again: Kai's modversions stuff has gone in, which is great (I consider it Kai's: the original patch is mine but he did the nasty work of making the build system work properly). I like it when other people do my work for me: I can be productive even when playing that damn gtetrinet game.

Some good linux.conf.au writeups, but if I had missed the conference I think I'd find them lacking. I suppose that's what the Oggs are for, but having one person summarize each session would be really nice too.

Monday Feb 3 2003

Chris introduced the office to gtetrinet. Talk about doing negative work...

Tuesday Jan 28 2003

Back to reading mail again. Andrew picked up Kai's modversions patch, which looked pretty good, but I reimplemented it to not rely on linker order. It'd be nice to zlib those symbols: my kernel exports about 1500 of them, and the strings really do add up.

Monday Jan 27 2003

Not much hacking today; had Greg Kroah-Hartman and his wife and daughter over for lunch, and Suparna, then drove with Tridge to Sydney to pick up Linus and his entourage.

Sunday Jan 26 2003

Great conference: I ran half an hour short on my keynote, but I don't think anyone minded, as they were all recovering from the conference dinner.

Next year is in Adelaide: they're going to have a tough time following this conference, but I think that linux.conf.au has nailed the "OLS of the Southern Hemisphere" title.

Monday Jan 20 2003

Gave a talk at the Debian mini-con: nothing dramatic, but I did some simply non-technical user surveys and presented them. The problem they have with "I want to do this: what do I install?" gave Bdale a(nother) excuse to talk about his idea of Debian Flavors, where proper subsets of Debian are used to address a particular task. Interesting idea, and seems to make sense to me.

Wednesday Jan 15 2003

Sparc exports ".div" and ".mul". Oh well, calling them "div" and "mul" should be fine; it uses its own macros to export them anyway.

Finally published my init race patch, which I suggested way back when the init change was reverted, and had sitting on my web page for a while. I don't look forward to the debates, explanations, and misunderstandings that it will bring: modules are racy, and it's nothing I did, and the other alternatives are worse. Oh well.

Tuesday Jan 14 2003

Richard posted a patch to support weak symbols, and also cleaned up some code. Unfortunately, the (as yet unmerged) PPC64 port requires things as they were, but Linus released .58 before I could argue. Oh well, I took the heart of it and implemented it, tested, and sent it off.

Oh well, it happens, it's noone's fault, and it's better than the alternative: noone hacking on my code because it's an impenetrable mess.

Refreshed the PPC64 support, and Anton put it in the kernel. Requires a little depmod tweak (modules ask for .foo, but "foo" is exported), but works fine. That'll hit Linus as soon as the RTH thing gets sorted.

Finally, tested Kai's version of the module version magic patch, tweaked and sent it back to him. That should go in soon, too.

LCA next week, and the Debian mini-con before that. I love going, but hate losing the time.

Monday Jan 13 2003

Andi reported it was an arch-specific bug, not my fault at all. Oh.

Tridge dropped by today to complain about four module things. He was having trouble building the VMWare modules: the kernel didn't report that the problem was the common symbols (minor patch). He was getting messages about a module failing to load, but didn't know which one (because the failure was not finding the module structure, so it didn't know its name): I tweaked my kmod patch to report when modprobe returns with non-zero exit status, which fixes this. Then, the latest Debian X modules didn't work (although they loaded), due to them being compiled with gcc 3.2, and the kernel with 2.95, which changes the size of spinlocks. I posted a patch to include a little version mask in the module struct (SMP-ness, PREEMPT and the major and minor kernel versions), but Linus suggested a special section with the version in it. Done, and posted.

The fourth thing was that sound didn't autoload any more. Turns out that the "sound" alias does nothing under 2.5: "sound-slot-0" is what you want. ie. not my fault, but once again, reporting the failure of the request_module allows diagnosis.

I also posted a couple of cleanup patches, now the lists and lock are all internal to kernel/module.c, which simplify things. On top of this goes a patch to get rid of the module semaphore altogether, but that patch locks up under stress_modules.sh, so I'm chasing that tomorrow.

All in all, not a bad day. We're getting there.

Saturday Jan 11 2003

Alli worked today, so I did too: I'd fallen behind from this week. Andi Kleen reported that x86_64 module loading gets wierd corruption: my userspace framework didn't have the same problem, but Richard reported problems on Alpha, too. 64 bit issue? Don't know, still chasing. I don't have a 64-bit crash box.

Lots of minor cleanups to go around. Kai pointed out that changing EXPORT_SYMBOL to use a flat structure (rather than a struct with a pointer to the string) caused unneccessary bloat: he's right, but pulling it out in general would break depmod (depmod being the part of modules which is tied to the kernel, and should probably be shipped with it). It's in 0.9.8 in case we decide to go that way.

Release 0.9.8 of module-init-tools, mainly because I'd not realized that depmod would not see the GPL-only symbols exported by a module. Obviously, few people use this, since I only had one report.

Linus dislikes variable length arrays on the stack. As long as it's constrained (like any stack usage), I don't see the problem, but he complained about code generation. I deleted my original "WTF are you on?" reply and asked for a less terse explanation: the gcc 3.0 code looks reasonable, and it's certainly nicer than kmalloc/kfree in a loop.

Friday Jan 10 2003

Linus has been really good about merging recently. Hopefully this means the new features (like SYSENTER) are behind us. Couple of minor slips, like thinking I didn't have to place the __gpl_ksymtab section, just because the linker generates the __start_ andd __stop_ symbols for me: of course many archs won't place it where we want it unless its mentioned in the linker script. I guess I got lucky on x86.

Adam Richter proposed replacing my locking scheme with a semaphore. Other than being useless for connection tracking netfilter modules (which currently use their own reference counter for exactly this reason), it's terribly slow on SMP: the entire point of the current scheme is to avoid cacheline bouncing and atomic ops (i386, at least, can define __HAVE_LOCAL_INC and avoid atomic ops as well).

My original dislike of module unloading was caused by two things: firstly, it was invasive in interfaces and modules. This I came to accept by using an already existing scheme and expanding it (meaning that most people already understand what is required), and indeed, most of the module refcount patches I have seen recently have been trivially correct. My second dislike was the performance implications of any existing reference count scheme: we went to great lengths in networking and elsewhere to avoid stomping on the same cachelines, and to introduce this penalty everywhere for what is statistically a "never happens" case is a really bad idea.

Of course, if you're dealing with filesystems, who cares. But modules are increasingly being used in areas without such a clearly defined "activate/deactivate" boundary: the obvious case is the security modules which poke their fingers in everywhere.

Wednesday Jan 8 2003

Had to fly to Cairns early Tuesday morning: Alli's Grandfather's funeral; the only flights we could get. The hotel phone line was too noisy to connect, so I only got one chance to grab mail until Friday night.

Sunday Jan 5 2003

Polished up and transmitted the license patch and the exception table cleanup patch, and sent to Linus. He queried the removal of the initialization of the exception tables, which is a feature, not a bug.

Having abandoned the modules as ET_DYN (ie. shared) objects approach until 2.7 (MIPS and MIPS64 need significant work for this), I'd best advertise my userspace test framework for those arch maintainers who still haven't implemented modules.

Friday Jan 3 2003

You can guess what my New Year's resolution was, huh?

Module code went in while I was travelling. Normally this would be a Good Thing, but it has taken me about six weeks to get off the back foot and back to being productive. I didn't anticipate the level of negativity which some people would bring: most of them never saw the old module code (and frankly, didn't read the new code either), so didn't seem to understand why I was rewriting "perfectly good" code.

Depressing, and I stopped reading lkml during that time. But things seem fairly sorted now. I even did the documentation.


Rusty's 2002 Diary