normal size = 31603726 (100%), rsync xfer = 29134042 (92%). uncompressed: size = 69849664 (221%), rsync xfer = 20563916 (65%). rsync size = 31750518 (100%), rsync xfer = 13132994 (41%).As you can see, `gzip --rsync -9' is slightly larger than using normal gzip -9, but gives a much reduced transfer.
Ran through old netfilter bug reports and patches, doing scores for them. Wow, have I let that go in the last couple of months. Sorry. Also, made the scoreboard 2-column, and cleaned up the web page.
Alan is slowly merging my set_bit audit results with Linus, but hasn't reached the netfilter one yet (and another Sparc64 report just showed up). I'll send that separately.
We actually started making them back in November, but the enthusiasm waned. My fault.
I'm going to have to do some work on parallelism to fill my slots at linux.conf.au.
Turns out I'm not alone in not knowing that `set_bit()' needs a long, and I sent a search-and-destroy patch to linux-kernel. Alan picked it up, so hopefully it'll be included in 2.4.
A bug was reported with 2.4's ipchains compatibility module; seems to be broken in 2.4.11, but I didn't get a chance to test it before I got on the plane (things have been really hectic).
FHS work continues. Hmmm, maybe I'm optimistic for a rapid release of FHS 2.2...
The Linuxcare Ozlabs social club has some money, so we're looking at buying a video game machine (or two...). Chris Yeoh and I would really love a Gauntlet machine, but a cheaper option will probably prevail.
Tridge came up with a much better method to make rsync scale: I'll be implementing that next week (basically, do one directory at a time). This means a version increment, so we'll fall back to the old code for older clients.
TDB work continues: Anton did the spinlock stuff, and I've done the endian-neutral work and lots of locking enhancements -- it's now impossible to deadlock the database (without using `tdb_chainlock()' which is only there to make Tridge happy). Watch for a 2.0 release in coming weeks (it's worth a major revision, according to local consensus).
OK, having written that, there's still a few bugs in the locking...
Accepted an invitation to speak in Dublin at the end of the month; should be fun, although going round the world in 9 days is not exactly a holiday...
Sent rsyncable gzip patch to the Debian gzip maintainer: he hasn't applied it yet AFAICT. Turned out to be relatively trivial (I originally assumed that the size of the window for calculating when to reset the Huffman tables would be much larger than the internal 32k window; this is false, and a value of ~16k is fine). Next step is to propose its use for .debs, then the world will be mine. Bwaahahaha!
Two new releases of apt-proxy this week: major new speedup, plus a one character fix.
I have preliminary gcc patches (against 2.95.2) for memset arg reversal warning and stricter enum checking. Here are the untested memset and -Wstrict-enums patches for the 20001002 CVS snapshot.
ipchains release had `make install' bug, reported by two people. Fixed and released under same release number (no source differences).
Speaking of which, we have to have a new iptables release. There's lots of stuff which could use a wider audience (patch-o-matic mainly) before people start using 2.4 in anger.
Been busy:
Switching to 6:10 sleep for a while to try to keep up. Haven't read netfilter list in a week. Must do that tomorrow.
I'm going to make OUTPUT NAT a config option: it doesn't work at all at the moment, and it's fiddly. I broke it when fixing ICMP error handling a while back, and didn't detect it for months.
Hiroyuki Yamamori sent another mailful of corrections to the Packet Filtering HOWTO which he is translating into Japanese. He went so far as to look in the code to check the against text! So there may be more corrections to come.
Spent Wednesday hacking on apt-proxy still: installed it on flame Wednesday night, which led to an afternoon of debugging today, interrupted by having to write the lecture I was giving for Operating Systems Implementation at ANU at 3:00pm. Lacked my usual panash, but my mind was still in shell scripts and rsync...
Hacking on --rsyncable option to gzip. The gzip maintainers are probably impossible to get changes past (gzip is ported to everything: the code is a mess), but I'm trying to make this as non-intrusive as possible. This is exciting because widespread deployment of this option when creating .deb files will reduce stress on the Debian mirror sites by an order of magnitude (once apt understands rsync backends: apt-proxy already does). In practice, this probably means that it will flatten the Debian bandwidth requirement curves for a couple of years as it gets rolled out.
Today I've been working on apt-proxy. New version will be tested on flame for a week or so, then I'll roll it out. I've nominated this as version 0.98. Expect an announce soon. Yes, it's still in sh.
Uncovered a bug on the testing with the Ultrasparc (thanks to Anton Blanchard), which can be fixed in userspace. Packet queuing doesn't seem to work at all though, leaving wide holes in the test suite.
Spent the last two days fixing up the test suite after the TCP changes, and getting it to run on the iMac (found two bugs: one minor one mine, and one major PPC one).
Two bug reports on ipchains within a week means it's time to roll 1.3.10; a very minor release. Sent the notification to the ipchains list, and the change contributors: the beta can be found here.
Big news is that Harald Welte made top of the scoreboard, by doing conntrack and NAT for IRC, miscellanous other patches, and ULOG, which I think will prove very popular among the more hard core users (it has the potential to become a significant project in its own right).
Looks like my ftp bug also bites the 2.2 (and presumably, the 2.0) ip_nat_ftp modules. Nasty. But because they don't drop packets containing a partial PORT pattern, they won't stop making progress until later (much less deterministic). Sent Juan Ciarlante a report.
It's interesting that two independent approaches hit the same bugs (I only ever glanced at the ftp code; most of this stuff I derived myself).
Note to self: keep diary more up-to-date.
I finally tracked down that damn FTP-fails-after-30-files bug which some people were experiencing. I had to walk through the code by hand, line-by-line, substituting example numbers, and finally it hit me. This bug has been in there since the beginning, but only happened when your packets were being shortened, and the total subtracted bytes exceeded one packet size. Then, boom; ack numbers got screwed, and wierd things happened.
Felt good to nail that damn thing (easy once I managed to replicate it in the test suite).
Since Coniston was booked out (sunny Saturday) as Alan Cox warned, Harald Welte (who gave me a lift from Manchester) and I are crashing here in the hall tonight. This place is much more of a chaotic user-group-meeting feel than OLS or any conference; arrived to see Duncan Grove, a friend of mine from my old home town, setting up ppp on the SparcStation 2 (the entire place is funnelling through a 56k dialup).
I'm running a public rproxy between my laptop and one of Harald's machines in Germany: it's really needed for this case, and serves as an excellent test. I also set up the DNS caching forwarder, then DHCP to enhance the peg-based system (and coincidentally, point them through the DNS proxy).
Having to expand the conntrack/NAT architecture a little for helpers: avoiding allowing two connections to expect the same range of connections, for IRC and to fix a possible ftp problem, and also to add a `your expected connection has arrived' callback for videoconferencing protocols which must wait for another potential stream. Multiple expecteds off one connection will not be supported yet (although may well be needed later).
| Sun 16 July 15:30 | AN 5968 | Canberra->Sydney (Arr 16:20) |
| Mon 17 July 9:35 | NZ14 | Sydney->Los Angeles |
| Mon 17 July 8:40 | NW932 | Los Angeles->Detroit |
| Mon 17 July 17:20 | NW3305 | Detroit->Ottawa (Arr 19:29) |
| OLS | ||
| Tue 25 July 15:00 | CP813 | Ottawa->Toronto |
| Tue 25 July 18:20 | KL692 | Toronto->Amsterdam (Arr 07:40 Wed) |
| Amsterdam | ||
| Sat 29 July 9:20 | UK2023 | Amsterdam->Manchester (Arr 09:40) |
| Linux Beer Hike | ||
| Mon 7 August 8:20 | UK2024 | Manchester->Amsterdam |
| Mon 7 August 13:30 | KL1655 | Amsterdam->Venice (Arr 15:20) |
| Linuxcare, Italia | ||
| Sun 13 August 06:45 | KL1650 | Venice->Amsterdam |
| Sun 13 August 11:50 | KL605 | Amsterdam->San Francisco (Arr 14:05) |
| LinuxWorld | ||
| Sat 19 August 19:35 | UA2067 | San Francisco->Los Angeles |
| Sat 19 August 22:15 | NZ15 | Los Angeles->Sydney |
| Mon 21 August 07:20 | AN5953 | Sydney->Canberra (Arr 08:10) |
This puts me in phase with the Europeans on IRC, which is always nice (on the linux channels, there is less `chatter' when the US are asleep, and if I wanted to chat, I'd call someone up).
Did a trial implementation of RST generation, but it doesn't work with NAT, so I'm abandoning it again (the connection is unconfirmed, since the packet hasn't passed through the box yet, so the connection tracking code refuses to associate the RST reply with it). This could be worked around by generating the RST myself, but this either requires a very naive approach (easy to fingerprint), or dragging in a couple of hundred lines of TCP internals (no thanks!).
There's never a good time to get sick, but probability is good that I'll be fine in time for OLS.
But I'm finally up-to-date on EMail and most incoming patches. I want to get the scoreboard more uptodate, review bug reports, and run the test suite on the iMac (Paul Mackerras is back, so now it will surely behave). Pulled aside into some local testing work as well.
There's a bug lurking somewhere: had too many hard lockups on module removal to be coincidence.
Just found another bug: ICMP handling in compatibility modules (found as I was trying to update scoreboard through ipchains.o module: we run a 1460 MTU tunnel here, so those ICMP Frag Needed errors must be translated correctly!). Practice of Programming says `never leave a bug until later'. Make that 6:15am.
Even with my complex solution, you end up with the possibility of an attacker using fake IPs 64 apart (which hash the same), to get a huge hash collision. Harder, but still fairly easy. Anyway, if you're doing DNAT out to N servers, the attack will be spread over N chains, halving the debilitating effect (we store a hash of the REPLY src ip, dst ip and protocol in the NAT code).
Can't get a damn 2.4.0-test1 kernel to boot on the damn iMac. non-Intel platforms hate me, and our experts are all out of the office. Next week...
Linus created 2.4 dir today: batton down the hatches.
Shorter day today: trying to keep body clock in sync with normality for Melbourne this weekend.
Got to sleep 7am. Good to be back at work.
Have to get back to Canberra (my return is on Saturday).
I've been putting off sending the `woah! break everything!' patch to Linus until I'm convinced that it fixes all those wierd platforms (fucking mac users!). I'm not in Canberra, so no access to Sparc or PPC.
Figured out rough itinerary today, for OLS and LinuxWorld:
Ottawa Linux Symposium: Wednesday July 19 - Saturday July 22 Linux Beer Hike: Sunday July 30 - Sunday August 6 LinuxWorld: Monday August 15 - Thursday August 17Spent much time with friends (old and new) back here in Adelaide. It's been great, but I have to get back to Canberra to go into deep hack mode again.
Worked on an DocBook form of the Kernel Locking HOWTO. I'm not sure that it's suited for inclusion in the mainstream kernel, but I'll humbly submit it to Alan, and see what he thinks. Jeff Garzik wants me to submit the kernel hacking HOWTO the same way: I need to revise it heavily (remove the locking stuff, clear up the context stuff, etc) before I do that.
Paul Mackerras reported a bug in iptables on the PowerPC. Must get back to Canberra soon.
Also looked at the code path used by tracking existing connections, and simplified it a little; cleaner code, fewer function calls. Profiling should help find furthur problems.
After much trial and tribulation, got sound working on my Thinkpad 600E. Also, first physio today. Ouch. But no more self-administered injections!
I am now battling to get him to Ottawa in July: Amlan gave me a contact for the people who got him to Singapore, and I'm chasing it up with them now.
Getting Alexey in a room with Dave M, AK, and Donald Becker would do more for Linux networking than a month of me hacking out code. Hence I'm spending a reasonable amount of effort on this, even though I'm no travel agent.
Some excitement last week with the Linuxcare announcement that the CEO was retiring: being out of the office I was probably the last person on the planet to know. I'm told we are still in the quiet period, so I can't say anything about it even if I had something intelligent to say...
The Wounded Knee Party (held in my absence at my apartment) is tonight: hope everyone has fun.
Saturday I went out with some great local friends and Marc Boucher
to a local club: didn't get much sleep, and injured my knee again
(there's a
Tonight had Richard Guy Briggs and his wife Carol over my parents' place for dinner. Nice wind-down.
I'm on holidays this week. Yeah right: must hack.
Sunday I finally found the `panic when builtin' stuff: sent it in to DaveM and netdev. Took out ipt_REJECT's tcp-reset option because (1) I didn't like it, and (2) I should have tested it before using the inbuilt kernel routines. Alexey has long felt that generating RST packets cavalierly is a real danger to the integrity of the protocol (as Microsoft did in this case).
People keep asking me what happened to 6:10. Still going, after something of a lapse. I know I can do it now, so it's just a matter of deciding that I need to. I have needed to this week.
It just occurred to me tonight that maybe apologies are *supposed* to be that way: so afterwards you don't feel any less sorry. Then you try as hard as you can to fix things.
Finished first cut of the NAT HOWTO, and working on the Networking Concepts HOWTO. Then the Packet Filtering HOWTO. Writing documentation is hard work, but this needs to be done soon. More patches from various people, and need to rework test suite as well, tomorrow.
At the conference, a John Adams turned up to talk with me, and his badge said `Netfilter'. Turns out that these guys have the trading name `netfilter' in Australia. Fortunately, it also turns out that John (one of their senior Sales/Marketing types) just wanted to talk to make sure I knew they were fine with sharing the name: he was pretty clued up, and obviously neither of us wants to get into a namespace war.
The end result is that we are swapping links on our pages, so if search engines send people to the wrong place, they don't get terribly confused. By the way, the do controlled (filtered, bandwidth-controlled, etc) ISP access for companies in Australia.
I hear there's been some grumbling on the list about netfilter not compiling. Marc is here in Canberra now, and we're doing the final fixes before the merge.
Tridge is back! I realized when I walked in and saw him at his desk today that there had been a big hole in the office without him.
Martin Pool gave me this quote out of ``The Reader'' which I just finished:
When an aeroplane's engines fail, it is not the end of the flight.
Aeroplanes don't fall out of the sky like stones. They glide on, the
enormous multi-engined passenger jets, for thirty, forty-five minutes,
only to smash themselves up when they attempt a landing. The
passengers don't notice a thing. Flying feels the same whether the
engines are working or not. It's quieter, but only slightly: the wind
drowns out the engines as it buffets the tail and wings. At some
point, the earth or sea look dangerously close through the window.
But perhaps the film is on, and the stewards and air hostesses have
closed the blinds. Maybe the very quietness of the flight strikes the
passengers as an improvement.
That summer was the glide path of our love.
Between the discussion with her on the experiences she had, and a fascinating discussion with Telsa Gwynn on IRC, I'm sure that a woman coming to work at Linuxcare Ozlabs in a technical role would have issues that the guys don't. And I think the Linux Kernel has the same kind of issues.
Imagine if you wanted to be a coder, but gave up because others made it unbearable? That is the polar opposite of the spirit of Free Software, and I won't be party to it.
Shot patch I off to the core team and James Morris for testing: if that's OK, I'll send
Can't believe that Igno and DaveM plan on introducing a new kind of lock this late in the 2.3 cycle. They must have an infinite supply of `get out of freeze' cards. Mind you, the brlocks are really cool, and I've changed the netfilter stuff over to use them, but fuck a brick, guys.
Saturday night, I went with David, Paulus and Martin Pool to a Bach Recital across the road from the office. It was really good, but I can't really tell the difference between great music and a great performance for that stuff. It was much fun though. Martin and I sought out the Canberra nightlife afterwards. I'm sure it's here somewhere.
Tired, but food (didn't eat lunch) and a shower should make me OK to go out with friends later tonight. Never turn down a chance to be sociable: you might not be asked next time.
0.90.2 release compiling up as I write. ne2000 doesn't work under 2.4.47, so I couldn't commit while testing. Helped Paulus with a tty locking bug which has been plaguing PPP on 2.2 SMP kernels, which made me feel less useless.
Bookings for Chateau de Rusty are currently as follows:
This is an event which must be marked.
I want to have Larry McVoy's lovechild, after his linux-kernel thread. Larry is da man on scalability, and he used to work for SGI (and Sun, and...): here he is arguing with a current SGI engineer. My favourite is probably:
... go talk to your hardware designers. I know there are some left.
Forgot to set alarm last night (set time, but didn't activate it), so I stole an extra 1:10 sleep. Still, I think that a Monday large-caveat 0.90 release is possible.
Martin's antennae are starting to bother me, but I don't think I should say anything about it: noone else has mentioned it so maybe he's sensitive. I can hear his evil avian laughter all the time now.
Trouble getting up this morning, but nothing too severe: this 6:10 thing seems to be working. Netfilter testing continues: I keep adding new tests, so it's more gradual than it might otherwise be, but this isn't a race.
Testsuite runs up to the NAT tests again (on UP: SMP has oopsen which I suspect aren't my fault). A solid day: working all weekend might put me in range of a 0.90 alpha release.
Was accepted for OLS : shaping up to be a fantastic conference. If I get ambitious I might go around the world in July...
I've decided to experiment with cutting down on sleep: I've decided to try a week on 6 hours, 10 minutes (I usually prefer an irregular 9). My coworkers have been warned that I might well be a raving psychotic by Wednesday.
Still hacking on new netfilter generics. ethertap is broken in 2.3.42 still, so I can't run my test suite, but I can still break things pretty badly. At least my keyboard works under X again. DaveM shocked my by putting userspace checksumming in 2.3.41, and he tells me that softnet is going in as well. Linus can't be serious about 1 month to 2.4. Try 4 or 5.
Of course, coming back, I was right back into it with my CLUG talk `Rusty's Remarkably Unreliable Guide to Kernel Hacking', which was extremely chaotic. Slides up RSN.
Netfilter hacking continues: check something into CVS, and am now busy crashing my kernel. This will go on for some time before I can actually make a release.
Packet filter generalization rewrite work continues. Slowly.
I am so much looking forward to hitting the road for a few days to get away; way more than I expected. Maybe this `holiday' concept has some merit. (OTOH, maybe after two days of not coding, I'll be itching to code again 8-).
The packet filtering kernel code now compiles again; I need to make NAT use it properly, and then the huge job of redoing userspace. I'm tempted to check in to CVS, but I have to decide how to rearrange files first.
Phone call was interrupted by Martin Pool dropping in to stay: I thought that was next week. Martin is joining Linuxcare, and rode his (motor-)bike down from Brisbane. Oh well, I hacked on Saturday (explicitly against My Rules), so I can't feel too guilty.
Submitted my proposal to OLS; I'm assured that it was really cool last year. Escaping winter in Canberra is always a good plan anyway.
Today, releasing a new version (kernel change in 2.3.38 caused minor breakage) wasted over two hours of my time trying to figure out CVS (gave up, created manual diff).
I expect a learning curve, but this is crazy. I'm between a rock and a hard place here: I didn't want to master CVS (which you need to do, to work around its severe limitations), but I've given up on Larry McVoy's Bitkeeper project. I wouldn't be so severely pissed off if I hadn't had expectations that all this pain wouldn't be neccessary.
Did an APM power-off hack which I'm not proud of; an SMP box here wants the APM BIOS calls done from CPU 0, and there's no simple way at shutdown to choose your CPU. I simply loop and create bogus kernel threads until I get scheduled on the right CPU. Linus will hate it.
Found the skb bug: it was introduced by my patch in 2.3.35 (pointed out by Alexey). I found it on New Year's Eve; it was a simple matter of lazy coding (who would have guessed that skb_realloc_headroom was sometimes used to reduce headroom?).
Very little response from the plea for testsuite enhancements, but maybe that's because it doesn't run on RedHat (patch sent in by James Morris).
Today's coding task is to start on the generic-packet-selection CVS branch, off the main branch (it's orthogonal to the skb-reservation stuff). These two are the ingredients for netfilter-modules 1.0, which is a prerequisite for the merge to Linus.
Read through the core of the softnet stuff. I think I'd like to document what it is and what it changes now I went to the trouble of understanding it. Basically, it makes no difference to the netfilter module requirements (as Alexey said on netdev); even the netfilter internals remain unaffected, since I didn't try to be really clever.
Basically, Alexey implemented soft irq's: they run with no constraints whenever triggered (no serialization control). So far NET_TX_SOFTIRQ and NET_RX_SOFTIRQ use this. Under that he implemented `tasklets': these are guarenteed only to be run once on each CPU at any time. Finally, he also implemented bh's, which are never reentrant (ie. only run on one CPU at a time). I'm not quite sure why he didn't implement bh()'s as tasklets, but I'm guessing that it's for efficiency reasons: bh()'s are still going to be important in the forseeable future even if Alexey's code gets into 2.3.