toofishes.net - Blog

Because it is what I've been spending most of my Arch free time on lately, I thought I'd make another quick post on what's happening with the Arch Linux website.

• Filelists are back! Check out a package page such as pacman. You'll see the Files header at the bottom; if you click the link inside you'll get a filelist. If you aren't crazy and using something like noscript, some nice AJAX will put the list right inside the page; otherwise you'll be sent off to the files page.
• Mirror management improvements. We manage all our (known) mirrors through a developer accessible UI, and we were missing some important fields to store things like what tier a mirror is or what rsync credentials it should be using. These are now available.
• reporead improvements. This script is the magic that takes a pacman database and shoves it into our web site database to get all the relevant information out there. It is a bit more robust and now allows for force updates if/when new fields are added.
• Speaking of new fields, you can now see the compressed size, installed size, and the build date of every package on the website.
• Courtesy of Evangelos Foutras, the developer todo list UI now has some AJAX to make package flagging slightly less painful.
• Many small touchups for efficiency, such as not fetching the full list of packages whenever a user clicks through on the Recent Updates box.
• Recent updates groups together multiple architectures in order to make more packages visible in the box.
• Feeds for package updates are a lot more granular. Check out the new feeds page for more details.

A few more things are planned down the road. These include:

• Advanced search. This will allow searching on every field we have in the database as well as finding a package with a given filename (or filename prefix).
• Remove some of the lesser used pieces of the site in favor of linking to important wiki articles. These include Arch-based Projects, Press (which has been unlinked for a long time), International Communities, and IRC Channels. Putting this stuff on the wiki will allow it to be kept much more up to date than it currently is.
• Other general cleanup. We have old CSS styles, hacks from the pre-1.0 Django days, and a lot of style embedded in our templates rather than in stylesheets.
• And of course, any contributions from you that will enhance the site for the whole Arch community. :)

Just wanted to drop a quick note saying Slicehost has finally released a new kernel that fixes all of the issues I had pointed out in the 2.6.31-302-rs build. It took a whole lot longer than I would have wanted—nearly three months—but it does seem to be working great so maybe one could say it was worth the wait.

The new kernel has a version of 2.6.32.1-rscloud. It includes both the xt_recent kernel module, used for my iptables rate-limiting rules on the SSH port, and also has a much smaller memory footprint with none of the JFS and XFS junk stealing valuable megabytes of RAM. If you are running Arch Linux on your slice, I would highly recommend updating to this kernel.

I finally got around to giving the site that is the face of Arch Linux its long-deserved refresher today. Of course, you probably just clicked that link and went "I don't see anything new!". Most of what I coded, merged, and deployed today was the behind the scenes stuff. However, it should help set up any future usability changes we want to make and make development in general a heck of a lot easier.

The biggest drawback up to this point with the site was its two-faced nature. It had been split a while back into two seperate archweb_pub and archweb_dev Django codebases. This was a bit before my time, but I believe it was for caching concerns. We want the main site to be heavily cached as it does get hit relatively heavily (side note: I forgot to enable caching right away today after deploying and it was quite noticeable). However, the developer site was meant to be uncached so when doing things like editing news items, you could immediately see the results of your work.

The fact that these two sites served slightly different audiences wasn't a problem. The problem came with keeping them in sync. They shared the same database, and thus the same model classes and even several helper functions and views. Thus making a change to the models was a huge chore as you would have to make it in both places (copy and paste, oh no!), then fix up the code, and then deploy them both…at the same time. Top that all off with the fact that no migration framework was in place, and you have a project that doesn't exactly scream "fun".

Well this weekend I finally decided to get this fixed on the live site. I should say at this point that Ismael Carnales, an Arch community member, started the hardest part of this work back in November by undertaking the task of merging the two applications back together. He put nearly 30 git patches together to accomplish this task. I talked with him about a lot of this stuff back then, and pulled in his work after we worked though a few of the trickier issues. However, his work sat for a long time without actually being deployed (pretty much all my fault).

Last week, Evangelos Foutras, an Arch Linux Trusted User, sent a few patches to the ML for archweb. This was a bit of a kick in the ass for me to get moving on this again. I looked at the patches, they looked good, and I pulled them into this new merged code base. I then added a few small patches and fixes of my own, and got the new, merged codebase deployed to our developer site yesterday.

Today I woke up to another email with a few more proposed patches, including the integration of south, a Django schema migrations tool, into the project. I grabbed these and ran with them as it would help fix a long-outstanding bug that bothered me- the inability to use foreign keys correctly with package maintainers. I then made more progress in a day than I had in six months on this web app, fixing quite a few of the easier bugs in Flyspray (FS#12752, FS#13166, FS#16752, FS#17141, FS#17144) as well as scratched a few other itches.

I'd like to thank Ismael and Evangelos for contributing a ton of work to this release that should get us pointed in a good direction. I think the project will be easier to pick up and run with now that it isn't two separate codebases, and the addition of south should make it less scary to change the models and have to worry about changing the data.

Next things that are on the radar are (1) figure out a better way to do package maintainers, and (2) try to get file lists back into the database and thus back on the website.

A while back, I posted about a long-needed Slicehost kernel upgrade from a 2.6.24 to a 2.6.31 build. Since I run Arch Linux on my slice, it is a lot easier on the maintenance if I am running a relatively up to date kernel. Given that 2.6.24 was released in January 2008, it was definitely time for something more recent.

Fast forward two and a half months. I've been running this 2.6.31-302-rs build of theirs the entire time, and I was a very early adopter. Given this, I found two non-trivial issues with the build and gave their support team a heads up. The first was an issue I noted in my earlier post regarding iptables and the recent connection tracking module changing names and thus no longer being included.

Ticket 14486, November 1, 2009

I upgraded to the 2.6.31 kernel and noticed my iptables rules didn't get loaded correctly. It looks like it is due to the xt_recent kernel module not being available. In 2.6.24 (through I think 2.6.27), it was known as ipt_recent, which is probably why it got omitted in the new build. Can we get it back?

dmcgee@toofishes /etc
$ locate xt_recent.ko

dmcgee@toofishes /etc
$ locate ipt_recent.ko
/lib/modules/2.6.24-24-xen/kernel/net/ipv4/netfilter/ipt_recent.ko

And the "offending" rules in case you were curious:

$ cat /etc/iptables/iptables.rules | grep recent
#-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 300 --hitcount 10 -j DROP
#-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set

The second ticket was an issue I found soon after. Although my slice now has 321 MB of memory, it also has a ton more overhead than it used to due to a bunch of unnecessary kernel-level processes.

Ticket 14489, November 2, 2009

Another note about the new 2.6.31 kernel- it ends up having a ton more overhead because it looks like JFS and XFS are built-in, and all of their kernel processes get started even if one isn't using either of these filesystems. Because they aren't modules I can't unload them and free up some memory.

Example 1: Process count spike, look at the thin line and the spike at the end of October

Example 2: Memory usage jump, note the application memory and swap in use

dmcgee@toofishes ~
$ ps -eLf | grep -E 'jfs|xfs'
root        42     2    42  0    1 Nov01 ?        00:00:00 [jfsIO]
root        43     2    43  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        44     2    44  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        45     2    45  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        46     2    46  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        47     2    47  0    1 Nov01 ?        00:00:00 [jfsSync]
root        48     2    48  0    1 Nov01 ?        00:00:00 [xfs_mru_cache]
root        49     2    49  0    1 Nov01 ?        00:00:00 [xfslogd/0]
root        50     2    50  0    1 Nov01 ?        00:00:00 [xfslogd/1]
root        51     2    51  0    1 Nov01 ?        00:00:00 [xfslogd/2]
root        52     2    52  0    1 Nov01 ?        00:00:00 [xfslogd/3]
root        53     2    53  0    1 Nov01 ?        00:00:00 [xfsdatad/0]
root        54     2    54  0    1 Nov01 ?        00:00:00 [xfsdatad/1]
root        55     2    55  0    1 Nov01 ?        00:00:00 [xfsdatad/2]
root        56     2    56  0    1 Nov01 ?        00:00:00 [xfsdatad/3]
root        57     2    57  0    1 Nov01 ?        00:00:00 [xfsconvertd/0]
root        58     2    58  0    1 Nov01 ?        00:00:00 [xfsconvertd/1]
root        59     2    59  0    1 Nov01 ?        00:00:00 [xfsconvertd/2]
root        60     2    60  0    1 Nov01 ?        00:00:00 [xfsconvertd/3]

I got very quick responses to both of these tickets. Both were along the lines of "thanks for letting us know, I'll get this over to our kernel guys". I got the impression there would be a new kernel in a week's time or so, especially given this more detailed follow-up response I received a few days later.

Slicehost Support, November 5, 2009

Hello Dan,

Thank you for your messages about the new kernel. We're working on a new kernel now that will take up much less resources and also provide the same amount of iptables functionality as the previous kernels. The new kernel is currently being tested in our testing environment and we hope to have it released very soon.

As soon as it is available, we will post an update on http://slicehost.com/blog/ and it will be available for you in the SliceManager. We apologize for the issues in the current kernel and we are working to fix the problems as quickly as possible.

Please let us know if we can assist you further at any time.

So what has happened since? Nothing. I sent them another follow-up email a month later asking on the status of the new kernel. I got a response telling me they are "working through some compatibility issues" and "we are in the final stages of the last round of testing". Wow, final stages of the last round of testing? That sounds good, they really do some good regression.

Only problem? This amazing new kernel still is nowhere to be seen. Slicehost, your good will meter is shrinking faster than you know right now. It has been nearly 50 days since that conversation and I still see no new kernel available. Will this blog post help? I have no idea, but know that your customers are trying real hard to stay loyal but Linode is looking awfully good right now.

It's already gone across the Arch Planet feed from a few other people's blogs, but an interview with the Arch Linux Team was published today. I think it is a good read because it wasn't just Aaron speaking for us; instead anyone on the dev team that wanted to contribute answers was more than welcome to. I have a few well-placed answers in there, but I am glad that they left it mostly unedited from what we submitted.

Anyway, enjoy the interview!