toofishes.net - Blog

I finally got around to giving the site that is the face of Arch Linux its long-deserved refresher today. Of course, you probably just clicked that link and went "I don't see anything new!". Most of what I coded, merged, and deployed today was the behind the scenes stuff. However, it should help set up any future usability changes we want to make and make development in general a heck of a lot easier.

The biggest drawback up to this point with the site was its two-faced nature. It had been split a while back into two seperate archweb_pub and archweb_dev Django codebases. This was a bit before my time, but I believe it was for caching concerns. We want the main site to be heavily cached as it does get hit relatively heavily (side note: I forgot to enable caching right away today after deploying and it was quite noticeable). However, the developer site was meant to be uncached so when doing things like editing news items, you could immediately see the results of your work.

The fact that these two sites served slightly different audiences wasn't a problem. The problem came with keeping them in sync. They shared the same database, and thus the same model classes and even several helper functions and views. Thus making a change to the models was a huge chore as you would have to make it in both places (copy and paste, oh no!), then fix up the code, and then deploy them both…at the same time. Top that all off with the fact that no migration framework was in place, and you have a project that doesn't exactly scream "fun".

Well this weekend I finally decided to get this fixed on the live site. I should say at this point that Ismael Carnales, an Arch community member, started the hardest part of this work back in November by undertaking the task of merging the two applications back together. He put nearly 30 git patches together to accomplish this task. I talked with him about a lot of this stuff back then, and pulled in his work after we worked though a few of the trickier issues. However, his work sat for a long time without actually being deployed (pretty much all my fault).

Last week, Evangelos Foutras, an Arch Linux Trusted User, sent a few patches to the ML for archweb. This was a bit of a kick in the ass for me to get moving on this again. I looked at the patches, they looked good, and I pulled them into this new merged code base. I then added a few small patches and fixes of my own, and got the new, merged codebase deployed to our developer site yesterday.

Today I woke up to another email with a few more proposed patches, including the integration of south, a Django schema migrations tool, into the project. I grabbed these and ran with them as it would help fix a long-outstanding bug that bothered me- the inability to use foreign keys correctly with package maintainers. I then made more progress in a day than I had in six months on this web app, fixing quite a few of the easier bugs in Flyspray (FS#12752, FS#13166, FS#16752, FS#17141, FS#17144) as well as scratched a few other itches.

I'd like to thank Ismael and Evangelos for contributing a ton of work to this release that should get us pointed in a good direction. I think the project will be easier to pick up and run with now that it isn't two separate codebases, and the addition of south should make it less scary to change the models and have to worry about changing the data.

Next things that are on the radar are (1) figure out a better way to do package maintainers, and (2) try to get file lists back into the database and thus back on the website.

A while back, I posted about a long-needed Slicehost kernel upgrade from a 2.6.24 to a 2.6.31 build. Since I run Arch Linux on my slice, it is a lot easier on the maintenance if I am running a relatively up to date kernel. Given that 2.6.24 was released in January 2008, it was definitely time for something more recent.

Fast forward two and a half months. I've been running this 2.6.31-302-rs build of theirs the entire time, and I was a very early adopter. Given this, I found two non-trivial issues with the build and gave their support team a heads up. The first was an issue I noted in my earlier post regarding iptables and the recent connection tracking module changing names and thus no longer being included.

Ticket 14486, November 1, 2009

I upgraded to the 2.6.31 kernel and noticed my iptables rules didn't get loaded correctly. It looks like it is due to the xt_recent kernel module not being available. In 2.6.24 (through I think 2.6.27), it was known as ipt_recent, which is probably why it got omitted in the new build. Can we get it back?

dmcgee@toofishes /etc
$ locate xt_recent.ko

dmcgee@toofishes /etc
$ locate ipt_recent.ko
/lib/modules/2.6.24-24-xen/kernel/net/ipv4/netfilter/ipt_recent.ko

And the "offending" rules in case you were curious:

$ cat /etc/iptables/iptables.rules | grep recent
#-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 300 --hitcount 10 -j DROP
#-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set

The second ticket was an issue I found soon after. Although my slice now has 321 MB of memory, it also has a ton more overhead than it used to due to a bunch of unnecessary kernel-level processes.

Ticket 14489, November 2, 2009

Another note about the new 2.6.31 kernel- it ends up having a ton more overhead because it looks like JFS and XFS are built-in, and all of their kernel processes get started even if one isn't using either of these filesystems. Because they aren't modules I can't unload them and free up some memory.

Example 1: Process count spike, look at the thin line and the spike at the end of October

Example 2: Memory usage jump, note the application memory and swap in use

dmcgee@toofishes ~
$ ps -eLf | grep -E 'jfs|xfs'
root        42     2    42  0    1 Nov01 ?        00:00:00 [jfsIO]
root        43     2    43  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        44     2    44  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        45     2    45  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        46     2    46  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        47     2    47  0    1 Nov01 ?        00:00:00 [jfsSync]
root        48     2    48  0    1 Nov01 ?        00:00:00 [xfs_mru_cache]
root        49     2    49  0    1 Nov01 ?        00:00:00 [xfslogd/0]
root        50     2    50  0    1 Nov01 ?        00:00:00 [xfslogd/1]
root        51     2    51  0    1 Nov01 ?        00:00:00 [xfslogd/2]
root        52     2    52  0    1 Nov01 ?        00:00:00 [xfslogd/3]
root        53     2    53  0    1 Nov01 ?        00:00:00 [xfsdatad/0]
root        54     2    54  0    1 Nov01 ?        00:00:00 [xfsdatad/1]
root        55     2    55  0    1 Nov01 ?        00:00:00 [xfsdatad/2]
root        56     2    56  0    1 Nov01 ?        00:00:00 [xfsdatad/3]
root        57     2    57  0    1 Nov01 ?        00:00:00 [xfsconvertd/0]
root        58     2    58  0    1 Nov01 ?        00:00:00 [xfsconvertd/1]
root        59     2    59  0    1 Nov01 ?        00:00:00 [xfsconvertd/2]
root        60     2    60  0    1 Nov01 ?        00:00:00 [xfsconvertd/3]

I got very quick responses to both of these tickets. Both were along the lines of "thanks for letting us know, I'll get this over to our kernel guys". I got the impression there would be a new kernel in a week's time or so, especially given this more detailed follow-up response I received a few days later.

Slicehost Support, November 5, 2009

Hello Dan,

Thank you for your messages about the new kernel. We're working on a new kernel now that will take up much less resources and also provide the same amount of iptables functionality as the previous kernels. The new kernel is currently being tested in our testing environment and we hope to have it released very soon.

As soon as it is available, we will post an update on http://slicehost.com/blog/ and it will be available for you in the SliceManager. We apologize for the issues in the current kernel and we are working to fix the problems as quickly as possible.

Please let us know if we can assist you further at any time.

So what has happened since? Nothing. I sent them another follow-up email a month later asking on the status of the new kernel. I got a response telling me they are "working through some compatibility issues" and "we are in the final stages of the last round of testing". Wow, final stages of the last round of testing? That sounds good, they really do some good regression.

Only problem? This amazing new kernel still is nowhere to be seen. Slicehost, your good will meter is shrinking faster than you know right now. It has been nearly 50 days since that conversation and I still see no new kernel available. Will this blog post help? I have no idea, but know that your customers are trying real hard to stay loyal but Linode is looking awfully good right now.

It's already gone across the Arch Planet feed from a few other people's blogs, but an interview with the Arch Linux Team was published today. I think it is a good read because it wasn't just Aaron speaking for us; instead anyone on the dev team that wanted to contribute answers was more than welcome to. I have a few well-placed answers in there, but I am glad that they left it mostly unedited from what we submitted.

Anyway, enjoy the interview!

Subversion was the first serious open-source (and free) version control systems to be a worthy rival to CVS. For anyone that has used CVS in the past and has moved on to better tools, you can understand where those who started the Subversion project were coming from. With CVS came no atomic commits, no easy way to rename files, and many other fun things I have since forgotten.

Fast forward 10 or so years. We now have a huge selection of version control systems, many of which have adopted the more distributed model. SVN still fills that niche (especially in the corporate world) of having a centralized repository while not being near as encumbered with restrictions as CVS. You'd think in 10 years of steady development, SVN would have done a pretty good job getting the kinks worked out. Compare this to git, which has only been around for five years. However, I've found SVN to have some of the worst performance ever when it comes to doing absolutely nothing, which is a fairly uncomplimentary thing to say.

I decided tonight to gather some basic numbers and performance characteristics of Subversion. As a comparison, I've done some similar tests with git and will show those here as well. I should note that both timing and tracing runs here were done after the respective update operation (svn update, git pull) actually did a refresh of the local copy- the timings and traces using strace you see below are of what turns out to be "no-ops".

Subversion timing and tracing

dmcgee@galway ~/projects/arch-repos
$ time svn up
At revision 62267.
Killed by signal 15.

real    0m13.375s
user    0m1.160s
sys     0m0.600s

dmcgee@galway ~/projects/arch-repos
$ strace -c svn update
At revision 62267.
Killed by signal 15.
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 65.46    0.019177           2     11598           unlink
 21.88    0.006409           0     46516        52 open
  3.74    0.001095           0     46468           close
  3.43    0.001005           0     23198           getdents
  3.24    0.000948           0     69601           fcntl
  1.79    0.000523           0     46478           read
  0.32    0.000094           1       176           brk
  0.08    0.000024          12         2           wait4
  0.07    0.000021           0       152           mmap
  0.00    0.000000           0        37           write
  0.00    0.000000           0        13         9 stat
<snip>
------ ----------- ----------- --------- --------- ----------------
100.00    0.029296                244451        64 total

Git timing and tracing

dmcgee@galway ~/projects/linux-2.6 (master)
$ time git pull
Already up-to-date.

real    0m0.636s
user    0m0.100s
sys     0m0.033s

dmcgee@galway ~/projects/linux-2.6 (master)
$ strace -cf git pull
Already up-to-date.
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.30    0.135659        2886        47        21 wait4
  0.36    0.000488          24        20           execve
  0.15    0.000208           8        26           clone
  0.13    0.000181           2       115           munmap
  0.02    0.000032           0       394         6 close
  0.02    0.000027           0       226        79 stat
  0.02    0.000026           4         7         2 connect
  0.00    0.000000           0      1216           read
  0.00    0.000000           0        21           write
  0.00    0.000000           0       438       162 open
  0.00    0.000000           0       233           fstat
  0.00    0.000000           0        55        27 lstat
<snip>
------ ----------- ----------- --------- --------- ----------------
100.00    0.136621                  4672       371 total

Looking for Answers

Let's sum up the test as the above raw data may not mean much just yet.

I've highlighted the two figures in the above table that I find astounding. Yes, I know these two repositories aren't identical. One has more files, the other more directories, and SVN definitely seems to struggle as you add directories as it sticks its own .svn metadirectory in each one. But that doesn't excuse its awful performance. Why on earth is it making over 11,000 unlink calls to do absolutely nothing? Then there are the other 230,000 syscalls that I haven't even begun to think about.

I wish I cared more about Subversion to help make it better, but I don't use it in my personal projects anymore because git is so quick and easy. It looks like it is time to move the Arch Linux package repositories to something that sucks less.

Bonus material

I ran this test too but it didn't really have a comparable operation in git, so it didn't fit in above. I'll put it here and let you draw your own conclusions.

dmcgee@galway ~/projects/arch-repos
$ strace -c svn cleanup
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 36.39    0.038920           1     46388           rmdir
 28.92    0.030931           1     46388           mkdir
 17.87    0.019107           2     11597           unlink
  8.17    0.008734           0    104492     11647 open
  3.26    0.003482           0    115970           getdents
  1.70    0.001821           0     92845           close
  1.69    0.001806           0     72307     11597 lstat
  0.95    0.001014           0     92799           fcntl
  0.91    0.000972           0     58076           read
  0.14    0.000155           0     11598           lseek

GCC has a rather crazy syntax that I discovered tonight for the first time, and I thought I'd share so you don't have to dig for a half hour like I did to find out what it is and how it works. The feature is compound statment expressions. I came across it in the Linux kernel code in include/linux/percpu.h:

#define per_cpu_ptr(ptr, cpu) ({ (void)(cpu); (ptr); })

The long and short of this expression is this- if you call per_cpu_ptr(myptr, 5), you will get myptr back. But the macro to do this confused me, with no explicit return.

With a compound statement expression, as opposed to just a compound statement (those things surrounded by curly braces), the last item in the block is used as the return value for the whole expression. Thus, in the above expression, the value of ptr will be returned. The key is including both the parenthesis and curly braces.

If you want to see a full example, check out the following (working) program.

#include <stdlib.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    int number;
    char *string;

    string = ({ "hidden"; "shown"; });
    number = ({ int a = 50; a += 50; a; });

    printf("%s %d\n", string, number);
    return 0;
}

If compiled with gcc test.c, it should produce the output "shown 100".