Skip to the Main Content

Note:These pages make extensive use of the latest XHTML and CSS Standards. They ought to look great in any standards-compliant modern browser. Unfortunately, they will probably look horrible in older browsers, like Netscape 4.x and IE 4.x. Moreover, many posts use MathML, which is, currently only supported in Mozilla. My best suggestion (and you will thank me when surfing an ever-increasing number of sites on the web which have been crafted to use the new standards) is to upgrade to the latest version of your browser. If that's not possible, consider moving to the Standards-compliant and open-source Mozilla browser.

August 19, 2009

nLab Migration

Posted by Urs Schreiber

As you may have heard, we had various reasons to migrate the nnLab wiki to a different server. Thanks to tremendous help from Andrew Stacey this is now essentially achieved.

We have a trial version of the migrated nnLab up and running now, here:

migrated nnLab (at nlab.mathforge.org)

Before we officially migrate, change DNS servers and redirect everything to the new server, we’d like to ask you to test the new installation by messing around with it.

Please, if you have a minute, go to the migrated installation and look around the entries, try editing some entries etc, as if the whole thing were a huge sandbox.

The point is to have a look around and see if anything looks or feels different to the original. Please notify us about anything weird that you spot, anything that might have been broken in the migration process.

Notify us by dropping a comment at the nnForum.

Beware. We have not yet officially migrated. Any changes made at the new migrated installation will be lost when we do the proper migration. This is just a test run.

For the time being, all actual content that you want to contribute is still to be edited at the nnLab at its original URL. In fact, in the end we will redirect URLs accordingly, so that you will never submit any genuine contribution elsewhere than at http://ncatlab.org/nlab.

Thanks for your help.

Posted at August 19, 2009 12:50 PM UTC

TrackBack URL for this Entry:   https://golem.ph.utexas.edu/cgi-bin/MT-3.0/dxy-tb.fcgi/2039



56 Comments & 1 Trackback

Re: nLab Migration

Went to Physics page; clicked on edit at bottom of page, waited several minutes; no response. Using Firefox, latest update.

Posted by: Charlie C on August 19, 2009 2:25 PM | Permalink | Reply to this

Re: nLab Migration

I can’t get the site to load at all! Just timed out.

Posted by: David Roberts on August 19, 2009 2:52 PM | Permalink | Reply to this

Re: nLab Migration

Hm. If we are lucky this means that Andrew just switched the migrated site off for a moment and will switch it back on in another moment.

If we are not so lucky this might mean that migration hasn’t yet solved our main problem…

Posted by: Urs Schreiber on August 19, 2009 2:55 PM | Permalink | Reply to this

Re: nLab Migration

No luck loading anything here either.

Posted by: Eric Forgy on August 19, 2009 3:07 PM | Permalink | Reply to this

Re: nLab Migration

Okay, everybody wait until Andrew Stacey gets back to us with news. Then we’ll know more.

Posted by: Urs Schreiber on August 19, 2009 3:13 PM | Permalink | Reply to this

Re: nLab Migration

Same here.

Posted by: Alex Hoffnung on August 19, 2009 3:30 PM | Permalink | Reply to this

Re: nLab Migration

Having been in on this migration stuff from the beginning, I can assure everybody that the new site used to work. But we'll have to here from Andrew about what's happening now, since only he has access to the server at the moment.

Posted by: Toby Bartels on August 19, 2009 6:17 PM | Permalink | Reply to this

Re: nLab Migration

Memo to self: in future, it’s probably a really bad idea to agree that Urs should make some announcement about the new nlab just before I go off for a violin lesson.

Y’all may be interested to know that before Urs made this announcement we’d been discussing whether or not it was worth doing some sort of test of how the new host coped under pressure. Urs and Toby said that if we made an announcement asking for help then everyone would do so, I thought that as it was just a test site then it would be unlikely that many people would go over and try it out.

It’s amazing how wrong you can be!

I’ve just been looking at the logs trying to figure out what happened. Between Urs making this announcement and the poor thing running for cover, it received nearly 300 hits from nearly 50 ips. We seemed to hit some sort of memory cap and it couldn’t cope with any more.

What seems to have happened is that it went into a loop trying to respawn the processes, but as it had reached its maximum memory threshold it couldn’t do it.

Bailing out like this is clearly Not Good. However, I’m not sure at this stage what was the problem step: it may be the hosts, it may be the web server, it may be instiki itself.

In the meantime, I’ve rebooted the VPS and it does seem to be live again.

It is still useful to have comments since even if we have to change the configuration somehow, it would be good to know if the migration itself works correctly as there are a few steps where human error could creep in.

I think we’ve just had a preview of what might happen if we get slashdotted …

Posted by: Andrew Stacey on August 19, 2009 7:42 PM | Permalink | Reply to this

Re: nLab Migration

it received nearly 300 hits from nearly 50 ips

Ah, so it is requests for testing that motivates the nn-Café dwellers!

Okay:

just purely for testing purposes, please everyone test if it is possible that you create an entry on one of your favorite topics in math, physics or philosophy at the true nnLab, and test if you can fill it with a brief 10-line quick overview and maybe a link to your favorite reference on it.

(Here is How To do that, if you don’t know.)

Let’s see if you can bring the currently running installation of the nnLab to its knees by flooding it with content!

Posted by: Urs Schreiber on August 19, 2009 7:56 PM | Permalink | Reply to this

Recently Revised

I went to the migrated nnLab a few times now, within some hours. I checked the “Recently Revised” page to look for activity.

It all worked so far (except that there was not much activity to be seen there…). But a minute ago, hitting Recently Revised produced this error message

Application error (Apache)

Something very bad just happened. I just know it. Do you smell smoke?

Hitting relaod produces

Internal Error

An application error occurred while processing your request.

Hitting reload once again… there it is, the Recently Revised page.

I got that smoke message before on the nnLab. Apart from being very funny indeed, I never understood where it came from and why it disappeared after a while.

And now if and when the Recently Revised page does appear on the migrate lab, it appears a bit less slowly (not to say: quicker) than it used to on the current nnLab before we turned it off there. But it’s pretty slow. At least it doesn’t seem to halt the entire system, but apparently the original source of the inefficiency of this page is within instiki somewhere.

I am guessing that calling this page makes the server go through its database in its entirety to check for changes. That would at least explain why its slow, and gets slower as the lab grows.

If we could re-program the software, we should make it have an auxiliary file into which each time an entry is changed a corresponding message is written. Calling “Recently Revised” should then simply display that file.

Posted by: Urs Schreiber on August 19, 2009 10:17 PM | Permalink | Reply to this

Re: Recently Revised

Recently revised is already cached. The problem is that the cache needs to be rebuilt every time someone revises something.

Looking at the logs, I get times of between 7ms and 13s for producing ‘recently revised’! I guess that’s ‘cached’ and ‘noncached’.

I expect that there are better ways of doing it, along the lines of what you suggest. I’m not familiar with the code so I don’t know how it currently works.

What would be interesting is whether calling ‘recently revised’ slows down the whole lab (on the new host) in the way that it did on the old one.

Posted by: Andrew Stacey on August 20, 2009 10:18 AM | Permalink | Reply to this

Re: Recently Revised

I forgot to say, the errors you saw were probably caused by me doing the upgrades. After berating John elsewhere about unsafe umask values, I then went and got them wrong myself (though my mistake was opposite to what I was warning against there: I made things too restrictive).

The ‘I can smell smoke’ is just a replacement for the ‘500’ error. Those are never informative to the user since they could potentially give away important information. It’s a signal to the maintainer to go and look in the logs. So on the migrated n-lab it’s very useful to tell me when you see it (and with times, as accurately as you can remember).

But this one was me mucking about ‘under the bonnet’ so “Nothing to see, folks, move along there”.

Posted by: Andrew Stacey on August 20, 2009 10:40 AM | Permalink | Reply to this

Re: nLab Migration

I’m rather hesitant at posting this, given what happened yesterday - and given that I’m about to go offline for a couple of hours while I give a lecture (incidentally, I’m using Instiki for that course as well).

Anyway, the migrated n-lab is back and better than before.

Specifically, I got some suggestions from the people at our hosts (Rails Playground) on how to cut down memory usage. I’ve put those in place so hopefully we’ll cope a bit better with spikes. If we can’t, then we need to buy a bit more memory but then I think that for our size we’re quite memory-intensive since we probably have a higher edit/view ratio than, say, Wikipedia (a function of our age, I’d say).

In the spirit of encouraging good behaviour, not that I expect this to be seen by anyone relevant, my first encounter with the support at Rails Playground was extremely positive. I asked a fairly vague question on their forum (since I didn’t know how to phrase it better) and got an answer from someone on their support team with good, specific suggestions. So they clearly monitor their forums and are good at figuring out what the customer meant to ask!

Back on topic, over at the forum, Toby’s already noted a couple of differences between the two labs. These seem to be to do with how many levels of escaping needs to be done with “unsafe” characters. I’d be interested in how widespread these errors are as to what kind of fix is appropriate.

Posted by: Andrew Stacey on August 20, 2009 10:27 AM | Permalink | Reply to this

Re: nLab Migration

I just tried to create a new page ‘database of categories’, and when I clicked on a link of that name in an existing page, I got… a blank page.

This happened a bunch at the old nnLab, too, and not just when going to not-yet-created pages: I click a link to a page and the nnLab sort of stalls out and gives me a blank white page.

In general the new nnLab seems sort of painfully slow, a lot like the old one.

Posted by: John Baez on August 20, 2009 8:29 PM | Permalink | Reply to this

Re: nLab Migration

The Unthinkable

Posted by: Eric Forgy on August 20, 2009 9:22 PM | Permalink | Reply to this

Re: nLab Migration

I get this from time to time; I can tell when it happens even if I'm in another tab, since the title doesn't get set. It seems especially common if I'm working on the Lab in several tabs at once; if I wait until everything's done failing and then try them again, then it's OK. (If not, then I restart instiki and everything really is OK.)

So if you get a blank page, then you should be all right if you just reload the page. Is that ever not true?

Posted by: Toby Bartels on August 20, 2009 10:34 PM | Permalink | Reply to this

Re: nLab Migration

Sometimes when I reload the page it works fine. Sometimes I click “Edit”, get a blank page, then reload, and get a mild error message saying something like “John Baez is already editing this page. Do you want to edit anyway?” I say yeah and go on editing.

So, it’s not a disastrous bug, but in conjunction with the overall slowness it’s a bit annoying. I had hoped these issues were problems with the old server, but I guess they’re not.

Posted by: John Baez on August 21, 2009 4:31 AM | Permalink | Reply to this

Re: nLab Migration

Although I disagree with Eric’s remedy, his diagnosis has a lot to it. From the Instiki ‘TODO’ page:

Let’s be blunt: Instiki is kinda slow. Some of this is because Ruby and Rails are kinda slow. Ruby 1.9 (when Rails becomes 1.9-ready) will provide a huge speed-boost.

Anyone who has an opinion as to a remedy for this should first read the discussion to which Eric refers up above.

Done that? Good.

I’m reluctant (I seem to be saying that a lot these days!) to open this again here, particularly as I’d like to focus on the migration of hosts project here rather than migration of wikis, but here’s my killer reason for not moving to Mediawiki: the number of people interested in getting mediawiki to the point where it is capable of serving MathML is extremely small, perhaps only one, whereas the number of people interesting in speeding up rails applications is much higher. Therefore we don’t have to do all the work in speeding up Instiki, whereas I suspect that we would have to do all the work in porting to Mediawiki. Of course, if we feel that progress is slow in speeding up Instiki then we could always join in:

Of course, there are always improvements that can be made. That’s where (I hope) you come in …

Posted by: Andrew Stacey on August 21, 2009 8:03 AM | Permalink | Reply to this

Re: nLab Migration

It would be helpful if people could timestamp these comments. If you say “it feels kinda slow” then all I can do is say “maybe the electrons were on strike in LA”. If you give me even an approximate time (including timezone!) then I can look in the logs and get some idea as to why it was slow. There are several steps in getting a page to you:

  1. Your browser sends the request to the host.
  2. The web server on the host works out what to do with it (in this case, pass it on to Instiki).
  3. Instiki processes it and returns it to the web server.
  4. The web server sends it back along the wire to your browser.
  5. Your browser interprets the code in the page and prints something that looks nice.

Every step there can get slowed down for one reason or another. If your computer is doing something extremely important while you’re browsing the web, then the first and fifth steps can be extremely slow (my ageing laptop almost halts when it’s updating software). If all the electrons in LA have gone on strike in sympathy with the scriptwriters, then the first and fourth can be extremely slow. As I commented above, the Instiki process is itself not the zippiest program on the block.

The choice of host contributes to the first and fourth: if they are overloaded, or their machinery isn’t capable, then that slows down data flowing to and from their architecture. I don’t get the impression that Rails Playground are guilty of this, but it’s hard to find out exact statistics. We can always try a different host if we feel that this is the rate determining step.

The second step is what I have most control over. At the moment, we’re running the apache webserver. Whilst this is the industry standard, it’s also fairly big which can slow things down a little. I’m considering trying out Nginx which seems to be smaller and a bit nippier.

As for Instiki itself, well I can’t do too much about that. There are small things that can be done such as shifting databases from sqlite3 to mysql (which I’ve done) and using phusion passenger rather than mongrel (again, done). It may be that postgresql will give better performance than mysql, it’s hard to get reliable data on these things and I’m not an expert. Also, there may be further optimisations to do with mysql, but I suspect that to do these will involve a trade-off between memory and speed. However, what happened earlier when Urs first announced the migration site was that we ran out of memory. So I’m reluctant to simply use up memory just to get a millisecond or so off the round trip.

Of course, we can always buy more memory on our VPS but then that’s a trade-off between cost and speed.

Posted by: Andrew Stacey on August 21, 2009 9:04 AM | Permalink | Reply to this

Re: nLab Migration

Sometimes I click “Edit”, get a blank page, then reload, and get a mild error message saying something like “John Baez is already editing this page. Do you want to edit anyway?” I say yeah and go on editing.

Yes, happens regularly to me, too.

Also when submitting, I often get some internal error message, but I learned that despite the error message, the content was often corrrectly submitted, as can be checked by viewing that page in another window.

So clearly clicking both “edit” and “submit changes” induce some reaction that happens in stages, and the command itself seems to always succeed, while sometimes the operation of displying the result fails.

In that context I reiterate two general advices to the general readership here that make life less painful when submitting material to blog and wiki:

First one:

Make it a habit to always copy-and-paste your edit window content to the clipboard before hitting submit.

When I hit submit, I always do it Ctrl-A/Ctrl-C/Submit.

Better yet, if you are preparing a long piece, make a security copy of it to some editor every two paragraphs.

Second one:

If your content disappears and only error messages are left, don’t forget to try the back button on your browser (or choose “back” from right-mouse-button-pulldown menu if it is a small edit window). At least with Firefox, the typed content can usually be recovered that way.

Posted by: Urs Schreiber on August 21, 2009 12:45 PM | Permalink | Reply to this

Re: nLab Migration

When I hit submit, I always do it Ctrl-A/Ctrl-C/Submit.

Since you're already using the keyboard, try Ctrl-A/Ctrl-C/Alt-Shift-S. Neat, eh?

Better yet, if you are preparing a long piece, make a security copy of it to some editor every two paragraphs.

Or just write it in some editor. This is particularly easy if you're using Firefox and install It’s All Text!.

Posted by: Toby Bartels on August 21, 2009 3:53 PM | Permalink | Reply to this

Re: nLab Migration

John:

I got… a blank page.

Toby:

I get this from time to time;

John:

Sometimes when I reload the page it works fine. Sometimes I click “Edit”, get a blank page,

It’s not clear to me from your discussion whether this is happening with the new host or not. Next time that this happens on the new host could you please note down the time (and timezone) and the page you were trying to load and send me those details.

Thanks.

Gosh, the cafe’s running slow today.

Posted by: Andrew Stacey on August 21, 2009 1:47 PM | Permalink | Reply to this

Re: nLab Migration

It’s not clear to me from your discussion whether this is happening with the new host or not.

I was talking about the old host because, well, that's where John's database appeared! (^_^)

Posted by: Toby Bartels on August 21, 2009 3:46 PM | Permalink | Reply to this

Re: nLab Migration

John wrote:

I just tried to create a new page ‘database of categories’, and when I clicked on a link of that name in an existing page, I got… a blank page.

I’ve been through the logs, I’ve searched in the database, and I can’t find a reference to ‘database of categories’ anywhere. There is no record of anyone trying to create the page and there is no record of a page with that string on it.

Are you sure you were doing this on the test site and not on the original one?

http://ncatlab.org/nlab is still the old nlab. We haven’t migrated the DNS yet so if you want to test the new one then you have to explicitly go to http://nlab.mathforge.org

Please can you give me some more details: on which page did you click on a link to create the page ‘database of categories’? When did you do this - even roughly!?

Thanks.

Posted by: Andrew Stacey on August 21, 2009 11:11 AM | Permalink | Reply to this

Re: nLab Migration

Please can you give me some more details: on which page did you click on a link to create the page ‘database of categories’? When did you do this - even roughly!?

A database of categories was requested by a comment on the Cat-Theory mailing list just yesterday, or the day before maybe. So I am guessing that John was being inspired by that.

In which case I hope he did it on the true nnLab. Because otherwise the effort would have been in vain.

Posted by: Urs Schreiber on August 21, 2009 12:24 PM | Permalink | Reply to this

Re: nLab Migration

He did, which is one thing that makes me rather suspicious about his original comment.

I’ve now done a full database search for the phrase ‘database of categories’ on mathforge and it does not appear anywhere. The word database only appears in three pages, and they’re all about the database underlying Instiki.

Posted by: Andrew Stacey on August 21, 2009 1:22 PM | Permalink | Reply to this

Re: nLab Migration

For the record, here's the database of categories.

Posted by: Toby Bartels on August 21, 2009 4:00 PM | Permalink | Reply to this

Re: nLab Migration

John wrote:

I just tried to create a new page ‘database of categories’, and when I clicked on a link of that name in an existing page, I got… a blank page.

Andrew wrote:

Are you sure you were doing this on the test site and not on the original one?

Umm, as you know by now, it was on the original one. I thought it was the new one. The confusion was entirely my fault; I didn’t read your original post carefully enough.

Futhermore, I’m sorry that I’m looking at this page again only now.

Worse still, I’m completely confused about what the URL of the migrated nnLab will be when it’s finally settled down, or even whether it’s officially migrated or not. On this thread you say ‘Now at last we migrated and things are getting better’, but here it sounds like something is still just being tested and isn’t really real yet.

If I had an emoticon that showed me committing harakiri, I would use it now.

Posted by: John Baez on August 27, 2009 6:08 AM | Permalink | Reply to this

Re: nLab Migration

Worse still, I’m completely confused about what the URL of the migrated nLab will be when it’s finally settled down, or even whether it’s officially migrated or not. On this thread you say ‘Now at last we migrated and things are getting better’, but here it sounds like something is still just being tested and isn’t really real yet.

Yeah, I found that comment strange as well. But I'm pretty on top of this, so I can tell you:

  • It hasn't been migrated yet.
  • When it has been, the old URL will still work.
Posted by: Toby Bartels on August 27, 2009 7:07 AM | Permalink | Reply to this

Re: nLab Migration

Okay, cool. Having a stable URL is important. I want to refer to Urs’ article on span trace in HDA7. My struggles to understand the correct future URL for this article led me back to this page, where to my dismay I found all the confusion I’d wrought!

Posted by: John Baez on August 27, 2009 8:02 AM | Permalink | Reply to this

Re: nLab Migration

I want to refer to Urs’ article on span trace in HDA7.

When you do that, be sure to put a date in the citation. (So ‘U. Schreiber, Span trace, $n$Lab article available at http://ncatlab.org/nlab/show/span+trace, retrieved 2009-08-27’ or the like.) Because while the URL will remain stable, the article won't, and someday it may not even be particularly Urs's article at all!

Posted by: Toby Bartels on August 27, 2009 8:25 AM | Permalink | Reply to this

Re: nLab Migration

Worse still, I’m completely confused about what the URL of the migrated nLab will be when it’s finally settled down,

On that at least there is a clear statement emphasized in the above entry: the official nnLab URL will always be the familiar one, no matter what happens behind the scenes.

Posted by: Urs Schreiber on August 27, 2009 2:05 PM | Permalink | Reply to this

Re: nLab Migration

When referring to pages on the nLab (which I think is awesome by the way), I would suggest not referring to the “current” page, e.g.

http://ncatlab.org/nlab/show/span+trace,

which is dynamic, but rather the specific revision. To do that, hit the “History” link on the bottom of the page. Then grab a link from one of the revisions, e.g.

http://ncatlab.org/nlab/revision/span+trace/6.

That way, as long as the nLab exists, the link in your paper will point to the content you intended.

Posted by: Eric Forgy on August 27, 2009 6:00 PM | Permalink | Reply to this

Re: nLab Migration

http://ncatlab.org/nlab/revision/span+trace/6

What if you want to link permanently to the current version as it is now? Then use your leet math skillz to add 11: http://ncatlab.org/nlab/revision/span+trace/7. Even though that is not linked from the history page, it works. (Actually, that is no longer the current version, since I just fixed a typo, but let's pretend.)

Beware, however: under Instiki as currently designed, these revision links will all fail if the page is ever moved. Hopefully Instiki will learn to automatically forward revision redirects before this becomes an issue.

Posted by: Toby Bartels on August 27, 2009 7:36 PM | Permalink | Reply to this

Re: nLab Migration

In that case, it is probably better just to point to the current page with an additional note (like you originally suggested) to the specific revision, “(Revision 7)”.

Posted by: Eric Forgy on August 27, 2009 7:52 PM | Permalink | Reply to this

Re: nLab Migration

Thanks, everyone, for the help about how to cite the nnLab in a paper. Maybe I’m the first person in the Universe to do this? Whee!

Posted by: John Baez on August 28, 2009 11:53 PM | Permalink | Reply to this

Re: nLab Migration

Also possibly useful is Wikipedia's advice.

Posted by: Toby Bartels on August 29, 2009 12:35 AM | Permalink | Reply to this

Re: nLab Migration

Umm, as you know by now, it was on the original one. I thought it was the new one. The confusion was entirely my fault; I didn’t read your original post carefully enough.

I didn’t know that for sure until I read this comment, so thanks for taking the time to confirm that (and apologies for all the sludge you may have waded through on this page).

Everything Toby said is correct so there’s no point in repeating it.

The current status of the migration is that we’re pretty sure that we know how to do it all (there will inevitably be one or two hiccoughs but we think they’ll be minor) so now we’re just waiting for a clear stretch when I won’t be off learning the violin to actually migrate.

Ideally, no one will notice that it happens until someone says, “Hey, the n-lab seems a little more stable these days”.

Posted by: Andrew Stacey on August 27, 2009 1:53 PM | Permalink | Reply to this

Re: nLab Migration

‘Now at last we migrated and things are getting better’, but here it sounds like something is still just being tested and isn’t really real yet.

Sorry for that misleading statement. I meant to express that we (Andrew Stacey mostly) finally went through all the technical trouble that a migration brings with it and that – as soon as Andrew presses some button – we have effectively migrated and that it seems that the new server does behave better than the old one.

Posted by: Urs Schreiber on August 27, 2009 2:02 PM | Permalink | Reply to this

Re: nLab Migration

I’ve written a script to get information from the logs about how long it takes to do stuff on the nLabs (both original and migrated). There are lots of different tasks that the Instiki process has to do and they take different lengths of time. I’ve put up tables from three Instiki processes (the two nlabs and one that I’m running for a course here) on my homepage somewhere roundabout here.

These are the times that Instiki records as how long it took to do what was asked of it. This doesn’t include any time taken by the webserver or for the request to arrive at the host or leave it again. So mainly it’s comparing the database speeds, and the fact that on mathforge I’m using an optimised version of ruby.

What’s interesting is that at the low level there isn’t a lot of difference. Sometimes the old nlab beats the new: showing a cached page, for example. The old one takes an average of .4s while the new one takes 1.2s. However, as things get more database-intense then one can really see the difference. A search on the old nlab takes an average of 40s. The new cuts that down to a mere 2s. The time for saving a page is cut from 3s on the old to 2s on the new. Listing stuff (which for some reason is marked as cached but I don’t understand that) takes 23s on the old but only 4s on the new.

Posted by: Andrew Stacey on August 21, 2009 11:37 AM | Permalink | Reply to this

Re: nLab Migration

Wow, Andrew, you are doing a signifcant job here.

A search on the old nlab takes an average of 40s. The new cuts that down to a mere 2s. The time for saving a page is cut from 3s on the old to 2s on the new. Listing stuff (which for some reason is marked as cached but I don’t understand that) takes 23s on the old but only 4s on the new.

Glad to see that the migration makes things quicker!

But John is right, overal reaction is still not as snappy as one would ideally wish.

I wish I knew what to do about this. But now, at least, with your and other people’s help, I am at least not pessimistic that eventually we might be able to improve things. Somehow.

I am thinking that if there is more in the instiki code that makes things scale inefficiently the way the Recently Revised page’s performance doesn’t scale, it should in principle be rather ease to change the software just a little. It was likely designed with a somewhat different optimization in mind than we need.

Posted by: Urs Schreiber on August 21, 2009 12:20 PM | Permalink | Reply to this

Re: nLab Migration

But John is right, overal reaction is still not as snappy as one would ideally wish.

I’m going to start getting snappy soon!

What do you expect?

I’d like to know what everyone is comparing the nLab to when they complain about speed. I have a suspicion that they’re comparing it to Wikipedia. That’s not a fair comparison for Oh so many reasons. One of the main ones is that at the moment, we probably have a much higher edit/show ratio than they do. Edits take much more time than just page views. Just think about how many times and how much data has to go back and forth when you do an edit. Also, they have a sophisticated set of mirrors in place. All our stuff goes to the one place, which is over there in the US.

If you look at the data I produced, you’ll see that most requests take on average less than a second for Instiki to process them. Those requests which hit the high numbers (eg list) have very skewed distributions (compare the means, medians, and modes). Looking at the last column in the table you’ll see that on the migrated nLab, only the search facility has ‘mode’ over 1s (the figures are a bit worse on the original). This says that the vast majority of the requests to instiki get processed in less than a second. I defy anyone to think that that is slow.

Every now and then one request seems to take an inordinately long time. I don’t know why this is, but it would be useful to track it down. This is probably what results in the blank pages that people see from time to time. Maybe we should start a “blank page” log so that when someone gets a blank page then I can look in the logs and see what it was that timed out.

I can also think of a few more optimisations that can be done in the meantime. For example, now that we are behind a proper webserver, we can leave the webserver to handle static files including the cached pages. That might speed things up a bit.

For example, I was a little bit surprised to see that the ‘list’ page is marked as cached but takes one of the longest average times to serve. I downloaded the cached page and the served page and they’re the same. But the logs show that Instiki “does” some processing on it. I suspect that this processing takes a long time because the page is so long, but that it actually ends up with doing nothing. So if this page were served statically when it is cached, zoom! everything speeds up.

I suspect that the reason this isn’t done by default is because Instiki is designed to run by itself rather than behind a web server so hasn’t been optimised for being run behind a server. It seems that it’s only recently that the big web servers (i.e. apache) have been adapted for rails apps.

Posted by: Andrew Stacey on August 21, 2009 1:42 PM | Permalink | Reply to this

Re: nLab Migration

What do you expect?

I’d like to know what everyone is comparing the nLab to when they complain about speed. I have a suspicion that they’re comparing it to Wikipedia. That’s not a fair comparison

Yes it is a fair comparison. Another fair comparison is every other single web page on the WWW. No one expects to sit and wait for anything these days.

Every now and then one request seems to take an inordinately long time. I don’t know why this is, but it would be useful to track it down. This is probably what results in the blank pages that people see from time to time. Maybe we should start a “blank page” log so that when someone gets a blank page then I can look in the logs and see what it was that timed out.

That is a good idea. If one of these “Every now and then” events occurs more than once, that is what sticks in our minds. Not the other 100 or so links that come back in 1 second or less.

I hope that someone reading this (especially lurkers!) will take on the challenge to create an itex2mml plugin for Mediawiki and make the whole world a better (mathematical) place. I started a little effort but life intervened. Some of my notes are here and here. Feel free to add to them.

Posted by: Eric Forgy on August 21, 2009 3:59 PM | Permalink | Reply to this

Re: nLab Migration

1, 2, 3, 4, 5, 6, 7, 8, 9, 10

If you advocate shifting to MediaWiki you must explain which bit of the process that is going to speed up. As Jacques has pointed out, it would take a phenomenal amount of work to get MediaWiki to the point where it is serving valid XHTML, let alone able to serve MathML. Basically, you’d have to rewrite MediaWiki. Doing the itexToMML plugin is the easy bit - Jacques has already told us how to do it somewhere else on this blog. I’m mildly interested in doing it, I just haven’t gotten round to it yet as I’ve been rather busy with other things.

Yes it is a fair comparison. Another fair comparison is every single web page on the WWW. No one expects to sit and wait for anything these days.

1, 2, 3, 4, 5, 6, 7, 8, 9, 10

No it is not. Wikipedia, and things like Google, have tremendous resources at their beck and call. Installing MediaWiki doesn’t mean that your site will work as wonderfully as Wikipedia does! Do you think that they use VPS? Somehow, I doubt it.

But the real reason it is not a fair comparison is that I suspect that most people who use the nLab are not Wikipedia contributors. So when they say that they “use Wikipedia” what they mean is that they look stuff up. But when they say that the nLab is slow, they mean that when the edit stuff then it is slow. That’s four times the data flowing across the network!

Google, Wikipedia, the BBC, all spend vast amounts of money ensuring that they don’t get those sporadic slowdowns because they know that one bad experience knocks out a whole slew of good ones. But then they’ve got the resources to do that! (Search for “UK license fee” to learn about the third one). Quite frankly, I’m glad that they do. If the BBC was on a VPS, they’d have completely crashed today as this happened.

If the nLab can find a sugar daddy with enough cash to sponsor a field of dedicated hosts, scattered about the world, then we may be able to get the response time down to that of Wikipedia. Back in the real world, Instiki ain’t necessarily the rate-determining step.

Posted by: Andrew Stacey on August 21, 2009 9:30 PM | Permalink | Reply to this

Re: nLab Migration

You are telling us WHY the nLab is slow. The fact that the nLab is slow is fine. Everyone participating understands why it is slow (or is capable of understanding after being told). Trying to argue that it is NOT slow is silly.

Posted by: Eric Forgy on August 21, 2009 11:08 PM | Permalink | Reply to this

Re: nLab Migration

You know why the nlab is slow? Great! Please let me know and I’ll have a look at what I can do about it. I thought I was going to have to do lots of log analysis to find out why it was slow.

What I was saying is that when a web page is slow then there are lots of reasons why it could be slow and figuring out what is holding things up is a complicated procedure. I disagree with your comment that everyone understands why it is slow. I certainly don’t fully understand it. If everyone else does understand then you’ve got the wrong guy doing the migration and someone else should take over.

I’m keen to speed up the nlab, but to do that I need decent reports. When a page is slow, or comes up blank, then I need to know what the link was and when it was. Then I can chase it up.

But to suggest shifting to MediaWiki is completely missing the point. I tried thinking of a suitable analogy. Here’s the closest I got.

It’s like you drive to work every day in your Ford. You notice that sometimes it takes a while to get in. You’re rich (or American) and have two cars. Sometimes, you take the Toyota out to the supermarket or the mall and that’s a quick trip. So you figure that you should use the Toyota to drive to work; that’ll save you time on the journey. But there’s a catch. You work for Ford. It’ll not be good if you turn up in a Toyota. So you figure that you’ll take the engine out of the Toyota and stick it in the Ford.

I could stretch the story further. For example, the closest approximation to a mechanic you have is someone who once fixed their car with duct tape and a cable tie.

To be honest, I think that doing the nLab as a Wiki is the wrong way to do it. The right way, which is also the fastest way, is to use a distributed version control system and not use a browser at all. Why send the entire page across the net four times just to add a full stop? The best, and fastest, system would be something like a git or bazaar repository that we each commit to. Then each of us has a local copy, we can make changes as and when we like - no need to be online - and commit them only when we’re happy with them. There’s none of the issues with simultaneous requests, locking pages, or any of that. Most of the work happens locally so is blindingly fast, and when you do send something over the net then you only send the diffs, not the whole thing. Moreover, we could then use the real LaTeX and whatever editory we liked (okay, I know that one can use things like “It’s all text” but if I edit a page and submit my changes then want to edit it again, I get a whole new instance of my editor! Can’t I just reuse the old one?)

The real problem with the wiki model is that all the work is done by the server. The local machine doesn’t get a chance to help out at all. It is assumed to be as thick as two short RAM sticks. It is frustrating to sit in front of an AMD 64-bit dual core machine with memory measuring in gigabytes, and have to send the document off to some machine half a world away just to see if I’ve figured out the correct number of spaces to indent a paragraph (something I never seem to get right).

So if you want to talk about speed, don’t talk about swapping Instiki for something the raccoons in the trash have dreamt up. If you really want the nLab to zip along, let’s talk about git, bazaar, svn, and (dare I say it?) arch. There’s a reason I chose mathforge as the domain name for the test site of the nLab.

But I’m not actually proposing to swap Instiki for Bazaar. There’s one extremely good reason for using a Wiki instead of a DVCS. I alluded to it above when I said that the local machine was considered to be, essentially, incompetent. Only it’s not the local machine that should be viewed as incompetent but the local user. The thought of trying to explain to mathematicians how public key cryptography works in practice is enough to put me off DVCS. The entry level for a Wiki is so much lower than for a DVCS that sometimes it’s worth the trade-off.

(By way of analogy, sometimes I will honestly recommend that someone install something other than Linux on their computer just because I know that if they install Linux then I’ll be their default tech support for life. At least if they install MacOSX then I can say “Sorry, don’t have a Mac.” (Actually, that’s a lie. I do have a Mac, but it’s running Linux).)

I’m not saying that the nLab can’t be speeded up. Of course it can. Migrating to new hosts isn’t a magic bullet that is suddenly going to make it go faster than a Harmisson delivery on day 5 of the final test. The real value in migrating is that it gives us the opportunity to speed up Instiki. I’ve already done some simple steps to do this, but there’s more that can be done. It’s a question of identifying which are going to give the maximum benefit. Rewriting Instiki in PHP (which is really what you are suggesting, whether you realise it or not) is going to give marginal benefit (if any) for a lot of work.

I think I’ve said enough on this. I could go on about how when you edit a page you send a copy of it across the net four times. I could talk about the importance of memory and asynchronous processes. But I’d rather get on with actually doing something to improve the current implementation of the nLab.

Posted by: Andrew Stacey on August 22, 2009 9:52 PM | Permalink | Reply to this

Re: nLab Migration

I don’t know much about maths or category theory, but I do know about computers and web servers.

If the nCatlab is slow it’s nothing to do with sending four pages across the network. That’s nothing. Comparing to Google and Wikipedia is an *entirely* fair comparison. They have plenty of resources at their disposal, but they also have correspondingly more users and network traffic.

There are a number of reasons it might be slow. The most likely are:

1. Insufficient CPU resources on the host
2. Slow database access (instiki uses SQLite[1] which is very slow on write access[2])
3. Instiki itself is a poor performer

Number 2 is by far the most likely. If you want I can help you with determining the source of the slowness.

Your DVCS idea is also very interesting, although without the benefit of hyperlinks, makes it hard for the uninitiated to find their way around.

[1] http://blogs.law.harvard.edu/hoanga/2007/02/07/migrating-instiki-from-one-database-type-to-another/

[2] http://www.rkblog.rk.edu.pl/w/p/sqlite-performance-and-django/

Posted by: Tom Ellis on August 22, 2009 11:40 PM | Permalink | Reply to this

Re: nLab Migration

Actually, I believe that Andrew is taking this opportunity to change from SQLite to MySQL. (Possibly [1] will be useful for this, although I think that we've gotten past that stage.)

Posted by: Toby Bartels on August 23, 2009 12:20 AM | Permalink | Reply to this

Re: nLab Migration

I spent yesterday without turning on a computer, then this morning saw this and this so was in a good mood.

Ah well, it was nice while it lasted.

I don’t know much about maths or category theory, but I do know about computers and web servers.

I know a little about all four. Not a lot, but a little. I have no idea about what everyone else knows, but I have my suspicions and so I sometimes simplify matters a little. On occasion that means that someone who knows more than I do will think that I know less than I do. That’s fine. I’d rather people put things too simply than too complicatedly.

If the nCatlab is slow it’s nothing to do with sending four pages across the network.

It was late. I said this a bit sloppily. What I should have said is that you have to go through the whole process four times. Not just send it four times. So if one step in the process is a little slow, then with an edit it gets four times slower. And my count may have been wrong as well. My point was that on the nLab we probably have a higher edit/show ratio than, say, Wikipedia so our priorities for fixing slowdowns is going to be different to theirs.

That’s nothing.

No. That’s four times nothing.

Comparing to Google and Wikipedia is an entirely fair comparison. They have plenty of resources at their disposal, but they also have correspondingly more users and network traffic.

Please can we stop this!

The reason that this is an unfair comparison is absolutely nothing to do with resources, number of users, or anything like that.

It’s to do with Aims and Objectives. For Wikipedia and Google, one of their primary aims must be (I’m guessing) that they almost never fail to provide the requested service. Since, particularly for Google, there are alternatives, if they get a bad rap then they lose customers. We don’t have that problem. Of course, we don’t want to be awful, but I suspect that most of our heavy users are also our contributors, there currently isn’t any competition, and everyone knows that we’re in the early stages of this project. So it’s unlikely that anyone is going to stop using the nLab just because on occasion they get a slow page.

I imagine that Wikipedia and Google have, somewhere, a simple list of aims and objectives. They probably boil down to:

  1. Reliability
  2. Speed
  3. Ease of Use

Sounds pretty good. And sounds like a good list for the nLab as well. But that’s probably too simple for Google and Wikipedia. They’d probably like to add the words “As darn near 100% as possible” to those (interpreting that as “Even an English cricketer the day after winning the ashes could use this service to look up how great an achievement it was” to the third). So they plough an incredible amount of resources into them and achieve them, or pretty close. We’re probably not that bothered about 100% on speed or ease of use. And “reliability” for us probably means the mathematical content rather than “always being able to connect even on a dodgy mobile in the middle of the Gobi desert”.

Of course we want to get rid of the slowdowns. But we have finite resources to put into this. So we need to think best how to deploy them and how to use them to make the most impact. I’m mainly trying to stifle the idea that it would be good to port Instiki to PHP (aka modify MediaWiki to be able to use itexToMML). Of all the possible uses of our resources, that strikes me as being one of the worst.

At the moment, our resources are currently being used up in answering blog posts.

There are a number of reasons it might be slow.

I know that. The problem is “might”. To properly allocate our resources, we need to identify what they actually are.

  1. Insufficient CPU resources on the host

Actually, it seems that memory is the limiter. But again, that’s resources.

  1. Slow database access (instiki uses SQLite[1] which is very slow on write access[2])

By default, it does use SQLite and the original (and current) nlab uses this. The new one uses MySQL. The ‘harvard’ link that you gave was one that I came across while doing the migration and it was, I should say, completely useless.

In fact, the whole point of this blog post was that I had to hack the database conversion and I wanted to know if I’d gotten it right. So I wanted people to go and have a look to see if things looked the same or different, since that would be a lot quicker than me doing it all by myself. So far all I’ve gotten is a load of random whinges about speed, which may not even have pertained to the new server! John hasn’t been back to confirm or deny the hypothesis that he was actually using the original server to do the ‘database of categories’ link.

  1. Instiki itself is a poor performer

No one disputes the fact that Instiki doesn’t exactly emulate a Steve Harmison delivery. But I think it’s best to spend my time tracking down the actual bottlenecks and sorting them out.

Number 2 is by far the most likely.

As Toby says, we’re now on MySQL.

If you want I can help you with determining the source of the slowness.

Do you mean this just with regard to the databases, or is this a more general offer? If the former, thanks but we’ve already dealt with that. If the latter, then great! What can you do?

Your DVCS idea is also very interesting, although without the benefit of hyperlinks, makes it hard for the uninitiated to find their way around.

Errr … last I looked, PDFs had hyperlink capabilities. In fact, the idea of doing all of this in a browser is just plain stupid apart from the fact that just about everyone knows how to use a browser (though I could tell you a story about someone who got the instruction “point the mouse at the browser icon” slightly but hilariously wrong) and educating them to use a DVCS would be an absolute nightmare. It’d be like telling them to use the \( syntax instead of $ for inline maths in LaTeX documents, only much, much worse.

The statement at the beginning of the Google Wave video got it just about right:

You will forget that you are looking at a browser.

Quite right! I’d love to forget about looking at a browser.

I use Emacs for writing papers, vim for emails and short documents, LaTeX for formatting, xournal for taking notes, refbase for organising them, gobby and jarnal for collaboration, bazaar for keeping track of revisions, mutt for reading email, perl for hacking, xpdf and xournal for displaying presentations, gimp for drawing, octave for computations, if I have to I might use openoffice or abiword. Why do I need firefox/opera/konqueror? Only because no-one else uses all those tools and on occasion I have to be compatible with everyone else. It’s like having to have access to a Windows machine because every so often someone sends you a document that openoffice or abiword can’t cope with. I’d far rather only ever use Unix, but I have to be able to talk to other people once in a while.

I apologise if it feels like I’ve bitten your head off here. But so far we’ve had 38 comments on this post (admittedly many by me) without any actual useful information. Meanwhile, over on the forum Toby has been busy finding discrepancies between the old lab and the migrated one. If I didn’t keep getting distracted I might be able to track down what was causing them and fix ‘em.

Posted by: Andrew Stacey on August 24, 2009 8:45 AM | Permalink | Reply to this

Re: nLab Migration

>If you want I can help you with determining the source of the slowness.

Do you mean this just with regard to the databases, or is this a more general offer? If the former, thanks but we’ve already dealt with that. If the latter, then great! What can you do?

I mean as a more general offer. With access to the server I could help you track down the source of the slowness.

Posted by: Tom on August 24, 2009 10:09 AM | Permalink | Reply to this

Re: nLab Migration

I mean as a more general offer.

Fantastic! Welcome on board. I hereby promote you to into the exalted group of Lab elves (service department).

Posted by: Andrew Stacey on August 24, 2009 10:51 AM | Permalink | Reply to this

Re: nLab Migration

I should say to anyone else who wants to join the technical team: please do! No particular skills required, there’s always something that can be done by whatever level.

Posted by: Andrew Stacey on August 24, 2009 10:53 AM | Permalink | Reply to this

Re: nLab Migration

Great! You can get my e-mail address from:

http://www.srcf.ucam.org/~te233/contact/

(I’ll be unavailable for approx the next week though).

Posted by: Tom on August 24, 2009 11:16 AM | Permalink | Reply to this

Re: nLab Migration

You know why the nlab is slow? Great! Please let me know and I’ll have a look at what I can do about it.

It’s fantastic that you are looking into the performance problem with so much energy and expertise.

When listening to experience reports from users currently, it may however be good to keep in mind that they will probably all report on the experience with the old installation of the nnLab. We should have this kind of discussion after we have seriously migrated. Maybe much of the performance problem will have been gone away. (Maybe not, but let’s see then.)

It seems that this particular discussion here was triggered by John’s comment that the new nnLab is as “painfully slow” as the old one. But it seems now that John was acutally accidentally using the old one after all when making that comment.

Personally, I could give a good estimate of how well the new Lab performs after I have seriously worked with it the way I do with the old one. But such “seious work” takes me more time than I can afford wasting for just a test run, so I won’t actually know before we have officially migrated.

Posted by: Urs Schreiber on August 25, 2009 6:56 PM | Permalink | Reply to this

Re: nLab Migration

Well, let’s see. Now at last we migrated and things are getting better.

On the instiki TODO page it says not only

Let’s be blunt: Instiki is kinda slow.

but also

Ruby 1.9 (when Rails becomes 1.9-ready) will provide a huge speed-boost.

But Instiki itself could probably be much accelerated, with some rewriting.

Can one say what the prospects are for “Rails to become 1.9-ready”?

Posted by: Urs Schreiber on August 21, 2009 4:48 PM | Permalink | Reply to this

Re: nLab Migration

In light of the tone of my previous comment, I should make it plain that this one is very definitely intended humorously.

It’s like shooting fish in a barrel (which, according to MythBusters, is actually very difficult) …

  1. Google “rails ruby 1.9”
  2. Click on the second link (Rails 2.3 release notes in case the order’s changed since I wrote this)
  3. There is no step 3 …

(read the Instiki installation instructions for part of the joke.)

Actually, step 3 involves scrolling down, or clicking on ‘Chapter 3’ in the sidebar.

If you do that, you’ll see that Rails 2.3 is ready for Ruby 1.9 but that any given application may not be. If you subscribe to Jacques’ RSS feed for Instiki updates, as I’m sure all of you do, you’ll see that Instiki now uses Rails 2.3. So in fact, all that’s needed is checking the plugins and probably modifying them a little to make them work with Ruby 1.9. But you’ll also see that upgrading Instiki to Rails 2.3 was not quite a walk in the park so given that Jacques is quite busy at the moment, it may take a little while before it gets upgraded to 1.9. On the plus side, since Jacques’ version of Instiki is now the official one then there’s probably more than just Jacques doing these updates so maybe someone else will do it for him. Wouldn’t that be nice?

Posted by: Andrew Stacey on August 21, 2009 9:41 PM | Permalink | Reply to this
Read the post nLab Migration Done
Weblog: The n-Category Café
Excerpt: The nLab wiki has successfully migrated to a better server.
Tracked: September 1, 2009 10:15 AM

Post a New Comment