Okay, so looking through the week’s log for bots (bot, spider, crawler), I get 33,517 hits (actual time period: 11th December 6:25am to 16th December 11:27am, so that’s an average of a little over 4 hits per minute). These break down as follows:
25690: show
2953: new
1345: history
1290: edit
1029: source
395: cancel_edit
81: files
67: atom_with_headlines
53: recently_revised
15: save
13: atom_with_content
There’s a few that I’ve missed out in between - there are clearly some bad links to the nlab.
I’d say that only show should show in that list. source could, but I don’t really see why. The saves are a bit worrying - I’m going to check those!
Is it obvious why the spider aren’t just hitting the cache (in which case, they should not slow down the system at all)?
Are they asking for all revisions of some page (or whatever), that would entail a large percentage of cache-misses?
I ask, just because it seems to me that, if they are operating correctly, spiders shouldn’t lead to an undue slowdown. Maybe I’ve been remiss about
<meta name="robots" content="noindex,nofollow" />
directives.
In any case, is it clear that your 3-queue scheme is better than having one queue, with a larger number of worker processes? (I.e., do these spiders insist on making multiple simultaneous connections, or do they access the nlab serially?)
That’s pretty strange. I have not been able to reproduce the problem.
Obviously, having dummy /app/helper/admin_helper.rb and /app/helper/file_helper.rb files does no harm, but I’m puzzled by the fact that they seem to be necessary for you.
I just created a fresh install of instiki (0.19.3) on mac osx snow leopard. Ruby 1.9.3 (ruby 1.9.3p6 (2011-12-13 revision 34018) x86_64-darwin10.8.0).
After running through the usual steps to download dependencies, tried kicking off ./instiki. Consistently got the following error: ...instiki/instiki-0.19.3/vendor/rails/activesupport/lib/active_support/dependencies.rb:186:in require’: cannot load such file – admin_helper (LoadError) from …instiki/instiki-0.19.3/vendor/rails/activesupport/lib/active_support/dependencies.rb:186:in require'
from ...instiki/instiki-0.19.3/vendor/rails/activesupport/lib/active_support/dependencies.rb:293:in require_or_load’ from …instiki/instiki-0.19.3/vendor/rails/activesupport/lib/active_support/dependencies.rb:252:in depend_on'
from ...instiki/instiki-0.19.3/vendor/rails/activesupport/lib/active_support/dependencies.rb:163:in require_dependency’ from …instiki/instiki-0.19.3/vendor/rails/actionpack/lib/action_controller/helpers.rb:198:in default_helper_module!'
After banging my head for several hours and inserting debug statements, I figured that it is looking for the file ‘admin_helper’ which does not exist in any folder. I created a dummy file /instiki-0.19.3/app/helpers/admin_helper.rb with an empty modue AdminHelper. Similarly, created another file /instiki-0.19.3/app/helpers/file_helper.rb
Now instiki starts up fine, and I am able to create a few pages.
From time to time, the nLab gets a whole host of spiders and other bots crawling all over it. While I understand that they’re part of what makes the internet work, they can be a bit annoying and slow down the server for everyone else. So I thought of channelling requests a little more cleverly than I currently do. At the moment, I use a global queue in passenger which is fine until all the slots get a slow request. So what I thought was to have a semi-global queue with slow requests (like feeds and lists) and bots being handled by a few dedicated processes, normal requests by some others, and maybe a “priority” list as well. Since passenger doesn’t do this itself (it either has global queue or individual queues) I think that what I’d have to do is to have three virtual versions of the nLab, at least as far as apache and passenger are concerned. Then apache would examine the request and classify it according to which type it was and send it to the right version of the nLab. Passenger wouldn’t know that these are the same so would have a global queue for each, and that way requests get segregated and so don’t hold up others in other segments. The way that I’d have three virtual versions is simply with symlinks in the filesystem: “nlab”, “nlabPriority”, and “nlabSlow” would all be symlinks to the same instiki installation.
Can you see any immediate problems with that? As far as Instiki is concerned, it’s just like being run under passenger as there will be multiple instances of instiki running concurrently, which is what already happens. So that shouldn’t be affected. Apache, also, eats this sort of thing for breakfast, and passenger can cope with different programs as well. So I don’t see an immediate flaw.
(Of course, it may be that this won’t solve the blockage, but it’s less drastic than moving servers which is the other option.)
I notice that Frederic has now closed this and says that the correct way to handle them is with movablelimits="false". So that makes this now an itex bug again!
I’ve create several forums with admin account. It seems to show up on the top with recently created orders. but how can I change the order of forum ? and how can I delete the specific forum ?
When I sign up for new user, notification mail is failed to deliver.
(development environment works fine. because it ignore to fail)
And site says:
We're sorry, but something went wrong.
We've been notified about this issue and we'll take a look at it shortly.
And production.log is
Started POST "/users" for 118.36.184.24 at Thu Dec 01 07:15:34 +0900 2011
Processing by UsersController#create as HTML
Parameters: {"commit"=>"Sign up", "authenticity_token"=>"ktzSyfq2jKdHmHII7m3EpvxHF3loOUR2rx9P8dkBfGA=",
"utf8"=>"\342\234\223", "user"=>{"openid_url"=>"", "password_confirmation"=>"[FILTERED]",
"password"=>"[FILTERED]", "login"=>"jufoot3", "email"=>"jufoot3@gmail.com"}}
Rendered user_mailer/signup_notification.text.erb (0.3ms)
Sent mail to jufoot3@gmail.com (44ms)
Completed 500 Internal Server Error in 102ms
Errno::ECONNREFUSED (Connection refused - connect(2)):
app/models/user/states.rb:41:in `do_activation'
app/models/user/activation.rb:8:in `set_first_user_as_activated'
app/controllers/users_controller.rb:40:in `create'
PS. Do I have to separate this post from this thread ?..
resolves to a “site” named “localhost”, which is what you created when you ran the bootstrap script (with the default responses).
Connecting to
http://127.0.0.1:3000/
gets you to a different site (“127.0.0.1”) which doesn’t exist.
Since you want the URL for your site to be
http://peppercode.net:3000
that’s what you have to input, when you run the bootstrap script.
Later, you could set up another site (by going to http://peppercode.net:3000/sites), say:
http://foo.peppercode.net:3000
(where foo.peppercode.net resolves to the same IP address as peppercode.net), and run both forum sites off the same HB installation. I’m running three different sites (each with multiple forums), off this one Heterotic Beast installation.
and respond that you want to keep the existing database.yml file. This will use SQLite3, instead of MySQL (it also uses separate database files for development and for production, but you could change that by editing the file).
If you can get Heterotic Beast to work with SQLite3, then we can narrow down the problem to your MySQL setup.
and I created two databases in MySQL with below command.
create database heterotic character set utf8;
create database heterotic_test character set utf8;
create user 'heterotic2'@'localhost' identified by 'my_password';
grant all privileges on heterotic.* to 'heterotic2'@'localhost';
grant all privileges on heterotic_test.* to 'heterotic2'@'localhost';
Try running the server in the foreground in development mode; ie
script/rails server
instead of
script/rails server -d -e production
and report back what you see on the commandline.
I’ve done a fresh install of Heterotic Beast twice, now, and I can’t seem to reproduce your behaviour. My suspicion is that something is f%#@’d up with the communication between HB and MySQL.
And, if you paid attention, as the bootstrap script executed, it told you exactly what the problem was:
Time to create your administrator account.
Login: |admin|
Password:
*****
Email: |admin@example.com|
admin@example.com
The user didn't validate for whatever reason. Fix and call user.save!
Password is too short (minimum is 6 characters)
You failed to create the initial “admin” user, whose presence is needed, in order for Heterotic Beast to function.
I’ve updated Rails to 3.1.1 now. and cloned from GitHub.
but same problem.
I’ve created MySQL database (heterotic, heterotic_test) manually.
Here is the app:bootstrap log. (sorry for long text again)
juhl@jufoot:/var/www/heterotic_beast$ sudo ruby bundle exec rake app:bootstrap
Bootstrapping Heterotic_beast...
1) Create database.yml config file.
2) Load Database Schema.
3) Setup the Application Database.
It looks like you already have a database.yml file.
Would you like to CLEAR it and start over? [y/n]
y
Host name: |localhost|
This same database will be used for your DEV and PRODUCTION environments.
The test database name will be inferred from this database name.
Database name:
heterotic
User name: |root|
heterotic2
Password:
********
Socket path: (blank by default)
Your databases:
Development: 'heterotic'
Production: 'heterotic'
Test: 'heterotic_test'
Now it's time to load the schema into the development and test databases. Any existing data will be
OVERWRITTEN. Are you sure you wish to continue? [y/n]
y
mkdir -p /var/www/heterotic_beast/log
Attempting to reset the database.
heterotic_test already exists
-- create_table("brain_busters", {:force=>true})
-> 0.0080s
-- create_table("forums", {:force=>true})
-> 0.0341s
-- add_index("forums", ["position", "site_id"], {:name=>"index_forums_on_position_and_site_id"})
-> 0.0571s
-- add_index("forums", ["site_id", "permalink"], {:name=>"index_forums_on_site_id_and_permalink"})
-> 0.0040s
-- create_table("moderatorships", {:force=>true})
-> 0.0034s
-- create_table("monitorships", {:force=>true})
-> 0.0036s
-- create_table("open_id_authentication_associations", {:force=>true})
-> 0.0034s
-- create_table("open_id_authentication_nonces", {:force=>true})
-> 0.0032s
-- create_table("open_id_authentication_settings", {:force=>true})
-> 0.0051s
-- create_table("posts", {:force=>true})
-> 0.0039s
-- add_index("posts", ["created_at", "forum_id"], {:name=>"index_posts_on_forum_id"})
-> 0.0039s
-- add_index("posts", ["created_at", "topic_id"], {:name=>"index_posts_on_topic_id"})
-> 0.0040s
-- add_index("posts", ["created_at", "user_id"], {:name=>"index_posts_on_user_id"})
-> 0.0042s
-- create_table("sites", {:force=>true})
-> 0.0042s
-- create_table("topics", {:force=>true})
-> 0.0049s
-- add_index("topics", ["forum_id", "permalink"], {:name=>"index_topics_on_forum_id_and_permalink"})
-> 0.0199s
-- add_index("topics", ["last_updated_at", "forum_id"],
{:name=>"index_topics_on_forum_id_and_last_updated_at"})
-> 0.0048s
-- add_index("topics", ["sticky", "last_updated_at", "forum_id"],
{:name=>"index_topics_on_sticky_and_last_updated_at"})
-> 0.0049s
-- create_table("users", {:force=>true})
-> 0.0071s
-- add_index("users", ["last_seen_at"], {:name=>"index_users_on_last_seen_at"})
-> 0.0046s
-- add_index("users", ["site_id", "permalink"], {:name=>"index_site_users_on_permalink"})
-> 0.0054s
-- add_index("users", ["site_id", "posts_count"], {:name=>"index_site_users_on_posts_count"})
-> 0.0057s
-- initialize_schema_migrations_table()
-> 0.0088s
-- assume_migrated_upto_version(20090317123901, ["/var/www/heterotic_beast/db/migrate"])
-> 0.0027s
We need to create a default 'site' for your users to blog and forum and whatnot.
Or for you to test on, if you're a developer.
If you are a developer, and you set the host to anything other than 'localhost', please make sure to
add an entry to your /etc/hosts file, f.e.: '127.0.0.1 test.local'
Host: |localhost|
Site Name: |localhost|
Site created successfully
#<Site id: 1, name: "localhost", host: "localhost", created_at: "2011-11-28 08:12:09", updated_at:
"2011-11-28 08:12:09", topics_count: 0, users_count: 0, posts_count: 0, description: nil, tagline: nil>
Time to create your administrator account.
Login: |admin|
Password:
*****
Email: |admin@example.com|
admin@example.com
The user didn't validate for whatever reason. Fix and call user.save!
Password is too short (minimum is 6 characters)
***** debugger statement ignored, use -d or --debug option to enable debugging
/var/www/heterotic_beast/lib/tasks/bootstrap.rake:39
User created successfully
#<User id: nil, login: "admin", email: "admin@example.com", crypted_password: nil, salt: nil,
created_at: nil, updated_at: nil, remember_token: nil, remember_token_expires_at: nil,
activation_code: nil, activated_at: nil, state: "passive", deleted_at: nil, admin: true, site_id:
1, last_login_at: nil, bio_html: nil, openid_url: nil, last_seen_at: nil, website: nil, posts_count:
0, bio: nil, display_name: nil, permalink: "admin">
==========================================================
Heterotic_beast is ready to roll.
Okay, thanks for bootstrapping! I know I felt some chemistry here, did you?
Now, start the application with 'rails server' and get to work!
I’ve just installed heterotic_beast on my ubuntu server. but It just redirect page many times.
domain:3000/sites <-> domain:3000/session
as a result, I can’t see any web page but browser connection error.
here is the server output log.
juhl$:/var/www/heterotic_beast$ sudo rails server
=> Booting Mongrel
=> Rails 3.0.7 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
Started GET "/" for 118.36.184.24 at Sun Nov 27 21:30:13 +0900 2011
Processing by ForumsController#index as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Completed in 3ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:13 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 3ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:13 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (31.0ms)
Completed in 45ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:13 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 2ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:14 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (11.0ms)
Completed in 24ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:14 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 2ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:14 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (11.8ms)
Completed in 26ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:14 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 2ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:14 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.3ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (11.6ms)
Completed in 26ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:15 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 2ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:15 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (11.0ms)
Completed in 25ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:15 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 2ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:15 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (11.9ms)
Completed in 26ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:15 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 2ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:15 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (10.2ms)
Completed in 23ms
Redirected to http://peppercode.net:3000/sites/new
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/sites/new" for 118.36.184.24 at Sun Nov 27 21:30:16 +0900 2011
Processing by SitesController#new as HTML
Redirected to http://peppercode.net:3000/session/new
Completed 302 Found in 2ms
/var/www/heterotic_beast/app/controllers/application_controller.rb:41: warning: already initialized constant HTML
/var/www/heterotic_beast/app/controllers/application_controller.rb:44: warning: already initialized constant XHTML
Started GET "/session/new" for 118.36.184.24 at Sun Nov 27 21:30:16 +0900 2011
Processing by SessionsController#new as HTML
Site Load (0.2ms) SELECT `sites`.* FROM `sites` WHERE (host = 'peppercode.net' or host = '')
Rendered sessions/new.html.erb within layouts/application (10.5ms)
Completed in 103ms
Redirected to http://peppercode.net:3000/sites/new
According to Frédéric, it’s a feature, not a bug. I could change \widehat{} (and its cousins) to wrap their output in an <mstyle displaystyle='true'>, but that might have unintended side-effects.
‘Cept it’s not a Firefox bug; it’s a Gecko Core (MathML Component) bug. As filed, no one relevant will see it. If you, as the Reporter of the bug, can reclassify it, maybe it will have a fighting chance of getting some attention.
When there are style_additions then the surrounding <style> ... </style> appears to get escaped. I noticed this on a published web: in the source, it says:
(My word, but that’s a complicated form. “What did you do?” I looked at a MathML web page. “What happened?” It didn’t look right. “What should have happened?” It should have looked right.)
Errr … the pages table doesn’t have a revised_at field. It has an updated_at field. But this seems to get updated whenever the record gets changed. So if I start editing a page then the pages table gains a locked_by and a locked_at entry, and the updated_at entry is set to the same as the locked_at entry. Then when I cancel editing, the locked_at entry is set to NULL (the locked_by isn’t, though that’s probably not an issue) but the updated_at entry is left as it is. So the updated_at entry in the pages table does not necessarily point to the timestamp of the latest revision.
292 posts found