Forum: Instiki – Topic: Idle Instiki was using 17% of CPU on a shared host

Problems not related. Sysadmins fixed a server problem. Instiki with high CPU usage has not recurred after that one incident.

Thanks

Forum: Instiki – Topic: How do I bulk-clear the Instiki cache

Thanks, I did that.

It seems to be working fine, but one thing puzzles me. About a day after I cleared everything out as per the above, the full tree under instiki/cache/views/azimuth got repopulated – presumably when the site got crawled.

But the storage folder didn’t get repopulated. Or rather, now there is just a single file in it: azimuth-markdownMML-2015-02-12-11-11-03.zip, with a timestamp of yesterday.

Before I cleared out storage, there were tons of files it in, it looked something like one file per page of the wiki, though I didn’t look closely.

Does this make sense; are there any signs of ill-health here? What is the function of the storage folder?

Thanks.

Forum: Instiki – Topic: How do I bulk-clear the Instiki cache

Can we make the instructions more specific, because I’m cautions about disrupting a structure that I don’t understand.

How about:

rm instiki/storage/*

rm -r instiki/cache/views/azimuth

There’s a whole structure of directories and files under the last directory, that’s okay to wipe out?

Forum: Instiki – Topic: How do I bulk-clear the Instiki cache

I did a bulk update in the database to change our forum URL from one location to another. (Updated the field Content in the Revisions table.)

Now a large number of entries in the cache are stale.

I grepped in the cache directory tree for the old url, and found it present in files in several subdirectories of the cache/views folder for the web in question. I saw it in the subdirectories source, tex, revision, …

In addition, the old URL is present in files in the top-level folder “storage.”

What is the cleanest / simplest / safest way to purge the entire cache?

Thanks!

Forum: Instiki – Topic: Idle Instiki was using 17% of CPU on a shared host

Good questions. Thanks. Research pending…

Forum: Instiki – Topic: Idle Instiki was using 17% of CPU on a shared host

But this isn’t my main concern tonight, which is off-topic: our shared host is crashing everyday, due to too many sleeping processes and not enough memory. They claim that 5G is enough memory, and are promising to fix it… Grr.

Forum: Instiki – Topic: Idle Instiki was using 17% of CPU on a shared host

Noticed this, then I touched tmp/restart.txt and usage went down to basically zero.

Forum: Instiki – Topic: Feature Requests

Thanks!

Forum: Instiki – Topic: Feature Requests

Background: The Azimuth wiki is getting consistently hit by a spammer promoting “coffre fort.” I’ll do what I can with keyword blocking, but the guy is creative with words.

Main point: The IPs are all in the database at www.stopforumspam.com.

Possible feature request: Option to have this web service called when validating an IP address.

One question is what the performance hit would be – what is the uptime of this service, and what is its response time.

A drawback of doing this is introducing a further dependence through the internet on an external system.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

After considering these requirements, and enumerating the commands that are on the Edit Webs page, let’s take a breath, and then move on to discuss the SQL interpretation of these commands in another discussion thread.

Thanks for listening!

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

Please if anyone sees any problems with these requirements, or has suggestions for simplifications, enhancements or new use cases – do tell!

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

I believe that these specifications cover the story I mentioned above, which is to publish from one installation to another based on category:publish. They also cover more. For instance, multiple people could each have their own staging installation, and with these scripting commands, they could publish page sets to a common target installation.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

c) Import a page set P into I’.

Options: (1) overwrite existing pages, and clobber their histories, (2) append to the existing histories, (3) fail if the pages already exist.

There is a constraint here. If the export contains full histories of pages, it doesn’t make sense to use option (2) to append these histories to an already existing history. So if option 2 is specified, then an error should be generated if the imported data contains full histories and the targets already exist.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

b) Delete the page set for (W,C) from the target installation I’.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

Functions to support Use-Case 2.

a) Export. Given an Instiki instance I, define a page set by PageSet(W,C), where W is a list of web identifiers and C is a set of categories. This consists of all pages in webs W that fall under one of the categories in C.

The export function can be given a list of page-set-specifiers (W1,C1), …, (WN,CN). Allow the wildcard specification * for any of the arguments Wi or Ci.

Allow the export to also specify whether the full history should be exported, or just the most recent versions of the pages.

Output of the export statement may be based arounda collection of SQL insert statements, which can be applied to the database for another Instiki installation I’. (To keep the levels clear, let’s defer talk about the SQL specifics to a followup thread.) But as the following functions suggest, some more meta-data may be needed.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

Functions to support Use-Case 1.

TODO: just read off the functions from the Edit Webs page, and specify them as abstract commands here.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

Use Case 3: Develop new and more powerful methods for manipulating the data that constitutes Instiki webs.

This is open-ended, and ideas can grow here as we go along.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

The story behind use-case 2.

I maintain one Instiki installation that is competely private, password protected, and not referenced from anywhere on the internet. This is the staging area for Instiki webs that am planning to publish. For some things, I’d rather not think aloud in the open, and would prefer to release when I deem them completed.

I will maintain another Instiki installation for the published webs. It’s also a security precaution, to make sure that my staging webs won’t get compromised if the public installation gets hacked, attacked or spammed.

Now in the staging webs, I will build a full image of what the published webs will contain. To accomplish this, I will just add the line “category:publish:” to any page in a staging web that I want to get published.

So: I’ll want a scripting function that exports all of the data from one installation that is tagged with “publish”, and another one that imports the data into an existing Instiki installation.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

Use Case 2: I want to be able to write scripts to export subsets of the data on one Instiki installation, and import them into other Instiki installations.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

Sidenote: The saga behind use case 1.

The concrete thing which is motivating me here is that I am running Instiki on hosted server account, and want to manage it securely using https, but my host – RailsPlayground – which has been fantastic in every regard except for this one is having a bunch of technical problems getting https to work well. First, they don’t yet support SNI (server name identification) in Apache, so a dedicated IP, for 2 bucks per month, is required in order to get SSL to work. But worse than that is that they are claiming that the dedicated IP, along with a wildcard SSL certificate for the whole domain, will not work for the whole domain – that actually I’ll need a 2 dollar per month dedicated IP for each application running on each subdomain. On top of this, they say that for each login, they can only provide one dedicated IP!

I was also willing to use lynx and regular http to localhost, but lynx isn’t supported on this shared host. It’s rather confused and annoying. I’ve pleaded my case with them, and the admins are sympathetic, so they are going to escalate the issue internally.

In all fairness, I should say that overall I’m really happy with their service, which provides 5 shell/cPanel accounts, lots of bandwidth and ok storage. Each account gets 200G per month of bandwidth, and 6G of storage. Stuff is backed up (so they say), and custom php.ini is supported. Leaving https aside, these guys are real experts on managing Rails applications, and general system administration, and their level of responsiveness is impressive. The Azimuth wiki has been running without incident (well, just one brief incident, where the web server had to be restarted because Passenger stopped spawning processes), appears more responsive than it was on the previous VPS (at RailsPlaygrond also) that was running the Azimuth wiki, and does not manifest the “Smoke” error to do with growing cookies.

But even apart from this Saga, I rather like writing scripts to do things, so I’d be pretty happy administering Instiki this way.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

Use Case 1. I want to write scripts to perform the standard administrative operations that are made available from the Edit Web page of Instiki.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

There are three general use cases that motivate this feature for me.

Forum: Instiki – Topic: Requirements for a server-side Instiki scripting package

I would like to develop a server-side scripting packaging for managing Instiki webs, which will be based on direct calls to the database.

If anyone knows of such a thing already, or even the rudiments of it, do tell – because I’d be glad to use something already existing that meets the bill.

In this thread, I will spell out the requirements that I have in mind. I welcome any comments or suggested modifications to these requirements. Then, in a followup thread, we can talk about how these requirements (for commands) will get translated to SQL update and query statements. Then, in another thread, we can talk about implementation approaches (at the moment I’m leaning towards using Python along with its interface to Mysql).

In any case, on with the requirements…

Forum: Instiki – Topic: How to develop an ABC music plugin for Instiki?

Thanks for the good information.

All I’ve worked with is the script abc2ly.py, which comes with Liliypond. It parses ABC pretty well – though not perfectly. It’s fairly simple, in that it generates a Lilypond script. Then the “fat” app Lilypond does the heavy lifting, when given this script as input, it produces a PDF with beautifully typeset music notation.

So a klunky solution would be to have a Maruku ABC extension that just calls this script, then fires off a Lilypond process, and stores the pdf somewhere in the file area. It would be good to employ some sort of caching scheme for the rendered files, so that this bloated process would only be run once for each exact instance of an ABC source file on a given page. To spell out the idea here more specifically, a crude implementation could be achieved by having a directory for each instiki page, which would contain the source text for all ABC snippets on that page, along with their associated rendered PDF files. When a page is reprocessed, it would match the new ABC snippets against the stored ones, and use the rendered PDF files if there was an exact match. Of course there are nicer ways to do this, but what I just said does define the idea.

If the Maruku filter generates PDF, can that be embedded in the Instiki output in a nice way?

This solution is klunky in the sense that it shells out to some heavy processes, and requires them to be installed on the server. On the other hand, there is a coding minimalism to it – music typesetting is a huge challenge, especially to do well – in that the whole buck gets passed to Lilypond.

This could be a default approach in the absence of something nice that would be more integrated with the Wiki.

Forum: Instiki – Topic: How to develop an ABC music plugin for Instiki?

Does Instiki have a general interface / API designed for writing plugins?

Forum: Instiki – Topic: How to develop an ABC music plugin for Instiki?

I’m interested in using Instiki for a wiki that includes sheet music. I like the ABC notation, which is a pure ASCII format, with a defined standard, that’s widely used.

For example, see this link for a sample of the format, the output it renders. (It’s also a really nice Balkan song, Jovano Jovanke.) The ABC converters also generally produce MIDI also, but this is of secondary interest to me.

I realize that this request is out of the context of a math-focused wiki, so potentially the only developer for this would be me. Not sure if I’m up for making the leap to Ruby / rails development, but possibly…

As a starting point, I’m interested in discussing here how one might go about implementing this as an extension / plugin to Instiki.

For reference, PmWIki implements an ABC plugin, here.

But I want to stick with – Instiki.

Still it would be interesting to see how they accomplish it.

A long time ago I wrote an ABC plugin for MediaWiki. I piggybacked on a script abc2ly which is distributed with LilyPond (GPL music typesetting program). This script takes an ABC file as input, and generates a PDF file. My extension was simple but heavy-handed – it called the script, cached up the generated PDF file, and then included it on the page. (Many years later they archived this extension, for being unsupported and a security risk.

Another source of ideas could be from the Traditional Tune Archive, which uses Semantic MediaWiki, and has an ABC plugin. That page has a link to an open source javascript component for rendering ABC notation. However, when I clicked on the link for the actual wiki, it took me to a page with some music and text on it, and then…froze completely. Which led me to wonder whether that whole rig is half-baked.

Forum: Instiki – Topic: Is there a server-side file-cache for internal files?

Answer to my question: there is an application caching mechanism in place.

Touching tmp/restart.txt forces it to restart.

Forum: Instiki – Topic: Is there a server-side file-cache for internal files?

Hi I’m using Instiki with Passenger, on a hosted account.

In config/database.yml, I had the incorrect credentials set, let’s say that I had the username coded as abc.

When I go to the Instiki home page, I get the Smoke error. Checking in log/production.log, I see the error message about the wrong credentials abc.

Then I go to fix database.yml, to change the username to xyz.

However, I get the same Smoke error, and the log file still complains about username abc.

Then I go out for a long walk, and when I come back, it fails with a message about xyz.

Do you know of any components (Passenger?) that would be caching up the contents of the database.yml file, and then expiring the cache after some time?

Forum: Instiki – Topic: Question marks in Instiki page titles?

I recall hearing some advice about avoiding question marks in page titles. Is this well-founded? What are the issues, and what are the workarounds.

One possibility that I could imagine are problems with the question mark in the URL string, leading to invalid http requests. Another is that certain parts of the code, e.g. search, interpret the question mark as part of a regular expression.

Forum: Heterotic Beast – Topic: Evaluating HB for Azimuth Forum

Also, I’m not evaluating now how important these features are. Once I have a clear picture of what HB can do now, and what it potentially could do without a lot of extra effort, then I’m hoping to adjust our Vanilla-based expectations to the HB.

Regarding categories, for example, we’re very used to it, and take it for granted, but I’m not convinced that it’s a really important feature. For example, although it’s nice to think that I can filter the posts by category, I have never actually done this in practice.

The other uses of categories I mentioned are still valid, though I could probably make a case for workarounds (a separate forum called Technical) or doing without them.

Previewing is just a nice-to-have.

Moreover, we can assess later, now I’m just trying to learn how it works and throw out ideas as they come up.

Thanks

By the way, I meant to change the title of this thread from “Evaluating HB for Azimuth Forum” to “Discussion of HB for Azimuth Forum,” but when I clicked on the Edit link, it takes me right to the Login screen. However, this bug is not present in the instance of HB that we now have running.