The n-Category Café

Both MathSciNet and the ArXiv index citations per paper, but the arXiv’s citeseer index seems to be disfunctional at the moment. Maybe the effort died as per lack of funds. These at least give an indication of the number of citations to the particular author or the paper in question.

Any citation datum is, at best, a gross measure of quality; sort of like “the woman with blonde hair” is a description of a particular female. Or that blonde hair is the only measure of attractiveness or intellectual ability. (Spare us the jokes, please).

Administrators want easy measures of quality. It is part of our job as mathematicians to educate the public (and our bosses) about the meaning of measure or the measure of meaning. Detailed measures of the quality of research require time, patience, and an extreme lack of prejudice on the part of the reviewer. Consequently, quality is best described in prose rather than numbers. On the other hand, a variety of numbers can help delineate a multi-dimensional image of quality.

The IMU released this report this report on the misuse of Citation indices. Department chairs should provide that information to their Deans.

An item of continuing concern are the roles of blogging and Wiki’ing in evaluation of Promotion and Tenure. While these venues are NOT peer reviewed, they are reviewed. Some very popular blogs, such as this one, are providing a service to the greater community and improving science and science communication. However, popularity and indeed the number of comments on a given blog are not measures of the quality of the entry. If they were, this blog might be the “Animals that are verbs”-category cafe .

Scott said:

Both MathSciNet and the ArXiv index citations per paper, but the arXiv’s citeseer index seems to be disfunctional at the moment.

If I look myself up on MathSciNet I am told that I have been cited by people other than myself precisely zero times. Fortunately I know this to be untrue, so I won’t slit my wrists just yet.

The MathSciNet citation count has the same problems as the others - the category theory journals are not among those considered to “count”. Presumably MathSciNet uses the same “AMS citation database” that Joachim refers to in his message. Anyway, the list of journals it deems worthy of counting can be found here and does not include TAC, Cahiers, Applied Categorical Structures or ENTCS (for example).

On top of this, I am falling foul of point (**) that Joachim makes:

The citation count…even fails to identify preprint citations to papers later indexed. (E.g. paper A cites preprint B. When B is published in an indexed journal the citation from A does
not count.)

I make my papers available on the arXiv long before they are officially published, and a google search shows I have quite a few citations for the preprint versions from before the published version has appeared. So these don’t count.

Furthermore, the “valid” journals are, in my experience, the ones that have the longest delay between submission and publication, which makes this particular problem even worse. I have a paper which was part of my 2002 thesis, which appeared in Math. Proc. Cam. Phil. Soc. in 2006. Another I put on the arxiv in 2003 and it will appear in JPAA in 2009! All the citations I’ve had in the interim don’t count, even if they were from papers in “valid” journals.

This is particularly bad for young researchers as the “invalid” period of time takes up a huge proportion of our active research life so far.

I recently had to list the number of citations (excluding auto-citations) for each of my publications for a grant proposal, so I became aware of this issue with MathSciNet although at the time I didn’t fully understand what the problem was. I spent hours (with the help of a nice friend) trawling the internet to try to find my true citation count. It wasn’t zero.

“If I look myself up on MathSciNet I am told that I have been cited by people other than myself precisely zero times. Fortunately I know this to be untrue, so I won’t slit my wrists just yet.”

OUCH!

But then again, as of this writing, you have 14,323 hits on a single entry of youtube! And Fame is a fickle food.

Is Citeseer functional now?

The UK Computing Research Committee (an expert panel comprising representatives of the British Computer Society, the Council of UK Professors and Heads of Computing, and the Institution of Engineering and Technology) made a submission to the UK Higher Education Funding Council (HEFCE) on their proposed use of citation metrics to assess research performance by UK academics. After reading this report, a rational policy maker would surely abandon any such policy, but I doubt that such will be the case for HEFCE.

www.ukcrc.org.uk/resource/reports/2008-01.pdf

The report includes these statements:

“It would be incompetent and unprofessional to introduce a citation-based Research Excellence Framework until it has been established that there is an adequately complete, consistent and auditable set of data, available from multiple sources free of any commercial bias, that can be relied on to be kept up to date, that includes citations in journals, conferences, PhD theses, industrial reports and institutional repositories — and that assessments based on citation counts from these sources leads to cost-effective assessment of research quality that does not lead to undesirable changes in the way research is carried out or published or on standards or variety of teaching. We do not see any convincing evidence that these criteria have been met.”

and

“We have conducted an exercise on one paper that has over a hundred and fifty “references” on Google Scholar. The ISI citation count is 24; the citation count from ACM is 12 (ACM is the major professional body in international computing). There are no citations in common between the ACM and ISI lists! Many of the extra Google Scholar references are in industrial reports, books, PhD theses and conferences. A study based on a single paper can only be illustrative but it shows that much wider analysis is needed to find robust and valid bibliometric indicators.” (Italics in original.)

I’ve just noticed yet another way in which I fall foul of the MathSciNet system: I have an article in Electronic Notes in Theoretical Computer Science (with Hyland and Power). However, this journal is not even part of the database at all, so the paper doesn’t count as a publication! To clarify: my papers in TAC do at least “count” as publications, even if citations from TAC don’t “count” as citations.

And guess what: in reality this non-publication is my most cited paper, but none of those citations “count”, even if it is from a “valid” journal. For example, this paper has been cited from the Journal of the LMS and Advances, each of which contributes zero to my total citation count of…zero!

I’m clearly going to spend the rest of the night pondering just how badly I’ve been playing this game.

This issue is in the air, here's a cartoon. (But the cartoonist forgot to correct for the number of times that your article or its citation appears in the premier journal of your field when that journal is not included in the citation indexes.)

31 Jan 2008 10:30 am
Why the h-index is little use

“In 2005 Jorge E. Hirsch published an article in the Proceedings of the National Academy of Science (link), proposing the ‘h-index’, a metric for the impact of an academic’s publications.”

“Your h-index is the largest number n such that you have n papers with n or more citations. So, for example, if you have 21 papers with 21 or more citations, but don’t yet have 22 papers with 22 or more citations, then your h-index is 21.”

“Hirsch claims that this measure is a better (or at least different) measure of impact than standard measures such as the total number of citations. He gives a number of apparently persuasive reasons why this might be the case….”

“The first class of exception is people with very few papers. Someone with 1-4 papers can easily evade the rule, simply because their distribution of citations across papers may be very unusual. In practice, though, this doesn’t much matter, since in such cases it’s possible to look at a person’s entire record, and measures of aggregate performance are not used so much in these cases, anyway.

The second class of exceptions is people who have one work which is vastly more cited than any other work. In that case the formula (*) tends to overstate the h-index….”

Be that as it may, at the August 2008 conference at Harvard for the 60th birthday of the geometer Shing-Tung Yau, Iz Singer and Edward Witten gave what were felt to be particularly interesting talks..

Singer’s – “The Interface between Geometry and Physics, 1967-2007″, summarized some of the advances in this area with which he’s been involved. 1967 was the year of a Battelle conference in Seattle on the intersection of mathematics and physics, organized by deWitt and Wheeler. Singer displayed a copy of a 1966 letter from Feynman to Wheeler turning down an invitation to
attend, with the explanation

“I am not interested in what today’s mathematicians find interesting.”

Feynman only had, what, 17 refereed papers published? That’s the way that I remember it, because 17 is my favorite integer. But surely he had more influence than SCI suggests.

And how to quantify the influence of Bohr, best remembered for something that was disbelieved almost a century ago, and simultaneously revered and underappreciated?

I, for one, go out of my way not to play the SCI game. I publish a relatively small number of refereed papers, mostly in venues that my coauthors want to be. I’ve been intentionally, since about 1975 when I started working with Ted Nelson, father of hypertext, putting millions of words (in the old magazine metric of 1 word = 6 alphanumeric characters including spaces) of implicit hypertext in print and radio and video and digital media.

I just don’t explicitly build all the hotlinks, because the proper link infrastructure designed by Ted Nelson was not implemented in the Woirld Wide Web (which omits, for instance, transclusion, bidirectionality, and micropayments).

I am trying to maximize the Darwinian Fitness of my writings.

The SCI hardly lives in the same metric space.

As 2009 is the bicentennial of Darwin, and sesquicentennial of “On the Origin of Species”, I wonder if we’ll have informed debate on the evolution of texts (including science and math papers) by artificial selection.

Since the “The Case of M. S. El Naschie, Continued” thread seems closed to comments, let me throw in this Scaience Fiction/Fantasy quotation which seems to carry sychronicity:

“She held Elsevier’s indigo eyes, willing Elsevier to understand her need, and her longing – and her regret.”

Joan D. Vinge, The Snow Queen, 1980.