The n-Category Café

guest post by Phillip Helbig

Monthly Notices of the Royal Astronomical Society is one of the oldest and most prestigious journals in the fields of astronomy, astrophysics, and cosmology. My latest MNRAS paper was not allowed to appear in the astro-ph category at the arXiv (https://arxiv.org, the main avenue of distribution for scientific articles in many fields) because it was reclassified to a category which is inappropriate for several reasons. This is definitely not due to some technical error, misunderstanding, or oversight. It took more than three months for me to even be told why it had been reclassified, and that only after a well known cosmologist threatened the Scientific Director of arXiv that he would complain to the arXiv sponsors if things weren’t cleared up. Also, there is evidence that the reason I was given is not the real one.

Although I would like my paper to appear in astro-ph, this in not about just my paper. Rather, it is about the question whether the community wants arXiv to decide which papers, and hence which people, are allowed to be part of that community, as opposed to peer review by respected journals such as MNRAS. Below, after some general background on arXiv, I mention some policies which are probably not as well known as they should be, before briefly describing my own odyssey.

Like it or not, many if not most astronomers rely on arXiv at least for learning about new papers; some rely on it exclusively, despite the facts that not everything is on arXiv, that that which is there is not always in the definitive version, and that even if the definitive version is there, then that might not be clear. The last two (and, in some cases, the first as well) can be due to lazy authors or to restrictions imposed by journals as to what version, such as the ‘author’s accepted manuscript’, is allowed to appear; more-definitive versions hence either don’t appear or if so then that fact is not advertised. At the same time, publication in a respected journal is generally recognized as a mark of quality. In fact, the main reason that the quality of papers at arXiv is so high is that most of them will eventually appear in respected journals. So essentially journals are for separating the wheat from the chaff while arXiv has become the main method of distribution, because no subscription is required and because a majority of articles can be found at one website with a reasonably useful interface (the former is crucial for those without access to a subscription to every journal they might want to access and the latter saves large amounts of time). There is thus a problem if standards of acceptance between journals and arXiv differ.

The main reason, at least for me, to have my papers on arXiv is visibility. All else being equal, papers on arXiv are almost certainly read more, and probably cited more, than those which are not. (In a field in which a large fraction are on arXiv, the reason can’t be that only the better papers are put on arXiv. Also, at least a few years after the paper has appeared, having it on arXiv before it has appeared in the journal probably won’t substantially increase the number of times it is read and/or cited due to the only slightly increased time during which it has been available; the increased citation rate is due to the higher visibility from being on arXiv.) The ‘stamp of approval’ comes from the journal. It is easy to distribute open-access versions of the paper, although implementing a robust long-term storage strategy is not. Finding them is more difficult; that would be easiest via arXiv, but author-supplied links at the corresponding ADS& abstract web page are good enough.

People often look for open-access versions of papers via links on such web pages, especially if they want to make sure that they find the official version, not whatever version might be on arXiv; arXiv itself is not an option for papers which are not on arXiv; of course, ADS can be and is used completely independently of arXiv. Lack of visibility at arXiv is a serious disadvantage to an author and such decisions should be made only in extreme cases. (Also, having the paper at arXiv but in the wrong category can be worse than not having it there at all.)

arXiv is under no obligation to allow even a paper which has been accepted by a leading journal in the field to appear in the appropriate category (e.g., astro-ph for astronomy / astrophysics / cosmology), or even to appear at all. There are also some other things which are documented but not as well known as they should be, some things which are at best poorly documented, and inconsistent and/or incomplete recommendations. I think that it is important to alert the community to those in order to counter the impression held by many that everything worth reading is on arXiv and/or if something is not on arXiv then it must be a matter of the author excluding himself from the community, rather than being excluded by arXiv (references intentionally not included to avoid public shaming). (Of course, most who claim that all papers in their field worth reading are on arXiv are not in a position to make that claim, because they don’t read any papers which are not on arXiv.) I suspect that at least some of those things are known by many, but also that there is a fear of criticizing arXiv in public for fear of getting banned, which is the modern-day equivalent of excommunication.

According to the submission agreement, “[t]he Submitter waives…[a]ny claims against arXiv…based upon actions…including…decisions to include the Work in, or exclude the Work from, the repository…the classification or characterization of the Work.” “arXiv reserves the right to reject or reclassify any submission.” In other words, the idea that any serious paper (‘serious’ being defined here as having appeared in a respected journal) can (assuming, of course, that the journal allows it) be uploaded to arXiv is wrong. Also, arXiv reserves the right to reclassify the article, e.g. a paper submitted to astro-ph can be reclassified to gen-ph. Moreover, after such a reclassification, the author is not allowed to withdraw the paper (Steinn Sigurdsson*, personal communication; Eleonora Presani@, personal communication), although that is technically possible (by first ‘unsubmitting’ it then ‘deleting’ it).

Of course, journals also decide which papers they accept and reject. However, the comparison of arXiv with journals is not appropriate, for several reasons: arXiv does not peer-review submissions and claims to do only a minimal amount of moderation. Also, journals offer something between acceptance and rejection, namely the possibility of revision, coupled with the opportunity to discuss the degree of revision, or even reasons for rejection, with the referee(s) and/or editor(s). Of course, revision of an article accepted by a journal doesn’t make sense, but the fact that it is not offered is another piece of evidence that interaction with arXiv shouldn’t be compared to interaction with a journal. Moreover, if an article is rejected by a journal, it is not automatically submitted to another journal, much less without any possibility for the author to choose to withdraw it completely, hence the claim that the various arXiv categories are comparable to various journals with different standards (Eleonora Presani, personal communication) is dubious at best. In addition, there is usually more than one journal of comparable reputation in a given field, so the author has the chance of getting an independent evaluation. In that case, competition between journals is good. In the case of arXiv, however, a monopoly is actually good, as long as it works, because one of the main advantages of arXiv is that there is only one place one needs to look in order to find most papers. This is the main point of my criticism: arXiv’s unique relevance to the community means that excluding a paper from its intended category should be done only under extreme circumstances. arXiv has become one of the most important resources for the astronomical community but that community has essentially no control over arXiv. Great power should be accompanied by great responsibility. Quis custodiet ipsos custodes?

It is possible to appeal a decision. However, the appeals process is not well documented, in part because astro-ph is sometimes seen as a top-level category, sometimes as one of the physics categories. As part of the appeals process, “[e]xtreme cases may be addressed to the appropriate advisory committee chair only”. The value of a successful appeal is questionable, because most rely on the abstract lists for recent papers in a particularly category, either sent via email or available at the arXiv website. As far as I know, a paper reclassified after a successful appeal would not appear in the ‘recent’ list for that category. The main problem with such an appeal, though, is that arXiv is policing itself.

For various reasons, in recent years so-called arXiv-overlay journals have sprung up. There is even one for astrophysics, The Open Journal of Astrophysics, and I have published a review paper there. The basic idea is that there is a robust distribution structure already in place, namely arXiv, so the job of the journal is essentially only to provide refereeing. Such journals usually assume that all potential authors could post their paper to arXiv before submitting it to the journal, but obviously that is not the case. (Some even use the arXiv category as a filter to determine whether the paper could even be considered to be appropriate for the journal.) It is sometimes possible, though usually not widely advertised, to submit to the journal first and submit the paper to arXiv only after acceptance, which is what I did (like many, I prefer to put papers on arXiv only after acceptance). That paper had no problems at arXiv, but based on the reasons I’m presenting here, arXiv-overlay journals are no longer an option for me. (I have long suggested not only that should the possibility to submit to the journal before submitting to arXiv be more widely advertised, but also that the journal should have some sort of agreement with arXiv that any paper accepted by the journal automatically qualifies for the corresponding category at arXiv (after all, the purpose of a journal is publication); alas, the Open Journal of Astrophysics does not plan to pursue that at all: “OJA has no power to compel arXiv to accept submissions, nor would we want to. We see arXiv as the most important resource in astrophysics….”.) Despite the longevity and robustness of some traditional journals, the scientific publishing landscape is changing rapidly. That is a topic for another discussion, but part of it involves arXiv-overlay journals, and wrong assumptions about arXiv mean that a substantial part of the new system is built on shaky foundations.

Those who are interested in high-quality, free-for-readers-and-authors, well organized, open-access journals should check out https://scipost.org/. Is there any valid reason to submit anywhere else? Their astronomy journals are just getting underway; please consider supporting them.

I learned about some of the things discussed above the hard way when my latest MNRAS paper was reclassified from astro-ph to gen-ph (general physics). Of course, I appealed the decision quickly, after discussing the matter with a few colleagues, some of whom assumed that it must have been some sort of technical glitch. It took more than three months before I was told a reason for the classification (after having escalated up to the highest levels of arXiv)§, and more than four before the appeals process finally ended. That paper is not on arXiv, and I don’t intend to post anything else to arXiv before the procedure becomes fairer, more transparent, and more accountable (if it ever does). I had escalated as highly as possible within arXiv before I asked Cornell University (which hosts arXiv) to investigate possible academic misconduct, which led to an email from Eleonora Presani. Her stance is essentially the same as that of Licia Verde#: my accusations themselves don’t seem to have been investigated and authors just have to live with the fact that arXiv can reclassify papers at will and even prevent authors from withdrawing them completely before announcement if they disagree with the reclassification. Unfortunately, Cornell takes the point of view that although Cornell maintains and sustains arXiv, it is not the university’s role to interfere in the moderation or appeal process.

There is evidence that I wasn’t told the real reason why my paper was reclassified$, and no-one with whom I have discussed the matter thinks that arXiv was right to reclassify my paper. (That doesn’t mean that they necessarily have a high opinion of my paper, but those are two separate issues. One colleague stated (though not in reference to my paper) that even the occasional papers which appear in respected journals obviously by mistake should appear on arXiv; that would put pressure on journals to be more careful and also benefit those wishing to critically discuss or refute them.) However, I will discuss that and other aspects (hopefully) unique to my case elsewhere (perhaps in the comments if there is interest), and here concentrate on problems which the astronomical community should recognize and try to correct.

I certainly regard reclassifying a paper which has appeared in MNRAS to a category other than astro-ph, giving reasons for the reclassification only after threat from a famous colleague, and then giving me a completely different reason, to be an extreme case. Thus, I did contact the chair of the physics advisory committee, Robert Seiringer; that he is the appropriate person was also confirmed by Licia Verde. Nevertheless, his response was that he could not investigate disputes involving individual submissions, which was also Verde’s reply to my complaint. Hence, not only is there disagreement between arXiv’s documented appeals procedure and how those involved actually behave, there seems to be no system of checks and balances within arXiv, not to mention the problem that the community, despite relying on arXiv, in practice has no way to arbitrate disputes with it; it is judge, jury, and executioner.

All who believe that my paper should be on arXiv in the astro-ph.CO category if I so desire are encouraged to contact the Scientific Director, the Executive Director, the Chair of the Scientific Advisory Board, and the Chair of the Physics Advisory Committee and complain. It is not necessary to think that my paper is great. It is enough if one thinks that it is not so bad that it should be banned from astro-ph, or even if one can point to worse papers which are in astro-ph. (Of course, if one agrees that my paper should appear in astro-ph.CO, the reason why arXiv has not (yet?) let it appear are irrelevant.)

Of course, my bad experience with arXiv is not the main point. The main point is that arXiv can, and does, make decisions which experts in the field (see third footnote; Tegmark wasn’t the only expert consulted by me) cannot understand at all. Due to fear of the consequences of criticizing arXiv, most of those probably go unnoticed. While arXiv does need the possibility to reject or reclassify some papers, that needs to be done transparently and fairly. However, in view of its value to the community, there should be some simple rules, such as a ‘white list’ of journals so that papers accepted by them automatically qualify for the corresponding category at arXiv. Fortunately, my own livelihood does not depend on submitting to arXiv (in either sense of the word). Imagine the consequences of a young scientist who, after a year or so of work, gets their first paper accepted by a serious journal, only to have it rejected by arXiv or reclassified into a category where no colleague, potential employer, and so on will see it. Not only that, but the decision is made by someone (or some thing; arXiv is now moving to classification based on machine learning, but that was not relevant to the reclassification of my paper (Licia Verde, personal communication)) via an untransparent algorithm and no reason is given. Any appeal is within arXiv itself and essentially consists of some people asking others if they are guilty and accepting the expected answer. Such behaviour should be an embarrassment to the scientific community.

I think that some action on the part of the community would be in order even if my paper were the only one affected. However, the problem is much larger. Many colleagues have told me that they disagree with the reclassification of my paper, but are afraid to say so publicly for fear of getting banned from arXiv themselves. Also, I have been told that I am far from the first person to make such complaints about arXiv. Since I have started discussing this with colleagues, a few other similar cases have been mentioned to me. Considering that many of those affected probably don’t mention it at all out of a false sense of shame, the number of people affected is probably larger than many might at first guess. (I am not on FaceBook, but I understand that a similar problem was recently discussed within a FaceBook group for professional astronomers.)

A new development is that arXiv, by its own admission, doesn’t have the necessary means to do its job properly, and that I am not the only one complaining about it:

• Daniel Garisto, ArXiv.org reaches a milestone and a reckoning, Scientific American, 10 January 2022.

A red herring is that the American Astronomical Society has made all of its journals (which are some of the major journals in cosmology/astrophysics/astronomy) open-access. That probably won’t diminish the importance of arXiv—and hence the importance of making sure that it is run responsibly—for several reasons. First, an attraction of arXiv is that it is a one-stop shop with a reasonable interface, and by following it one can keep of with much of the literature in one’s field (though of course not all papers are posted to arXiv, but if it is run responsibly then there should be no reason for them not to be, except if the journal forbids posting (some version of) the paper to arXiv). Even if all papers were open-access, that would mean following websites, or RSS feeds, of several or even dozens of websites, not nearly as convenient as the abstract listings at arXiv. Second, the AAS journals have rather expensive publication fees, which are becoming increasingly hard to justify, especially in the case of online-only publications. (Note that there are journals with no publication fees which actually encourage the author to post something equivalent to the final version on arXiv with no embargo period; MNRAS is an example.) Third, items which would otherwise have limited circulation, such as theses and conference proceedings, can (in principle) be on arXiv.

I’m all for giving arXiv more support, but first my paper needs to be rehabilitated by being allowed into astro-ph, and the policies should be changed, and publicly communicated, so that such problems do not happen in the future (neither to me nor anyone else); I could then post my backlog. The evidence is that the goof is so large that a public apology is called for. The minimum which needs to be done:

1. When a paper is reclassified, authors should be informed (now, there is not even an automatic email; that makes sense because arXiv thinks that it needs to reclassify some papers against the will of the submitter) and given a chance to approve the reclassification, delete the submission entirely, suggest another reclassification, or appeal. Until the matter is resolved, the submission should stay in the ‘hold’ status with no action required to keep it there (now, one has to unsubmit and resubmit it to keep it from going away).

2. When a paper is reclassified, the submitter must be given concrete reasons.

3. The appeals process needs to be overseen with some authority outside of arXiv which has the power to overrule arXiv’s decisions, otherwise it is more or less a farce. It seems to me that some committee in the corresponding professional organization would be a good choice, e.g. the International Astronomical Union for papers on cosmology / astrophysics / astronomy. There can be an internal appeals process, but the final authority of arXiv’s decisions should not reside with arXiv if arXiv is to provide a meaningful service to the community.

4. Papers from the major journals should be essentially white-listed. If a paper is really so bad that it is obvious that it somehow slipped in by mistake, arXiv should request the journal to formally withdraw it. If the journal does so, then arXiv shouldn’t accept it either. If not, then it should go onto arXiv. (It should go on even if it is bad, to put pressure on journals to uphold quality and so that it can be discussed and rebutted).

5. arXiv needs to publicly apologize for reclassifying papers for reasons other than quality or content (e.g. my case), and invite those papers to be resubmitted after the other points above have been implemented.

6. The points above should make (re)submissions by wrong authors viable, but perhaps some sort of special protection is needed for whistle-blowers such as myself.

7. I was going to call for the resignation of Seiringer, Verde, and Presani, but it seems that they have all no longer in the posts they were when interacting with me. The main guilty person, though, Sigurdsson, is still Scientific Director. How anyone can be aware of my story (which can be backed up with evidence, in court if necessary) and still think that Sigurdsson should have anything at all to do with arXiv is beyond me. Also, although they have chosen (probably with good reason) to remain nameless, if arXiv were not drastically wrong on this point, the distinguished colleagues who put in a lot of time and effort trying to get arXiv to reverse its decision would not have done so. I am extremely grateful to them for their courage.

Of course, a boycott will not put pressure on arXiv. (It would actually remove pressure if people who are critical of arXiv stop using it.) If really famous people publicly announce that they will stop posting to arXiv until the points I raise have been cleared up, that might lead to something.

It is not clear how large the problem is, in part because not everyone feels able to complain. I don’t think that my case is a one-off, or even part of a small minority, because otherwise arXiv would not have invested so much time and effort to prevent one more abstract from appearing in astro-ph. I have given them several opportunities to revert their decision and hence cut their losses, but never even received a reply to such requests. Thus, the problem is probably substantial, and hence should be of interest to the entire community.

Information based on the web pages pointed to by the URLs in the reference list reflects the state of those pages on 28 August 2020; that based on the technical behaviour of the arXiv interface reflects my experiences between 20 April and 25 July 2020. References to ‘arXiv’ reflect my experience with the astro-ph category.

I would be interested in hearing anything relevant to this topic by email (my address is easy enough to find). Please indicate the degree of confidentiality you wish.

Please point as many people as possible, by all means at your disposal, to this post and related discussion. I am probably taking a big risk by going public, but if I do so, I want it to have the maximum effect. I see the lack of accountability of arXiv as a serious problem in modern academia.

Footnotes

* Steinn Sigurdsson is the Scientific Director of arXiv.

@ Eleonora Presani was the first Executive Director of arXiv, the post having been created only in 2020, while arXiv itself was created in 1991. She used to work for Elsevier. On 21 December 2021, it was announced that she would step down. According to the same announcement, Steinn Sigurdsson is still Scientific Director. Robert Seiringer is no longer Chair of the physics committee. I don’t see a new Executive Director listed on the arXiv Leadership Team web page.

§ Even that happened only after noted cosmologist Max Tegmark had threatened to complain to arXiv’s sponsors if my paper wasn’t taken out of limbo. Before, I had received only an extremely brief reply from Sigurdsson, and that only after a colleague who has known him for a long time discussed my complaints with him. Tegmark not only agrees that arXiv is overstepping its bounds by essentially overriding the refereeing process of a respected journal, but also that there is no reason that my paper should not be allowed to appear in astro-ph. He was also kind enough and brave enough to give me permission to quote from his emails to me. These do contain quotations of emails he received from arXiv. Ethically, I think that trying to correct the tremendous harm done to me and others because of wrong reclassification overrides any concerns about quoting without permission (which of course would not be given), especially since such quotations make my case much stronger than merely paraphrasing what others have told me or even just my own suspicions; this is a typical whistle-blower situation.

$ The only reason which I was given is the alleged lack of “substantiveness” of the paper. Max Tegmark, on the other hand, wasn’t told that, but was told that my case is “complicated” and that “[t]he reason for this [arXiv not automatically accepting a paper accepted by a journal] is partly the SCOAP3 agreement, which arXiv is not party to but still put certain obligations on us, and partly because we can not privilege any one journal or publisher for legal reasons. We get sued.” (Max Tegmark, personal communication.) I certainly don’t think that arXiv should automatically accept a paper just because it has been accepted by any journal, but do think that rejecting or reclassifying a paper which has been accepted by a respected journal should be done only under extreme circumstances, via a transparent and fair process, and for reasons which can be explained. Also, no one I have talked to has any idea how SCOAP3 could be relevant to my paper. Apart from Max Tegmark, several other colleagues (all full professors of cosmology / astrophysics / astronomy at major research universities) tried to intervene with arXiv (which did not want even discuss the matter with a low-life such as myself). That none of them want their names mentioned publicly is a problem in itself: the people whom arXiv is supposed to serve do not feel free to offer constructive criticism in public. Between the lines (or even in them, if one is allowed to see them), it seems that, in my case, the reclassification was not due to the contents or quality of my paper, but rather indicates another, possibly even more serious, problem: arXiv appears to be afraid of getting sued by crackpots. Apparently they abuse the gen-ph category (which is a mix of papers about general physics, papers which at first or even second or third glance obviously belong in another category and have nothing obviously wrong with them, and genuine crackpot stuff) by reclassifying some real papers to it and also letting through a few crackpot papers, thus avoiding the accusation of white-listing the major journals (which shouldn’t be a problem) and the crackpots can be appeased by having their papers in the same category as some major-journal papers. Of course this is not a policy which arXiv has published, but when several people get the same message behind the scenes, it is as certain as it needs to be to make my case. Although I believe that the concept still would have been deeply flawed, I offered to leave the paper in gen-ph but get have it cross-listed to astro-ph, but that suggestion was rejected by arXiv. Of course, if their goal is to appease the crackpots but at the same time keep them out of the major categories, that strategy wouldn’t work, because they would then have to cross-list crackpot papers or make a distinction, which is what they are trying to avoid (or rather they want to have a few alibi papers with no distinction).

# Licia Verde was Chair of the arXiv Scientific Advisory Committee. The Chair is now Ralph Wijers, who is also chair of the Physics Advisory Committee. I did contact him, but he sees no reason to investigate my case, as it happened before his posts as Chairman.

& The SAO/NASA Astrophysics Data system is the most important bibliographic database in astronomy/astrophysics/cosmology, operated by the Smithsonian Astrophysical Observatory (part of the Harvard/Smithsonian Center for Astrophysics, which also includes the Harvard College Observatory) under a grant from the National Aeronautics and Space Administration.