University ranking analysis

Warren Smart has analysed the recent performance of Australian and NZ universities in the three most prominent international university rankings (ARWU, QS, THE). There is a lot of detail there, not all of it depressing. It is going to be hard for NZ to keep being satisfied with “punching above its weight” in the face of lower income per student than just about anywhere we want to compare ourselves with. As a country, we may indeed have too many universities for them all to rank well internationally. But the good thing about NZ is that change can happen rapidly. So, please consider the policies of all parties in the areas of tertiary education, research, innovation, etc when voting in the general election on 20 September 2014.

Edit: the situation with NZ university rankings has been discussed quite widely recently. Some links:

Kiwifoo 2014

I was invited this year to KiwiFoo camp run 11-13 April by Nat Torkington and his crew in Warkworth. Before I went, I expected from reading others’ accounts of past camps that it would be (over?)stimulating and not to be missed, and so it proved to be. The opportunity to mingle with and listen to a diverse group of around 200 intelligent and articulate people (mostly with a common belief that technology can make the world better) doesn’t come along often. Certainly it is the first time I have seen journalists, bureaucrats, politicians, scientists, entrepreneurs, programmers and teachers thrown together in this way. Although there is always the danger of sessions degenerating into discussions about society that generalize from the experience of the participants (who are certainly not representative of NZ society) without sufficient data, this must be how major changes in society start. I hope that many good actions are inspired by our discussions.

If you ever get an invitation to KiwiFoo, accept it!

Open access news

Sometimes it is easy to forget that there may still be people who are not informed about this issue.

  • A nice summary by Samuel Gershman. He doesn’t mention one reason for the status quo being so hard to change: each journal has a monopoly on papers, and publisher packages (“the Big Deal”) make it hard to cancel individual journals – in any case, authors are insulated from having to make decisions about publication venues based on price.
  • Peter Murray-Rust is rightly angry about devious/incompetent publishers getting in the way
  • A great title: Causes for the persistence of impact factor mania
  • Meanwhile, Elsevier (anagram of Evil Seer) just keeps on going, with rising profits
  • The University of Waikato now has an open access “mandate” (a bit toothless for that name, really a policy, but a reasonable start). I have seen claims that it is the first in NZ, but it seems Lincoln University beat them to that. I know the University of Auckland has a working group on this issue. So, some slow progress, and maybe in my lifetime we will get where we ought to be already.

Is there a better way to fund research?

As I submit yet another low-probability grant bid that took up too much of my time, once again thoughts that “there must be a better way” come to mind. It seems that many colleagues feel the same way. Some interesting reading:

I have always felt that adding a random component to the grant award system, so as not to waste so much time trying to distinguish between very similar proposals, and giving out more, but smaller, grants, would be improvements. Reading comments on the above articles shows that several others agree. Perhaps it is time to try out some more modelling!

The peer review system for research

The disclaimer

In the last few years I have often read about the crisis in scholarly (mostly scientific) peer review.
I share the belief that the current system is surely suboptimal and must be changed. Much of what I say below is not original: I have read so many posts and books by Tim Gowers, Bjoern Brembs, Mike Taylor, Michael Nielsen, Michael Eisen, Noam Nisan, and many other people, that I can’t remember them all. I have a year’s experience as managing editor of an open access no-fee journal, two years’ experience as an editor (= referee) for a Hindawi journal, and many years experience as a journal referee. My research area is mathematics and various applications, so there may be some discipline-specific assumptions that don’t work for other fields. And it is not possible to cover every issue in a blog post, so I don’t claim to be comprehensive.

The latest online furore was occasioned by a so-called “sting” operation published in Science (unlike most articles in that magazine, this one is freely readable). I don’t think it worth commenting in detail on that specific article. It tells us little we didn’t know already, and missed a big opportunity to do a more comprehensive study. It does show by example that pre-publication peer review can fail spectacularly. Some other (often amusing) instances from the last few years involve computer-generated papers that are much low quality than the one submitted by Science, presumably accepted by computer-generated editors (even mathematics is not immune and some journals have done this more than once).

Some people have claimed that these weaknesses in peer review are exacerbated by the pay-to-publish model (they are certainly not exclusive to such journals, as the examples above, some published by Elsevier in toll access journals, show). This model certainly does lead to clear incentives for “Gold OA” journals to publish very weak papers. However, since authors have to pay, there are countervailing incentives for authors. If the reward system is poorly organized (as it seems to be in China, for example), then authors may still choose these predatory journals. But since papers in them are unlikely to be read or cited much, it seems unlikely to create a large problem. Journal reputation counts too – most predatory journals receive few submissions, for good reason. The existence of many low quality outlets (which predates the rise of open access journals) is a nuisance and possible trap for inexperienced researchers, and reduces the signal/noise ratio, but is not the main problem.

The main problem is: the currently dominant model of pre-publication peer review by a small number of people who don’t receive any proper payment for their time, either in money or reputation, is unlikely to achieve the desired quality control, no matter how expert these reviewers are. Furthermore, our post-publication system of review to ensure reliability is rudimentary, and corrections and retractions are not well integrated into the literature.

Both deliberate fraud (still quite rare, given the reputational risks, but apparently much more common than I would have thought) and works that are “not even wrong” and thus can’t be checked (poorly designed experiments, mathematical gibberish, etc) slip through far too often. It is bad enough that there are too many interesting papers to read, and then a lot of solid but uninteresting ones. Having to waste time with, or be fooled by, papers that are unreliable is inefficient for readers, and allowing this to go on creates wrong incentives for unscrupulous authors.

It seems now that “publication” doesn’t mean much, since the barrier is so low. A research paper now has no more status than a seminar talk (perhaps less in many cases). Self-publication on the internet is simple. There are so many journals that almost anything can be published eventually. How can we find the interesting and reliable research?

The only good solution that I can see involves the following steps. Clear proposals along these lines have been made by by Tim Gowers and by Yann LeCun.

  • adopt the open research model

    This means more than just making the polished research article freely available. It includes circulation of preliminary results and data. Certainly a paper that doesn’t allow readers to make their own conclusions from the data should be considered anecdotal and not even wrong. Imagine a mathematics paper that doesn’t give any proofs.

  • decouple “peer review” from publication

    There can be two kinds of services: assistance (writing tips, pointers to literature, spotting errors) with the paper before it is ready for “publication”, and comment and rating services (which can give more refined grades on quality, not just the current yes/no score.)

    Journal peer review focuses on the second type, but only gives yes/no scores (sometimes, a recommendation to submit to another journal). Computer science conferences are good for the first type of review, in my experience, but bad at the second. The first type of service was is offered by Rubriq, Peerage of Science, Science Open Reviewed. The second type is currently offered by Publons, SelectedPapers.net (no ratings yet), PubPeer.

    This allows people with more time to specialize in reviewing, rather than writing. And they should get credit for it! A colleague in our mathematics department told me in June that he had just received his 40th referee request for the year. He is too busy writing good papers to do anything like that amount of work. Yet PhD students and postdocs, or retired researchers, or those with good training whose job description does include intensive research (such as teaching colleges) could do this job well. To keep this post from getting even longer, I will not discuss anonymity in reviewing, but it is an important topic.

    Other advantages are that post-publication review boards could bid for papers, so the best ones are reviewed quickly by the best reviewers, multiple review boards could review papers, and reviews are not wasted by being hidden in a particular journal’s editorial process.

  • decouple “significance” from inherent quality measures

    Journals also routinely reject on grounds of their own idea of “significance”, which is inefficient (especially when they publish “important” work that is “not even wrong”). The real determination of how important and interesting a paper is can only be done after publication and takes a long time. In some fields, replication must be attempted before importance can be determined. PLoS does this kind of filtering and seems to be successful. Pre-registration of experimental trials which will lead to publication whatever the result, and registered replication reports, are other ways to reduce the bias toward “glamour mag science”.

  • if you want attention for your work, you may have to pay for it

    There ought to be a barrier to consuming expert time. It is limited, and refereeing junk papers for free is a big waste of it. I would like to see a situation where it costs authors something (money, reputation points, in-kind work) to command attention from someone else (if the work is exciting enough that people will do it for free, then so much the better). This doesn’t preclude authors making drafts available and seeking freely given feedback. However, more detailed pre-publication attention might be obtained by various means: give seminar talks and present at conferences, pay via money or formalized reciprocal arrangement. Post-publication attention is another matter.

  • complete the feedback loop

    No system can work well unless information on performance and opinions is allowed to flow freely. Reviewers must themselves be able to be reviewed and compared. Strong ethical guidelines for reviewers should be set, and enforced. The current system allows anonymous referees to do a poor job or an excellent one, and only their editor knows both who they are and their performance level.

Freedom and security

I have followed the PRISM revelations with dismay. Despite attempts to downplay its significance by those who assert that “privacy is old-fashioned” or “if you have done nothing wrong then you have nothing to fear”, a line has been crossed that ought not to have been without major public discussion. There has been a presumption of privacy for hundreds of years, and totalitarianism is not unthinkable in our so-called “free” societies.

In New Zealand the increasingly unimpressive-looking government has put forward legislation in this area that seems ill-conceived and is at the very least far too rushed.

There is Public meeting tonight, and a national protest planned for Saturday. It is true that some people attend far too many protests, but it seems to me that if you are ever going to protest anything, it should be this. Selling off state assets seems potentially much less serious. I really wonder what Richard Nixon, or even Robert Muldoon, would have done with the proposed spying powers.

Au revoir Berkeley

I will be leaving later this week after 2 months here. I will miss the lack of rain and the relaxed lifestyle and high education level off campus (not having to teach or listen to colleagues in meetings certainly helps – I suppose sabbaticals are supposed to be better than “real life”). Yesterday I saw two full large shelves of mathematics books, many very high level, in a used bookstore, something impossible to find at home. Not to mention a shop selling only dog collars, nurses on strike, the local street people, poor roads, compostable plates and forks.

Two interesting Berkeley institutions are worth mentioning. The Berkeley Math Circle has been running for over 15 years, modelled on East European practice and driven by dedicated people such as Zvezdelina Stankova. It is tempting to try to replicate something like this at home, starting small of course. The Simons Institute for Theory of Computing is up and running, and promises to be a great venue for collaboration.

Reverse liaison

As I understand it, many languages are pronounced “incorrectly” in such a way as to make it easier. Liaison in French is very common: saying “les amis” without pronouncing the final “s” of “les” would require a pause or glottal stop. I have just noticed that it is very common in American English to do the reverse in some situations, and I have absolutely no idea why. I have heard many people pronounce phrases like “get off” with a glottal stop instead of the “t” of “get”, and also words like “button” have a break between the two “t”s. This seems very weird – why pronounce words in a nonstandard way when that makes it harder to say, not to mention ugly-sounding? I first noticed it in an episode of Dora the Explorer several years ago, and assumed it was an idiosyncrasy of the voice actor in question. But it seems to be very common, and I even heard a reporter on National Public Radio doing it today.

What theory accounts for this illogical and inefficient behaviour? Wikipedia tells us that it is not confined to the US, which I had known for much longer, having as a child seen too many TV programmes involving Cockney characters.