Some thoughts on my ideal publication system.
Two common users of the scientific literature:
- The newbie/shallow learner: they are just getting into a field and need to understand some subject.
- The deep researcher: they spend their days doing research in this area and need to keep up to date with what other researchers are doing.
Currently, the scientific literature doesn’t really satisfy either type of user:
Newbies need more review articles and textbooks. These are often not available for concepts and research from the last 10 years. Review articles are currently quite rare especially for new or fast moving fields and are not well incentivized by the career process. They are normally written by altruistic, often tenured, senior researchers1. Textbooks are similar.
- Both opinionated review articles and encyclopedic review articles would be useful. Opinionated review articles would explain a particular researcher’s position. Encyclopedic review articles might cover the last year or two of research progress and how particular relevant papers fit into that progress.
- I’d like to see something like a 5:1 or 10:1 ratio of research papers to review articles. I don’t think we’re anywhere close to this right now.
- I don’t currently have any good ideas for how to change the system to better incentivize this kind of work.
- With a higher density of review articles to research papers, it might be possible to do away with the typical paper introductions2.
Deep researchers need fast updates and fast publication and need to avoid publication-driven scope creep. Slow publication puts up barriers for the writer _and _the reader.
- Currently, many fields have publication processes that take many months or years. This hinders progress because it’s hard to build on existing research. A substitute for publication is either personal communication or conference talks. Both of these processes put up artificial barriers. It’s especially hard if you’re not connected to the “old boy’s club” of senior researchers or are in a far away country without access to top researchers.
An alternative for faster updates: “clarity and honesty” in publication.
- Currently, an article can get rejected for many reasons.
- Instead, default to accepting papers but allow edits for clarity and honest: allow reviewers to require changes to the tone and discussion to emphasize ways in which the reviewers view the study as weak, but do not allow rejection solely based on content3.
- Include a “Next steps” section instead of requiring extra work: oftentimes, a researcher has moved on from a project but must waste time returning to the project because the review process requires redoing or extending the project in some way. Instead of requiring such additions, the article could be amended to have a “next steps” section that summarizes the extensions requested by the reviewer.4
- arxiv and similar websites partially solve this problem since they help researchers share their work quickly. But, the papers those researchers write are still written with reviewers in mind. This results in convoluted and very defensive writing. That means a “clarity and honesty” standard might still be helpful and lead to better writing even in a world of pre-prints or no journals at all.
Notes
Unsurprisingly, it seems like good introductory material is more common in fields with strong industry presence since companies and researchers have a large incentive to get their team members up to speed. See, for example, deep learning or almost any practical software topic. Question for readers: is the same true in other practically relevant engineering domains in atom-space rather than bit-space? E.g. materials? ↩︎
An unnecessary friction. In world where there are lots and lots of review articles, you can just cite the most relevant one as introductory material to your sub-sub-sub-sub-field and then explain which parts you are focused on improving. This might be different in a pre-paradigmatic field where there is no relevant review article. ↩︎
I know that this is not realistic under the current system! ↩︎
Or they decide not to return to the project, which has two costs: 1) People don’t find out what substantive facts they learned. 2) People don’t learn the meta fact that this kind of work doesn’t always go well. I’m specifically thinking of a friend who, as a seventh-year, said “My results don’t make any sense, just like no one else’s make sense. This is why no one does field work in [my field]!” ↩︎