Ep77 The Academic Journal System is Broken, Here’s How to Fix It

Overview of the Academic Journal System and Its Challenges

• The topic of discussion is the academic journal system and whether it is still the best system for academic debate, considering it has been in place for 150-200 years, and new technology may offer better alternatives 10s.
• The current system is perceived to be in crisis, but the reasons for this crisis may differ from those reported in the popular press, with the main problem being the outdated distribution technology 4m42s.
• Fifty years ago, research was distributed through paper journals, which was a costly and time-consuming process, resulting in a rigorous selection process for published papers, with a limited number of elite journals in each field 6m6s.
• In economics, for example, there were only a few elite journals, such as the American Economic Review, the Journal of Political Economy, and Econometrica, making it feasible for researchers to keep up with new publications 8m30s.

Evolution of Research Distribution and Its Impact on the Journal System

• The introduction of technology, including photocopying machines and the internet, has fundamentally changed the distribution of research, making it free and efficient to share papers electronically 10m30s.
• The peer review system, which decides which papers to publish, has never been perfect, and there is a need to reevaluate the system, considering that many more papers are written than published, and only a select few are chosen for publication 14m20s.
• The academic journal system has been imperfect, with many bad papers and wrong ideas getting published, while good papers often do not, and this issue has been present for a long time, with the peer review system being biased against new ideas 10s.
• The old system was able to mitigate this issue to some extent, as people would only see a curated selection of papers, and bad papers would be ignored, while good papers would be referenced by everyone, but this system broke down as distribution costs decreased and people started distributing their own papers 2m6s.

Variation in Peer Review Processes Across Academic Fields

• Different fields in academia have different review processes, with economics being an extreme example, where the review process can take up to four years, while medicine has a much shorter lag, but a more flawed review process, with many papers claiming incorrect information getting published 2m6s.
• The incentives of individual players in the system, such as editors and referees, play a crucial role in evaluating the system, and editors may be more inclined to avoid publishing a potentially groundbreaking paper if it is too new and risky, rather than publishing a paper that may be wrong 4m6s.

Incentives and Challenges for Referees in the Peer Review System

• Referees, who are responsible for evaluating papers, are not adequately compensated for their work, with the financial compensation being nowhere near their hourly rate, and the fact that referees are anonymous, meaning authors never find out who they are, raises questions about the incentive for referees to write good reports 6m6s.
• In the field of economics, the journal system relies on a board of editors and referees, who are regular academics, to evaluate papers and write reports, with the Journal of Finance being a high-prestige journal that uses this system, and the lack of compensation for referees makes it costly for them to do a good job 8m6s.
• The academic journal system has incentives for referees to do a good job, but these incentives are largely implicit, with referees hoping that writing good reports will give them a good reputation and potentially lead to benefits, such as more favorable treatment of their own papers by editors 10s.
• There is no formal quid pro quo system, and referees are not scored on their reports, but many believe that writing good reports will lead to rewards, such as editors remembering them as competent and using high-quality referees for their papers 1m42s.
• Junior faculty members who write deep and insightful referee reports may be remembered by editors as competent, increasing the chances that their own papers will be treated favorably, and this can be a significant incentive for them to do a good job 2m6s.

Pressure and Inefficiencies in the Journal Submission and Review Process

• The acceptance rate for the Journal of Finance is around 8%, making it a highly competitive and significant publication, with a single publication often being enough to secure tenure at some universities 4m30s.
• The low acceptance rate and high stakes for faculty members, particularly junior ones, create a high-pressure environment, but despite this, the quality of referee reports is not always high, with referees often focusing on finding low-hanging fruit to impress editors rather than doing a thorough job 6m10s.
• The referee report process has become a list of tasks for authors to complete, with referees often not putting in the effort to find deeper issues with the paper, and instead focusing on easy-to-find problems that can be used to impress editors 8m0s.

Flaws in the Referee Report Process and Their Consequences

• The academic journal system is flawed, as it forces authors to make revisions based on referee demands, regardless of whether they make sense, resulting in a huge amount of wasted effort, particularly for junior faculty who spend a significant amount of time revising papers instead of conducting new research 10s.
• A potential survey of published papers could ask authors if the refereeing process improved or worsened their paper, which could provide insight into the effectiveness of the current system, although authors may be biased in their responses 2m6s.
• The refereeing process may lead to a small increase in paper quality, but the effort required to achieve this increase is substantial, resulting in a negative effort-to-quality trade-off, and externalities such as the increasing length and complexity of papers 4m40s.

Negative Externalities of the Current Journal System on Research Quality

• The demand for more information and literature reviews in economics papers has led to them becoming overly long, jargon-filled, and difficult to read, with a negative externality being the obstruction of clear and concise communication of ideas 6m10s.
• The time it takes for curation, or the decision on whether a paper is worth reading, is delayed by around four years, making it useless for deciding whether to read a paper, resulting in an enormous number of papers being published, and individuals relying on introductions and abstracts rather than reading the full paper 8m40s.
• The overall quality of papers has dropped dramatically due to the emphasis on writing engaging introductions without necessarily having the academic rigor to support it, leading to a crisis in the profession, as noted by Jonathan Berk and Jules Van Binsbergen 10m50s.

Challenges in Curation and the Proliferation of Low-Quality Research

• The academic journal system is facing challenges due to the increased demand for teaching and research output, which has exploded and made it difficult to separate high-quality research from low-quality research, with information technology making it easy to distribute a large amount of work 10s.
• The current journal system is not well-equipped to separate good research from bad, and there is no efficient way to share crucial knowledge in real-time, as seen in an example where a paper with a fatal flaw was published without the editors being aware of a note that pointed out the issue 2m6s.

Introduction to the Informed Discourse Initiative

• A new initiative, called Informed Discourse, has been started to address these issues by using technology to facilitate the distribution of knowledge and crucial information about papers, allowing experts to review and comment on research 4m6s.
• The Informed Discourse website is designed to be a platform where experts can discuss research, with features such as only allowing true experts to contribute, no anonymity, and a reputation system to incentivize high-quality contributions, with the goal of creating a trusted source for research 6m6s.
• The website aims to curate research in a way that relevant papers are easily accessible to knowledgeable researchers, creating a go-to place for research and addressing the issue of signal jamming in the current academic journal system 8m6s.

Founders and Collaborators of Informed Discourse

• The Informed Discourse initiative is a collaborative effort between Jonathan Berk and three other researchers: Matt Ringgenberg, Martin Lettau, and Willy Fuchs, who are working together to create a platform that promotes informed discourse and high-quality research 6m6s.

Proposed Features and Goals of the Informed Discourse System

• A proposed system for academic journals would provide users with a personalized homepage, curating content relevant to their interests based on their reading, writing, and citing history, as well as their reviews and comments on papers 10s.
• The system, called Informed Discourse, aims to create a community that facilitates scientific discourse and real-time sharing of knowledge, improving the knowledge and research processes by presenting users with papers closest to their interests, making them more likely to write reviews and comments 42s.
• The curation process in this system is automated and based on topic, not opinion, reducing the potential for abuse and increasing the likelihood of users engaging with relevant content 2m6s.
• In contrast to the current system, where editors have significant power and can abuse it, the proposed system relies on crowdsourcing and community engagement to determine the relevance and quality of research, reducing the influence of individual biases 2m6s.

Design Philosophy and Advantages of the New System

• The proposed system also aims to address the issue of negative reviews in traditional journals, instead focusing on positive and constructive feedback, and using the lack of attention or reviews as an indicator of a paper's relevance and influence 4m30s.
• The idea is that a paper's influence should be determined by its ability to generate genuine interest, rather than being forced through the review process, and that the proposed system can provide a more accurate and helpful assessment of a paper's value 6m40s.

Discussion of Implementation and Potential Challenges

• Jonathan Berk and Jules Van Binsbergen discuss the potential benefits and challenges of this system, including the need to balance community engagement with the potential for abuse, and the importance of using the system to provide meaningful and relevant feedback to authors 8m0s.

Limitations of the Current System and the Need for Alternatives

• The current academic journal system has issues, including the problem of too many submissions, and increasing the price is not a suitable solution as it may unfairly discriminate against those with good ideas but limited means to submit 10s.
• A proposed alternative is to let a curated crowd decide, but this approach still distributes power in a particular way, and it is essential to be precise about the selection process 42s.

Launch and Development of Informed Discourse

• A new system, informeddiscourse.com, is being launched, which aims to address the issues with the current academic journal system and is looking for enthusiastic first-time users to help build the system 2m6s.
• The system has signed up thought leaders in the profession, including 20 influential people in financial economics, who will write reviews and help curate the content 2m6s.
• The system will also feature a conference system, where reviewers can easily access and review papers, and the program chairman will be assisted in assigning papers and reviewers 2m6s.
• The goal of the new system is to provide a service to the profession, and it will be endogenous, relying on users to write reviews and engage with the content 2m6s.
• The system is being promoted, and listeners are encouraged to visit the website, informeddiscourse.com, and provide feedback 2m6s.