ACADEMIA SHOULD STOP USING BEALL’S LISTS AND REVIEW THEIR USE IN PREVIOUS STUDIES

Academics (should) strive to submit to journals which are academically sound and scholarly. To achieve this, they could either submit to journals that appear exclusively on safelists (occasionally referred to as whitelists, although this term tends to be avoided), or avoid submitting to journals on watchlists (occasionally referred to as blacklists, although this term tends to be avoided). The most well-known of these lists was curated by Jeffrey Beall. Beall’s Lists (there are two, one for stand-alone journals and one for publishers) were taken offline by Beall himself in January 2017. Prior to 2017, Beall’s Lists were widely cited and utilized, including to make quantitative claims about scholarly publishing. Even after Beall’s Lists became obsolete (they have not been maintained for the past six years), they continue to be widely cited and used. This paper argues that the use of Beall’s Lists, pre-and post-2017, may constitute a methodological error and, even if papers carry a disclaimer or limitations section noting this weakness, their conclusions cannot always be relied upon. This paper also argues for the need to conduct a detailed post-publication assessment of reports in the literature that used Beall’s Lists to validate their findings and conclusions, assuming that it becomes accepted that Beall’s Lists are not a reliable resource for scientific investigation. Finally, this paper contends that any papers that have identified methodological errors should be corrected. Several lists that were cloned from Beall’s Lists have also emerged and are also being cited. These should also be included in any post-publication investigation that is conducted.


INTRODUCTION
Beall's Lists, one for stand-alone Open Access (OA) journals, the other for OA publishers, used to be maintained on a blog.That blog, as well as Beall's Lists, became a popular source of advice and guidance, and were widely debated in academic papers that were supposedly peer reviewed and approved by experts [1].
Beall's Lists became the "go to" place for those seeking to identify journals and publishers that they deemed to be predatory.It was an invaluable resource, in the absence of anything else.Although those lists were closed down in 2017 by Beall himself, many scholars still access and reference this resource even though it is no longer maintained and is increasingly out of date.
We wish to acknowledge the contribution that Beall made to highlighting the issue of predatory publishing, and the legacy that he has left behind [2], but there are several reasons why Beall's Lists were academically or potentially scientifically flawed: 1.The lists were maintained on a personal blog, and not on an institutional website nor on the website of a recognized scholarly entity [3]. 2. The lists exclusively profiled OA journals and publishers, ignoring that hybrid or subscription journals or publishers might exhibit similar issues [4].3. The criteria for inclusion were flawed and were not transparent, and there were no inclusion criteria for each included journal/publisher [5].4. The lists were incomplete, that is, not all OA journals and publishers had been assessed.This means that many were not evaluated (as either being on the list or not).Some journals or publishers were "lucky" for not being included in the lists, even though they should have been, had they been judged by the same criteria.Many entries were journals that had either ceased publication or had not published any content at all [6], so it is unclear how their academic quality or integrity was assessed.5. Beall's Lists were only a list of journals and publishers that Beall had identified as predatory.There were no lists of journals or publishers that he had evaluated as non-predatory.6.The lists carried false negatives [1,5].Among the hundreds or early thousands of OA journals and publishers, the suggestion was that all of them were equivalent, and thus unscholarly, or with no scholarly characteristics [1,5].7. The lists were seen by some as promoting a culture of bias and discrimination, even more so when associated with accusations [7,8].
Whilst recognizing the significant contribution made by Beall in raising awareness of the issue of predatory publishing, this paper notes the concerns in adding journals or publishers to the lists and also question their use as a methodological tool by other researchers.

THE USE OF BEALL'S LISTS BY AUTHORS
It is not unusual to find studies that use Beall's Lists, or a revised version of them, as a data source.These papers, which will be assessed in the future, may add a limitation section indicating the use of those lists, but using it nonetheless, stating in a paraphrased manner something along the lines of: "Although we know that Beall's Lists have their critiques, many of them justified, we used them anyway for our analysis".If a scientist knows a priori that a method is flawed or has errors, why would they use it anyway?Perhaps the publish or perish environment has motivated the use of flawed data because it is readily available?
In the case of students, why would senior advisors or principal investigators encourage their students to use Beall's Lists?One of the authors of this paper has previously stated "… the author should carry out their own due diligence, as well as seeking advice from their peers/supervisor(s).If there are any doubts, look for another journal as the world is not short of scientific journals" [9].This was our way of stating that too many people relied on those lists, despite there being a lot of knowledge and experience among their supervisors, peers and colleagues.We would add to that advicealbeit cautiously -by looking at where internationally renowned scholars are publishing and consider using those journals as a starting point for selection.If peers and supervisors suggest Beall's Lists as a preferred journal selection (or journal elimination) mechanism, are they meeting their responsibilities and/or harming their student's career and reputation by offering unsound advice?
Why do authors continue to use Beall's Lists, either as a data source for studies or as a journal selection mechanism?Despite the criticisms that were levelled at those lists, they were nonetheless seen as a useful resource, up to the point when they were taken offline in 2017.Yet, over six years later, their usefulness is increasingly decreasing, and we would argue that they can no longer be relied upon.Even if Beall's Lists are used, many other journals or publishers have appeared since 2017 and there were many that Beall did not consider.Thus, the fact that a journal or publisher did not appear on Beall's Lists does not necessarily mean that they are not predatory.
There is, perhaps, an even more worrying practice, where authors intentionally submit to predatory journals [11,12,13] -in these authors' cases, submitting to the journals included on Beall's Lists -knowing that their article will likely undergo little (if any) peer review and will be published very quickly.The onus is then on those scrutinizing a curriculum vitae (CV), whether they may be a potential new employer, a funding agency, or a promotion panel to spot truly predatory journals.Those looking at such CVs are likely to be non-domain experts, so the chances of articles in truly predatory journals being spotted is minimal.Even so, the scientific content and integrity of each paper, rather than its publishing venue, should be emphasized when evaluating a candidate based on their CV.

BEALL'S LISTS AS USED BY EDITORS AND REVIEWERS
Editors of journals who claim to conduct peer review, as well as peer reviewers who review papers, should be knowledgeable individuals (so called experts) in the topic of the journal and the subject of the paper being presented.They should also be aware of the publishing process and its ethics.
If a paper is submitted that draws on the work of Beall, specifically his lists, the reviewers should question whether the methodology is valid and robust.We do not wish to say that every study that has drawn on the work of Beall's Lists is flawed (that is not our role, as we have neither looked at every paper, nor have we seen the reviewers' comments) but it should no longer be permissible to use Beall's Lists and justify their use by acknowledging that the dataset has problems or other issues, which are known.Any analysis or conclusions must be supported by evidence, an approach that was adopted by Kendall and Linacre [9], who revisited the first 18 OA publishers that Beall analyzed.That paper's conclusions were supported by updated evidence collected about each of those publishers.
Any editor or reviewer that acts on a paper that uses Beall's Lists as source data, should question whether simply using the lists "as they are" is methodologically acceptable and whether the paper's conclusions are valid based on how the Lists are used, criticisms that have been levelled at them, and the fact that the Lists were shut down over six years ago, i.e., outdated.
Perhaps, it is pertinent to ask why would an editor or reviewer approve the publication of a study that used a flawed data set or methodology, even if it carries a limitation-related statement?There are at least three possible explanations: 1) the editors or reviewers are not suitably qualified to review and decide on the topic of predatory publishing; 2) the editors or reviewers are biased, perhaps subconsciously, and/or are pro-Beall, without closely considering the evidence, and thus may be inclined to support any study that relates to Beall's Lists; 3) the editors or reviewers believe that publication of the paper will draw novel attention to predatory publishing and see this reason as more important than the robustness of the paper itself.Independent of the reason, we are of the opinion that approving a study that uses a flawed methodology (or tool) reflects poorly on the level of editorial (and peer) quality control.

Pre-2017 versus post-2017 studies
As noted above, Beall's Lists were taken down by Beall himself on January 15, 2017 [14].A possible reason why academics prior to this date did not know that the Lists had flaws was because they were unaware of the evidence or discussion that had taken place or because, as the blog was still active, it was considered to be an active and thus credible resource.
From the date the Lists were taken offline, discounting the existence of fragmentary mementos on the Internet Archive, any individual who attempted to access the blog (https://scholarlyoa.com/) would have observed a 404 error (page not found) or some other irrelevant information (i.e., not Beall's Lists), indicating that the domain is now being used for a different purpose (as is still the case today [accessed 14 April 2023]).This is a good example of why the scientific archive is very important.Anything that the scientific community relies upon should be exactly the same as when it was written (with any changes being handled by an erratum).For example, the landmark 1953 paper by Watson and Crick [15] can still be accessed via the journal Nature.While a DOI (in this case 10.1038/171737a0) can make access easier, it does not replace the underlying premise that a peer-reviewed paper should always be accessible through the journal that published it and the reader knows that i) it has been peer reviewed and ii) it is the same as the final version that was accepted by the editor and journal.This is why blogs, newspapers, and Wikipedia are not reliable scientific references because they can either be changed post-publication or they may cease to be available.
Three problems with Beall's Lists were/are as follows: i) they were not peer reviewed, ii) they could change over time such that the content and listings of journals or publishers could change overnight, and iii) they are no longer available in their original format.The conclusion we draw is that Beall's Lists cannot be considered to form part of the scientific archive.We suspect that Beall never intended this to be the case.We note, however, that Beall called on predatory journals and publishers to be banned [16], presumably referring to those on his Lists.
In fact, we are of the belief that Beall was merely providing what he believed was a useful (advisory or alert) service which, in many respects it was.However, it is mostly other scholars who have used Beall's Lists as if they form part of the scientific archive.For this reason, peer reviewers and editors have a responsibility to properly screen the cited sources and literature.

Beall's Lists and the studies that have used them
We reiterate that use of Beall's Lists as data sources in a peer-reviewed paper constitutes a methodological error, based on the observations we presented above.We further suggest that any advice or conclusions need to be checked and possibly corrected, even more so if the journal (or its publisher) is a member of the Committee on Publication Ethics (COPE) or the International Committee of Medical Journal Editors (ICMJE), both of which have guidelines related to the correction of erroneous papers [17,18].
More specifically, we propose the following: 1. Authors who used Beall's Lists for quantitative studies (both before and after January 2017) could, upon reflection, have published a flawed study.We suggest that those authors revisit their work, in the light of the arguments in this paper, to determine whether they believe that their study is still valid.2. Studies that employed Beall's Lists to compare journals or publishers should be identified.Other studies that made predictions on the volume of "predatory" journals or publishers, OA or non-OA, locally or internationally, and that relied on Beall's Lists to base their findings, estimates or comparisons should be identified.The papers from these groups should be re-examined to gain an appreciation of the extent of reliance on Beall's Lists.Papers that have made a significant use of those Lists, without additional supporting evidence, may need to be challenged.For example, if papers used those Lists to predict the number of "predatory" journals or publishers, without providing any additional evidence, then such papers would be open to challenge.Furthermore, papers that compared journals or publishers on Beall's Lists and journals or publishers that are widely considered to be legitimate, could also be challenged, depending on the robustness of the additional evidence presented, if any, and the conclusions that were drawn.3. It would be interesting to assess whether any Expressions of Concern (EoC) were issued and if any papers that relied on Beall's Lists have ever been retracted, particularly with regards to existing guidelines for EoCs and retractions [17,18,19,20], in order to address if any errors, methodological or other, arose specifically from the use of Beall's Lists.4. Papers which cite another paper's statistics, for example, in the Introduction, would be regarded as "secondary" users, and depending on their level of reliance on Beall's Lists and the context, they may require a correction (via an erratum).Other papers that were more reliant on the results of a paper that used Beall's Lists would require an investigation to explore whether their findings are invalidated by the use of the "Beall's Lists"-reliant paper that it cites.

ACCOUNTABILITY
Beall's Lists were probably never designed to be part of the scientific literature, and since they no longer exist in their original form, and since their content is becoming increasingly outdated, the scientific community needs to resist the temptation to assume that Beall's Lists constitute datasets that have scientific merit or validity.
We hope that the majority of the scientific community can appreciate what Beall was trying to achieve.If his aim was to raise awareness of the pernicious practice of predatory publishing, as was suggested by Beall himself [14], then this was achieved.If, however, it was to produce lists that could be referred to in order to identify predatory journals or publishers, then this was also achieved, although not without serious caveats since the lists were flawed (as discussed above).Thus, anybody using them should not only be aware of their shortcomings and should only use them as a starting point for their own personal investigation, rather than as a way to find a definitive answer to the question "Is this journal (or publisher) predatory?" The scientific community has a collective responsibility and should be held accountable for assuming that Beall's Lists are a definitive source for identifying predatory journals or publishers.

WHAT ARE THE ALTERNATIVES TO BEALL'S LISTS?
We believe that there is currently no way to reliably and consistently identify predatory journals or publishers that is open to all.All existing lists themselves carry flaws and weaknesses that reduce their reliability and functionality [21,22,23].A recent artificial intelligence-driven software program [24], which relies on Beall's Lists among others to make categorization decisions regarding the predatory nature of journals, has severe classification errors [25].
That study questioned whether reports produced using the stated criteria could be relied upon and opined that Cabells' Predatory Reports is thus not a reliable source of information.Other academics, supported by previous findings, also found classification errors [27].We note that even if Cabell's is considered a reliable resource, it is a subscription-based service, so it is not available to all scholars.We also note that as Beall's blog and Lists were open, this may have aided in their popularization.
Another post-Beall blog is the Dolos list which appeared in 2018, but it also disappeared similarly to Beall's Lists, although the exact date of disappearance is unclear [28].Moreover, its creator and/or manager, a seemingly fictitious or pseudonymous name "Professor Alexandre Georges", has not yet provided any public explanation, unlike Beall [14].
The authors are of the opinion that at the time of writing (April 2023), there are no reliable, robust, and free-toaccess lists that can accurately warn academics of unscholarly journals or publishers.

HOW DO ACADEMICS PROTECT THEMSELVES AGAINST PREDATORY PUBLISHERS?
This is a difficult question to answer because currently, no reliable solutions exist.Possibly, in the knowledge of this gap, some scholars may be complicit in their use of truly predatory publishers, including those that rebrand to avoid identification and detection [29], as a way of enhancing their CV.
Many papers that discuss predatory publishing offer some advice, often in the form of checklists, as to how they can be avoided.This topic alone would merit its own article, but there are some examples [30,31,32,33].The issue with this approach is that it requires the scholar to be familiar with the literature on predatory publishing which, of course, most will likely not be.This is one reason why Beall's Lists were so attractive.It provided a single "go-to" location and resource that gave a binary answer to our question.Even so, some academics have attempted to approach the issue via the refinement of predatory criteria [34,35,36,37,38,39], or scholarly criteria [40].
Some academics might believe in structures that are presented to them by established entities, perhaps drawing some level of comfort in the notion that the lists (negative or positive) that they rely on to make their choice of journal or publisher was created by a famous scientist, a respectable organization, an ethics-related organization, a powerful media organization, influential media-like lobbyists [41], a prestigious group of academics or self-professed "experts", or that provided definitions are valid and a safe mechanism of protection because dozens of like-minded (or perhaps equally biased?) "specialists", "experts" or policy makers were involved in their creation [42,43].
However, over-simplistic definitions or an attempt to collectively define a wide range of behaviors in a limited definition does not offer academics any robust form of protection or guidance, and merely serves to elevate the egos and academic profiles of those who created such definitions and/or policies.In other words, some academics feel comfort in believing something tangible, and so they want to believe in the credibility of a list, even though the evidence shows that the list they are using is not credible.Perhaps believing blindly in such lists, even if they are flawed, brings a level of comfort and sense of protection against the fear of a "predator" [44,45]?
Of equally great concern, what if these "specialists", "experts", professionals, policy makers and/or ethics organizations have erred, or are wrong?What if the scholarly advice that has been provided to date (2023) has not been entirely scholarly, and may have, to some extent, been misleading?Have librarians also been involved in the dissemination of misleading information related to "predatory" publishing [46,47], despite their responsibilities towards academia [48]?What if genuine scholarly entities may have been unfairly tarnished, while genuinely unscholarly or predatory entities have been lucky to have not suffered the same listing, characterization and fate, through either their legal prowess, or their ability to suppress criticism?We believe that academia is in a current state of "limbo" or adjustment with respect to the issue of "predatory" publishing.Moreover, not wanting to recognize that there is a body of journals and publishers that might display some unscholarly or imperfect properties, but not necessarily predatory ones, leaves a gray zone of incorrectly characterized or mischaracterized journals and publishers [29,49].This gray zone is complicated by the ability of some potentially predatory entities to rebrand, change hands, or modify names [50].
So, how do academics protect themselves?How do they confidently know that a journal or publisher they wish to submit a paper to is genuinely scholarly, or if it is predatory?How do they differentiate exploitation from predation [51], or are these now synonymous?Most likely, there is neither an absolute level of protection, nor a legitimate claim that we can assert as being able to offer sound advice.Even legitimate journals are now proving themselves to be poor bastions of the "peerreviewed" label [52], as indicated by high levels of retractions, many in highly ranked journals that claim a high or respectable journal impact factor, that claim to be peer reviewed and thus exercise stringent scholarly scrutiny of work that they publish, and that espouse adherence to COPE and/or ICMJE (or other) ethical or scholarly principles.In other words, an academic might look at a supposed legitimate journal, thinking that it is compliant with COPE/ICMJE guidelines, adhering to "best principles", claiming to conduct peer review, or displaying impressive impact factors, or essentially encompassing these or other markers of academic quality and scholarly behavior, when in fact, they may not necessarily be.
The two authors of this paper have (independently) proposed methodologies, which we briefly present here for consideration and discussion.
Teixeira da Silva et al. [53] suggested a layer of classification, such as a credit-like rating, that can be used alone, or as an overlay to a more quantitative system of ranking behavior, such as the "Predatory Score" [54].In such a system, concrete criteria are ranked, and a qualitative system of terminology, or color coding, is overlaid, to provide a score that reflects the attribution of positive points for scholarly behavior, but discounts points for unscholarly behavior.
Kendall [55] proposed a solution that draws inspiration from the fine art world that uses a Catalogue Raisonné to guard against forgeries.In that proposal, publishers (both predatory and legitimate) are subject to scrutiny via an evidence-based, peer-reviewed paper.If the paper is accepted, it would have undergone the same process as any other paper that is accepted into the scientific literature.

FUTURE WORK
As proposed in the present paper, it would be useful to revisit every paper that has employed Beall's Lists as sources of data for studies that have appeared in the peer review literature.If it is accepted that Beall's Lists should not be used for such studies, then are the conclusions stated in those papers still valid?Moreover, have papers that cited those papers been based their own assumptions or on the conclusion of the paper?Even if all the "Beall List" papers are found to be valid, it would still be interesting to have a definitive list, in the form of a survey, or a post-publication review or assessment, which would be a useful resource for future scholars.Challenges associated with identifying and eliminating truly predatory journals and publishers still remain, and anything the scientific community can do to address this would be welcomed.

CONCLUSION
We wish to acknowldge the important role that Beall played in raising awareness about predatory publishing, and we applaud his sustained effort, and presumably peer-reviewed publication record, that addressed this important topic.Beall's Lists (both for OA journals and publishers) had their methodological flaws, although they provided a crude service to many people during a part of their academic lifetime and, for all their faults, they were for a time (a few years) the only resources that were available to scholars.Unfortunately, since those Lists hosted on a blog were closed in January 2017, many people still continue to rely on them as definitive sources to appreciate whether an OA journal or publisher is predatory or not.Even if the Lists were valid in January 2017 [56], six years on, they cannot (and should not) be used as sources of reliable data and recommendations.Moreover, as the main focus of this paper, Beall's Lists (pre-or post-January 2017) should neither be used as datasets nor be as bases for drawing conclusions that are presented in the scientific peer-reviewed literature.Any such papers should be revisited to verify whether their conclusions are valid, if the suggestion that Beall's Lists should not be utilized in this way becomes widely accepted.Since the closure of Beall's Lists, to the best of our knowledge, there has been no development or emergence of any robust and free service that can be openly used to identify truly predatory journals or publishers.This should be of great concern to the scientific community and requires urgent attention and dedication to find solutions.