Irreproducibility in searches of scientific literature: A comparative analysis
Abstract Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment a...
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | article |
| Lenguaje: | EN |
| Publicado: |
Wiley
2021
|
| Materias: | |
| Acceso en línea: | https://doaj.org/article/974e63d33f5e404dbf0641d64b6fff65 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| id |
oai:doaj.org-article:974e63d33f5e404dbf0641d64b6fff65 |
|---|---|
| record_format |
dspace |
| spelling |
oai:doaj.org-article:974e63d33f5e404dbf0641d64b6fff652021-11-08T17:10:40ZIrreproducibility in searches of scientific literature: A comparative analysis2045-775810.1002/ece3.8154https://doaj.org/article/974e63d33f5e404dbf0641d64b6fff652021-11-01T00:00:00Zhttps://doi.org/10.1002/ece3.8154https://doaj.org/toc/2045-7758Abstract Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results. We present a comparative analysis of time‐synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms. We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions. To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine‐readable search metadata.Gábor PozsgaiGábor L. LöveiLiette VasseurGeoff GurrPéter BatáryJános KorponaiNick A. LittlewoodJian LiuArnold MóraJohn ObryckiOlivia ReynoldsJenni A. StockanHeather VanVolkenburgJie ZhangWenwu ZhouMinsheng YouWileyarticledatabaseevidence synthesis methodsinformation retrievalrepeatabilityreproducibilitysearch engineEcologyQH540-549.5ENEcology and Evolution, Vol 11, Iss 21, Pp 14658-14668 (2021) |
| institution |
DOAJ |
| collection |
DOAJ |
| language |
EN |
| topic |
database evidence synthesis methods information retrieval repeatability reproducibility search engine Ecology QH540-549.5 |
| spellingShingle |
database evidence synthesis methods information retrieval repeatability reproducibility search engine Ecology QH540-549.5 Gábor Pozsgai Gábor L. Lövei Liette Vasseur Geoff Gurr Péter Batáry János Korponai Nick A. Littlewood Jian Liu Arnold Móra John Obrycki Olivia Reynolds Jenni A. Stockan Heather VanVolkenburg Jie Zhang Wenwu Zhou Minsheng You Irreproducibility in searches of scientific literature: A comparative analysis |
| description |
Abstract Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results. We present a comparative analysis of time‐synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms. We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions. To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine‐readable search metadata. |
| format |
article |
| author |
Gábor Pozsgai Gábor L. Lövei Liette Vasseur Geoff Gurr Péter Batáry János Korponai Nick A. Littlewood Jian Liu Arnold Móra John Obrycki Olivia Reynolds Jenni A. Stockan Heather VanVolkenburg Jie Zhang Wenwu Zhou Minsheng You |
| author_facet |
Gábor Pozsgai Gábor L. Lövei Liette Vasseur Geoff Gurr Péter Batáry János Korponai Nick A. Littlewood Jian Liu Arnold Móra John Obrycki Olivia Reynolds Jenni A. Stockan Heather VanVolkenburg Jie Zhang Wenwu Zhou Minsheng You |
| author_sort |
Gábor Pozsgai |
| title |
Irreproducibility in searches of scientific literature: A comparative analysis |
| title_short |
Irreproducibility in searches of scientific literature: A comparative analysis |
| title_full |
Irreproducibility in searches of scientific literature: A comparative analysis |
| title_fullStr |
Irreproducibility in searches of scientific literature: A comparative analysis |
| title_full_unstemmed |
Irreproducibility in searches of scientific literature: A comparative analysis |
| title_sort |
irreproducibility in searches of scientific literature: a comparative analysis |
| publisher |
Wiley |
| publishDate |
2021 |
| url |
https://doaj.org/article/974e63d33f5e404dbf0641d64b6fff65 |
| work_keys_str_mv |
AT gaborpozsgai irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT gaborllovei irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT liettevasseur irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT geoffgurr irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT peterbatary irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT janoskorponai irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT nickalittlewood irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT jianliu irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT arnoldmora irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT johnobrycki irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT oliviareynolds irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT jenniastockan irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT heathervanvolkenburg irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT jiezhang irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT wenwuzhou irreproducibilityinsearchesofscientificliteratureacomparativeanalysis AT minshengyou irreproducibilityinsearchesofscientificliteratureacomparativeanalysis |
| _version_ |
1718441495701225472 |