This month in the US, The National Science Foundation awarded AI startup scite 224,559 dollars in funding for their 'deep learning platform to identify confirming and refuting citations' (link, link). Go to the scite.ai website, enter a DOI and find out how many publications either mentioned the article, agreed with the article or disagreed with it / contradicted it. There is even a browser plugin that works like a virus scanner: it automatically warns you if you open an article that has been refuted. Let's say an early-stage Retraction Watch. Is it possible to use artificial intelligence and deep learning to discover which of millions of articles is contradicting the article you are reading?
This blog has been covering then fresh articles for over 10 years now. Let's find out if anyone of them have been contradicted since then. A quick sample from the 2008-2009 period has the following results. Few citations per article featured, this blog must have an esoteric taste when it comes to chemistry. According to the scite website the number averages around 5 to 15. But hold on, according to the official article pages at the publishing house the number of citations is much higher, between 20 and 80. Clearly scite is missing a lot of citations. The most important find: none of the articles tested have uh.. contracted a contradiction. That should be good news because that means this blog has been not been wasting anyone's time with reporting on sloppy science. On the other hand, on several occasions the blog did discuss contradicting articles. One of them discussed an article on the possible existence of mercury(IV) link but the critics do not get a mention. Same with a zwitterion paper featured here or the protobranching debate featured here.
Part of the problem is the missing citations. This effort requires data mining on a large scale and the scientific publishers are not going to hand over millions of their papers to just anyone. This blog has been covering this topic before here. I am not reading anywhere that scite has signed agreements with any of them so worst case scenario the startup will have to spend all of their 225 thousand dollars on journal subscriptions. Probably is not going to be enough.
Side note. This week newspaper The Guardian had this article out on consensus on global warming. The article mentions the well-cited article from 2013 (DOI) from which the claim is derived that 97% of all scientists agree that global warming is man-made. Surely scite will uncover that 100% of all scientist agree with the article that states that all scientists agree that global warming is man-made? Not so, just one citation in support and another 100 merely mentions. And this is odd, now that we are on the abstract page of the John Cook article anyway: why does it say with respect to anthropogenic global warming (AGW) that "We find that 66.4% of abstracts expressed no position on AGW, 32.6% endorsed AGW, 0.7% rejected AGW and 0.3% were uncertain about the cause of global warming". 97% is not 32.6%?