[Mitch]

Dissociating the good literature from the bad literature is an endeavor we all do individually(but not for long: ChemRank). If only there was some website where we could tell whether that syn. prep. is accurate or that physical model is valid. The current methods to determine the validity of the literature are: to perform the same experiment, try to determine this from how many people cite that article, go ask around the department for someone who did something similar, try to relate the quality of the paper from the h-index of the author.

But, what if you still want more? What if you want something more than a numerical qualifier of a paper's worthiness, or of an author's scientific quality? What if you want to have a discussion beyond the scope of your labmates? What if you want to have an intellectual discussion about the current chemical literature with every other chemist on the planet? How do you do this and where do you begin?

The chemical blogosphere has done a rather good job in keeping the community of internet savvy chemists a breasted on some of the latest and coolest research (http://wiki.cubic.uni-koeln.de/cb/index.php). But, unless you have a huge audience and build a faithful readership, no one will ever know your views and it can be rather difficult to have an intellectual conversation with just yourself.

With all this in mind, I set out to create a website that would overcome these historic limitations to academic communication. I created a new website called ChemRank: http://www.chemrank.com Screen Shot shown Below



At ChemRank you can add a paper to the database and then vote whether you find it a good or bad paper. You can also leave comments critiquing the yield or congratulating the authors on a job well done. Most importantly, there is a public record of your views! Your pain does not need to be repeated if you point out the problematic reaction or the incorrect eqn in the comments section. 

An other cool feature of ChemRank is the building of a database that knows the good chemical literature from the bad chemical literature. Currently, there is no database to my knowledge, except your own brain or PI's brain, that tracks this information. Although, Noel's experiments with Connotea is a step in the right direction.

There is even something for the already established pioneers in online chemical literature discussions (aka chemical bloggers) too. For each paper added to the database a script, behind the scenes, will generate a little code snippet to allow voting on the literature your discussing on your blog too. For example, if you are discussing the recent paper "Atomic Structure of Graphene on SiO2" you can generate the following from copy/pasting a code snippet. Assuming you've already added the paper to the database and clicked the link: "add to your blog"



This by no means is a completed project, there is still much more to do.

• Make links that show the papers posted last 7days, 1month, 1 year, all time
• Make users register before posting.
• Add the ability to tag the articles with relevant keywords
• Make a pipe's rss feed of the author's most recent papers, on the fly and behind the scenes, display in the description as well.
• Make an api so others can access the database and use it for cool-new mashups.
• Add a search feature, but that is kind of pointless while the database has less than 10 papers in it.

All of the above can be done in time, but it depends on the feedback from the community and the popularity of the website. As with all projects, I don't necessarily expect it to catch on right away if at all. A lot of times I code, simply to show how to do it, and how to do it well. Allowing public comment on the current literature is something the chemical publishers should be doing anyways, now we no longer have to wait for them...

Mitch

[Mitch]

Comments / feedback?

[Yggdrasil]

Perhaps put the name of the journal somewhere in the summary box.

[Mitch]

Unfortunately there is no easy way to grab that information from a database quickly and on the fly. Should I add a field to the submission form to include it?

[excimer]

Running into one of my personal problems: five people have knocked down the paper I enjoyed. Why? No comments made. Yay for scientific discussion on the internet! 

[Mitch]

For every 10 thumb ups/thumb downs there will a comment left. That is just the nature of Web 2.0 unfortunately.

What is the percentage of unique people that go to your website and leave a comment?

[Ψ×Ψ]

We average about 200ish hits a day most of the time.  You can see how many comments we get.  In the day following a post, we may get ten or so; on days we don't post, we maybe get one or two.

[Mitch]

That ratio seems about right. I would expect something similar for ChemRank too.

[kayesdee]

Mitch,

the basic idea is very cool. However: with the current system it is just too much fun to click on the red/green flags of articles without any further action. No wonder people are just messing with this. As of now I would consider the whole thing to be untrustworthy. And since you are doing the voting via form anyone could, if mean enough, just write a little bot that POSTs random votes. Well enough you check IP addresses (cookies? havent checked it) that noone votes twice. But this way you will not really get any number of comments – and basically this would be the place where comments are essential. i.e. the rating system right now does not work as intended, right?

What chances do you have? Force people to add a comment on every vote? Will not increase votes, but it will keep scores from getting messed up by "vandals".

I strongly believe that in the end we will have to settle for a different system: anyone willing to comment on a paper will do so in her/his blog, utilizing microformats (see structuredblogging) -- which would then be aggregated by a site that collect all those reviews and summarizes/averages the scores. In this way one could even provide a nice template for those reviews incorporating scores for creativity, reproducibility, good writing, etc.

decentralized voting. centralized reporting.

Whattaya think? 

[Mitch]

the basic idea is very cool. However: with the current system it is just too much fun to click on the red/green flags of articles without any further action. No wonder people are just messing with this. As of now I would consider the whole thing to be untrustworthy. And since you are doing the voting via form anyone could, if mean enough, just write a little bot that POSTs random votes. Well enough you check IP addresses (cookies? havent checked it) that noone votes twice. But this way you will not really get any number of comments – and basically this would be the place where comments are essential. i.e. the rating system right now does not work as intended, right?

Trust in what sense? Perhaps the goal wasn't stated to clearly. What I wanted to see, eventually, was recent good chemical literature rise up on ChemRank so more people could see it and then after a week on the front page it would go into the archive. It really was just meant as a Digg.com clone not a definitive algorithm for telling everyone what good papers are, necessarily. In a way, if a paper in that week creeps up to the top, that probably means it is of some quality, but that would be about as much information you could get from it.

What chances do you have? Force people to add a comment on every vote? Will not increase votes, but it will keep scores from getting messed up by "vandals".

A comments section was added so there would be a record of why a person voted a paper up or down if they elected to give a reason. I don't believe in forcing people to do this. Most won't, and that is just the nature of things.

I strongly believe that in the end we will have to settle for a different system: anyone willing to comment on a paper will do so in her/his blog, utilizing microformats (see structuredblogging) -- which would then be aggregated by a site that collect all those reviews and summarizes/averages the scores. In this way one could even provide a nice template for those reviews incorporating scores for creativity, reproducibility, good writing, etc.

I've tried to understand microformats, but they make no sense to me. If I can't get it, most bloggers probably won't adopt it anytime soon either.

decentralized voting. centralized reporting.

Whattaya think? 

Sounds fine. ChemRank can be made to aggregate just as well as any other site.

[kayesdee]

Trust in what sense? Perhaps the goal wasn't stated to clearly. What I wanted to see, eventually, was recent good chemical literature rise up on ChemRank so more people could see it and then after a week on the front page it would go into the archive. It really was just meant as a Digg.com clone not a definitive algorithm for telling everyone what good papers are, necessarily. In a way, if a paper in that week creeps up to the top, that probably means it is of some quality, but that would be about as much information you could get from it.

Well, digg and the like can work with economies of scale. If you only get 5 votes - what is that going to say? There aren't that many scientists out there rating papers, much less chemists and much less organic chemists or any other even more specific subgroup. And thus, as you can already see, random voting vandalism becomes even more grave than in those cases. It is just a much more diverse group of topics to vote on, as well as potential voters are rare. At least on digg you need to be registered to vote. (oh, btw. see florida).

A comments section was added so there would be a record of why a person voted a paper up or down if they elected to give a reason. I don't believe in forcing people to do this. Most won't, and that is just the nature of things.

Yes, you are right it is the nature of such things. But if you cannot count on getting rid of vandals by massive voter numbers, as well as if you really don't care about thumbs up or down, but real criticism, comments and rants on a paper (I would, else I wouldn't even look at a system like that but rather ignore that article. My approach to this is much more a "if there are people saying good things about the article, I might read it if it is of general interest or someone points out a specific hilight in the way the article is written").

You gotta admit: if the article somehow relates to your area of interest you read it anyhow, no matter what the head count is going to be.

I've tried to understand microformats, but they make no sense to me. If I can't get it, most bloggers probably won't adopt it anytime soon either.

You get RSS, right? XML? Just some standardized way of marking up a review blog post, so that machines can easily understand it and aggregate it and potentially summarize or categorize it. Or some other apps can use the data (such as in ical oder hcal or whatever)

The blogger per se doesn't need to understand this at all from the technical viewpoint. Just look at chemical blogspace and how it pulls out liks to papers, molecules and conferences. THAT I would love to see for comments on papers: pull out the corresponding data that gets put into the comments, all posted in their individual blogs. Make it easy like HTML markup, provide templates for wordpress, typepad (you name it, see structured blogging again). And then on chemrank you can just display a paper, with links to all comments on blogs and a summary (maybe in some numerical way or via tags or whatever). But the actual evaluation is done on a place where the author has some known credibility, personality, reputation -- identity. THEN I can start trusting an aggregated/summarized ranking of papers. THERE is valuable data. But, sorry, just a heads up or down like chemrank has it at the moment does have little value for me ;(

decentralized voting. centralized reporting.

Sounds fine. ChemRank can be made to aggregate just as well as any other site.

Mitch, I am sorry to sound a bit too critical and pessimistic about this thing as it is. Maybe my usage scenario is different from yours, maybe I am projecting much more into it than is intended?

[Mitch]

Well, digg and the like can work with economies of scale. If you only get 5 votes - what is that going to say? There aren't that many scientists out there rating papers, much less chemists and much less organic chemists or any other even more specific subgroup. And thus, as you can already see, random voting vandalism becomes even more grave than in those cases. It is just a much more diverse group of topics to vote on, as well as potential voters are rare. At least on digg you need to be registered to vote. (oh, btw. see florida).

Of course it works on scale. Often I feel, I make tools and resources for the future professional Chemists that are still in high school. If you want to view it as a weekly what's hot or not in the chemical literature from a small set of net savvy chemists that will most likely have divergent interests than you. Then, at this level in its development, that is exactly what it is. But, there is a potential for something unique, that I think you are missing. Already, you can prune by submitter and tags and all that nifty stuff. You also have a web developer, who is a chemist, actively seeking input from you on how to make it better and more relevant for you, that is also important.

If the question is a TrustRank. You can already view submissions based on the submitter and soon you'll be able to track their comments as well. You can trust the people who have submitted and commented on the articles to the database that you like.

Yes, you are right it is the nature of such things. But if you cannot count on getting rid of vandals by massive voter numbers, as well as if you really don't care about thumbs up or down, but real criticism, comments and rants on a paper (I would, else I wouldn't even look at a system like that but rather ignore that article. My approach to this is much more a "if there are people saying good things about the article, I might read it if it is of general interest or someone points out a specific hilight in the way the article is written").

The comments section is there for you to see what comments people have put out about the article, so we are both in agreement?

You gotta admit: if the article somehow relates to your area of interest you read it anyhow, no matter what the head count is going to be.

Yes, and now you have a place to put your views about it without the need for a blog.

The blogger per se doesn't need to understand this at all from the technical viewpoint. Just look at chemical blogspace and how it pulls out liks to papers, molecules and conferences. THAT I would love to see for comments on papers: pull out the corresponding data that gets put into the comments, all posted in their individual blogs. Make it easy like HTML markup, provide templates for wordpress, typepad (you name it, see structured blogging again). And then on chemrank you can just display a paper, with links to all comments on blogs and a summary (maybe in some numerical way or via tags or whatever). But the actual evaluation is done on a place where the author has some known credibility, personality, reputation -- identity. THEN I can start trusting an aggregated/summarized ranking of papers. THERE is valuable data. But, sorry, just a heads up or down like chemrank has it at the moment does have little value for me ;(

There are a lot of different ways to view ChemRank. One way is as a huge group blog. It is very possible to export the comments made on papers at ChemRank to the Chemical Blogspace in the same manner that you are saying a regular single blog does. The potential is easily there. That's why I like having this discussion with you, because it lets me see out of the narrow box that I may view my own project in.

Mitch, I am sorry to sound a bit too critical and pessimistic about this thing as it is. Maybe my usage scenario is different from yours, maybe I am projecting much more into it than is intended?

Its okay. I like pessimistic people, they keep all of us optimists grounded in reality. Here is my personal take on the matter. When I made Chemical Forums, there already existed many-many other chemistry forums on the internet. But, I knew I could do it better or I wouldn't of made it. People always said, "yeah Chemical Forums is nice, but why use it when I frequent these other more populated chemistry forums." The reason why chemical forums became the premier chemistry forum, is because of the community of people that helped answer questions and helped make this place a great site. The eventual success of ChemRank will be on building a community. I agree it is difficult, but you can't argue that I haven't had some success in doing just that in the past...

Mitch

[Mitch]

The feed can now be found here: http://feeds.feedburner.com/chemrank

With DOI's and ranks all embedded into the feed for those big aggregation sites out there.

[Bronwen Dekker]

I really like this idea. If it works it might make the job of finding good potential authors a little easier too.

Perhaps it might be good to make it so that the person who adds the article needs to comment on why they have added it? At the moment, there is nothing stopping someone like me from adding all the articles from 'my' journal - would you want that to happen?