Sunday, October 04, 2009

Zotero plugin: auto-fetching number of citations from Google Scholar

Due to my study I have to read a lot of papers and, therefore, organize the bibliography in some way. The first software that I found for this purpose was EndNote. However, I don't like the idea of storing all data on a local drive, especially when you need to access the information for several computers. I started to look for something else and found a very nice and convenient tool called Zotero (http://www.zotero.org/). All data are stored on their server and you only need to install Zotero's extension for Firefox that will automatically synchronize with the server.


Everything is fine except absence of data about number of citations. Usually, you need to choose the most recent or the most cited papers. The first solution was just to find the number of citations in Google Scholar and enter it manually for each paper. Unsurprisingly, after a hundred of papers I found this solution inefficient. So I decided to develop a simple plugin for Zotero that will do all work for me :)

All you need is to install the plugin and add the field Call Number to the papers listing table. I chose this field because I don't use it and Zotero doesn't have any special field for the number of citations. Citations will be automatically fetched from Google Scholar for each new paper. For old papers it is possible to update this number, which is useful because it may change over time. You can update it for a particular paper, a collection or all papers at once.

Finally, here is the most important thing — the download link :)

https://addons.mozilla.org/en-US/firefox/addon/14667/

75 Comments:

Bilal Abdul Kader said...

Greetings,
It is a nice piece of software indeed. However, I suggest storing the citations in a different table that is linked by itemID to the main table.

This is far more efficient and structured. Anyway, you would be installing a new plugin so let the plugin creates its own dataspace.

bilal

Anton said...

Bilal, thank you for the feedback.

It is possible to store the data in a separate table, but the problem is to display it in the documents listing. According to Dan Stillman (http://groups.google.com/group/zotero-dev/msg/12b5f147796a94b5) a special field can be added in the future. In this case it will be not difficult to change current behavior.

Shantanu said...

ZSC 1.0 disables in FF3.6, waiting for update!!

Anton said...

Shantanu,

I have updated the addon, please update your version from this page https://addons.mozilla.org/en-US/firefox/addon/14667/

If you have any problem, please contact me.

Unknown said...

Doesn't work for me. I imported my references from EndNote into Zotero. When I run 'Update all citations' the Call Number field is set to 0000 for all references, altough in Google Scholar a different amount is presented...
Any idea for a solution?

Windows XP
Firefox 3.6
Zotero 2.0 (installed today)
Zotero Scholar Citations 1.1 (installed today)
Zotero WinWord Integration 3.0a3 (installed today)

Ronald said...

Try opening the english version of google scholar first. Happened to me also because the dutch version of google scholar was opened instead of the english version.

Unknown said...
This comment has been removed by the author.
Unknown said...

Hi Anton, with the new Firefox update, I installed version 1.1 but it's still disabled and all of the citations I had previously input seem to have vanished. Help!

Anton said...

Hi Anna,

I'm not sure what is the reason. I have the latest versions of both Firefox and Zotero, and everything works fine. You can try to remove and reinstall Zotero and Zotero Scholar Citations addon. Please let me know if this doesn't help.

Unknown said...
This comment has been removed by the author.
Unknown said...

great, will try it asap!

would be even greater, if directly integrated in the zotero client/plugin. maybe you can contribute the code to the zotero community/developers.

Unknown said...

This add-on is great. However, there are a few problems I'm running into with citations for books and book sections:

1. It seems to be unable to distinguish between citations of a book review of a book and citations of the actual book. The authors are different, so maybe that is part of a solution.
2. Same thing with a book whose title is the name of another author. e.g., The book "Merleau-Ponty" by Stephen Priest (ISBN 041530864X) is listed as having 8799 citations. Google Scholar shows only 51.
3. Same thing with book sections that have a very generic title. e.g. the book section "Postmodernism" by Noel Carroll, in the anthology "Oxford Companion to United States History" edited by Paul Boyer, is listed as having 7798 citations. There's just no way.
4. Similar problems with magazine articles.

Since the add-on is probably designed for journal articles only, perhaps you could have an option to only have it look up journal article citations.

Anton said...

Hi Jonathan,

I'm aware of this problem and going to update the add-on to make more specific queries where possible. I'll do this as soon as I get some spare time to put my hands on it again.

Shantanu said...

Dear Anton,

Waiting eagerly for ZSC upgrade for FF4...

Shantanu

Anton said...

Dear Shantanu,

The add-on has already been updated for FF4 and currently is under review. Once the review is completed, it should become available.

Best regards,
Anton

Shantanu said...

Thanks Anton, very impatient for the add on :D

Ben said...

Anton,
I may be being obtuse, but I have installed plugin version 1.2 on FF4.0.1 with Zotero 2.1.6 and I can't for the life of me figure out how to batch fetch citation numbers for the existing library. It doesn't appear in any of the existing menus, preference panes, or on right click of an existing record. I really love the idea of this plugin. If you could point me in the right direction I would really appreciate it.
-Ben

Anton said...

Hi Ben,

I have received a report about this problem from another person. However, I have exactly the same setup (same versions) and couldn't reproduce the problem. The option "Update citations" should appear at the bottom of the menu when you right-click on a record, set of selected records or a collection. Unfortunately, at this moment I have no idea of what may cause the problem you described. I will let you know if I get any new information.

Thanks,
Anton

Som said...

Similar problem + 1 more. On my notebook after install I could find the menu on right click of the 'My Library' collection and it updated all my citations. However, now the right click option is gone! So I tried to to install the add-on on my Desktop and I get the following error "The add-on from addons.mozilla.org could not be installed because it is not the add-on Firefox expected".

Som said...

Update: I was able to install the add-on by downloading to desktop and then using Firefox 'Install from File'. Still not menu though.

Som said...

OK I found the problem and the solution...
The popup items appear just fine when Zotero is loaded in the 'Browser Pane' not in its 'Own Tab'. When Zotero is in 'Own Tab' mode, the menus disappear. Cause: The overlay.xul that this add-on appends applies only to the 'Browser Pane' mode. To add menus to the 'Own Tab' mode, the ZoteroPane.xul needs to be appended by replacing the line in Chrome.Manifest by this one:
overlay chrome://zotero/content/zoteroPane.xul chrome://zotfile/content/overlay.xul

Som said...

Made an error in the previous post. The like should be...
overlay chrome://zotero/content/zoteroPane.xul chrome://zoteroscholarcitations/content/overlay.xul

Anton said...

Hi Som,

Thank you for finding the solution! For some reason, I haven't seen this problem myself, everything worked fine for me. Nevertheless, I've updated the add-on according to your suggestion and uploaded it. The new version should become downloadable soon once it's reviewed. Please let me know whether the new version works for you.

Thanks,
Anton

Som said...

Anton, Thanks for updating your add-on. It is very useful.

I did find another unrelated issue: When I attach a snapshot of a webpage to an existing item, the add-on tries to update the citation count for the web page and never times out thus forcing me to exit Firefox.

Anton said...

Hi Som,

Thank you for reporting the problem. I'll try to find a solution and update the add-on accordingly.

Michael X said...

Any plans to update it so that it runs from Firefox 5?

cejote said...

It is a great plug-in. I would really appreciate if it would work on Firefox 5 as well :D

Anton said...

A new version of the plugin supporting Firefox 5 has finally been reviewed and is now available for download!

Amin said...
This comment has been removed by the author.
Amin said...

Thanks for the great work.
For some reason it fetches zero for some papers from springer.
e.x.
http://www.springerlink.com/content/vvn64u2qdhtt0dhv/

I would really appreciate if you take a look at this.

Anton said...

Hi Amin,

Thank you for reporting the problem! I'll have a look at it as soon as I get some spare time.

Cheers

VSMU said...

Hi Anton,

I just use this add-on because I like zotero. I have try to update all citations at the first time after install this add-on. There is no any diaglog show the progress. Then, I try to update one by one but it said 'Some of the request to Google Scholar failed. Probably due to large number of requests'. I cannot stop this request. Then, I exit Firefox and hope that the request will be cancelled but no. I have about 1400 citations in the database.

How to solve this problem?

Thank you.

Anton said...

Hello,

I've removed the progress dialog in the recent version, as some people didn't like that it blocks Zotero until the updating process is over.

The only way to avoid being blocked by Google for multiple requests is to update fewer citations at once. Maybe around 100 at once should be alright, but I'm not sure about the exact number.

Cheers.

Anton said...

Maybe a good idea could be to insert say a 30 seconds delay between every 100 requests. I'll think about it and maybe add into the new version.

VSMU said...

It is a nice idea to use time delay. However, it will be better if you allow user to set options by themselves. As you don't have screen of option, you should provide this screen. We don't know the exactly number or time of request that google scholar allowed. It is good to let user try and share. Then, you can use that for default value in option.

Anton said...

This sounds good, I'll see what I can do. Thank you for a suggestion.

VSMU said...

I got a problem of updating. Now, I cannot update any paper. When I click 'update citation', I got only '0000'. I don't what's happened.

Do you have the same problem? Is there any way to solve this problem?

Thanks.

Anton said...

Do you still have that problem? Maybe it was a temporary blocking by Google. Unfortunately, by today I haven't got any time to work on improving the add-on.

VSMU said...

I still have that problem. However, I can visit Google Scholar and check the citation. There is no problem. Some paper that I already got the number of citation, then I update, the result is '0000' only.

Take a time to check. I just inform you the problem.

Thanks.

Anton said...

Hi Titi,

Unfortunately, I haven't had time to make any updates in the addon. But the source is indeed available online: https://github.com/beloglazov/zotero-scholar-citations

If you would be able to make any improvements, I will be happy to merge them with the main repository and update the addon.

Thanks.

Anton said...

Can you please give me an example of a paper that the plugin returns 0000 for? Please tell me the title, authors and data exactly as they are entered in the corresponding fields in Zotero. I'll try to test it.

Anton said...

I've just released a new version - 1.5. Here is the list of changes:

- Relaxed the constraints of Google Scholar queries: only the first author's last name, and only the lower bound in the year range.
- Added URL encoding of titles and author names, which should help avoid query mistakes due to special characters.
- Added a 1 minute delay after every 50 items to attempt avoiding being blocked by Google Scholar for multiple subsequent queries.

Anton said...

Hi Titi,

I've just tested the new version with the paper you mentioned, and it works correctly. Please note that the new version (1.5) is not approved yet on addons.mozilla.org, as it usually takes some time for the moderators to review new versions. However, it should become available in a few days.

Cheers,
Anton

Anton said...

Great! :)

Charl P. Botha said...

Great plugin, except that mine has recently stopped working with Zotero 3.0.3 in Firefox 11.0 with the following error message when I try to update citations:

JavaScript Error: "Zotero is undefined" {file: "chrome://zoteroscholarcitations/content/scripts/zoteroscholarcitations.js" line: 62}]

Any ideas?

(by the way, your embedding of this comment frame makes it REALLY hard to fill in the google account captcha. I had to use the HTML inspector to set the height of the blogger-iframe-colorize to 500 instead of 275, which was clipping and not scrolling)

Charl P. Botha said...

I just restarted Firefox, now it seems to work again. There was no internet connection when FF started, that could have caused the problem.

Your comment box still needs to be fixed however. :)

Anton said...

I'm glad the plugin works for you. Sorry for the comment box, it's the default one, I don't have much control here :) (but it looks ok in Chrome).

Anonymous said...

Hello and thank you! (that said, I am experiencing technical difficulties)

I can update one reference at a time by right clicking and selection of Update Citation. Following that, I receive the error message "Some of the requests to Google Scholar failed. Probably due to large number of requests."

By closing and reloading Firefox, I can collect another citation. I need to continue this cycle to keep updating my collection. While a nice distraction from dissertation writing, I'd prefer to go for a walk outside...

I attempted to update all citations. That did not work.

My setup:
Windows 7; Firefox 12.0; Zotero 3.0.7

Any help would be appreciated.

I also had problems with the captcha in Chrome and IE.

Anton said...

Hi, unfortunately, there is no way to avoid being blocked by Google. I suppose it calculates the frequency of requests and blocks the IP address if the frequency is higher than the threshold.

Usually, the threshold is high enough to allow you to update a number of references at once. However, in your case, the situation might worse due the fact that a single IP address is shared by multiple people in your university. So as I said, unfortunately, I can't do anything to solve this problem.

Good luck with your dissertation!

Anonymous said...

Thanks Anton,

Great suggestion! I got off the University proxy and was able to update larger numbers of references. Off the proxy, Google Scholar popped up a captcha stating that I had an unusual pattern of requests, or something to that effect and it wanted to check that I was human. This had not happened on the proxy.

I still am unable to update my entire collection but can now select larger number of references for update which is working fine for me.

If you are looking for an additional challenge, I am guessing everyone has errors due to typos that made it into their Zotero library. It would be nice to search with several combinations of fields to identify the combination that yields the greatest number of citations as a response variable. If an error (ie misspelled author) was identified, it would be nice if a new entry was created in the same Zotero collection with the corrected entry and the citation #. That would be fantastic and would stop me from needing to weed out errors manually to some effect.

Thank you for the best wishes on the dissertation. It is by far the biggest persistence game that I have ever played.

Anton said...

I'm glad you've found a workaround for your problem.

Thanks for the idea about error correction, it's interesting. However, the highest number of citations may not be an indicator of correctness, as using a subset of fields may lead to a completely wrong search result. Anyway, this is something to think about. Cheers!

Ben said...

Hi Anton,
I've been using your plugin for about a year, however with the switch to "Zotero Standalone" the plugin seems broken. I no longer have the update citations contextual menus.

I know it is a plugin for firefox, but I tried loading the XPI thought the Zotero plugin page, which failed. Is there anyway to support the new "Standalone Zotero"?
-Ben

Anton said...

Hi Ben,

Thanks for pointing this out. I'll see if it's possible to adapt the plugin to the standalone version of Zotero.

Cheers

GreatLeapingCrab said...

Noticed that it's stopped working, using Firefox 19.0.2 and Zotero 3.0.14. Any chance of an update?

Anton said...

Thanks for reporting the problem. I'll have a look at it soon.

Anonymous said...

How to export the "Extra" field from Zotero? I want to add this information to a bibliometric database in Excel. I tried exporting as RIS but this strips the Extra field.

Anton said...

Sorry, I'm not sure how to do that. Try searching for plugins for exporting data.

simon said...

Anton, thank you for this great Zotero plug-in! I had been using it for quite a while; however, I had to stop using it after you changed the field for storing the citations from "call number" to "extra". This is for two reasons: First, some of my Zotero entries contained actual data there, such as "in press", "submitted for review", etc. The plug-in simply replaced them (if I remember correctly). Second, I usually use Zotero together with LyX (which interchanges information via the BibTeX format). What happend for me then, was, that in bibliography entries that I created in an actual document, suddenly the citation number started to appear --- which is something that definitely should not happen!

For these reasons, I would like to ask: Is there any chance to implement a configuration option for the users to decide which field to use? I would love to switch back to "call number" in order to be able to use your great plug-in again! And just out of curiosity: what was the reason for you to change the field?

Anton said...

Hi Simon,

Thanks for your feedback! The change from call number to extra was suggested by a Zotero developer, as using the call number field for storing the number of citations is not semantically correct. I understand your concerns with the extra field. Probably the best solution would be to provide a configuration option as you suggested. I'll try to implement that in the next version as soon as I get some time to work on the plugin.

Cheers

Unknown said...

Anton,
thank you for this nice plugin, it works quite well for me.

I just don't get how to add the field "Extra" to the papers listing table because this field does not belong to the list of addable fields when I click the top right icon of the central pannel. (I can add fields "Type", "Date", "Year", ... but not "Extra").

Do you have any advice to make it appear into the list?
Thank you again.

Anton said...

Hi Remy,

Hmm, I'm not sure why you don't see the Extra field in the list. I've just installed the latest version of Zotero and Zotero Scholar Citations, and when I click on that top right icon I can see the Extra field in the bottom of the list between "Date Modified" and "Attachments". Can you see these fields?

Unknown said...

Hi Anton,

I updated from version 3.0 to version 4.0 (latest) and now I can see the "Extra" field into the list.
surprisingly enough I could not see it using version 3.0.

Anyway, now it works perfectly for me, thank you for your answer and for your nice plugin :)

Anton said...

Hi Remy,

No worries, I'm glad you got it working!

Anton said...

I've just updated the plugin, it's currently under review but can be downloaded from https://github.com/beloglazov/zotero-scholar-citations/raw/master/builds/zotero-scholar-citations-1.8.5-fx.xpi

I've changed the logic of updating the Extra field:
- if it's empty, the number of citations is set
- if it's non-empty but only contains the number of citations, the number of citations is updated
- if it's non-empty and contains data other than the number of citations, the number of citations is prepended to the existing data

This way the existing data do not get overwritten, and the option for select the field is not required.

Unknown said...

I'm getting 00000 counts for citations that the Google Scholar website list as having more citations, and this is still the case after more than 48 hours. I've tried emptying cookies out manually and via CCleaner, I've reset my modem so that my ip is different, I tried using a vpn, tried using it logged into my google account and logged out.

I switched to my work computer and that worked for about 45 references and then stopped working and was again replacing older citations with numbers in them with 00000.

I don't understand what it is blocking when even a change of IP and clearing of cookies isn't working?

Anton said...

Hi Lune, do you still have this issue? I've just tested the addon on my machine, and don't have any problems. Each time the blocking happens, a new tab with a Captcha should open, which after solving should allow you to continue updating the citations.

Unknown said...

Yes, I'm still having it. Sometimes I get a captcha, sometimes it's just a block notice, sometimes there's not even any notice, just all searches return 00000, and checking at the Google Scholar website reveals they're wrong.

I've tried pacing it out by updating 5, pause, updating the next 5 and so on.

I think I read one patch update that said that they'd stopped making Zotero pause 2 minutes for every 50 references? In any case, I'm spacing it out more than that.

Anton said...

Did increasing the pause help in any way? If you find a solution, please feel free to send a pull request on github (https://github.com/beloglazov/zotero-scholar-citations) -- I'll be happy to merge your changes in.

Unknown said...

No, I don't think it made a difference. The weird thing is this: from home, I was initially hitting the wall at around 45-55 references. At work, I could do more references.

Over this last weekend, though, I think I got to around 100 references (done in lots of 5-10) before I hit the wall. So when I came to work, I thought I'd get as much but then hit the wall early on, around 40 something.

Here's another observation - not only did filling the captcha not work but also clearing the cache, even changing my ip by resetting my modem, using a vpn, using cc cleaner to clean out my cookies... all these did not work and I just had to wait more than 24 hours for Google to reset. This was the same at work and home - when Google decided, "Nope", that was it for me.

I'm trying to thin out my Zotero collections using a combination of citation counts and qualitative judgements about relevancy and value after more than 5 years using it, so this is a bummer...

Anton said...

I'm sorry to hear about the problems you are having. Unfortunately, I'm not aware of a solution to them.

Kevin said...

Thank you for this very useful plugin! I am using version 1.9.1.1 with Zotero standalone. For the most part, it has worked really well and the citation count is extremely useful for sorting and filtering. However, several papers in the library will erroneously display 00000 citations. A common thread between those papers is that they have superscripts or symbols in the article title. As a test, I modified the titles to avoid those characters and then the plugin was able to retrieve the correct result. I am not sure if there is an easy solution, but is there a way to differentiate a true 00000 result from a false 00000. Thanks again!

Unknown said...

This is a great add-on and helped me a lot! However, it is incompatible wiht the new Zotero 5.0.28, is anyone else suffering from the same problem? Is there a way to solve this problem? Thanks!

Unknown said...

It is not working for the current version of Zotero.

Unknown said...

Hi, thanks for the amazing add-on! I have exactly the same problem as above: getting 00000 counts for citations. For me, it stopped extracting the number of citations after approximately 10-15 publications. I don't get any notification or Captcha. Have you ever found a solution to this issue?

shamsuddeen hassan muhammad said...

Note: The plugin is no longer maintain by th author... See here who the maintained version here: https://www.zotero.org/support/plugins

Post a Comment