Inspec query and getting pdf files
Jochen Schroeder <[email protected]> Sat, 08 Oct 2005 22:07:20 +1300
| Newsgroups | gmane.comp.gnome.apps.pybliographer |
|---|---|
| Message-ID | <[email protected]> |
Hi all, I recently discovered pybligrapher and found it really great. Well I added some functionallity which I needed. The attached patch adds a query to the inspec database and a new option in the file menu, that lets you get a pdf from physical review osa and ieee and adds the save location as an url entry. the physical review and osa get_pdf functions use the volume, number and page entry to get the pdf. unfortunatly ieee does not let search for a certain page number, so i had to use the title field, this sometimes causes problems, when there are more than one article for the title. the inspec query currently only supports the inspec syntax. The urls for the journal pages and the inspec query can be configured in the preferences. I have only used urls different from the defaults, as we have proxy acces here at my uni (auckland). The defaults should work however. The other thing is that I added an option for Inspec to use password authentification, this is untested, as I don't have password authentification, if somebody could test it and tell me if it works, would be great. I also had to change something in the ovid import format, because the way the import was, it would not open my inspec files. I'm not really an experienced programmer, so the code might be really ugly. I also used BeautifulSoup and ClientCookie, so that are two more dependencies. The actions are still quite rough around the edges, ATM errors if it cannot find the results, etc are not very meaningful. This is the first time I created a patch I hope I have done it correctly. The patch is agains 1.2.6.2. I hope somebody finds this useful. If somebody has any ideas on how I could improve this, I am glad to here about it, I'm still very much a python novice. Cheers Jochen Jochen
inspecquery-getpdf.patch
(text/x-patch, 46.3 KB)
diff -Naur pybliographer-1.2.6.2/glade/inspec.glade pybliographer-new/glade/inspec.glade
--- pybliographer-1.2.6.2/glade/inspec.glade 1970-01-01 12:00:00.000000000 +1200
+++ pybliographer-new/glade/inspec.glade 2005-10-08 19:51:35.037498824 +1300
@@ -0,0 +1,329 @@
+<?xml version="1.0" standalone="no"?> <!--*- mode: xml -*-->
+<!DOCTYPE glade-interface SYSTEM "http://glade.gnome.org/glade-2.0.dtd">
+
+<glade-interface>
+<requires lib="gnome"/>
+
+<widget class="GtkDialog" id="_w_inspec">
+ <property name="title" translatable="yes">Inspec Query</property>
+ <property name="type">GTK_WINDOW_TOPLEVEL</property>
+ <property name="window_position">GTK_WIN_POS_NONE</property>
+ <property name="modal">False</property>
+ <property name="resizable">True</property>
+ <property name="destroy_with_parent">False</property>
+ <property name="decorated">True</property>
+ <property name="skip_taskbar_hint">False</property>
+ <property name="skip_pager_hint">False</property>
+ <property name="type_hint">GDK_WINDOW_TYPE_HINT_DIALOG</property>
+ <property name="gravity">GDK_GRAVITY_NORTH_WEST</property>
+ <property name="focus_on_map">True</property>
+ <property name="has_separator">True</property>
+
+ <child internal-child="vbox">
+ <widget class="GtkVBox" id="dialog-vbox1">
+ <property name="visible">True</property>
+ <property name="homogeneous">False</property>
+ <property name="spacing">0</property>
+
+ <child internal-child="action_area">
+ <widget class="GtkHButtonBox" id="dialog-action_area1">
+ <property name="visible">True</property>
+ <property name="layout_style">GTK_BUTTONBOX_END</property>
+
+ <child>
+ <widget class="GtkButton" id="cancelbutton1">
+ <property name="visible">True</property>
+ <property name="can_default">True</property>
+ <property name="can_focus">True</property>
+ <property name="label">gtk-cancel</property>
+ <property name="use_stock">True</property>
+ <property name="relief">GTK_RELIEF_NORMAL</property>
+ <property name="focus_on_click">True</property>
+ <property name="response_id">-6</property>
+ </widget>
+ </child>
+
+ <child>
+ <widget class="GtkButton" id="okbutton1">
+ <property name="visible">True</property>
+ <property name="can_default">True</property>
+ <property name="has_default">True</property>
+ <property name="can_focus">True</property>
+ <property name="label">gtk-ok</property>
+ <property name="use_stock">True</property>
+ <property name="relief">GTK_RELIEF_NORMAL</property>
+ <property name="focus_on_click">True</property>
+ <property name="response_id">-5</property>
+ <accelerator key="Return" modifiers="0" signal="clicked"/>
+ </widget>
+ </child>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">False</property>
+ <property name="fill">True</property>
+ <property name="pack_type">GTK_PACK_END</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GtkVBox" id="vbox1">
+ <property name="visible">True</property>
+ <property name="homogeneous">False</property>
+ <property name="spacing">5</property>
+
+ <child>
+ <widget class="GtkLabel" id="label5">
+ <property name="visible">True</property>
+ <property name="label" translatable="yes">Enter your Inspec Query</property>
+ <property name="use_underline">False</property>
+ <property name="use_markup">False</property>
+ <property name="justify">GTK_JUSTIFY_LEFT</property>
+ <property name="wrap">False</property>
+ <property name="selectable">False</property>
+ <property name="xalign">0.5</property>
+ <property name="yalign">0.5</property>
+ <property name="xpad">0</property>
+ <property name="ypad">0</property>
+ <property name="ellipsize">PANGO_ELLIPSIZE_NONE</property>
+ <property name="width_chars">-1</property>
+ <property name="single_line_mode">False</property>
+ <property name="angle">0</property>
+ </widget>
+ <packing>
+ <property name="padding">10</property>
+ <property name="expand">False</property>
+ <property name="fill">False</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GtkHBox" id="hbox1">
+ <property name="visible">True</property>
+ <property name="homogeneous">False</property>
+ <property name="spacing">5</property>
+
+ <child>
+ <widget class="GtkLabel" id="label1">
+ <property name="visible">True</property>
+ <property name="label" translatable="yes">Search
+Inspec for:</property>
+ <property name="use_underline">False</property>
+ <property name="use_markup">False</property>
+ <property name="justify">GTK_JUSTIFY_LEFT</property>
+ <property name="wrap">False</property>
+ <property name="selectable">False</property>
+ <property name="xalign">0.5</property>
+ <property name="yalign">0.5</property>
+ <property name="xpad">0</property>
+ <property name="ypad">0</property>
+ <property name="ellipsize">PANGO_ELLIPSIZE_NONE</property>
+ <property name="width_chars">-1</property>
+ <property name="single_line_mode">False</property>
+ <property name="angle">0</property>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">False</property>
+ <property name="fill">False</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GnomeEntry" id="_w_keyword">
+ <property name="visible">True</property>
+ <property name="history_id">medline</property>
+ <property name="max_saved">10</property>
+
+ <child internal-child="entry">
+ <widget class="GtkEntry" id="keyword1">
+ <property name="visible">True</property>
+ <property name="can_focus">True</property>
+ <property name="editable">True</property>
+ <property name="visibility">True</property>
+ <property name="max_length">0</property>
+ <property name="text" translatable="yes"></property>
+ <property name="has_frame">True</property>
+ <property name="invisible_char">*</property>
+ <property name="activates_default">False</property>
+ </widget>
+ </child>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">True</property>
+ <property name="fill">True</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GtkLabel" id="label2">
+ <property name="visible">True</property>
+ <property name="label" translatable="yes">Maximum number
+of results:</property>
+ <property name="use_underline">False</property>
+ <property name="use_markup">False</property>
+ <property name="justify">GTK_JUSTIFY_LEFT</property>
+ <property name="wrap">False</property>
+ <property name="selectable">False</property>
+ <property name="xalign">0.5</property>
+ <property name="yalign">0.5</property>
+ <property name="xpad">0</property>
+ <property name="ypad">0</property>
+ <property name="ellipsize">PANGO_ELLIPSIZE_NONE</property>
+ <property name="width_chars">-1</property>
+ <property name="single_line_mode">False</property>
+ <property name="angle">0</property>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">False</property>
+ <property name="fill">False</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GtkSpinButton" id="_w_max_results">
+ <property name="visible">True</property>
+ <property name="can_focus">True</property>
+ <property name="climb_rate">1</property>
+ <property name="digits">0</property>
+ <property name="numeric">False</property>
+ <property name="update_policy">GTK_UPDATE_ALWAYS</property>
+ <property name="snap_to_ticks">False</property>
+ <property name="wrap">True</property>
+ <property name="adjustment">100 1 1000 1 10 10</property>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">True</property>
+ <property name="fill">True</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GtkLabel" id="label3">
+ <property name="visible">True</property>
+ <property name="label" translatable="yes">Starting listing at
+result number:</property>
+ <property name="use_underline">False</property>
+ <property name="use_markup">False</property>
+ <property name="justify">GTK_JUSTIFY_LEFT</property>
+ <property name="wrap">False</property>
+ <property name="selectable">False</property>
+ <property name="xalign">0.5</property>
+ <property name="yalign">0.5</property>
+ <property name="xpad">0</property>
+ <property name="ypad">0</property>
+ <property name="ellipsize">PANGO_ELLIPSIZE_NONE</property>
+ <property name="width_chars">-1</property>
+ <property name="single_line_mode">False</property>
+ <property name="angle">0</property>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">False</property>
+ <property name="fill">False</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GtkSpinButton" id="_w_start_results">
+ <property name="visible">True</property>
+ <property name="can_focus">True</property>
+ <property name="climb_rate">1</property>
+ <property name="digits">0</property>
+ <property name="numeric">False</property>
+ <property name="update_policy">GTK_UPDATE_ALWAYS</property>
+ <property name="snap_to_ticks">False</property>
+ <property name="wrap">False</property>
+ <property name="adjustment">1 0 100 1 10 10</property>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">True</property>
+ <property name="fill">True</property>
+ </packing>
+ </child>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">False</property>
+ <property name="fill">False</property>
+ </packing>
+ </child>
+
+ <child>
+ <widget class="GtkFrame" id="frame1">
+ <property name="visible">True</property>
+ <property name="label_xalign">0</property>
+ <property name="label_yalign">0.5</property>
+ <property name="shadow_type">GTK_SHADOW_ETCHED_IN</property>
+
+ <child>
+ <widget class="GtkAlignment" id="alignment1">
+ <property name="visible">True</property>
+ <property name="xalign">0.0500000007451</property>
+ <property name="yalign">0.0500000007451</property>
+ <property name="xscale">1</property>
+ <property name="yscale">1</property>
+ <property name="top_padding">0</property>
+ <property name="bottom_padding">0</property>
+ <property name="left_padding">0</property>
+ <property name="right_padding">0</property>
+
+ <child>
+ <widget class="GtkScrolledWindow" id="scrolledwindow1">
+ <property name="visible">True</property>
+ <property name="can_focus">True</property>
+ <property name="hscrollbar_policy">GTK_POLICY_AUTOMATIC</property>
+ <property name="vscrollbar_policy">GTK_POLICY_AUTOMATIC</property>
+ <property name="shadow_type">GTK_SHADOW_IN</property>
+ <property name="window_placement">GTK_CORNER_TOP_LEFT</property>
+
+ <child>
+ <widget class="GtkTextView" id="textview1">
+ <property name="visible">True</property>
+ <property name="can_focus">True</property>
+ <property name="editable">False</property>
+ <property name="overwrite">False</property>
+ <property name="accepts_tab">True</property>
+ <property name="justification">GTK_JUSTIFY_LEFT</property>
+ <property name="wrap_mode">GTK_WRAP_WORD</property>
+ <property name="cursor_visible">True</property>
+ <property name="pixels_above_lines">0</property>
+ <property name="pixels_below_lines">0</property>
+ <property name="pixels_inside_wrap">0</property>
+ <property name="left_margin">0</property>
+ <property name="right_margin">0</property>
+ <property name="indent">0</property>
+ <property name="text" translatable="yes">Currently only searches using the Ovid syntax are supported, so if you want to eg. search for the keyword fiber from author J. Smith type: fiber.mp and smith j.au. The default search is mp (mp=title, abstract, subject heading words, key phrase identifiers)</property>
+ </widget>
+ </child>
+ </widget>
+ </child>
+ </widget>
+ </child>
+
+ <child>
+ <placeholder/>
+ </child>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">True</property>
+ <property name="fill">True</property>
+ </packing>
+ </child>
+ </widget>
+ <packing>
+ <property name="padding">0</property>
+ <property name="expand">True</property>
+ <property name="fill">True</property>
+ </packing>
+ </child>
+ </widget>
+ </child>
+</widget>
+
+</glade-interface>
diff -Naur pybliographer-1.2.6.2/glade/Makefile.am pybliographer-new/glade/Makefile.am
--- pybliographer-1.2.6.2/glade/Makefile.am 2005-02-22 06:50:42.000000000 +1300
+++ pybliographer-new/glade/Makefile.am 2005-10-08 19:51:35.037498824 +1300
@@ -7,6 +7,7 @@
sort.glade \
format.glade \
medline.glade \
+ inspec.glade \
config1.glade
EXTRA_DIST = $(glade_DATA)
diff -Naur pybliographer-1.2.6.2/glade/Makefile.in pybliographer-new/glade/Makefile.in
--- pybliographer-1.2.6.2/glade/Makefile.in 2005-02-22 06:51:37.000000000 +1300
+++ pybliographer-new/glade/Makefile.in 2005-10-08 20:24:17.819110664 +1300
@@ -97,6 +97,7 @@
sort.glade \
format.glade \
medline.glade \
+ inspec.glade \
config1.glade
diff -Naur pybliographer-1.2.6.2/glade/pyblio.glade pybliographer-new/glade/pyblio.glade
--- pybliographer-1.2.6.2/glade/pyblio.glade 2005-02-22 06:50:42.000000000 +1300
+++ pybliographer-new/glade/pyblio.glade 2005-10-08 19:51:35.025500648 +1300
@@ -17,6 +17,7 @@
<property name="skip_pager_hint">False</property>
<property name="type_hint">GDK_WINDOW_TYPE_HINT_NORMAL</property>
<property name="gravity">GDK_GRAVITY_NORTH_WEST</property>
+ <property name="focus_on_map">True</property>
<property name="enable_layout_config">True</property>
<signal name="delete_event" handler="close_or_exit" last_modification_time="Thu, 19 Feb 2004 20:39:16 GMT"/>
<signal name="key_press_event" handler="key_pressed" last_modification_time="Wed, 01 Dec 2004 19:53:14 GMT"/>
@@ -70,7 +71,7 @@
<accelerator key="g" modifiers="GDK_CONTROL_MASK" signal="activate"/>
<child internal-child="image">
- <widget class="GtkImage" id="image63">
+ <widget class="GtkImage" id="image81">
<property name="visible">True</property>
<property name="stock">gtk-convert</property>
<property name="icon_size">1</property>
@@ -94,6 +95,25 @@
</child>
<child>
+ <widget class="GtkMenuItem" id="inspec_query1">
+ <property name="visible">True</property>
+ <property name="label" translatable="yes">Inspec Query...</property>
+ <property name="use_underline">True</property>
+ <signal name="activate" handler="query_inspec" last_modification_time="Mon, 09 Jun 2003 18:56:54 GMT"/>
+ <accelerator key="i" modifiers="GDK_CONTROL_MASK" signal="activate"/>
+ </widget>
+ </child>
+
+ <child>
+ <widget class="GtkMenuItem" id="get_pdf1">
+ <property name="visible">True</property>
+ <property name="label" translatable="yes">Get PDF...</property>
+ <property name="use_underline">True</property>
+ <signal name="activate" handler="pdf_get" last_modification_time="Mon, 04 Jul 2005 06:04:12 GMT"/>
+ </widget>
+ </child>
+
+ <child>
<widget class="GtkImageMenuItem" id="_w_save_mnu">
<property name="visible">True</property>
<property name="stock_item">GNOMEUIINFO_MENU_SAVE_ITEM</property>
@@ -204,7 +224,7 @@
<signal name="activate" handler="add_entry" last_modification_time="Mon, 09 Jun 2003 18:56:54 GMT"/>
<child internal-child="image">
- <widget class="GtkImage" id="image64">
+ <widget class="GtkImage" id="image82">
<property name="visible">True</property>
<property name="stock">gtk-add</property>
<property name="icon_size">1</property>
@@ -235,7 +255,7 @@
<accelerator key="Delete" modifiers="0" signal="activate"/>
<child internal-child="image">
- <widget class="GtkImage" id="image65">
+ <widget class="GtkImage" id="image83">
<property name="visible">True</property>
<property name="stock">gtk-remove</property>
<property name="icon_size">1</property>
@@ -270,7 +290,7 @@
<signal name="activate" handler="sort_entries" last_modification_time="Mon, 09 Jun 2003 18:56:54 GMT"/>
<child internal-child="image">
- <widget class="GtkImage" id="image66">
+ <widget class="GtkImage" id="image84">
<property name="visible">True</property>
<property name="stock">gtk-sort-ascending</property>
<property name="icon_size">1</property>
@@ -304,7 +324,7 @@
<signal name="activate" handler="lyx_cite" last_modification_time="Mon, 09 Jun 2003 18:56:54 GMT"/>
<child internal-child="image">
- <widget class="GtkImage" id="image67">
+ <widget class="GtkImage" id="image85">
<property name="visible">True</property>
<property name="stock">gtk-jump-to</property>
<property name="icon_size">1</property>
@@ -325,7 +345,7 @@
<signal name="activate" handler="format_entries" last_modification_time="Mon, 09 Jun 2003 18:56:54 GMT"/>
<child internal-child="image">
- <widget class="GtkImage" id="image68">
+ <widget class="GtkImage" id="image86">
<property name="visible">True</property>
<property name="stock">gtk-execute</property>
<property name="icon_size">1</property>
@@ -405,7 +425,7 @@
<accelerator key="F1" modifiers="0" signal="activate"/>
<child internal-child="image">
- <widget class="GtkImage" id="image69">
+ <widget class="GtkImage" id="image87">
<property name="visible">True</property>
<property name="stock">gtk-help</property>
<property name="icon_size">1</property>
diff -Naur pybliographer-1.2.6.2/Pyblio/ConfDir/Inspec.py pybliographer-new/Pyblio/ConfDir/Inspec.py
--- pybliographer-1.2.6.2/Pyblio/ConfDir/Inspec.py 1970-01-01 12:00:00.000000000 +1200
+++ pybliographer-new/Pyblio/ConfDir/Inspec.py 2005-10-08 20:30:34.642824752 +1300
@@ -0,0 +1,14 @@
+from Pyblio import Config
+
+Config.define ('inspec/authentification',"""Use password authentification
+yes/no""", Config.Boolean())
+Config.define ('inspec/settings',
+ """ URL of Inspec database and password""",
+ Config.Dict (Config.String (), Config.String ()))
+
+Config.set ('inspec/authentification',0)
+Config.set ('inspec/settings', {
+ 'URL' : 'http://gateway.ovid.com/ovidweb.cgi',
+ 'Passwd' : '',
+ 'ID' : ''
+ })
diff -Naur pybliographer-1.2.6.2/Pyblio/ConfDir/Journals.py pybliographer-new/Pyblio/ConfDir/Journals.py
--- pybliographer-1.2.6.2/Pyblio/ConfDir/Journals.py 1970-01-01 12:00:00.000000000 +1200
+++ pybliographer-new/Pyblio/ConfDir/Journals.py 2005-10-08 20:29:22.302822104 +1300
@@ -0,0 +1,12 @@
+from Pyblio import Config
+
+Config.define ('journals/URLs',"""Journal sites to get the pdf-filesi""",
+ Config.Dict (Config.String(), Config.String()))
+
+Config.set ('journals/URLs', {
+ 'Physical Review' :
+ "http://link.aps.org.nz/ejnls/volpage",
+ 'OSA Journals' : "http://ol.osa.org",
+ 'IEEE Journals' :
+ "http://www.ieeexplore.ieee.org"
+ })
diff -Naur pybliographer-1.2.6.2/Pyblio/ConfDir/Makefile.am pybliographer-new/Pyblio/ConfDir/Makefile.am
--- pybliographer-1.2.6.2/Pyblio/ConfDir/Makefile.am 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/ConfDir/Makefile.am 2005-10-08 19:51:36.345300008 +1300
@@ -29,6 +29,8 @@
GnomeUI.py \
Medline.py \
Ovid.py \
+ Inspec.py \
+ Journals.py \
Refer.py
pyb_DATA = $(EXTRA_DIST)
diff -Naur pybliographer-1.2.6.2/Pyblio/ConfDir/Makefile.in pybliographer-new/Pyblio/ConfDir/Makefile.in
--- pybliographer-1.2.6.2/Pyblio/ConfDir/Makefile.in 2005-02-22 06:51:36.000000000 +1300
+++ pybliographer-new/Pyblio/ConfDir/Makefile.in 2005-10-08 20:24:17.825109752 +1300
@@ -120,6 +120,8 @@
GnomeUI.py \
Medline.py \
Ovid.py \
+ Inspec.py \
+ Journals.py \
Refer.py
diff -Naur pybliographer-1.2.6.2/Pyblio/Config.py pybliographer-new/Pyblio/Config.py
--- pybliographer-1.2.6.2/Pyblio/Config.py 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/Config.py 2005-10-08 19:51:36.391293016 +1300
@@ -147,7 +147,7 @@
return
-def set (key, value):
+def set (key, value):
try:
ConfigItems [key].set (value)
except KeyError:
diff -Naur pybliographer-1.2.6.2/Pyblio/Format/OvidLike.py pybliographer-new/Pyblio/Format/OvidLike.py
--- pybliographer-1.2.6.2/Pyblio/Format/OvidLike.py 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/Format/OvidLike.py 2005-10-08 19:51:35.948360352 +1300
@@ -161,28 +161,79 @@
# parse a source field
elif type == SourceField:
- dict_key = ' '.join(dict[key].split('\n'))
- m = self.source_re.match (dict_key.strip())
- if m:
- year, month, day = None, None, None
- j, v, s, n, p, o, y, d = m.group(
- 'journal', 'volume', 'inseries', 'number',
- 'pages', 'other', 'year', 'month')
-
- if s: ### article in a monograph series
- entry['booktitle'] = Fields.Text (j)
- if d:
- entry['beigabevermerk'] = Fields.LongText (d)
- entry.type = Types.get_entry('incollection')
-
- elif j:
- entry ['journal'] = Fields.Text (j)
- if d and not d.isspace():
- dates = d.split ()
- try:
- month = long_month [dates[0]]
- except KeyError:
- pass
+ fields = string.split(dict [key], ',')
+ if len (fields) == 1:
+ print "warning: can't parse source"
+ continue
+ journalName = string.strip(fields [0])
+ # extract volume, number, pages, ...
+ i=len(fields)-1
+ while i>=1:
+ fs = string.strip(fields[i])
+ if fs[0:4] == 'vol.':
+ entry ['volume'] = Fields.Text (fs[4:])
+ elif fs[0:3] == "no.":
+ entry ['number'] = Fields.Text (fs[3:])
+ elif fs[0:3] == "pp.":
+ fss = string.split(fs,'.')
+ entry ['pages'] = Fields.Text(fss[1])
+ journalName = journalName + ","\
+ + string.join(fss[2:], '.')
+
+ # the date field precedes pages
+ fss = string.split(fields[i-1])
+ # we have to work from the end since there may be
+ # characters unrelated to the date at the start of the
+ # field
+ try:
+ year = int(fss[-1])
+ except:
+ year = None
+ print "warning: cannot parse year"
+ print "offending line:", dict[key]
+ try:
+ month = long_month [fss[-2][:3]]
+ except:
+ month = None
+ try:
+ day = int(fss[-3])
+ except:
+ day = None
+ entry ['date'] = Fields.Date((year, month, day))
+ if year is not None or month is not None or \
+ day is not None:
+ del fields[i-1]
+ i-=1
+ else:
+ # additional information we do not want to loose
+ journalName = journalName + ", " + fs
+ i-=1
+
+ # the journal name and additional information
+ entry ['journal'] = Fields.Text (journalName)
+ continue
+
+
+
+ #year, month, day = None, None, None
+ #j, v, s, n, p, o, y, d = m.group(
+ #'journal', 'volume', 'inseries', 'number',
+ #'pages', 'other', 'year', 'month')
+
+ #if s: ### article in a monograph series
+ #entry['booktitle'] = Fields.Text (j)
+ #if d:
+ #entry['beigabevermerk'] = Fields.LongText (d)
+ #entry.type = Types.get_entry('incollection')
+
+ #elif j:
+ #entry ['journal'] = Fields.Text (j)
+ #if d and not d.isspace():
+ #dates = d.split ()
+ #try:
+ #month = long_month [dates[0]]
+ #except KeyError:
+ #pass
## import warnings
## warnings.filterwarnings ('once',
## message='date',
@@ -190,33 +241,36 @@
## warnings.warn (
## 'OVID: %s is not representable in date '
## 'field %s' %(dates[0], d), stacklevel=2)
- if len(dates) > 1:
- day = int (dates[1])
-
- if v:
- entry ['volume'] = Fields.Text (v)
+ #if len(dates) > 1:
+ #day = int (dates[1])
- if n:
- entry ['number'] = Fields.Text (n)
+ #if v:
+ #entry ['volume'] = Fields.Text (v)
- if p:
- entry ['pages'] = Fields.Text (p)
+ #if n:
+ #entry ['number'] = Fields.Text (n)
- if o:
- entry ['other-note'] = Fields.Text(o)
-
- if y:
- year = int(y)
-
- entry ['date'] = Fields.Date((year, month, day))
- else:
- print '>>> Error: Source field does not parse correctly:'
- print dict_key
- print entry
- continue
+ #if p:
+ #entry ['pages'] = Fields.Text (p)
+
+ #if o:
+ #entry ['other-note'] = Fields.Text(o)
+
+ #if y:
+ #year = int(y)
+
+ #entry ['date'] = Fields.Date((year, month, day))
+ #else:
+ #print '>>> Error: Source field does not parse correctly:'
+ #print dict_key
+ #print entry
+ #continue
return entry
+# raise TypeError, "unknown field type `%d'" % type
+
+
def parse_author (self, text):
ag = Fields.AuthorGroup ()
diff -Naur pybliographer-1.2.6.2/Pyblio/GnomeUI/Config.py pybliographer-new/Pyblio/GnomeUI/Config.py
--- pybliographer-1.2.6.2/Pyblio/GnomeUI/Config.py 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/GnomeUI/Config.py 2005-10-08 19:51:36.714243920 +1300
@@ -607,10 +607,13 @@
m, row = self.s.get_selected()
key = self.keyw.get()
#print 'UPDATE:', self.valuew.get()
- try:
+ type_string = "%s" %self.valuew.type
+
+ if type_string.split()[0] == 'Tuple':
val = tuple(self.valuew.get())
- except TypeError:
+ else:
val = self.valuew.get()
+
if self.dict.has_key (key):
for i in m:
if i[0] == key:
diff -Naur pybliographer-1.2.6.2/Pyblio/GnomeUI/Document.py pybliographer-new/Pyblio/GnomeUI/Document.py
--- pybliographer-1.2.6.2/Pyblio/GnomeUI/Document.py 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/GnomeUI/Document.py 2005-10-08 19:51:36.710244528 +1300
@@ -35,6 +35,7 @@
from Pyblio.GnomeUI import Search, Format
from Pyblio.GnomeUI.Sort import SortDialog
from Pyblio.GnomeUI.Medline import MedlineUI
+from Pyblio.GnomeUI.Medline import InspecUI
from Pyblio import Connector, Open, Exceptions, Selection, Sort, Base, Config
from Pyblio import version, Fields, Types, Query
@@ -316,6 +317,24 @@
self.open_document (url, 'medline', no_name = True)
return
+ def query_inspec (self, * arg):
+ ''' callback corresponding to the "Inspec Query..." button '''
+ if not self.confirm (): return
+
+ data = InspecUI (self.w).run ()
+ if data is None: return
+
+ url = apply (Query.inspec_query, data)
+
+ if url is None:
+ # no result.
+ self.w.error (_("Your query returned no result"))
+ return
+
+ self.open_document (url, 'ovid', no_name = True)
+ return
+
+
def merge_database (self, * arg):
''' add all the entries of another database to the current one '''
@@ -536,6 +555,36 @@
self.index.set_scroll (entries [-1])
return
+ def pdf_get (self, * arg):
+ entries = self.index.selection ()
+ if not entries: return
+
+ for entry in entries:
+ if entry.has_key('url'):
+ if not Utils.Callback (_("The selected entry already contains a URL field. Really get PDF-file?"), \
+ parent = self.w).answer ():
+ return
+
+ (url, how) = FileSelector.URLFileSelection (_("Save As..."),\
+ url = False, has_auto = False, is_save = \
+ True , is_paper = True).run ()
+
+ if url is None: continue
+
+ if os.path.exists (url):
+ if not Utils.Callback (_("The file `%s' already exists.\nOverwrite it ?") % url, parent = self.w).answer ():
+ return
+
+ try:
+ newentry=Query.get_pdf (entry,url)
+ except (Exceptions.FileError,Exceptions.SimpleError), error:
+ Utils.error_dialog (_("Error getting PDF"),\
+ error,parent=self.w)
+ return
+ self.commit_edition(entry,newentry)
+
+ return
+
def cut_entry (self, * arg):
entries = self.index.selection ()
diff -Naur pybliographer-1.2.6.2/Pyblio/GnomeUI/FileSelector.py pybliographer-new/Pyblio/GnomeUI/FileSelector.py
--- pybliographer-1.2.6.2/Pyblio/GnomeUI/FileSelector.py 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/GnomeUI/FileSelector.py 2005-10-08 19:51:36.710244528 +1300
@@ -45,7 +45,7 @@
def __init__(self, title = _("File"),
url = True, modal = True, has_auto = True,is_save = False,
- directory = None):
+ directory = None, is_paper = False):
if has_file_chooser:
FileSelect.__init__ (self)
@@ -95,14 +95,25 @@
vbox.pack_start (hbox, expand = False, fill = False)
# type selector
- hbox = gtk.HBox ()
- hbox.set_spacing (5)
- hbox.set_border_width (5)
- hbox.pack_start (gtk.Label (_("Bibliography type:")),
- expand = False, fill = False)
+ if is_paper:
+ hbox = gtk.HBox ()
+ hbox.set_spacing (5)
+ hbox.set_border_width (5)
+ hbox.pack_start (gtk.Label (_("File type:")),
+ expand = False, fill = False)
+
+ self.menu = gtk.OptionMenu ()
+ hbox.pack_start (self.menu)
+ else:
+ hbox = gtk.HBox ()
+ hbox.set_spacing (5)
+ hbox.set_border_width (5)
+ hbox.pack_start (gtk.Label (_("Bibliography type:")),
+ expand = False, fill = False)
+
+ self.menu = gtk.OptionMenu ()
+ hbox.pack_start (self.menu)
- self.menu = gtk.OptionMenu ()
- hbox.pack_start (self.menu)
if not has_file_chooser:
vbox.pack_start (hbox, expand = False, fill = False)
else:
@@ -112,10 +123,16 @@
menu = gtk.Menu ()
self.menu.set_menu (menu)
- liste = Autoload.available ('format')
+ if is_paper:
+ liste = ['pdf']
+ else:
+ liste = Autoload.available ('format')
liste.sort ()
- if has_auto:
+ if has_auto:
+ Utils.popup_add (menu, _(' - According to file suffix - '), self.menu_select, None)
+ self.ftype = None
+ elif is_paper:
Utils.popup_add (menu, _(' - According to file suffix - '), self.menu_select, None)
self.ftype = None
else:
diff -Naur pybliographer-1.2.6.2/Pyblio/GnomeUI/Medline.py pybliographer-new/Pyblio/GnomeUI/Medline.py
--- pybliographer-1.2.6.2/Pyblio/GnomeUI/Medline.py 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/GnomeUI/Medline.py 2005-10-08 19:51:36.666251216 +1300
@@ -145,3 +145,40 @@
self._w_medline.destroy ()
return data
+
+class InspecUI (Utils.GladeWindow):
+
+ gladeinfo = { 'name': 'inspec',
+ 'file': 'inspec.glade',
+ 'root': '_w_inspec'
+ }
+
+ def __init__ (self, parent = None):
+
+ Utils.GladeWindow.__init__ (self, parent)
+
+ # Fill in the combo boxes
+
+ self._w_inspec.show ()
+ return
+
+
+ def run (self):
+
+ ret = self._w_inspec.run ()
+
+ if ret != gtk.RESPONSE_OK:
+ self._w_inspec.destroy ()
+ return None
+
+ data = (
+ self._w_keyword.entry.get_text (),
+ self._w_max_results.get_value_as_int (),
+ self._w_start_results.get_value_as_int (),
+ )
+
+ self._w_keyword.append_history (True, data [0])
+
+ self._w_inspec.destroy ()
+
+ return data
diff -Naur pybliographer-1.2.6.2/Pyblio/Query.py pybliographer-new/Pyblio/Query.py
--- pybliographer-1.2.6.2/Pyblio/Query.py 2005-02-22 06:50:41.000000000 +1300
+++ pybliographer-new/Pyblio/Query.py 2005-10-08 20:36:53.086292600 +1300
@@ -19,17 +19,390 @@
#
#
+#TODO: have to make get_pdf edit the field in a way that pyblio realizes it has
+# changed
"""
-Search a keyword in a medline database
+Search a keyword in a medline or inspec database
This code has been contributed by: John Vu <[email protected]>
"""
# The time module is added for querying date ranges of publications
-import urllib, sys, re, string, time
+import urllib, sys, re, string, time, StringIO,\
+ types,os,ClientCookie,urlparse
+from BeautifulSoup import BeautifulSoup
+from Pyblio import Fields, Exceptions, Config
+
+#medline urls
+m_query_url = 'http://www.ncbi.nlm.nih.gov/entrez/utils/pmqty.fcgi'
+m_fetch_url = 'http://www.ncbi.nlm.nih.gov/entrez/utils/pmfetch.fcgi'
+
+
+def into_table(soup):
+ '''creates a nested list from an html table'''
+ table=[]
+ for row in soup:
+ rows=[]
+ for col in row:
+ if type(col) is types.InstanceType:
+ rows.append(col)
+ if rows:
+ table.append(rows)
+ return table
+
+def find_resultsno(table):
+ '''find the number of results'''
+ rows=0
+ cols=0
+ while rows<len(table):
+ cols=0
+ while cols<len(table[0]):
+ if table[rows][cols].string=='Results':
+ results_no=int(table[rows+1][cols].string)
+ return results_no
+ cols=cols+1
+ rows=rows+1
+ return -1
+
+
+def inspec_query(searchterm,displaynum,displaystart):
+ """query the inspec database as I see it, search for search term display
+ displaynum results and start at result number displaystart"""
+ #inspec urls
+ i_query_url = Config.get ("inspec/settings").data['URL']
+
+ #inspec authentification
+ i_auth = Config.get ("inspec/authentification").data
+
+ searchterm = searchterm.strip() #strip leading and trailing whitespace
+ #create temp files
+ result_file_name = os.tempnam() #tempory file for the query result
+
+
+ # create post statement for query
+ if i_auth == 0:
+ post={'T':'JS','MODE':'ovid','D':'insz','PAGE':'main',\
+ 'NEWS':'N',\
+ 'SEARCH':searchterm} # these are ovid jumpstart parameters,
+ # they can be found on the ovid support pages
+ else:
+ i_passwd = Config.get ("inspec/settings").data['Passwd']
+ i_id = Config.get ("inspec/settings").data['ID']
+ post={'T':'JS','MODE':'ovid','D':'insz','PAGE':'main',\
+ 'NEWS':'N', 'SEARCH':searchterm,'ID': i_id,\
+ 'PASSWORD' : i_passwd}
+
+ # posting does not work, need to jumpstart url directly
+ url=i_query_url + '?' + urllib.urlencode(post)
+
+ file = ClientCookie.urlopen(url)
+
+ # begin parsing the document
+ #soup=BeautifulSoup(page1.getvalue())
+ soup = BeautifulSoup(file.read())
+ file.close()
+ file=open("test.html",'w')
+ file.write(str(soup))
+ file.close()
+ # try to get the log key for the database
+ try:
+ key=soup.first('input',{'name':'S'})['value']
+ except:
+ Exceptions.SimpleError (_("could not get the log key"))
+ return
+
+ #the number of results is in a table underneath the heading Results,
+ # find that heading and put the table into a list, then pick the
+ # cell underneath the Results cell, that should be fairly robust against
+ #changes of the Layout
+ resultsentry=soup.firstText('Results')
+ results_table=resultsentry.findParent('table')
+ results_table=into_table(results_table)
+
+ results_no = find_resultsno(results_table)
+ if results_no <1:
+ raise Exception.SimpleError (_("$d results found (negative numbers indicate an error)")%results_no)
+ return
+
+ # these are needed to get the actual citation data
+ Select = soup('input',{'name' : 'SELECT'})[0]['value']
+ citman = soup('input',{'name' : 'CitMan'})[0]['value']
+ citmanprev = soup('input',{'name' : 'CitManPrev'})[0]['value']
+
+ #specify the range of results to get
+ range = "%d-%d" % (displaystart, displaystart+displaynum)
+
+ # create the second post statement to get the actual entries
+ post2={'S':key,'SELECT':Select,'CitMan':citman,'CitManPrev':citmanprev,\
+ 'cmRecordSelect':'SELECTED','cmRecords':range,\
+ 'cmFields':'ALL','cmFormat':'ovid','cmsave.x':'26',\
+ 'cmsave.y':'15'}# don't know what all these entries
+ #stand for cmFormat is the format to
+ #save in, cmRecords specifies which ones
+ #to save, cmFields specifies which fields
+ #shall be in the results
+
+
+ file = ClientCookie.urlopen(i_query_url,urllib.urlencode(post2))
+
+ # parse next page for information, need the hidden inputs which specify
+ # essentially only repeat what we specified in the last post
+ soup=BeautifulSoup(file.read())
+ file.close()
+ inputs=soup('input',{'type':'hidden'})
+ # need to decided if we want unix, mac or dos newlines
+ post3={'CRLF':'unix'}
+ for input in inputs:
+ post3[input['name']]=input['value']
+
+ file = ClientCookie.urlopen(i_query_url,urllib.urlencode(post3))
+ content = file.read()
+ file.close()
+
+ file=open(result_file_name,'w')
+ file.write(content)
+ file.close()
+ return result_file_name
+
+
+def get_pdf (entry,filename):
+ '''get a pdf file for an entry '''
+
+ #journal urls for getting pdfs
+ journals_url = Config.get ("journals/URLs").data
+
+ new=entry
+ try:
+ fulljournal=str(new['journal']).lower()
+ except:
+ raise Exceptions.SimpleError (_("The entry is missing a Journal field"))
+ return
+
+ pages=str(new['pages'])
+ vol=str(new['volume'])
+ title=str(new['title'])
+ journal=re.split('[^ a-zA-Z0-9]',fulljournal)[0].lower()
+ pages=re.split("([^0-9])",pages)[0]
+ vol=re.search("[0-9]+",vol).group()
+
+ if (pages == "" or vol == "") and title == "":
+ raise Exceptions.SimpleError (_("Missing pages or volume field, and title field"))
+ return
+
+ if journal.split(' ')[0] == 'physical' and \
+ journal.split(' ')[1] == 'review':
+ journal=journal.split(' ')
+ get_pdf_from_pr (journal,vol,pages,filename,journals_url)
+ elif journal == 'optics letters':
+ get_pdf_from_osa (journal,vol,pages,title,filename,journals_url)
+ elif re.search ('optical society of america',journal):
+ get_pdf_from_osa (journal,vol,pages,title,filename,journals_url)
+ elif journal == 'applied optics':
+ get_pdf_from_osa (journal,vol,pages,title,filename,journals_url)
+ elif re.search ('ieee',fulljournal): # ieee journals don't necessarily
+ #have the ieee in the journal title, but as publisher etc
+ get_pdf_from_ieee (title,filename,journals_url)
+ elif re.search ('lightwave technology',journal):
+ get_pdf_from_ieee (title,filename,journals_url)
+ elif journal == 'electronics letters':
+ get_pdf_from_ieee (title,filename,journals_url)
+ else:
+ raise Exceptions.SimpleError (_("%s is an unknown journal")%journal)
+ return
+
+ #test if file is pdf
+ fp=open(filename,'r')
+ identifier=fp.readline()
+ if not re.search('PDF',identifier):
+ raise Exceptions.FileError (_("The retrieved file is not a PDF"))
+ return
+ fp.close()
+
+
+ new['url']=Fields.URL(filename)
+ return new
+
+
+def get_pdf_from_pr (journal,vol,page,filename,journals_url):
+ '''get pdf from physical review'''
+
+ pr_search_url = journals_url['Physical Review']
+
+ if len(journal) < 3:
+ journalcode=journal[0][0]+journal[1][0]
+ else:
+ journalcode=journal[0][0]+journal[1][0]+journal[2][0]
+
+ journalcode=journalcode.upper()
+
+ post={'journal_code':journalcode,'volume':vol,'pageno':page}
+ url = pr_search_url+'?'+urllib.urlencode(post)
+ file = ClientCookie.urlopen(url)
+
+ get_url=file.url#in case of redirect
+
+ soup=BeautifulSoup(file.read())
+ file.close()
+
+ # get all links with text string PDF
+ links=soup.fetch('a')
+ pdflinks=[]
+ for link in links:
+ if link.string == 'PDF':
+ pdflinks.append(link['href'])
+
+ #We should only have one result
+ if len(pdflinks) < 1:
+ raise Exception.SimpleError (_("Less than 1 search result"))
+ return
+ else:
+ pdfpath=pdflinks[0]
+
+ url=urlparse.urljoin(get_url,pdfpath)
+ print url
+ file = ClientCookie.urlopen(url)
+ content = file.read()
+ file.close()
+
+ pdffile = open(filename,'w')
+ pdffile.write(content)
+ pdffile.close()
+
+ return
+
+def get_pdf_from_osa(journal,volume,page,title,filename,journals_url):
+ '''get pdf file from Optical Society of America website'''
+
+ osa_get_url = journals_url['OSA Journals']
+ osa_search_url = osa_get_url + '/search.cfm'
+ #osa journals
+ journals={'optics letters':3,\
+ 'journal of the optical society of america b':2,\
+ 'applied optics':9,\
+ 'journal of the optical society of america a':1}
+ try:
+ j_id=journals[journal]
+ except:
+ raise Exceptions.SimpleError (_("%s is not an OSA journal")\
+ %journal)
+
+ #use only volume + page or title in case one is wrong
+ if volume == "" or page == "":
+ volume = ""
+ page = ""
+ else:
+ title = ""
+
+ post={'action':'jsearch','ocis':'',\
+ 'journal':j_id,\
+ 'term1':'title',\
+ 'match1':'match',\
+ 'text1':title,\
+ 'join1':'and',\
+ 'term2':'abstract',\
+ 'match2':'match',\
+ 'text2':'',\
+ 'join2':'and',\
+ 'term3':'author',\
+ 'match3':'match',\
+ 'text3':'',\
+ 'year':'',\
+ 'startyear':'',\
+ 'endyear':'',\
+ 'volume':volume,\
+ 'issue':'',\
+ 'page':page,\
+ 'ocisjoin':'and',\
+ 'sort':'journal'}
+
+ url=osa_search_url+'?'+urllib.urlencode(post)
+
+ file = ClientCookie.urlopen(url)
+ soup=BeautifulSoup(file.read())
+ get_url=file.url # in case we get redirected
+ file.close()
+
+ abstract_links=soup.fetch('a',{'href': re.compile('id=')})
+
+ #we want only one search result
+ if len(abstract_links) > 1:
+ raise Exception.SimpleError (_("More than 1 search result"))
+ return
+ elif len(abstract_links) < 1:
+ raise Exception.SimpleError (_("Less than 1 search result"))
+ return
+ else:
+ link=abstract_links[0]
+
+
+ url=urlparse.urljoin(get_url , link['href'])
+
+ file = ClientCookie.urlopen(url)
+ get_url=file.url
+ soup=BeautifulSoup(file.read())
+ file.close()
+
+
+ #find link with id in the href that will be the pdf
+ pdflink=soup.first('a',{'href':re.compile('id=')})
+
+ url=urlparse.urljoin(get_url , pdflink['href'])
+
+ #get and save the pdf file
+ file = ClientCookie.urlopen(url)
+ content = file.read()
+ file.close
+
+
+ file=open(filename,'w')
+ file.write(content)
+ file.close()
+
+ return
+
+def get_pdf_from_ieee (title,filename,journals_url):
+ '''get pdf file from ieeexplore using the title of the entry'''
+
+ ieee_get_url = journals_url['IEEE Journals']
+ ieee_search_url = ieee_get_url + '/search/searchresult.jsp'
+ # the ieee page cannot search for page, we therefore search with the
+ # title
+ post={'queryText':title + '<in>ti'}
+
+ url=ieee_search_url + '?' + urllib.urlencode(post)
+
+ file=ClientCookie.urlopen(url) #need cookies
+ get_url=file.url
+ soup=BeautifulSoup(file.read())
+ file.close()
+
+ #find links with text PDF, that should link to the pdf-file
+ links=soup.fetch('a')
+
+ pdflinks=[]
+ for link in links:
+ if link.string == 'PDF':
+ pdflinks.append(link['href'])
+
+ #result is valid if we found only one pdf
+ if len(pdflinks) > 1:
+ raise Exception.SimpleError (_("More than 1 search result"))
+ return
+ elif len(pdflinks) <1:
+ raise Exception.SimpleError (_("Less than 1 search result"))
+ return
+ else:
+ pdfpath=pdflinks[0]
+
+ #get and save the pdf file
+ url=urlparse.urljoin(get_url , pdfpath)
+ file=ClientCookie.urlopen(url)
+ contents=file.read()
+ file.close()
+ fp=open(filename,'w')
+ fp.write(contents)
-query_url = 'http://www.ncbi.nlm.nih.gov/entrez/utils/pmqty.fcgi'
-fetch_url = 'http://www.ncbi.nlm.nih.gov/entrez/utils/pmfetch.fcgi'
+ return
def query_info (searchterm, field, displaynum, displaystart, edate):
@@ -53,7 +426,7 @@
'dispstart' : displaystart - 1
})
- f = urllib.urlopen ("%s?%s" % (query_url, params))
+ f = urllib.urlopen ("%s?%s" % (m_query_url, params))
uids = []
in_body = 0
uid_re = re.compile (r'^([\d]+)<br>')
@@ -267,7 +640,7 @@
'mode' : 'text'
})
- url = "%s?%s&id=%s" % (fetch_url, params, str(uids))
+ url = "%s?%s&id=%s" % (m_fetch_url, params, str(uids))
file, data = urllib.urlretrieve (url)
diff -Naur pybliographer-1.2.6.2/setup-check.py pybliographer-new/setup-check.py
--- pybliographer-1.2.6.2/setup-check.py 2005-02-22 06:50:43.000000000 +1300
+++ pybliographer-new/setup-check.py 2005-10-08 20:36:21.837043208 +1300
@@ -51,6 +51,8 @@
import gtk.glade
import gnome.ui
import gconf
+ import BeautifulSoup
+ import ClientCookie
v = string.join (map (str, gtk.pygtk_version), '.')