Re: importing bank statements

I have been working on a solution off and on (more off than on) for a 
few years now. There are a few different approaches to take.

1. Automate your browser
2. Write a cURL script
3. Write a program in your favorite computer language

Writing a program in your favorite computer language is a boatload of 
work.

Writing a cURL script is often the recommended approach for automating 
downloading of files from any web site. The usual advice is to use the 
LiveHTTPHeaders plugin for Firefox to capture the HTTP requests sent 
from Firefox to your bank and then rebuild each of those requests in 
cURL. This would work for most web sites, but banks tend to have lots 
of security on their web sites. Some of those security measures are 
designed to prevent automated scripts from accessing the bank's web 
site. For example, I found that using cURL to login to one bank's web 
site would fail unless I had also used curl to download one tiny little 
.gif file. Yes, the bank's server would fail any login attempt if that 
one .gif file had not been downloaded. That can be overcome but then 
you'll quickly run in to other problems. One such problem is that banks 
often have the browser execute some JavaScript to calculate a value 
that then gets sent back to the bank's server. cURL doesn't have a 
JavaScript engine so you'd need to find some other tool that can 
execute the JavaScript for you.

Automating a browser is the best approach I've found so far. At first I 
tried using the iMacros plugin for Firefox, but found it very difficult 
to handle branching on certain conditions. Eventually I gave up on 
iMacros. After I learned to write JavaScript I realized that 
JavaScript's very purpose is browser automation. All I needed was a 
browser that would let me "inject" some JavaScript into the downloaded 
copy of the bank's web page. Unless you are using a browser debugger 
most browsers specifically do not allow this for security reasons. 
Using a debugger I was able to write a complete script to login to one 
specific bank, parse each page, navigate to the downloads page and 
download the transactions as an .ofx file.  I became convinced that 
JavaScript is absolutely the correct approach.

So I had decided to write my own special browser app that would allow 
me to inject JavaScript. Earlier I claimed that writing a special 
program would be a lot of work, and it would have been. Fortunately 
before I invested too much time in writing my own program I found a 
commercial product named "Fake."  Fake is a program for OS X. Fake is a 
"fake" browser. Fake lets you create a workflow for automating 
navigation of a web site. One of the things Fake allows you to do is 
inject JavaScript into a bank's web page. Using Fake I was able to 
fully automate downloading transactions from one specific bank.

What I need to do now is break the JavaScript code into modules so that 
code specific to one bank is separate from code that is generic to all 
banks. I'd be happy to collaborate with anyone who wants to help build 
up a library of code that can automate downloading transactions and 
statements from any bank's web site. Let me know if you are interested.

-SteveL








On Tue, 17 May 2011 20:37:30 -0600, Matt&Katrina Funk wrote:
> Mmmh,
> looks like there is no such solution out there then. Maybe i will give
> it a shot and see what comes about.
> 
> thanks
> matt
> 
> On 5/17/2011 10:30 AM, Steve Lessard wrote:
>> Unfortunately the PocketSense scripts won't help Matt because those 
>> scripts work by connecting to a bank's OFX server which is something 
>> Moneydance already does. The problem Matt (and I) would like to find a 
>> solution for is how to automate downloading transactions from a bank's 
>> web site and not its OFX server.
>> 
>> 
>> 
>> 
>> 
>> On Mon, 16 May 2011 12:10:40 -0400, Kevin Nugent wrote:
>>> Hi Mat,  
>>> There's a fella named Robert who has done some nice work
>>> with Python on a program to download OFX files from financial 
>>> institutions.  
>>> The program is set up to work with MS Money.  
>>> It downloads OFX files to a folder where you could easily import them 
>>> into Moneydance.  
>>> http://sites.google.com/site/pocketsense/home  
>>> HTH -Kevin N. 
>>>  
>>>> Date: Sat, 14 May 2011 17:07:05 -0600
>>>> From: [email protected]
>>>> To: [email protected]; [email protected]
>>>> Subject: [moneydance] importing bank statements
>>>> 
>>>> Hi,
>>>> i had a question with regards to importing bank/credit card stuff. My
>>>> bank (a local bank) does not support any sort of OFX. Other bank charge
>>>> fees that are 'unacceptable' to some people. However, my bank and the
>>>> 'fee' charging banks provide online access and the option to manually
>>>> download the statements in quicken format or somesuch.
>>>> 
>>>> Anyway, so as i would like to be able to have things automated as much
>>>> as possible, i was thinking about using python's mechanize module to
>>>> possible download the statements automatically and have them imported
>>>> automatically.
>>>> 
>>>> As i have never really worked with moneydance i was just wondering if
>>>> this sounds like a reasonable thing to do (and whether it is possible)?
>>>> 
>>>> Any input would be appreciated.
>>>> 
>>>> thanks
>>>> matt
>>>> _______________________________________________
>>>> moneydance-info mailing list
>>>> [email protected]
>>>> http://moneydance.com/mailman/listinfo/moneydance-info
>>>  		 	   		  
>>> _______________________________________________
>>> moneydance-info mailing list
>>> [email protected]
>>> http://moneydance.com/mailman/listinfo/moneydance-info
> 
> _______________________________________________
> moneydance-info mailing list
> [email protected]
> http://moneydance.com/mailman/listinfo/moneydance-info