Re: Lightweight IPC for scripting

Brian Mastenbrook writes:
> I didn't see anything about "button" in his post. Care to explain where you
> started to connect this to the presentation (view) layer instead of the
> model layer?
> 
> If this statement made no sense to you then you are probably unqualified to
> comment on anything related to GUI programming.

Well,  I'm no  specialist  of X,  but  one thing  I  remember from  my
browsing  of its documention  is that  in X  windows, everything  is a
window. Any widget, button, text  field, whatever, can be designed and
drawn  as a X  window, identified  as such.   So you  can put  in your
.Xresources statements to customize colors, sizes and fonts widget per
widget, and you can dynamically identify it and send it mssages. Also,
you could have  an external application open its  window as an embeded
widget in your own window (isn't that how the window managers work?).

I wouls underlining the fact  that non X toolboxes, like GNUstep don't
take advantage  of this  and just  display pixels in  one X  window. I
don't know about  KDE or GNOME, it's possible  they're doing X widget,
but perhaps not.

> > So  to be  general, it  would be  better to  develo an  AI  that would
> > analyse the screen bitmap,  recognize the buttons, and other controls,
> > the data fields,  OCR and "see" them (hopefully we  can get 100% here,
> > since there's  not much noise,  only some anti-aliasing...),  and then
> > you could try to drive the mouse and click and drag around.
> 
> I think I can phrase a bit more specifically what you're asking for, given
> that I've worked for a little while in this weird "AI" field. Let's assume
> that the user of the application uses some decision procedure to interact
> with the program, and that this decision procedure is given inputs, called
> "goals", and uses some existing data, called "knowledge", to figure out how
> to translate these inputs into actions. Now further suppose that programs
> are designed to facilitate this translation procedure by presenting
> information in a way that coherently adds "knowledge" which is designed to
> be processed by this procedure. Thus we only need to discern the operation
> of this function as it is implemented by humans, and we can generally
> perform such translation from the presentation of knowledge to the effects,
> and work in the reverse direction.
> 
> Congratulations! That's brilliant! I'll let you know then when I can perform
> reliable, correct goal->action translation, and then you can modify it to
> perform knowledge->effect inference.

On the  other hand, scripting is  here to specify  more precisely what
needs to be done.  If the IA  can't build an action list to execute an
order such as "take a rendez-vous with my dentist", you could at least
write:

    (let* ((mydentist (get-address-book-entry 
                       (address-book *myself*)
                       :function "dentist"))
           (email     (send-email
                       :to (email-address mydentist)
                       :subject "Rendez-vous"
                       :text "Could you please receive me on ~A"
                          (find-a-common-free-slot 
                             (fetch-calendar mydentist)
                             (fetch-calendar *myself*)))))
      (tell-me (wait-for-response-to-email (email message-id))))

And  the IA/2  could  know how  to  drive your  email  program to  put
(email-address  mydentist)  into the  "To:"  field  of the  send-email
window, to put "Rendez-Vous" into the "Subject:" field, etc.

In any case, that's almost  done with some web scripting applications.
Also, on  MacOS, it's been  tried too, but using  absolute coordinates
instead of relative identification of the fields.

The only  difference is  that instead of  identifying the  fields with
hidden  data  (the  tagging  of  hypertext pages),  the  system  would
identify them  with geographic information  and labels of  fixed texts
around the data entry widgets.


> On second thought, let's not.
> 
> > Perhaps ambitious, but one can drean!
> 
> I'm sorry, but my simply AI program written in Lisp (which I used just
> because of Norvig's book) has no knowledge of what "drean" means. Perhaps I
> should start using Python like he is now, and then it will be able to learn
> these words.

I would have  sworn that I typed "develop",  "dream", etc.  But that's
what you get when you write emails just before going to sleep...

-- 
__Pascal_Bourguignon__                   http://www.informatimago.com/
----------------------------------------------------------------------
Do not adjust your mind, there is a fault in reality.