Re: Seeking advice for a couple of issues
Gerd Stolpmann <[email protected]> Mon, 13 May 2013 16:36:24 +0200
| Newsgroups | gmane.comp.lang.ocaml.lib.net.devel |
|---|---|
| Message-ID | <1368455784.5134.2@samsung> |
Am 02.04.2013 20:47:36 schrieb(en) [email protected]: > The fix for chunked request processing seems to work well. Quite a > drastic improvement in processing time. My latest tests are showing > processing time of slightly greater than 2 seconds to process 10,000 > rows/documents from a CouchDb get operation. Just a thought, is it > necessary for each chunk to be processed by the event system? If all > the chunks were treated as one event (decoupled), I'd bet that the > processing time would drop significantly. > > I'm having some trouble using the skip_challenge option on the Basic > Auth handler. > > The following snippet is my latest attempt: Hi, Sorry for the late response. You can fix it with: > let () = > let keys = new Http_client.key_ring() in > let key = Http_client.key ~user:"admin" ~password:"admin" ~realm:"" > ~domain:["http://localhost:5984"] in ~realm:"anywhere" ~domain:["http://localhost:5984/"] I did not mention the "anywhere" change in my latest email, but it is documented in the mli. The domain string must include a final slash. Gerd > keys#add_key key; > let auth = new Http_client.basic_auth_handler > ~enable_auth_in_advance:true > ~skip_challenge:(Some "*") keys in > let pipeline = new Http_client.pipeline in > pipeline # add_auth_handler auth; > let call = new Http_client.get "http://localhost:5984/camldb" in > let hdr = call#request_header `Base in > hdr#update_field "Content-type" "application/json"; > pipeline#add call; > pipeline#run(); > printf "rc=%d\n" (call#response_status_code); > let hdr = call#request_header `Effective in > List.iter (fun (k, v) -> printf "%s: %s\n" k v) hdr#fields; > > This yields a return code 401 with no Authorization header added to > the sent GET request. No luck with PUT or DELETE requests either. > > Thanks again for all your help. I've enjoyed working with your > library. > > Regards, > John > > Quoting "Gerd Stolpmann" <[email protected]>: > > > Am 29.03.2013 01:11:39 schrieb(en) John Derrick: > >> Thanks for the reply, Gerd. > >> > >> First, I should mention I'm using Ocaml 4.001, the Mingw build on > >> Win7. I'm > >> using Ocamlnet version 3.6.1 (or newer). > >> > >> I tried using ~enable_auth_in_advance as you suggested but was > unable > >> to get > >> it working. I must be doing something wrong. > >> Here is my latest attempt: > > > > Confirmed, this does not work. I had this wrong in my mind - > > auth_in_advance does not work when the server never generates a > > challenge. > > > > I've added a new mode skip_challenge, e.g. > > > > let auth = new Http_client.basic_auth_handler ~skip_challenge:(Some > > "*") keys > > > > This is set to "*" or to the domain URL (e.g. "http://localhost/") > to > > which it applies. > > > >> method private do_run_pipe call = > >> let keys = new Http_client.key_ring() in > >> let auth = new Http_client.basic_auth_handler > >> ~enable_auth_in_advance:true > >> keys in > >> let key = Http_client.key ~user:"admin" ~password:"admin" > ~realm:"" > >> ~domain:["localhost"] in > > > > This is also a problem in the current code: you would have to set > > domain to "http://localhost:80/". Not really well documented. The > new > > version also allows now "*", and you can omit the port number. > > > > That the domain is a full URL has to do with the definitions in the > > HTTP protocol. > > > >> keys#add_key key; > >> let pipeline = new Http_client.pipeline in > >> pipeline # add_auth_handler auth; > >> let hdr = call#request_header `Base in > >> hdr#update_field "Content-type" "application/json"; > >> pipeline#add call; > >> pipeline#run(); > >> call#response_status_code > >> > >> Packet capture shows that no Authorization header was added to a > PUT > >> request > >> with the above block of code. Return code is 401. > >> > >> On the second issue, I have TCP_NODELAY set to true in the CouchDb > >> config. > >> The process is using heavy CPU while it runs. Do you think the > issue > >> could > >> be related to OS and/or OCaml build? > > > > Turns out this is in deed a problem with event handling, namely that > > there was some progress counting, and this resulted in a performance > > bug when the chain of event handling gets long. > > > > The new version is not yet released, but you can grab it from the > svn > > server: https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/ > > > > Gerd > > > > > >> Thanks again for your help, > >> John > >> > >> -----Original Message----- > >> From: Gerd Stolpmann > >> Sent: Thursday, March 28, 2013 9:06 AM > >> To: John Derrick > >> Cc: [email protected] > >> Subject: AW: [Ocamlnet-devel] Seeking advice for a couple of issues > >> > >> Am 27.03.2013 23:41:36 schrieb(en) John Derrick: > >> > Hi All, > >> > > >> > I've been working with Http_client to develop a no-frills CouchDb > >> > driver. I'm relatively new to OCaml, but have many years of > >> experience > >> > with various imperative langs. > >> > > >> > I'm looking for help with a couple of issues. My first issue is > >> Basic > >> > Authentication. I've noticed that Basic Auth and Basic Auth > Session > >> > do not include the "Authorization:" header when making GET or PUT > >> > requests UNLESS a challenge is received. Is this the expected > >> > behavior? Not a serious issue as it's easy to add my own > >> > Authorization entry to the header. Apparently CouchDb does not > >> send an > >> > auth challenge. > >> > >> It is in deed possible to enable this behavior (it's off by > default): > >> > >> - Create the authentication handler like this: > >> > >> let keys = new Http_client.key_ring() > >> let auth = new Http_client.basic_auth_handler > >> ~enable_auth_in_advance:true keys > >> > >> The enable_auth_in_advance option makes it working. > >> > >> - Add credentials: > >> > >> let key = Http_client.key ... > >> keys # add_key key > >> > >> - Add this to the pipeline: > >> > >> pipeline # add_auth_handler auth > >> > >> Well, this behavior has always been a matter of discussion. > Normally, > >> you don't want to send credentials before you know that the server > is > >> really the server (remember that you can run http over ssl, and the > >> ssl > >> connection authenticates the server), and this way it is specified > in > >> RFC 2617. Historically, though, browsers have always implemented > some > >> form of sending the credentials without prior challenge, namely > when > >> the web page was visited the second time. > >> > >> > The second issue is chunked request performance. When peforming a > >> > query against CouchDb (say _all_docs for example) via a GET > request, > >> > Ocamlnet seems to have issues processing the returned data. > CouchDb > >> > returns many small blocks of data in chunked format. Processing a > >> > collection of 1000 docs can take several minutes. I'm not sure if > >> > this relates to the event system performance or socket > performance. > >> > Does any one know of a solution to this issue? > >> > >> I don't think it has to do with the event processing, as the OS > >> buffers > >> data up if the client is not fast enough, so for the next read() > the > >> client gets many chunks at once. Also, 1000 events (if they are > really > >> generated) are not really much, and should not take minutes, but > only > >> milliseconds. > >> > >> Does the client consume a lot of CPU? Or is it just waiting? > >> > >> In theory there could be a bad interaction with the TCP buffering, > >> especially when CouchDB disables the Nagle algorithm (TCP_NODELAY), > >> which it probably does. But you report a really bad behavior, so > I've > >> actually no clue here. > >> > >> Gerd > >> > >> > >> > > >> > Thanks to all who contributed to the development of Ocamlnet. > It's > >> an > >> > impressive collection of work. > >> > > >> > Regards, > >> > John > >> > > >> > > >> > > >> > ------------------------------------------------- > >> > > >> > > >> > > >> > > >> > > >> > VFEmail.net - http://www.vfemail.net > >> > > >> > > >> > $14.95 ONETIME Lifetime accounts with Privacy Features! > >> > > >> > > >> > 15GB disk! No bandwidth quotas! > >> > > >> > > >> > Commercial and Bulk Mail Options! > >> > > >> > > >> > > >> > >> ------Zitierte Anlage------ > >> > > >> > ------------------------------------------------------------------------------ > >> > Own the Future-Intel® Level Up Game Demo Contest 2013 > >> > Rise to greatness in Intel's independent game demo contest. > >> > Compete for recognition, cash, and the chance to get your game > >> > on Steam. $5K grand prize plus 10 genre and skill prizes. > >> > Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d > >> > >> ------Zitierte Anlage------ > >> > _______________________________________________ > >> > Ocamlnet-devel mailing list > >> > [email protected] > >> > https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel > >> > > >> > >> > >> > >> -- > >> ------------------------------------------------------------ > >> Gerd Stolpmann, Darmstadt, Germany [email protected] > >> Creator of GODI and camlcity.org. > >> Contact details: http://www.camlcity.org/contact.html > >> Company homepage: http://www.gerd-stolpmann.de > >> ------------------------------------------------------------ > >> > >> > >> ------------------------------------------------- > >> > >> VFEmail.net - http://www.vfemail.net > >> $14.95 ONETIME Lifetime accounts with Privacy Features! > >> 15GB disk! No bandwidth quotas! > >> Commercial and Bulk Mail Options! > >> > >> > ------------------------------------------------------------------------------ > >> Own the Future-Intel(R) Level Up Game Demo Contest 2013 > >> Rise to greatness in Intel's independent game demo contest. Compete > >> for recognition, cash, and the chance to get your game on Steam. > >> $5K grand prize plus 10 genre and skill prizes. Submit your demo > >> by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2 > >> _______________________________________________ > >> Ocamlnet-devel mailing list > >> [email protected] > >> https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel > >> > >> > > > > > > > > -- > > ------------------------------------------------------------ > > Gerd Stolpmann, Darmstadt, Germany [email protected] > > Creator of GODI and camlcity.org. > > Contact details: http://www.camlcity.org/contact.html > > Company homepage: http://www.gerd-stolpmann.de > > ------------------------------------------------------------ > > > > > > ------------------------------------------------- > > VFEmail.net - http://www.vfemail.net > $14.95 ONETIME Lifetime accounts with Privacy Features! > 15GB disk! No bandwidth quotas! > Commercial and Bulk Mail Options! > > ------------------------------------------------------------------------------ > Minimize network downtime and maximize team effectiveness. > Reduce network management and security costs.Learn how to hire > the most talented Cisco Certified professionals. Visit the > Employer Resources Portal > http://www.cisco.com/web/learning/employer_resources/index.html > _______________________________________________ > Ocamlnet-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel > > -- ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany [email protected] Creator of GODI and camlcity.org. Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------ ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may