Re: Seeking advice for a couple of issues
[email protected] Tue, 02 Apr 2013 14:47:36 -0400
| Newsgroups | gmane.comp.lang.ocaml.lib.net.devel |
|---|---|
| Message-ID | <[email protected]> |
The fix for chunked request processing seems to work well. Quite a drastic improvement in processing time. My latest tests are showing processing time of slightly greater than 2 seconds to process 10,000 rows/documents from a CouchDb get operation. Just a thought, is it necessary for each chunk to be processed by the event system? If all the chunks were treated as one event (decoupled), I'd bet that the processing time would drop significantly. I'm having some trouble using the skip_challenge option on the Basic Auth handler. The following snippet is my latest attempt: let () = let keys = new Http_client.key_ring() in let key = Http_client.key ~user:"admin" ~password:"admin" ~realm:"" ~domain:["http://localhost:5984"] in keys#add_key key; let auth = new Http_client.basic_auth_handler ~enable_auth_in_advance:true ~skip_challenge:(Some "*") keys in let pipeline = new Http_client.pipeline in pipeline # add_auth_handler auth; let call = new Http_client.get "http://localhost:5984/camldb" in let hdr = call#request_header `Base in hdr#update_field "Content-type" "application/json"; pipeline#add call; pipeline#run(); printf "rc=%d\n" (call#response_status_code); let hdr = call#request_header `Effective in List.iter (fun (k, v) -> printf "%s: %s\n" k v) hdr#fields; This yields a return code 401 with no Authorization header added to the sent GET request. No luck with PUT or DELETE requests either. Thanks again for all your help. I've enjoyed working with your library. Regards, John Quoting "Gerd Stolpmann" <[email protected]>: > Am 29.03.2013 01:11:39 schrieb(en) John Derrick: >> Thanks for the reply, Gerd. >> >> First, I should mention I'm using Ocaml 4.001, the Mingw build on >> Win7. I'm >> using Ocamlnet version 3.6.1 (or newer). >> >> I tried using ~enable_auth_in_advance as you suggested but was unable >> to get >> it working. I must be doing something wrong. >> Here is my latest attempt: > > Confirmed, this does not work. I had this wrong in my mind - > auth_in_advance does not work when the server never generates a > challenge. > > I've added a new mode skip_challenge, e.g. > > let auth = new Http_client.basic_auth_handler ~skip_challenge:(Some > "*") keys > > This is set to "*" or to the domain URL (e.g. "http://localhost/") to > which it applies. > >> method private do_run_pipe call = >> let keys = new Http_client.key_ring() in >> let auth = new Http_client.basic_auth_handler >> ~enable_auth_in_advance:true >> keys in >> let key = Http_client.key ~user:"admin" ~password:"admin" ~realm:"" >> ~domain:["localhost"] in > > This is also a problem in the current code: you would have to set > domain to "http://localhost:80/". Not really well documented. The new > version also allows now "*", and you can omit the port number. > > That the domain is a full URL has to do with the definitions in the > HTTP protocol. > >> keys#add_key key; >> let pipeline = new Http_client.pipeline in >> pipeline # add_auth_handler auth; >> let hdr = call#request_header `Base in >> hdr#update_field "Content-type" "application/json"; >> pipeline#add call; >> pipeline#run(); >> call#response_status_code >> >> Packet capture shows that no Authorization header was added to a PUT >> request >> with the above block of code. Return code is 401. >> >> On the second issue, I have TCP_NODELAY set to true in the CouchDb >> config. >> The process is using heavy CPU while it runs. Do you think the issue >> could >> be related to OS and/or OCaml build? > > Turns out this is in deed a problem with event handling, namely that > there was some progress counting, and this resulted in a performance > bug when the chain of event handling gets long. > > The new version is not yet released, but you can grab it from the svn > server: https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/ > > Gerd > > >> Thanks again for your help, >> John >> >> -----Original Message----- >> From: Gerd Stolpmann >> Sent: Thursday, March 28, 2013 9:06 AM >> To: John Derrick >> Cc: [email protected] >> Subject: AW: [Ocamlnet-devel] Seeking advice for a couple of issues >> >> Am 27.03.2013 23:41:36 schrieb(en) John Derrick: >> > Hi All, >> > >> > I've been working with Http_client to develop a no-frills CouchDb >> > driver. I'm relatively new to OCaml, but have many years of >> experience >> > with various imperative langs. >> > >> > I'm looking for help with a couple of issues. My first issue is >> Basic >> > Authentication. I've noticed that Basic Auth and Basic Auth Session >> > do not include the "Authorization:" header when making GET or PUT >> > requests UNLESS a challenge is received. Is this the expected >> > behavior? Not a serious issue as it's easy to add my own >> > Authorization entry to the header. Apparently CouchDb does not >> send an >> > auth challenge. >> >> It is in deed possible to enable this behavior (it's off by default): >> >> - Create the authentication handler like this: >> >> let keys = new Http_client.key_ring() >> let auth = new Http_client.basic_auth_handler >> ~enable_auth_in_advance:true keys >> >> The enable_auth_in_advance option makes it working. >> >> - Add credentials: >> >> let key = Http_client.key ... >> keys # add_key key >> >> - Add this to the pipeline: >> >> pipeline # add_auth_handler auth >> >> Well, this behavior has always been a matter of discussion. Normally, >> you don't want to send credentials before you know that the server is >> really the server (remember that you can run http over ssl, and the >> ssl >> connection authenticates the server), and this way it is specified in >> RFC 2617. Historically, though, browsers have always implemented some >> form of sending the credentials without prior challenge, namely when >> the web page was visited the second time. >> >> > The second issue is chunked request performance. When peforming a >> > query against CouchDb (say _all_docs for example) via a GET request, >> > Ocamlnet seems to have issues processing the returned data. CouchDb >> > returns many small blocks of data in chunked format. Processing a >> > collection of 1000 docs can take several minutes. I'm not sure if >> > this relates to the event system performance or socket performance. >> > Does any one know of a solution to this issue? >> >> I don't think it has to do with the event processing, as the OS >> buffers >> data up if the client is not fast enough, so for the next read() the >> client gets many chunks at once. Also, 1000 events (if they are really >> generated) are not really much, and should not take minutes, but only >> milliseconds. >> >> Does the client consume a lot of CPU? Or is it just waiting? >> >> In theory there could be a bad interaction with the TCP buffering, >> especially when CouchDB disables the Nagle algorithm (TCP_NODELAY), >> which it probably does. But you report a really bad behavior, so I've >> actually no clue here. >> >> Gerd >> >> >> > >> > Thanks to all who contributed to the development of Ocamlnet. It's >> an >> > impressive collection of work. >> > >> > Regards, >> > John >> > >> > >> > >> > ------------------------------------------------- >> > >> > >> > >> > >> > >> > VFEmail.net - http://www.vfemail.net >> > >> > >> > $14.95 ONETIME Lifetime accounts with Privacy Features! >> > >> > >> > 15GB disk! No bandwidth quotas! >> > >> > >> > Commercial and Bulk Mail Options! >> > >> > >> > >> >> ------Zitierte Anlage------ >> > >> ------------------------------------------------------------------------------ >> > Own the Future-Intel® Level Up Game Demo Contest 2013 >> > Rise to greatness in Intel's independent game demo contest. >> > Compete for recognition, cash, and the chance to get your game >> > on Steam. $5K grand prize plus 10 genre and skill prizes. >> > Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d >> >> ------Zitierte Anlage------ >> > _______________________________________________ >> > Ocamlnet-devel mailing list >> > [email protected] >> > https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel >> > >> >> >> >> -- >> ------------------------------------------------------------ >> Gerd Stolpmann, Darmstadt, Germany [email protected] >> Creator of GODI and camlcity.org. >> Contact details: http://www.camlcity.org/contact.html >> Company homepage: http://www.gerd-stolpmann.de >> ------------------------------------------------------------ >> >> >> ------------------------------------------------- >> >> VFEmail.net - http://www.vfemail.net >> $14.95 ONETIME Lifetime accounts with Privacy Features! >> 15GB disk! No bandwidth quotas! >> Commercial and Bulk Mail Options! >> >> ------------------------------------------------------------------------------ >> Own the Future-Intel(R) Level Up Game Demo Contest 2013 >> Rise to greatness in Intel's independent game demo contest. Compete >> for recognition, cash, and the chance to get your game on Steam. >> $5K grand prize plus 10 genre and skill prizes. Submit your demo >> by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2 >> _______________________________________________ >> Ocamlnet-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel >> >> > > > > -- > ------------------------------------------------------------ > Gerd Stolpmann, Darmstadt, Germany [email protected] > Creator of GODI and camlcity.org. > Contact details: http://www.camlcity.org/contact.html > Company homepage: http://www.gerd-stolpmann.de > ------------------------------------------------------------ > ------------------------------------------------- VFEmail.net - http://www.vfemail.net $14.95 ONETIME Lifetime accounts with Privacy Features! 15GB disk! No bandwidth quotas! Commercial and Bulk Mail Options! ------------------------------------------------------------------------------ Minimize network downtime and maximize team effectiveness. Reduce network management and security costs.Learn how to hire the most talented Cisco Certified professionals. Visit the Employer Resources Portal http://www.cisco.com/web/learning/employer_resources/index.html