Re: netcamlbox: checking the mail box without waiting

Gerd Stolpmann wrote:
> On Di, 2010-03-23 at 19:08 +0000, Hugo Ferreira wrote:
>> Hello,
>>
>> Gerd Stolpmann wrote:
>>> On Di, 2010-03-23 at 15:50 +0000, Hugo Ferreira wrote:
>> snip..
>>>> Fatal error: exception Unix.Unix_error(20, "shm_open", "/camlbox_9838_0")
>>>> Raised by primitive operation at unknown location
>>>> Called from file "netcamlbox.ml", line 322, characters 4-21
>>> 20 is ENOENT, so I guess you unlinked the shm object before.
>>>
>> Correct. I unlink the mbox as per your example.
>> No need to keep it around.
>>
>>>> For testing purposes I used:
>> snip...
>>>> My question is: how do I go about avoiding the block
>>>> on a mailing box?
>>> Well, generally you need some trigger from the worker, and a counter.
>>> The trigger should be a message the worker sends out when it freed a
>>> slot. The counter could be implemented inside the coordinator, or as a
>>> shared counter (i.e. as semaphore). That depends on a lot of details, so
>>> without knowing the code I cannot give an advice.
>>>
>> Interesting. I was looking at it the other way around.
>> In other words workers wait on their mboxes for work.
>> In fact I had envisioned a common mbox for all task dispatch.
> 
> Well, in a message passing world messages are the only triggers that can
> wake up components. So, often you end up having both ends of the
> communication channel waiting on each other (not at the same time, of
> course, but with changing roles).
> 

Very true. But my problem is simply that:
1. I am iterating through 'n' worker processes dishing out work
2. I want to keep all process busy but don't know which is ready
3. If I come to a process that has not completed yet I block on a send
4. But additional processes may be free to process a task at hand
5. Hence I loose a chance to "schedule" the task
6. And therefore don't maximize CPU utilization

Note that increasing the number of mbox slots may alleviate the problem
but it will not solve it.The task distribution would still be
unbalanced.

So I first check which processes can take a task and send that to it
knowing it will not block. Seeing as this info is already in the message
boxes I tried to use that 8-).

Other alternatives include:
a) A single task queue
b) Work stealing

Both solutions however add needless complexity 8-(.

>> In the meantime I have looked at the mbox source code.
>> I realized that with a small change to camlbox_messages
>> my problem is solved. I have altered it to:
>>
>> let camlbox_messages box =
>>    Netsys_posix.sem_getvalue box.sem.s_free_slots
>>
>> Upon reflection I have come to realize that:
>>
>> a) I am in fact using a global counter to push work
>>     onto idle workers (basically does as you suggested -
>>     use a counting semaphore).
>>
>> b) I think that "camlbox_messages" should use a box because
>>     it is a:
>>     i) a value of a specific mbox and
>>     ii) allows faster access to the same information.
>>
>> Might I suggest the above alteration to the API?
> 
> Ok, I've added this as a second function. (See svn repository at
> https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/)
> 

I see "camlbox_bmessages box". Great.

Thank you very much.

Regards,
Hugo F.


> Gerd
> 
>> Thanks for the suggestions.
>>
>> Regards,
>> Hugo F.
>>
>>
>>> Gerd
>>>
>>>> TIA,
>>>> Hugo F.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download Intel® Parallel Studio Eval
>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>> proactively, and fine-tune applications for parallel performance.
>>>> See why Intel Parallel Studio got high marks during beta.
>>>> http://p.sf.net/sfu/intel-sw-dev
>>>> _______________________________________________
>>>> Ocamlnet-devel mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel
>>>>
>>>
>>
> 
> 


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev