Re: futures and parallel forms in Guile and Bigloo

Hello Joseph.

Thanks for your explanations.
I now have a small helper function par-map (parallel map) that does not impose
any thread limit,
while para-map/cores limits it to the number of physical cores (overwritable).
(If you are interested, I can send you the code for reading or inclusion.)

I hope that Manuel will accept your PR needed for bigloo-concurrent, soon.

Ciao
Sven

Your wrote, 2020-09-27 00:57:
> Hello, Sven,
>
> I am glad that it is working well for you. Most of the credit lies with Takashi Kato and Manuel Serrano for their work on the scheme-concurrent and pthread libraries, respectively.
>
> As for how the number of available threads is determined if the code contains only calls of future and future-get from bigloo-concurrent, like scheme-concurrent, a separate thread is launched for every future in this case. It would be better to have a default threadpool in a lot of cases, and this is something we
> can experiment with. I also think it would be useful to add features similar to those provided by CompletableFuture in Java. It supports a number of useful methods for composing futures.
>
> Best Regards,
> Joe
>
> On Friday, September 25, 2020, 7:54:18 AM PDT, Sven Hartrumpf <[email protected]> wrote:
>
> Hi again.
>
> Some more feedback.
>
> SH, 2020-09-21 19:35:
>> Hello Joseph.
>>
>> You wrote, 2020-09-20 18:43:
>>> Hello, Sven,
>>>
>>> I got a chance to look at the jvm issues this weekend and have made an update that results in a mostly working implementation; there still remains a race condition when shutting down an executor that results in futures being in a finished state instead of the expected terminated state at times. I will continue
> to
>>> look into this but don't see it as super critical. As before, changes were required to the bigloo pthread library, I refactored the bglpmutex class to use ReentrantLock. This gives the jvm version of the pthread library proper recursive mutexes. The changes can be found at
>>> https://github.com/donaldsonjw/bigloo-1/tree/pthread_modifications.
>>>
>>> If you get a chance to try it out, let me know what you think.
>>
>> I am only using the C backend; so I cannot talk about your improvements for the Java
>> backend.
>> For the C backend, I have tried some smaller examples.
>> Your library seems to be solid and correct!
>> The performance is a little bit disappointing so far,
>> but maybe this
>> will change when I look at some more advanced/real-life code segments.
>
> It changed and improved :-)
> I have some other examples that show impressive performance.
> On a CPU with 16 physical cores (32 virtual cores, AMD Ryzen 9 3950X),
> the parallel version with 128 similar compute tasks is 20 times faster than
> the serial version.
> With 16 similar compute tasks, the speed-up is 15, which is almost the
> theoretical optimum of 16.
> So, the overhead of the library seems to be small.
>
>> How does your library choose the number of available threads (if the code
>> contains only calls of future and future-get from bigloo-concurrent)?
>> Number of physical cores, numer of virtual cores, ...?
>
> Greetings
> Sven
>
>>> Best Regards,
>>> Joseph Donaldson
>>>
>>> .
>>>
>>> On Sunday, September 6, 2020, 12:25:14 PM PDT, Joseph Donaldson <[email protected]> wrote:
>>>
>>> Hello, Sven,
>>>
>>> I took a look at porting scheme-concurrent to Bigloo. The results can be found at https://github.com/donaldsonjw/bigloo-concurrent. The native back-end passes all the tests but there are still some issues with the jvm back-end; I will continue to work on it. While doing the port, I made a few changes to Bigloo.
>>> These are linked from the bigloo-concurrent README. I have made a pull request to have the changes included in the Bigloo mainline.
>>>
>>> Let me know what you think.
>>>
>>> Best Regards,
>>> Joseph Donaldson
>>>
>>>  On Thu, Aug 20, 2020 at 8:34 AM, Sven Hartrumpf
>>>  <[email protected]> wrote:
>>>  Hello Joseph.
>>>
>>>  Thanks for your message and the link to hop code.
>>>  I must admit that this looks a little bit scary to me.
>>>  Maybe it is easier to port https://github.com/ktakashi/scheme-concurrent
>>>  (which already runs on several Schemes) to Bigloo?
>>>  It was presented at the Scheme Workshop 2016:
>>>  http://schemeworkshop.org/2016/ (it's the second talk)
>>>
>>>  Greetings
>>>  Sven
>>>
>>>  Joseph wrote, 2020-08-06 17:30:
>>>  > Hello, Sven,
>>>  >
>>>  > Given Bigloo's support for threads (pthreads or java threads), efficiently implementing Guile's futures and parallel forms is definitely possible. As mentioned in the Guile documentation, the most obvious implementation would be to introduce a threadpool abstraction and
>>>  > build the futures and parallel-forms on top of that. Manuel created a similar abstraction for his hop executors. See https://github.com/manuel-serrano/hop/blob/master/src/queue_scheduler.scm. Taking a look at Java's ThreadPoolExecutor abstraction may also be helpful.
>>>  >
>>>  > I think this would be an interesting project and would be willing to contribute to it.
>>>  >
>>>  > Best Regards,
>>>  > Joseph Donaldson.
>>>  >
>>>  > On Wednesday, August 5, 2020, 3:04:46 AM PDT, Sven Hartrumpf <[email protected]> wrote:
>>>  >
>>>  > Dear Bigloo users.
>>>  >
>>>  > I read about futures in Guile (see https://www.gnu.org/software/guile/manual/html_node/Futures.html)
>>>  > and the parallel forms built on-top, e.g. parallel, par-map and par-for-each
>>>  > (see https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html).
>>>  >
>>>  > They look very promising,
>>>  > especially because 8, 16 or more CPU cores are available on more and more
>>>  > standard CPUs.
>>>  >
>>>  > Can these constructs be efficiently implemented in Bigloo, too?
>>>  >
>>>  > Greetings
>>>  > Sven