Re: [mh] Process_item_queue <more partly solved>

If you look at my raZberry code (around line 512), I use a multi-dimensional array in a hash to hold various attributes related to the process_item queue. Hopefully that might help a bit.

For my moduled, it seemed to me that keeping a queue array as part of the process_item object was a simpler approach. It kept the data and methods encapsulated.

> On Feb 24, 2021, at 9:32 AM, Giles Godart-Brown <[email protected]> wrote:
> 
> I've made a lot of progress this week, this version still uses a single queue, but it sorts it by process_name and time and then runs ones with different names in parallel, so it appears like there is a queue for each process.
> 
> It currently has a lot of debug prints and tests in it as I try to check all the edge cases (no guarantees that I've succeeded) .  I will remove these if you think it worth creating a pull to incorporate it into MisterHouse Master. If so I'll need some help.
> 
> I've still failed to be able to build a 3D hash array to include process with more than one command, so it only works for processes with a single command and throws an error if you try to queue a process with more than one.
> 
> Giles
> 
> On 20/02/2021 15:08, Giles Godart-Brown wrote:
>> I've now fixed the problem with changes to the process item before it was removed from the queue. 
>> 
>> There are 2 limitations with this solution;
>> 
>> 1) its a single FIFO, all process_items are executed in the order they arrive on the queue
>> 
>> 2) I struggled with array references and arrays of arrays, so there is a limit to a single command  process_item i.e. no add('program3 arguments', 'program4 arguments', ...). It simply runs the first command assigned to the process_item.
>> 
>> I think this can be improved by adding a time_added_to_queue column and sorting the queue on process_name and time_added_to_queue.
>> 
>> Then when it processes the queue instead of it just taking the top of the queue it takes the first entry for each process_name i.e. a sort of parallel queue.
>> 
>> If I get some time over the weekend I'll see if I can implement this.
>> 
>> I'm no expert on the innards of MisterHouse and how the modules work, so If you think it worthwhile I'd like some help turning this into a module for a future pull request.
>> 
>> Thanks in anticipation
>> 
>> Giles
>> 
>> 
>> 
>> On 19/02/2021 19:17, Giles Godart-Brown wrote:
>>> Attached is a first go at building a process_item_queue. 
>>> 
>>> Its a single queue, and I'm sure one queue per process_item would be better, but I've run out of talent :-) 
>>> 
>>> Giles 
>>> 
>>> On 18/02/2021 18:28, Giles Godart-Brown wrote: 
>>>> I was going to build it for myself, then if its good enough (?) and fulfils backwards compatibility I will add a pull 
>>>> 
>>>> G 
>>>> 
>>>> On 18/02/2021 17:25, Jeff Siddall via misterhouse-users wrote: 
>>>>> Giles, 
>>>>> 
>>>>> When you say "I'll build a queue" do you mean for your project or adding it onto process_items?  As long as it was backwards compatible with the existing behavior of a process_item, it would be nice to add that onto the process_item so there is a standard way for anyone to push items onto a process_item queue instead of everyone implementing it their own way. 
>>>>> 
>>>>> Jeff 
>>>>> 
>>>>> 
>>>>> On 2021-02-18 11:47 a.m., Giles Godart-Brown wrote: 
>>>>>> 
>>>>>> Thanks H 
>>>>>> 
>>>>>> I'll build a queue and update the docs over the weekend 
>>>>>> 
>>>>>> Giles 
>>>>>> 
>>>>>> On 18/02/2021 14:24, H Plato wrote: 
>>>>>>> As far as I know that’s how it works, the process_item has a single thread. You can check if the process_item is still active by calling$p_longproc->done(), and then queuing the additional commands to an array. That’s how I’ve dealt with this in some of my modules 
>>>>>>> 
>>>>>>>> On Feb 18, 2021, at 1:22 AM, Giles Godart-Brown <[email protected] <mailto:[email protected]> <mailto:[email protected]> <mailto:[email protected]>> wrote: 
>>>>>>>> 
>>>>>>>> I use process_items to send messages and urls, occasionally (usually when I write bad code) I need to send a lot of emails in quick succession but MisterHouse kills any unfinished ones, rather than running them in parallel which is what I expected. To test this I built a minimum reproducible case below where it launches a process that simply sleeps for 5 seconds twice in succession. As you can see from the print log, he first one gets killed. 
>>>>>>>> 
>>>>>>>> Is this normal? 
>>>>>>>> 
>>>>>>>> Giles 
>>>>>>>> 
>>>>>>>> <mytest.pl> 
>>>>>>>> 
>>>>>>>> print_log( "Longproc run 1", "INFORMATIONAL", "p_longproc" ); 
>>>>>>>> start $p_longproc; 
>>>>>>>> print_log( "Longproc run 2", "INFORMATIONAL", "p_longproc" ); 
>>>>>>>> start $p_longproc; 
>>>>>>>> </mytest.pl> 
>>>>>>>> 
>>>>>>>> <longproc.sh> 
>>>>>>>> 
>>>>>>>> #!/bin/sh 
>>>>>>>> # shell to sleep 
>>>>>>>> echo longproc start 5 second sleep pid= $$ 
>>>>>>>> sleep 5 
>>>>>>>> echo "longproc end" 
>>>>>>>> exit 0 
>>>>>>>> </longproc.sh> 
>>>>>>>> 
>>>>>>>> <print.log> 
>>>>>>>> 
>>>>>>>> 18/02/2021 08:10:27 [p_longproc] INFO Longproc run 1 
>>>>>>>> 18/02/2021 08:10:27 [p_longproc] INFO Longproc run 2 
>>>>>>>> Warning, a previous 'start' on this process has not finished yet 
>>>>>>>> Killing unfinished process id 9380 cmd=//home/pi/mh/GGBcode/procs/longproc.sh 
>>>>>>>> 
>>>>>>>> ... 
>>>>>>>> 
>>>>>>>> 18/02/2021 08:10:33 [p_longproc] longproc start 5 second sleep pid= 9381 
>>>>>>>> 18/02/2021 08:10:33 [p_longproc] longproc end 
>>>>>>>> 
>>>>>>>> </print.log> 
>>>>>>>> 
>>>>>>>> ________________________________________________________ 
>>>>>>>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users <https://lists.sourceforge.net/lists/listinfo/misterhouse-users> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>     Virus-free. www.avg.com <http://www.avg.com/> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> 
>>>>>> 
>>>>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> 
>>>>>> 
>>>>>> 
>>>>>> ________________________________________________________ 
>>>>>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users <https://lists.sourceforge.net/lists/listinfo/misterhouse-users> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> ________________________________________________________ 
>>>>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users <https://lists.sourceforge.net/lists/listinfo/misterhouse-users> 
>>>>> 
>>> 
>>> 
> <process_item_queue.pl><longproc.sh>________________________________________________________
> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users