[mh] Process_item_queue <partly solved>
Giles Godart-Brown <[email protected]> Sat, 20 Feb 2021 15:08:01 +0000
| Newsgroups | gmane.comp.misc.misterhouse.user |
|---|---|
| Message-ID | <[email protected]> |
I've now fixed the problem with changes to the process item before it
was removed from the queue.
There are 2 limitations with this solution;
1) its a single FIFO, all process_items are executed in the order they
arrive on the queue
2) I struggled with array references and arrays of arrays, so there is a
limit to a single command process_item i.e. no add('program3 arguments',
'program4 arguments', ...). It simply runs the first command assigned to
the process_item.
I think this can be improved by adding a time_added_to_queue column and
sorting the queue on process_name and time_added_to_queue.
Then when it processes the queue instead of it just taking the top of
the queue it takes the first entry for each process_name i.e. a sort of
parallel queue.
If I get some time over the weekend I'll see if I can implement this.
I'm no expert on the innards of MisterHouse and how the modules work, so
If you think it worthwhile I'd like some help turning this into a module
for a future pull request.
Thanks in anticipation
Giles
On 19/02/2021 19:17, Giles Godart-Brown wrote:
> Attached is a first go at building a process_item_queue.
>
> Its a single queue, and I'm sure one queue per process_item would be
> better, but I've run out of talent :-)
>
> Giles
>
> On 18/02/2021 18:28, Giles Godart-Brown wrote:
>> I was going to build it for myself, then if its good enough (?) and
>> fulfils backwards compatibility I will add a pull
>>
>> G
>>
>> On 18/02/2021 17:25, Jeff Siddall via misterhouse-users wrote:
>>> Giles,
>>>
>>> When you say "I'll build a queue" do you mean for your project or
>>> adding it onto process_items? As long as it was backwards
>>> compatible with the existing behavior of a process_item, it would be
>>> nice to add that onto the process_item so there is a standard way
>>> for anyone to push items onto a process_item queue instead of
>>> everyone implementing it their own way.
>>>
>>> Jeff
>>>
>>>
>>> On 2021-02-18 11:47 a.m., Giles Godart-Brown wrote:
>>>>
>>>> Thanks H
>>>>
>>>> I'll build a queue and update the docs over the weekend
>>>>
>>>> Giles
>>>>
>>>> On 18/02/2021 14:24, H Plato wrote:
>>>>> As far as I know that’s how it works, the process_item has a
>>>>> single thread. You can check if the process_item is still active
>>>>> by calling$p_longproc->done(), and then queuing
>>>>> the additional commands to an array. That’s how I’ve dealt with
>>>>> this in some of my modules
>>>>>
>>>>>> On Feb 18, 2021, at 1:22 AM, Giles Godart-Brown
>>>>>> <[email protected] <mailto:[email protected]>> wrote:
>>>>>>
>>>>>> I use process_items to send messages and urls, occasionally
>>>>>> (usually when I write bad code) I need to send a lot of emails in
>>>>>> quick succession but MisterHouse kills any unfinished ones,
>>>>>> rather than running them in parallel which is what I expected. To
>>>>>> test this I built a minimum reproducible case below where it
>>>>>> launches a process that simply sleeps for 5 seconds twice in
>>>>>> succession. As you can see from the print log, he first one gets
>>>>>> killed.
>>>>>>
>>>>>> Is this normal?
>>>>>>
>>>>>> Giles
>>>>>>
>>>>>> <mytest.pl>
>>>>>>
>>>>>> print_log( "Longproc run 1", "INFORMATIONAL", "p_longproc" );
>>>>>> start $p_longproc;
>>>>>> print_log( "Longproc run 2", "INFORMATIONAL", "p_longproc" );
>>>>>> start $p_longproc;
>>>>>> </mytest.pl>
>>>>>>
>>>>>> <longproc.sh>
>>>>>>
>>>>>> #!/bin/sh
>>>>>> # shell to sleep
>>>>>> echo longproc start 5 second sleep pid= $$
>>>>>> sleep 5
>>>>>> echo "longproc end"
>>>>>> exit 0
>>>>>> </longproc.sh>
>>>>>>
>>>>>> <print.log>
>>>>>>
>>>>>> 18/02/2021 08:10:27 [p_longproc] INFO Longproc run 1
>>>>>> 18/02/2021 08:10:27 [p_longproc] INFO Longproc run 2
>>>>>> Warning, a previous 'start' on this process has not finished yet
>>>>>> Killing unfinished process id 9380
>>>>>> cmd=//home/pi/mh/GGBcode/procs/longproc.sh
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> 18/02/2021 08:10:33 [p_longproc] longproc start 5 second sleep
>>>>>> pid= 9381
>>>>>> 18/02/2021 08:10:33 [p_longproc] longproc end
>>>>>>
>>>>>> </print.log>
>>>>>>
>>>>>> ________________________________________________________
>>>>>> To unsubscribe from this list, go to:
>>>>>> https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>>>>
>>>>>
>>>>
>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>> Virus-free. www.avg.com
>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>>
>>>>
>>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>
>>>>
>>>> ________________________________________________________
>>>> To unsubscribe from this list, go to:
>>>> https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>>
>>>
>>>
>>> ________________________________________________________
>>> To unsubscribe from this list, go to:
>>> https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>
>
>
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
process_item_queue.pl
(text/x-perl-script, 6.1 KB)
=head1 B<Process_Item_queue>
=head2 SYNOPSIS
=head2 DESCRIPTION
use this object to add process items to a queue.
This version has a single queue called @process_item_queue, the next item on the queue is forked when the previous one finishes
=head2 INHERITS
<>
=head2 METHODS
add_process_item_to_queue add a process item to the queue, passed the Process_item
check_process_item_queue This is called regularly to see if it has any items to process on the queue
=over
=cut
#-----------------------------------------------------------------------
# tests
#-----------------------------------------------------------------------
# noloop=start
$my_test1 = new Voice_Cmd 'Run test [1,2,3,4]';
$p_longproc = new Process_Item;
set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh";
$p_longproc->set_output( $config_parms{data_dir} . "/logs/longproc.txt" );
set_timeout $p_longproc 5;
# noloop=stop
if ( $state = said $my_test1) {
if ( $state == 1 ) {
set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 2";
set_timeout $p_longproc 5;
add_process_item_to_queue($p_longproc);
}
elsif ( $state == 2 ) {
print_log( "Longproc run 1 3sec", "INFORMATIONAL", "p_longproc" );
set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 3";
set_timeout $p_longproc 5;
add_process_item_to_queue($p_longproc);
print_log( "Longproc run 2 6sec", "INFORMATIONAL", "p_longproc" );
set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 6";
set_timeout $p_longproc 5;
add_process_item_to_queue($p_longproc);
print_log( "Longproc run 3 2sec", "INFORMATIONAL", "p_longproc" );
set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 2";
set_timeout $p_longproc 5;
add_process_item_to_queue($p_longproc);
}
elsif ( $state == 3 ) {
purge_process_item_queue();
}
elsif ( $state == 4 ) {
}
}
if ( done_now $p_longproc) {
print_log( "Longproc done", "INFORMATIONAL", "p_longproc" );
if ( timed_out $p_longproc) {
print_log( 'longproc - timed out', "ERROR", "p_longproc" );
}
print_process_log( "longproc.txt", "p_longproc" );
}
#-----------------------------------------------------------------------
# every seconds check the queue
#-----------------------------------------------------------------------
if ($New_Second) {
check_process_item_queue();
}
#-----------------------------------------------------------------------
# globals
#-----------------------------------------------------------------------
#-----------------------------------------------------------------------
# Add a process item to the queue
#-----------------------------------------------------------------------
sub add_process_item_to_queue {
my $this_process_object = $_[0];
print_log(
"Adding "
. substr( $this_process_object->{object_name}, 1 )
. " cmds"
. $this_process_object->{cmds}[0]
. " timeout"
. $this_process_object->{timeout}
. " to the queue",
"INFORMATIONAL", "add_process_item_to_queue"
);
my $counter = scalar @process_item_queue;
$process_item_queue[$counter][0] = substr( $this_process_object->{object_name}, 1 );
$process_item_queue[$counter][1] = $this_process_object->{cmds}[0];
$process_item_queue[$counter][2] = $this_process_object->{timeout};
if ( scalar @process_item_queue == 1 ) {
start_process_object($process_item_queue[0][0], $process_item_queue[0][1], $process_item_queue[0][2]);
}
}
#-----------------------------------------------------------------------
# check the process_item_queue
# if the previous one is done, delete it and action the next
#-----------------------------------------------------------------------
sub check_process_item_queue {
my ( $this_queued_process_name, $this_queued_process, $this_process_object, $next_process_object );
my $count_items_in_queue = scalar @process_item_queue;
if ( $count_items_in_queue > 0 ) {
print_log(
"there are " . $count_items_in_queue . " items in the queue",
"INFORMATIONAL", "check_process_item_queue" );
$this_queued_process_name = $process_item_queue[0][0];
$this_process_object = get_object_by_name( $this_queued_process_name );
if ( !defined( $this_process_object->{done} ) )
{ # this process is still running
return;
}
else {
print_log(
$this_queued_process_name
. " has finished",
"INFORMATIONAL", "check_process_item_queue"
);
if ( $count_items_in_queue == 1 ) {
@process_item_queue = ();
}
else {
my $first = shift @process_item_queue; # delete the firstitem
print_log(
"Starting new first process item "
. $process_item_queue[0][0] . " cmds"
. $process_item_queue[0][1],
"INFORMATIONAL", "check_process_item_queue"
);
start_process_object($process_item_queue[0][0], $process_item_queue[0][1], $process_item_queue[0][2]);
}
}
return;
}
else {
return;
}
}
#-----------------------------------------------------------------------
# start a process item given its name, cmds and timeout
#-----------------------------------------------------------------------
sub start_process_object {
my ($this_process_name,$this_process_cmds,$this_process_timeout) = @_;
my $command;
$command = "set \$" . $this_process_name . " \"" . $this_process_cmds . "\";";
print_log( "running " . $command, "INFORMATIONAL", "start_process_name" );
eval $command;
$command = "set_timeout \$" . $this_process_name . " " . $this_process_timeout . ";";
print_log( "running " . $command, "INFORMATIONAL", "start_process_object" );
eval $command;
$command = "start \$" . $this_process_name . ";";
print_log( "running " . $command, "INFORMATIONAL", "start_process_object" );
eval $command;
return;
}
#-----------------------------------------------------------------------
# purge process item queue
#-----------------------------------------------------------------------
sub purge_process_item_queue {
print_log( "Purging process item queue ",
"INFORMATIONAL", "purge_process_item_queue" );
@process_item_queue = ();
return;
}
########################################################################
# History:
# 19-02-2021 new code
# 20-02-2021 fixed for changes on each queue entry
proc queue test.log
(text/plain, 2.4 KB)
20/02/2021 14:43:19 Ran telnet command: run test 2 20/02/2021 14:43:19 [p_longproc] INFO Longproc run 1 3sec 20/02/2021 14:43:19 [add_process_item_to_queue] INFO Adding p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 3 timeout5 to the queue 20/02/2021 14:43:19 [start_process_object] INFO running start $p_longproc; 20/02/2021 14:43:19 [p_longproc] INFO Longproc run 2 6sec 20/02/2021 14:43:19 [add_process_item_to_queue] INFO Adding p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 6 timeout5 to the queue 20/02/2021 14:43:19 [p_longproc] INFO Longproc run 3 2sec 20/02/2021 14:43:19 [add_process_item_to_queue] INFO Adding p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 2 timeout5 to the queue 20/02/2021 14:43:20 [check_process_item_queue] INFO there are 3 items in the queue 20/02/2021 14:43:22 [p_longproc] INFO Longproc done 20/02/2021 14:43:22 [p_longproc] 14:43:19 longproc start 3 second sleep pid= 9401 20/02/2021 14:43:22 [p_longproc] 14:43:22 longproc end 20/02/2021 14:43:23 [check_process_item_queue] INFO there are 3 items in the queue 20/02/2021 14:43:23 [check_process_item_queue] INFO p_longproc has finished 20/02/2021 14:43:23 [check_process_item_queue] INFO Starting new first process item p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 6 20/02/2021 14:43:23 [start_process_object] INFO running start $p_longproc; 20/02/2021 14:43:24 [check_process_item_queue] INFO there are 2 items in the queue 20/02/2021 14:43:29 [p_longproc] INFO Longproc done 20/02/2021 14:43:29 [p_longproc] ERROR longproc - timed out 20/02/2021 14:43:29 [p_longproc] 14:43:23 longproc start 6 second sleep pid= 9407 20/02/2021 14:43:29 [check_process_item_queue] INFO there are 2 items in the queue 20/02/2021 14:43:29 [check_process_item_queue] INFO p_longproc has finished 20/02/2021 14:43:29 [check_process_item_queue] INFO Starting new first process item p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 2 20/02/2021 14:43:29 [start_process_object] INFO running start $p_longproc; 20/02/2021 14:43:30 [check_process_item_queue] INFO there are 1 items in the queue 20/02/2021 14:43:31 [p_longproc] INFO Longproc done 20/02/2021 14:43:31 [p_longproc] 14:43:29 longproc start 2 second sleep pid= 9414 20/02/2021 14:43:31 [p_longproc] 14:43:31 longproc end 20/02/2021 14:43:32 [check_process_item_queue] INFO there are 1 items in the queue 20/02/2021 14:43:32 [check_process_item_queue] INFO p_longproc has finished 20/02