[mh] Process_item_queue <partly solved>

Giles Godart-Brown <[email protected]> Sat, 20 Feb 2021 15:08:01 +0000
Newsgroups gmane.comp.misc.misterhouse.user
Message-ID <[email protected]>
I've now fixed the problem with changes to the process item before it 
was removed from the queue.

There are 2 limitations with this solution;

1) its a single FIFO, all process_items are executed in the order they 
arrive on the queue

2) I struggled with array references and arrays of arrays, so there is a 
limit to a single command process_item i.e. no add('program3 arguments', 
'program4 arguments', ...). It simply runs the first command assigned to 
the process_item.

I think this can be improved by adding a time_added_to_queue column and 
sorting the queue on process_name and time_added_to_queue.

Then when it processes the queue instead of it just taking the top of 
the queue it takes the first entry for each process_name i.e. a sort of 
parallel queue.

If I get some time over the weekend I'll see if I can implement this.

I'm no expert on the innards of MisterHouse and how the modules work, so 
If you think it worthwhile I'd like some help turning this into a module 
for a future pull request.

Thanks in anticipation

Giles


On 19/02/2021 19:17, Giles Godart-Brown wrote:
> Attached is a first go at building a process_item_queue.
>
> Its a single queue, and I'm sure one queue per process_item would be 
> better, but I've run out of talent :-)
>
> Giles
>
> On 18/02/2021 18:28, Giles Godart-Brown wrote:
>> I was going to build it for myself, then if its good enough (?) and 
>> fulfils backwards compatibility I will add a pull
>>
>> G
>>
>> On 18/02/2021 17:25, Jeff Siddall via misterhouse-users wrote:
>>> Giles,
>>>
>>> When you say "I'll build a queue" do you mean for your project or 
>>> adding it onto process_items?  As long as it was backwards 
>>> compatible with the existing behavior of a process_item, it would be 
>>> nice to add that onto the process_item so there is a standard way 
>>> for anyone to push items onto a process_item queue instead of 
>>> everyone implementing it their own way.
>>>
>>> Jeff
>>>
>>>
>>> On 2021-02-18 11:47 a.m., Giles Godart-Brown wrote:
>>>>
>>>> Thanks H
>>>>
>>>> I'll build a queue and update the docs over the weekend
>>>>
>>>> Giles
>>>>
>>>> On 18/02/2021 14:24, H Plato wrote:
>>>>> As far as I know that’s how it works, the process_item has a 
>>>>> single thread. You can check if the process_item is still active 
>>>>> by calling$p_longproc->done(), and then queuing 
>>>>> the additional commands to an array. That’s how I’ve dealt with 
>>>>> this in some of my modules
>>>>>
>>>>>> On Feb 18, 2021, at 1:22 AM, Giles Godart-Brown 
>>>>>> <[email protected] <mailto:[email protected]>> wrote:
>>>>>>
>>>>>> I use process_items to send messages and urls, occasionally 
>>>>>> (usually when I write bad code) I need to send a lot of emails in 
>>>>>> quick succession but MisterHouse kills any unfinished ones, 
>>>>>> rather than running them in parallel which is what I expected. To 
>>>>>> test this I built a minimum reproducible case below where it 
>>>>>> launches a process that simply sleeps for 5 seconds twice in 
>>>>>> succession. As you can see from the print log, he first one gets 
>>>>>> killed.
>>>>>>
>>>>>> Is this normal?
>>>>>>
>>>>>> Giles
>>>>>>
>>>>>> <mytest.pl>
>>>>>>
>>>>>> print_log( "Longproc run 1", "INFORMATIONAL", "p_longproc" );
>>>>>> start $p_longproc;
>>>>>> print_log( "Longproc run 2", "INFORMATIONAL", "p_longproc" );
>>>>>> start $p_longproc;
>>>>>> </mytest.pl>
>>>>>>
>>>>>> <longproc.sh>
>>>>>>
>>>>>> #!/bin/sh
>>>>>> # shell to sleep
>>>>>> echo longproc start 5 second sleep pid= $$
>>>>>> sleep 5
>>>>>> echo "longproc end"
>>>>>> exit 0
>>>>>> </longproc.sh>
>>>>>>
>>>>>> <print.log>
>>>>>>
>>>>>> 18/02/2021 08:10:27 [p_longproc] INFO Longproc run 1
>>>>>> 18/02/2021 08:10:27 [p_longproc] INFO Longproc run 2
>>>>>> Warning, a previous 'start' on this process has not finished yet
>>>>>> Killing unfinished process id 9380 
>>>>>> cmd=//home/pi/mh/GGBcode/procs/longproc.sh
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> 18/02/2021 08:10:33 [p_longproc] longproc start 5 second sleep 
>>>>>> pid= 9381
>>>>>> 18/02/2021 08:10:33 [p_longproc] longproc end
>>>>>>
>>>>>> </print.log>
>>>>>>
>>>>>> ________________________________________________________
>>>>>> To unsubscribe from this list, go to: 
>>>>>> https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>>>>
>>>>>
>>>>
>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> 
>>>>     Virus-free. www.avg.com 
>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> 
>>>>
>>>>
>>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>
>>>>
>>>> ________________________________________________________
>>>> To unsubscribe from this list, go to: 
>>>> https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>>
>>>
>>>
>>> ________________________________________________________
>>> To unsubscribe from this list, go to: 
>>> https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>
>
>

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
process_item_queue.pl (text/x-perl-script, 6.1 KB)
=head1 B<Process_Item_queue>

=head2 SYNOPSIS

  
=head2 DESCRIPTION

use this object to add process items to a queue. 
This version has a single queue called @process_item_queue, the next item on the queue is forked when the previous one finishes

=head2 INHERITS

<>

=head2 METHODS

add_process_item_to_queue add a process item to the queue, passed the  Process_item 
check_process_item_queue This is called regularly to see if it has any items to process on the queue

=over

=cut

#-----------------------------------------------------------------------
# tests
#-----------------------------------------------------------------------
# noloop=start
$my_test1   = new Voice_Cmd 'Run test [1,2,3,4]';
$p_longproc = new Process_Item;
set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh";
$p_longproc->set_output( $config_parms{data_dir} . "/logs/longproc.txt" );
set_timeout $p_longproc 5;

# noloop=stop

if ( $state = said $my_test1) {
	if ( $state == 1 ) {
		set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 2";
		set_timeout $p_longproc 5;
		add_process_item_to_queue($p_longproc);
	}
	elsif ( $state == 2 ) {
		print_log( "Longproc run 1 3sec", "INFORMATIONAL", "p_longproc" );
		set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 3";
		set_timeout $p_longproc 5;		
		add_process_item_to_queue($p_longproc);
		print_log( "Longproc run 2 6sec", "INFORMATIONAL", "p_longproc" );
		set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 6";
		set_timeout $p_longproc 5;	
		add_process_item_to_queue($p_longproc);
		print_log( "Longproc run  3 2sec", "INFORMATIONAL", "p_longproc" );
		set $p_longproc $config_parms{"code_dir"} . "/procs/longproc.sh 2";
		set_timeout $p_longproc 5;		
		add_process_item_to_queue($p_longproc);

	}

	elsif ( $state == 3 ) {
		purge_process_item_queue();
	}
	elsif ( $state == 4 ) {

	}
}

if ( done_now $p_longproc) {
	print_log( "Longproc done", "INFORMATIONAL", "p_longproc" );
	if ( timed_out $p_longproc) {
		print_log( 'longproc - timed out', "ERROR", "p_longproc" );
	}
	print_process_log( "longproc.txt", "p_longproc" );
}

#-----------------------------------------------------------------------
# every  seconds check the queue
#-----------------------------------------------------------------------

if ($New_Second) {
	check_process_item_queue();
}

#-----------------------------------------------------------------------
# globals
#-----------------------------------------------------------------------

#-----------------------------------------------------------------------
# Add a process item to the queue
#-----------------------------------------------------------------------

sub add_process_item_to_queue {
	my $this_process_object = $_[0];
	print_log(
		"Adding "
		  . substr( $this_process_object->{object_name}, 1 ) 
		  . " cmds"
		  . $this_process_object->{cmds}[0]
		  . " timeout"
		  . $this_process_object->{timeout}
		  . " to the queue",
		"INFORMATIONAL", "add_process_item_to_queue"
	);
	my $counter = scalar @process_item_queue;
	$process_item_queue[$counter][0] = substr( $this_process_object->{object_name}, 1 );
	$process_item_queue[$counter][1] = $this_process_object->{cmds}[0];
	$process_item_queue[$counter][2] = $this_process_object->{timeout};
	
	if ( scalar @process_item_queue == 1 ) {
		start_process_object($process_item_queue[0][0], $process_item_queue[0][1], $process_item_queue[0][2]);		    
	}
	

}

#-----------------------------------------------------------------------
# check the process_item_queue
# if the previous one is done, delete it and action the next
#-----------------------------------------------------------------------
sub check_process_item_queue {

	my ( $this_queued_process_name, $this_queued_process, $this_process_object, $next_process_object );
	my $count_items_in_queue = scalar @process_item_queue;
	if ( $count_items_in_queue > 0 ) {
		print_log(
			"there are  " . $count_items_in_queue . " items in the queue",
			"INFORMATIONAL", "check_process_item_queue" );
		$this_queued_process_name = $process_item_queue[0][0];
		$this_process_object = get_object_by_name( $this_queued_process_name );
		if ( !defined( $this_process_object->{done} ) )
		{    # this process is still running
			return;
		}
		else {
			print_log(
				$this_queued_process_name
				  . " has finished",
				"INFORMATIONAL", "check_process_item_queue"
			);
			if ( $count_items_in_queue == 1 ) {
				@process_item_queue = ();
			}
			else {
				my $first = shift @process_item_queue; # delete the firstitem
				print_log(
					"Starting new first  process item "
					  . $process_item_queue[0][0] . " cmds"
					  . $process_item_queue[0][1],
					"INFORMATIONAL", "check_process_item_queue"
				);
				start_process_object($process_item_queue[0][0], $process_item_queue[0][1], $process_item_queue[0][2]);
			    
			}
		}
		return;
	}
	else {
		return;
	}

}

#-----------------------------------------------------------------------
# start a process item given its name, cmds and timeout
#-----------------------------------------------------------------------
sub start_process_object {
	my ($this_process_name,$this_process_cmds,$this_process_timeout) = @_;
	my $command;
	$command = "set \$" . $this_process_name . " \"" . $this_process_cmds . "\";";
	print_log( "running " . $command, "INFORMATIONAL", "start_process_name" );
	eval $command;
	$command = "set_timeout \$" . $this_process_name . " " . $this_process_timeout . ";";
	print_log( "running " . $command, "INFORMATIONAL", "start_process_object" );
	eval $command;
	$command = "start \$" . $this_process_name . ";";
	print_log( "running " . $command, "INFORMATIONAL", "start_process_object" );
	eval $command;
	return;
}

#-----------------------------------------------------------------------
# purge process item queue
#-----------------------------------------------------------------------
sub purge_process_item_queue {
	print_log( "Purging process item queue ",
		"INFORMATIONAL", "purge_process_item_queue" );
	@process_item_queue = ();
	return;
}
########################################################################
# History:
# 19-02-2021 new code
# 20-02-2021 fixed for changes on each queue entry
proc queue test.log (text/plain, 2.4 KB)
20/02/2021 14:43:19 Ran telnet command: run test 2
20/02/2021 14:43:19 [p_longproc] INFO Longproc run 1 3sec
20/02/2021 14:43:19 [add_process_item_to_queue] INFO Adding p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 3 timeout5 to the queue
20/02/2021 14:43:19 [start_process_object] INFO running start $p_longproc;
20/02/2021 14:43:19 [p_longproc] INFO Longproc run 2 6sec
20/02/2021 14:43:19 [add_process_item_to_queue] INFO Adding p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 6 timeout5 to the queue
20/02/2021 14:43:19 [p_longproc] INFO Longproc run  3 2sec
20/02/2021 14:43:19 [add_process_item_to_queue] INFO Adding p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 2 timeout5 to the queue
20/02/2021 14:43:20 [check_process_item_queue] INFO there are  3 items in the queue
20/02/2021 14:43:22 [p_longproc] INFO Longproc done
20/02/2021 14:43:22 [p_longproc] 14:43:19 longproc start 3 second sleep pid= 9401
20/02/2021 14:43:22 [p_longproc] 14:43:22 longproc end
20/02/2021 14:43:23 [check_process_item_queue] INFO there are  3 items in the queue
20/02/2021 14:43:23 [check_process_item_queue] INFO p_longproc has finished
20/02/2021 14:43:23 [check_process_item_queue] INFO Starting new first  process item p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 6
20/02/2021 14:43:23 [start_process_object] INFO running start $p_longproc;
20/02/2021 14:43:24 [check_process_item_queue] INFO there are  2 items in the queue
20/02/2021 14:43:29 [p_longproc] INFO Longproc done
20/02/2021 14:43:29 [p_longproc] ERROR longproc - timed out
20/02/2021 14:43:29 [p_longproc] 14:43:23 longproc start 6 second sleep pid= 9407
20/02/2021 14:43:29 [check_process_item_queue] INFO there are  2 items in the queue
20/02/2021 14:43:29 [check_process_item_queue] INFO p_longproc has finished
20/02/2021 14:43:29 [check_process_item_queue] INFO Starting new first  process item p_longproc cmds/home/pi/mh/GGBcode/procs/longproc.sh 2
20/02/2021 14:43:29 [start_process_object] INFO running start $p_longproc;
20/02/2021 14:43:30 [check_process_item_queue] INFO there are  1 items in the queue
20/02/2021 14:43:31 [p_longproc] INFO Longproc done
20/02/2021 14:43:31 [p_longproc] 14:43:29 longproc start 2 second sleep pid= 9414
20/02/2021 14:43:31 [p_longproc] 14:43:31 longproc end
20/02/2021 14:43:32 [check_process_item_queue] INFO there are  1 items in the queue
20/02/2021 14:43:32 [check_process_item_queue] INFO p_longproc has finished
20/02