Re: spot/remove repeated elements in XML documet

Manuel Souto Pico wrote:

 > My question is: Is there any easy way to spot and/or remove repeated
 > elements? I suppose it could be done with XSL or a perl module for XML...

OK, I'll take a crack at it using XML::Twig.

I assume that you know the name of the repeated elements, otherwise it becomes a 
little more complicated (and you need to define some rules to determine the 
candidates).

The principle is just to take the md5 of the inner xml of each entry, and to 
check that you have not seen it previously. If you have, delete the element, if 
not, flush it out (which also keeps memory usage low). Then store the md5 for 
future reference.

Additional features would be to process the inner xml string to taste, although 
as it is all extra spaces outside the tags are discarded (but the pretty_print 
option ensures that the output looks good).

In the current public release of XML::Twig you need the last flush, although the 
development version on xmltwig.com takes care of this for you (I kept forgetting 
it).

Does this help?

#!/usr/bin/perl

use strict;
use warnings;

use XML::Twig;
use Digest::MD5 qw(md5);

my %seen;

XML::Twig->new( twig_handlers => { entry => \&entry },
                 pretty_print => 'indented',
               )
          ->parsefile( 'my_long_document.xml' )
          ->flush;

sub entry
   { my( $t, $entry)= @_;
     my $md5= md5( $entry->inner_xml);
     if( $seen{$md5}) { $entry->delete; }
     else             { $t->flush; }
     $seen{$md5}=1;
   }
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs