Re: aerb "file has been altered" errors after removing files in branch
Calvin Webster <[email protected]> Mon, 25 Apr 2016 12:08:22 -0400
| Newsgroups | gmane.comp.version-control.aegis.user |
|---|---|
| Message-ID | <1461600502.14245.169.camel@klink> |
--===============1162712530197071158==
Content-Type: multipart/alternative; boundary="=-322O5dSsE3nO4O82jwKE"
--=-322O5dSsE3nO4O82jwKE
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
On Mon, 2016-04-25 at 15:08 +0200, Ralf Fassel wrote:
> A different approach would be to temporarily alter the project config
> to go directly to being_reviewed instead of awaiting_review and see
> whether this makes a difference. It could be that only 'aerb' has a
> problem with this step, and that other steps are ok.
That's an excellent idea Ralf! I'll give that a shot. Thanks!
> | Yes, thanks. While searching the Aegis source for "crypto" this
> | weekend I found the aefp man page. After reading it I discovered, as
> | you say, it's not installed. I ran "rpmbuild -bi" on the source RPM
> | but the binary doesn't appear to have been built there and I'm not
> | sure how to get it built. In case it was a built-in without an alias
> | I tried using as "aegis -fp buildinfo" but no cigar.
>
> I downloaded the aegis source distribution from aegis.sourceforge.net
> and ran configure/make. There aefp is built in bin/aefp during the
> make step. I'm on Linux with gcc version 4.8.1 and had to add
> -fpermissive to CXXFLAGS to work around some const/non-const return
> value errors during compilation.
>
> sh ./configure CFLAGS='-g -fpermissive' CXXFLAGS='-g -fpermissive'
> make
> ls -l bin/aefp
> -rwxr-xr-x 1 ralf abg 15114942 Apr 25 14:34 bin/aefp*
>
> The binary produces the same crypto output as listed in the .fs files
> for my changes.
I was eventually able to rebuild Aegis from the Source RPM and retrieve
the "aefp" executable. See below for results.
> | If there is a simpler way to rid myself of this blocking error I'd
> | be happy to hear it. I don't care about the history of this file or
> | its content in this location.
>
> I would start with a recompile of aegis, find the position where the
> error in question is generated, and add some diagnostics, especially
> show the two crypto strings which do not match, and try to find out
> how aegis gets them, and then try to find out whether it's a genuine
> error or a flaw in the aegis process with branches (a possibility
> someone else pointed out earlier in this discussion). Eg. it might
> make no sense to check "crpyto" for files which have been removed.
> Or aegis compares against a non-existing crypto. Or...
I'll use this approach when I come back to revisit this after I put this
project to bed. I've started work on a new simulator and must keep my
attention focused on the task at hand.
> As to hacking aegis' database...
> Myself, I would make a backup of the aegis project directory, then
> search all files for the filename in question, then simply remove
> every section which refers to that file. This way aegis would never
> have known the file.
>
> cd /path/to/aegis/projectdir
> find info -type f | xargs grep FILENAME
>
> Eg:
> info/change/0/004.branch/0/000.branch/0/020.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/0/010.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/3/333.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/3/387.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/3/330.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/3/366.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/1/198.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/4/418.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/7/701.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.branch/7/706.fs: file_name = "include/si++/hdsys.h";
> info/change/0/004.branch/0/000.fs: file_name = "include/si++/hdsys.h";
>
> This breaks the ability to recreate changes from the past which still
> had the file in question, but since you said you don't care...
I think this is the way I'll end up handling this issue for now. I think
it may be simplest for me to go ahead and remove all traces of these
misplaced derived files at this point. I found that removing both the
stanzas containing the base file and path (CymSOUND93/bin/buildinfo) in
the 15053.fs file eliminated the original error. One stanza was for the
"aemv" and the other for the "aerm". However, as I suspected, another
error took its place citing another file that had previously been moved
and removed.
I just hope it ends here, in this branch. If I can get through this I
definitely would never again add derived files to an Aegis project. I
suspect this was the main cause when Aegis config was set to retain
derived files. I'll also be cautious about using "aemv" and "aerm" on
the same files in the same branch.
> Also note that I have not used different branches, and I am still
> using a very old version of aegis (3.29), so I'm probably missing
> something here, but looking at the output, I would expect the branches
> to show up in parallel.
Yes, there were two parallel, sibling branches involved in this
modification, both under a parent branch of the project. See below for
more.
> Of course, I would never recommend this to others to do :-)
Neither would I, unless they backup everything and accept responsibility
for their work.
Thanks for taking time to give me your insights Ralf. I always welcome
your suggestions and/or criticisms.
Below is a summary of where I'm at right now with this issue. I'll
update with another post when I resolve it.
./Cal
============================================
Issue Summary
Aegis may be having trouble reconciling the "aemv" AND "aerm" of a file
in the same branch in the presence of retained derived files.
Ours is probably a unique case so I'll try to explain how this happened.
All our Aegis projects are configured in aegis.conf with the following
development_directory_style options:
source_file_symlink = true;
derived_file_copy = true;
derived_file_st_start_only = true;
New code was introduced in a branch (av8b93.23.11117) representing an
engineering change - a "minor revision", so to speak. This code would
initially not build in our build environment so we just dumped the
entire tree, including all derived files, into our Aegis project. Other
changes were made to other code and the branch integrated without
incident.
In a sister branch (av8b93.23.15023) we found a way to build this code
in our environment. This required renaming the parent directory and
removing all but the source files.
In one change we renamed the directory that as yet still had all the
derived files as part of the project.
In a subsequent change we removed all the derived files from the
project.
Aegis config had not yet been modified so that any new changes still
showed derived files. Now, however, they're coming from the most recent
successful (integrated) build.
No errors or warnings were raised in any of these changes.
When preparing to integrate this sister branch (av8b93.23.15023) the
following error was raised:
[root@pegasus2 bin]# su - mtrev
Last login: Fri Apr 22 08:05:10 EDT 2016 on pts/0
[mtrev@pegasus2 ~]$ aerb -p av8b93.23 -c 15023
aegis: project "av8b93.23": change 15023: file
"CymSOUND93/bin/buildinfo,D" has been altered
[mtrev@pegasus2 ~]$
At first I thought this was odd because *,D files are always altered and
I wasn't aware that Aegis tracked the contents of ,D files except during
a change where the base file is being altered.
At this point I investigated the error, looking for the offending file
but could not find it. This diff file in that particular relative path
no longer exists anywhere. It must have existed in the development
directory when we removed the base file in a change within this branch
but that has long since been integrated.
Then I started looking at the base file, which was listed in the "info/"
files for the project. That's when I came upon the "crypto" hash and
began to wonder how it was being used.
I was eventually able to rebuild Aegis from the source RPM while
retaining the build tree. I copied over the "aefp" command to the
development server and generated a fingerprint on the base file. I then
replaced the "crypto" entry for the ,D file in the corresponding "info/"
file (15023.fs) but this changed nothing.
I then manually created the CymSOUND93/bin/buildinfo,D file (diff -U10
buildinfo), and generated a fingerprint with "aefp buildinfo,D". After
replacing the "crypto" enty in 15023.fs with this new hash, I got the
exact same error.
Apparently, Aegis is not looking in the branch.15023/baseline/ for this
file. The trouble is, I don't really know where aegis is looking for it
or what value it's using to compare to the value in 15023.fs. This is
what leads me to believe that Aegis was not designed to accommodate this
particular scenario ("aemv directory/file new_directory/file", followed
by "aerm new_directory/file" in a subsequent change within the same
branch)
I would like to investigate further and learn what Aegis is really
trying to do here. Then maybe I'd be able to contribute some useful
information about how to fix this problem to prevent future issues. At
this point, however, I don't have a lot of time to go down this rabbit
hole. That'll have to wait until this project is put to bed.
--=-322O5dSsE3nO4O82jwKE
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 7bit
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.32.2">
</HEAD>
<BODY>
On Mon, 2016-04-25 at 15:08 +0200, Ralf Fassel wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
A different approach would be to temporarily alter the project config
to go directly to being_reviewed instead of awaiting_review and see
whether this makes a difference. It could be that only 'aerb' has a
problem with this step, and that other steps are ok.
</PRE>
</BLOCKQUOTE>
<PRE>
</PRE>
<B>That's an excellent idea Ralf! I'll give that a shot. Thanks!</B><BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
| Yes, thanks. While searching the Aegis source for "crypto" this
| weekend I found the aefp man page. After reading it I discovered, as
| you say, it's not installed. I ran "rpmbuild -bi" on the source RPM
| but the binary doesn't appear to have been built there and I'm not
| sure how to get it built. In case it was a built-in without an alias
| I tried using as "aegis -fp buildinfo" but no cigar.
I downloaded the aegis source distribution from aegis.sourceforge.net
and ran configure/make. There aefp is built in bin/aefp during the
make step. I'm on Linux with gcc version 4.8.1 and had to add
-fpermissive to CXXFLAGS to work around some const/non-const return
value errors during compilation.
sh ./configure CFLAGS='-g -fpermissive' CXXFLAGS='-g -fpermissive'
make
ls -l bin/aefp
-rwxr-xr-x 1 ralf abg 15114942 Apr 25 14:34 bin/aefp*
The binary produces the same crypto output as listed in the .fs files
for my changes.
</PRE>
</BLOCKQUOTE>
<BR>
<B>I was eventually able to rebuild Aegis from the Source RPM and retrieve the "aefp" executable. See below for results.</B><BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
| If there is a simpler way to rid myself of this blocking error I'd
| be happy to hear it. I don't care about the history of this file or
| its content in this location.
I would start with a recompile of aegis, find the position where the
error in question is generated, and add some diagnostics, especially
show the two crypto strings which do not match, and try to find out
how aegis gets them, and then try to find out whether it's a genuine
error or a flaw in the aegis process with branches (a possibility
someone else pointed out earlier in this discussion). Eg. it might
make no sense to check "crpyto" for files which have been removed.
Or aegis compares against a non-existing crypto. Or...
</PRE>
</BLOCKQUOTE>
<PRE>
</PRE>
<B>I'll use this approach when I come back to revisit this after I put this project to bed. I've started work on a new simulator and must keep my attention focused on the task at hand.</B><BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
As to hacking aegis' database...
Myself, I would make a backup of the aegis project directory, then
search all files for the filename in question, then simply remove
every section which refers to that file. This way aegis would never
have known the file.
cd /path/to/aegis/projectdir
find info -type f | xargs grep FILENAME
Eg:
info/change/0/004.branch/0/000.branch/0/020.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/0/010.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/3/333.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/3/387.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/3/330.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/3/366.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/1/198.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/4/418.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/7/701.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.branch/7/706.fs: file_name = "include/si++/hdsys.h";
info/change/0/004.branch/0/000.fs: file_name = "include/si++/hdsys.h";
This breaks the ability to recreate changes from the past which still
had the file in question, but since you said you don't care...
</PRE>
</BLOCKQUOTE>
<PRE>
</PRE>
<B>I think this is the way I'll end up handling this issue for now. I think it may be simplest for me to go ahead and remove all traces of these misplaced derived files at this point. I found that removing both the stanzas containing the base file and path (CymSOUND93/bin/buildinfo) in the 15053.fs file eliminated the original error. One stanza was for the "aemv" and the other for the "aerm". However, as I suspected, another error took its place citing another file that had previously been moved and removed.</B><BR>
<BR>
<B>I just hope it ends here, in this branch. If I can get through this I definitely would never again add derived files to an Aegis project. I suspect this was the main cause when Aegis config was set to retain derived files. I'll also be cautious about using "aemv" and "aerm" on the same files in the same branch.</B><BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
Also note that I have not used different branches, and I am still
using a very old version of aegis (3.29), so I'm probably missing
something here, but looking at the output, I would expect the branches
to show up in parallel.
</PRE>
</BLOCKQUOTE>
<BR>
<B>Yes, there were two parallel, sibling branches involved in this modification, both under a parent branch of the project. See below for more.</B><BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
Of course, I would never recommend this to others to do :-)
</PRE>
</BLOCKQUOTE>
<BR>
<B>Neither would I, unless they backup everything and accept responsibility for their work.</B><BR>
<BR>
<B>Thanks for taking time to give me your insights Ralf. I always welcome your suggestions and/or criticisms.</B><BR>
<BR>
<B>Below is a summary of where I'm at right now with this issue. I'll update with another post when I resolve it.</B><BR>
<BR>
<B>./Cal</B><BR>
<BR>
============================================<BR>
Issue Summary<BR>
<BR>
Aegis may be having trouble reconciling the "aemv" AND "aerm" of a file in the same branch in the presence of retained derived files.<BR>
<BR>
Ours is probably a unique case so I'll try to explain how this happened.<BR>
<BR>
All our Aegis projects are configured in aegis.conf with the following development_directory_style options:<BR>
<BR>
source_file_symlink = true;<BR>
derived_file_copy = true;<BR>
derived_file_st_start_only = true;<BR>
<BR>
New code was introduced in a branch (av8b93.23.11117) representing an engineering change - a "minor revision", so to speak. This code would initially not build in our build environment so we just dumped the entire tree, including all derived files, into our Aegis project. Other changes were made to other code and the branch integrated without incident.<BR>
<BR>
In a sister branch (av8b93.23.15023) we found a way to build this code in our environment. This required renaming the parent directory and removing all but the source files.<BR>
<BR>
In one change we renamed the directory that as yet still had all the derived files as part of the project.<BR>
In a subsequent change we removed all the derived files from the project.<BR>
Aegis config had not yet been modified so that any new changes still showed derived files. Now, however, they're coming from the most recent successful (integrated) build.<BR>
<BR>
No errors or warnings were raised in any of these changes.<BR>
<BR>
When preparing to integrate this sister branch (av8b93.23.15023) the following error was raised:<BR>
<BR>
[<A HREF="mailto:root@pegasus2">root@pegasus2</A> bin]# su - mtrev<BR>
Last login: Fri Apr 22 08:05:10 EDT 2016 on pts/0<BR>
[<A HREF="mailto:mtrev@pegasus2">mtrev@pegasus2</A> ~]$ aerb -p av8b93.23 -c 15023<BR>
aegis: project "av8b93.23": change 15023: file "CymSOUND93/bin/buildinfo,D" has been altered<BR>
[<A HREF="mailto:mtrev@pegasus2">mtrev@pegasus2</A> ~]$ <BR>
<BR>
At first I thought this was odd because *,D files are always altered and I wasn't aware that Aegis tracked the contents of ,D files except during a change where the base file is being altered. <BR>
<BR>
At this point I investigated the error, looking for the offending file but could not find it. This diff file in that particular relative path no longer exists anywhere. It must have existed in the development directory when we removed the base file in a change within this branch but that has long since been integrated.<BR>
<BR>
Then I started looking at the base file, which was listed in the "info/" files for the project. That's when I came upon the "crypto" hash and began to wonder how it was being used.<BR>
<BR>
I was eventually able to rebuild Aegis from the source RPM while retaining the build tree. I copied over the "aefp" command to the development server and generated a fingerprint on the base file. I then replaced the "crypto" entry for the ,D file in the corresponding "info/" file (15023.fs) but this changed nothing.<BR>
<BR>
I then manually created the CymSOUND93/bin/buildinfo,D file (diff -U10 buildinfo), and generated a fingerprint with "aefp buildinfo,D". After replacing the "crypto" enty in 15023.fs with this new hash, I got the exact same error.<BR>
<BR>
Apparently, Aegis is not looking in the branch.15023/baseline/ for this file. The trouble is, I don't really know where aegis is looking for it or what value it's using to compare to the value in 15023.fs. This is what leads me to believe that Aegis was not designed to accommodate this particular scenario (<TT>"aemv directory/file new_directory/file</TT>", followed by "<TT>aerm new_directory/file</TT>" in a subsequent change within the same branch)<BR>
<BR>
I would like to investigate further and learn what Aegis is really trying to do here. Then maybe I'd be able to contribute some useful information about how to fix this problem to prevent future issues. At this point, however, I don't have a lot of time to go down this rabbit hole. That'll have to wait until this project is put to bed.<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
</BODY>
</HTML>
--=-322O5dSsE3nO4O82jwKE--
--===============1162712530197071158==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KQWVnaXMtdXNl
cnMgbWFpbGluZyBsaXN0CkFlZ2lzLXVzZXJzQGxpc3RzLmF1dWcub3JnLmF1Cmh0dHBzOi8vbGlz
dHMuYXV1Zy5vcmcuYXUvbGlzdGluZm8vYWVnaXMtdXNlcnMK
--===============1162712530197071158==--