makehistory can generate duplicate lines was:[Re: ovsqlite - bug?]

Kamil Jońca <[email protected]> writes:
[...]

>
> So the first thigs I should do is to make history consistent with spool
> and then test further.
> But for now I am pretty sure, that sqlite worked properly as, it forces
> uniqueness of  (group,artnum )  in overview (and other methods not) -
> and this causes error messages.
> Sorry for bothering you.
>

I tried to rebuild history:

--8<---------------cut here---------------start------------->8---
kjonca@alfa:/var/lib/news%sudo -u news env PATH=$PATH:/usr/lib/news/bin makehistory -f history.n                                                                                                                  
makehistory: dbzstore duplicate message-id history.n:[2FFF2E79E03CA12C3E3C4F7E398EBAF7]@60072283 Success
makehistory: dbzstore duplicate message-id history.n:[C1067871BE1F25B369114E7C94635D22]@121782076 Success
makehistory: dbzstore duplicate message-id history.n:[6282400F8A293D047E9052468792AF18]@176267656 Success
makehistory: dbzstore duplicate message-id history.n:[65B607B90549808E3D4A4E3E33DD6D4A]@226791333 Success
makehistory: dbzstore duplicate message-id history.n:[E55B9C6F2E865C431F4466F1085D7596]@352606103 Success
makehistory: dbzstore duplicate message-id history.n:[28464C771BBE6A7D471737C28E6C7F6A]@411231076 Success
makehistory: dbzstore duplicate message-id history.n:[1F375DE27593F75A6022530840232809]@470885597 Success
--8<---------------cut here---------------end--------------->8---
You can see that there is 7 messages about duplicates ... (anyone surprised?)

I took a look in new history file and end with command:

--8<---------------cut here---------------start------------->8---
kjonca@alfa:/var/lib/news%grep -n -A2 -B2 '2FFF2E79E03CA12C3E3C4F7E398EBAF7\|C1067871BE1F25B369114E7C94635D22\|6282400F8A293D047E9052468792AF18\|65B607B90549808E3D4A4E3E33DD6D4A\|E55B9C6F2E865C431F4466F1085D7596\|28464C771BBE6A7D471737C28E6C7F6A\|1F375DE27593F75A6022530840232809' history.n 
1:[2FFF2E79E03CA12C3E3C4F7E398EBAF7]    1643424642~-~1538492029 @03FF30000000000000000000004400000008@
2-[50AFBD2AD3C16FA233D6E88D7D5B600C]    1643424642~-~1538492036 @03FF30000000000000000000005E00000007@
3-[4CA26D1C22BC1FCB52DCAA5B56F36666]    1643424642~-~1538492212 @03FF30000000000000000000005F00000007@
--
612979-[11F1E04DF20F6533C381DB6C649B374D]       1643427583~-~1566810064 @03FF3000000000000000001FFFE500000007@
612980-[D121941BCEA5E037523AE35801A53596]       1643427583~-~1566810077 @03FF3000000000000000001FFFE800000007@
612981:[2FFF2E79E03CA12C3E3C4F7E398EBAF7]       1643424642~-~1538492029 @03FF30000000000000000000004400000008@
612982:[C1067871BE1F25B369114E7C94635D22]       1643427583~-~1566810112 @03FF31000000000000000000004400000008@
612983-[82BFD6304D1012333F22C559B8AF46A3]       1643427583~-~1566810146 @03FF31000000000000000000009100000007@
612984-[B2EAA6504DB45DC7F33C214880D99A50]       1643427583~-~1566810185 @03FF31000000000000000000009200000007@
--
1242667-[4F893D038B278138E8AFB8E1C899BFFA]      1643431380~-~1595582940 @03FF3100000000000000001FFFF400000007@
1242668-[B22F99E507F22F30250D4A2D28742A19]      1643431380~-~1595582974 @03FF3100000000000000001FFFFD00000007@
1242669:[C1067871BE1F25B369114E7C94635D22]      1643427583~-~1566810112 @03FF31000000000000000000004400000008@
1242670:[6282400F8A293D047E9052468792AF18]      1643431380~-~1595583001 @03FF32000000000000000000004400000008@
1242671-[A0615E54374BB1263836C289CAB75EF9]      1643431380~-~1595583061 @03FF32000000000000000000004600000007@
1242672-[C07606242B08C0D7FC987945FE0CECA1]      1643431380~-~1595583073 @03FF32000000000000000000004700000007@
--
1798639-[BE079E8E4CA8D10A0ED4C8761C04EE9B]      1643433853~-~1623959587 @03FF3200000000000000001FFFF900000007@
1798640-[C52CD86396DB88B01887D56389ECC6E7]      1643433853~-~1623959629 @03FF3200000000000000001FFFFC00000007@
1798641:[6282400F8A293D047E9052468792AF18]      1643431380~-~1595583001 @03FF32000000000000000000004400000008@
1798642:[65B607B90549808E3D4A4E3E33DD6D4A]      1643433853~-~1623959630 @03FF33000000000000000000004400000008@
1798643-[7D4F42A4C19BC04239EBB1C1538E64AE]      1643433853~-~1623959631 @03FF33000000000000000000004800000007@
1798644-[F49DEA0F8095B1B96440EF800427A797]      1643433853~-~1623959632 @03FF33000000000000000000004B00000007@
--
2314183-[BC90903F831BFB1DE8CA39C98F8ED2C9]      1653058806~-~1653058751 @03FF3300000000000000001FFFF600000007@
2314184-[A505379E708A59F16DAA62541CCFDB67]      1653058866~-~1653058823 @03FF3300000000000000001FFFFE00000007@
2314185:[65B607B90549808E3D4A4E3E33DD6D4A]      1643433853~-~1623959630 @03FF33000000000000000000004400000008@
2314186-[1AA051033A3D96E8E79F6F42C2159419]      1643409332~-~1454972341 @03FF3400000000000000000EE11400000006@
2314187-[8FCA5D33F8688E4E75445850D7CCE398]      1643409332~-~1454972378 @03FF3400000000000000000EE11600000006@
--
2921682-[8735D546AC02468CEC974116E2291BBA]      1669003873~-~1669003801 @03FF3400000000000000000EE10E00000007@
2921683-[4A8E1E3096F262C460C7963971964912]      1669004073~-~1669003319 @03FF3400000000000000000EE11000000007@
2921684:[E55B9C6F2E865C431F4466F1085D7596]      1643412102~-~1465897595 @03FF35000000000000000000004400000007@
2921685-[00F1642A04FEAA3F23269EAF258BDC4A]      1643412102~-~1465897596 @03FF35000000000000000000004800000006@
2921686-[9025840433BA049A341F2D7BF71F44AE]      1643412102~-~1465897596 @03FF35000000000000000000004A00000006@
--
3597988-[D8DB3B14DB0DD47F9344179C17974EF7]      1643417459~-~1487957728 @03FF3500000000000000001FFFCF00000006@
3597989-[EB84472A4704BB56196FAE8D119B6102]      1643417459~-~1487957745 @03FF3500000000000000001FFFD100000006@
3597990:[E55B9C6F2E865C431F4466F1085D7596]      1643412102~-~1465897595 @03FF35000000000000000000004400000007@
3597991:[28464C771BBE6A7D471737C28E6C7F6A]      1643417459~-~1487957763 @03FF36000000000000000000004400000007@
3597992-[25B7BD351BFC0DCA37C9E8D143C74313]      1643417459~-~1487957771 @03FF36000000000000000000008800000006@
3597993-[A4C478830BCD26EDF9206CF4DC18CE29]      1643417459~-~1487957812 @03FF36000000000000000000008B00000006@
--
4196192-[E6B5FC05ED4C9BDB2DC0452382676E79]      1643421414~-~1512585723 @03FF3600000000000000001FFFD300000006@
4196193-[C2574DC662BA26F96D83668966312137]      1643421414~-~1512585730 @03FF3600000000000000001FFFD500000006@
4196194:[28464C771BBE6A7D471737C28E6C7F6A]      1643417459~-~1487957763 @03FF36000000000000000000004400000007@
4196195:[1F375DE27593F75A6022530840232809]      1643421414~-~1512585730 @03FF37000000000000000000004400000007@
4196196-[37B8A01C3D7A5849FC290DA5D3CF91BB]      1643421414~-~1512585769 @03FF37000000000000000000007900000006@
4196197-[578459C8F2A952BC0782C85CCC270CD5]      1643421414~-~1512585775 @03FF37000000000000000000007B00000006@
--
4804906-[9A0C78E7194A7BBA54EB5CB20962BCAA]      1643424642~-~1538491933 @03FF3700000000000000001FFFE700000006@
4804907-[55DB71AEF999C1BCD62FAE9BECC0ED6C]      1643424642~-~1538492145 @03FF3700000000000000001FFFE800000006@
4804908:[1F375DE27593F75A6022530840232809]      1643421414~-~1512585730 @03FF37000000000000000000004400000007@
4804909-[D712318D392955A986E84DCC54C5CC1D]      1651442596~1657056641~1651440641        @040000626F03000200000000000000000000@
4804910-[99690AB74014DD13A932527753D33814]      1646094796~-~1646093042 @040000621D69000200000000000000000000@
--8<---------------cut here---------------end--------------->8---

This is clear (at least for me) that for each cycbuff file (except this
named "4") makehistory, after scanning whole buffer, it additionally try
to write entry about "first" message in buffer (compare lines 1-612981,
612982-1242669 and so on)

Here is additional information - my cycbuff config:
--8<---------------cut here---------------start------------->8---
cycbuff:0:/var/spool/news/cycbuffs/0:8388608
cycbuff:1:/var/spool/news/cycbuffs/1:8388608
cycbuff:2:/var/spool/news/cycbuffs/2:8388608
cycbuff:3:/var/spool/news/cycbuffs/3:8388608
cycbuff:4:/var/spool/news/cycbuffs/4:8388608
cycbuff:5:/var/spool/news/cycbuffs/5:8388608
cycbuff:6:/var/spool/news/cycbuffs/6:8388608
cycbuff:7:/var/spool/news/cycbuffs/7:8388608
metacycbuff:0:0,1,2,3,4,5,6,7:SEQUENTIAL
--8<---------------cut here---------------end--------------->8---

note the "SEQUENTIAL" line - this means that all buffers except one are
"fully rotated" (if you know what I mean) i.e.:

--8<---------------cut here---------------start------------->8---
sudo -u news /usr/lib/news/bin/cnfsstat   
Class 0 for groups matching "*"
 Buffer 0, size:  8.00 GBytes, position:   272 kBytes  7.00 cycles
  Newest: 2022-01-29  4:39:43,  296 days,  4:06:37 ago
 Buffer 1, size:  8.00 GBytes, position:   272 kBytes  7.00 cycles
  Newest: 2022-01-29  5:43:00,  296 days,  3:03:20 ago
 Buffer 2, size:  8.00 GBytes, position:   272 kBytes  7.00 cycles
  Newest: 2022-01-29  6:24:13,  296 days,  2:22:07 ago
 Buffer 3, size:  8.00 GBytes, position:   272 kBytes  7.00 cycles
  Newest: 2022-05-20 17:01:09,  184 days, 16:45:11 ago
 Buffer 4, size:  8.00 GBytes, position:  3.72 GBytes  6.47 cycles
  Newest: 2022-11-21  8:45:05,    0 days,  0:01:15 ago
 Buffer 5, size:  8.00 GBytes, position:   272 kBytes  6.00 cycles
  Newest: 2022-01-29  1:50:59,  296 days,  6:55:21 ago
 Buffer 6, size:  8.00 GBytes, position:   272 kBytes  6.00 cycles
  Newest: 2022-01-29  2:56:54,  296 days,  5:49:26 ago
 Buffer 7, size:  8.00 GBytes, position:   272 kBytes  6.00 cycles
  Newest: 2022-01-29  3:50:42,  296 days,  4:55:38 ago
--8<---------------cut here---------------end--------------->8---

(please ignore the dates, this spool was recreated from archive/backup)

So I think problem is with "makehistory" when comes to "fully rotated"
cycbuff.

KJ

-- 
http://wolnelektury.pl/wesprzyj/teraz/
-- 
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers