Re: Got write_block_to_dev error on device
"Martin Juhl Prendergast" <[email protected]>
| Newsgroups | gmane.comp.bacula.user |
|---|---|
| Message-ID | <[email protected]> |
Hi Arno
Traceback is inserted below..
Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue..
I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes..
Regards
If you need any more info, please
Check the log files for more information.
Please install a debugger (gdb) to receive a traceback.
Attempt to dump locks
threadid=0x7fc5477fe640 max=1 current=-1
threadid=0x7fc564ff9640 max=1 current=-1
threadid=0x7fc5467fc640 max=1 current=-1
threadid=0x7fc547fff640 max=2 current=-1
threadid=0x7fc5457fa640 max=1 current=-1
threadid=0x7fc545ffb640 max=2 current=-1
threadid=0x7fc59d80e640 max=1 current=-1
threadid=0x7fc59e00f640 max=2 current=-1
threadid=0x7fc59e810640 max=0 current=-1
threadid=0x7fc59f1ff640 max=0 current=-1
threadid=0x7fc5a000bf40 max=1 current=-1
Attempt to dump current JCRs. njcrs=3
threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01
use_count=1 killable=0
JobType=I JobLevel=
sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30
end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0
threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45
use_count=2 killable=1
JobType=B JobLevel=F
sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35
end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
db=0x7fc570027348 db_batch=(nil) batch_started=0
wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1
BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true
cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16
RWLOCK=0x7fc570027360 w_active=0 w_wait=0
threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00
use_count=1 killable=0
JobType=U JobLevel=
sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36
end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0
List plugins. Hook count=0
On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <[email protected]> wrote:
> Hi Martin,
>
> Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast:
> > Using Debug on bacula-dir, I get this:
> >
> > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25)
> > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan
> > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation
>
> Definitely deserves a thorough investigation. It's unlilkely to be
> caused by configuration.
>
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958]
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked...
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback
>
> That file will now be relevant.
>
> /var/spool/bacula/bacula.2464062.traceback should contain information
> the developers can work with.
>
> Also interesting will be to know where you installed the packages from,
> or how you built the software.
>
> Cheers,
>
> Arno
>
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director.
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected
> >
> >
> > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <[email protected]> wrote:
> >
> >> Also.. on the server running bacula/bacularis I get:
> >>
> >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000]
> >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping.
> >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000]
> >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0)
> >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00
> >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000]
> >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0)
> >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
> >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000]
> >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000]
> >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping.
> >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000]
> >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000]
> >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0)
> >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
> >>
> >>
> >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <[email protected]> wrote:
> >>
> >>> Oh, I can see that..
> >>>
> >>> The storage daemon says:
> >>>
> >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
> >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director.
> >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
> >>>
> >>> Config:
> >>>
> >>> Director {
> >>> Name = "bacula-dir"
> >>> Password = "@@SD_PASSWORD@@"
> >>> }
> >>> Director {
> >>> Name = "bacula-mon"
> >>> Password = "@@MON_SD_PASSWORD@@"
> >>> Monitor = yes
> >>> }
> >>> Storage {
> >>> Name = "bacula-sd"
> >>> WorkingDirectory = "/var/spool/bacula"
> >>> PidDirectory = "/var/run"
> >>> PluginDirectory = "/usr/lib64/bacula"
> >>> MaximumConcurrentJobs = 20
> >>> }
> >>> Device {
> >>> Name = "AlwaysIncrement"
> >>> Description = ""
> >>> MediaType = "AlwaysIncrement"
> >>> DeviceType = "File"
> >>> ArchiveDevice = "/home/bacula/autoincrement"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> Autochanger = no
> >>> ReadOnly = no
> >>> MaximumConcurrentJobs = 5
> >>> DriveIndex = 0
> >>> }
> >>> Device {
> >>> Name = "FileChgr1-Dev1"
> >>> MediaType = "File1"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Device {
> >>> Name = "FileChgr1-Dev2"
> >>> MediaType = "File1"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Device {
> >>> Name = "FileChgr2-Dev1"
> >>> MediaType = "File2"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Device {
> >>> Name = "FileChgr2-Dev2"
> >>> MediaType = "File2"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Messages {
> >>> Name = "Standard"
> >>> Director = bacula-dir = All
> >>> }
> >>> Autochanger {
> >>> Name = "FileChgr1"
> >>> Device = "FileChgr1-Dev1"
> >>> Device = "FileChgr1-Dev2"
> >>> ChangerDevice = "/dev/null"
> >>> ChangerCommand = ""
> >>> }
> >>> Autochanger {
> >>> Name = "FileChgr2"
> >>> Device = "FileChgr2-Dev1"
> >>> Device = "FileChgr2-Dev2"
> >>> ChangerDevice = "/dev/null"
> >>> ChangerCommand = ""
> >>> }
> >>> Device {
> >>> DeviceType = "File"
> >>> RemovableMedia = no
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> MaximumConcurrentJobs = 5
> >>> RandomAccess = yes
> >>> Name = "Consolidate"
> >>> Description = ""
> >>> DriveIndex = 0
> >>> ArchiveDevice = "/home/bacula/consolidate"
> >>> MediaType = "Consolidate"
> >>> ReadOnly = no
> >>> Autochanger = no
> >>> }
> >>>
> >>> Please say if I need to provide more configuration
> >>>
> >>> /Martin
> >>>
> >>>
> >>> Martin,
> >>>
> >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says".
> >>>
> >>> Regards,
> >>> Robert Gerber
> >>> 402-237-8692
> >>> [email protected]
> >>>
> >>>
> >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <[email protected]> wrote:
> >>>
> >>> Hi guys
> >>>
> >>> Hope someone can help me..
> >>>
> >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10..
> >>>
> >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup.
> >>>
> >>> On the hosts I get:
> >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted.
> >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer
> >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103
> >>>
> >>>
> >>>
> >>> The Storage daemon says
> >>>
> >>>
> >>> _______________________________________________
> >>> Bacula-users mailing list
> >>> [email protected]
> >>> https://lists.sourceforge.net/lists/listinfo/bacula-users
> >
> >
> >
> > _______________________________________________
> > Bacula-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
>
> --
> Arno Lehmann
>
> IT-Service Lehmann
> Sandstr. 6, 49080 Osnabrück
>
>
>
> _______________________________________________
> Bacula-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users