Re: Still seeing Failed assertion: "p[i] == 0" on armv7 buildworld [Notes from another example core dump: #5] [Common range #0..#10 __je_pa_alloc]
Mark Millard <[email protected]>
| Newsgroups | gmane.os.freebsd.current,gmane.os.freebsd.devel.arm |
|---|---|
| Message-ID | <[email protected]> |
On Nov 19, 2025, at 09:53, Mark Millard <[email protected]> wrote: > > On Nov 18, 2025, at 22:10, Mark Millard <[email protected]> wrote: > >> I'm only sending notes from testing of how similar other failures appear >> to the 2 lists. Folks can ask that I do otherwise for them if they want. >> >> This one also does not have area_malloc involved at all. >> >> This one is for size 8192 (2 pages). It looks like #0..#10 are similar to >> the prior reports. #10 is __je_pa_alloc. >> >> #11 is: arena_slab_alloc >> >> (gdb) bt >> #0 thr_kill () at thr_kill.S:4 >> #1 0x2a08ef24 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:48 >> #2 0x2a145f38 in abort () at /usr/src/lib/libc/stdlib/abort.c:61 >> #3 0x2a196128 in ehooks_debug_zero_check (addr=addr@entry=0x2e12b000, size=size@entry=8192) at /usr/src/contrib/jemalloc/include/jemalloc/internal/ehooks.h:170 >> #4 0x2a191f60 in ehooks_alloc (tsdn=0x2a2e4060, ehooks=0x2a600080, new_addr=0x0, size=<optimized out>, alignment=4096, zero=0xffff9747, commit=<optimized out>) >> at /usr/src/contrib/jemalloc/include/jemalloc/internal/ehooks.h:208 >> #5 __je_extent_alloc_wrapper (tsdn=tsdn@entry=0x2a2e4060, pac=0x2a601810, ehooks=<optimized out>, new_addr=<optimized out>, size=8192, alignment=4096, zero=true, commit=0xffff97a7, >> growing_retained=<optimized out>) at jemalloc_extent.c:1003 >> #6 0x2a1916e0 in __je_ecache_alloc_grow (tsdn=<optimized out>, tsdn@entry=0x2a2e4060, pac=pac@entry=0x2a601810, ehooks=ehooks@entry=0x2a600080, ecache=<optimized out>, ecache@entry=0x2a603dd0, >> expand_edata=0x0, size=8192, alignment=4096, zero=<optimized out>, guarded=<optimized out>) at jemalloc_extent.c:126 >> #7 0x2a1c9680 in pac_alloc_real (tsdn=0x2a2e4060, pac=0x2a601810, ehooks=0x2a600080, size=8192, alignment=4096, zero=<optimized out>, guarded=false) at jemalloc_pac.c:124 >> #8 pac_alloc_impl (tsdn=tsdn@entry=0x2a2e4060, self=0x2a601810, size=size@entry=8192, alignment=4096, zero=<optimized out>, guarded=false, frequent_reuse=<optimized out>, >> deferred_work_generated=<optimized out>) at jemalloc_pac.c:178 >> #9 0x2a1c7ae8 in pai_alloc (tsdn=0x2a2e4060, self=0x0, size=8192, alignment=2147483615, zero=<optimized out>, guarded=false, frequent_reuse=true, deferred_work_generated=<optimized out>) >> at /usr/src/contrib/jemalloc/include/jemalloc/internal/pai.h:43 >> #10 __je_pa_alloc (tsdn=tsdn@entry=0x2a2e4060, shard=shard@entry=0x2a601800, size=8192, alignment=<optimized out>, slab=true, szind=35, zero=<optimized out>, guarded=false, >> deferred_work_generated=0xffff986f) at jemalloc_pa.c:139 >> #11 0x2a16b9f8 in arena_slab_alloc (tsdn=tsdn@entry=0x2a2e4060, arena=0x2a600540, binind=35, binshard=0, bin_info=0x2a2200ec <__je_bin_infos+1680>) at jemalloc_arena.c:839 >> #12 0x2a16ac98 in __je_arena_cache_bin_fill_small (tsdn=0x2a2e4060, arena=0x2a600540, cache_bin=cache_bin@entry=0x2a2e4618, cache_bin_info=0x2a600506, binind=35, nfill=10) at jemalloc_arena.c:1034 >> #13 0x2a1b5694 in __je_tcache_alloc_small_hard (tsdn=0x0, tsdn@entry=0x2a2e4060, arena=0x0, arena@entry=0x2a600540, tcache=tcache@entry=0x2a2e42c8, cache_bin=cache_bin@entry=0x2a2e4618, binind=35, >> tcache_success=0xffff991f) at jemalloc_tcache.c:238 >> #14 0x2a16cef4 in tcache_alloc_small (tsd=<optimized out>, arena=0x2a600540, tcache=0x2a2e42c8, size=<optimized out>, binind=35, zero=false, slow_path=true) >> at /usr/src/contrib/jemalloc/include/jemalloc/internal/tcache_inlines.h:68 >> #15 arena_malloc (tsdn=<optimized out>, arena=<optimized out>, size=8192, ind=35, zero=false, tcache=0x2a2e42c8, slow_path=true) >> at /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:151 >> #16 0x2a16cb88 in __je_arena_palloc (tsdn=0x0, tsdn@entry=0x2a2e4060, arena=<optimized out>, usize=<optimized out>, usize@entry=8192, alignment=alignment@entry=8, zero=false, tcache=0x2a2e42c8) >> at jemalloc_arena.c:1224 >> #17 0x2a16559c in ipallocztm (tsdn=0x2a2e4060, tsdn@entry=0x2a2e42c8, usize=8192, alignment=8, zero=false, tcache=0x2a2e42c8, is_internal=false, arena=0x0) >> at /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h:80 >> #18 ipalloct (tsdn=0x0, tsdn@entry=0x2a2e4060, usize=8192, alignment=8, zero=false, tcache=0x2a2e42c8, arena=0x0) >> at /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h:91 >> #19 0x2a1651f4 in imalloc_no_sample (sopts=0xffff9a14, dopts=0xffff99f4, tsd=0x2a2e4060, size=8192, usize=8192, ind=<optimized out>) at jemalloc_jemalloc.c:2398 >> #20 imalloc_body (sopts=0xffff9a14, dopts=0xffff99f4, tsd=0x2a2e4060) at jemalloc_jemalloc.c:2577 >> #21 0x2a156188 in imalloc (sopts=sopts@entry=0xffff9a14, dopts=<optimized out>, dopts@entry=0xffff99f4) at jemalloc_jemalloc.c:2693 >> #22 0x2a15677c in __aligned_alloc (alignment=8, size=8192) at jemalloc_jemalloc.c:2821 >> #23 0x29e61a00 in std::__1::__libcpp_aligned_alloc[abi:se190107](unsigned int, unsigned int) (__alignment=8, __size=<optimized out>) >> at /usr/src/contrib/llvm-project/libcxx/include/__memory/aligned_alloc.h:43 >> #24 operator_new_aligned_impl (size=<optimized out>, alignment=8) at /usr/src/contrib/llvm-project/libcxx/src/new.cpp:129 >> #25 operator new (size=<optimized out>, alignment=<optimized out>) at /usr/src/contrib/llvm-project/libcxx/src/new.cpp:141 >> #26 0x20ff35f8 in Allocate () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/AllocatorBase.h:92 >> #27 StartNewSlab () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/Allocator.h:344 >> #28 AllocateSlow () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/Allocator.h:200 >> #29 0x26c0ab48 in Allocate () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/Allocator.h:176 >> #30 Allocate () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/Allocator.h:214 >> #31 operator new<llvm::MallocAllocator, 4096U, 4096U, 128U> () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/Allocator.h:448 >> #32 getMachineMemOperand () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/MachineFunction.cpp:496 >> #33 0x2705c62c in getStore () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp:9015 >> #34 0x270941b4 in visitStore () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp:4746 >> #35 0x2708d80c in visit () at /usr/src/contrib/llvm-project/llvm/include/llvm/IR/Instruction.def:173 >> #36 0x2708c9e4 in visit () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp:1346 >> #37 0x270f53c8 in SelectBasicBlock () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:838 >> #38 0x270f4b84 in SelectAllBasicBlocks () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:1863 >> #39 0x270f1f24 in runOnMachineFunction () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:631 >> #40 0x28300224 in runOnMachineFunction () at /usr/src/contrib/llvm-project/llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp:70 >> #41 0x270efd6c in runOnMachineFunction () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:374 >> #42 0x26c15e88 in runOnFunction () at /usr/src/contrib/llvm-project/llvm/lib/CodeGen/MachineFunctionPass.cpp:94 >> #43 0x276a9e74 in runOnFunction () at /usr/src/contrib/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1440 >> #44 0x276b0d40 in runOnModule () at /usr/src/contrib/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1486 >> #45 0x276aa5e0 in runOnModule () at /usr/src/contrib/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1555 >> --Type <RET> for more, q to quit, c to continue without paging-- >> #46 run () at /usr/src/contrib/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:541 >> #47 0x2216d2e8 in RunCodegenPipeline () at /usr/src/contrib/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1157 >> #48 EmitAssembly () at /usr/src/contrib/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1180 >> #49 EmitBackendOutput () at /usr/src/contrib/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1341 >> #50 0x225cbca0 in HandleTranslationUnit () at /usr/src/contrib/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:354 >> #51 0x22cff8e4 in ParseAST () at /usr/src/contrib/llvm-project/clang/lib/Parse/ParseAST.cpp:184 >> #52 0x22b5a7b8 in Execute () at /usr/src/contrib/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1078 >> #53 0x22adb800 in ExecuteAction () at /usr/src/contrib/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1061 >> #54 0x22bf6a90 in ExecuteCompilerInvocation () at /usr/src/contrib/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:280 >> #55 0x0002afc8 in cc1_main () at /usr/src/contrib/llvm-project/clang/tools/driver/cc1_main.cpp:284 >> #56 0x00038548 in ExecuteCC1Tool () at /usr/src/contrib/llvm-project/clang/tools/driver/driver.cpp:215 >> #57 0x227877ec in operator() () at /usr/src/contrib/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68 >> #58 operator() () at /usr/src/contrib/llvm-project/clang/lib/Driver/Job.cpp:440 >> #59 callback_fn<(lambda at /usr/src/contrib/llvm-project/clang/lib/Driver/Job.cpp:440:22)>(void) () at /usr/src/contrib/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45 >> #60 0x27d88624 in operator() () at /usr/src/contrib/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68 >> #61 RunSafely () at /usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:426 >> #62 0x22786e90 in Execute () at /usr/src/contrib/llvm-project/clang/lib/Driver/Job.cpp:440 >> #63 0x22748074 in ExecuteCommand () at /usr/src/contrib/llvm-project/clang/lib/Driver/Compilation.cpp:199 >> #64 0x227483d0 in ExecuteJobs () at /usr/src/contrib/llvm-project/clang/lib/Driver/Compilation.cpp:253 >> #65 0x22765bb8 in ExecuteCompilation () at /usr/src/contrib/llvm-project/clang/lib/Driver/Driver.cpp:1943 >> #66 0x00037ba4 in clang_main () at /usr/src/contrib/llvm-project/clang/tools/driver/driver.cpp:391 >> #67 0x000363a8 in main () at /usr/src/usr.bin/clang/clang/clang-driver.cpp:17 >> >> >> >> 0x2e12afd0: 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 >> 0x2e12afe0: 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 >> 0x2e12aff0: 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 >> (gdb) x /1024x ((size_t*)addr)+0 >> 0x2e12b000: 0x00000000 0x00000000 0x00000000 0x00000000 >> 0x2e12b010: 0x00000000 0x00000000 0x00000000 0x00000000 >> 0x2e12b020: 0x00000000 0x00000000 0x00000000 0x00000000 >> . . . >> 0x2e12b650: 0x00000000 0x00000000 0x00000000 0x00000000 >> 0x2e12b660: 0x00000000 0x00000000 0x00000000 0x00000000 >> 0x2e12b670: 0x00000000 0x00000000 0x00000000 0x00000000 >> 0x2e12b680: 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a >> 0x2e12b690: 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a >> 0x2e12b6a0: 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a >> . . . >> 0x2e12cfd0: 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a >> 0x2e12cfe0: 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a >> 0x2e12cff0: 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a 0x5a5a5a5a >> (gdb) x /1024x ((size_t*)addr)+2048 >> 0x2e12d000: Cannot access memory at address 0x2e12d000 >> >> >> >> For #0..#10: The prior examples and the above >> agree about: >> >> #5 __je_extent_alloc_wrapper zero=true >> >> But also there was in this example: >> #14 tcache_alloc_small zero=false >> >> (The others before #14 are optimized out.) >> >> >> So summarizing some of the failure results so far . . . >> >> The common part of the call chain is: >> >> #0 thr_kill () at thr_kill.S:4 >> #1 __raise (s=6) at /usr/src/lib/libc/gen/raise.c:48 >> #2 abort () at /usr/src/lib/libc/stdlib/abort.c:61 >> #3 ehooks_debug_zero_check at /usr/src/contrib/jemalloc/include/jemalloc/internal/ehooks.h:170 >> #4 ehooks_alloc at /usr/src/contrib/jemalloc/include/jemalloc/internal/ehooks.h:208 >> #5 __je_extent_alloc_wrapper at jemalloc_extent.c:1003 (argument zero=true) >> #6 __je_ecache_alloc_grow at jemalloc_extent.c:126 >> #7 pac_alloc_real at jemalloc_pac.c:124 >> #8 pac_alloc_impl at jemalloc_pac.c:178 >> #9 pai_alloc at /usr/src/contrib/jemalloc/include/jemalloc/internal/pai.h:43 >> #10 __je_pa_alloc at jemalloc_pa.c:139 >> Note: some #11+ can show arguments with a zero=false >> >> All the non-zero Bytes in the pages being checked are 0x5a bytes. >> The zero Bytes (if any) come first so far. > > By no means do I know if the below is the actual problem. > > > The one place I see zero possibly becoming true for > __je_extent_alloc_wrapper by the time of the backtrace > is via: > > void * > extent_alloc_mmap(void *new_addr, size_t size, size_t alignment, bool *zero, > bool *commit) { > assert(alignment == ALIGNMENT_CEILING(alignment, PAGE)); > void *ret = pages_map(new_addr, size, alignment, commit); > if (ret == NULL) { > return NULL; > } > assert(ret != NULL); > if (*commit) { > *zero = true; > } > return ret; > } > > (So pages_map behavior vs. commit usage is relevant. > See jemalloc/src/pages.c .) > > extent_alloc_mmap called via: > > /* > * If the caller specifies (!*zero), it is still possible to receive zeroed > * memory, in which case *zero is toggled to true. arena_extent_alloc() takes > * advantage of this to avoid demanding zeroed extents, but taking advantage of > * them if they are returned. > */ > static void * > extent_alloc_core(tsdn_t *tsdn, arena_t *arena, void *new_addr, size_t size, > size_t alignment, bool *zero, bool *commit, dss_prec_t dss_prec) { > void *ret; > > assert(size != 0); > assert(alignment != 0); > > /* "primary" dss. */ > if (have_dss && dss_prec == dss_prec_primary && (ret = > extent_alloc_dss(tsdn, arena, new_addr, size, alignment, zero, > commit)) != NULL) { > return ret; > } > /* mmap. */ > if ((ret = extent_alloc_mmap(new_addr, size, alignment, zero, commit)) > != NULL) { > return ret; > } > /* "secondary" dss. */ > if (have_dss && dss_prec == dss_prec_secondary && (ret = > extent_alloc_dss(tsdn, arena, new_addr, size, alignment, zero, > commit)) != NULL) { > return ret; > } > > /* All strategies for allocation failed. */ > return NULL; > } > > in jemalloc/src/ehooks.c > > called via: > > void * > ehooks_default_alloc_impl(tsdn_t *tsdn, void *new_addr, size_t size, > size_t alignment, bool *zero, bool *commit, unsigned arena_ind) { > arena_t *arena = arena_get(tsdn, arena_ind, false); > /* NULL arena indicates arena_create. */ > assert(arena != NULL || alignment == HUGEPAGE); > dss_prec_t dss = (arena == NULL) ? dss_prec_disabled : > (dss_prec_t)atomic_load_u(&arena->dss_prec, ATOMIC_RELAXED); > void *ret = extent_alloc_core(tsdn, arena, new_addr, size, alignment, > zero, commit, dss); > if (have_madvise_huge && ret) { > pages_set_thp_state(ret, size); > } > return ret; > } > > in jemalloc/src/ehooks.c > > called via (so is ehooks_debug_zero_check): > > static inline void * > ehooks_alloc(tsdn_t *tsdn, ehooks_t *ehooks, void *new_addr, size_t size, > size_t alignment, bool *zero, bool *commit) { > bool orig_zero = *zero; > void *ret; > extent_hooks_t *extent_hooks = ehooks_get_extent_hooks_ptr(ehooks); > if (extent_hooks == &ehooks_default_extent_hooks) { > ret = ehooks_default_alloc_impl(tsdn, new_addr, size, > alignment, zero, commit, ehooks_ind_get(ehooks)); > } else { > ehooks_pre_reentrancy(tsdn); > ret = extent_hooks->alloc(extent_hooks, new_addr, size, > alignment, zero, commit, ehooks_ind_get(ehooks)); > ehooks_post_reentrancy(tsdn); > } > assert(new_addr == NULL || ret == NULL || new_addr == ret); > assert(!orig_zero || *zero); > if (*zero && ret != NULL) { > ehooks_debug_zero_check(ret, size); > } > return ret; > } > > called via: > > edata_t * > extent_alloc_wrapper(tsdn_t *tsdn, pac_t *pac, ehooks_t *ehooks, > void *new_addr, size_t size, size_t alignment, bool zero, bool *commit, > bool growing_retained) { > witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn), > WITNESS_RANK_CORE, growing_retained ? 1 : 0); > > edata_t *edata = edata_cache_get(tsdn, pac->edata_cache); > if (edata == NULL) { > return NULL; > } > size_t palignment = ALIGNMENT_CEILING(alignment, PAGE); > void *addr = ehooks_alloc(tsdn, ehooks, new_addr, size, palignment, > &zero, commit); > . . . > There are just 2 calls to (__je_)pa_alloc, one of which pass false for zero ( jemalloc/src/arena.c ). The pattern used below also matches hpa_alloc but ignore those: # grep -r 'pa_alloc\>' /usr/src/contrib/jemalloc/src/ /usr/src/contrib/jemalloc/src/pa.c:pa_alloc(tsdn_t *tsdn, pa_shard_t *shard, size_t size, size_t alignment, /usr/src/contrib/jemalloc/src/hpa.c:static edata_t *hpa_alloc(tsdn_t *tsdn, pai_t *self, size_t size, /usr/src/contrib/jemalloc/src/hpa.c: shard->pai.alloc = &hpa_alloc; /usr/src/contrib/jemalloc/src/hpa.c: assert(self->alloc = &hpa_alloc); /usr/src/contrib/jemalloc/src/hpa.c:hpa_alloc(tsdn_t *tsdn, pai_t *self, size_t size, size_t alignment, bool zero, /usr/src/contrib/jemalloc/src/arena.c: edata_t *edata = pa_alloc(tsdn, &arena->pa_shard, esize, alignment, /usr/src/contrib/jemalloc/src/arena.c: edata_t *slab = pa_alloc(tsdn, &arena->pa_shard, bin_info->slab_size, The calling code looks like the below, where it is the arena_slab_alloc routine that passes false directly. Examples 1, 2, 3, and 5 that I sent to the lists have arena_slab_alloc as the caller of (__je_)pa_alloc. Yet they end up with #5 (__je_)extent_alloc_wrapper showing zero=true in the backtrace and ehooks_debug_zero_check being called (which requires ehooks_alloc to see its zero with the relevant value indicating true). That is my evidence for extent_alloc_mmap possibly causing ehooks_alloc to see a true for zero in its check for if it should call ehooks_debug_zero_check . For reference: edata_t * arena_extent_alloc_large(tsdn_t *tsdn, arena_t *arena, size_t usize, size_t alignment, bool zero) { bool deferred_work_generated = false; szind_t szind = sz_size2index(usize); size_t esize = usize + sz_large_pad; bool guarded = san_large_extent_decide_guard(tsdn, arena_get_ehooks(arena), esize, alignment); edata_t *edata = pa_alloc(tsdn, &arena->pa_shard, esize, alignment, /* slab */ false, szind, zero, guarded, &deferred_work_generated); . . . static edata_t * arena_slab_alloc(tsdn_t *tsdn, arena_t *arena, szind_t binind, unsigned binshard, const bin_info_t *bin_info) { bool deferred_work_generated = false; witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn), WITNESS_RANK_CORE, 0); bool guarded = san_slab_extent_decide_guard(tsdn, arena_get_ehooks(arena)); edata_t *slab = pa_alloc(tsdn, &arena->pa_shard, bin_info->slab_size, /* alignment */ PAGE, /* slab */ true, /* szind */ binind, /* zero */ false, guarded, &deferred_work_generated); . . . (I do have some more saved core dumps now, but I doubt publication would be all that useful: too similar to those already published.) === Mark Millard marklmi at yahoo.com