How does GDB load source line information during dlopen calls

Hi all,
First time poster to this list - so please bear with me :)
I am currently working on a project that has a custom dynamic loader for
shared libraries in the ELF format - it does NOT use dlopen. When debugging
this dynamically loaded library, I am hoping to be able to single step
through the C source code of functions in the library in tui mode.

If I had use dlopen this would work fine, but this does not work using the
custom dynamic loader. I am able to get the libraries symbols loaded with
"add-symbol-file". However the src line information is still not working.
For example "list someFunctionInLibrary" prints nothing and "layout src"
shows a blank screen

I am hoping for any information or suggestions. Is there some way to hint
to gdb to load the source line information from a particular file at a
particular address?

(Also please let me know if there is a better venue for the question).

Thanks!
Shravan