Re: Peculiar Win/11 Behavior?

Hi Kurt,

I suppose you might be correct. Odd though, I do not allocate any memory that should migrate to highest level generations. Mostly scratch bindings of complex number values and occasional transient vectors for plotting. None of those have any substantial lifetime.

One other possibility arises… I just discovered that M$’s SetThreadExecutionState(), used for preventing the computer going to sleep while a thread is alive, now has additional gradations of “sleep”. So I’m using a new one (to me) called ES_AWAYMODE_REQUIRED. That may help a bit, as their documentation talks about multi-media services needing this while running in the background.

I migrated this SetThreadExecutionState down into the C thread that gets spawned on live audio init to talk with the MMIO, instead of the Lisp layer. We’ll see in about 8 hours…



> On Jan 17, 2025, at 22:48, Kurt Geisel <[email protected]> wrote:
> 
> Thanks, David. Apologies for being slow on this. This adds a lot of clarity.
>  
> My “last suggestion” is, if you do believe that some your Lisp process’ reference set is getting paged out, you should consider what is giving the system memory pressure. Unless there is a request to trim working set, one normally expects it to happen if the system thinks it’s under pressure. I suppose this could be some memory intensive system process that runs periodically. I don’t know much about Lispworks’ particular GC bag of tricks, but some basic GC approaches cause the working set to grow before it shrinks (e.g., copying live objects during mark-and-sweep or stop-and-copy). Even with occasional edge-of-threshold effects, it’s conceivable that your frequent manual GC requests at unlucky times cause your working set to temporarily compete more with whatever else is going on.
>  
> - Kurt
>  
> From: David McClain <[email protected]>
> Sent: Friday, January 17, 2025 6:51 PM
> To: David McClain <[email protected]>
> Cc: Kurt Geisel <[email protected]>; Toomas Altosaar <[email protected]>; Yuri Davidovsky <[email protected]>; Lisp HUG <[email protected]>
> Subject: Re: Peculiar Win/11 Behavior?
>  
> More properly stated, what I measure is the time interval between successive low-level buffer requests. That measures the runtime of the Lisp layer just above the C I/O routines, but nothing about those C level routines is being measured. This timing is the duration of conversion to double-float, channel separation, telemetry computation and audio encoding, recording of the augmented buffer with telemetry sub-band, and then 3 passes of heterodyning + decimation.
>  
> What I do measure about the C layer is number of samples dropped between successive buffer requests from Lisp. I have to assume that the samples are coming in at 8 kHz. There is no time tagging, or sample id labeling on these MMIO buffers. They are simply presented to me when they are full. It is then up to me to enqueue them back to MMIO when I finish with their contents.
>  
> The MMIO buffers in the C layer have been allocated and locked down, to prevent them from being paged. But my Lisp code has not been protected like this. And that Lisp code is what I see being paged out, and delayed back in.
>  
> Similarly, when the analysis routines in Lisp are triggered, once every 15 sec, I measure the interval from data arrival to the next data request, which covers all the frequency analysis routines, the plotting, and waterfall display. But I’m not measuring the interval of how long it takes for the data to arrive. That would be roughly 15-16 sec.
> 
> 
> On Jan 17, 2025, at 18:41, David McClain (as dbm at refined-audiometrics dot com) <[email protected] <mailto:[email protected]>> wrote:
>  
> Hi Kurt,
>  
> No, I’m doing live capture of audio. But I also have the facility to replay recordings of prior sessions. I capture on the Win/11 machine, then review on an M1 iMac in the other room.
>  
> The problems arise with live capture, not with recording or playback. And in playback, I don’t have to respond in real time. I can take as long as I need in between successive buffer grabs.
>  
> This is a time sensitive system, but not like any I have worked on before. This thing has a typical inter-frame time of 0.5s. And a full analysis window of 15s.  Each MMIO buffer is 4096 (stereo) samples long, and I decided today to allocate and lock down 6 of these MMIO buffers. The logic in my mind has at most, 1 in play with Lisp requestor, 2 in FIFO waiting for Lisp request, 1 in use by MMIO at any instant, and 2 backup buffers ready for MMIO when it finishes the one it is working on now.
>  
> Up above, in Lisp, my main unit of information is a buffer of 512 samples. And I take these MMIO buffers handed to the Lisp process on request, add some telemetry in a 3-4 kHz sub-band, record the buffer to disk, convert the samples to double-float, left and right channels separated. Then 3 passes of heterodyne + decimation into a final 512 sample buffer. Then the MMIO buffer is returned to the C code to await or grab the next physical buffer. 3 passes of heterodyne and downsample because my SOI is at 1500 Hz, while my two reference carriers are at 99.5 Hz and 2900.5 Hz (SOI ± 1400.5 Hz). 
>  
> That’s for the left channel. On the right I have IRIG-B coming in for time tagging, and I send the right buffer over to the IRIG-B reader. 
>  
> The aim here is to accumulate decimated data buffers such that any two adjacent buffers are phase continuous. To pull that off, I can’t allow any incoming buffers to have a dropout. Above these half-buffers, we have Lisp analysis client code which needs two adjacent, phase continuous, half buffers of 512 decimated samples. It is okay if the upper level analysis can’t complete in time for a solid continuous readout, there will be no phase discontinuities visible to it. (so long as the low level has no dropouts).
>  
> The low-level buffer filling takes about 512 ms at 8 kHz sample rate. The recording, telemetry gen, heterodyning and decimation take about 4-20 ms. The upper level analysis takes on average about 200-400 ms to present a graph and a waterfall display.
>  
> When things go bonkers, I see instances where the low-level Lisp - C interface, normally 4-20 ms, took 3.5s, thereby losing some valid buffers, and causing a severe phase discontinuity. Stopping the whole thing, shutting down MMIO, and restarting, does not fix the slippage problem. The only thing that fixes anything is to exit the Lisp process completely, shutting down the MMIO altogether, then firing up the Lisp from code start all over again. Then we will be good for another 6-8 hours.
>  
> It really looks like Win/11 decides that my working set of memory pages need to get swapped to disk, and it takes its time restoring them for a continuation of what had been a continuous regular process with little variation.
>  
> I really detest these computer companies making so many decisions about what is best for us…
>  
>  
> 
> 
> On Jan 17, 2025, at 16:47, Kurt Geisel (as kgeisel at outlook dot com) <[email protected] <mailto:[email protected]>> wrote:
>  
> OH! When I heard MMIO my first thought was reading multimedia files but dismissed this interpretation because you were talking about the device. I thought you were doing direct I/O with the device (say, in real time). But am I to understand you are reading already-captured WAV files? So, one might expect the device doesn’t matter (unless it’s ADC problems we’re observing).
>  
> So perhaps my suggestion is less practical—if you have to mmioClose part way through, you would have to re-seek to where you were. Perhaps you are processing very large files.
>  
> Could you keep WAV file reading in the Lisp realm (riff-wave [https://github.com/eugeneia/riff-wave] sounds like it might not be robust enough for all the format variations one might encounter—and you’ve already mentioned fake sample rates)?
>  
> I guess form your description of a continuous time-sensitive process, perhaps you are trying to read a WAV file concurrently with it being written? Hmmm. I might think about replacing the device with a test process that writes a continuous WAV file with a known repeating pattern, then perhaps you can rig it to detect when you observe corruption. Then, you might catch mmio “red-handed” or narrow down where it is going wrong. Another, more involved approach to isolate mmio would be to have two consumer processes—one running in Lisp and one a dumb harness around your C code. They both just write a data stream that you can later compare to see if was only the Lisp version that observed corruption. If both observe it, I think you pretty much painted mmio into a corner.
>  
> - Kurt
>  
> From: David McClain <[email protected] <mailto:[email protected]>>
> Sent: Friday, January 17, 2025 3:08 PM
> To: David McClain <[email protected] <mailto:[email protected]>>
> Cc: Kurt Geisel <[email protected] <mailto:[email protected]>>; Toomas Altosaar <[email protected] <mailto:[email protected]>>; Yuri Davidovsky <[email protected] <mailto:[email protected]>>; Lisp HUG <[email protected] <mailto:[email protected]>>
> Subject: Re: Peculiar Win/11 Behavior?
>  
> When I say that I restart the app from Lisp, I mean that the MMIO stuff is closed out, then reopened with fresh buffers, threads, etc. The whole MMIO should be back to init state. 
>  
> Getting to a point where a debugger could be used is very tedious. It takes typically 6-8 hours for something interesting (bad) to happen.
>  
> I have gone through the Lisp with a fine-toothed comb looking for problems. So far, I’m not seeing any.  But that shouldn’t be too surprising - how often does the Fox smell its own hole?
>  
>  
>  
> 
> 
> 
> On Jan 17, 2025, at 16:00, David McClain (as dbm at refined-audiometrics dot com) <[email protected] <mailto:[email protected]>> wrote:
>  
> Here is my code. I’m using plain old MMIO - the very oldest API that M$ defined. 
>  
> WASAPI? Oh dear… I was hoping to avoid diving too deeply into Windows. I have an allergic reaction whenever I get too deeply into Windows or VMS code. 
>  
> The WAV file stuff is at the top, the realtime acquisition is in the middle, and some high-perf subroutines for Lisp are at the end.
>  
> <lispwave.cpp>
>  
> 
> 
> 
> On Jan 17, 2025, at 15:37, Kurt Geisel <[email protected] <mailto:[email protected]>> wrote:
>  
> I'm still not sure which specific API you are using. For example, if using WASAPI, you could try to make two test runs separated by IAudioClient::Stop() and see if that alone recovers from your phase jitter. If not, you can incrementally tear down more things (e.g., Release() all the interfaces) between the two runs until you've found the suspect. If you can't find any way to recover from within the Lisp process, this might not tell you much more than what it is *not* :-/.
> 
> - Kurt
> 
> -----Original Message-----
> From: [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> On Behalf Of David McClain (as dbm at refined-audiometrics dot com)
> Sent: Friday, January 17, 2025 2:10 PM
> To: Toomas Altosaar <[email protected] <mailto:[email protected]>>
> Cc: Yuri Davidovsky <[email protected] <mailto:[email protected]>>; Lisp HUG <[email protected] <mailto:[email protected]>>
> Subject: Re: Peculiar Win/11 Behavior?
> 
> GC should not be an issue. Every 15 sec or so I make a call to (HCL:GC-GENERATION T).
> 
> And yes, I am using Multi-media subsystem (MMIO). I can open the incoming A/D and request a sample rate of 8 kHz, even though it is happening physically at 48 kHz. There is nothing in the signal above 3 kHz. So, no matter how sloppily M$ pulls off that sample rate conversion, I’ll be okay.
> 
> (That’s another reason I hesitate to use a Mac here. I don’t believe the Mac will accommodate the fake 8 kHz sample rate.
> 
> The Windows machine also hooks up to the plethora of RS-232 Serial I/O ports needed by other equipment and software. Yes, EE’s seem to be heavily wedded to Windows PC’s and Serial I/O - now done with USB interfaces.)
> 
> After this experience, I’m finding that in many ways the M1 iMac and the Windows/11 AMD Ryzen 5 5600G with Radeon Graphics at 3.9 GHz, are both remarkably the same effective compute speed.
> 
> 
> 
> 
> 
> 
> On Jan 17, 2025, at 14:36, Toomas Altosaar (as toomas dot altosaar at gmail dot com) <[email protected] <mailto:[email protected]>> wrote:
> 
> Garbage collection came to mind as well: make a substantial amount of garbage, then time how long (gc) takes to run (or whatever a full gc is called in LW). David: are you running with ephemeral gc enabled?
> 
> 
> 
> On 17. Jan 2025, at 23.25, Yuri Davidovsky (as work at disclosure dot ie) <[email protected] <mailto:[email protected]>> wrote:
> 
> 
> Could that be the garbage collector secretly trying to stab you in the back?
> 
> 
> 
> On 17 Jan 2025, at 20:21, David McClain (as dbm at refined-audiometrics dot com) <[email protected] <mailto:[email protected]>> wrote:
> 
> So, theoretically, I can stretch 3 buffer periods in bursts, so long as I catch up on average to less than one buffer period. My processing takes 4-20 ms.
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users
> [email protected] <mailto:[email protected]>
> http://www.l/
> ispworks.com <http://ispworks.com/>%2Fsupport%2Flisp-hug.html&data=05%7C02%7C%7Cabe57c1f8fd8
> 4aa48b5508dd3743ecac%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638
> 727487075972181%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiO
> iIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7
> C%7C%7C&sdata=2Lw%2Bp8c8gZnSK90AupMahRLiiucEur3SPkMozbczasI%3D&reserv
> ed=0
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users [email protected] <mailto:[email protected]>
> http://www.li/
> spworks.com <http://spworks.com/>%2Fsupport%2Flisp-hug.html&data=05%7C02%7C%7Cabe57c1f8fd84a
> a48b5508dd3743ecac%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638727
> 487075991088%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL
> jAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%
> 7C&sdata=03RQeVZNuGMNRmDFffgE4aIu6N6i8M03glGHqyIMyIk%3D&reserved=0
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users [email protected] <mailto:[email protected]>
> http://www.lispworks.com/support/lisp-hug.html