[PR] lavc/aacenc: encoder improvements (PR #23837)
Lynne via ffmpeg-devel <[email protected]> Fri, 17 Jul 2026 06:35:10 -0000
| Newsgroups | gmane.comp.video.ffmpeg.devel |
|---|---|
| Message-ID | <178427011130.59.14179892932976252818@29965ddac10e> |
PR #23837 opened by Lynne URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23837 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23837.patch Rate control & allocation (NMR coder) - Integral-windup servo replaced with stateless pressure + per-frame lambda slew limiter; reservoir seeded full at stream start (fixes start-of-file crackle and post-loud quality craters). - CPE pairs now solved jointly under one shared lambda against a pooled bit budget (was equal per-channel split that starved the mid). - Isolated transients get a bit-burst held across the short run; dense beats get a starvation-scaled boost; START frames clamp thresholds against backward masking. - Coded bandwidth retuned upward at >- 48 kbps/ch (18.5 kHz at 96k stereo, 20 kHz at 128k); lower rates untouched. Stereo - M/S adoption now content-driven and rate-free (side < 0.5×mid), with per-grid decision banks, EMA-smoothed statistics, and leave-hysteresis — image no longer churns or wanders. - I/S actually works: competes with M/S above 6.1 kHz (was structurally dead), engages only under sustained starvation (pressure times lambda-floor gate), falls back to M/S when unengaged. - Decorrelated pairs (applause-class) decouple: per-channel windows, no M/S. Windows (psy) - Block switching synced across CPE pairs — common_window survives transients (fixes one-channel garble at beats). - Novelty veto kills false short runs on periodic content (trumpet/Berling flutter class). TNS / PNS - Short-block TNS enabled: pooled per-group filters (no more shared-scalefactor dropout holes), run-scoped Schmitt accepts (no gravel). - PNS candidacy widened to near-masked bands — texture the allocator would delete becomes matched noise instead; entry/exit debounced. CLI - Native AAC at low CBR bitrates without -ar auto-resamples (32/24/16 kHz below 44/28/22 kbps/ch) with a warning; -ar overrides. From 95ecba94a32934180b4926dff63c2009b1b293a0 Mon Sep 17 00:00:00 2001 From: Lynne <[email protected]> Date: Fri, 17 Jul 2026 14:18:41 +0800 Subject: [PATCH 1/7] avcodec/psymodel: add pair-synced window switching hook and pair state Divergent block switching between the two channels of a CPE disables common_window, and with it every joint stereo tool, on exactly the frames that need them most; the lone short channel then codes its transient alone, which is audible as one-channel garble at beats. Add an optional window_pair() model hook so a psy model can decide both channels of a pair together, and encoder-fed per-pair state: the joint-tool candidacy fraction EMA and a Schmitt-gated decouple flag, letting the model fall back to independent per-channel switching on content whose joint tools are dead (diffuse, decorrelated pairs) where forcing a common grid only costs quality. --- libavcodec/psymodel.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/libavcodec/psymodel.h b/libavcodec/psymodel.h index 9dbce89780..302d25e1a2 100644 --- a/libavcodec/psymodel.h +++ b/libavcodec/psymodel.h @@ -106,6 +106,8 @@ typedef struct FFPsyContext { } bitres; void* model_priv_data; ///< psychoacoustic model implementation private data + float pair_joint[16]; ///< encoder-fed per-CPE joint-tool candidacy fraction EMA (bands M/S would adopt or I/S renders; ~0 = joint tools dead); 0 = unseeded + uint8_t pair_decoupled[16]; ///< Schmitt state over pair_joint: 1 = joint tools dead on this pair, joint windowing/M-S coupling suspended } FFPsyContext; /** @@ -128,6 +130,17 @@ typedef struct FFPsyModel { */ FFPsyWindowInfo (*window)(FFPsyContext *ctx, const float *audio, const float *la, int channel, int prev_type); + /** + * Suggest window sequences for both channels of a CPE with block switching + * synchronized across the pair (either channel's attack switches both), so + * common_window and the joint stereo tools stay available. Optional; when + * NULL the encoder decides each channel independently via window(). + */ + void (*window_pair)(FFPsyContext *ctx, const float *audio0, const float *la0, + const float *audio1, const float *la1, + int channel0, int channel1, + int prev_type0, int prev_type1, FFPsyWindowInfo wi[2]); + /** * Perform psychoacoustic analysis and set band info (threshold, energy) for a group of channels. * -- 2.52.0 From e297e71675edf8b15ac3602434f06d3427e53878 Mon Sep 17 00:00:00 2001 From: Lynne <[email protected]> Date: Fri, 17 Jul 2026 14:19:01 +0800 Subject: [PATCH 2/7] avcodec/aacenc: rework NMR rate control, pool CPE budgets, add decision memory Rate control: replace the integral servo with a stateless pressure offset (exp2(-K*fill/R)) so a drained reservoir cannot wind up and crater quality after loud stretches; slew-limit the final operating lambda per frame (bits deviate instead, the reservoir absorbs); seed the reservoir full at stream start; and track rate strain explicitly: a long-frame lambda EMA against anchors that scale up when the achieved distortion/mask ratio flags noise-class content (whose psy masks are wholesale violated and lambda reads inflated), plus a lambda min-tracker separating sustained starvation from transient spikes at a comfortable operating point. The resulting pressure ramp gates every pressure-adaptive tool from one place. CPE budget pooling: solve both channels of a pair jointly under one shared lambda against a pooled budget (NMRSlot defer/solve/commit) instead of an equal per-channel split, which starved the mid/carrier while the side gold-plated. Mono and VBR output are unchanged. Transients: isolated onsets are coded uniformly finer across the short run and repaid from steady stretches; dense-beat runs get a starvation-scaled boost; a transition premask clamps START-frame thresholds toward the previous long frame (an attack cannot mask backwards). Decision memory: marginal per-frame re-decisions oscillate audibly, so every stateful choice now carries hysteresis - band zeroing, PNS enter/leave with debounce (and near-masked bands staying noise until a loudness guard), per-grid stereo mode banks that survive window switches, and the short-TNS accept state. TNS-covered bands price distortion by the synthesis filter's re-amplification gain, so the trellis spends where noise will actually be heard. --- libavcodec/aaccoder_nmr.h | 996 +++++++++++++++++++++++++------------- libavcodec/aacenc.h | 58 ++- 2 files changed, 705 insertions(+), 349 deletions(-) diff --git a/libavcodec/aaccoder_nmr.h b/libavcodec/aaccoder_nmr.h index 7a01a57570..224cb24d7d 100644 --- a/libavcodec/aaccoder_nmr.h +++ b/libavcodec/aaccoder_nmr.h @@ -83,15 +83,29 @@ * smooth while per-frame demand is tracked; 1.5 cuts lambda jitter ~25%. */ #define NMR_RC_CORR 1.5f -/* Leaky-bucket half-depth (bits/ch); 512 is the sweet spot — tighter rebounds as - * frames cannot hit the narrow window. Clamped to the 6144 bits/ch decoder buffer. */ -#define NMR_CBR_BUF 512 +/* Reservoir half-window (bits/ch); swept 512/1536/3072, 1536 optimal. */ +#define NMR_CBR_BUF 1536 +/* Slew limit on the FINAL operating lambda per frame; bits deviate instead, + * the reservoir absorbs. See memory: aac-castanets-transient-rc. */ +#define NMR_SLEW 1.6f +#define NMR_SLEW_RUN 1.15f /* within short runs */ #define NMR_RC_CITERS 3 /* corridor coarse-pass iters */ +/* Transition premask: an attack cannot mask backwards; clamp a START frame's + * thresholds toward the previous long frame's. */ +#define NMR_TRANS_PM 2.0f + +/* Zero-decision hysteresis: previously-coded bands need this margin below + * threshold to zero (marginal bands flicker audibly otherwise). */ +#define NMR_ZERO_STICKY 0.5f + /* Transient bit-burst: an isolated onset (preceded by >= NMR_BURST_GAP long frames) * is coded NMR_BURST_GAIN x finer, held uniform across the run, repaid from steady stretches. */ #define NMR_BURST_GAP 10 #define NMR_BURST_GAIN 8.0f +/* Dense-beat boost: short runs with gap < NMR_BURST_GAP get a budget factor + * ramping with the gap (starvation-scaled at the use site). */ +#define NMR_SHORT_BOOST 2.0f #define NMR_RC_FITERS 4 /* corridor fine-pass iters */ #define NMR_RC_TRACK 0.1f /* per-frame pull of the corridor centre toward the realized lambda */ @@ -106,6 +120,14 @@ * substituting real texture for 9 signalling bits is net-negative. */ #define NMR_PNS_LAM 100.0f +/* PNS decision hysteresis: enter and leave both cost a margin. */ +#define NMR_PNS_ENTER 0.7f +#define NMR_PNS_STAY 1.4f +/* PNS debounce: enter after NMR_PNS_ON consecutive wants, leave after + * NMR_PNS_OFF (chronically marginal bands never qualify). */ +#define NMR_PNS_ON 8 +#define NMR_PNS_OFF 4 + /** * Viterbi over the coding sequence act[0..nact-1] (indices into the per-band * curves nd/nb), with lambda binary-searched so the coded size ~ destbits. @@ -201,128 +223,86 @@ static int nmr_band_curve(AACEncContext *s, SingleChannelElement *sce, int w, in return ncand; } -static void search_for_quantizers_nmr(AVCodecContext *avctx, - AACEncContext *s, - SingleChannelElement *sce, - const float lambda) +/* Zero a channel with nothing codeable; stale band_types would resurrect + * bands with chain-illegal scalefactors. */ +static void nmr_bail_channel(SingleChannelElement *sce) { - int bch = ((avctx->flags & AV_CODEC_FLAG_QSCALE) ? 2.0f : avctx->ch_layout.nb_channels); - int destbits = avctx->bit_rate * 1024.0 / avctx->sample_rate / bch * (lambda / 120.f); + for (int i = 0; i < 128; i++) { + if (sce->band_type[i] == INTENSITY_BT || sce->band_type[i] == INTENSITY_BT2) + continue; + sce->zeroes[i] = 1; + sce->band_type[i] = 0; + } +} + +/* Per-channel setup into slot t: short-block threshold shaping, the + * allocation law, zero decisions, and the PASS 1 coarse candidate curves. + * Returns the coded-band count; 0 = nothing codeable (caller bails). */ +static int nmr_setup_channel(AVCodecContext *avctx, AACEncContext *s, + SingleChannelElement *sce, NMRSlot *t) +{ + float (*nd)[NMR_NCAND] = s->nmr->nd[t->si]; + int (*nb)[NMR_NCAND] = s->nmr->nb[t->si]; + const int cstep = NMR_COARSE > 0 ? NMR_COARSE : NMR_STEP; int allz = 0, cutoff = 1024, nbnd = 0; - float thr[128]; /* allocation-law effective threshold (drives the trellis) */ - float thr_real[128]; /* real masking threshold (perceptual gates: PNS) */ - float pener[128]; /* band energy (for PNS noise target) */ - float pspread[128]; /* band tonality spread (1 = noise) */ - int minsf[128]; - float maxvals[128]; - - /* coded-band trellis state (indexed 0..nbnd-1) */ - int bidx[128]; /* sce band index (w*16+g) */ - int bw[128], bg[128], bst[128]; /* window group, swb, coef start per coded band */ - int blo[128]; /* finest candidate scalefactor */ - int bnc[128]; /* number of candidates */ - int chosen[128]; - int act[128]; /* active (non-PNS) band coding order */ - uint8_t is_pns[128]; /* trellis band coded as noise */ - - float (*nd)[NMR_NCAND] = s->nmr->nd; /* dist / threshold per candidate (heap) */ - int (*nb)[NMR_NCAND] = s->nmr->nb; /* spectral bits per candidate (heap) */ - - /* two-pass coarse->fine grid step (see NMR_COARSE), the lambda search runs on - * the cheap coarse grid, PASS 2 refines the winner at NMR_STEP granularity */ - const int cstep = NMR_COARSE > 0 ? NMR_COARSE : NMR_STEP; - - s->nmr->counted[s->cur_channel] = 0; - - /* Global-lambda RC: one solve per frame at a servoed centre lambda; the reservoir - * holds the long-run mean rate. Bypassed for VBR (-q:a) and the bootstrap frame. */ - int rc_eligible = !(avctx->flags & AV_CODEC_FLAG_QSCALE) && avctx->bit_rate > 0 && - avctx->bit_rate_tolerance != 0; - /* Leaky-bucket reservoir: rc_fill (signed +-rc_bmax); the spend-floor/cap below force - * lambda so no frame banks past +rc_bmax or borrows past -rc_bmax. */ - int rc_rate_frame = avctx->bit_rate * 1024.0 / avctx->sample_rate; - int rc_bmax = FFMIN(FFMAX(6144 * s->channels - rc_rate_frame, 256), NMR_CBR_BUF * s->channels); - if (rc_eligible && avctx->frame_num != s->nmr->rc_frame_num) { - if (s->nmr->rc_frame_num > 0 && s->nmr->lam_rc > 0.0f) - s->nmr->rc_fill = av_clip(s->nmr->rc_fill + rc_rate_frame - s->last_frame_pb_count, - -rc_bmax, rc_bmax); - s->nmr->rc_frame_num = avctx->frame_num; - - /* Transient burst run state: set at run start and held across the run so - * coding stays uniform; repaid from the reservoir's steady stretches. */ - int is_short = sce->ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE; - if (is_short) { - if (!s->nmr->prev_was_short) /* run start */ - s->nmr->run_burst = s->nmr->frames_since_short >= NMR_BURST_GAP - ? NMR_BURST_GAIN : 1.0f; - s->nmr->frames_since_short = 0; - } else { - s->nmr->run_burst = 1.0f; - s->nmr->frames_since_short++; - } - s->nmr->prev_was_short = is_short; + uint8_t *zprev = s->nmr->zero_prev[s->cur_channel & 15]; + if (s->nmr->zero_nw[s->cur_channel & 15] != sce->ics.num_windows) { + memset(zprev, 1, 128); + s->nmr->zero_nw[s->cur_channel & 15] = sce->ics.num_windows; } - int rc_global = rc_eligible && s->nmr->lam_rc > 0.0f; - if (s->psy.bitres.alloc >= 0) - destbits = s->psy.bitres.alloc * - (lambda / (avctx->global_quality ? avctx->global_quality : 120)); - if (rc_global && s->psy.bitres.alloc >= 0) - /* uniform CBR target: nominal rate plus fast reservoir repayment */ - destbits = (avctx->bit_rate * 1024.0 / avctx->sample_rate - + s->nmr->rc_fill / 2.0) / s->channels; - destbits = FFMIN(destbits, 5800); - /* honest budget: subtract the measured non-trellis overhead (section data, ICS, - * sf/PNS signalling), which is rate-dependent hence adaptive. */ - if (s->nmr->side_inited) - destbits = av_clip(destbits - (int)(s->nmr->side_ema / s->channels), 64, 5800); - - /* Apply the held transient burst factor (set in the run-state machine above). */ - if (sce->ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE && s->nmr->run_burst > 1.0f) - destbits = av_clip((int)(destbits * s->nmr->run_burst), 64, 6800); + t->sce = sce; + t->cur_ch = s->cur_channel; + t->is8 = sce->ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE; + t->nbnd = t->nact = 0; /* band cutoff index for this frame's window size; the bandwidth is fixed * at init and shared with the psy model */ cutoff = s->bandwidth * 2 * (1024 / sce->ics.num_windows) / avctx->sample_rate; - /* Short-block transient noise shaping (pairs with short-block TNS): temporal - * premasking clamps each window's threshold toward the preceding windows' - * (Apple's preEchoReduction), and flat-residual flattens each window's thresholds - * to their per-window mean so TNS synthesis has a white floor to concentrate. */ + /* Short-block shaping: temporal premask + per-window threshold flatten. */ if (sce->ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE) { const float pm_p1 = 0.1f, pm_p2 = 2.0f, pm_p3 = 4.0f; for (int g = 0; g < sce->ics.num_swb; g++) { float t1 = FLT_MAX, t2 = FLT_MAX; /* original thr of w-1, w-2 */ for (int w = 0; w < sce->ics.num_windows; w++) { FFPsyBand *b = &s->psy.ch[s->cur_channel].psy_bands[w*16+g]; - float t = b->threshold; - float c = FFMIN(t, FFMIN(t1*pm_p2, t2*pm_p3)); - b->threshold = FFMAX(c, t*pm_p1); - t2 = t1; t1 = t; + float th = b->threshold; + float c = FFMIN(th, FFMIN(t1*pm_p2, t2*pm_p3)); + b->threshold = FFMAX(c, th*pm_p1); + t2 = t1; t1 = th; } } { for (int w = 0; w < sce->ics.num_windows; w++) { - float sum = 0.0f; int n = 0; + float sum = 0.0f, esum = 0.0f; int n = 0; for (int g = 0; g < sce->ics.num_swb; g++) { FFPsyBand *b = &s->psy.ch[s->cur_channel].psy_bands[w*16+g]; - if (b->energy > b->threshold && b->threshold > 0.0f) { sum += b->threshold; n++; } + if (b->energy > b->threshold && b->threshold > 0.0f) { sum += b->threshold; esum += b->energy; n++; } } if (n > 0) { - float mean = sum / n; + /* keep each window codeable: cap the mean 12dB under the + * window's mean audible energy */ + float mean = FFMIN(sum / n, (esum / n) * expf(-12.0f * (float)M_LN10 / 10.0f)); for (int g = 0; g < sce->ics.num_swb; g++) { FFPsyBand *b = &s->psy.ch[s->cur_channel].psy_bands[w*16+g]; if (b->energy > b->threshold && b->threshold > 0.0f) - b->threshold = mean; + b->threshold = FFMIN(mean, b->threshold * 1e9f); } } } } } - /* Allocation curve to favour high frequencies */ - const float a_ae = 0.443f, a_at = 0.111f; + /* Allocation law; short frames blend to softer energy exponents under + * pressure (roll anti-starvation, see memory). */ + float a_ae = 0.443f, a_at = 0.111f; + if (sce->ics.num_windows == 8 && s->nmr) { + /* blend to mask-weighted exponents under rate pressure */ + a_ae += (0.35f - a_ae) * s->nmr->press; + a_at += (0.3f - a_at) * s->nmr->press; + } for (int w = 0; w < sce->ics.num_windows; w += sce->ics.group_len[w]) { int start = 0; for (int g = 0; g < sce->ics.num_swb; start += sce->ics.swb_sizes[g++]) { @@ -336,69 +316,128 @@ static void search_for_quantizers_nmr(AVCodecContext *avctx, sce->zeroes[(w+w2)*16+g] = 0; continue; } + float zthr_mul = zprev[w*16+g] ? 1.0f : NMR_ZERO_STICKY; + /* M/S side bands: zero-reluctance scaled by side/mid ratio (a tiny + * side IS the image; zeroing it flickers). */ + if ((t->cur_ch & 1) && s->nmr && s->nmr->pair && + s->nmr->smode_band[(t->cur_ch >> 1) & 7][w*16+g] == 1) { + const FFPsyBand *mb = &s->psy.ch[s->cur_channel - 1].psy_bands[w*16+g]; + float ratio = 0.0f; + float eside = 0.0f; + for (int w2 = 0; w2 < sce->ics.group_len[w]; w2++) { + const FFPsyBand *bb = &s->psy.ch[s->cur_channel].psy_bands[(w+w2)*16+g]; + eside += bb->energy; + } + ratio = eside / FFMAX(mb->energy * sce->ics.group_len[w], 1e-9f); + zthr_mul *= 0.25f + 0.75f * av_clipf(ratio / 0.3f, 0.0f, 1.0f); + } for (int w2 = 0; w2 < sce->ics.group_len[w]; w2++) { FFPsyBand *band = &s->psy.ch[s->cur_channel].psy_bands[(w+w2)*16+g]; ener += band->energy; spread = FFMIN(spread, band->spread); - if (start >= cutoff || band->energy <= band->threshold || band->threshold == 0.0f) { + if (start >= cutoff || band->energy <= band->threshold * zthr_mul || + band->threshold == 0.0f) { sce->zeroes[(w+w2)*16+g] = 1; continue; } uplim += band->threshold; nz = 1; } + zprev[w*16+g] = !nz; sce->zeroes[w*16+g] = !nz; - thr_real[w*16+g] = uplim; /* real mask, before the allocation law (PNS gate) */ - if (nz && ener > 0.0f && uplim > 0.0f) + t->thr_real[w*16+g] = uplim; /* real mask, before the allocation law (PNS gate) */ + if (nz && ener > 0.0f && uplim > 0.0f) /* allocation law */ uplim = expf(a_ae * logf(ener) + a_at * logf(uplim)); - thr[w*16+g] = uplim; - pener[w*16+g] = ener; - pspread[w*16+g] = spread; + t->thr[w*16+g] = uplim; + t->pener[w*16+g] = ener; + t->pspread[w*16+g] = spread; allz |= nz; } } if (!allz) - goto bail; + return 0; + + /* transition premask (see NMR_TRANS_PM) */ + if (sce->ics.num_windows == 1) { + int ci = t->cur_ch & 15; + if (sce->ics.window_sequence[0] == LONG_START_SEQUENCE && + s->nmr->thr_prev_ok[ci]) { + for (int g = 0; g < sce->ics.num_swb && g < 64; g++) + if (t->thr[g] > 0.0f && s->nmr->thr_prev[ci][g] > 0.0f) + t->thr[g] = FFMIN(t->thr[g], s->nmr->thr_prev[ci][g] * NMR_TRANS_PM); + } + for (int g = 0; g < sce->ics.num_swb && g < 64; g++) + s->nmr->thr_prev[ci][g] = t->thr[g]; + s->nmr->thr_prev_ok[ci] = 1; + } else { + s->nmr->thr_prev_ok[t->cur_ch & 15] = 0; + } s->aacdsp.abs_pow34(s->scoefs, sce->coeffs, 1024); ff_quantize_band_cost_cache_init(s); + /* TNS synthesis gain per band: the decoder re-amplifies residual-domain + * quantization noise by the whitening gain (shorts only). */ + for (int i = 0; i < 128; i++) + t->tnsg[i] = 1.0f; + if (sce->ics.num_windows == 8 && sce->tns.present) { + const int mmm2 = FFMIN(sce->ics.tns_max_bands, sce->ics.max_sfb ? sce->ics.max_sfb : sce->ics.num_swb); + for (int w = 0; w < 8; w++) { + int bottom2 = sce->ics.num_swb; + for (int filt = 0; filt < sce->tns.n_filt[w]; filt++) { + int top2 = bottom2; + bottom2 = FFMAX(0, top2 - sce->tns.length[w][filt]); + if (!sce->tns.order[w][filt]) + continue; + for (int g = FFMIN(bottom2, mmm2); g < FFMIN(top2, mmm2); g++) { + int s0 = sce->ics.swb_offset[g] + w*128; + int s1 = sce->ics.swb_offset[g+1] + w*128; + float eres = 0.0f; + const FFPsyBand *pb = &s->psy.ch[s->cur_channel].psy_bands[w*16+g]; + for (int k = s0; k < s1; k++) + eres += sce->coeffs[k]*sce->coeffs[k]; + t->tnsg[w*16+g] = av_clipf(pb->energy / FFMAX(eres, 1e-12f), 1.0f, 64.0f); + } + } + } + } + /* finest codeable scalefactor and max value per band */ for (int w = 0; w < sce->ics.num_windows; w += sce->ics.group_len[w]) { int start = w*128; for (int g = 0; g < sce->ics.num_swb; g++) { - maxvals[w*16+g] = find_max_val(sce->ics.group_len[w], sce->ics.swb_sizes[g], s->scoefs + start); - minsf[w*16+g] = maxvals[w*16+g] > 0 ? coef2minsf(maxvals[w*16+g]) : 0; + t->maxvals[w*16+g] = find_max_val(sce->ics.group_len[w], sce->ics.swb_sizes[g], s->scoefs + start); + t->minsf[w*16+g] = t->maxvals[w*16+g] > 0 ? coef2minsf(t->maxvals[w*16+g]) : 0; start += sce->ics.swb_sizes[g]; } } - /* PASS 1: - * precompute each coded band's cost curve at the coarse candidate step + /* PASS 1: coarse candidate curves per coded band * (the lambda search runs on this cheap grid, PASS 2 refines the winner) */ { for (int w = 0; w < sce->ics.num_windows; w += sce->ics.group_len[w]) { int start = w*128; for (int g = 0; g < sce->ics.num_swb; g++) { - if (!sce->zeroes[w*16+g] && maxvals[w*16+g] > 0 && nbnd < 128) { - int lo = av_clip(minsf[w*16+g], 0, SCALE_MAX_POS); - float invthr = 1.0f / FFMAX(thr[w*16+g], 1e-9f); + if (!sce->zeroes[w*16+g] && t->maxvals[w*16+g] > 0 && nbnd < 128) { + int lo = av_clip(t->minsf[w*16+g], 0, SCALE_MAX_POS); + float invthr = 1.0f / FFMAX(t->thr[w*16+g], 1e-9f); int ncand = nmr_band_curve(s, sce, w, g, start, lo, cstep, NMR_NCAND, - invthr, maxvals[w*16+g], nd[nbnd], nb[nbnd]); + invthr, t->maxvals[w*16+g], nd[nbnd], nb[nbnd]); + if (t->tnsg[w*16+g] > 1.0f) + for (int o = 0; o < ncand; o++) + nd[nbnd][o] *= t->tnsg[w*16+g]; if (ncand == 0) { - /* nothing codeable -> drop the whole group band. The - * subwindow flags must be cleared too: the encoder later - * re-derives the group flag by ANDing them, which would - * resurrect the band with a never-assigned scalefactor. */ + /* nothing codeable: drop the group band incl. subwindow + * flags (group flag is re-derived by ANDing) */ for (int w2 = 0; w2 < sce->ics.group_len[w]; w2++) sce->zeroes[(w+w2)*16+g] = 1; } else { - bidx[nbnd] = w*16+g; - bw[nbnd] = w; - bg[nbnd] = g; - bst[nbnd] = start; - blo[nbnd] = lo; - bnc[nbnd] = ncand; + t->bidx[nbnd] = w*16+g; + t->bw[nbnd] = w; + t->bg[nbnd] = g; + t->bst[nbnd] = start; + t->blo[nbnd] = lo; + t->bnc[nbnd] = ncand; nbnd++; } } @@ -406,223 +445,94 @@ static void search_for_quantizers_nmr(AVCodecContext *avctx, } } } - if (!nbnd) - goto bail; + t->nbnd = nbnd; + for (int b = 0; b < nbnd; b++) { + t->act[b] = b; + t->is_pns[b] = 0; + } + t->nact = nbnd; + return nbnd; +} - /* solve the trellis over all coded bands, then offer PNS at the operating - * lambda and re-solve over the survivors with the freed budget */ - { - int nact = nbnd, pns_count = 0; - float lam0 = s->nmr->lam[s->cur_channel]; - float lam; +/* total bits of a slot's current chosen[] on grid `step`, incl. sf deltas */ +static int nmr_slot_bits(const NMRSlot *t, const int (*nb)[NMR_NCAND], int step) +{ + int tot = 0; + for (int k = 0; k < t->nact; k++) + tot += nb[t->act[k]][t->chosen[t->act[k]]]; + for (int k = 1; k < t->nact; k++) + tot += NMR_SFBITS((t->blo[t->act[k]]+t->chosen[t->act[k]]*step) - + (t->blo[t->act[k-1]]+t->chosen[t->act[k-1]]*step)); + return tot; +} - for (int b = 0; b < nbnd; b++) { - act[b] = b; - is_pns[b] = 0; +/* Run every slot's trellis at one fixed lambda; returns the pooled bits. */ +static int nmr_eval_slots(AACEncContext *s, NMRSlot *const *sl, int nsl, int step, float lam) +{ + int total = 0; + for (int k = 0; k < nsl; k++) { + NMRSlot *t = sl[k]; + if (!t->nact) + continue; + nmr_solve(s, s->nmr->nd[t->si], s->nmr->nb[t->si], t->blo, t->bnc, step, + t->act, t->nact, 0, t->chosen, lam, lam, 1); + total += nmr_slot_bits(t, s->nmr->nb[t->si], step); + } + return total; +} + +/* Bisect ONE shared lambda across the slots so the POOLED bits meet destbits. + * This is the CPE budget pool: bits flow to whichever channel of the pair has + * demand at the common operating point, instead of an equal per-channel split. */ +static float nmr_solve_slots(AACEncContext *s, NMRSlot *const *sl, int nsl, int step, + int destbits, float lo_l, float hi_l, int iters) +{ + float lam = 1.0f; + for (int it = 0; it < iters; it++) { + lam = sqrtf(lo_l * hi_l); + int total = nmr_eval_slots(s, sl, nsl, step, lam); + if (it == iters - 1) + break; + /* over budget -> go coarser */ + if (total > destbits) + lo_l = lam; + else + hi_l = lam; + } + return lam; +} + +/* Write a solved slot back into its channel: band types, scalefactors, and the + * SCALE_MAX_DIFF legality fixups. Verbatim from the pre-pool single-channel tail. */ +static void nmr_commit_channel(AACEncContext *s, NMRSlot *t) +{ + SingleChannelElement *sce = t->sce; + const int (*nb)[NMR_NCAND] = (const int (*)[NMR_NCAND])s->nmr->nb[t->si]; + + for (int b = 0; b < t->nbnd; b++) { + int bi = t->bidx[b]; + if (t->is_pns[b]) { + sce->band_type[bi] = NOISE_BT; + sce->zeroes[bi] = 0; + sce->pns_ener[bi] = t->pener[bi] * FFMIN(1.0f, t->pspread[bi]*t->pspread[bi]); + } else { + sce->sf_idx[bi] = av_clip(t->blo[b] + t->chosen[b]*NMR_STEP, 0, SCALE_MAX_POS); } - if (rc_global) { - /* bisect to this frame's bit demand within the corridor around the - * servoed lambda: per-frame psy demand is tracked, but lambda cannot - * jump, which keeps quality smooth across frames */ - float lo = s->nmr->lam_rc / NMR_RC_CORR; - /* Transient burst: widen the lower lambda bound so the bisection can actually - * pour the boosted destbits into an onset frame (finer coding kills the - * pre-echo); reservoir servo repays it from the steady frames. run_burst==1 on - * non-onset frames leaves the corridor unchanged. */ - if (sce->ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE && s->nmr->run_burst > 1.0f) - lo /= s->nmr->run_burst; - lam = nmr_solve(s, nd, nb, blo, bnc, cstep, act, nact, destbits, chosen, - lo, s->nmr->lam_rc * NMR_RC_CORR, - NMR_RC_CITERS); + } - int tot = 0; - for (int k = 0; k < nact; k++) - tot += nb[act[k]][chosen[act[k]]]; - for (int k = 1; k < nact; k++) - tot += NMR_SFBITS((blo[act[k]]+chosen[act[k]]*cstep) - (blo[act[k-1]]+chosen[act[k-1]]*cstep)); - int hardcap = av_clip((int)(5800.f * FFMIN(1.f, lambda / 120.f)), 256, 5800); - /* leaky-bucket window: don't borrow past -rc_bmax (cap) or bank past +rc_bmax (floor) */ - int rc_cap = FFMIN(hardcap, (s->nmr->rc_fill + rc_rate_frame + rc_bmax) / s->channels); - int rc_floor = FFMAX(0, (s->nmr->rc_fill + rc_rate_frame - rc_bmax) / s->channels); - if (tot > rc_cap) - lam = nmr_solve(s, nd, nb, blo, bnc, cstep, act, nact, rc_cap, chosen, - lam, 1e4f, NMR_CITERS); - else if (tot < rc_floor) - lam = nmr_solve(s, nd, nb, blo, bnc, cstep, act, nact, rc_floor, chosen, - 1e-9f, lam, NMR_CITERS); - } else if (NMR_COARSE > 0 && lam0 > 0.0f) { - /* per-frame bisection; lambda is strongly frame-correlated, so when a - * previous frame's operating lambda exists, bisect a narrow bracket - * around it. A result near the bracket edge means the budget crossing - * lies outside (hard content transition) == redo the full search. */ - lam = nmr_solve(s, nd, nb, blo, bnc, cstep, act, nact, destbits, chosen, - lam0/32.0f, lam0*32.0f, NMR_CWARM); - if (lam < lam0/16.0f || lam > lam0*16.0f) - lam0 = 0.0f; - } - if (!rc_global && lam0 <= 0.0f) - lam = nmr_solve(s, nd, nb, blo, bnc, cstep, act, nact, destbits, chosen, - 1e-9f, 1e4f, NMR_COARSE > 0 ? NMR_CITERS : NMR_ITERS); - - /* PASS 2: - * refine each band at full granularity (NMR_STEP) in a +/-cstep window - * around the coarse pick, then re-solve. Recovers single-pass quality while the - * lambda search stayed cheap on the coarse grid. */ - if (NMR_COARSE > 0) { - /* nmr_speed, 0 = slowest/best, higher = faster. It narrows the fine - * refine +/-window (scalefactors) below NMR_COARSE: at speed 0 the window - * spans the whole coarse-grid gap, so the two-pass result matches the - * exhaustive single-pass search. - * Each speed level shaves one sf off the window. - * At @64k mono (Zim / xRT): speed 0 -> 0.00095/15x, - * 2 -> 0.00096/18x, 3 -> 0.00100/20x, 4 -> 0.00103/22x */ - int win = NMR_COARSE - av_clip(s->options.nmr_speed, 0, 4); - for (int b = 0; b < nbnd; b++) { - int center = blo[b] + chosen[b]*cstep; - int flo = av_clip(center - win, av_clip(minsf[bidx[b]], 0, SCALE_MAX_POS), SCALE_MAX_POS); - int maxn = FFMIN(NMR_NCAND, 2*win/NMR_STEP + 1); - float invthr = 1.0f / FFMAX(thr[bidx[b]], 1e-9f); - int ncand = nmr_band_curve(s, sce, bw[b], bg[b], bst[b], flo, NMR_STEP, maxn, - invthr, maxvals[bidx[b]], nd[b], nb[b]); - blo[b] = flo; - bnc[b] = FFMAX(1, ncand); - } - /* fine pass: narrow corridor around the coarse solve */ - if (rc_global) - lam = nmr_solve(s, nd, nb, blo, bnc, NMR_STEP, act, nact, destbits, chosen, - lam/2.0f, lam*2.0f, NMR_RC_FITERS); - else - lam = nmr_solve(s, nd, nb, blo, bnc, NMR_STEP, act, nact, destbits, chosen, - lam/16.0f, lam*16.0f, NMR_IFINE); - } - - if (rc_global) { - /* leaky-bucket clamp: keep the frame within [rc_floor, rc_cap] so the reservoir - * stays in +-rc_bmax -- clamp lambda UP if it would borrow past the cap, DOWN if it - * would bank past the floor (spend-floor). The hard cap follows the encoder's outer - * lambda so the (rare) hard-overflow re-encode -- which shrinks that lambda -- always - * converges; on the first pass lambda is nominal and this is 5800. */ - int hardcap = av_clip((int)(5800.f * FFMIN(1.f, lambda / 120.f)), 256, 5800); - int tot = 0; - for (int k = 0; k < nact; k++) - tot += nb[act[k]][chosen[act[k]]]; - for (int k = 1; k < nact; k++) - tot += NMR_SFBITS((blo[act[k]]+chosen[act[k]]*NMR_STEP) - (blo[act[k-1]]+chosen[act[k-1]]*NMR_STEP)); - int rc_cap = FFMIN(hardcap, (s->nmr->rc_fill + rc_rate_frame + rc_bmax) / s->channels); - int rc_floor = FFMAX(0, (s->nmr->rc_fill + rc_rate_frame - rc_bmax) / s->channels); - if (tot > rc_cap) - lam = nmr_solve(s, nd, nb, blo, bnc, NMR_STEP, act, nact, rc_cap, chosen, - lam, 1e4f, NMR_RC_ITERS); - else if (tot < rc_floor) - lam = nmr_solve(s, nd, nb, blo, bnc, NMR_STEP, act, nact, rc_floor, chosen, - 1e-9f, lam, NMR_RC_ITERS); - } - - s->nmr->lam[s->cur_channel] = lam; /* warm start for the next frame */ - if (rc_global) { - /* drag the corridor centre toward the realized lambda so it follows - * content drift faster than the reservoir term alone */ - float c = s->nmr->lam_rc * powf(lam / s->nmr->lam_rc, NMR_RC_TRACK); - /* then servo the centre off the reservoir error so the long-run rate - * returns to nominal. rc_fill>0 = bits banked (undershooting) -> lower - * lambda to spend them; <0 -> raise it. This is what holds the mean; - * the corridor tracking alone has no rate authority and a bad centre - * would otherwise drift for dozens of frames, starving each one. */ - float R = avctx->bit_rate * 1024.0 / avctx->sample_rate; - c *= exp2f(-NMR_RC_K_CBR * s->nmr->rc_fill / R); - s->nmr->lam_rc = av_clipf(c, 1e-6f, 1e4f); - } else if (rc_eligible && nbnd >= 8) { - /* bootstrap the servo off the first substantive frame; near-silent - * lead-in frames have degenerate budgets that rail the bisection to - * a nonsense lambda and would poison the whole stream */ - s->nmr->lam_rc = av_clipf(lam, 1e-4f, 10.0f); - } - - { /* PNS */ - const float pns_lam = NMR_PNS_LAM; - /* band 0 (lowest freq) is kept as the global-gain / sf-chain anchor */ - for (int b = 1; b < nbnd; b++) { - int bi = bidx[b]; - float spread = pspread[bi]; - float nmr_pns, cost_keep, cost_pns, frac; - if (!sce->can_pns[bi]) - continue; - - /* Loud-band guard: never substitute a band whose energy is far above the - * masking threshold -- energy-matched noise on a dominant band clips/pops - * (and is audibly wrong). PNS is for near-masked noise only. */ - if (pener[bi] > NMR_PNS_MAX_ET * thr_real[bi]) - continue; - - /* Struggle gate: no PNS at all unless the encoder is genuinely under bit - * pressure (high operating lambda). */ - if (lam <= pns_lam) - continue; - - /* Spectral-hole fill: a noise-like band the trellis left mostly empty */ - frac = nd[b][chosen[b]] * thr[bi] / FFMAX(pener[bi], 1e-9f); - if (spread > NMR_PNS_HOLE_SPREAD && frac > NMR_PNS_HOLE_FRAC) { - is_pns[b] = 1; - pns_count++; - continue; - } - - /* Only replace a band that is being coded audibly badly */ - if (nd[b][chosen[b]] * thr[bi] <= NMR_PNS_NDGATE * thr_real[bi]) - continue; - - /* perceptual cost of replacing the band with energy-matched noise: - * the non-noise-like fraction of its energy, in dist/threshold units */ - nmr_pns = FFMAX(0.0f, pener[bi] * (1.0f - spread*spread)) - / FFMAX(thr[bi], 1e-9f); - cost_keep = nd[b][chosen[b]] + lam * nb[b][chosen[b]]; - cost_pns = nmr_pns + lam * NMR_PNS_BITS; - if (cost_pns < cost_keep) { - is_pns[b] = 1; - pns_count++; - } - } - if (pns_count) { - int budget2 = destbits - pns_count * NMR_PNS_BITS; - nact = 0; - for (int b = 0; b < nbnd; b++) - if (!is_pns[b]) - act[nact++] = b; - /* re-solve over the survivors: at fixed lambda the allocation is - * the same except for the repaired sf-delta chain; in bisection - * mode re-spend the freed budget */ - if (rc_global) - nmr_solve(s, nd, nb, blo, bnc, NMR_STEP, act, nact, budget2, chosen, - lam, lam, 1); - else - nmr_solve(s, nd, nb, blo, bnc, NMR_STEP, act, nact, budget2, chosen, - 1e-9f, 1e4f, NMR_ITERS); - } - } - for (int b = 0; b < nbnd; b++) { - int bi = bidx[b]; - if (is_pns[b]) { - sce->band_type[bi] = NOISE_BT; - sce->zeroes[bi] = 0; - sce->pns_ener[bi] = pener[bi] * FFMIN(1.0f, pspread[bi]*pspread[bi]); - } else { - sce->sf_idx[bi] = av_clip(blo[b] + chosen[b]*NMR_STEP, 0, SCALE_MAX_POS); - } - } - - { /* record the bits this solve accounted for; the encoder compares them - * against the channel's real output to keep the budget honest */ - int tot = 0, prevb = -1; - for (int b = 0; b < nbnd; b++) { - if (is_pns[b]) - continue; - tot += nb[b][chosen[b]]; - if (prevb >= 0) - tot += NMR_SFBITS((blo[b]+chosen[b]*NMR_STEP) - (blo[prevb]+chosen[prevb]*NMR_STEP)); - prevb = b; - } - s->nmr->counted[s->cur_channel] = tot; + + { /* record the bits this solve accounted for; the encoder compares them + * against the channel's real output to keep the budget honest */ + int tot = 0, prevb = -1; + for (int b = 0; b < t->nbnd; b++) { + if (t->is_pns[b]) + continue; + tot += nb[b][t->chosen[b]]; + if (prevb >= 0) + tot += NMR_SFBITS((t->blo[b]+t->chosen[b]*NMR_STEP) - (t->blo[prevb]+t->chosen[prevb]*NMR_STEP)); + prevb = b; } + s->nmr->counted[t->cur_ch] = tot; } /* SCALE_MAX_DIFF condition: @@ -645,7 +555,7 @@ static void search_for_quantizers_nmr(AVCodecContext *avctx, if (prev != -1) sce->sf_idx[w*16+g] = av_clip(sce->sf_idx[w*16+g], prev - SCALE_MAX_DIFF, prev + SCALE_MAX_DIFF); - sce->band_type[w*16+g] = find_min_book(maxvals[w*16+g], sce->sf_idx[w*16+g]); + sce->band_type[w*16+g] = find_min_book(t->maxvals[w*16+g], sce->sf_idx[w*16+g]); if (sce->band_type[w*16+g] <= 0) { if (!ff_sfdelta_can_remove_band(sce, nextband, prev, w*16+g)) { sce->band_type[w*16+g] = 1; @@ -663,11 +573,8 @@ static void search_for_quantizers_nmr(AVCodecContext *avctx, } } - /* Every band, coded or not, must carry a chain-legal scalefactor: the - * codebook trellis (encode_window_bands_info) may later absorb a dropped - * band into a nonzero section, resurrecting it, and its sf then gets - * coded. Forward-fill with the previous coded sf (delta 0, cheapest); - * leading bands get the global gain. */ + /* every band must carry a chain-legal scalefactor (re-clamp, codebook + * fixup, global gain) */ if (prev != -1) { int last = sce->sf_idx[0]; for (int w = 0; w < sce->ics.num_windows; w += sce->ics.group_len[w]) { @@ -681,19 +588,414 @@ static void search_for_quantizers_nmr(AVCodecContext *avctx, } } } - return; +} -bail: - /* Nothing codeable in this channel. Leave a fully consistent state: any - * stale nonzero band_type acts as a codebook lower bound in the encoder's - * section trellis (encode_window_bands_info), which would forbid the zero - * section and resurrect the band with a stale, chain-illegal scalefactor. - * Pre-decided intensity bands keep their signalling. */ - for (int i = 0; i < 128; i++) { - if (sce->band_type[i] == INTENSITY_BT || sce->band_type[i] == INTENSITY_BT2) - continue; - sce->zeroes[i] = 1; - sce->band_type[i] = 0; +/* Solve one element group (a solo channel, or a CPE pair pooled under one + * shared lambda and one pooled budget), then PNS and commit. */ +static void nmr_solve_group(AVCodecContext *avctx, AACEncContext *s, + const float lambda, NMRSlot *const *sl, int nsl, + int chans, int rc_eligible, int rc_global, + int rc_rate_frame, int rc_bmax) +{ + const int cstep = NMR_COARSE > 0 ? NMR_COARSE : NMR_STEP; + int bch = ((avctx->flags & AV_CODEC_FLAG_QSCALE) ? 2.0f : avctx->ch_layout.nb_channels); + int destbits = avctx->bit_rate * 1024.0 / avctx->sample_rate / bch * (lambda / 120.f) * chans; + int is8_any = 0; + float lam; + float rc_off = 1.0f, lam_dem = 0.0f; + + for (int k = 0; k < nsl; k++) + is8_any |= sl[k]->is8; + + if (s->psy.bitres.alloc >= 0) + destbits = s->psy.bitres.alloc * + (lambda / (avctx->global_quality ? avctx->global_quality : 120)) * chans; + if (rc_global && s->psy.bitres.alloc >= 0) { + /* CBR target: nominal + repayment, bounded +-30%/frame */ + double rr = avctx->bit_rate * 1024.0 / avctx->sample_rate; + destbits = (rr + av_clipd(s->nmr->rc_fill / 2.0, -0.3 * rr, 0.3 * rr)) * chans / s->channels; + } else if (rc_eligible && s->psy.bitres.alloc >= 0) { + /* pre-bootstrap CBR frames: target nominal (psy bitres is cold) */ + destbits = (avctx->bit_rate * 1024.0 / avctx->sample_rate) * chans / s->channels; + } + destbits = FFMIN(destbits, 5800 * chans); + /* honest budget: subtract the measured non-trellis overhead (section data, ICS, + * sf/PNS signalling), which is rate-dependent hence adaptive. */ + if (s->nmr->side_inited) + destbits = av_clip(destbits - (int)(s->nmr->side_ema * chans / s->channels), 64, 5800 * chans); + + /* Held transient burst, bank-aware: spend banked bits, never borrow deep + * (payback troughs starve the next transient). */ + if (s->nmr->run_burst > 1.0f) { + int extra = destbits * (s->nmr->run_burst - 1.0f); + int avail = FFMAX(0, (int)((s->nmr->rc_fill + rc_bmax / 2) * (int64_t)chans / s->channels)); + destbits = av_clip(destbits + FFMIN(extra, avail), 64, 6800 * chans); + } + + if (rc_global) { + /* corridor bisect around the servoed centre; pressure = stateless + * rc_off multiplier (folding it into lam_rc winds up) */ + float R = avctx->bit_rate * 1024.0 / avctx->sample_rate; + float cen; + int tot, hardcap, rc_cap; + float lo; + rc_off = exp2f(-NMR_RC_K_CBR * s->nmr->rc_fill / R); + cen = s->nmr->lam_rc * rc_off; + lo = cen / NMR_RC_CORR; + /* transient burst: widen the lower bound so the boosted destbits can + * actually pour into the onset frame */ + if (is8_any && s->nmr->run_burst > 1.0f) + lo /= s->nmr->run_burst; + lam = nmr_solve_slots(s, sl, nsl, cstep, destbits, + lo, cen * NMR_RC_CORR, NMR_RC_CITERS); + + tot = 0; + for (int k = 0; k < nsl; k++) + tot += nmr_slot_bits(sl[k], s->nmr->nb[sl[k]->si], cstep); + hardcap = av_clip((int)(5800.f * FFMIN(1.f, lambda / 120.f)), 256, 5800) * chans; + /* legality cap only; no spend-floor (rc_off spends the bank) */ + rc_cap = FFMIN(hardcap, (s->nmr->rc_fill + rc_rate_frame + rc_bmax) * chans / s->channels); + if (tot > rc_cap) { + lam = nmr_solve_slots(s, sl, nsl, cstep, rc_cap, lam, 1e4f, NMR_CITERS); + } + } else { + /* per-frame bisection, warm-started off the previous frame's lambda; + * a result at the bracket edge means redo the full search */ + float lam0 = s->nmr->lam[sl[0]->cur_ch]; + lam = 1.0f; + if (NMR_COARSE > 0 && lam0 > 0.0f) { + lam = nmr_solve_slots(s, sl, nsl, cstep, destbits, lam0/32.0f, lam0*32.0f, NMR_CWARM); + if (lam < lam0/16.0f || lam > lam0*16.0f) + lam0 = 0.0f; + } + if (lam0 <= 0.0f) + lam = nmr_solve_slots(s, sl, nsl, cstep, destbits, + 1e-9f, 1e4f, NMR_COARSE > 0 ? NMR_CITERS : NMR_ITERS); + } + + /* PASS 2: + * refine each band at full granularity (NMR_STEP) in a +/-cstep window + * around the coarse pick, then re-solve. Recovers single-pass quality while the + * lambda search stayed cheap on the coarse grid. */ + if (NMR_COARSE > 0) { + /* nmr_speed, 0 = slowest/best, higher = faster; see the option docs. */ + int win = NMR_COARSE - av_clip(s->options.nmr_speed, 0, 4); + for (int k = 0; k < nsl; k++) { + NMRSlot *t = sl[k]; + float (*ndk)[NMR_NCAND] = s->nmr->nd[t->si]; + int (*nbk)[NMR_NCAND] = s->nmr->nb[t->si]; + if (!t->nact) + continue; + /* the pow34 spectrum and the quantize cache are per-channel state */ + s->aacdsp.abs_pow34(s->scoefs, t->sce->coeffs, 1024); + ff_quantize_band_cost_cache_init(s); + for (int b = 0; b < t->nbnd; b++) { + int center = t->blo[b] + t->chosen[b]*cstep; + int flo = av_clip(center - win, av_clip(t->minsf[t->bidx[b]], 0, SCALE_MAX_POS), SCALE_MAX_POS); + int maxn = FFMIN(NMR_NCAND, 2*win/NMR_STEP + 1); + float invthr = 1.0f / FFMAX(t->thr[t->bidx[b]], 1e-9f); + int ncand = nmr_band_curve(s, t->sce, t->bw[b], t->bg[b], t->bst[b], flo, NMR_STEP, maxn, + invthr, t->maxvals[t->bidx[b]], ndk[b], nbk[b]); + if (t->tnsg[t->bidx[b]] > 1.0f) + for (int o = 0; o < ncand; o++) + ndk[b][o] *= t->tnsg[t->bidx[b]]; + t->blo[b] = flo; + t->bnc[b] = FFMAX(1, ncand); + } + } + /* fine pass: narrow corridor around the coarse solve */ + if (rc_global) + lam = nmr_solve_slots(s, sl, nsl, NMR_STEP, destbits, lam/2.0f, lam*2.0f, NMR_RC_FITERS); + else + lam = nmr_solve_slots(s, sl, nsl, NMR_STEP, destbits, lam/16.0f, lam*16.0f, NMR_IFINE); + } + + lam_dem = lam; /* demand-solved lambda, pre bucket clamp: what content wants */ + + if (rc_global) { + /* legality clamp, then the quality slew limiter */ + int hardcap = av_clip((int)(5800.f * FFMIN(1.f, lambda / 120.f)), 256, 5800) * chans; + int tot = 0, rc_cap; + for (int k = 0; k < nsl; k++) + tot += nmr_slot_bits(sl[k], s->nmr->nb[sl[k]->si], NMR_STEP); + rc_cap = FFMIN(hardcap, (s->nmr->rc_fill + rc_rate_frame + rc_bmax) * chans / s->channels); + if (tot > rc_cap) { + lam = nmr_solve_slots(s, sl, nsl, NMR_STEP, rc_cap, lam, 1e4f, NMR_RC_ITERS); + } + if (s->nmr->lam_slew > 0.0f) { + float kup, kdn; + /* hold lambda near-constant within short runs; bits follow content */ + kup = (is8_any && s->nmr->prev_was_short) ? NMR_SLEW_RUN : NMR_SLEW; + /* a deliberate onset burst may dive as far as its widened corridor + * allows; the RECOVERY back up is what must stay gradual */ + kdn = (is8_any && s->nmr->run_burst > 1.0f) ? NMR_SLEW * s->nmr->run_burst : + (is8_any && s->nmr->prev_was_short) ? NMR_SLEW_RUN : NMR_SLEW; + if (lam > s->nmr->lam_slew * kup || lam < s->nmr->lam_slew / kdn) { + lam = av_clipf(lam, s->nmr->lam_slew / kdn, s->nmr->lam_slew * kup); + tot = nmr_eval_slots(s, sl, nsl, NMR_STEP, lam); + /* never at the price of an illegal reservoir excursion */ + if (tot > rc_cap) { + lam = nmr_solve_slots(s, sl, nsl, NMR_STEP, rc_cap, lam, 1e4f, NMR_RC_ITERS); + } + } + } + s->nmr->lam_slew = lam; + } + + for (int k = 0; k < nsl; k++) + s->nmr->lam[sl[k]->cur_ch] = lam; /* warm start for the next frame */ + { /* nd: mean achieved dist/real-mask (dimensionless starvation + + * noise-class signal) */ + float ndsum = 0.0f; int ndn = 0; + for (int k = 0; k < nsl; k++) { + NMRSlot *t = sl[k]; + float (*ndk)[NMR_NCAND] = s->nmr->nd[t->si]; + for (int b_ = 0; b_ < t->nact; b_++) { + int b = t->act[b_], bi = t->bidx[b]; + if (t->thr_real[bi] > 0.0f && t->thr[bi] > 0.0f) { + ndsum += ndk[b][t->chosen[b]] * t->thr[bi] / t->thr_real[bi]; + ndn++; + } + } + } + /* long frames only (short groups inflate the ratio) */ + if (ndn >= 8 && !is8_any) { + float nd = ndsum / ndn; + s->nmr->nd_ema = s->nmr->nd_ema > 0.0f ? + 0.95f * s->nmr->nd_ema + 0.05f * nd : nd; + } + } + { /* track short vs long operating lambda (dense-beat boost scaling) */ + float *ema = is8_any ? &s->nmr->lam_short_ema : &s->nmr->lam_long_ema; + *ema = *ema > 0.0f ? 0.9f * *ema + 0.1f * lam : lam; + /* sustained-strain floor: snaps down at any comfortable moment, + * recovers only slowly, so bursty content cannot bank pressure + * credit between its lambda valleys. */ + s->nmr->lam_floor = s->nmr->lam_floor > 0.0f ? + fminf(s->nmr->lam_floor * 1.02f, lam) : lam; + } + { /* shared rate-pressure ramp: lambda vs nd-scaled anchors */ + float scale, ramp; + scale = 1.0f + av_clipf(s->nmr->nd_ema / 50.0f, 0.0f, 8.0f); + ramp = s->nmr->lam_long_ema > 0.0f ? + av_clipf((s->nmr->lam_long_ema - 120.0f * scale) / + (350.0f * scale - 120.0f * scale), 0.0f, 1.0f) : 0.0f; + /* transparency veto: lambda*nd below ~74 = comfortable */ + if (s->nmr->nd_ema > 0.0f) + ramp *= av_clipf((s->nmr->lam_long_ema * s->nmr->nd_ema - 60.0f) / + (120.0f - 60.0f), 0.0f, 1.0f); + s->nmr->press = ramp; + } + if (rc_global) { + /* track the centre toward the CONTENT lambda (demand-solved, pressure + * divided out); clamped lambda is rate noise, not content */ + float c = s->nmr->lam_rc * powf(lam_dem / rc_off / s->nmr->lam_rc, NMR_RC_TRACK); + s->nmr->lam_rc = av_clipf(c, 1e-6f, 1e4f); + } else if (rc_eligible) { + /* bootstrap the servo off the first substantive frame (silent lead-ins + * have degenerate budgets) */ + int nbnd_max = 0; + for (int k = 0; k < nsl; k++) + nbnd_max = FFMAX(nbnd_max, sl[k]->nbnd); + if (nbnd_max >= 8) { + s->nmr->lam_rc = av_clipf(lam, 1e-4f, 1e4f); + s->nmr->lam_slew = s->nmr->lam_rc; + } + } + + { /* PNS, per channel at the group's operating lambda */ + const float pns_lam = NMR_PNS_LAM; + int pns_total = 0; + for (int k = 0; k < nsl; k++) { + NMRSlot *t = sl[k]; + const float (*ndk)[NMR_NCAND] = (const float (*)[NMR_NCAND])s->nmr->nd[t->si]; + const int (*nbk)[NMR_NCAND] = (const int (*)[NMR_NCAND])s->nmr->nb[t->si]; + int pns_count = 0; + /* band 0 (lowest freq) is kept as the global-gain / sf-chain anchor */ + for (int b = 1; b < t->nbnd; b++) { + int bi = t->bidx[b]; + float spread = t->pspread[bi]; + float nmr_pns, cost_keep, cost_pns, frac; + if (!t->sce->can_pns[bi]) + continue; + + int was = s->nmr->pns_prev[t->cur_ch & 15][bi]; + float bias = was ? NMR_PNS_STAY : NMR_PNS_ENTER; + int want = 0, force_exit = 0; + + /* (can_pns was already checked above; gates below fill `want`) */ + if (t->pener[bi] > NMR_PNS_MAX_ET * t->thr_real[bi]) { + force_exit = 1; /* loud-band guard */ + } else if (lam > pns_lam) { + /* Spectral-hole fill: a noise-like band left mostly empty */ + frac = ndk[b][t->chosen[b]] * t->thr[bi] / FFMAX(t->pener[bi], 1e-9f); + if (spread > NMR_PNS_HOLE_SPREAD && + frac > NMR_PNS_HOLE_FRAC * (was ? 0.7f : 1.0f)) { + want = 1; + } else if (ndk[b][t->chosen[b]] * t->thr[bi] > + NMR_PNS_NDGATE * t->thr_real[bi] * (was ? 0.5f : 1.0f)) { + /* replace only a band coded audibly badly; cost of + * energy-matched noise = its non-noise-like fraction */ + nmr_pns = FFMAX(0.0f, t->pener[bi] * (1.0f - spread*spread)) + / FFMAX(t->thr[bi], 1e-9f); + cost_keep = ndk[b][t->chosen[b]] + lam * nbk[b][t->chosen[b]]; + cost_pns = nmr_pns + lam * NMR_PNS_BITS; + want = cost_pns < cost_keep * bias; + } + } + { /* debounce; near-mask deletion candidates skip entry + * (noise beats the ~silent rendition they'd get) */ + uint8_t *ron = &s->nmr->pns_run_on [t->cur_ch & 15][bi]; + uint8_t *roff = &s->nmr->pns_run_off[t->cur_ch & 15][bi]; + int near = t->pener[bi] < 2.0f * t->thr_real[bi]; + if (want) { if (*ron < 255) (*ron)++; *roff = 0; } + else { if (*roff < 255) (*roff)++; *ron = 0; } + if (force_exit) + want = 0; + else if (!was) + want = near ? want : *ron >= NMR_PNS_ON; + else if (near) + want = 1; /* physics-hysteresis: noise until audible */ + else + want = !(*roff >= NMR_PNS_OFF); + } + if (want) { + t->is_pns[b] = 1; + pns_count++; + } + } + if (pns_count) { + t->nact = 0; + for (int b = 0; b < t->nbnd; b++) + if (!t->is_pns[b]) + t->act[t->nact++] = b; + } + pns_total += pns_count; + } + if (pns_total) { + /* re-solve over the survivors: at fixed lambda the allocation is + * the same except for the repaired sf-delta chain; in bisection + * mode re-spend the freed budget */ + if (rc_global) + nmr_eval_slots(s, sl, nsl, NMR_STEP, lam); + else + nmr_solve_slots(s, sl, nsl, NMR_STEP, destbits - pns_total * NMR_PNS_BITS, + 1e-9f, 1e4f, NMR_ITERS); + } + } + + for (int k = 0; k < nsl; k++) { + NMRSlot *t = sl[k]; + uint8_t *pp = s->nmr->pns_prev[t->cur_ch & 15]; + uint8_t now[128] = {0}; + for (int b = 0; b < t->nbnd; b++) + if (t->is_pns[b]) + now[t->bidx[b]] = 1; + memcpy(pp, now, 128); + } + for (int k = 0; k < nsl; k++) + nmr_commit_channel(s, sl[k]); + +} + +static void search_for_quantizers_nmr(AVCodecContext *avctx, + AACEncContext *s, + SingleChannelElement *sce, + const float lambda) +{ + AACNMRCurves *n = s->nmr; + /* Global-lambda RC: one solve per frame at a servoed centre lambda; the reservoir + * holds the long-run mean rate. Bypassed for VBR (-q:a) and the bootstrap frame. */ + int rc_eligible = !(avctx->flags & AV_CODEC_FLAG_QSCALE) && avctx->bit_rate > 0 && + avctx->bit_rate_tolerance != 0; + /* Signed reservoir; soft steering (bounded repay + rc_off), hard cap = + * legality only. */ + int rc_rate_frame = avctx->bit_rate * 1024.0 / avctx->sample_rate; + int rc_bmax = FFMIN(FFMAX(6144 * s->channels - rc_rate_frame, 256), NMR_CBR_BUF * s->channels); + + int rc_global, defer; + NMRSlot *t; + + s->nmr->counted[s->cur_channel] = 0; + + if (rc_eligible && !n->rc_fill_seeded) { + /* the decoder bit reservoir starts FULL: seed it so the head may frontload */ + n->rc_fill = rc_bmax; + n->rc_fill_seeded = 1; + } + if (rc_eligible && avctx->frame_num != n->rc_frame_num) { + if (n->rc_frame_num > 0 && n->lam_rc > 0.0f) + n->rc_fill = av_clip(n->rc_fill + rc_rate_frame - s->last_frame_pb_count, + -rc_bmax, rc_bmax); + n->rc_frame_num = avctx->frame_num; + n->pending = 0; /* a deferred first channel never crosses a frame */ + /* latch the RC mode per frame: a mid-frame bootstrap must not flip + * the CPE defer logic between channels */ + n->rc_gl = rc_eligible && n->lam_rc > 0.0f; + + /* Transient burst run state: set at run start and held across the run so + * coding stays uniform; repaid from the reservoir's steady stretches. */ + int is_short = sce->ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE; + if (is_short) { + if (!n->prev_was_short) { /* run start */ + if (n->frames_since_short >= NMR_BURST_GAP) { + n->run_burst = NMR_BURST_GAIN; + } else { + /* dense-beat boost, scaled by measured short-frame starvation */ + float imb = 0.0f; + if (n->lam_long_ema > 0.0f && n->lam_short_ema > 0.0f) + imb = av_clipf(n->lam_short_ema / n->lam_long_ema - 1.0f, + 0.0f, 1.0f); + n->run_burst = 1.0f + (NMR_SHORT_BOOST - 1.0f) * imb * + n->frames_since_short / (float)NMR_BURST_GAP; + } + } + n->frames_since_short = 0; + } else { + /* the frame closing a run (the STOP) absorbs the corridor recoil + * of the boosted shorts; give it half the run's factor so the + * repayment spreads into the steady stretch instead */ + n->run_burst = n->prev_was_short ? sqrtf(n->run_burst) : 1.0f; + n->frames_since_short++; + } + n->prev_was_short = is_short; + } + rc_global = rc_eligible && n->rc_gl; + + /* CPE budget pool: under global-lambda RC, defer the pair's first channel + * and solve both against one pooled budget when the second one arrives. */ + defer = n->pair && rc_global; + + t = &n->slot[(defer && n->pending) ? 1 : 0]; + t->si = (defer && n->pending) ? 1 : 0; + + if (!nmr_setup_channel(avctx, s, sce, t)) { + nmr_bail_channel(sce); + t->nbnd = t->nact = 0; + } + + if (defer && !n->pending) { + n->pending = 1; /* wait for the partner channel */ + return; + } + + { + NMRSlot *sl[2]; + int nsl = 0, chans = 1; + if (defer) { + n->pending = 0; + chans = 2; + if (n->slot[0].nact) + sl[nsl++] = &n->slot[0]; + if (n->slot[1].nact) + sl[nsl++] = &n->slot[1]; + } else if (t->nact) { + sl[nsl++] = t; + } + if (!nsl) + return; /* nothing codeable in the group */ + nmr_solve_group(avctx, s, lambda, sl, nsl, chans, + rc_eligible, rc_global, rc_rate_frame, rc_bmax); } } diff --git a/libavcodec/aacenc.h b/libavcodec/aacenc.h index 67069cbdf1..ec04465a6d 100644 --- a/libavcodec/aacenc.h +++ b/libavcodec/aacenc.h @@ -173,9 +173,57 @@ typedef struct AACQuantizeBandCostCacheEntry { /** * NMR coder per-band candidate cost curves (~96 KiB) and rate-control carry-over */ +/** + * Per-channel trellis state for one solve. A channel pair (CPE) is solved + * jointly against a pooled budget: the first channel's setup is stored here + * and committed together with the second channel under one shared lambda. + */ +typedef struct NMRSlot { + struct SingleChannelElement *sce; + int si; ///< curve-bank index (nd/nb slot) + int cur_ch; ///< encoder channel index (psy/cache context) + int nbnd; ///< coded-band count, 0 = nothing codeable + int is8; ///< EIGHT_SHORT frame + int bidx[128]; ///< sce band index (w*16+g) + int bw[128], bg[128], bst[128]; ///< window group, swb, coef start + int blo[128]; ///< finest candidate scalefactor + int bnc[128]; ///< number of candidates + int chosen[128]; + int act[128]; ///< active (non-PNS) band coding order + int nact; + int minsf[128]; + float maxvals[128]; + float thr[128]; ///< allocation-law effective threshold + float thr_real[128]; ///< real masking threshold (PNS gates) + float tnsg[128]; ///< TNS synthesis gain per band for THIS solve (1 = uncovered), M/S-aware (pair max) + float pener[128]; ///< band energy (PNS noise target) + float pspread[128]; ///< band tonality spread (1 = noise) + uint8_t is_pns[128]; ///< band coded as noise +} NMRSlot; + typedef struct AACNMRCurves { - float nd[128][NMR_NCAND]; ///< dist / threshold per candidate - int nb[128][NMR_NCAND]; ///< spectral bits per candidate + float nd[2][128][NMR_NCAND]; ///< dist / threshold per candidate, per pair slot + int nb[2][128][NMR_NCAND]; ///< spectral bits per candidate, per pair slot + NMRSlot slot[2]; ///< pair slots (solo solves use slot 0) + int pair; ///< current element is a CPE: pool the pair budget + int rc_gl; ///< rc_global latched at frame start: the corridor bootstrap must not flip the CPE defer logic between channels of one frame + int rc_fill_seeded; ///< reservoir seeded full at stream start (decoder buffer starts full) + int pending; ///< slot 0 holds a deferred first channel + uint8_t zero_prev[16][128]; ///< per-channel band zero state last frame (zeroing hysteresis) + int zero_nw[16]; ///< window count zero_prev was recorded on + float thr_prev[16][64]; ///< per-channel long-grid law thresholds of the previous frame + uint8_t thr_prev_ok[16]; ///< thr_prev holds a long-frame measurement + uint8_t pns_prev[16][128]; ///< per-channel PNS state last frame (decision hysteresis) + uint8_t pns_run_on[16][128]; ///< consecutive frames the band has WANTED PNS + uint8_t pns_run_off[16][128]; ///< consecutive frames the band has wanted OUT + uint8_t smode[16][128]; ///< per-pair previous stereo mode per band, two banks per pair (long/short grid): each grid's memory persists across the other's frames instead of being wiped at window switches + uint8_t smode_band[8][128]; ///< last decided stereo mode per band index (side-band tests) + uint8_t tns8_prev[16]; ///< short-TNS accepted last frame (per channel): Schmitt state for the accept bar + uint8_t sinit[16]; ///< stereo state bank initialized + int smode_nw[8]; ///< window count the stored modes were decided on + float sema_es[16][128]; ///< smoothed side energy per band (stereo-decision EMA) + float sema_em[16][128]; ///< smoothed mid energy per band + float sema_img[16][128]; ///< smoothed I/S image-error/mask ratio per band float lam[16]; ///< per-channel operating lambda of the previous frame, 0 = none yet int counted[16]; ///< per-channel bits the trellis accounted for in the last solve float side_ema; ///< running estimate of real-minus-counted bits per frame @@ -187,6 +235,12 @@ typedef struct AACNMRCurves { int frames_since_short; ///< long-block frames since the last short run (the "gap"): large = isolated transient int prev_was_short; ///< previous frame was a short block (for run-start detection) float run_burst; ///< transient bit-burst factor, set at run start and held across the short run + float lam_slew; ///< final operating lambda of the previous RC frame (slew-limiter state) + float nd_ema; ///< smoothed achieved distortion/real-mask over long-frame coded bands (1 = at threshold; >>1 flags psy-unreliable noise-class content) + float press; ///< rate-pressure ramp [0,1]: lambda EMA against anchors that scale up when nd_ema flags noise-class content (psy masks unreliable there, lambda reads inflated) + float lam_short_ema; ///< smoothed operating lambda of short frames + float lam_long_ema; ///< smoothed operating lambda of long frames + float lam_floor; ///< lambda min-tracker (snaps down, +2%/frame up): sustained-strain floor; bursty spikes at a comfortable rate cannot raise it } AACNMRCurves; typedef struct AACPCEInfo { -- 2.52.0 From 714d38879c1bf575e77a90882af13ceb96554a74 Mon Sep 17 00:00:00 2001 From: Lynne <[email protected]> Date: Fri, 17 Jul 2026 14:19:15 +0800 Subject: [PATCH 3/7] avcodec/aaccoder: widen PNS candidacy to near-masked bands Bands whose energy sits between threshold/4 and 2x threshold are deletion candidates for the trellis: coded, they round to silence under pressure, and an audible hole is worse than energy-matched noise. Let them qualify for PNS with a relaxed uniformity requirement, so the noise substitution keeps the texture the allocator was about to delete. --- libavcodec/aaccoder.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/libavcodec/aaccoder.c b/libavcodec/aaccoder.c index 7aead4d8b4..086e5e2f0a 100644 --- a/libavcodec/aaccoder.c +++ b/libavcodec/aaccoder.c @@ -664,10 +664,20 @@ static void mark_pns(AACEncContext *s, AVCodecContext *avctx, SingleChannelEleme * 3. on short window groups, all windows have similar energy (variations in energy would be destroyed by PNS) */ sce->pns_ener[w*16+g] = sfb_energy; - if (sfb_energy < threshold*sqrtf(1.5f/freq_boost) || spread < spread_threshold || min_energy < pns_transient_energy_r * max_energy) { - sce->can_pns[w*16+g] = 0; - } else { - sce->can_pns[w*16+g] = 1; + { + /* near-mask PNS class (E in [thr/4, 2*thr]): deletion + * candidates go to noise, not silence (AAC_PNSHOLE) */ + int near = sfb_energy < 2.0f * threshold && + sfb_energy > threshold * 0.25f; + if (near) { + /* deletion candidate: noise beats the ~silent rendition */ + sce->can_pns[w*16+g] = spread >= spread_threshold && + min_energy >= 0.2f * max_energy; + } else if (sfb_energy < threshold*sqrtf(1.5f/freq_boost) || spread < spread_threshold || min_energy < pns_transient_energy_r * max_energy) { + sce->can_pns[w*16+g] = 0; + } else { + sce->can_pns[w*16+g] = 1; + } } } } -- 2.52.0 From 5c351998d12f6039219bd46d495aaa1369b4c15d Mon Sep 17 00:00:00 2001 From: Lynne <[email protected]> Date: Fri, 17 Jul 2026 14:19:35 +0800 Subject: [PATCH 4/7] avcodec/aacenc_tns: enable short-block TNS via pooled per-group filters Per-window filters combined with group-shared scalefactors made short-block TNS produce decoder-synthesized silence: a weak residual in one sub-window rounds to zero under the group's scalefactor and the synthesis filter has nothing to re-amplify, audible as 1.5-3 ms dropouts on beat content. Fit ONE filter per window group instead, on the concatenated weighted spectra, and require every window in the group to pass the prediction-gain bar. Accepts are Schmitt-gated and run-scoped: within a short run the enter bar is raised and the hold bar lowered (accept flicker was audible as gravel), while isolated transient frames - where TNS matters most and cannot flicker - use the base bar. --- libavcodec/aacenc_tns.c | 145 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 143 insertions(+), 2 deletions(-) diff --git a/libavcodec/aacenc_tns.c b/libavcodec/aacenc_tns.c index c1308b1ebf..e7cbd60161 100644 --- a/libavcodec/aacenc_tns.c +++ b/libavcodec/aacenc_tns.c @@ -27,6 +27,7 @@ #include "libavutil/libm.h" #include "aacenc.h" +#include <float.h> #include "aacenc_tns.h" #include "aactab.h" #include "aacenc_utils.h" @@ -123,6 +124,7 @@ void ff_aac_apply_tns(AACEncContext *s, SingleChannelElement *sce) for (w = 0; w < ics->num_windows; w++) { bottom = ics->num_swb; for (filt = 0; filt < tns->n_filt[w]; filt++) { + int b0, e0; top = bottom; bottom = FFMAX(0, top - tns->length[w][filt]); order = tns->order[w][filt]; @@ -132,8 +134,10 @@ void ff_aac_apply_tns(AACEncContext *s, SingleChannelElement *sce) // tns_decode_coef compute_lpc_coefs(tns->coef[w][filt], 0, order, lpc, 0, 0, 0, NULL); - start = ics->swb_offset[FFMIN(bottom, mmm)]; - end = ics->swb_offset[FFMIN( top, mmm)]; + b0 = FFMIN(bottom, mmm); + e0 = FFMIN( top, mmm); + start = ics->swb_offset[b0]; + end = ics->swb_offset[e0]; if ((size = end - start) <= 0) continue; if (tns->direction[w][filt]) { @@ -150,6 +154,7 @@ void ff_aac_apply_tns(AACEncContext *s, SingleChannelElement *sce) sce->coeffs[start] += lpc[i-1]*hist[start - i*inc]; } } + } } } @@ -171,6 +176,108 @@ static inline void quantize_coefs(double *coef, int *idx, float *lpc, int order, /* * 3 bits per coefficient with 8 short windows */ +/* Short blocks, pooled per group: one filter per scalefactor group so the + * shared sf sees uniform residuals (per-window filters caused silent + * sub-windows); all-or-none accept. */ +static void search_for_tns_short_pooled(AACEncContext *s, SingleChannelElement *sce) +{ + TemporalNoiseShaping *tns = &sce->tns; + const int mmm = tns_max_nonpns(sce, FFMIN(sce->ics.tns_max_bands, sce->ics.max_sfb ? sce->ics.max_sfb : sce->ics.num_swb)); + const int sfb_start = av_clip(tns_min_sfb[1][s->samplerate_index], 0, mmm); + const int sfb_end = av_clip(sce->ics.num_swb, 0, mmm); + const int c_bits = TNS_Q_BITS_IS8 == 4; + int count = 0; + FFPsyBand *const psy_bands = &s->psy.ch[s->cur_channel].psy_bands[0]; + + memset(tns, 0, sizeof(*tns)); + if (sfb_end - sfb_start <= 0) + return; + const int c_lo = sce->ics.swb_offset[sfb_start]; + const int c_hi = sce->ics.swb_offset[sfb_end]; + const int clen = c_hi - c_lo; + const int ord_g = 7; + if (clen <= 2*ord_g) + return; + + for (int wh = 0; wh < sce->ics.num_windows; wh += sce->ics.group_len[wh]) { + int gl = sce->ics.group_len[wh]; + double coefs[MAX_LPC_ORDER]; + float pooled[1024], lpc_q[TNS_MAX_ORDER]; + float gain, gmin; + int ok = 1; + + /* per-window weighted spectra, concatenated over the group */ + for (int w2 = 0; w2 < gl; w2++) { + int w = wh + w2; + float maxrms = 0.0f, floorrms; + for (int g = sfb_start; g < sfb_end; g++) { + int s0 = sce->ics.swb_offset[g], s1 = sce->ics.swb_offset[g+1]; + float rms = sqrtf(FFMAX(psy_bands[w*16 + g].threshold, 0.0f) / FFMAX(s1 - s0, 1)); + maxrms = FFMAX(maxrms, rms); + } + floorrms = FFMAX(maxrms * TNS_WEIGHT_FLOOR, 1e-9f); + for (int g = sfb_start; g < sfb_end; g++) { + int s0 = sce->ics.swb_offset[g], s1 = sce->ics.swb_offset[g+1]; + float rms = sqrtf(FFMAX(psy_bands[w*16 + g].threshold, 0.0f) / FFMAX(s1 - s0, 1)); + float wgt = 1.0f / FFMAX(rms, floorrms); + for (int k = s0; k < s1; k++) + pooled[w2*clen + (k - c_lo)] = sce->coeffs[w*128 + k] * wgt; + } + } + + gain = ff_lpc_calc_ref_coefs_f(&s->lpc, pooled, clen*gl, ord_g, coefs, 0); + if (!isfinite(gain) || gain < TNS_PREDGAIN_GATE || gain > TNS_PG_CLAMP) + continue; + for (int i = 0; i < ord_g; i++) + coefs[i] = -coefs[i]; + + quantize_coefs(coefs, tns->coef_idx[wh][0], tns->coef[wh][0], ord_g, c_bits); + compute_lpc_coefs(tns->coef[wh][0], 0, ord_g, lpc_q, 0, 0, 0, NULL); + + /* every window must clear the measured post-quantization bar */ + gmin = FLT_MAX; + for (int w2 = 0; w2 < gl; w2++) { + const float *msrc = pooled + w2*clen; + float orig_e = 0.0f, filt_e = 0.0f; + for (int m = 0; m < clen; m++) { + float acc = msrc[m]; + for (int i = 1; i <= FFMIN(m, ord_g); i++) + acc += lpc_q[i-1] * msrc[m - i]; + orig_e += msrc[m]*msrc[m]; + filt_e += acc*acc; + } + gmin = FFMIN(gmin, orig_e / FFMAX(filt_e, 1e-9f)); + } + { + /* accept Schmitt, run-scoped: hard entry / easy hold inside + * short runs (anti-gravel); isolated frames use the base bar */ + int in_run = s->nmr ? s->nmr->prev_was_short : 0; + int prev_on = s->nmr ? s->nmr->tns8_prev[s->cur_channel & 15] : 0; + float bar = TNS_PG_C1_SHORT * (!in_run ? 1.0f : prev_on ? 0.5f : 1.8f); + if (gmin < bar) + ok = 0; + } + + if (ok) { + for (int w2 = 0; w2 < gl; w2++) { + int w = wh + w2; + tns->n_filt[w] = 1; + tns->length[w][0] = sfb_end - sfb_start; + tns->order[w][0] = ord_g; + tns->direction[w][0] = 0; + if (w2) { + memcpy(tns->coef_idx[w][0], tns->coef_idx[wh][0], sizeof(tns->coef_idx[w][0])); + memcpy(tns->coef[w][0], tns->coef[wh][0], sizeof(tns->coef[w][0])); + } + count++; + } + } + } + sce->tns.present = !!count; + if (s->nmr) + s->nmr->tns8_prev[s->cur_channel & 15] = !!count; +} + void ff_aac_search_for_tns(AACEncContext *s, SingleChannelElement *sce) { TemporalNoiseShaping *tns = &sce->tns; @@ -197,11 +304,17 @@ void ff_aac_search_for_tns(AACEncContext *s, SingleChannelElement *sce) sce->tns.present = 0; return; } + if (is8) { + search_for_tns_short_pooled(s, sce); + return; + } /* time-domain window length backing one coding window: a long MDCT block is * fed 2048 windowed samples (current 1024 + overlap), each short block 256. */ const int tlen = is8 ? 256 : 2048; + float mgain[8] = {0}; + for (w = 0; w < sce->ics.num_windows; w++) { int filt, any = 0; @@ -302,11 +415,39 @@ void ff_aac_search_for_tns(AACEncContext *s, SingleChannelElement *sce) tns->order[w][filt] = ord_g; tns->direction[w][filt] = dir; + mgain[w] = orig_e / filt_e; any = 1; } tns->n_filt[w] = any ? n_filt : 0; if (any) count++; } + + /* per-window path: group-uniformity gate (mismatched whitening within a + * shared-sf group silences sub-windows) */ + if (is8 && count) { + const float gspread = 2.0f; + count = 0; + for (w = 0; w < sce->ics.num_windows; w += sce->ics.group_len[w]) { + int gl = sce->ics.group_len[w], drop = 0; + float gmin = FLT_MAX, gmax = 0.0f; + for (int w2 = w; w2 < w + gl; w2++) { + if (!tns->n_filt[w2] || mgain[w2] <= 0.0f) { drop = 1; break; } + gmin = FFMIN(gmin, mgain[w2]); + gmax = FFMAX(gmax, mgain[w2]); + } + if (!drop && gmax > gspread * gmin) + drop = 1; + for (int w2 = w; w2 < w + gl; w2++) { + if (drop) { + tns->n_filt[w2] = 0; + for (int f2 = 0; f2 < n_filt; f2++) + tns->order[w2][f2] = 0; + } else if (tns->n_filt[w2]) { + count++; + } + } + } + } sce->tns.present = !!count; } -- 2.52.0 From e362e243b8f46eb1faeb7441d3efe50b979e6bd4 Mon Sep 17 00:00:00 2001 From: Lynne <[email protected]> Date: Fri, 17 Jul 2026 14:19:52 +0800 Subject: [PATCH 5/7] avcodec/aacpsy: sync block switching across channel pairs Implement the window_pair() hook: detect both channels of a CPE, then decide together - if either channel attacks, both switch, with one merged attack map so the grouping matches across the pair and common_window survives transients. Pairs the encoder has flagged as decoupled (joint tools dead) keep independent per-channel decisions, where forcing the steady channel short at every event in the other only costs quality. Attacks must also be NOVEL against the trailing high-pass max-envelope: a steady pitch-pulse train repeats its peak every period and re-fires the ratio test forever, producing long false short runs on periodic content (low brass and the like); a genuine onset towers over the recent past. The pre-echo threshold relaxation for gentler isolated onsets is kept, and START frames entering a short sequence clamp their thresholds toward the previous long frame's per-band values on the mapped long grid. --- libavcodec/aacpsy.c | 146 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 136 insertions(+), 10 deletions(-) diff --git a/libavcodec/aacpsy.c b/libavcodec/aacpsy.c index e38c43a323..e0faa2a19a 100644 --- a/libavcodec/aacpsy.c +++ b/libavcodec/aacpsy.c @@ -143,6 +143,9 @@ typedef struct AacPsyChannel{ int prev_attack; ///< attack value for the last short block in the previous sequence int next_attack0_zero; ///< whether attack[0] of the next frame is zero int frames_since_short; ///< consecutive long frames (pre-echo-aware isolated-onset gate) + float prev_frame_energy; ///< previous frame's full-band lookahead energy (attack veto) + int64_t win_count; ///< window() calls so far (frame counter for pair sync) + int64_t last_att; ///< win_count value of this channel's last own attack /* rate-loop re-analysis rewind state, see psy_3gpp_analyze() */ int64_t rc_frame_num; ///< frame this channel last saved rewind state for @@ -677,6 +680,23 @@ static void psy_3gpp_analyze_channel(FFPsyContext *ctx, int channel, float pe = pctx->chan_bitrate > 32000 ? 0.0f : FFMAX(50.0f, 100.0f - pctx->chan_bitrate * 100.0f / 32000.0f); const int num_bands = ctx->num_bands[wi->num_windows == 8]; const uint8_t *band_sizes = ctx->bands[wi->num_windows == 8]; + uint8_t s2l[16] = {0}; + int start_after_long = wi->num_windows == 8 && + wi->window_type[1] == LONG_START_SEQUENCE; + { /* short->long grid band map for cross-transition pre-echo control */ + if (start_after_long) { + const uint8_t *ls = ctx->bands[0]; + const int ln = ctx->num_bands[0]; + const uint8_t *ss = ctx->bands[1]; + int lacc = 0, sacc = 0, gl = 0; + for (int gs = 0; gs < num_bands && gs < 16; gs++) { + int center8 = (sacc + ss[gs] / 2) * 8; + while (gl < ln - 1 && lacc + ls[gl] <= center8) { lacc += ls[gl]; gl++; } + s2l[gs] = gl; + sacc += ss[gs]; + } + } + } AacPsyCoeffs *coeffs = pctx->psy_coef[wi->num_windows == 8]; const float avoid_hole_thr = wi->num_windows == 8 ? PSY_3GPP_AH_THR_SHORT : PSY_3GPP_AH_THR_LONG; const int bandwidth = ctx->cutoff ? ctx->cutoff : AAC_CUTOFF(ctx->avctx); @@ -708,6 +728,12 @@ static void psy_3gpp_analyze_channel(FFPsyContext *ctx, int channel, if (!(wi->window_type[0] == LONG_STOP_SEQUENCE || (!w && wi->window_type[1] == LONG_START_SEQUENCE))) band->thr = FFMAX(PSY_3GPP_RPEMIN*band->thr, FFMIN(band->thr, PSY_3GPP_RPELEV*pch->prev_band[w+g].thr_quiet)); + else if (!w && start_after_long) + /* w0 after a START frame: grid-mapped, scaled continuity + * clamp instead of the spec's skip (cannot bind on noise + * content - see memory - but correct for tonal) */ + band->thr = FFMAX(PSY_3GPP_RPEMIN*band->thr, FFMIN(band->thr, + PSY_3GPP_RPELEV*pch->prev_band[s2l[FFMIN(g,15)]].thr / 8.0f)); /* 5.6.1.3.1 "Preparatory steps of the perceptual entropy calculation" */ pe += calc_pe_3gpp(band); @@ -925,16 +951,16 @@ static void lame_apply_block_type(AacPsyChannel *ctx, FFPsyWindowInfo *wi, int u ctx->next_window_seq = blocktype; } -static FFPsyWindowInfo psy_lame_window(FFPsyContext *ctx, const float *audio, - const float *la, int channel, int prev_type) +/* Attack detection half of the LAME window decision: everything up to (and + * excluding) the block-type state machine. Fills attacks[] and returns the raw + * uselongblock; mutates only the detection history. Split out so a channel + * pair can be detected first and DECIDED together (synced block switching). */ +static int psy_lame_detect(AacPsyContext *pctx, AacPsyChannel *pch, + const float *la, int channel, int prev_type, + int attacks[AAC_NUM_BLOCKS_SHORT + 1]) { - AacPsyContext *pctx = (AacPsyContext*) ctx->model_priv_data; - AacPsyChannel *pch = &pctx->ch[channel]; - int grouping = 0; int uselongblock = 1; - int attacks[AAC_NUM_BLOCKS_SHORT + 1] = { 0 }; int i; - FFPsyWindowInfo wi = { { 0 } }; if (la) { float hpfsmpl[AAC_BLOCK_SIZE_LONG]; @@ -975,8 +1001,15 @@ static FFPsyWindowInfo psy_lame_window(FFPsyContext *ctx, const float *audio, attack_intensity[i + PSY_LAME_NUM_SUBBLOCKS] = p; } - { /* pre-echo-aware threshold relaxation, see PSY_LAME_PE_* */ + { /* pre-echo-aware threshold relaxation + periodicity/novelty veto (a + * pitch-pulse train repeats its peak; a real onset towers) */ float frame_peak = 1.0f; + float eh[8 + (AAC_NUM_BLOCKS_SHORT + 1) * PSY_LAME_NUM_SUBBLOCKS]; + const float nov_gate = 1.25f; + for (i = 0; i < 8; i++) + eh[i] = pch->prev_energy_subshort[8 + i]; + for (i = 0; i < (AAC_NUM_BLOCKS_SHORT + 1) * PSY_LAME_NUM_SUBBLOCKS; i++) + eh[8 + i] = energy_subshort[i]; for (i = PSY_LAME_NUM_SUBBLOCKS; i < (AAC_NUM_BLOCKS_SHORT + 1) * PSY_LAME_NUM_SUBBLOCKS; i++) frame_peak = FFMAX(frame_peak, energy_subshort[i]); for (i = 0; i < (AAC_NUM_BLOCKS_SHORT + 1) * PSY_LAME_NUM_SUBBLOCKS; i++) @@ -986,8 +1019,16 @@ static FFPsyWindowInfo psy_lame_window(FFPsyContext *ctx, const float *audio, pch->frames_since_short >= PSY_LAME_PE_GAP && energy_subshort[i - PSY_LAME_NUM_SUBBLOCKS] < PSY_LAME_PE_QUIET * frame_peak) thr *= PSY_LAME_PE_RED; - if (attack_intensity[i] > thr) + if (attack_intensity[i] > thr) { + if (nov_gate > 0.0f && i >= PSY_LAME_NUM_SUBBLOCKS) { + float prevmax = 1.0f; + for (int k = 3; k <= 8; k++) + prevmax = FFMAX(prevmax, eh[8 + i - k]); + if (energy_subshort[i] < nov_gate * prevmax) + continue; /* periodic, not an onset */ + } attacks[i / PSY_LAME_NUM_SUBBLOCKS] = (i % PSY_LAME_NUM_SUBBLOCKS) + 1; + } } } @@ -1009,6 +1050,26 @@ static FFPsyWindowInfo psy_lame_window(FFPsyContext *ctx, const float *audio, att_sum += attacks[i]; } + { /* novelty of each attacking sub-block against the trailing HP + * max-envelope (~1-2 pitch periods): a periodic pulse train repeats + * its peak every period (novelty ~1), a genuine onset towers over + * the recent past. Instrumentation only. */ + float eh[8 + (AAC_NUM_BLOCKS_SHORT + 1) * PSY_LAME_NUM_SUBBLOCKS]; + float novmax = 0.0f; + for (i = 0; i < 8; i++) + eh[i] = pch->prev_energy_subshort[8 + i]; + for (i = 0; i < (AAC_NUM_BLOCKS_SHORT + 1) * PSY_LAME_NUM_SUBBLOCKS; i++) + eh[8 + i] = energy_subshort[i]; + for (i = PSY_LAME_NUM_SUBBLOCKS; i < (AAC_NUM_BLOCKS_SHORT + 1) * PSY_LAME_NUM_SUBBLOCKS; i++) { + if (attacks[i / PSY_LAME_NUM_SUBBLOCKS] == (i % PSY_LAME_NUM_SUBBLOCKS) + 1) { + float prevmax = 1.0f; + for (int k = 3; k <= 8; k++) + prevmax = FFMAX(prevmax, eh[8 + i - k]); + novmax = FFMAX(novmax, eh[8 + i] / prevmax); + } + } + } + if (pch->next_attack0_zero) attacks[0] = 0; pch->next_attack0_zero = !attacks[AAC_NUM_BLOCKS_SHORT]; @@ -1028,11 +1089,26 @@ static FFPsyWindowInfo psy_lame_window(FFPsyContext *ctx, const float *audio, attacks[i] = 0; } - pch->frames_since_short = uselongblock ? pch->frames_since_short + 1 : 0; } else { /* We have no lookahead info, so just use same type as the previous sequence. */ uselongblock = !(prev_type == EIGHT_SHORT_SEQUENCE); } + return uselongblock; +} + +/* Decision half: the block-type state machine and window/grouping fill, + * given the (possibly pair-synced) final uselongblock. */ +static FFPsyWindowInfo psy_lame_apply(AacPsyContext *pctx, AacPsyChannel *pch, + int uselongblock, + const int attacks[AAC_NUM_BLOCKS_SHORT + 1], + int prev_type, int have_la) +{ + int grouping = 0; + int i; + FFPsyWindowInfo wi = { { 0 } }; + + if (have_la) + pch->frames_since_short = uselongblock ? pch->frames_since_short + 1 : 0; lame_apply_block_type(pch, &wi, uselongblock); @@ -1077,6 +1153,55 @@ static FFPsyWindowInfo psy_lame_window(FFPsyContext *ctx, const float *audio, return wi; } +static FFPsyWindowInfo psy_lame_window(FFPsyContext *ctx, const float *audio, + const float *la, int channel, int prev_type) +{ + AacPsyContext *pctx = (AacPsyContext*) ctx->model_priv_data; + AacPsyChannel *pch = &pctx->ch[channel]; + int attacks[AAC_NUM_BLOCKS_SHORT + 1] = { 0 }; + int uselongblock = psy_lame_detect(pctx, pch, la, channel, prev_type, attacks); + + return psy_lame_apply(pctx, pch, uselongblock, attacks, prev_type, !!la); +} + +/* Pair-synced block switching: either channel's attack switches both. */ +static void psy_lame_window_pair(FFPsyContext *ctx, + const float *audio0, const float *la0, + const float *audio1, const float *la1, + int channel0, int channel1, + int prev_type0, int prev_type1, + FFPsyWindowInfo wi[2]) +{ + AacPsyContext *pctx = (AacPsyContext*) ctx->model_priv_data; + AacPsyChannel *pch0 = &pctx->ch[channel0]; + AacPsyChannel *pch1 = &pctx->ch[channel1]; + int att0[AAC_NUM_BLOCKS_SHORT + 1] = { 0 }; + int att1[AAC_NUM_BLOCKS_SHORT + 1] = { 0 }; + int merged[AAC_NUM_BLOCKS_SHORT + 1]; + int u0 = psy_lame_detect(pctx, pch0, la0, channel0, prev_type0, att0); + int u1 = psy_lame_detect(pctx, pch1, la1, channel1, prev_type1, att1); + int u = u0 && u1; + + if (ctx->pair_decoupled[(channel0 >> 1) & 15]) { + /* Joint tools are dead on this pair (encoder-fed state): each channel + * windows for ITS transients - divergence costs nothing there, while + * union-syncing forces the steady channel short at every event in + * the other. Correlated content keeps the sync. */ + wi[0] = psy_lame_apply(pctx, pch0, u0, att0, prev_type0, !!la0); + wi[1] = psy_lame_apply(pctx, pch1, u1, att1, prev_type1, !!la1); + return; + } + + /* One merged attack map for both channels: the grouping (and with it + * common_window) must match across the pair, and the group boundary + * should isolate the first attack heard in EITHER channel. */ + for (int i = 0; i < AAC_NUM_BLOCKS_SHORT + 1; i++) + merged[i] = att0[i] ? att0[i] : att1[i]; + + wi[0] = psy_lame_apply(pctx, pch0, u, merged, prev_type0, !!la0); + wi[1] = psy_lame_apply(pctx, pch1, u, merged, prev_type1, !!la1); +} + const FFPsyModel ff_aac_psy_model = { .name = "3GPP TS 26.403-inspired model", @@ -1084,4 +1209,5 @@ const FFPsyModel ff_aac_psy_model = .window = psy_lame_window, .analyze = psy_3gpp_analyze, .end = psy_3gpp_end, + .window_pair = psy_lame_window_pair, }; -- 2.52.0 From 1143476b1ae28e9efa5fa9550c4a58fa896787bb Mon Sep 17 00:00:00 2001 From: Lynne <[email protected]> Date: Fri, 17 Jul 2026 14:20:18 +0800 Subject: [PATCH 6/7] avcodec/aacenc: rework NMR stereo decisions, retune coded bandwidth Stereo decisions are made per band before quantization, from the psy model's spectra, and carry cross-frame memory (EMA-smoothed statistics, per-grid mode banks, leave-hysteresis) so the image holds instead of churning: * M/S adopts content-driven and rate-free (side under half the mid); for mid-dominant bands M/S is simply the better coding at every rate, and a wandering L/R fraction reads as image instability. * I/S competes with M/S above 6.1 kHz instead of only seeing M/S rejects (which are exactly the wide bands it cannot render), and engages under SUSTAINED strain only: the pressure ramp gated by the lambda floor, so pressure spikes at a comfortable operating point cannot flood it onto content where coding the side is affordable and strictly better. Unengaged candidates fall back to M/S. * Pairs whose joint-tool candidacy fraction stays low decouple: block switching goes per-channel and M/S stops, matching how independent coding wins on diffuse decorrelated content. * PNS in a pair is reserved for clearly-wide bands (it renders uncorrelated noise per channel). Block switching for a pair is decided through the psy window_pair() hook so common_window survives transients. The NMR rate-to-bandwidth table is retuned upward at >= 48 kbps/ch to track the bandwidth strong encoders deliver; both quality metrics improve on a 16-clip battery and wider regresses. Lower rates are unchanged. --- libavcodec/aacenc.c | 237 +++++++++++++++++++++++++++++--------------- 1 file changed, 156 insertions(+), 81 deletions(-) diff --git a/libavcodec/aacenc.c b/libavcodec/aacenc.c index 6fc9254ae8..3ace4067dc 100644 --- a/libavcodec/aacenc.c +++ b/libavcodec/aacenc.c @@ -577,36 +577,31 @@ static void apply_intensity_stereo(ChannelElement *cpe) } } -/* Intensity stereo is only allowed when its irreducible image error */ -#define NMR_IS_IMG_GATE 0.5f +/* I/S acceptance level for the image-error EMA at full rate pressure */ +#define NMR_IS_IMG_GATE 8000.0f /* Frequency in Hz for the lower limit of intensity stereo */ #define NMR_IS_LOW_LIMIT 6100 -/* Rate ceiling (bits/sample/channel) above which intensity is skipped, ~145kbps */ -#define NMR_IS_MAXBPS 1.52f - -/* The rate ceiling is lifted on hard-to-code frames. The signal is the bit - * reservoir going into deficit: a negative fill means the trellis is spending - * more than the nominal rate to hold quality (operating lambda has climbed). */ -#define NMR_IS_FILLGAIN 0.27f -#define NMR_IS_FILLMAX 0.40f - -/* M/S thresholds: a band is recoded as mid+side when the side is negligible */ -#define NMR_MS_EQUIV 0.01f +/* M/S adoption: es < 0.5*em, content-driven and rate-free */ +#define NMR_MS_EQUIV 0.5f #define NMR_MS_MASK 0.0f -/* PNS-stereo decorrelation gate: a band may be noise-substituted in a CPE only if its - * side energy is at least this fraction of its mid energy, i.e. the image is genuinely - * wide (channels decorrelated). PNS renders uncorrelated noise per channel, so it only - * preserves the image on already-wide bands; a much stricter bar than I/S (which can - * collapse correlated bands). Lower = more PNS / more imaging risk. */ +/* Pair decouple threshold on the joint-tool candidacy fraction EMA: pairs + * whose joint tools are mostly dead (diffuse decorrelated content) window + * per-channel and skip M/S; recouple above 1.3x. */ +#define NMR_DECORR_LO 0.20f + +/* Stereo-decision hysteresis: leaving a joint mode costs a margin. */ +#define NMR_STICKY 2.0f + +/* Decision statistics are EMA-smoothed across frames. */ +#define NMR_SDEC_EMA 0.75f + +/* PNS-stereo gate: substitute only clearly-decorrelated (wide) bands. */ #define NMR_PNS_STEREO_DECORR 0.6f -/* Recode one band's window group as mid+side in place, updating the psy band - * energies/thresholds to the M/S spectra. The threshold is halved as a coarse guard - * against L/R unmasking of the independently-quantized M/S noise (M/S is a lossless - * rotation but lossy coding). Used for the M/S decision and the intensity fallback. */ +/* Recode one band's window group as mid+side in place. */ static void nmr_apply_ms_band(AACEncContext *s, ChannelElement *cpe, int w, int g, int start, int len, int gl) { @@ -624,25 +619,22 @@ static void nmr_apply_ms_band(AACEncContext *s, ChannelElement *cpe, R[i] = m - R[i]; L[i] = m; em += L[i]*L[i]; es += R[i]*R[i]; } - b0->threshold = b1->threshold = FFMIN(b0->threshold, b1->threshold) * 0.5f; + b0->threshold = FFMIN(b0->threshold, b1->threshold) * 0.5f; + b1->threshold = b0->threshold; b0->energy = em; b1->energy = es; } } -/* Intensity-stereo perceptual test for one band's window group: collapse the pair - * to a single carrier (L + p*R)*scale that the decoder rescales per channel, and - * check that the irreducible image error, which no bit budget can reduce, is - * masked in both channels. On success returns 1 and fills the carrier scale, the - * decoder's R/carrier ratio sr_, and the phase p. The caller restricts this to HF - * bands with energy in both channels. */ +/* I/S perceptual test: reconstruction image error vs the pair's masks. */ static int nmr_is_image_masked(AACEncContext *s, ChannelElement *cpe, int w, int g, int start, int len, int gl, float ener0, float ener1, float dot, - float minthr0, float minthr1, + float minthr0, float minthr1, float *ratio_out, float *scale_out, float *sr_out, int *p_out) { int p = dot >= 0.0f ? 1 : -1; float ener01 = ener0 + ener1 + 2*p*dot; /* energy of L + p*R */ + *ratio_out = FLT_MAX; if (ener01 <= FLT_MIN) return 0; float scale = sqrtf(ener0 / ener01); /* carrier = (L + p*R)*scale */ @@ -657,9 +649,8 @@ static int nmr_is_image_masked(AACEncContext *s, ChannelElement *cpe, img0 += dl*dl; img1 += dr*dr; } } - if (img0 >= NMR_IS_IMG_GATE * minthr0 * gl || - img1 >= NMR_IS_IMG_GATE * minthr1 * gl) - return 0; + *ratio_out = FFMAX(img0 / FFMAX(minthr0 * gl, FLT_MIN), + img1 / FFMAX(minthr1 * gl, FLT_MIN)); *scale_out = scale; *sr_out = sr_; *p_out = p; return 1; } @@ -704,31 +695,36 @@ static void nmr_decide_stereo(AACEncContext *s, ChannelElement *cpe) IndividualChannelStream *ics = &sce0->ics; const AVCodecContext *avctx = s->psy.avctx; const float freq_mult = avctx->sample_rate / (1024.0f / ics->num_windows) / 2.0f; - const float bps = avctx->bit_rate > 0 ? - (float)avctx->bit_rate / avctx->sample_rate / avctx->ch_layout.nb_channels : 0.0f; int is_count = 0; - /* Stereo decision, with no bitrate dependence. Start from full L/R and depart from - * it only where the change is inaudible. M/S and I/S differ in what they trade: - * M/S recodes the pair as mid+side -- an invertible rotation, but the M and S - * are quantized independently, so it is lossy coding whose noise un-mixes - * back to L/R. Used where it barely changes the result (the side is - * negligible vs the mid, so it is ~equivalent to L/R at the same rate) -- - * OR where the doubled side energy is masked. - * I/S drops the side phase and keeps its energy, where the residual image error - * is masked. Used for the decorrelated HF that M/S cannot help. - * Both tests are content/perceptual and frame-stable, so the image holds. */ + if (s->nmr) { + int pi = (s->cur_channel >> 1) & 7; + pi = pi * 2 + (ics->num_windows == 8); /* per-grid state bank */ + if (!s->nmr->sinit[pi]) { + /* one-time init; per-grid banks persist across window switches + * (wiping them churned stereo modes audibly) */ + memset(s->nmr->smode[pi], 0, sizeof(s->nmr->smode[pi])); + for (int b = 0; b < 128; b++) { + s->nmr->sema_em[pi][b] = 0.0f; + s->nmr->sema_img[pi][b] = -1.0f; + } + s->nmr->sinit[pi] = 1; + } + } - /* I/S rate gate: eligible at/below ~128 kbps, with the ceiling lifted on hard - * frames (bit reservoir in deficit) so a starved high-rate passage can still - * call on intensity. Where an I/S candidate is found but IS is not eligible, fall - * back to M/S: not free, but ~equivalent to L/R there and it lets the energy - * compact into the mid. */ - const float rate_frame = avctx->bit_rate * 1024.0f / FFMAX(avctx->sample_rate, 1); - const float deficit = (s->nmr && rate_frame > 0.0f) - ? FFMAX(0.0f, -(float)s->nmr->rc_fill / rate_frame) : 0.0f; - const float is_bonus = FFMIN(NMR_IS_FILLMAX, NMR_IS_FILLGAIN * deficit); - const int allow_is = s->options.intensity_stereo && bps < NMR_IS_MAXBPS + is_bonus; + /* Per-band stereo decision (L/R vs M/S vs I/S), made pre-quantization from + * the psy model so the trellis allocates on the coded spectra. */ + + /* I/S engages under SUSTAINED strain only: rate pressure gated by the + * lambda floor (pressure spikes at a comfortable operating point must + * not admit it). Unengaged candidates fall back to M/S. */ + float is_ramp = s->nmr ? s->nmr->press * + av_clipf((s->nmr->lam_floor - 40.0f) / (120.0f - 40.0f), 0.0f, 1.0f) : 0.0f; + const int allow_is = s->options.intensity_stereo && is_ramp > 0.0f; + + const int pidx = (s->cur_channel >> 1) & 15; + const int decoupled = s->psy.pair_decoupled[pidx]; + int njoint = 0, nbands = 0; /* joint-tool candidacy census, decouple feed */ for (int w = 0; w < ics->num_windows; w += ics->group_len[w]) { int start = 0; @@ -758,39 +754,97 @@ static void nmr_decide_stereo(AACEncContext *s, ChannelElement *cpe) } float thr_g = FFMIN(minthr0, minthr1) * gl; /* group masking budget */ - /* PNS-stereo reservation. Reserve a band for noise substitution only if it - * is noise-like in both channels (intersected can_pns) and clearly - * decorrelated (wide image). */ - if (cpe->ch[0].can_pns[w*16+g] && cpe->ch[1].can_pns[w*16+g] && - es_tot > NMR_PNS_STEREO_DECORR * em_tot) - continue; + /* PNS-stereo reservation: keep clearly-wide noise bands for PNS. */ + const int sidx = w*16+g; + { + float es_w = es_tot, em_w = em_tot; + if (s->nmr) { + int pi_ = ((s->cur_channel >> 1) & 7) * 2 + (cpe->ch[0].ics.num_windows == 8); + float pe = s->nmr->sema_es[pi_][sidx]; + float pm = s->nmr->sema_em[pi_][sidx]; + if (pm > 0.0f) { + es_w = NMR_SDEC_EMA * pe + (1.0f - NMR_SDEC_EMA) * es_tot; + em_w = NMR_SDEC_EMA * pm + (1.0f - NMR_SDEC_EMA) * em_tot; + } + } + if (cpe->ch[0].can_pns[w*16+g] && cpe->ch[1].can_pns[w*16+g] && + es_w > NMR_PNS_STEREO_DECORR * em_w) + continue; + } cpe->ch[0].can_pns[w*16+g] = cpe->ch[1].can_pns[w*16+g] = 0; - int ms_ok = s->options.mid_side && - (s->options.mid_side == 1 || - es_tot < NMR_MS_EQUIV * em_tot || - es_tot < NMR_MS_MASK * thr_g); - float scale, sr_; int p; - int is_ok = !ms_ok && - start * freq_mult > NMR_IS_LOW_LIMIT && - ener0 > FLT_MIN && ener1 > FLT_MIN && - nmr_is_image_masked(s, cpe, w, g, start, len, gl, - ener0, ener1, dot, minthr0, minthr1, - &scale, &sr_, &p); + int pi = ((s->cur_channel >> 1) & 7) * 2 + (cpe->ch[0].ics.num_windows == 8); + uint8_t *pmode = s->nmr ? s->nmr->smode[pi] : NULL; + int prev = pmode ? pmode[sidx] : 0; + float eqgate = NMR_MS_EQUIV * (prev == 1 ? 1.5f : 1.0f); /* stay-until es>0.75em */ + /* I/S = lossy economy: image-error budget scales with pressure */ + float imgate = NMR_IS_IMG_GATE * is_ramp * (prev == 2 ? NMR_STICKY : 1.0f); + float es_d = es_tot, em_d = em_tot; + if (s->nmr) { + float *ees = &s->nmr->sema_es[pi][sidx]; + float *eem = &s->nmr->sema_em[pi][sidx]; + if (*eem <= 0.0f) { *ees = es_tot; *eem = em_tot; } + else { + *ees = NMR_SDEC_EMA * *ees + (1.0f - NMR_SDEC_EMA) * es_tot; + *eem = NMR_SDEC_EMA * *eem + (1.0f - NMR_SDEC_EMA) * em_tot; + } + es_d = *ees; em_d = *eem; + } + int ms_would = s->options.mid_side && + (s->options.mid_side == 1 || + es_d < eqgate * em_d || + es_tot < NMR_MS_MASK * thr_g); + int ms_ok = ms_would && !decoupled; + float scale, sr_, imgratio; int p; + /* I/S competes with M/S above the frequency limit (candidacy must + * not be gated on !ms_ok - that leaves only unrenderable bands) */ + int is_cand = start * freq_mult > NMR_IS_LOW_LIMIT && + ener0 > FLT_MIN && ener1 > FLT_MIN && + nmr_is_image_masked(s, cpe, w, g, start, len, gl, + ener0, ener1, dot, minthr0, minthr1, + &imgratio, &scale, &sr_, &p); + int is_ok = is_cand; + if (s->nmr && start * freq_mult > NMR_IS_LOW_LIMIT) { + /* smoothed image-error; updated only while candidate (fail-value + * feeding jammed it permanently high) */ + float *eim = &s->nmr->sema_img[pi][sidx]; + if (is_cand) { + /* seed from first measurement; freeze when not candidate */ + if (*eim < 0.0f) *eim = imgratio; + else *eim = NMR_SDEC_EMA * *eim + (1.0f - NMR_SDEC_EMA) * FFMIN(imgratio, 100.0f * NMR_IS_IMG_GATE); + } + is_ok = is_cand && *eim >= 0.0f && *eim < imgate; + } - if (ms_ok) { - nmr_apply_ms_band(s, cpe, w, g, start, len, gl); - } else if (is_ok && allow_is) { + njoint += ms_would || is_ok; nbands++; + if (pmode) { + int m_ = (is_ok && allow_is) ? 2 : ms_ok ? 1 : + (is_ok && s->options.mid_side) ? 1 : 0; + pmode[sidx] = m_; + s->nmr->smode_band[(s->cur_channel >> 1) & 7][w*16+g] = m_; + } + if (is_ok && allow_is) { nmr_apply_is_band(s, cpe, w, g, start, len, gl, scale, sr_, p, ener0, ener1); is_count++; - } else if (is_ok && s->options.mid_side) { + } else if (ms_ok || (is_ok && s->options.mid_side)) { nmr_apply_ms_band(s, cpe, w, g, start, len, gl); } /* else: keep full L/R stereo */ } } cpe->is_mode = !!is_count; + + if (nbands > 0) { + /* Pair joint-tool value, read next frame by the psy pair-synced window + * decision and the M/S candidacy above. Measured as CANDIDACY (not + * adoption) so decoupling cannot starve its own signal and self-lock. */ + float r = (float)njoint / nbands; + float *pj = &s->psy.pair_joint[pidx]; + *pj = *pj > 0.0f ? 0.95f * *pj + 0.05f * r : r; + s->psy.pair_decoupled[pidx] = *pj < + (s->psy.pair_decoupled[pidx] ? 1.3f * NMR_DECORR_LO : NMR_DECORR_LO); + } } static void apply_mid_side_stereo(ChannelElement *cpe) @@ -1043,7 +1097,22 @@ static int aac_encode_frame(AVCodecContext *avctx, AVPacket *avpkt, tag = s->chan_map[i+1]; chans = tag == TYPE_CPE ? 2 : 1; cpe = &s->cpe[i]; - for (ch = 0; ch < chans; ch++) { + { + int wi_paired = 0; + /* Synced pair windows: decide both channels of a CPE together so + * their block switching never diverges (see psy window_pair). */ + if (chans == 2 && tag != TYPE_LFE && s->psy.model->window_pair && frame) { + const float *ov0 = &samples[start_ch][0], *ov1 = &samples[start_ch + 1][0]; + s->psy.model->window_pair(&s->psy, + ov0 + 1024, ov0 + 1024 + 448 + 64, + ov1 + 1024, ov1 + 1024 + 448 + 64, + start_ch, start_ch + 1, + cpe->ch[0].ics.window_sequence[0], + cpe->ch[1].ics.window_sequence[0], + wi); + wi_paired = 1; + } + for (ch = 0; ch < chans; ch++) { int k; float clip_avoidance_factor; sce = &cpe->ch[ch]; @@ -1066,7 +1135,7 @@ static int aac_encode_frame(AVCodecContext *avctx, AVPacket *avpkt, * being used for 11.025kHz to 16kHz sample rates. */ ics->num_swb = s->samplerate_index >= 8 ? 1 : 3; - } else { + } else if (!wi_paired) { wi[ch] = s->psy.model->window(&s->psy, samples2, la, s->cur_channel, ics->window_sequence[0]); } @@ -1124,6 +1193,7 @@ static int aac_encode_frame(AVCodecContext *avctx, AVPacket *avpkt, } avoid_clipping(s, sce); } + } start_ch += chans; } if ((ret = ff_alloc_packet(avctx, avpkt, 8192 * s->channels)) < 0) @@ -1218,6 +1288,10 @@ static int aac_encode_frame(AVCodecContext *avctx, AVPacket *avpkt, s->cur_channel = start_ch; nmr_decide_stereo(s, cpe); } + /* NMR pools the CPE bit budget: both channels of a pair are solved + * jointly under one shared lambda (see aaccoder_nmr.h). */ + if (s->options.coder == AAC_CODER_NMR && s->nmr) + s->nmr->pair = (chans == 2); for (ch = 0; ch < chans; ch++) { s->cur_channel = start_ch + ch; /* NMR PNS is mono-only */ @@ -1587,11 +1661,12 @@ static av_cold int aac_encode_init(AVCodecContext *avctx) (avctx->bit_rate / 2.0f * (s->lambda / 120.f) * 1.5f) : (avctx->bit_rate / avctx->ch_layout.nb_channels); - /* For NMR, the rate to bandwidth conversion was tuned to maximize metrics - * over a variable cutoff x bitrate combo */ + /* For NMR, the rate to bandwidth conversion was tuned to maximize + * metrics over a variable cutoff x bitrate combo (>= 48kbps/ch tracks + * the bandwidth the strongest encoders deliver; wider regresses). */ if (s->options.coder == AAC_CODER_NMR && frame_br >= 32000) { static const int rates[] = { 32000, 48000, 64000, 96000, 192000 }; - static const int bws[] = { 14000, 15000, 16000, 18000, 20000 }; + static const int bws[] = { 14000, 18500, 20000, 21000, 22000 }; int bw_i = 0; for (; bw_i < FF_ARRAY_ELEMS(rates) - 2 && frame_br > rates[bw_i + 1]; bw_i++); s->bandwidth = bws[bw_i] + (int)((int64_t)(bws[bw_i + 1] - bws[bw_i]) * -- 2.52.0 From 45822c69541b79576a56d74fd591f35e38ea7272 Mon Sep 17 00:00:00 2001 From: Lynne <[email protected]> Date: Fri, 17 Jul 2026 14:20:31 +0800 Subject: [PATCH 7/7] fftools/ffmpeg: resample low-bitrate native AAC to a rate the budget fills Encoding a full-band spectrum at low per-channel bitrates wastes resolution on a top octave the budget cannot fill; downsampling first is standard practice among high-quality AAC encoders. When the native encoder is used with CBR and no explicit -ar, pick 32/24/16 kHz below 44/28/22 kbps per channel and warn, with -ar overriding. Explicit -ar, VBR, external encoders and filter graphs that set a rate are untouched. --- fftools/ffmpeg_mux_init.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/fftools/ffmpeg_mux_init.c b/fftools/ffmpeg_mux_init.c index 7a238b10ff..1f33ee24f7 100644 --- a/fftools/ffmpeg_mux_init.c +++ b/fftools/ffmpeg_mux_init.c @@ -1478,6 +1478,40 @@ static int ost_add(Muxer *mux, const OptionsContext *o, enum AVMediaType type, if (ret < 0) goto fail; + /* Native AAC at a low bitrate and no explicit -ar: pick a lower sample + * rate suited to the bitrate, as is standard encoder practice. */ + if (ost->enc && type == AVMEDIA_TYPE_AUDIO && + ost->enc->enc_ctx->codec && + ost->enc->enc_ctx->codec->id == AV_CODEC_ID_AAC && + !ost->enc->enc_ctx->codec->wrapper_name && + !ost->enc->enc_ctx->sample_rate && + !(ost->enc->enc_ctx->flags & AV_CODEC_FLAG_QSCALE)) { + AVCodecContext *ec = ost->enc->enc_ctx; + /* codec options are already applied, so -b:a is visible here */ + if (ec->bit_rate > 0) { + int ch = ec->ch_layout.nb_channels; + int in_rate = 0; + if (ost->ist) { + if (!ch) + ch = ost->ist->par->ch_layout.nb_channels; + in_rate = ost->ist->par->sample_rate; + } + if (ch > 0 && in_rate > 0) { + int64_t bpc = ec->bit_rate / ch; + int target = bpc >= 44000 ? 0 : + bpc >= 28000 ? 32000 : + bpc >= 22000 ? 24000 : 16000; + if (target && target < in_rate) { + ec->sample_rate = target; + av_log(ost, AV_LOG_WARNING, + "aac: resampling %d Hz input to %d Hz for " + "%d kb/s, use -ar to override\n", + in_rate, target, (int)(ec->bit_rate / 1000)); + } + } + } + } + if (ost->enc && (type == AVMEDIA_TYPE_VIDEO || type == AVMEDIA_TYPE_AUDIO)) { ret = ost_bind_filter(mux, ms, ofilter, o, enc_tb, vsync_method, -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]