From 76a4356a75fd1bb79101ac00bfb73303c369dfc8 Mon Sep 17 00:00:00 2001 From: Andreas Rheinhardt Date: Thu, 5 Nov 2020 02:29:26 +0100 Subject: [PATCH] avcodec/atrac9dec: Don't create VLCs that are never used The ATRAC9 decoder creates VLCs with parameters contained in HuffmanCodebooks; some of these HuffmanCodebooks are empty and yet VLCs (that were completely unused*) were created from them. Said VLC contained a single table with 512 VLC_TYPE[2] entries, each of which indicated that this is an invalid code. This commit stops creating said VLCs. *: read_coeffs_coarse() uses the HuffmanCodebook at9_huffman_coeffs[cb][prec][cbi]. prec is c->precision_coarse[i] + 1 and every precision_coarse entry is in the 1..15 range after calc_precision(), so prec is >= 2 (all codebooks with prec < 2 are empty). The remaining empty codebooks are those with cb == 1 and cbi == 0, yet this is impossible, too: cb is given by c->codebookset[i] and this is always 0 if i < 8 (because those are never set to anything else in calc_codebook_idx()) and cbi is given by at9_q_unit_to_codebookidx[i] which is never zero if i >= 8. Signed-off-by: Andreas Rheinhardt --- libavcodec/atrac9dec.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavcodec/atrac9dec.c b/libavcodec/atrac9dec.c index b7a46784fb..c7c6a12f01 100644 --- a/libavcodec/atrac9dec.c +++ b/libavcodec/atrac9dec.c @@ -964,8 +964,8 @@ static av_cold int atrac9_decode_init(AVCodecContext *avctx) /* Coefficient VLCs */ tab = at9_coeffs_tab; for (int i = 0; i < 2; i++) { - for (int j = 0; j < 8; j++) { - for (int k = 0; k < 4; k++) { + for (int j = 2; j < 8; j++) { + for (int k = i; k < 4; k++) { const HuffmanCodebook *hf = &at9_huffman_coeffs[i][j][k]; ret = ff_init_vlc_from_lengths(&s->coeff_vlc[i][j][k], 9, hf->size, &tab[0][1], 2, -- GitLab