All MDCT outputs must be checked in case of 128point MDCTs
Fixes: out of array read
Fixes: 04442da73d935b776d2236282588d4f9/signal_sigsegv_2625a69_351_52ca6226eb83547a2d26e322ce84ed84.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Use the ability to invert phase with ms_mask instead of changing
the codebook when possible, to avoid having to switch codebooks
if some bands are INTENSTY_BT and others are INTENSITY_BT2, since
usually a set ms_mask uses less bits that a codebook change. While
it may not always be a win (ie: if it causes an ms_mask bitmap
to be sent when it wouldn't have been otherwise), it's unlikely
since the ms_mask bitmap will almost always be there already for
M/S itself.

The relative error between two encoding strategies is the simple
difference of rate-distortion values, and not the absolute
difference. An absolute measure would allow worsening of the
quantization error as well as improving.

1. Fix sf_idx and band_type addressing to address only the first
subwindow in the group (others could hold garbage values)

2. Don't step on ms_mask when is_mask is set. I/S selection
already sets the ms_mask properly and shouldn't be overridden.

3. Use mid/sid cb/sf when computing coding error, as should be
since those are the cb/sfs that will eventually be set.

4. Fix distortion computation on multi-subwindow groups (was
subtracting the bits terms multiple times)

5. Clear ms_mask when one side uses PNS and the other doesn't.
When using PNS, ms_mask signals correlated noise, which can be
detected just like regular M/S detection, so we don't skip
noise bands, but when only one side uses PNS setting the flag
can confuse some encoders, so avoid that.

Correct addressing of sf_idx and band_type arrays in I/S code.

Both arrays don't guarantee valid values for subwindows at all
times, depending on which coder is being used, which could result
in reading of garbage values. Instead, only the first subwindow
in the window group has to be used, ie: use w*16+g instead of
(w+w2)*16+g

Change the condition for application of the M/S transform to match
that of the decoder. Namely, that no special coding books must be
in use in either channel. While the condition ought to be
equivalent to the current one when the invariant of is_mask is
kept, matching the decoder's condition is safer and easier to
maintain.

In rare corner cases it could still fail an assert on sf_diff due
to failure to update prev_sf in some code paths. Fix that case.

Fixes an assertion error reported in #2686 that happens when
using prediction (either explicitly or implicitly by setting
the AAC main profile), since prediction code would allow
creating new zeroes or removing existing ones, without
properly checking for SF delta violations.

This patch forbids creating/removing zeroes, perhaps an
overly conservative approach, but a safe one. More permissive
and sophisticated approaches may be attempted in the future.

Fixes ticket #5151.

Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes out of array read
Fixes: 03c406ec9530e594a074ce2979f8a1f0/asan_heap-oob_7dec26_4664_37c52495b2870a2eaac65f53958e76c1.flac

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

since:
cbcc88c0 libvpx: Support setting color range for vp9.
Reviewed-by: NRonald S. Bultje <rsbultje@gmail.com>
Signed-off-by: NJames Zern <jzern@google.com>

Signed-off-by: NPaul B Mahol <onemda@gmail.com>

Signed-off-by: NPaul B Mahol <onemda@gmail.com>

Signed-off-by: NPaul B Mahol <onemda@gmail.com>

Fixes integer overflow
Fixes: 0063df8be3aaa30dd6d76f59c8f818c8/signal_sigsegv_7b7b59_3634_bf418b6822bbfa68734411d96b667be3.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes: 0063df8be3aaa30dd6d76f59c8f818c8/signal_sigsegv_7b7b59_3634_bf418b6822bbfa68734411d96b667be3.mov
Fixes assertion failure

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes memleaks
Fixes: 245dae71bc28fe8f3f454300277e575a/signal_sigabrt_7ffff6ae7cb7_5062_f7d9dfa385454806586076f9db48c0bf.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Use av_realloc() rather than av_malloc() when normalizing noncompliant
private data in get_qt_codec().
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Found-by: Leo Izen <leo.izen@gmail.com> as thebombzen on IRC
Signed-off-by: NTimothy Gu <timothygu99@gmail.com>

Fixes assertion failure
Fixes out of memory access

Fixes: test_casex.ivf
Found-by: NTyson Smith <twsmith@mozilla.com>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Either disabling or init'ing secure memory is required after the use
of gcry_check_version. From a look at the functions rtmpdh uses, I
noticed none require the use of secure memory, so we disable it [1][2].

This resolves some errors returned by rtmpdh code with uninitialized
gcrypt, especifically:
Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument

Version "1.5.4" was arbitrarily chosen. An older version probably works
as well, but I couldn't compile older versions to test on my machine.

[1]
https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
[2]
https://www.gnupg.org/documentation/manuals/gcrypt/Controlling-the-library.htmlSigned-off-by: NRicardo Constantino <wiiaboo@gmail.com>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

The transformation to bytes must happen after alignment to get the same
resulting pointers as before.

This fixes segmentation faults in the assembler code.

The regression was introduced in commit 95536898.
Reviewed-by: NKieran Kunhya <kierank@obe.tv>
Signed-off-by: NAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

This exploits an approach based on the sieve of Eratosthenes, a popular
method for generating prime numbers.

Tables are identical to previous ones.

Tested with FATE with/without --enable-hardcoded-tables.

Sample benchmark (Haswell, GNU/Linux+gcc):
prev:
7860100 decicycles in cbrt_tableinit,       1 runs,      0 skips
7777490 decicycles in cbrt_tableinit,       2 runs,      0 skips
[...]
7582339 decicycles in cbrt_tableinit,     256 runs,      0 skips
7563556 decicycles in cbrt_tableinit,     512 runs,      0 skips

new:
2099480 decicycles in cbrt_tableinit,       1 runs,      0 skips
2044470 decicycles in cbrt_tableinit,       2 runs,      0 skips
[...]
1796544 decicycles in cbrt_tableinit,     256 runs,      0 skips
1791631 decicycles in cbrt_tableinit,     512 runs,      0 skips

Both small and large run count given as this is called once so small run
count may give a better picture, small numbers are fairly consistent,
and there is a consistent downward trend from small to large runs,
at which point it stabilizes to a new value.
Reviewed-by: NMichael Niedermayer <michael@niedermayer.cc>
Signed-off-by: NGanesh Ajjanagadde <gajjanagadde@gmail.com>

It is used to store the difference between pointers, so ptrdiff_t is the
correct type.

This prevents potential overflows.
Reviewed-by: NMichael Niedermayer <michael@niedermayer.cc>
Signed-off-by: NAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

This fixes segmentation faults due to out of bounds writes, when
color_start is interpreted as negative number.

This regression was introduced in commit 57631f.
Reviewed-by: NMats Peterson <matsp888@yahoo.com>
Signed-off-by: NAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

They are stored in a byte, thus more than 255 is not possible
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes: 1b8b83a53bfa751f01b1daa65a4758db/signal_sigabrt_7ffff6ae7cb7_7488_403f71d1a2565b598d01b6cb110fac8f.aiff
Fixes: assertion failure

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes: 07fa81f300736485e585b63e7b1cd155/signal_sigabrt_7ffff6ae7cb7_16_94ee815791a1357826524072db5bc2dc.dfa

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

This patch adds a new static function get_qt_codec() that takes care of
the initial retrieval of the fourcc and codec ID for A_QUICKTIME and
V_QUICKTIME. It also normalizes noncompliant private data found in some
older files that incorrectly starts with the fourcc by expanding/shifting
the data by 4 bytes, and storing the data size at the start. This is
necessary in order for the rest of the code in the A_QUICKTIME and
V_QUICKTIME blocks (and most likely other code as well) to correctly
parse the private data.
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes index out of bounds error
Fixes: aac_index_out_of_bounds.wmv
Found-by: NPiotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes: dvdec_left_shift.avi
Found-by: NPiotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes invalid shift
Fixes: g2m_left_shift_2.wmv
Found-by: NPiotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Use monospaced font, and a black box outline.

Fixes: mjpeg_left_shift.avi
Found-by: NPiotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>