The MBS only flag was not taken into account when checking macroblock dimensions.
Also removes the unneeded check in init_dimensions for slices.

In order to be able to make SPS const in H264ParamSets,
modify decode_scaling_matrices so that it returns if the scaling
matrix are present in the SPS, instead of altering the input SPS
structure.

Signed-off-by: NDiego Biurrun <diego@biurrun.de>

Make the SPS/PPS parsing independent of the H264Context, to allow
decoupling the parser from the decoder. The change is modelled after the
one done earlier for HEVC.

Move the dequant buffers to the PPS to avoid complex checks whether they
changed and an expensive copy for frame threads.

Fall back to maximum DPB size if the level is unknown.

This should be more spec-compliant and does not depend on the caller
setting has_b_frames before opening the decoder.

The old behaviour, when the delay is supplied by the caller setting
has_b_frames, can still be obtained by setting strict_std_compliance
below normal.

Fixes: https://trac.ffmpeg.org/attachment/ticket/685/movie.264

In the available testcase the actual PPS only uses a few bits
while there are 7kbyte of apparently random data after it
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

We assume an upper bound of 4096 bytes for each raw SPS/PPS. It's hard
to determine an exact maximum size, but this value was was considered
high enough and safe.

Needed for the following VideotoolBox commit.

Convert doxygen to multiline and express bitfields more simply.
Signed-off-by: NVittorio Giovara <vittorio.giovara@gmail.com>

Bug-Id: CID 1026763

This probably makes no big difference but it is more correct
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

The spec madandate both time_scale and num_units_in_tick greater than 0,
however since they are not essential for decoding, just ignore the whole
block and try to finish parsing the VUI.

Related to Ticket4445.
Signed-off-by: NVittorio Giovara <vittorio.giovara@gmail.com>

Fixes Ticket4445
Tested-by: NVittorio Giovara <vittorio.giovara@gmail.com>
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

This more completely checks for truncation
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

There is in general no reason for the currently active SPS to be the one
referenced by the PPS being parsed.

CC: libav-stable@libav.org

Fixes out of array read
Fixes: asan_static-oob_30328b6_719_cov_3325483287_H264_artifacts_motion.h264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Needed for filters that may need it, like an SPS bsf.
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Signed-off-by: NDiego Biurrun <diego@biurrun.de>

This avoids compiler warnings about argument and specifier type mismatch.

Also comment all previous profiles.

Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

In preparation for MVC support.

Also make error style consistent and drop redundant information.

An invalid VUI is not considered a fatal error, so the SPS containing it
may still be used. Leaving an invalid value of num_reorder_frames there
can result in writing over the bounds of H264Context.delayed_pic.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org

There is no point in delaying the check and it avoids bugs with a
half-initialized context.

Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org

A VUI doesn't contain anything strictly necessary for decoding.
Apparently there are many samples with truncated VUIs in the wild, this
commit should allow decoding them.

Fixes Ticket2763
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Fixes: CID1026763
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>