Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

avcodec/h264_ps: Check offset_for_non_ref_pic, offset_for_top_to_bottom_field and offset_for_ref_frame

Fixes: signed integer overflow: -2147483648 + -1 cannot be represented in type 'int'
Fixes: 14444/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_H264_fuzzer-5675880333967360

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: NMichael Niedermayer <michael@niedermayer.cc>

All three may be up to 32 bits wide.
Signed-off-by: NJames Almer <jamrial@gmail.com>

Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

This merges commit c3e84820 from libav,
originally written by Anton Khirnov and skipped in
fc63d5ce.

 libavcodec/h264_picture.c |  3 ---
 libavcodec/h264_ps.c      |  9 ---------
 libavcodec/h264_slice.c   | 25 +++++++++++++++++++------
 libavcodec/h264dec.c      | 13 +------------
 libavcodec/h264dec.h      |  9 +++++----
 5 files changed, 25 insertions(+), 34 deletions(-)

avcodec/h264_ps: Fix runtime error: signed integer overflow: 2147483647 + 26 cannot be represented in type 'int'

Fixes: 902/clusterfuzz-testcase-4561155144024064

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegReviewed-by: N"Ronald S. Bultje" <rsbultje@gmail.com>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes: 647/clusterfuzz-testcase-5195745823031296

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Reviewed-by: BBB
Reviewed-by: NPaul B Mahol <onemda@gmail.com>
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: 5 + 2147483646 cannot be represented in type 'int'
Fixes: 634/clusterfuzz-testcase-5285420445204480

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: NMichael Niedermayer <michael@niedermayer.cc>

avcodec/h264_ps: Fix runtime error: signed integer overflow: -1094995528 * 2 cannot be represented in type 'int'

Fixes: 615/clusterfuzz-testcase-5488002644049920
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Signed-off-by: NVittorio Giovara <vittorio.giovara@gmail.com>

Currently, SPS.mb_height is actually what the spec calls
PicHeightInMapUnits, which is half the frame height when interlacing is
allowed. Calling this 'mb_height' is quite confusing, and there are at
least two associated bugs where this field is treated as the actual
frame height - in the h264 parser and in the code computing maximum
reordering buffer size for a given level.

Fix those issues (and avoid possible future ones) by exporting the real
frame height in this field.

The MBS only flag was not taken into account when checking macroblock dimensions.
Also removes the unneeded check in init_dimensions for slices.

In order to be able to make SPS const in H264ParamSets,
modify decode_scaling_matrices so that it returns if the scaling
matrix are present in the SPS, instead of altering the input SPS
structure.

This field (which the spec calls max_num_ref_frames) must be less than
or equal to MaxDpbFrames, which is at most 16.

The PS parsing code is independent from the decoder, so it makes more
sense for it to have its own separate header.

This is more consistent with the naming of other decoders.

Signed-off-by: NAnton Khirnov <anton@khirnov.net>

Signed-off-by: NDiego Biurrun <diego@biurrun.de>

Make the SPS/PPS parsing independent of the H264Context, to allow
decoupling the parser from the decoder. The change is modelled after the
one done earlier for HEVC.

Move the dequant buffers to the PPS to avoid complex checks whether they
changed and an expensive copy for frame threads.

Fall back to maximum DPB size if the level is unknown.

This should be more spec-compliant and does not depend on the caller
setting has_b_frames before opening the decoder.

The old behaviour, when the delay is supplied by the caller setting
has_b_frames, can still be obtained by setting strict_std_compliance
below normal.

Fixes: https://trac.ffmpeg.org/attachment/ticket/685/movie.264

In the available testcase the actual PPS only uses a few bits
while there are 7kbyte of apparently random data after it
Signed-off-by: NMichael Niedermayer <michael@niedermayer.cc>

We assume an upper bound of 4096 bytes for each raw SPS/PPS. It's hard
to determine an exact maximum size, but this value was was considered
high enough and safe.

Needed for the following VideotoolBox commit.

Convert doxygen to multiline and express bitfields more simply.
Signed-off-by: NVittorio Giovara <vittorio.giovara@gmail.com>

Bug-Id: CID 1026763

This probably makes no big difference but it is more correct
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

The spec madandate both time_scale and num_units_in_tick greater than 0,
however since they are not essential for decoding, just ignore the whole
block and try to finish parsing the VUI.

Related to Ticket4445.
Signed-off-by: NVittorio Giovara <vittorio.giovara@gmail.com>

Fixes Ticket4445
Tested-by: NVittorio Giovara <vittorio.giovara@gmail.com>
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

This more completely checks for truncation
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>