Fixes: signed integer overflow: 65535 * 55335 cannot be represented in type 'int'
Fixes: 21955/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_PGMYUV_fuzzer-5669206981083136

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: NMichael Niedermayer <michael@niedermayer.cc>

Regression since cdb5479c
Reported by irc user tTh from Mixart-Myrys

Otherwise put_bits can be called with a value that doesn't fit in the
sample_len, causing an assertion failure.
Reviewed-by: NMichael Niedermayer <michael@niedermayer.cc>
Signed-off-by: NAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

Signed-off-by: NPaul B Mahol <onemda@gmail.com>

Express bitfields more simply.
Signed-off-by: NVittorio Giovara <vittorio.giovara@gmail.com>

Fixes fate-lavf-pam on alpha
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Fixes Ticket1897

Some hunks that have become unneeded due to previous changes removed by the commiter.
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

This simplifies the code
Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

The patch worked on little endian because pgm decoding was broken
but it was not correct in any way.

This reverts commit 7651c0e4.

libjpeg v6b apparently does not initialize the most significant bits
to zero when writing 12bit pgm's with maxval 4095.

Coccinelle profile used:

  @@
  expression r, ctx, f, loglevel, str, flags;
  @@

  -if ((r = ff_get_buffer(ctx, f, flags)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_get_buffer(ctx, f, flags)) < 0)
  +    return r;

  @@
  expression r, ctx, f, loglevel, str;
  @@

  -if ((r = ff_reget_buffer(ctx, f)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_reget_buffer(ctx, f)) < 0)
  +    return r;

  @@
  expression r, ctx, f, loglevel, str, flags;
  @@

  -if ((r = ff_thread_get_buffer(ctx, f, flags)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_thread_get_buffer(ctx, f, flags)) < 0)
  +    return r;

...along with some manual patches for the remaining ones.

It's got_frame, not data size

It will be useful in the upcoming transition to refcounted AVFrames.

Signed-off-by: NPaul B Mahol <onemda@gmail.com>

Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Also break some long lines, remove codec function placeholder comments
and add spaces in sample/pixel format lists.
Signed-off-by: NMartin Storsjö <martin@martin.st>

Signed-off-by: NPaul B Mahol <onemda@gmail.com>
Signed-off-by: NJustin Ruggles <justin.ruggles@gmail.com>

Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

Reviewed-by: Paul B Mahol

Signed-off-by: NMichael Niedermayer <michaelni@gmx.at>

It's more readable and less prone to breakage.

Signed-off-by: NMans Rullgard <mans@mansr.com>

When the file to decode contains a sequence of binary values like
"1101110...", decode_frame() was reading the sequence of digits like a
unique integer value, which was resulting in integer overflows.

The change add support for parsing non-space-separated pixel digits
for mono formats, in particular fix decoding of file battrace.pbm, and
fix trac issue #154.

Signed-off-by: NDiego Biurrun <diego@biurrun.de>