libobs: Reset frame cache if it gets too big

When buffering is enabled for an async video source, sometimes minor
drift in timestamps or unexpected delays to frames can cause frames to
slowly buffer more and more in memory, in some cases eventually causing
the system to run out of memory.

The circumstances in which this can happen seems to depend on both the
computer and the devices in use.  So far, the only known circumstances
in which this happens are with heavily buffered devices, such as
hauppauge, where decoding can sometimes take too long and cause
continual frame playback delay, and thus continual buffering until
memory runs out.  I've never been able to replicate it on any of my
machines however, even after hours of testing.

This patch is a precautionary measure that puts a hard limit on the
number of async frames that can be currently queued to prevent any case
where memory might continually build for whatever reason.  If it goes
over the limit, it clears the cache to reset the buffering.

I had a user with this problem test this patch with success and positive
feedback, and the intervals between buffering resets were long to where
it wasn't even noticeable while streaming/recording.

Ideally when decoding frames (such as from those devices), frame
dropping should be used to ensure playback doesn't incur extra delay,
although this sort of hard limit on the frame cache should still be
implemented regardless just as a safety precaution.  For DirectShow
encoded devices I should just switch to faruton's libff for decoding and
enable the frame dropping options.  It would probably explain why no
one's ever reported it for the media source, and pretty much only from
DirectShow device usage.

libobs/obs-source.c

@@ -1831,6 +1831,8 @@ static void clean_cache(obs_source_t *source)
 	}
 }
 
+#define MAX_ASYNC_FRAMES 30
+
 static inline struct obs_source_frame *cache_video(struct obs_source *source,
 		const struct obs_source_frame *frame)
 {
@@ -1838,6 +1840,13 @@ static inline struct obs_source_frame *cache_video(struct obs_source *source,
 
 	pthread_mutex_lock(&source->async_mutex);
 
+	if (source->async_frames.num >= MAX_ASYNC_FRAMES) {
+		free_async_cache(source);
+		source->last_frame_ts = 0;
+		pthread_mutex_unlock(&source->async_mutex);
+		return NULL;
+	}
+
 	if (async_texture_changed(source, frame)) {
 		free_async_cache(source);
 		source->async_cache_width  = frame->width;