Created by: maxxfan

Hi, our CPU memory is running out when the training data set is big and Paddle stopped. Will Paddle load all the data from train_list into memory at once even when dataprovider is set to process only one data file?

We tried to set in provider cache=CacheType.NO_CACHE and it had no difference comparing to cache=CacheType.CACHE_PASS_IN_MEM

Since batch training does not need all data to be loaded during training. Is there a way to only load part of the data, i.e. load one file in train_list at one time? (The batch size is much smaller than each training data.) Or is there a way to set the CPU memory limit?

Thanks a lot!