gradient exploding issue when using rnn
Created by: Slyne
Hi, I'm not sure if this is normal. Here's the configuration and log:
----------- Configuration Arguments -----------
augment_conf_path: conf/augmentation.config
batch_size: 160
dev_manifest: data/librispeech/manifest.dev-clean
init_model_path: None
is_local: 1
learning_rate: 0.0005
max_duration: 27.0
mean_std_path: data/librispeech/mean_std.npz
min_duration: 0.0
num_conv_layers: 3
num_iter_print: 100
num_passes: 50
num_proc_data: 16
num_rnn_layers: 7
output_model_dir: ./checkpoints/libri_complex
rnn_layer_size: 2048
share_rnn_weights: 1
shuffle_method: batch_shuffle_clipped
specgram_type: linear
test_off: 0
train_manifest: data/librispeech/manifest.train
trainer_count: 8
use_gpu: 1
use_gru: 0
use_sortagrad: 1
vocab_path: data/librispeech/vocab.txt
======================= ................................................................................................... Pass: 0, Batch: 100, TrainCost: 125.596261 ................................................................................................... Pass: 0, Batch: 200, TrainCost: 204.322346 ................................................................................................... Pass: 0, Batch: 300, TrainCost: 279.806530 ................................................................................................... Pass: 0, Batch: 400, TrainCost: 360.684450 ................................................................................................... Pass: 0, Batch: 500, TrainCost: 399.957560 ................................................................................................... Pass: 0, Batch: 600, TrainCost: 440.331799 ................................................................................................... Pass: 0, Batch: 700, TrainCost: 466.695379 ................................................................................................... Pass: 0, Batch: 800, TrainCost: 485.230144 ................................................................................................... Pass: 0, Batch: 900, TrainCost: 499.748687 ................................................................................................... Pass: 0, Batch: 1000, TrainCost: 512.164312 ................................................................................................... Pass: 0, Batch: 1100, TrainCost: 523.656208 ................................................................................................... Pass: 0, Batch: 1200, TrainCost: 533.018596 ................................................................................................... Pass: 0, Batch: 1300, TrainCost: 541.667011 ................................................................................................... Pass: 0, Batch: 1400, TrainCost: 552.249161 ................................................................................................... Pass: 0, Batch: 1500, TrainCost: 561.308075 ................................................................................................... Pass: 0, Batch: 1600, TrainCost: 572.026342 ................................................................................................... Pass: 0, Batch: 1700, TrainCost: 583.196865 .......................................................... ------- Time: 6633 sec, Pass: 0, ValidationCost: 1524.97344964 ................................................................................................... Pass: 1, Batch: 100, TrainCost: 569.899187 ................................................................................................... Pass: 1, Batch: 200, TrainCost: 455.813797 ................................................................................................... Pass: 1, Batch: 300, TrainCost: 459.514104 ................................................................................................... Pass: 1, Batch: 400, TrainCost: 468.573062