How to apply gradient clipping in TensorFlow ?

How to apply gradient clipping in TensorFlow ?

Asked on December 15, 2018 in Tensorflow.
Add Comment


  • 1 Answer(s)

    By using global norm you will clip the whole gradient:

    optimizer = tf.train.AdamOptimizer(1e-3)
    gradients, variables = zip(*optimizer.compute_gradients(loss))
    gradients, _ = tf.clip_by_global_norm(gradients, 5.0)
    optimize = optimizer.apply_gradients(zip(gradients, variables))
    

    While Clipping each gradient matrix it will change their relative scale:

    optimizer = tf.train.AdamOptimizer(1e-3)
    gradients, variables = zip(*optimizer.compute_gradients(loss))
    gradients = [
        None if gradient is None else tf.clip_by_norm(gradient, 5.0)
        for gradient in gradients]
    optimize = optimizer.apply_gradients(zip(gradients, variables))
    
    Answered on December 15, 2018.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.