1. 07 8月, 2023 1 次提交
    • U
      [cherry-pick] Integration flash attention 2 (#56015) · cc9a7688
      umiswing 提交于
      * [FlashAttn] add flash randomness control (#52902)
      
      * add flash randomness control
      
      * fix VLOG undefied
      
      * [WIP] Integration flash attention 2 (#55758)
      
      * Work for fa-2 padded fwd. Code to be cleaned.
      
      * Work for fa2 unpadded fwd.
      
      * Work for padded-bwd, dk get small diff on np.random.seed(0)
      
      * Anyway I pass paddle's utest, except return softmax without dropout.
      
      * Clean code.
      
      * Modify interface.
      
      * Clean code and add some check.
      
      * Easy compile for dev.
      
      * Fix ci.
      
      * Fix ci-build.
      
      * Add std c++17 option again.
      
      * Limit max job when compiling fa2.
      
      * Remove const_cast
      
      * Add fwd params, to be cleaned.
      
      * Clean code.
      
      * Add bwd params.
      
      * Clean code.
      
      * Add enforce.
      
      * Use v2.0.4
      
      * Pass RNG state to fa2 capi
      
      * Fix review.
      
      * Add assert
      
      * Skip compile for sm less than 80.
      
      ---------
      Co-authored-by: NChitsing KUI <kuizhiqing@msn.com>
      cc9a7688
  2. 25 3月, 2023 1 次提交
  3. 16 3月, 2023 1 次提交
  4. 10 3月, 2023 1 次提交
  5. 01 3月, 2023 1 次提交
    • C
      Integration flash attention (#49869) · 61611786
      Chitsing KUI 提交于
      * flash attn
      
      * seed
      
      * almost
      
      * softmax
      
      * fix workspace
      
      * add unitest; linux only
      
      * fix setup
      
      * fix datatype include
      
      * fix setup typo
      
      * fix def scope
      
      * new error api
      
      * use paddle fork
      
      * fix attr bug; complete ut
      
      * update flash hash
      
      * fix rng reset
      
      * fix offset
      
      * fix comments
      61611786