diff --git a/test/diff/README.md b/test/diff/README.md new file mode 100644 index 0000000000000000000000000000000000000000..15a2a6df77b1098588c669dfedaae859f4256648 --- /dev/null +++ b/test/diff/README.md @@ -0,0 +1,31 @@ +# Three.js automatic regression testing with CI + +You probably shouldn't run this tests on PC because right now it's not optimized for local usage and you can get different results on different GPUs. Goal is to make quick automated testing inside CI and keep screenshot pack updated for it. + +### Local usage +```shell +# generate new screenshots +npm run make-screenshot ... + +# check examples +npm run test-diff ... + +# check all examples in browser +npx cross-env VISIBLE=ture npm run test-diff +``` + +### How it works +- ci configs with parallelism +- deterministic random/timer/rAF/video for screenshots +- increased robustness with hided text, datgui, different flags and timeouts. +- pipeline: turn off rAF -> 'networkidle0' -> networkTax -> turn on rAF -> render promise +- added 3 progressive attempts for robustness + +### Status +97% examples are covered with tests. Random robusness in CI ~85%. Robustness on different machines ~97%. For example in Windows webgl_effects_ascii example always fails or on integrated GPU you will have additional artifacts: webgl_materials_texture_anisotropy, webgl_postprocessing_procedural, webgl_shaders_tonemapping. + +### Probably wrong screenshots +webgl2_multisampled_renderbuffers, webgl_simple_gi, webgl_postprocessing_dof2, webgl_loader_texture_pvrtc + +### Contribution +You can help to simplify puppeteer script by suggesting example with [HeadlessExperimental.beginFrame](https://chromedevtools.github.io/devtools-protocol/tot/HeadlessExperimental) CDP API. diff --git a/test/diff/puppeteer.js b/test/diff/puppeteer.js index 4ab49c73b238cd38adbf617f8a5aafe7d5325dbb..45a08c47f7e9a5303afb22a67b3ae58f2f190dbf 100644 --- a/test/diff/puppeteer.js +++ b/test/diff/puppeteer.js @@ -14,7 +14,7 @@ const port = 1234; const pixelThreshold = 0.2; const maxFailedPixels = 0.05; const networkTimeout = 600; -const networkTax = 2000; // additional timout tax for resources size +const networkTax = 2000; // additional timeout for resources size const pageSizeMinTax = 1.0; // in mb, when networkTax = 0 const pageSizeMaxTax = 5.0; // in mb, when networkTax = networkTax const renderTimeout = 1200; @@ -23,14 +23,14 @@ const maxAttemptId = 3; // progresseve attempts const exceptionList = [ 'index', - 'webgl_loader_texture_pvrtc', // not supported in CI, usless + 'webgl_loader_texture_pvrtc', // not supported in CI, useless 'webgl_materials_envmaps_parallax', 'webgl_test_memory2', // gives fatal error in puppeteer 'webgl_worker_offscreencanvas', // in a worker, not robust ].concat( ( process.platform === "win32" ) ? [ - 'webgl_effects_ascii' // windows fonts + 'webgl_effects_ascii' // windows fonts not supported ] : [] );