[tests] refactor caching tests.#13235

sayakpaul wants to merge 10 commits into

mainhuggingface/diffusers:mainfrom

refactor-caching-testshuggingface/diffusers:refactor-caching-testsCopy head branch name to clipboard