Fix cache digest comparison by using oras manifest fetch instead of client-side hash

Fix cache digest comparison by using oras manifest fetch instead of client-side hash #537