Configuring cache using a parameter

[fountaindive]

There is this issue from last year which is closed but i'm not sure if a fix was ever released?

Does anyone have a workaround for how to configure caching using a pipeline parameter? See the issue for an example.

The error you get is

TypeError: 'PipelineParameterChannel' object cannot be interpreted as an integer

[milosjava]

As far as I know, you can't do that currently. set_caching_options expects a literal bool at compile time. I understand the need for this functionality. Cache is sometimes a blessing and sometimes a pain in the neck.

[fountaindive]

Thanks very much for commenting!

I don't suppose you could elaborate on when you have found it to be a pain?

[milosjava]

Typically, cache was very useful during pipeline development. It allowed our DS team to quickly re-run updated versions of the pipeline (for example, when adding or modifying components), saving both time and compute resources.

However, once the pipeline was more or less finalized, caching started to cause issues — especially for components that read from external sources like databases. Those components are often among the first in the pipeline, so stale cache entries could easily cause us to work with outdated data.

We tried a workaround by tying the cache behavior to a pipeline parameter — enabling it (True) while developing for speed, and disabling it when we needed to fetch the latest data but you already know how that ended up :)

[fountaindive]

Hey,

yeah that all makes sense.

I guess it's down changing the source code during dev and not forgetting to change it in prod 🙃

thanks for taking the time again, it's greatly appreciated!

[milosjava]

One thing to add here. It is possible to use the KFP SDK and run the pipeline "directly". Please check here. If that approach is ok for you then you can set enable_caching to either True or False.
We used that approach before, but now we value a proper CI/CD (e.g. PR merge , github action to deploy pipeline to KF) way.

[fountaindive]

Hmm that might work, I will take a look thanks!

[milosjava]

sure, let us know how it worked for you and please mark any of the responses as the answer if you found them helpful.

[fountaindive]

ah I don't think it will work for me because the way we run the pipeline is by compiling the yaml and at that point the set_caching_options are set and we don't have an equivalent of the run_pipeline in java so we can't override the set_caching_options values. Although, I suppose it would be possible by re-writing the compiled yaml?