Clean up requirements in lumen

[ahuang11]

I think there are unnecessary dependencies that can be moved to optional, e.g. https://github.com/holoviz/lumen/blob/main/pyproject.toml#L37