Skip to content

TODO.md

Latest commit

 

History

History
13 lines (12 loc) · 1.33 KB

TODO.md

File metadata and controls

13 lines (12 loc) · 1.33 KB
Raw

  1. EXO_BOOTSTRAP_PEERS is currently broken

  2. I'd like to see profiled network latency / bandwidth.

  3. I'd like to see how much bandwidth each link is using.

  4. Solve the problem of in continuous batching when a new prompt comes in, it will block decode of the current batch until the prefill is complete.

  5. We want people to be able to copy models over to a new device without ever connecting EXO to the internet. Right now EXO require internet connection once to cache some files to check if a download is complete. Instead, we should simply check if there is a non-empty model folder locally with no .partial files. This indicates it's a fully downloaded model that can be loaded.

  6. Memory pressure instead of memory used.

  7. Show the type of each connection (TB5, Ethernet, etc.) in the UI. Refer to old exo:

    exo/exo/helpers.py

    Line 251 in 56f783b

    async def get_macos_interface_type(ifname: str) -> Optional[Tuple[int, str]]:

  8. Dynamically switch to higher priority connection when it becomes available. Probably bring back InstanceReplacedAtomically.

  9. Faster model loads by streaming model from other devices in cluster.

  10. Add support for specifying the type of network connection to use in a test. Depends on 15/16.

  11. Log cleanup - per-module log filters and default to DEBUG log levels

  12. Validate RDMA connections with ibv_devinfo in the info gatherer