Refactor inference and graph code (#56)

This PR puts the graph I/O code in its own module and the inference code
in its own module. It doesn't change any outwards-facing functionality

Author: cthoyt