Add neighbor list detection

[cbkerr]

Detect the neighbor list of jobs in a project or neighbors of an individual job. This is useful for navigating in high dimensional signac projects, especially for enabling more intuitive navigation in signac-dashboard.

The idea for abstracting this code developed out of the signac dashboard Navigator module and the need to ignore certain state point parameters when building the neighbor list.

It allows new kinds of analysis based on nearby jobs.

Description

Neighbor detection is based on cache lookups by calculating job ids of possible neighbor jobs. The possible neighbors are found by changing the state point parameters to values listed in the schema.

Ignoring keys changes job ids, so I create a mapping to the related "shadow" project that has these new job ids. This is explained in detail in the docstring for prepare_shadow_project.

It was challenging to work around the conversion between "nested keys" to "dotted keys". The convention is only documented under the query syntax. Schema returns values associated with state point parameters in dotted key format, so I convert state points in the state point caches to dotted keys. However, we have to convert back to nested keys to compute the job id.

The ability to get neighbors of a single job is intended mainly for command line usage.

Motivation and Context

The Navigator module of dashboard has been very helpful, and the general concept of neighbors of jobs could help with analysis like identifying regions of parameter space.

Basic example:

neighbor_list = project.get_neighbors()
for job in project:
    neighbors = neighbor_list[job.id]
    print(f"Job {job.id}")
    for key,v in job.sp.items():
        print(f"has {key}={v} with neighbor jobs {key}-->{f" and {key}-->".join(
        f"{new_val} at job id {jid}" for new_val,jid in neighbors[key].items())}")

Complex example: show a graph of jobs (above)

import itertools
import signac
import networkx as nx
from pyvis.network import Network

project = signac.init_project("graph")
for a,b in itertools.product([1,2,3], [5,6,7]):
    project.open_job({"a": a, "b": b, "2b": 2*b}).init()

for b in [8,9,10]:
    project.open_job({"a": 1, "b": b}).init()

# works across data types
for b in ["eight", "nine", None]:
    project.open_job({"a": 1, "b": b}).init()

# see isolated jobs
project.open_job({"a": 1, "c": True, "b": "eight"}).init()

for c,b in itertools.product([True,False], ["eight", "nine"]):
    project.open_job({"c": c, "b": b}).init()

# works on lists
for c,b in itertools.product([True,False], [[1,2], [1,5]]):
    project.open_job({"c": c, "b": b}).init()

# works on nested values
# although they appear in the neighbor list like nl[id]["x.n"] because of internal limitations with schema
for x in [{"n": "nested"}, {"n": "values"}]:
    project.open_job({"x": x}).init()

neighbor_list = project.get_neighbors(ignore = ["2b"])

DG = nx.DiGraph()
for job in project:
    DG.add_node(job.id, label = f"{job.sp}")
for job in project:
    neighbors = neighbor_list[job.id]
    for key, neighbor_vals in neighbors.items():
        for neighbor_value, neighbor_job_id in neighbor_vals.items():
            DG.add_edge(job.id, neighbor_job_id, title = f"{key}-->{neighbor_value}")

nt = Network('800px', '1000px', directed = True)
nt.from_nx(DG)
nt.show('signac_graph.html', notebook=False)

Checklist:

• I am familiar with the Contributing Guidelines.
• I agree with the terms of the Contributor Agreement.
• My name is on the list of contributors.
• The changes introduced by this pull request are covered by existing or newly introduced tests.
• The package documentation and framework documentation in signac-docs are up to date with these changes.
• I have updated the changelog and added any related issue and pull request numbers for future reference.

[joaander]

Thanks! The code is clean and the intent is clear. I think the documentation should be more explicit. Other than that, it looks great.

[bdice]

Thanks for the continued contributions! This is an interesting feature. I haven't evaluated anything for performance but I did a quick read-through for my own curiosity. I left a few small comments but I don't plan to make a second pass of review, so I leave my approval. Feel free to accept or reject my suggestions however you see fit.