diff --git a/.agents/context/testing.md b/.agents/context/testing.md index cbead5272..185965334 100644 --- a/.agents/context/testing.md +++ b/.agents/context/testing.md @@ -11,8 +11,10 @@ pytest -m "not server" # skip tests that need a live server (CI ``` Config lives in `pyproject.toml` (`[tool.pytest.ini_options]`): discovery is scoped to -`py/tests/` and `pythonpath = ["py"]` makes `import visdom` work without an editable install. -Experimental `test_*.py` scripts in the repo root (and `test/`) are intentionally out of scope. +`py/tests/`, and `pythonpath = ["py", "py/tests"]` makes both `import visdom` and +`import testutils` work without an editable install. Because discovery is scoped by `testpaths`, +experimental `test_*.py` scripts in the repo root (and `test/`) stay out of scope; `testutils/` is +excluded via `norecursedirs` so helpers are importable but never collected. ## Run E2E / Visual Tests (Cypress) @@ -38,12 +40,90 @@ Always use port `8098` and `-env_path /tmp` for isolation. ## Writing Python Tests -- Place in `py/tests/`, name files `test_*.py`, classes `Test*` (unittest `TestCase` or plain - pytest functions both work — pytest auto-discovers unittest). -- Keep them hermetic (no running server). A test that genuinely needs a live server must be - marked `@pytest.mark.server` so CI can deselect it. -- Start with simple hermetic tests (e.g., `test_smoke.py`) and add focused unit tests for - server/window/env lifecycle code as coverage grows. +### Where a test goes + +``` +py/tests/ + conftest.py shared fixtures, auto-loaded by pytest + testutils/ importable helpers (fakes, payload builders, HTTP base class) + unit/ pure logic: no Application, no I/O beyond tmp_path + integration/ in-process Application, real HTTP, or handler dispatch +``` + +`py/tests/` has **no** `__init__.py` on purpose — `setup.py` runs `find_packages(where="py")`, +so a package there would ship a top-level `tests` distribution to users. `testutils/` is a +package and is reachable because `py/tests` is on `pythonpath`. + +Everything under `py/tests/` must be **hermetic and collectable**: no externally launched +server, no browser, no assertion a human has to make. A script that needs a live server and is +judged by looking at the UI goes in `example/manual/` instead — see +`example/manual/visual_check.py`. Pixel correctness is Playwright's and Cypress's job, not +pytest's. + +- Name a file after what it covers — `integration/window_types.py`, not + `integration/test_window_types.py`. The `unit/` and `integration/` directories already say these + are tests, so the filename does not repeat it; `python_files = ["*.py"]` in `pyproject.toml` + collects them, and `norecursedirs` keeps `testutils/` importable but uncollected. Test + *functions* and `Test*` classes still need their usual prefixes. +- **Which style you use depends on whether the test needs HTTP.** + + | Test needs | Write | Why | + |---|---|---| + | no `Application`, or a handler object | plain `def test_*()` functions | fixtures and `parametrize` both work | + | a real HTTP round trip | a `VisdomHTTPTestCase` subclass | `tornado.testing.AsyncHTTPTestCase` is a `unittest.TestCase`, and that is what starts the app | + + **pytest cannot inject fixtures into `TestCase` methods** — `def test_x(self, app)` fails, and + only autouse fixtures reach them. `@pytest.mark.parametrize` does not work on them either; use + a small `_assert_*` helper called from several one-line test methods instead. A module-level + `pytestmark = pytest.mark.integration` **does** apply to `TestCase` classes, so always set one. +- Keep them hermetic. A test that needs an **externally launched** server must be marked + `@pytest.mark.server` so CI can deselect it; nothing in the tracked suite needs one today. + +### Shared fixtures (`py/tests/conftest.py`) + +| Fixture | Gives you | +|---|---| +| `env_path` | disposable environment directory | +| `store` / `spy_store` | `JSONStore` / one that records backend calls | +| `app` / `app_factory` | `Application` on a temp `env_path`; factory for reload assertions | +| `handler` / `app_handler` | duck-typed handler, standalone or sharing an `Application`'s state | +| `fake_socket` | records `write_message`; `.commands()` and `.last(cmd)` for assertions | +| `offline_client` | `Visdom(send=False)` — never opens a connection | +| `capture_send` | runs a client call and returns the payload it would have sent | + +`reset_warn_once` is autouse: `shared_utils.warn_once` dedupes against a module-level set, so +without it a warning raised by one test silently suppresses the same warning in another. + +### HTTP tests + +Subclass `testutils.VisdomHTTPTestCase`. It starts the app in-process on an ephemeral port, gives +every test a fresh `env_path` that is cleaned up in `tearDown`, and provides `post_json`, +`create_window`, `create_text_window`, `update`, `close_window`, `win_exists`, `get_win_data`, +`get_envs`, `save` and `panes`, on top of `AsyncHTTPTestCase`'s own `fetch`. Override the +`app_kwargs` class attribute to vary server configuration: + +```python +class TestReadonlyRoutes(VisdomHTTPTestCase): + app_kwargs = {"readonly": True} +``` + +`AsyncHTTPTestCase` already runs the `Application` in-process on its own `IOLoop`, driving each +request through `io_loop.run_sync`. **Do not replace it with a background thread, a hand-rolled +`asyncio` loop, or an out-of-process server** — none of that buys anything, and it was tried and +reverted. The `TestCase` style is the accepted cost of using it. + +Because fixtures cannot reach these tests, anything shared goes on the class: `self.env_path` for +the temp directory, and a small base class between `VisdomHTTPTestCase` and your test classes for +helpers several of them need (see `WindowTypeTestCase` in `integration/window_types.py`). +Need a second `Application` over the same directory, for a reload assertion? Construct it directly +with `Application(port=8097, env_path=self.env_path)` — `app_factory` is not available here. + +### Markers + +`unit`, `integration`, `slow`, and `server` are registered in `pyproject.toml`. Set +`pytestmark = pytest.mark.unit` or `pytest.mark.integration` at the top of every new file; it +works for both plain functions and `TestCase` classes. Many older files predate this and carry no +marker, so `-m integration` currently under-selects. ## CI diff --git a/example/manual/README.md b/example/manual/README.md new file mode 100644 index 000000000..13ac1d619 --- /dev/null +++ b/example/manual/README.md @@ -0,0 +1,31 @@ + + +# Manual checks + +Scripts here drive a **running** visdom server and are judged by looking at the +browser. They are not tests: pytest does not collect them (`testpaths` is +`py/tests`), and nothing in CI runs them. + +Automated checks belong elsewhere: + +| Question | Where it is answered | +|---|---| +| Does the server behave correctly? | `py/tests/` (pytest) | +| Does the UI still render the same pixels? | `playwright/`, `cypress/` | +| Does this *look* right to a person? | here | + +## `visual_check.py` + +Creates one window of each visualization type in the `visual_check` environment, +then prints a checklist to walk through in the browser. Useful after a +dependency bump or a change to the plotting payloads. + +```bash +visdom -port 8097 -env_path /tmp # in one shell +python example/manual/visual_check.py +# then open http://localhost:8097/env/visual_check +``` diff --git a/example/manual/visual_check.py b/example/manual/visual_check.py new file mode 100644 index 000000000..693639b7f --- /dev/null +++ b/example/manual/visual_check.py @@ -0,0 +1,330 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Manual visual check: plot one window of every kind, then look at them. + +This is a script for a human, not a test. It needs a running server and its +result is a judgement about how the browser rendered the panes, which is why it +lives in ``example/manual/`` rather than under ``py/tests/`` — pytest collects +nothing here. Automated rendering checks belong to the Playwright and Cypress +visual-regression suites. + +Usage: + 1. Start a server: visdom -port 8097 + 2. Run this: python example/manual/visual_check.py + 3. Open a browser: http://localhost:8097/env/visual_check +""" + +import math +import sys +import time +import numpy as np + +from visdom import Visdom + +ENV = "visual_check" + + +def main(): + viz = Visdom(server="http://localhost", port=8097, env=ENV) + if not viz.check_connection(timeout_seconds=5): + print("ERROR: Cannot connect to Visdom server.") + print("Start it with: visdom -port 8097") + sys.exit(1) + + print(f"Connected to Visdom. Creating visualizations in env '{ENV}'...") + print(f"Open http://localhost:8097/env/{ENV} to view.\n") + + # =========================================================== + # 1. LINE PLOT — Single line + # =========================================================== + x = np.linspace(0, 4 * math.pi, 200) + viz.line( + Y=np.sin(x), + X=x, + opts=dict( + title="1. Sine Wave (Line Plot)", + xlabel="x", + ylabel="sin(x)", + ), + win="line_single", + ) + print("[OK] 1. Single line plot (sine wave)") + + # =========================================================== + # 2. LINE PLOT — Multiple lines + # =========================================================== + viz.line( + Y=np.column_stack((np.sin(x), np.cos(x), np.sin(x) * np.cos(x))), + X=np.column_stack((x, x, x)), + opts=dict( + title="2. Multiple Lines (sin, cos, sin*cos)", + legend=["sin(x)", "cos(x)", "sin(x)*cos(x)"], + xlabel="x", + ylabel="y", + ), + win="line_multi", + ) + print("[OK] 2. Multiple line plot") + + # =========================================================== + # 3. LINE PLOT — Append update (streaming data) + # =========================================================== + win = viz.line( + Y=np.array([0.0]), + X=np.array([0]), + opts=dict(title="3. Streaming Line (append update)"), + win="line_streaming", + ) + for i in range(1, 50): + viz.line( + Y=np.array([np.sin(i * 0.2) + np.random.randn() * 0.1]), + X=np.array([i]), + win=win, + update="append", + ) + print("[OK] 3. Streaming line plot (50 appended points)") + + # =========================================================== + # 4. SCATTER — 2D with classes + # =========================================================== + n = 200 + x_scatter = np.random.randn(n, 2) + labels = (x_scatter[:, 0] > 0).astype(int) + 1 # 1 or 2 + viz.scatter( + X=x_scatter, + Y=labels, + opts=dict( + title="4. 2D Scatter (two classes)", + legend=["Class A", "Class B"], + markersize=8, + xlabel="Feature 1", + ylabel="Feature 2", + ), + win="scatter_2d", + ) + print("[OK] 4. 2D scatter plot with classes") + + # =========================================================== + # 5. SCATTER — 3D + # =========================================================== + viz.scatter( + X=np.random.rand(100, 3), + opts=dict( + title="5. 3D Scatter Plot", + markersize=5, + ), + win="scatter_3d", + ) + print("[OK] 5. 3D scatter plot") + + # =========================================================== + # 6. BAR CHART — Simple + # =========================================================== + viz.bar( + X=np.array([28, 55, 43, 91, 72, 35]), + opts=dict( + title="6. Bar Chart", + rownames=["Jan", "Feb", "Mar", "Apr", "May", "Jun"], + ), + win="bar_simple", + ) + print("[OK] 6. Simple bar chart") + + # =========================================================== + # 7. BAR CHART — Stacked + # =========================================================== + viz.bar( + X=np.random.rand(5, 3) * 100, + opts=dict( + title="7. Stacked Bar Chart", + stacked=True, + legend=["Product A", "Product B", "Product C"], + rownames=["Q1", "Q2", "Q3", "Q4", "Q5"], + ), + win="bar_stacked", + ) + print("[OK] 7. Stacked bar chart") + + # =========================================================== + # 8. HEATMAP + # =========================================================== + hm_data = np.outer(np.arange(1, 11), np.arange(1, 11)) + viz.heatmap( + X=hm_data, + opts=dict( + title="8. Heatmap (Multiplication Table)", + columnnames=[str(i) for i in range(1, 11)], + rownames=[str(i) for i in range(1, 11)], + colormap="Viridis", + ), + win="heatmap", + ) + print("[OK] 8. Heatmap") + + # =========================================================== + # 9. HISTOGRAM + # =========================================================== + viz.histogram( + X=np.random.randn(1000), + opts=dict( + title="9. Histogram (Normal Distribution)", + numbins=40, + ), + win="histogram", + ) + print("[OK] 9. Histogram") + + # =========================================================== + # 10. BOX PLOT + # =========================================================== + viz.boxplot( + X=np.column_stack( + ( + np.random.randn(100) * 1 + 5, + np.random.randn(100) * 2 + 3, + np.random.randn(100) * 0.5 + 7, + ) + ), + opts=dict( + title="10. Box Plot (3 groups)", + legend=["Group A", "Group B", "Group C"], + ), + win="boxplot", + ) + print("[OK] 10. Box plot") + + # =========================================================== + # 11. SURFACE PLOT + # =========================================================== + xx = np.linspace(-3, 3, 50) + yy = np.linspace(-3, 3, 50) + X_grid, Y_grid = np.meshgrid(xx, yy) + Z = np.sin(np.sqrt(X_grid**2 + Y_grid**2)) + viz.surf( + X=Z, + opts=dict( + title="11. 3D Surface (sin(sqrt(x²+y²)))", + colormap="Hot", + ), + win="surface", + ) + print("[OK] 11. 3D surface plot") + + # =========================================================== + # 12. CONTOUR PLOT + # =========================================================== + viz.contour( + X=Z, + opts=dict(title="12. Contour Plot"), + win="contour", + ) + print("[OK] 12. Contour plot") + + # =========================================================== + # 13. PIE CHART + # =========================================================== + viz.pie( + X=np.array([30, 25, 20, 15, 10]), + opts=dict( + title="13. Pie Chart (Market Share)", + legend=["Chrome", "Firefox", "Safari", "Edge", "Other"], + ), + win="pie", + ) + print("[OK] 13. Pie chart") + + # =========================================================== + # 14. IMAGE — Random RGB + # =========================================================== + # Create a gradient image + img = np.zeros((3, 128, 256)) + img[0] = np.linspace(0, 1, 256).reshape(1, -1) # Red gradient → + img[1] = np.linspace(0, 1, 128).reshape(-1, 1) # Green gradient ↓ + img[2] = 0.5 # Constant blue + viz.image( + img, + opts=dict( + title="14. RGB Gradient Image", + caption="Red increases left→right, Green increases top→bottom", + ), + win="image_rgb", + ) + print("[OK] 14. RGB gradient image") + + # =========================================================== + # 15. IMAGE GRID — Multiple images + # =========================================================== + viz.images( + np.random.rand(16, 3, 64, 64), + nrow=4, + opts=dict(title="15. Image Grid (4x4)"), + win="image_grid", + ) + print("[OK] 15. Image grid") + + # =========================================================== + # 16. TEXT + # =========================================================== + viz.text( + "

Visual Test Report

" + "

All visualizations created successfully.

" + "" + "

All dependency tests PASSED

", + opts=dict(title="16. Text/HTML Window"), + win="text_report", + ) + print("[OK] 16. HTML text window") + + # =========================================================== + # 17. LINE PLOT — With markers and fill + # =========================================================== + x2 = np.linspace(0, 10, 50) + viz.line( + Y=np.column_stack((np.exp(-x2 / 3), np.exp(-x2 / 5))), + X=np.column_stack((x2, x2)), + opts=dict( + title="17. Line with Markers & Fill", + markers=True, + fillarea=True, + legend=["Fast decay", "Slow decay"], + markersize=5, + ), + win="line_markers_fill", + ) + print("[OK] 17. Line with markers and fill area") + + # =========================================================== + # DONE + # =========================================================== + print(f"\n{'='*60}") + print(f"ALL 17 VISUALIZATIONS CREATED SUCCESSFULLY") + print(f"{'='*60}") + print(f"\nView at: http://localhost:8097/env/{ENV}") + print("Manually verify each plot renders correctly in the browser.") + print("\nChecklist:") + print(" [ ] Line plots show smooth curves with correct legends") + print(" [ ] Scatter plots show colored points in 2D and 3D") + print(" [ ] Bar/histogram/box/pie charts render with labels") + print(" [ ] Heatmap shows color gradient with axis labels") + print(" [ ] Surface/contour show 3D data") + print(" [ ] Images render (gradient + grid)") + print(" [ ] Text window shows formatted HTML") + print(" [ ] Windows are draggable and resizable") + print(" [ ] Plots are zoomable and interactive (hover tooltips)") + + +if __name__ == "__main__": + main() diff --git a/py/tests/conftest.py b/py/tests/conftest.py new file mode 100644 index 000000000..7be7c11cc --- /dev/null +++ b/py/tests/conftest.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Fixtures shared by the whole suite. + +Everything here is hermetic: temporary directories only, no listening sockets, +no network. Tests that genuinely need an externally launched visdom must carry +``@pytest.mark.server`` so CI can deselect them. +""" + +from unittest.mock import patch + +import pytest + +from visdom.data_model.json_store import JSONStore +from visdom.server.app import Application +from visdom.utils import shared_utils + +from testutils.fakes import FakeHandler, FakeSocket, SpyStore + + +@pytest.fixture +def env_path(tmp_path): + """Disposable environment directory.""" + return str(tmp_path) + + +@pytest.fixture +def store(env_path): + """JSONStore backed by a temporary directory.""" + return JSONStore(env_path) + + +@pytest.fixture +def spy_store(env_path): + """JSONStore that records which backend methods the server calls.""" + return SpyStore(env_path) + + +@pytest.fixture +def app(env_path): + """Application instance with temporary persistence and no listener. + + The port is only recorded on the instance; nothing binds it, so this is + safe to construct in parallel with other tests. + """ + return Application(port=8097, env_path=env_path) + + +@pytest.fixture +def app_factory(env_path): + """Build Applications sharing one ``env_path``, for reload assertions.""" + + def build(**kwargs): + kwargs.setdefault("port", 8097) + kwargs.setdefault("env_path", env_path) + return Application(**kwargs) + + return build + + +@pytest.fixture +def fake_socket(): + return FakeSocket() + + +@pytest.fixture +def handler(env_path): + """Duck-typed handler with its own empty state and store.""" + return FakeHandler(env_path=env_path) + + +@pytest.fixture +def app_handler(app): + """Handler sharing a real Application's state, storage and subscribers.""" + return FakeHandler( + state=app.state, + storage=app.storage, + subs=app.subs, + sources=app.sources, + env_path=app.env_path, + ) + + +@pytest.fixture +def offline_client(): + """Visdom client that never opens a connection. + + With ``send=False`` the client's ``_send`` returns the payload instead of + performing I/O, so plot methods can be asserted as pure functions. + """ + import visdom + + return visdom.Visdom(send=False, use_incoming_socket=False) + + +@pytest.fixture +def capture_send(offline_client): + """Run a client call and return the payload it would have transmitted. + + Usage:: + + sent = capture_send(lambda v: v.line(Y=[1, 2, 3])) + assert sent["payload"]["data"][0]["type"] == "scatter" + """ + + def run(call, win_exists=None): + sent = {} + + def capture(msg, endpoint="events", **_): + sent["payload"] = msg + sent["endpoint"] = endpoint + return "win_capture" + + patches = [patch.object(offline_client, "_send", side_effect=capture)] + if win_exists is not None: + patches.append( + patch.object(offline_client, "win_exists", return_value=win_exists) + ) + for p in patches: + p.start() + try: + call(offline_client) + finally: + for p in reversed(patches): + p.stop() + return sent + + return run + + +@pytest.fixture(autouse=True) +def reset_warn_once(): + """Isolate ``shared_utils.warn_once`` between tests. + + It dedupes against a module-level set, so without this a warning raised by + an earlier test silently suppresses the same warning in a later one, and + assertions on it pass or fail depending on execution order. + """ + previous = set(shared_utils._seen_warnings) + shared_utils._seen_warnings.clear() + yield + shared_utils._seen_warnings.clear() + shared_utils._seen_warnings.update(previous) diff --git a/py/tests/integration/auth.py b/py/tests/integration/auth.py new file mode 100644 index 000000000..71c0d2686 --- /dev/null +++ b/py/tests/integration/auth.py @@ -0,0 +1,449 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Who may talk to the server, and what they may ask it to do. + +Two independent gates, both off by default: + +* ``-enable_login`` makes the server issue a signed ``user_password`` cookie + from ``IndexHandler.post`` and refuse every ``@check_auth`` route without + one. The browser sends ``sha256(password)``, never the password itself, and + the server stores ``hash_password`` of that. +* ``-readonly`` leaves reads alone and refuses writes, over sockets and now + over HTTP too. + +A login-enabled ``Application`` reads its cookie secret from +``DEFAULT_ENV_PATH``, which is the developer's real ``~/.visdom``. Every +fixture here points that at a temporary directory first, so nothing touches a +real installation. +""" + +import hashlib +import hmac +import json +import os +import unittest +from unittest.mock import patch + +import pytest + +from visdom.server import app as app_module +from visdom.server.app import Application +from visdom.server.handlers import web_handlers +from visdom.server.handlers.socket_handlers import SocketWrapper, VisSocketWrapper +from visdom.utils.server_utils import hash_password + +from testutils import socket_double +from testutils.http import VisdomHTTPTestCase + +pytestmark = pytest.mark.integration + +USERNAME = "visdom_user" +PASSWORD = "correct horse battery staple" +# What the login page actually posts: sjcl.hash.sha256 of the typed password. +CLIENT_HASH = hashlib.sha256(PASSWORD.encode("utf-8")).hexdigest() +COOKIE_SECRET = "test_cookie_secret" + +CREDENTIAL = {"username": USERNAME, "password": hash_password(CLIENT_HASH)} + + +class LoginTestCase(VisdomHTTPTestCase): + """Server started the way ``visdom -enable_login`` starts it.""" + + app_kwargs = {"user_credential": CREDENTIAL} + + def get_app(self): + with open(os.path.join(self.env_path, "COOKIE_SECRET"), "w") as secret_file: + secret_file.write(COOKIE_SECRET) + # Application reads DEFAULT_ENV_PATH + "COOKIE_SECRET" at construction + # and never again, so the patch only has to span the build. + with patch.object(app_module, "DEFAULT_ENV_PATH", self.env_path + os.sep): + return super().get_app() + + def login(self, username=USERNAME, password=CLIENT_HASH): + return self.fetch( + "/", + method="POST", + body=json.dumps({"username": username, "password": password}), + headers={"Content-Type": "application/json"}, + ) + + def session_headers(self): + """Headers carrying the cookie a successful login just issued.""" + resp = self.login() + self.assertEqual(resp.code, 200) + cookies = resp.headers.get_list("Set-Cookie") + self.assertTrue(cookies, "login issued no cookie") + return { + "Cookie": "; ".join(c.split(";")[0] for c in cookies), + "Content-Type": "application/json", + } + + def post_as_user(self, path, body): + return self.fetch( + path, method="POST", body=json.dumps(body), headers=self.session_headers() + ) + + +# -- The login page and its form --------------------------------------------- + + +class TestLoginPage(LoginTestCase): + def test_the_root_serves_the_login_page_to_a_stranger(self): + resp = self.fetch("/") + + self.assertEqual(resp.code, 200) + self.assertIn("Visdom Login", resp.body.decode()) + + def test_the_dashboard_is_not_served_to_a_stranger(self): + self.assertNotIn("visdom-container", self.fetch("/").body.decode()) + + def test_the_dashboard_is_served_once_logged_in(self): + resp = self.fetch("/", headers=self.session_headers()) + + self.assertEqual(resp.code, 200) + self.assertNotIn("Visdom Login", resp.body.decode()) + + +class TestLoginCredentials(LoginTestCase): + def test_the_right_credentials_are_accepted(self): + self.assertEqual(self.login().code, 200) + + def test_the_right_credentials_issue_a_cookie(self): + cookies = self.login().headers.get_list("Set-Cookie") + + self.assertTrue(any("user_password" in c for c in cookies)) + + def _assert_rejected(self, resp): + self.assertEqual(resp.code, 400) + self.assertEqual(resp.headers.get_list("Set-Cookie"), []) + + def test_a_wrong_password_is_rejected(self): + self._assert_rejected(self.login(password=hashlib.sha256(b"nope").hexdigest())) + + def test_a_wrong_username_is_rejected(self): + self._assert_rejected(self.login(username="somebody_else")) + + def test_the_raw_password_is_not_the_password(self): + """The server only ever sees the client-side sha256.""" + self._assert_rejected(self.login(password=PASSWORD)) + + def test_an_empty_password_is_rejected(self): + self._assert_rejected(self.login(password="")) + + def test_both_halves_are_always_compared(self): + """A wrong username must still cost a password comparison. + + The regression: ``==`` short-circuited, so a bad username skipped the + second check entirely and the response came back measurably sooner. + """ + with patch.object( + web_handlers.hmac, "compare_digest", wraps=hmac.compare_digest + ) as compare: + self.login(username="somebody_else") + + self.assertEqual(compare.call_count, 2) + + def test_the_stored_credential_is_not_the_client_hash(self): + """What is kept in memory is salted and derived, not the wire value.""" + stored = CREDENTIAL["password"] + + self.assertNotIn(CLIENT_HASH, stored) + self.assertEqual(len(stored.split("$")), 2) + + +# -- Authorization on the routes --------------------------------------------- + + +class TestUnauthenticatedRequests(LoginTestCase): + """``@check_auth`` answers 401 and does nothing else.""" + + def _assert_401(self, path, body): + resp = self.post_json(path, body) + self.assertEqual(resp.code, 401, path) + return resp + + def test_creating_a_window_is_unauthorized(self): + self._assert_401( + "/events", {"eid": "main", "data": [{"type": "text", "content": "hi"}]} + ) + self.assertEqual(self.panes(), {}) + + def test_updating_a_window_is_unauthorized(self): + self._assert_401( + "/update", {"eid": "main", "win": "w", "data": [{"content": "hi"}]} + ) + + def test_closing_a_window_is_unauthorized(self): + self._assert_401("/close", {"eid": "main", "win": "w"}) + + def test_deleting_an_environment_is_unauthorized(self): + self._app.state["expt"] = {"jsons": {}, "reload": {}} + + self._assert_401("/delete_env", {"eid": "expt"}) + + self.assertIn("expt", self._app.state) + + def test_saving_is_unauthorized(self): + self._assert_401("/save", {"data": ["main"]}) + + def test_forking_is_unauthorized(self): + self._assert_401("/fork_env", {"prev_eid": "main", "eid": "copy"}) + self.assertNotIn("copy", self._app.state) + + def test_reading_window_data_is_unauthorized(self): + self._assert_401("/win_data", {"eid": "main", "win": None}) + + def test_asking_whether_a_window_exists_is_unauthorized(self): + self._assert_401("/win_exists", {"eid": "main", "win": "w"}) + + def test_reading_the_environment_list_is_unauthorized(self): + self._assert_401("/env_state", {}) + + def test_logging_an_experiment_is_unauthorized(self): + self._assert_401("/experiments/log", {"eid": "main", "params": {"lr": 0.1}}) + + def test_an_unauthorized_response_carries_no_body(self): + self.assertEqual(self._assert_401("/env_state", {}).body, b"") + + +class TestAuthenticatedRequests(LoginTestCase): + """The same routes, with the cookie the login just issued.""" + + def test_a_window_can_be_created(self): + resp = self.post_as_user( + "/events", {"eid": "main", "data": [{"type": "text", "content": "hi"}]} + ) + + self.assertEqual(resp.code, 200) + self.assertIn(resp.body.decode(), self.panes()) + + def test_the_environment_list_can_be_read(self): + resp = self.post_as_user("/env_state", {}) + + self.assertEqual(resp.code, 200) + self.assertEqual(json.loads(resp.body), ["main"]) + + def test_a_forged_cookie_is_not_a_session(self): + resp = self.fetch( + "/env_state", + method="POST", + body="{}", + headers={"Cookie": "user_password=made_up"}, + ) + + self.assertEqual(resp.code, 401) + + +class TestHealthIsPublic(LoginTestCase): + def test_health_answers_without_a_session(self): + """Deliberately unauthenticated: it is what a load balancer polls.""" + resp = self.fetch("/health") + + self.assertEqual(resp.code, 200) + self.assertEqual(json.loads(resp.body), {"status": "ok"}) + + +class TestAuthDisabledByDefault(VisdomHTTPTestCase): + def test_no_login_means_no_cookie_is_needed(self): + self.assertEqual(self.post_json("/env_state", {}).code, 200) + + def test_the_root_serves_the_dashboard(self): + self.assertNotIn("Visdom Login", self.fetch("/").body.decode()) + + +# -- Sockets under login ----------------------------------------------------- + + +@pytest.fixture +def login_app(env_path, monkeypatch): + """Login-enabled Application whose cookie secret lives in ``env_path``.""" + with open(os.path.join(env_path, "COOKIE_SECRET"), "w") as secret_file: + secret_file.write(COOKIE_SECRET) + monkeypatch.setattr(app_module, "DEFAULT_ENV_PATH", env_path + os.sep) + return Application(port=8097, env_path=env_path, user_credential=CREDENTIAL) + + +def test_login_is_enabled_by_supplying_a_credential(login_app): + assert login_app.login_enabled is True + + +def test_an_unauthenticated_subscriber_socket_is_closed(login_app): + """``open`` closes before registering, so the socket never joins subs.""" + sub = socket_double(SocketWrapper, login_app) + + sub.open() + + assert login_app.subs == {} + assert list(sub.messages) == [] + + +def test_an_unauthenticated_source_socket_is_closed(login_app): + source = socket_double(VisSocketWrapper, login_app) + + source.open() + + assert login_app.sources == {} + assert list(source.messages) == [] + + +def test_a_socket_opens_when_login_is_disabled(app): + """The same double registers fine against a server without login.""" + sub = socket_double(SocketWrapper, app) + + sub.open() + + assert list(app.subs) == [sub.sid] + + +# -- Readonly enforcement ---------------------------------------------------- + + +class TestReadonlyRejectsWrites(VisdomHTTPTestCase): + """Every mutating route answers 403 and changes nothing.""" + + app_kwargs = {"readonly": True} + + def _assert_403(self, path, body): + resp = self.post_json(path, body) + self.assertEqual(resp.code, 403, path) + self.assertFalse(json.loads(resp.body)["success"]) + self.assertIn("readonly", json.loads(resp.body)["error"]) + return resp + + def _seed(self, eid="main"): + """Put a pane in state directly, since /events is refused here.""" + self._app.state.setdefault(eid, {"jsons": {}, "reload": {}}) + self._app.state[eid]["jsons"]["win_seed"] = { + "id": "win_seed", + "type": "text", + "content": "seeded", + "i": 0, + "version": 1, + } + return "win_seed" + + def test_creating_a_window_is_refused(self): + self._assert_403( + "/events", {"eid": "main", "data": [{"type": "text", "content": "hi"}]} + ) + + self.assertEqual(self.panes(), {}) + + def test_updating_a_window_is_refused(self): + win = self._seed() + + self._assert_403( + "/update", + {"eid": "main", "win": win, "data": [{"type": "text", "content": "new"}]}, + ) + + self.assertEqual(self.panes()[win]["content"], "seeded") + + def test_closing_a_window_is_refused(self): + win = self._seed() + + self._assert_403("/close", {"eid": "main", "win": win}) + + self.assertIn(win, self.panes()) + + def test_deleting_an_environment_is_refused(self): + self._app.state["expt"] = {"jsons": {}, "reload": {}} + + self._assert_403("/delete_env", {"eid": "expt"}) + + self.assertIn("expt", self._app.state) + + def test_saving_is_refused(self): + """The pane held in memory never reaches the env file on disk. + + ``main.json`` itself exists either way — ``load_state`` writes it at + startup, before any request has been served — so the assertion is on + its contents. + """ + self._seed() + + self._assert_403("/save", {"data": ["main"]}) + + with open(os.path.join(self.env_path, "main.json")) as env_file: + self.assertEqual(json.load(env_file)["jsons"], {}) + + def test_forking_is_refused(self): + self._assert_403("/fork_env", {"prev_eid": "main", "eid": "copy"}) + + self.assertNotIn("copy", self._app.state) + + def test_pushing_window_data_is_refused(self): + """The write behind /win_data, which also serves reads.""" + self._assert_403( + "/win_data", {"eid": "main", "win": None, "data": json.dumps({})} + ) + + def test_uploading_an_environment_is_refused(self): + resp = self.post_json("/upload_env", {}) + + self.assertEqual(resp.code, 403) + self.assertFalse(json.loads(resp.body)["success"]) + + def test_logging_an_experiment_is_refused(self): + resp = self.post_json("/experiments/log", {"eid": "main", "params": {"a": 1}}) + + self.assertEqual(resp.code, 403) + + +class TestReadonlyAllowsReads(VisdomHTTPTestCase): + app_kwargs = {"readonly": True} + + def test_the_environment_list_is_served(self): + resp = self.post_json("/env_state", {}) + + self.assertEqual(resp.code, 200) + self.assertEqual(json.loads(resp.body), ["main"]) + + def test_window_data_is_served(self): + resp = self.post_json("/win_data", {"eid": "main", "win": None}) + + self.assertEqual(resp.code, 200) + self.assertEqual(json.loads(resp.body), {}) + + def test_window_existence_is_answered(self): + resp = self.post_json("/win_exists", {"eid": "main", "win": "nope"}) + + self.assertEqual(resp.code, 200) + self.assertEqual(resp.body, b"false") + + def test_the_environment_page_renders(self): + self.assertEqual(self.fetch("/env/main").code, 200) + + def test_health_is_served(self): + self.assertEqual(self.fetch("/health").code, 200) + + +class TestWritesSurviveWithoutReadonly(VisdomHTTPTestCase): + """The decorator must not be refusing writes on an ordinary server.""" + + def test_every_guarded_route_still_answers_200(self): + win = self.create_text_window(content="writable") + + self.assertEqual( + [ + self.update(win, [{"type": "text", "content": "more"}]).code, + self.post_json("/save", {"data": ["main"]}).code, + self.post_json("/fork_env", {"prev_eid": "main", "eid": "copy"}).code, + self.post_json( + "/win_data", {"eid": "main", "win": None, "data": json.dumps({})} + ).code, + self.close_window(win).code, + self.post_json("/delete_env", {"eid": "copy"}).code, + ], + [200] * 6, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/integration/build_verification.py b/py/tests/integration/build_verification.py new file mode 100644 index 000000000..392513e33 --- /dev/null +++ b/py/tests/integration/build_verification.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Checks that the frontend build output exists and is served. + +Two kinds of asset live under ``py/visdom/static``: + +* **tracked** — ``js/main.js`` and its source map, the three HTML pages, and the + stylesheets under ``css/``. They are committed, so they are asserted outright. +* **downloaded** — jQuery, bootstrap and d3, fetched by ``build.py`` and ignored + by git (``.gitignore:17,19``). CI installs the package and runs pytest without + ever building the frontend, so these are absent there. Asserting them would + make the job red for a reason that has nothing to do with the code under test, + so each is skipped when it is not on disk and checked when it is. +""" + +import os + +import pytest + +from testutils import VisdomHTTPTestCase +from visdom.utils.shared_utils import get_visdom_path + +pytestmark = pytest.mark.integration + +# Committed by the repo; a missing one means the build output was clobbered. +TRACKED_ASSETS = [ + "index.html", + "login.html", + "error.html", + os.path.join("js", "main.js"), + os.path.join("js", "main.js.map"), + os.path.join("css", "style.css"), + os.path.join("css", "login.css"), + os.path.join("css", "error.css"), + os.path.join("css", "network.css"), + os.path.join("css", "rc-tree-select-overrides.css"), +] + +# Written by build.py at install time; absent in a plain checkout. +DOWNLOADED_ASSETS = [ + "js/jquery.min.js", + "js/d3.v3.min.js", + "css/bootstrap.min.css", +] + +# webpack emits well over a megabyte; anything near zero means a broken build +# that still produced a file. +MIN_BUNDLE_BYTES = 100_000 + + +def _static(relpath): + return os.path.join(get_visdom_path("static"), relpath) + + +@pytest.mark.parametrize("relpath", TRACKED_ASSETS) +def test_tracked_asset_exists(relpath): + """Every committed static asset is present in the installed package.""" + assert os.path.exists(_static(relpath)), f"missing build artifact: {relpath}" + + +def test_main_bundle_is_not_a_stub(): + """main.js is a real bundle, not an empty or truncated file.""" + size = os.path.getsize(_static(os.path.join("js", "main.js"))) + assert size > MIN_BUNDLE_BYTES, f"bundle suspiciously small: {size} bytes" + + +class TestStaticServing(VisdomHTTPTestCase): + """The Application serves the build output over HTTP.""" + + def _assert_served(self, url): + resp = self.fetch(url) + self.assertEqual(resp.code, 200, f"{url} returned {resp.code}") + return resp + + def test_index_page_returns_html(self): + body = self._assert_served("/").body.decode() + self.assertIn("alert(1)') + + +class TestDeleteMissingEnv(VisdomHTTPTestCase): + def test_deleting_an_unknown_env_is_a_no_op(self): + self.assertEqual( + self.post_json("/delete_env", {"eid": "never_existed"}).code, 200 + ) + + def test_deleting_a_none_env_is_a_no_op(self): + self.assertEqual(self.post_json("/delete_env", {"eid": None}).code, 200) + + +class TestErrorPageDetails(VisdomHTTPTestCase): + """What a 500 tells the client on an ordinary server. + + The error page renders the exception, its traceback and the request the + handler was serving whenever the app is built with error details on. That + used to be unconditional, so ``error.html``'s own "what happened" branch + for a production server was unreachable. + """ + + def test_the_status_is_still_reported(self): + body = self.fetch("/error/500").body.decode() + + self.assertIn("500", body) + self.assertIn("Internal Server Error", body) + + def test_the_production_message_is_shown_instead(self): + self.assertIn("What happened", self.fetch("/error/500").body.decode()) + + def test_the_traceback_is_not_rendered(self): + self.assertNotIn("Traceback", self.fetch("/error/500").body.decode()) + + def test_the_source_paths_are_not_rendered(self): + self.assertNotIn("web_handlers.py", self.fetch("/error/500").body.decode()) + + def test_the_request_is_not_rendered(self): + body = self.fetch("/error/500").body.decode() + + self.assertNotIn("Remote IP", body) + self.assertNotIn("127.0.0.1", body) + + +class TestErrorPageUnderDebugLogging(VisdomHTTPTestCase): + """``-logging_level DEBUG`` is the operator asking for the detail back.""" + + def get_app(self): + root = logging.getLogger() + previous = root.level + root.setLevel(logging.DEBUG) + try: + return super().get_app() + finally: + root.setLevel(previous) + + def test_the_traceback_is_rendered(self): + self.assertIn("Traceback", self.fetch("/error/500").body.decode()) + + def test_the_request_is_rendered(self): + body = self.fetch("/error/500").body.decode() + + self.assertIn("Remote IP", body) + self.assertIn("127.0.0.1", body) + + +class TestRenderedPages(VisdomHTTPTestCase): + def test_env_page_renders(self): + self.assertEqual(self.fetch("/env/main").code, 200) + + def test_compare_page_renders(self): + self.assertEqual(self.fetch("/compare/main+main").code, 200) + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/integration/environment_lifecycle.py b/py/tests/integration/environment_lifecycle.py new file mode 100644 index 000000000..1572649e5 --- /dev/null +++ b/py/tests/integration/environment_lifecycle.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Environment lifecycle over real HTTP: create, fork, save, delete, reload. + +An environment is never created explicitly -- it appears the moment a pane is +addressed to it. From there ``/fork_env`` deep-copies it, ``/save`` writes it +through the storage backend, ``/delete_env`` removes both the state and the +file, and a fresh ``Application`` over the same directory picks it back up. +That the writes go through the backend rather than around it is asserted +separately in ``storage_wiring``. +""" + +import json +import os +import unittest + +import pytest + +from visdom.server.app import Application + +from testutils.http import VisdomHTTPTestCase + +pytestmark = pytest.mark.integration + + +class TestStartupState(VisdomHTTPTestCase): + def test_main_env_exists_and_is_empty_on_startup(self): + self.assertIn("main", self.get_envs()) + self.assertEqual(self.get_win_data(), {}) + + def test_env_state_returns_a_list(self): + self.assertIsInstance(self.get_envs(), list) + + +class TestImplicitCreation(VisdomHTTPTestCase): + def test_addressing_an_unknown_env_creates_it(self): + self.create_text_window(eid="new_env", content="hello") + self.assertIn("new_env", self.get_envs()) + + def test_env_state_lists_every_env_created(self): + self.create_text_window(eid="x_env") + self.create_text_window(eid="y_env") + self.assertTrue({"main", "x_env", "y_env"} <= set(self.get_envs())) + + +class TestForkEnv(VisdomHTTPTestCase): + def test_fork_copies_the_panes_across(self): + self.create_text_window(eid="main", content="original", win="w1") + resp = self.post_json("/fork_env", {"prev_eid": "main", "eid": "fork1"}) + self.assertEqual(resp.code, 200) + self.assertEqual(resp.body.decode(), "fork1") + self.assertEqual(self.get_win_data("w1", eid="fork1")["content"], "original") + + def test_fork_is_independent_of_its_source(self): + self.create_text_window(eid="main", content="original", win="w1") + self.post_json("/fork_env", {"prev_eid": "main", "eid": "fork2"}) + + self.update("w1", [{"content": "modified"}], eid="fork2") + + self.assertEqual(self.get_win_data("w1", eid="main")["content"], "original") + + def test_forking_an_unknown_env_is_a_bad_request(self): + resp = self.post_json("/fork_env", {"prev_eid": "no_exist", "eid": "new"}) + self.assertEqual(resp.code, 400) + self.assertIn("env to be forked doesn't exist", resp.reason) + self.assertNotIn("new", self.get_envs()) + + +class TestSaveEnv(VisdomHTTPTestCase): + def test_save_writes_a_file_named_for_the_env(self): + self.create_text_window(eid="main", content="save me") + self.save(["main"]) + self.assertTrue(os.path.exists(os.path.join(self.env_path, "main.json"))) + + def test_saved_file_carries_jsons_and_reload(self): + self.create_text_window(eid="main", content="save me") + self.save(["main"]) + with open(os.path.join(self.env_path, "main.json")) as handle: + saved = json.load(handle) + self.assertIn("jsons", saved) + self.assertIn("reload", saved) + + def test_save_handles_several_envs_at_once(self): + self.create_text_window(eid="env_a", content="a") + self.create_text_window(eid="env_b", content="b") + self.save(["env_a", "env_b"]) + for eid in ("env_a", "env_b"): + self.assertTrue(os.path.exists(os.path.join(self.env_path, eid + ".json"))) + + def test_save_reports_only_the_envs_that_existed(self): + self.create_text_window(eid="main", content="x") + saved = json.loads(self.save(["main", "nonexistent"]).body) + self.assertIn("main", saved) + self.assertNotIn("nonexistent", saved) + + +class TestDeleteEnv(VisdomHTTPTestCase): + def test_delete_removes_the_env_from_state(self): + self.create_text_window(eid="del_me", content="bye") + self.post_json("/delete_env", {"eid": "del_me"}) + self.assertNotIn("del_me", self.get_envs()) + + def test_delete_removes_the_saved_file_too(self): + self.create_text_window(eid="del_file", content="bye") + self.save(["del_file"]) + path = os.path.join(self.env_path, "del_file.json") + self.assertTrue(os.path.exists(path)) + + self.post_json("/delete_env", {"eid": "del_file"}) + + self.assertFalse(os.path.exists(path)) + + def test_main_env_cannot_be_deleted(self): + self.post_json("/delete_env", {"eid": "main"}) + self.assertIn("main", self.get_envs()) + + +class TestReload(VisdomHTTPTestCase): + def test_a_saved_env_is_reloaded_by_a_fresh_application(self): + self.create_text_window(eid="persist", content="I survive", win="w1") + self.save(["persist"]) + + restarted = Application(port=8097, env_path=self.env_path) + + self.assertIn("persist", restarted.state) + self.assertEqual( + restarted.state["persist"]["jsons"]["w1"]["content"], "I survive" + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/test_experiment_log_handler.py b/py/tests/integration/experiment_log_handler.py similarity index 100% rename from py/tests/test_experiment_log_handler.py rename to py/tests/integration/experiment_log_handler.py diff --git a/py/tests/integration/polling_parity.py b/py/tests/integration/polling_parity.py new file mode 100644 index 000000000..8c34cb27e --- /dev/null +++ b/py/tests/integration/polling_parity.py @@ -0,0 +1,448 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""The polling transport, and its parity with the WebSocket one. + +``AGENTS.md`` requires every socket feature to work over both transports. +Polling replaces the socket with two routes: ``/socket_wrap`` for subscribers +and ``/vis_socket_wrap`` for sources. A client asks for a sid, then POSTs +``message_type: "send"`` to push a command and ``message_type: "query"`` to +drain whatever the server has queued for it. Server-side each sid owns a +``SocketWrapper`` whose ``write_message`` appends to a deque instead of writing +to a connection, so the same ``on_message`` dispatch runs either way. + +The HTTP round trip is what is under test here, so these are +``VisdomHTTPTestCase`` subclasses. The reaper and the queue are reachable +without a request and stay plain functions. +""" + +import json +import time +import types +import unittest + +import pytest + +from visdom.server.defaults import MAX_SOCKET_WAIT +from visdom.server.handlers.socket_handlers import SocketWrapper, VisSocketWrapper + +from testutils import socket_double +from testutils.http import VisdomHTTPTestCase + +pytestmark = pytest.mark.integration + + +class PollingTestCase(VisdomHTTPTestCase): + """Server configured the way ``visdom -use_frontend_client_polling`` runs. + + ``sub_sid`` / ``source_sid`` mint an id the way the two client kinds do: + the browser GETs ``/socket_wrap``, the python client POSTs to + ``/vis_socket_wrap`` without one. + """ + + app_kwargs = {"use_frontend_client_polling": True} + + def sub_sid(self): + return json.loads(self.fetch("/socket_wrap").body)["sid"] + + def source_sid(self): + return json.loads(self.poll("/vis_socket_wrap", None, "query").body)["sid"] + + def poll(self, route, sid, message_type, message=None): + body = {"sid": sid, "message_type": message_type} + if message is not None: + body["message"] = message + return self.post_json(route, body) + + def query(self, sid, route="/socket_wrap"): + """Drain the queued messages for ``sid``, decoded.""" + resp = json.loads(self.poll(route, sid, "query").body) + self.assertTrue(resp["success"], resp) + return [json.loads(m) for m in resp["messages"]] + + def send(self, sid, command, route="/socket_wrap"): + return self.poll(route, sid, "send", json.dumps(command)) + + +# -- Minting a connection ---------------------------------------------------- + + +class TestSubscriberHandshake(PollingTestCase): + def test_get_mints_a_sid(self): + resp = json.loads(self.fetch("/socket_wrap").body) + + self.assertTrue(resp["success"]) + self.assertTrue(resp["sid"]) + + def test_each_client_gets_its_own_sid(self): + self.assertNotEqual(self.sub_sid(), self.sub_sid()) + + def test_the_sid_is_registered_as_a_subscriber(self): + sid = self.sub_sid() + + self.assertIn(sid, self._app.subs) + self.assertEqual(self._app.sources, {}) + + def test_the_first_query_returns_the_register_handshake(self): + """register, layouts, envs — then a fourth, redundant layout push. + + ``SocketHandlerOrWrapper.initialize`` broadcasts layouts to every + subscriber, and on this path the socket has already registered itself + by then, because ``AnySocketWrapper.initialize`` opens it first. So it + receives its own broadcast. A WebSocket subscriber opens *after* + ``initialize`` and sees three messages. Idempotent either way, but it + is the one place the transports differ, so it is pinned here rather + than left to surprise the next reader. + """ + sid = self.sub_sid() + + messages = self.query(sid) + + self.assertEqual( + [m["command"] for m in messages], + ["register", "layout_update", "env_update", "layout_update"], + ) + self.assertEqual(messages[0]["data"], sid) + + def test_a_new_client_re_pushes_layouts_to_the_existing_ones(self): + """Same cause as above, seen from the other side of the registry.""" + first = self.sub_sid() + self.query(first) + + self.sub_sid() + + self.assertEqual([m["command"] for m in self.query(first)], ["layout_update"]) + + def test_the_handshake_is_delivered_only_once(self): + sid = self.sub_sid() + self.query(sid) + + self.assertEqual(self.query(sid), []) + + +class TestSourceHandshake(PollingTestCase): + def test_a_post_without_a_sid_mints_one(self): + resp = json.loads(self.poll("/vis_socket_wrap", None, "query").body) + + self.assertTrue(resp["success"]) + self.assertTrue(resp["sid"]) + + def test_the_sid_is_registered_as_a_source(self): + sid = self.source_sid() + + self.assertIn(sid, self._app.sources) + self.assertEqual(self._app.subs, {}) + + def test_the_source_handshake_is_an_alive_ping(self): + sid = self.source_sid() + + self.assertEqual( + self.query(sid, route="/vis_socket_wrap"), + [{"command": "alive", "data": "vis_alive"}], + ) + + def test_minting_twice_yields_two_sources(self): + self.source_sid() + self.source_sid() + + self.assertEqual(len(self._app.sources), 2) + + +# -- Protocol errors --------------------------------------------------------- + + +class TestPollingProtocolErrors(PollingTestCase): + def _assert_failure(self, resp, reason): + body = json.loads(resp.body) + self.assertEqual(resp.code, 200) + self.assertFalse(body["success"]) + self.assertEqual(body["reason"], reason) + self.assertTrue(body["detail"]) + return body + + def test_an_unknown_sid_is_reported_as_closed(self): + body = self._assert_failure( + self.poll("/socket_wrap", "never_minted", "query"), "closed" + ) + + self.assertIn("never_minted", body["message"]) + + def test_a_subscriber_sid_is_not_a_source_sid(self): + """The two registries are separate; a sid is only valid on its route.""" + sid = self.sub_sid() + + self._assert_failure(self.poll("/vis_socket_wrap", sid, "query"), "closed") + + def test_send_without_a_message_is_rejected(self): + sid = self.sub_sid() + + self._assert_failure(self.poll("/socket_wrap", sid, "send"), "no msg") + + def test_an_unknown_message_type_is_rejected(self): + sid = self.sub_sid() + + body = self._assert_failure( + self.poll("/socket_wrap", sid, "subscribe"), "invalid" + ) + + self.assertIn("subscribe", body["message"]) + + def test_a_missing_message_type_is_rejected(self): + sid = self.sub_sid() + + self._assert_failure(self.post_json("/socket_wrap", {"sid": sid}), "invalid") + + def test_a_closed_socket_stops_answering(self): + sid = self.sub_sid() + self._app.subs[sid].close() + + self._assert_failure(self.poll("/socket_wrap", sid, "query"), "closed") + + +# -- Commands over the polling transport ------------------------------------- + + +class TestPollingCommandParity(PollingTestCase): + """Each command has the same effect as it does over a WebSocket.""" + + def test_close_removes_the_pane(self): + win = self.create_text_window(content="polled") + sid = self.sub_sid() + + resp = self.send(sid, {"cmd": "close", "eid": "main", "data": win}) + + self.assertTrue(json.loads(resp.body)["success"]) + self.assertNotIn(win, self.panes()) + + def test_close_reaches_the_sources(self): + win = self.create_text_window(content="polled") + source_sid = self.source_sid() + self.query(source_sid, route="/vis_socket_wrap") + sid = self.sub_sid() + + self.send(sid, {"cmd": "close", "eid": "main", "data": win}) + + forwarded = self.query(source_sid, route="/vis_socket_wrap") + self.assertEqual(forwarded[0]["event_type"], "close") + self.assertEqual(forwarded[0]["target"], win) + self.assertIsNotNone(forwarded[0]["pane_data"]) + + def test_undo_restores_the_pane(self): + win = self.create_text_window(content="polled") + sid = self.sub_sid() + self.send(sid, {"cmd": "close", "eid": "main", "data": win}) + + self.send(sid, {"cmd": "undo", "eid": "main"}) + + self.assertIn(win, self.panes()) + + def test_delete_env_removes_the_environment(self): + self.create_text_window(eid="expt", content="polled") + sid = self.sub_sid() + + self.send(sid, {"cmd": "delete_env", "eid": "expt"}) + + self.assertNotIn("expt", self._app.state) + + def test_save_layouts_updates_the_application(self): + sid = self.sub_sid() + layouts = '[["view A", {"win_0": [0, 0, 3, 3]}]]' + + self.send(sid, {"cmd": "save_layouts", "data": layouts}) + + self.assertEqual(self._app.layouts, layouts) + + def test_save_layouts_is_broadcast_back_to_the_poller(self): + sid = self.sub_sid() + self.query(sid) + layouts = '[["view B", {}]]' + + self.send(sid, {"cmd": "save_layouts", "data": layouts}) + + self.assertEqual( + self.query(sid), [{"command": "layout_update", "data": layouts}] + ) + + def test_a_source_can_save_layouts_too(self): + """The command is handled on the shared base, not the subscriber.""" + sid = self.source_sid() + layouts = '[["view C", {}]]' + + resp = self.send( + sid, {"cmd": "save_layouts", "data": layouts}, "/vis_socket_wrap" + ) + + self.assertTrue(json.loads(resp.body)["success"]) + self.assertEqual(self._app.layouts, layouts) + + def test_update_comment_lands_on_the_pane(self): + win = self.create_text_window(content="polled") + sid = self.sub_sid() + + self.send( + sid, {"cmd": "update_comment", "eid": "main", "win": win, "data": "note"} + ) + + self.assertEqual(self.panes()[win]["comment"], "note") + + def test_layout_item_update_is_recorded(self): + win = self.create_text_window(content="polled") + sid = self.sub_sid() + + self.send( + sid, + {"cmd": "layout_item_update", "eid": "main", "win": win, "data": [0, 0]}, + ) + + self.assertEqual(self._app.state["main"]["reload"][win], [0, 0]) + + def test_echo_comes_back_on_the_source_route(self): + sid = self.source_sid() + self.query(sid, route="/vis_socket_wrap") + + self.send(sid, {"cmd": "echo", "data": "ping"}, "/vis_socket_wrap") + + self.assertEqual( + self.query(sid, route="/vis_socket_wrap"), + [{"cmd": "echo", "data": "ping"}], + ) + + def test_a_broadcast_reaches_every_polling_subscriber(self): + first, second = self.sub_sid(), self.sub_sid() + self.query(first) + self.query(second) + + self.create_text_window(content="broadcast") + + self.assertEqual(len(self.query(first)), 1) + self.assertEqual(len(self.query(second)), 1) + + +class TestPollingUnderReadonly(PollingTestCase): + app_kwargs = {"use_frontend_client_polling": True, "readonly": True} + + def test_the_handshake_reports_the_mode(self): + sid = self.sub_sid() + + self.assertTrue(self.query(sid)[0]["readonly"]) + + def test_a_command_is_accepted_but_dropped(self): + self._app.state["expt"] = {"jsons": {}, "reload": {}} + sid = self.sub_sid() + + resp = self.send(sid, {"cmd": "delete_env", "eid": "expt"}) + + self.assertTrue(json.loads(resp.body)["success"]) + self.assertIn("expt", self._app.state) + + +# -- The idle reaper --------------------------------------------------------- +# +# ``socket_wrap_monitor_thread`` runs on a 15s PeriodicCallback and closes +# polling clients that have stopped reading. Calling it directly needs no +# loop, only a monitor object to stop. + + +def _reaper(app, sockets): + """Register ``sockets`` and return the wrapper the monitor runs on.""" + stopped = [] + app.socket_wrap_monitor = types.SimpleNamespace(stop=lambda: stopped.append(True)) + for sock in sockets: + sock.open() + return sockets[0] if sockets else socket_double(SocketWrapper, app), stopped + + +def test_the_reaper_closes_an_idle_subscriber(app): + sub, _ = _reaper(app, [socket_double(SocketWrapper, app)]) + sub.last_read_time = time.time() - MAX_SOCKET_WAIT - 1 + + sub.socket_wrap_monitor_thread() + + assert app.subs == {} + + +def test_the_reaper_keeps_a_subscriber_that_still_polls(app): + sub, _ = _reaper(app, [socket_double(SocketWrapper, app)]) + sub.last_read_time = time.time() + + sub.socket_wrap_monitor_thread() + + assert list(app.subs) == [sub.sid] + + +def test_the_reaper_closes_an_idle_source(app): + source, _ = _reaper(app, [socket_double(VisSocketWrapper, app)]) + source.last_read_time = time.time() - MAX_SOCKET_WAIT - 1 + + source.socket_wrap_monitor_thread() + + assert app.sources == {} + + +def test_the_reaper_leaves_the_other_clients_alone(app): + idle = socket_double(SocketWrapper, app) + active = socket_double(SocketWrapper, app) + _reaper(app, [idle, active]) + idle.last_read_time = time.time() - MAX_SOCKET_WAIT - 1 + active.last_read_time = time.time() + + idle.socket_wrap_monitor_thread() + + assert list(app.subs) == [active.sid] + + +def test_the_reaper_stops_itself_once_everyone_has_gone(app): + sub, stopped = _reaper(app, [socket_double(SocketWrapper, app)]) + sub.on_close() + + sub.socket_wrap_monitor_thread() + + assert stopped == [True] + + +def test_a_query_postpones_the_reaper(app): + sub, _ = _reaper(app, [socket_double(SocketWrapper, app)]) + sub.last_read_time = time.time() - MAX_SOCKET_WAIT - 1 + + sub.get_messages() + sub.socket_wrap_monitor_thread() + + assert list(app.subs) == [sub.sid] + + +# -- The pending-message queue ----------------------------------------------- + + +def test_the_queue_holds_everything_written_between_polls(app): + """Nothing bounds ``messages``; a client that stops polling accumulates. + + Pinned down as current behaviour. The reaper above is what keeps it from + growing forever, so the two belong together: drop the reaper and this deque + is an unbounded leak per abandoned client. + """ + sub = socket_double(SocketWrapper, app) + for n in range(200): + sub.write_message(json.dumps({"n": n})) + + drained = sub.get_messages() + + assert len(drained) == 200 + assert [json.loads(m)["n"] for m in drained] == list(range(200)) + + +def test_the_queue_is_drained_in_arrival_order(app): + sub = socket_double(SocketWrapper, app) + sub.write_message("first") + sub.write_message("second") + + assert sub.get_messages() == ["first", "second"] + assert sub.get_messages() == [] + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/integration/socket_commands.py b/py/tests/integration/socket_commands.py new file mode 100644 index 000000000..532c892a7 --- /dev/null +++ b/py/tests/integration/socket_commands.py @@ -0,0 +1,845 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Every command ``on_message`` dispatches, driven through a real socket. + +``AnySocketHandlerOrWrapper.on_message`` is one long ``elif`` chain and is the +only entry point for pane closing, undo, environment deletion, layout edits, +comments and the embeddings drill-down. None of it needed a WebSocket to test: +``testutils.socket_double`` builds the real handler classes over a real +``Application`` with a recording ``write_message``. + +Two commands hand work to ``IOLoop.current().run_in_executor``. The +``inline_executor`` fixture replaces that with a synchronous stand-in, so the +side effect has happened by the time the call returns and the assertions do not +race a thread pool. +""" + +import json +import types + +import pytest +import tornado.ioloop + +from visdom.server.defaults import DEFAULT_MAX_UNDO_HISTORY +from visdom.utils.server_utils import count_deleted, push_deleted + +from testutils import commands, last, open_source, open_sub, sent + +pytestmark = pytest.mark.integration + + +@pytest.fixture +def inline_executor(monkeypatch): + """Run ``IOLoop.current().run_in_executor`` calls immediately, in-thread. + + Records ``(func, args)`` so a test can assert what was scheduled as well as + what it did. + """ + scheduled = [] + + def run_in_executor(_self_executor, func, *args): + scheduled.append((func, args)) + return func(*args) + + loop = types.SimpleNamespace( + run_in_executor=lambda executor, func, *args: run_in_executor( + executor, func, *args + ) + ) + monkeypatch.setattr(tornado.ioloop.IOLoop, "current", staticmethod(lambda: loop)) + return scheduled + + +def send(sock, **msg): + """Dispatch one command through the real ``on_message``.""" + sock.on_message(json.dumps(msg)) + + +def pane(win="win_0", ptype="plot", content=None, **extra): + """A pane as it sits in ``state[eid]["jsons"]``.""" + p = { + "id": win, + "i": 0, + "type": ptype, + "content": {"data": [], "layout": {}} if content is None else content, + } + p.update(extra) + return p + + +@pytest.fixture +def env(app): + """``app`` with an ``expt`` environment holding one pane.""" + app.state["expt"] = {"jsons": {"win_0": pane()}, "reload": {}} + return app + + +# -- close ------------------------------------------------------------------- + + +def test_close_removes_the_pane(env): + sub = open_sub(env) + send(sub, cmd="close", eid="expt", data="win_0") + + assert env.state["expt"]["jsons"] == {} + + +def test_close_pushes_the_pane_onto_the_undo_stack(env): + sub = open_sub(env) + send(sub, cmd="close", eid="expt", data="win_0") + + assert count_deleted(env.storage, "expt") == 1 + + +def test_close_forwards_the_pane_data_to_sources(env): + """The close event carries the pane that was removed, not ``None``. + + The pane used to be popped a second time under the unescaped eid, so by the + time the event was built there was nothing left to attach and every source + received ``pane_data: None``. + """ + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="close", eid="expt", data="win_0") + + event = sent(source)[-1] + assert event["event_type"] == "close" + assert event["target"] == "win_0" + assert event["eid"] == "expt" + assert event["pane_data"]["id"] == "win_0" + + +def test_close_of_an_unknown_pane_reports_no_pane_data(env): + """Closing something that is not there still tells the sources.""" + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="close", eid="expt", data="ghost") + + assert sent(source)[-1]["pane_data"] is None + assert count_deleted(env.storage, "expt") == 0 + + +def test_close_broadcasts_the_undo_count(env): + sub = open_sub(env) + sub.eid = "expt" + + send(sub, cmd="close", eid="expt", data="win_0") + + undo_state = last(sub, "undo_state") + assert undo_state["eid"] == "expt" + assert undo_state["count"] == 1 + + +def test_close_escapes_the_environment_id(app): + """A slash in the eid is escaped before the state lookup.""" + app.state["a_b"] = {"jsons": {"win_0": pane()}, "reload": {}} + source = open_source(app) + sub = open_sub(app) + + send(sub, cmd="close", eid="a/b", data="win_0") + + assert app.state["a_b"]["jsons"] == {} + assert sent(source)[-1]["eid"] == "a_b" + + +def test_close_of_an_unknown_environment_is_a_noop(env): + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="close", eid="ghost", data="win_0") + + assert sent(source) == [{"command": "alive", "data": "vis_alive"}] + assert env.state["expt"]["jsons"] != {} + + +def test_close_without_a_target_is_ignored(env): + sub = open_sub(env) + + send(sub, cmd="close", eid="expt") + + assert "win_0" in env.state["expt"]["jsons"] + + +# -- undo -------------------------------------------------------------------- + + +def test_undo_restores_the_last_closed_pane(env): + sub = open_sub(env) + send(sub, cmd="close", eid="expt", data="win_0") + + send(sub, cmd="undo", eid="expt") + + assert "win_0" in env.state["expt"]["jsons"] + + +def test_undo_puts_the_pane_at_the_end_of_the_order(env): + """A restored pane gets a fresh index so it cannot collide.""" + env.state["expt"]["jsons"]["win_1"] = pane("win_1") + env.state["expt"]["jsons"]["win_1"]["i"] = 5 + sub = open_sub(env) + send(sub, cmd="close", eid="expt", data="win_0") + + send(sub, cmd="undo", eid="expt") + + assert env.state["expt"]["jsons"]["win_0"]["i"] == 6 + + +def test_undo_broadcasts_the_restored_pane(env): + sub = open_sub(env) + sub.eid = "expt" + send(sub, cmd="close", eid="expt", data="win_0") + + send(sub, cmd="undo", eid="expt") + + restored = [m for m in sent(sub) if m.get("id") == "win_0"] + assert restored + assert restored[-1]["eid"] == "expt" + + +def test_undo_on_an_empty_stack_only_reports_the_count(env): + sub = open_sub(env) + sub.eid = "expt" + + send(sub, cmd="undo", eid="expt") + + assert last(sub, "undo_state")["count"] == 0 + assert env.state["expt"]["jsons"].keys() == {"win_0"} + + +def test_undo_stack_is_capped(env): + """Closing more panes than the cap keeps only the most recent ones.""" + for i in range(DEFAULT_MAX_UNDO_HISTORY + 3): + push_deleted(env.storage, "expt", f"win_{i}", pane(f"win_{i}")) + + assert count_deleted(env.storage, "expt") == DEFAULT_MAX_UNDO_HISTORY + + +def test_undo_of_an_unknown_environment_is_a_noop(env): + sub = open_sub(env) + + send(sub, cmd="undo", eid="ghost") + + assert commands(sub) == ["register", "layout_update", "env_update"] + + +# -- delete_env -------------------------------------------------------------- + + +def test_delete_env_drops_the_environment(env): + sub = open_sub(env) + + send(sub, cmd="delete_env", eid="expt") + + assert "expt" not in env.state + assert not env.storage.env_exists("expt") + + +def test_delete_env_clears_the_undo_history(env): + sub = open_sub(env) + send(sub, cmd="close", eid="expt", data="win_0") + + send(sub, cmd="delete_env", eid="expt") + + assert count_deleted(env.storage, "expt") == 0 + + +def test_delete_env_announces_the_new_environment_list(env): + sub = open_sub(env) + + send(sub, cmd="delete_env", eid="expt") + + assert last(sub, "env_update")["data"] == ["main"] + + +def test_delete_env_refuses_to_remove_main(env): + sub = open_sub(env) + + send(sub, cmd="delete_env", eid="main") + + assert "main" in env.state + + +def test_delete_env_escapes_the_environment_id(app): + app.state["a_b"] = {"jsons": {}, "reload": {}} + sub = open_sub(app) + + send(sub, cmd="delete_env", eid="a/b") + + assert "a_b" not in app.state + + +# -- save -------------------------------------------------------------------- + + +def test_save_forks_the_previous_environment(env): + sub = open_sub(env) + + send(sub, cmd="save", eid="copy", prev_eid="expt", data={"win_0": [0, 0, 3, 3]}) + + assert "win_0" in env.state["copy"]["jsons"] + assert env.state["copy"]["reload"] == {"win_0": [0, 0, 3, 3]} + + +def test_save_leaves_the_source_environment_alone(env): + sub = open_sub(env) + send(sub, cmd="save", eid="copy", prev_eid="expt", data={}) + + env.state["copy"]["jsons"]["win_1"] = pane("win_1") + + assert "win_1" not in env.state["expt"]["jsons"] + + +def test_save_persists_the_new_environment(env): + sub = open_sub(env) + + send(sub, cmd="save", eid="copy", prev_eid="expt", data={}) + + assert env.storage.env_exists("copy") + + +def test_save_repoints_the_socket_at_the_new_environment(env): + sub = open_sub(env) + + send(sub, cmd="save", eid="copy", prev_eid="expt", data={}) + + assert sub.eid == "copy" + + +def test_save_without_a_known_source_is_a_noop(env): + sub = open_sub(env) + + send(sub, cmd="save", eid="copy", prev_eid="ghost", data={}) + + assert "copy" not in env.state + + +def test_save_without_a_previous_id_is_a_noop(env): + sub = open_sub(env) + + send(sub, cmd="save", eid="copy", data={}) + + assert "copy" not in env.state + + +# -- save_all ---------------------------------------------------------------- + + +def test_save_all_persists_every_environment(env, inline_executor): + sub = open_sub(env) + + send(sub, cmd="save_all") + + assert env.storage.env_exists("expt") + assert env.storage.env_exists("main") + + +def test_save_all_is_handed_to_the_executor(env, inline_executor): + """The write is offloaded rather than blocking the socket loop.""" + sub = open_sub(env) + + send(sub, cmd="save_all") + + func, args = inline_executor[0] + assert func == env.storage.save_all + assert args == (env.state,) + + +# -- save_layouts ------------------------------------------------------------ + + +def test_save_layouts_stores_the_payload(env): + sub = open_sub(env) + + send(sub, cmd="save_layouts", data='[["view A", {}]]') + + assert env.layouts == '[["view A", {}]]' + assert env.storage.load_layouts() == '[["view A", {}]]' + + +def test_save_layouts_tells_the_subscribers(env): + sub = open_sub(env) + + send(sub, cmd="save_layouts", data="[]") + + assert last(sub, "layout_update")["data"] == "[]" + + +def test_a_source_socket_can_save_layouts(env): + """A source connection sending ``save_layouts`` used to raise ValueError. + + Only subscriber sockets carried an implementation of ``broadcast_layouts``; + the base class raised, and the exception escaped the message loop. + """ + sub = open_sub(env) + source = open_source(env) + + send(source, cmd="save_layouts", data='[["from a source", {}]]') + + assert env.layouts == '[["from a source", {}]]' + assert last(sub, "layout_update")["data"] == '[["from a source", {}]]' + + +def test_save_layouts_without_data_is_a_noop(env): + sub = open_sub(env) + before = env.layouts + + send(sub, cmd="save_layouts") + + assert env.layouts == before + + +# -- layout_item_update ------------------------------------------------------ + + +def test_layout_item_update_records_the_position(env): + sub = open_sub(env) + + send(sub, cmd="layout_item_update", eid="expt", win="win_0", data=[0, 0, 4, 4]) + + assert env.state["expt"]["reload"]["win_0"] == [0, 0, 4, 4] + + +@pytest.mark.parametrize( + "msg", + [ + {"cmd": "layout_item_update", "win": "win_0", "data": []}, + {"cmd": "layout_item_update", "eid": "expt", "data": []}, + {"cmd": "layout_item_update", "eid": "ghost", "win": "win_0", "data": []}, + ], +) +def test_layout_item_update_drops_malformed_messages(env, msg): + sub = open_sub(env) + + sub.on_message(json.dumps(msg)) + + assert env.state["expt"]["reload"] == {} + + +# -- update_plot_layout ------------------------------------------------------ + + +def test_update_plot_layout_patches_the_layout(env): + sub = open_sub(env) + + send(sub, cmd="update_plot_layout", eid="expt", win="win_0", data={"title": "new"}) + + assert env.state["expt"]["jsons"]["win_0"]["content"]["layout"]["title"] == "new" + + +def test_update_plot_layout_merges_into_an_existing_layout(env): + env.state["expt"]["jsons"]["win_0"]["content"]["layout"] = {"title": "old", "a": 1} + sub = open_sub(env) + + send(sub, cmd="update_plot_layout", eid="expt", win="win_0", data={"title": "new"}) + + layout = env.state["expt"]["jsons"]["win_0"]["content"]["layout"] + assert layout == {"title": "new", "a": 1} + + +def test_update_plot_layout_targets_one_history_frame(app): + """A ``plot_history`` pane holds a list of frames; only ``frame`` changes.""" + frames = [{"data": [], "layout": {}}, {"data": [], "layout": {}}] + app.state["expt"] = { + "jsons": {"win_0": pane(ptype="plot_history", content=frames)}, + "reload": {}, + } + sub = open_sub(app) + + send( + sub, + cmd="update_plot_layout", + eid="expt", + win="win_0", + frame=1, + data={"title": "second"}, + ) + + stored = app.state["expt"]["jsons"]["win_0"]["content"] + assert stored[1]["layout"] == {"title": "second"} + assert stored[0]["layout"] == {} + + +@pytest.mark.parametrize("frame", [None, -1, 2, "1", 1.0, True, [1]]) +def test_update_plot_layout_rejects_a_bad_history_frame(app, frame): + """The frame index must be an in-range int, and is checked as one. + + Out-of-range and missing frames were already rejected, but the comparison + ran before any type check, so a string or list frame from a client raised + ``TypeError`` out of the socket loop. ``True`` is excluded too: it would + otherwise sail through as index 1. + """ + frames = [{"data": [], "layout": {}}, {"data": [], "layout": {}}] + app.state["expt"] = { + "jsons": {"win_0": pane(ptype="plot_history", content=frames)}, + "reload": {}, + } + sub = open_sub(app) + + send( + sub, + cmd="update_plot_layout", + eid="expt", + win="win_0", + frame=frame, + data={"title": "nope"}, + ) + + assert all( + f["layout"] == {} for f in app.state["expt"]["jsons"]["win_0"]["content"] + ) + + +def test_update_plot_layout_rejects_a_non_dict_patch(env): + sub = open_sub(env) + + send(sub, cmd="update_plot_layout", eid="expt", win="win_0", data=["title"]) + + assert env.state["expt"]["jsons"]["win_0"]["content"]["layout"] == {} + + +def test_update_plot_layout_rejects_a_pane_without_plot_content(env): + env.state["expt"]["jsons"]["win_0"]["content"] = "just text" + sub = open_sub(env) + + send(sub, cmd="update_plot_layout", eid="expt", win="win_0", data={"title": "x"}) + + assert env.state["expt"]["jsons"]["win_0"]["content"] == "just text" + + +def test_update_plot_layout_drops_an_unknown_pane(env): + sub = open_sub(env) + + send(sub, cmd="update_plot_layout", eid="expt", win="ghost", data={"title": "x"}) + + assert env.state["expt"]["jsons"]["win_0"]["content"]["layout"] == {} + + +# -- update_comment ---------------------------------------------------------- + + +def test_update_comment_stores_the_text(env, inline_executor): + sub = open_sub(env) + + send(sub, cmd="update_comment", eid="expt", win="win_0", data="looks good") + + assert env.state["expt"]["jsons"]["win_0"]["comment"] == "looks good" + + +def test_update_comment_bumps_the_version(env, inline_executor): + sub = open_sub(env) + + send(sub, cmd="update_comment", eid="expt", win="win_0", data="first") + send(sub, cmd="update_comment", eid="expt", win="win_0", data="second") + + assert env.state["expt"]["jsons"]["win_0"]["version"] == 3 + + +def test_update_comment_broadcasts_a_json_patch(env, inline_executor): + sub = open_sub(env) + sub.eid = "expt" + + send(sub, cmd="update_comment", eid="expt", win="win_0", data="looks good") + + packet = last(sub, "window_update") + assert packet["win"] == "win_0" + assert packet["eid"] == "expt" + assert packet["content"] == [ + {"op": "add", "path": "/comment", "value": "looks good"}, + {"op": "replace", "path": "/version", "value": packet["version"]}, + ] + + +def test_update_comment_persists_the_environment(env, inline_executor): + sub = open_sub(env) + + send(sub, cmd="update_comment", eid="expt", win="win_0", data="looks good") + + saved = env.storage.load_env("expt") + assert saved["jsons"]["win_0"]["comment"] == "looks good" + + +@pytest.mark.parametrize("comment", [42, None, ["a"], {"text": "a"}]) +def test_update_comment_rejects_a_non_string(env, comment): + sub = open_sub(env) + + send(sub, cmd="update_comment", eid="expt", win="win_0", data=comment) + + assert "comment" not in env.state["expt"]["jsons"]["win_0"] + + +def test_update_comment_drops_an_unknown_pane(env): + sub = open_sub(env) + + send(sub, cmd="update_comment", eid="expt", win="ghost", data="hi") + + assert "comment" not in env.state["expt"]["jsons"]["win_0"] + + +def test_update_comment_drops_an_unknown_environment(env): + sub = open_sub(env) + + send(sub, cmd="update_comment", eid="ghost", win="win_0", data="hi") + + assert "comment" not in env.state["expt"]["jsons"]["win_0"] + + +# -- forward_to_vis ---------------------------------------------------------- + + +def test_forward_to_vis_reaches_the_sources(env): + source = open_source(env) + sub = open_sub(env) + + send( + sub, + cmd="forward_to_vis", + data={"eid": "expt", "target": "win_0", "event_type": "Click"}, + ) + + assert sent(source)[-1]["event_type"] == "Click" + + +def test_forward_to_vis_attaches_the_pane(env): + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="forward_to_vis", data={"eid": "expt", "target": "win_0"}) + + assert sent(source)[-1]["pane_data"]["id"] == "win_0" + + +def test_forward_to_vis_honours_an_opt_out(env): + """``pane_data: False`` means the source does not want the pane inlined.""" + source = open_source(env) + sub = open_sub(env) + + send( + sub, + cmd="forward_to_vis", + data={"eid": "expt", "target": "win_0", "pane_data": False}, + ) + + assert sent(source)[-1]["pane_data"] is False + + +@pytest.mark.parametrize("packet", ["a string", 42, ["a"], None]) +def test_forward_to_vis_rejects_a_non_dict_payload(env, packet): + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="forward_to_vis", data=packet) + + assert commands(source) == ["alive"] + + +@pytest.mark.parametrize("packet", [{"target": "win_0"}, {"eid": "expt"}, {}]) +def test_forward_to_vis_rejects_an_incomplete_packet(env, packet): + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="forward_to_vis", data=packet) + + assert commands(source) == ["alive"] + + +def test_forward_to_vis_warns_the_sender_about_a_missing_env(env): + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="forward_to_vis", data={"eid": "ghost", "target": "win_0"}) + + notification = last(sub, "notification") + assert notification["data"]["type"] == "warning" + assert "ghost" in notification["data"]["message"] + assert commands(source) == ["alive"] + + +def test_forward_to_vis_warns_the_sender_about_a_missing_pane(env): + source = open_source(env) + sub = open_sub(env) + + send(sub, cmd="forward_to_vis", data={"eid": "expt", "target": "ghost"}) + + notification = last(sub, "notification") + assert notification["data"]["type"] == "warning" + assert "ghost" in notification["data"]["message"] + assert commands(source) == ["alive"] + + +def test_forward_to_vis_notifies_only_the_sender(env): + other = open_sub(env) + sub = open_sub(env) + + send(sub, cmd="forward_to_vis", data={"eid": "ghost", "target": "win_0"}) + + assert last(sub, "notification") is not None + assert last(other, "notification") is None + + +# -- pop_embeddings_pane ----------------------------------------------------- + + +def embeddings_pane(history=None): + """An embeddings pane mid-drilldown, with previous states to pop back to.""" + return pane( + ptype="embeddings", + content={"data": [[9, 9]], "selected": 3, "has_previous": True}, + old_content=[[[1, 1]], [[5, 5]]] if history is None else history, + ) + + +def test_pop_embeddings_restores_the_previous_points(app): + app.state["expt"] = {"jsons": {"win_0": embeddings_pane()}, "reload": {}} + sub = open_sub(app) + + send(sub, cmd="pop_embeddings_pane", data={"eid": "expt", "target": "win_0"}) + + content = app.state["expt"]["jsons"]["win_0"]["content"] + assert content["data"] == [[5, 5]] + assert content["selected"] is None + assert content["has_previous"] is True + + +def test_pop_embeddings_clears_has_previous_on_the_last_step(app): + app.state["expt"] = { + "jsons": {"win_0": embeddings_pane([[[1, 1]]])}, + "reload": {}, + } + sub = open_sub(app) + + send(sub, cmd="pop_embeddings_pane", data={"eid": "expt", "target": "win_0"}) + + assert app.state["expt"]["jsons"]["win_0"]["content"]["has_previous"] is False + + +def test_pop_embeddings_issues_a_new_content_id(app): + app.state["expt"] = {"jsons": {"win_0": embeddings_pane()}, "reload": {}} + app.state["expt"]["jsons"]["win_0"]["contentID"] = "old" + sub = open_sub(app) + + send(sub, cmd="pop_embeddings_pane", data={"eid": "expt", "target": "win_0"}) + + assert app.state["expt"]["jsons"]["win_0"]["contentID"] != "old" + + +def test_pop_embeddings_broadcasts_the_pane(app): + app.state["expt"] = {"jsons": {"win_0": embeddings_pane()}, "reload": {}} + sub = open_sub(app) + sub.eid = "expt" + + send(sub, cmd="pop_embeddings_pane", data={"eid": "expt", "target": "win_0"}) + + broadcast = [m for m in sent(sub) if m.get("id") == "win_0"][-1] + assert broadcast["eid"] == "expt" + assert broadcast["content"]["data"] == [[5, 5]] + + +@pytest.mark.parametrize("drop_key", [False, True], ids=["empty", "absent"]) +def test_pop_embeddings_survives_an_exhausted_history(app, drop_key): + """Popping with nothing left used to raise out of the message loop. + + The pane is left untouched and the socket stays usable, rather than an + ``IndexError`` (empty list) or ``KeyError`` (absent key) escaping into + Tornado's WebSocket callback. + """ + p = embeddings_pane([]) + if drop_key: + del p["old_content"] + app.state["expt"] = {"jsons": {"win_0": p}, "reload": {}} + sub = open_sub(app) + + send(sub, cmd="pop_embeddings_pane", data={"eid": "expt", "target": "win_0"}) + + assert app.state["expt"]["jsons"]["win_0"]["content"]["data"] == [[9, 9]] + + send(sub, cmd="layout_item_update", eid="expt", win="win_0", data=[0, 0, 1, 1]) + assert app.state["expt"]["reload"]["win_0"] == [0, 0, 1, 1] + + +@pytest.mark.parametrize( + "packet", + [ + "a string", + None, + {"target": "win_0"}, + {"eid": "expt"}, + {"eid": "ghost", "target": "win_0"}, + {"eid": "expt", "target": "ghost"}, + ], +) +def test_pop_embeddings_drops_malformed_messages(app, packet): + app.state["expt"] = {"jsons": {"win_0": embeddings_pane()}, "reload": {}} + sub = open_sub(app) + + send(sub, cmd="pop_embeddings_pane", data=packet) + + assert app.state["expt"]["jsons"]["win_0"]["content"]["data"] == [[9, 9]] + + +# -- echo, and the source/subscriber split ----------------------------------- + + +def test_echo_returns_the_message_to_the_sources(env): + source = open_source(env) + + send(source, cmd="echo", data="ping") + + assert sent(source)[-1] == {"cmd": "echo", "data": "ping"} + + +def test_echo_reaches_every_source(env): + first = open_source(env) + second = open_source(env) + + send(first, cmd="echo", data="ping") + + assert sent(second)[-1]["data"] == "ping" + + +def test_echo_does_not_fall_through_to_the_base_commands(env): + """``echo`` returns early; it is not also treated as an unknown command.""" + source = open_source(env) + sub = open_sub(env) + + send(source, cmd="echo", data="ping") + + assert commands(sub) == ["register", "layout_update", "env_update"] + + +def test_a_source_socket_handles_the_shared_commands_too(env): + """Anything that is not ``echo`` falls through to the shared dispatch.""" + source = open_source(env) + + send(source, cmd="delete_env", eid="expt") + + assert "expt" not in env.state + + +def test_echo_from_a_subscriber_is_not_a_command(env): + """``echo`` only exists on the source socket class.""" + sub = open_sub(env) + + send(sub, cmd="echo", data="ping") + + assert commands(sub) == ["register", "layout_update", "env_update"] + + +# -- unknown commands -------------------------------------------------------- + + +@pytest.mark.parametrize("cmd", ["not_a_command", "", None]) +def test_an_unrecognised_command_is_ignored(env, cmd): + sub = open_sub(env) + + send(sub, cmd=cmd) + + assert commands(sub) == ["register", "layout_update", "env_update"] + assert env.state["expt"]["jsons"].keys() == {"win_0"} diff --git a/py/tests/integration/socket_lifecycle.py b/py/tests/integration/socket_lifecycle.py new file mode 100644 index 000000000..6e2a70ff7 --- /dev/null +++ b/py/tests/integration/socket_lifecycle.py @@ -0,0 +1,313 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Socket connection lifecycle: open, register, close. + +Subscriber sockets (``SocketHandlerOrWrapper``) and source sockets +(``VisSocketHandlerOrWrapper``) share a base ``open`` that mints a sid and adds +the socket to one of the app's two registries, then each layers its own +handshake on top. These tests drive the real classes through +``testutils.socket_double``; see that module for why a duck type is not enough. + +There is no HTTP here, but there is a real ``Application`` and real handler +dispatch, which is what ``integration`` means in this suite. +""" + +import json + +import pytest + +from visdom.server.app import Application +from visdom.server.handlers.socket_handlers import ( + SocketFailureReason, + SocketWrapper, + VisSocketWrapper, +) + +from testutils import commands, last, open_source, open_sub, sent, socket_double + +pytestmark = pytest.mark.integration + + +@pytest.fixture +def readonly_app(env_path): + """Application started the way ``visdom -readonly`` starts it.""" + return Application(port=8097, env_path=env_path, readonly=True) + + +# -- Opening a subscriber ---------------------------------------------------- + + +def test_open_assigns_a_sid(app): + """Every socket gets its own id at open time.""" + first = open_sub(app) + second = open_sub(app) + + assert first.sid + assert second.sid + assert first.sid != second.sid + + +def test_open_registers_the_subscriber(app): + """The socket lands in ``app.subs`` under its own sid.""" + sub = open_sub(app) + + assert app.subs[sub.sid] is sub + assert app.sources == {} + + +def test_open_defaults_the_environment_to_main(app): + """A fresh socket is subscribed to ``main`` until it says otherwise.""" + assert open_sub(app).eid == "main" + + +def test_register_payload_carries_the_sid(app): + """The first message back is the register handshake.""" + sub = open_sub(app) + register = sent(sub)[0] + + assert register["command"] == "register" + assert register["data"] == sub.sid + + +def test_register_payload_reports_writable_server(app): + """``readonly`` is false on a normal server.""" + assert sent(open_sub(app))[0]["readonly"] is False + + +def test_register_payload_reports_readonly_server(readonly_app): + """A ``-readonly`` server says so in the handshake.""" + assert sent(open_sub(readonly_app))[0]["readonly"] is True + + +def test_register_payload_lists_environments_sorted(app): + """``envList`` is the sorted env ids, so the client need not sort.""" + app.state["zebra"] = {"jsons": {}, "reload": {}} + app.state["alpha"] = {"jsons": {}, "reload": {}} + + assert sent(open_sub(app))[0]["envList"] == ["alpha", "main", "zebra"] + + +def test_open_follows_up_with_layouts_and_environments(app): + """Register, then the saved layouts, then the env list — in that order.""" + assert commands(open_sub(app)) == ["register", "layout_update", "env_update"] + + +def test_layout_update_carries_the_saved_layouts(app): + """The layout message ships whatever the app has loaded.""" + app.layouts = '[["view A", {"win_0": [0, 0, 3, 3]}]]' + sub = open_sub(app) + + assert last(sub, "layout_update")["data"] == app.layouts + + +def test_env_update_carries_the_environment_ids(app): + """The env message ships the live state keys.""" + app.state["expt"] = {"jsons": {}, "reload": {}} + sub = open_sub(app) + + assert sorted(last(sub, "env_update")["data"]) == ["expt", "main"] + + +def test_opening_only_notifies_the_new_socket(app): + """An existing subscriber is not re-sent the handshake of a new one.""" + first = open_sub(app) + before = len(first.messages) + + open_sub(app) + + assert len(first.messages) == before + + +# -- Opening a source -------------------------------------------------------- + + +def test_source_registers_separately_from_subscribers(app): + """Sources and subscribers live in different registries.""" + source = open_source(app) + + assert app.sources[source.sid] is source + assert app.subs == {} + + +def test_source_handshake_is_an_alive_ping(app): + """A source is told the server is up; it gets no layouts or env list.""" + source = open_source(app) + + assert sent(source) == [{"command": "alive", "data": "vis_alive"}] + + +def test_both_socket_kinds_can_be_open_at_once(app): + """The two registries do not collide.""" + sub = open_sub(app) + source = open_source(app) + + assert list(app.subs) == [sub.sid] + assert list(app.sources) == [source.sid] + + +# -- Closing ----------------------------------------------------------------- + + +def test_close_deregisters_a_subscriber(app): + """``on_close`` removes the socket from ``app.subs``.""" + sub = open_sub(app) + sub.on_close() + + assert app.subs == {} + + +def test_close_deregisters_a_source(app): + """``on_close`` removes the socket from ``app.sources``.""" + source = open_source(app) + source.on_close() + + assert app.sources == {} + + +def test_close_leaves_other_sockets_registered(app): + """Closing one subscriber does not disturb the others.""" + first = open_sub(app) + second = open_sub(app) + first.on_close() + + assert list(app.subs) == [second.sid] + + +def test_close_on_an_unopened_socket_is_a_noop(app): + """A socket that never registered can still be closed.""" + socket_double(SocketWrapper, app).on_close() + socket_double(VisSocketWrapper, app).on_close() + + assert app.subs == {} + assert app.sources == {} + + +def test_close_twice_is_a_noop(app): + """A second close does not raise on the already-removed sid.""" + sub = open_sub(app) + sub.on_close() + sub.on_close() + + assert app.subs == {} + + +def test_polling_close_routes_through_on_close(app): + """``close()`` on a polling wrapper is the deregistration path.""" + sub = open_sub(app) + sub.close() + + assert app.subs == {} + + +def test_reopening_keeps_the_socket_under_its_first_sid(app): + """A second ``open`` mints a new sid but does not re-register. + + ``open`` assigns ``self.sid`` before checking membership, so an already + registered socket keeps its registry entry under the *old* sid while + carrying the new one. ``on_close`` pops ``self.sid``, so the entry then + outlives the socket. Pinned down here as current behaviour; nothing in the + server reopens a live socket today. + """ + sub = open_sub(app) + first_sid = sub.sid + + sub.open() + + assert sub.sid != first_sid + assert list(app.subs) == [first_sid] + + sub.on_close() + assert list(app.subs) == [first_sid] + + +# -- Message queue ----------------------------------------------------------- + + +def test_get_messages_drains_the_queue(app): + """A polling client reads each message once.""" + sub = open_sub(app) + + drained = sub.get_messages() + + assert len(drained) == 3 + assert sub.get_messages() == [] + + +def test_get_messages_stamps_the_read_time(app): + """Reading refreshes the idle timer the reaper watches.""" + sub = open_sub(app) + sub.last_read_time = 0 + + sub.get_messages() + + assert sub.last_read_time > 0 + + +# -- Readonly short-circuit -------------------------------------------------- + + +def test_readonly_socket_ignores_commands(readonly_app): + """A readonly server drops every socket command before dispatch.""" + readonly_app.state["expt"] = {"jsons": {}, "reload": {}} + sub = open_sub(readonly_app) + + sub.on_message(json.dumps({"cmd": "delete_env", "eid": "expt"})) + + assert "expt" in readonly_app.state + + +def test_readonly_socket_ignores_an_unknown_command(readonly_app): + """The short-circuit happens before the command is even recognised.""" + sub = open_sub(readonly_app) + before = len(sub.messages) + + sub.on_message(json.dumps({"cmd": "not_a_command"})) + + assert len(sub.messages) == before + + +# -- SocketFailureReason ----------------------------------------------------- + + +@pytest.mark.parametrize( + "reason,value", + [ + (SocketFailureReason.CONNECTION_CLOSED, "closed"), + (SocketFailureReason.MISSING_MESSAGE, "no msg"), + (SocketFailureReason.INVALID_MESSAGE_TYPE, "invalid"), + ], +) +def test_failure_reason_wire_value(reason, value): + """The short code is what goes on the wire; the detail explains it.""" + assert reason.value == value + assert reason.detail + + +def test_failure_response_shape(): + """A failure response always reports success false with a reason.""" + resp = SocketFailureReason.CONNECTION_CLOSED.to_failure_response() + + assert resp["success"] is False + assert resp["reason"] == "closed" + assert resp["detail"] == SocketFailureReason.CONNECTION_CLOSED.detail + assert "message" not in resp + + +def test_failure_response_carries_an_optional_message(): + """Callers can attach the offending value for debugging.""" + resp = SocketFailureReason.MISSING_MESSAGE.to_failure_response("sid=None") + + assert resp["message"] == "sid=None" + + +def test_failure_reasons_have_distinct_codes(): + """No two reasons share a wire value.""" + values = [reason.value for reason in SocketFailureReason] + + assert len(values) == len(set(values)) diff --git a/py/tests/test_storage_wiring.py b/py/tests/integration/storage_wiring.py similarity index 79% rename from py/tests/test_storage_wiring.py rename to py/tests/integration/storage_wiring.py index 0daed4bf7..0c00fb8e5 100644 --- a/py/tests/test_storage_wiring.py +++ b/py/tests/integration/storage_wiring.py @@ -11,14 +11,13 @@ (the DataStore backend) rather than writing environment files directly. As end-to-end save/fork/reload behavior is already -covered in ``test_environment_lifecycle``; here we assert the abstraction itself +covered in ``environment_lifecycle``; here we assert the abstraction itself is in place so a future refactor cannot silently bypass the backend. """ import json import os import tempfile -import types import unittest from unittest import mock @@ -39,10 +38,8 @@ push_deleted, ) - -def _env(win_id="win_0"): - """Minimal environment payload (one window).""" - return {"jsons": {win_id: {"id": win_id}}, "reload": {}} +from testutils.fakes import FakeHandler, FakeSocket, SpyStore +from testutils.payloads import env_payload class TestStorageWiring(unittest.TestCase): @@ -71,31 +68,12 @@ def spy(state, eids): self.app.storage.save_envs = spy - written = [] - handler = types.SimpleNamespace( - storage=self.app.storage, state=self.app.state, write=written.append - ) + handler = FakeHandler(state=self.app.state, storage=self.app.storage) SaveHandler.wrap_func(handler, {"data": ["main"]}) self.assertEqual(len(calls), 1) self.assertIn("main", calls[0]) - self.assertIn("main", json.loads(written[0])) - - -class _SpyStore(JSONStore): - """JSONStore that records load-path calls while delegating to the real impl.""" - - def __init__(self, env_path): - super().__init__(env_path) - self.calls = {"list_envs": 0, "load_env": []} - - def list_envs(self): - self.calls["list_envs"] += 1 - return super().list_envs() - - def load_env(self, eid): - self.calls["load_env"].append(eid) - return super().load_env(eid) + self.assertIn("main", handler.json_body()) class TestLoadStateWiring(unittest.TestCase): @@ -109,26 +87,28 @@ def tearDown(self): self._tmp.cleanup() def _seed(self, eid, env=None): - JSONStore(self.env_path).save_env(eid, env if env is not None else _env()) + JSONStore(self.env_path).save_env( + eid, env if env is not None else env_payload() + ) def test_saved_envs_load_into_state(self): self._seed("main") - self._seed("expt", _env("w1")) + self._seed("expt", env_payload("w1")) app = Application(port=8097, env_path=self.env_path) self.assertIn("expt", app.state) - self.assertEqual(dict(app.state["expt"]), _env("w1")) + self.assertEqual(dict(app.state["expt"]), env_payload("w1")) def test_lazy_by_default(self): self._seed("expt") app = Application(port=8097, env_path=self.env_path) self.assertIsInstance(app.state["expt"], LazyEnvData) - self.assertEqual(app.state["expt"]["jsons"], _env()["jsons"]) + self.assertEqual(app.state["expt"]["jsons"], env_payload()["jsons"]) def test_eager_loads_plain_dicts(self): self._seed("expt") app = Application(port=8097, env_path=self.env_path, eager_data_loading=True) self.assertIsInstance(app.state["expt"], dict) - self.assertEqual(app.state["expt"], _env()) + self.assertEqual(app.state["expt"], env_payload()) def test_eager_preserves_extra_env_keys(self): """Eager loading keeps env keys the server itself does not read. @@ -137,13 +117,13 @@ def test_eager_preserves_extra_env_keys(self): dropping unknown keys here would lose it on the next full-env save. """ experiment = {"env_id": "expt", "name": "run-1", "status": "running"} - self._seed("expt", dict(_env("w1"), experiment=experiment)) + self._seed("expt", dict(env_payload("w1"), experiment=experiment)) app = Application(port=8097, env_path=self.env_path, eager_data_loading=True) self.assertEqual(app.state["expt"]["experiment"], experiment) def test_load_state_routes_through_storage(self): self._seed("expt") - with mock.patch("visdom.server.app.JSONStore", _SpyStore): + with mock.patch("visdom.server.app.JSONStore", SpyStore): app = Application(port=8097, env_path=self.env_path) self.assertEqual(app.storage.calls["list_envs"], 1) self.assertEqual(app.storage.calls["load_env"], []) @@ -184,19 +164,15 @@ def spy(eid): def _handler(self, state, **extra): """A stand-in handler exposing only what the delete paths read.""" - return types.SimpleNamespace( - storage=self.store, - state=state, - env_path=self.env_path, - subs={}, - **extra, + return FakeHandler( + state=state, storage=self.store, env_path=self.env_path, **extra ) def test_web_delete_routes_through_storage(self): - self.store.save_env("expt", _env()) + self.store.save_env("expt", env_payload()) self.assertTrue(self.store.env_exists("expt")) calls = self._spy_delete() - handler = self._handler({"expt": _env()}) + handler = self._handler({"expt": env_payload()}) DeleteEnvHandler.wrap_func(handler, {"eid": "expt"}) self.assertEqual(calls, ["expt"]) self.assertNotIn("expt", handler.state) @@ -204,17 +180,17 @@ def test_web_delete_routes_through_storage(self): self.assertFalse(os.path.exists(os.path.join(self.env_path, "expt.json"))) def test_web_delete_protects_main(self): - self.store.save_env("main", _env()) + self.store.save_env("main", env_payload()) calls = self._spy_delete() - handler = self._handler({"main": _env()}) + handler = self._handler({"main": env_payload()}) DeleteEnvHandler.wrap_func(handler, {"eid": "main"}) self.assertEqual(calls, []) self.assertTrue(self.store.env_exists("main")) def test_socket_delete_routes_through_storage(self): - self.store.save_env("expt", _env()) + self.store.save_env("expt", env_payload()) calls = self._spy_delete() - fake = self._handler({"expt": _env()}, readonly=False) + fake = self._handler({"expt": env_payload()}, readonly=False) AnySocketHandlerOrWrapper.on_message( fake, json.dumps({"cmd": "delete_env", "eid": "expt"}) ) @@ -310,7 +286,7 @@ def tearDown(self): self._tmp.cleanup() def test_defers_until_access_then_caches(self): - self._store.save_env("main", _env()) + self._store.save_env("main", env_payload()) reads = [] real_load = self._store.load_env @@ -321,7 +297,7 @@ def spy(eid): self._store.load_env = spy lazy = LazyEnvData(self._store, "main") self.assertEqual(reads, []) - self.assertEqual(lazy["jsons"], _env()["jsons"]) + self.assertEqual(lazy["jsons"], env_payload()["jsons"]) self.assertEqual(reads, ["main"]) _ = lazy["reload"] self.assertEqual(reads, ["main"]) @@ -332,55 +308,44 @@ def test_missing_env_raises_value_error(self): _ = lazy["jsons"] -class _FakeSocket: - """Minimal stand-in for a client socket used by the read helpers.""" - - def __init__(self): - self.messages = [] - self.eid = None - - def write_message(self, msg): - self.messages.append(msg) - - class TestReadHelperWiring(unittest.TestCase): """``load_env`` reads a cold env through the store, not raw env_path.""" def setUp(self): self._tmp = tempfile.TemporaryDirectory() - self.store = _SpyStore(self._tmp.name) + self.store = SpyStore(self._tmp.name) def tearDown(self): self._tmp.cleanup() def test_load_env_reads_cold_env_through_store(self): - JSONStore(self.store.env_path).save_env("expt", _env()) + JSONStore(self.store.env_path).save_env("expt", env_payload()) state = {} - socket = _FakeSocket() + socket = FakeSocket() load_env(state, "expt", socket, self.store) self.assertEqual(self.store.calls["load_env"], ["expt"]) - self.assertEqual(dict(state["expt"]), _env()) + self.assertEqual(dict(state["expt"]), env_payload()) def test_load_env_skips_store_when_already_in_state(self): - state = {"expt": _env()} - socket = _FakeSocket() + state = {"expt": env_payload()} + socket = FakeSocket() load_env(state, "expt", socket, self.store) self.assertEqual(self.store.calls["load_env"], []) def test_gather_envs_lists_through_store(self): - JSONStore(self.store.env_path).save_env("on_disk", _env()) - items = gather_envs({"in_memory": _env()}, self.store) + JSONStore(self.store.env_path).save_env("on_disk", env_payload()) + items = gather_envs({"in_memory": env_payload()}, self.store) self.assertEqual(self.store.calls["list_envs"], 1) self.assertEqual(items, ["in_memory", "on_disk"]) def test_gather_envs_in_memory_only(self): - store = _SpyStore(None) - self.assertEqual(gather_envs({"main": _env()}, store), ["main"]) + store = SpyStore(None) + self.assertEqual(gather_envs({"main": env_payload()}, store), ["main"]) def test_compare_envs_reads_cold_env_through_store(self): - JSONStore(self.store.env_path).save_env("cold", _env()) - state = {"warm": _env("w1")} - socket = _FakeSocket() + JSONStore(self.store.env_path).save_env("cold", env_payload()) + state = {"warm": env_payload("w1")} + socket = FakeSocket() compare_envs(state, ["warm", "cold"], socket, self.store) self.assertEqual(self.store.calls["load_env"], ["cold"]) self.assertIn("cold", state) diff --git a/py/tests/integration/update_plots.py b/py/tests/integration/update_plots.py new file mode 100644 index 000000000..766fe048b --- /dev/null +++ b/py/tests/integration/update_plots.py @@ -0,0 +1,433 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""``POST /update`` for plot panes -- heatmaps, traces, layout and opts. + +Once ``UpdateHandler.update`` is past the content panes it works on +``p["content"]["data"]``, and which branch runs depends on how many traces the +``name`` argument selects: none injects a trace, exactly one heatmap takes the +``updateDir`` path, and anything else falls through to the positional trace +update. Every one of those branches is driven here over HTTP. +""" + +import unittest + +import pytest + +from testutils.http import VisdomHTTPTestCase + +pytestmark = pytest.mark.integration + + +class PlotUpdateTestCase(VisdomHTTPTestCase): + """Creates the two plot shapes the update paths distinguish between.""" + + def create_heatmap(self, z=None, x=None, y=None): + return self.create_window( + [ + { + "type": "heatmap", + "z": [[1, 2], [3, 4]] if z is None else z, + "x": ["a", "b"] if x is None else x, + "y": ["c", "d"] if y is None else y, + } + ], + layout={"title": "hm"}, + ) + + def create_scatter(self, name="t1", x=None, y=None): + return self.create_window( + [ + { + "type": "scatter", + "x": [1, 2] if x is None else x, + "y": [3, 4] if y is None else y, + "name": name, + } + ], + layout={"title": "scatter"}, + ) + + def update_heatmap(self, win, z, update_dir, x=None, y=None): + """Mirror the client: ``append`` is set for every directional update. + + ``Visdom.heatmap`` derives it from ``updateDir`` (``__init__.py:3006``) + and it is load-bearing -- without it the opts loop at the end of the + heatmap branch overwrites the labels it just extended. + """ + return self.update( + win, + [{"type": "heatmap", "z": z, "x": x, "y": y}], + name=None, + updateDir=update_dir, + append=update_dir != "replace", + ) + + def traces(self, win): + return self.get_win_data(win)["content"]["data"] + + def heatmap_z(self, win): + return self.traces(win)[0]["z"] + + +class TestHeatmapRowUpdates(PlotUpdateTestCase): + """``appendRow`` and ``prependRow`` grow ``z`` along the y axis.""" + + def test_append_row_adds_to_the_bottom(self): + win = self.create_heatmap() + self.update_heatmap(win, [[5, 6]], "appendRow", y=["e"]) + z = self.heatmap_z(win) + self.assertEqual(len(z), 3) + self.assertEqual(z[2], [5, 6]) + + def test_append_row_extends_the_row_labels(self): + win = self.create_heatmap() + self.update_heatmap(win, [[5, 6]], "appendRow", y=["e"]) + self.assertEqual(self.traces(win)[0]["y"], ["c", "d", "e"]) + + def test_prepend_row_adds_to_the_top(self): + win = self.create_heatmap() + self.update_heatmap(win, [[5, 6]], "prependRow", y=["e"]) + z = self.heatmap_z(win) + self.assertEqual(len(z), 3) + self.assertEqual(z[0], [5, 6]) + + def test_prepend_row_extends_the_row_labels(self): + win = self.create_heatmap() + self.update_heatmap(win, [[5, 6]], "prependRow", y=["e"]) + self.assertEqual(self.traces(win)[0]["y"], ["e", "c", "d"]) + + +class TestHeatmapColumnUpdates(PlotUpdateTestCase): + """``appendColumn`` and ``prependColumn`` grow every row of ``z``.""" + + def test_append_column_adds_to_the_right(self): + win = self.create_heatmap() + self.update_heatmap(win, [[7], [8]], "appendColumn", x=["e"]) + z = self.heatmap_z(win) + self.assertEqual(len(z[0]), 3) + self.assertEqual([row[2] for row in z], [7, 8]) + + def test_prepend_column_adds_to_the_left(self): + win = self.create_heatmap() + self.update_heatmap(win, [[7], [8]], "prependColumn", x=["e"]) + z = self.heatmap_z(win) + self.assertEqual(len(z[0]), 3) + self.assertEqual([row[0] for row in z], [7, 8]) + + def test_column_update_extends_the_column_labels(self): + win = self.create_heatmap() + self.update_heatmap(win, [[7], [8]], "appendColumn", x=["e"]) + self.assertEqual(self.traces(win)[0]["x"], ["a", "b", "e"]) + + +class TestHeatmapReplace(PlotUpdateTestCase): + """``replace`` swaps the matrix outright, dimensions and all.""" + + def test_replace_swaps_the_matrix(self): + win = self.create_heatmap() + self.update_heatmap(win, [[9]], "replace", x=["only"], y=["one"]) + self.assertEqual(self.heatmap_z(win), [[9]]) + + def test_replace_swaps_the_labels(self): + win = self.create_heatmap() + self.update_heatmap(win, [[9]], "replace", x=["only"], y=["one"]) + trace = self.traces(win)[0] + self.assertEqual(trace["x"], ["only"]) + self.assertEqual(trace["y"], ["one"]) + + +class TestHeatmapMismatch(PlotUpdateTestCase): + """A shape or label conflict logs and leaves the plot untouched.""" + + def test_wrong_column_count_is_a_no_op(self): + win = self.create_heatmap() + resp = self.update_heatmap(win, [[5, 6, 7]], "appendRow", y=["e"]) + self.assertEqual(resp.code, 200) + self.assertEqual(self.heatmap_z(win), [[1, 2], [3, 4]]) + + def test_wrong_row_count_is_a_no_op(self): + win = self.create_heatmap() + resp = self.update_heatmap(win, [[7]], "appendColumn", x=["e"]) + self.assertEqual(resp.code, 200) + self.assertEqual(self.heatmap_z(win), [[1, 2], [3, 4]]) + + def test_duplicate_labels_are_a_no_op(self): + win = self.create_heatmap() + self.update_heatmap(win, [[5, 6]], "appendRow", y=["c"]) + self.assertEqual(self.heatmap_z(win), [[1, 2], [3, 4]]) + + def test_missing_labels_are_a_no_op(self): + win = self.create_heatmap() + self.update_heatmap(win, [[5, 6]], "appendRow", y=None) + self.assertEqual(self.heatmap_z(win), [[1, 2], [3, 4]]) + + +class TestTraceUpdates(PlotUpdateTestCase): + """A ``name`` selects which traces the update applies to.""" + + def create_two_traces(self): + return self.create_window( + [ + {"type": "scatter", "x": [1], "y": [2], "name": "t1"}, + {"type": "scatter", "x": [3], "y": [4], "name": "t2"}, + ], + layout={"title": "multi"}, + ) + + def test_named_update_replaces_the_trace(self): + win = self.create_scatter() + self.update( + win, + [{"type": "scatter", "x": [10, 20], "y": [30, 40], "name": "t1"}], + name="t1", + append=False, + ) + self.assertEqual(self.traces(win)[0]["x"], [10, 20]) + + def test_named_update_appends_to_the_trace(self): + win = self.create_scatter() + self.update( + win, + [{"type": "scatter", "x": [5], "y": [6], "name": "t1"}], + name="t1", + append=True, + ) + trace = self.traces(win)[0] + self.assertEqual(trace["x"], [1, 2, 5]) + self.assertEqual(trace["y"], [3, 4, 6]) + + def test_delete_removes_only_the_named_trace(self): + win = self.create_two_traces() + self.update(win, [{}], name="t1", delete=True) + remaining = self.traces(win) + self.assertEqual(len(remaining), 1) + self.assertEqual(remaining[0]["name"], "t2") + + def test_an_unknown_name_injects_a_new_trace(self): + win = self.create_scatter() + self.update( + win, + [{"type": "scatter", "x": [10], "y": [20], "name": "new_trace"}], + name="new_trace", + ) + traces = self.traces(win) + self.assertEqual(len(traces), 2) + self.assertEqual(traces[1]["name"], "new_trace") + self.assertEqual(traces[1]["x"], [10]) + + def test_a_named_update_carrying_several_entries_is_rejected(self): + win = self.create_scatter() + resp = self.update( + win, + [ + {"type": "scatter", "x": [5], "y": [6], "name": "t1"}, + {"type": "scatter", "x": [7], "y": [8], "name": "t1"}, + ], + name="t1", + ) + self.assertEqual(resp.code, 400) + + +class TestMarkerUpdates(PlotUpdateTestCase): + """Marker colours concatenate alongside the points they belong to.""" + + def create_coloured_scatter(self, colors): + return self.create_window( + [ + { + "type": "scatter", + "x": [1], + "y": [2], + "name": "t1", + "marker": {"color": colors}, + } + ], + layout={"title": "markers"}, + ) + + def append_colour(self, win, color, append=True): + return self.update( + win, + [ + { + "type": "scatter", + "x": [3], + "y": [4], + "name": "t1", + "marker": {"color": [color]}, + } + ], + name="t1", + append=append, + ) + + def test_append_concatenates_the_colours(self): + win = self.create_coloured_scatter(["red"]) + self.append_colour(win, "blue") + self.assertEqual(self.traces(win)[0]["marker"]["color"], ["red", "blue"]) + + def test_replace_swaps_the_colours(self): + win = self.create_coloured_scatter(["red"]) + self.append_colour(win, "blue", append=False) + self.assertEqual(self.traces(win)[0]["marker"]["color"], ["blue"]) + + +class TestLayoutAndOptsUpdates(PlotUpdateTestCase): + """An update with no ``data`` still applies the layout and opts.""" + + def test_layout_only_update_changes_the_title(self): + win = self.create_scatter() + self.update(win, None, layout={"title": "new title"}) + pane = self.get_win_data(win) + self.assertEqual(pane["content"]["layout"]["title"], "new title") + + def test_layout_only_update_bumps_the_version(self): + win = self.create_scatter() + self.assertEqual(self.get_win_data(win)["version"], 1) + self.update(win, None, layout={"title": "new title"}) + self.assertEqual(self.get_win_data(win)["version"], 2) + + def test_opts_legend_renames_the_trace(self): + win = self.create_scatter() + self.update(win, None, opts={"legend": ["renamed_trace"]}) + self.assertEqual(self.traces(win)[0]["name"], "renamed_trace") + + +class TestCategoricalXUpdate(PlotUpdateTestCase): + """A categorical x axis must survive the all-missing-points check. + + That check exists to skip a trace whose points are all None/NaN/Inf. Only + numbers can be either, so handing a label to ``math.isnan`` used to raise + ``TypeError`` and answer 500. + """ + + def create_categorical(self): + return self.create_scatter(x=["a", "b"], y=[1, 2]) + + def test_appending_to_a_categorical_axis_succeeds(self): + win = self.create_categorical() + resp = self.update( + win, + [{"type": "scatter", "x": ["c"], "y": [3], "name": "t1"}], + name="t1", + append=True, + ) + self.assertEqual(resp.code, 200, resp.body) + + def test_appending_to_a_categorical_axis_keeps_the_labels(self): + win = self.create_categorical() + self.update( + win, + [{"type": "scatter", "x": ["c"], "y": [3], "name": "t1"}], + name="t1", + append=True, + ) + trace = self.traces(win)[0] + self.assertEqual(trace["x"], ["a", "b", "c"]) + self.assertEqual(trace["y"], [1, 2, 3]) + + def test_a_wholly_missing_numeric_update_is_still_skipped(self): + win = self.create_scatter() + self.update( + win, + [{"type": "scatter", "x": [None, None], "y": [9, 9], "name": "t1"}], + name="t1", + append=False, + ) + self.assertEqual(self.traces(win)[0]["x"], [1, 2]) + + +class TestEmptyDataUpdate(PlotUpdateTestCase): + """Updates carrying fewer data entries than the plot has traces. + + Each of these indexed into ``data`` without checking its length first and + answered 500 with an ``IndexError``. + """ + + def test_an_empty_data_list_applies_the_layout(self): + win = self.create_scatter() + resp = self.update(win, [], layout={"title": "new title"}) + self.assertEqual(resp.code, 200, resp.body) + pane = self.get_win_data(win) + self.assertEqual(pane["content"]["layout"]["title"], "new title") + + def test_an_empty_data_list_leaves_the_traces_alone(self): + win = self.create_scatter() + self.update(win, [], layout={"title": "new title"}) + self.assertEqual(self.traces(win)[0]["x"], [1, 2]) + + def test_a_trace_can_be_injected_into_an_emptied_plot(self): + win = self.create_scatter() + self.update(win, [{}], name="t1", delete=True) + self.assertEqual(self.traces(win), []) + + resp = self.update( + win, + [{"type": "scatter", "x": [7], "y": [8], "name": "t2"}], + name="t2", + ) + self.assertEqual(resp.code, 200, resp.body) + traces = self.traces(win) + self.assertEqual(len(traces), 1) + self.assertEqual(traces[0]["name"], "t2") + self.assertEqual(traces[0]["x"], [7]) + + def test_an_unnamed_delete_still_empties_the_plot(self): + """``Visdom.heatmap(update="remove")`` names no trace. + + It posts ``data: []`` with ``delete`` set (``__init__.py:3008``), which + the empty-data shortcut above must not mistake for an opts-only update. + """ + win = self.create_heatmap() + resp = self.update(win, [], name=None, delete=True) + self.assertEqual(resp.code, 200, resp.body) + self.assertEqual(self.traces(win), []) + + def test_an_unnamed_update_may_cover_only_the_first_traces(self): + win = self.create_window( + [ + {"type": "scatter", "x": [1], "y": [2], "name": "t1"}, + {"type": "scatter", "x": [3], "y": [4], "name": "t2"}, + ], + layout={"title": "multi"}, + ) + resp = self.update( + win, + [{"type": "scatter", "x": [9], "y": [9]}], + append=True, + ) + self.assertEqual(resp.code, 200, resp.body) + traces = self.traces(win) + self.assertEqual(traces[0]["x"], [1, 9]) + self.assertEqual(traces[1]["x"], [3]) + + +class TestUnsupportedUpdate(PlotUpdateTestCase): + """Only a handful of pane types accept ``/update`` at all.""" + + def test_a_bar_pane_reports_the_type_it_is(self): + win = self.create_window( + [{"type": "bar", "x": ["a"], "y": [1], "name": "b1"}], + layout={"title": "bar"}, + ) + resp = self.update( + win, + [{"type": "bar", "x": ["b"], "y": [2], "name": "b1"}], + name="b1", + ) + self.assertIn(b"win is not scatter", resp.body) + self.assertIn(b"was bar", resp.body) + + def test_an_update_to_a_missing_window_is_reported(self): + resp = self.update("no_such_win", [{"type": "scatter", "x": [1], "y": [2]}]) + self.assertEqual(resp.body, b"win does not exist") + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/integration/update_text_media.py b/py/tests/integration/update_text_media.py new file mode 100644 index 000000000..d63f70c99 --- /dev/null +++ b/py/tests/integration/update_text_media.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""``POST /update`` for the panes that carry content rather than traces. + +``UpdateHandler.update`` dispatches on ``p["type"]`` before it ever looks at +plot data, and the first three branches -- ``text``, ``image_history`` and the +separate ``update_embeddings_packet`` route -- each mutate the pane in a +different shape. Text concatenates, image history appends and moves a cursor, +and embeddings swap ``content["data"]`` while stacking the old value onto +``old_content``. All three are asserted here through a real HTTP round trip. +""" + +import unittest + +import pytest + +from testutils.http import VisdomHTTPTestCase +from testutils.payloads import content_args + +pytestmark = pytest.mark.integration + + +class TestTextUpdate(VisdomHTTPTestCase): + """Text updates concatenate onto the existing content with ``
``.""" + + def test_update_appends_after_a_line_break(self): + win = self.create_text_window(content="hello") + self.update(win, [{"type": "text", "content": "world"}]) + self.assertEqual(self.get_win_data(win)["content"], "hello
world") + + def test_repeated_updates_accumulate(self): + win = self.create_text_window(content="a") + self.update(win, [{"content": "b"}]) + self.update(win, [{"content": "c"}]) + self.assertEqual(self.get_win_data(win)["content"], "a
b
c") + + def test_update_leaves_the_pane_type_alone(self): + win = self.create_text_window(content="a") + self.update(win, [{"content": "b"}]) + self.assertEqual(self.get_win_data(win)["type"], "text") + + +class EmbeddingsTestCase(VisdomHTTPTestCase): + """Creates a two-point embeddings pane for the selection tests below.""" + + def create_embeddings(self, data=None, labels=None): + content = { + "data": [[1, 2], [3, 4]] if data is None else data, + "labels": ["a", "b"] if labels is None else labels, + "selected": None, + } + args = content_args("embeddings", content) + return self.create_window(args["data"], layout=args["layout"]) + + def select_entity(self, win, index): + return self.update(win, {"update_type": "EntitySelected", "selected": index}) + + def select_region(self, win, points): + return self.update(win, {"update_type": "RegionSelected", "points": points}) + + +class TestEmbeddingsUpdate(EmbeddingsTestCase): + """``update_embeddings_packet`` handles both selection kinds in place.""" + + def test_entity_selection_records_the_index(self): + win = self.create_embeddings() + self.select_entity(win, 1) + self.assertEqual(self.get_win_data(win)["content"]["selected"], 1) + + def test_entity_selection_leaves_the_points_alone(self): + win = self.create_embeddings() + self.select_entity(win, 1) + pane = self.get_win_data(win) + self.assertEqual(pane["content"]["data"], [[1, 2], [3, 4]]) + self.assertEqual(pane["old_content"], []) + + def test_region_selection_replaces_the_points(self): + win = self.create_embeddings() + self.select_region(win, [[5, 6]]) + pane = self.get_win_data(win) + self.assertEqual(pane["content"]["data"], [[5, 6]]) + self.assertTrue(pane["content"]["has_previous"]) + + def test_region_selection_stacks_the_previous_points(self): + win = self.create_embeddings() + self.select_region(win, [[5, 6]]) + pane = self.get_win_data(win) + self.assertEqual(pane["old_content"], [[[1, 2], [3, 4]]]) + + def test_region_selection_clears_the_selected_entity(self): + win = self.create_embeddings() + self.select_entity(win, 1) + self.select_region(win, [[5, 6]]) + self.assertIsNone(self.get_win_data(win)["content"]["selected"]) + + def test_entity_selection_survives_a_preceding_region_selection(self): + win = self.create_embeddings() + self.select_region(win, [[5, 6]]) + self.select_entity(win, 0) + pane = self.get_win_data(win) + self.assertEqual(pane["content"]["selected"], 0) + self.assertTrue(pane["content"]["has_previous"]) + + def test_unknown_update_type_is_a_no_op(self): + win = self.create_embeddings() + resp = self.update(win, {"update_type": "NothingLikeThis"}) + self.assertEqual(resp.code, 200) + self.assertEqual(self.get_win_data(win)["content"]["data"], [[1, 2], [3, 4]]) + + +class ImageHistoryTestCase(VisdomHTTPTestCase): + """Creates a single-frame image history pane and appends frames to it.""" + + def create_image_history(self, caption="c0"): + args = content_args( + "image_history", + {"src": "data:image/png;base64,{}".format(caption), "caption": caption}, + ) + return self.create_window(args["data"], layout=args["layout"]) + + def append_image(self, win, caption): + return self.update( + win, + [ + { + "type": "image_history", + "content": { + "src": "data:image/png;base64,{}".format(caption), + "caption": caption, + }, + } + ], + ) + + def select_image(self, win, index): + return self.update(win, [{"type": "image_update_selected", "selected": index}]) + + +class TestImageHistoryUpdate(ImageHistoryTestCase): + """Appends grow ``content`` and drag ``selected`` along with them.""" + + def test_append_grows_the_history(self): + win = self.create_image_history() + self.append_image(win, "c1") + pane = self.get_win_data(win) + self.assertEqual(len(pane["content"]), 2) + self.assertEqual(pane["content"][1]["caption"], "c1") + + def test_append_selects_the_newest_frame(self): + win = self.create_image_history() + self.append_image(win, "c1") + self.append_image(win, "c2") + pane = self.get_win_data(win) + self.assertEqual(len(pane["content"]), 3) + self.assertEqual(pane["selected"], 2) + + +class TestImageHistorySelection(ImageHistoryTestCase): + """``image_update_selected`` clamps the requested index into range. + + The original of this file asserted only that ``selected`` was 0 or 1 after + an append, which no implementation could fail. The real contract is the + clamp in ``UpdateHandler.update``. + """ + + def test_selection_moves_the_cursor(self): + win = self.create_image_history() + self.append_image(win, "c1") + self.select_image(win, 0) + self.assertEqual(self.get_win_data(win)["selected"], 0) + + def test_selection_past_the_end_clamps_to_the_last_frame(self): + win = self.create_image_history() + self.append_image(win, "c1") + self.select_image(win, 99) + self.assertEqual(self.get_win_data(win)["selected"], 1) + + def test_negative_selection_clamps_to_the_first_frame(self): + win = self.create_image_history() + self.append_image(win, "c1") + self.select_image(win, -5) + self.assertEqual(self.get_win_data(win)["selected"], 0) + + def test_selection_does_not_change_the_history(self): + win = self.create_image_history() + self.append_image(win, "c1") + self.select_image(win, 0) + self.assertEqual(len(self.get_win_data(win)["content"]), 2) + + def test_selection_on_a_text_pane_is_rejected(self): + win = self.create_text_window(content="not an image") + resp = self.select_image(win, 0) + self.assertEqual(resp.code, 400) + self.assertIn(b"win is not image_history", resp.body) + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/integration/window_lifecycle.py b/py/tests/integration/window_lifecycle.py new file mode 100644 index 000000000..eb14b996e --- /dev/null +++ b/py/tests/integration/window_lifecycle.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Window CRUD over real HTTP: create, exists, read, write, close. + +Covers the routes a client touches for the whole life of a pane -- +``/events``, ``/win_exists``, ``/win_data``, ``/update`` and ``/close`` -- +against a real ``Application``. Pane construction itself is unit-tested in +``unit/window_builder.py``; here we assert what survives the round trip +through the server's state. +""" + +import json +import unittest + +import pytest + +from testutils.http import VisdomHTTPTestCase + +pytestmark = pytest.mark.integration + + +class TestWindowCreate(VisdomHTTPTestCase): + def test_create_returns_nonempty_id(self): + self.assertTrue(len(self.create_text_window()) > 0) + + def test_auto_generated_id_is_prefixed(self): + self.assertTrue(self.create_text_window().startswith("window_")) + + def test_supplied_id_is_used_verbatim(self): + self.assertEqual(self.create_text_window(win="my_id"), "my_id") + + +class TestWindowExists(VisdomHTTPTestCase): + def test_window_exists_after_creation(self): + self.assertTrue(self.win_exists(self.create_text_window())) + + def test_window_does_not_exist_when_never_created(self): + self.assertFalse(self.win_exists("no_such_win")) + + def test_window_does_not_exist_in_another_env(self): + win = self.create_text_window(eid="env_a") + self.assertFalse(self.win_exists(win, eid="main")) + + +class TestWindowRead(VisdomHTTPTestCase): + def test_read_single_window(self): + win = self.create_text_window(content="get me") + self.assertEqual(self.get_win_data(win)["content"], "get me") + + def test_read_every_window_at_once(self): + first = self.create_text_window(content="first") + second = self.create_text_window(content="second") + self.assertEqual(set(self.get_win_data()), {first, second}) + + +class TestWindowWrite(VisdomHTTPTestCase): + def test_window_data_can_be_replaced(self): + win = self.create_text_window(content="original") + replacement = { + "type": "text", + "content": "replaced", + "id": win, + "command": "window", + } + resp = self.post_json( + "/win_data", {"eid": "main", "win": win, "data": json.dumps(replacement)} + ) + self.assertEqual(resp.code, 200) + self.assertEqual(self.get_win_data(win)["content"], "replaced") + + +class TestWindowClose(VisdomHTTPTestCase): + def test_close_removes_only_the_named_window(self): + keep = self.create_text_window(content="keep") + drop = self.create_text_window(content="drop") + self.assertEqual(self.close_window(drop).code, 200) + self.assertTrue(self.win_exists(keep)) + self.assertFalse(self.win_exists(drop)) + + def test_close_with_no_window_clears_the_env(self): + self.create_text_window(content="a") + self.create_text_window(content="b") + self.close_window(None) + self.assertEqual(self.get_win_data(), {}) + + +class TestUpdateMissingWindow(VisdomHTTPTestCase): + def test_update_missing_window_is_reported_not_created(self): + resp = self.update("no_such_win", [{"type": "text", "content": "nope"}]) + self.assertEqual(resp.code, 200) + self.assertEqual(resp.body.decode(), "win does not exist") + self.assertFalse(self.win_exists("no_such_win")) + + def test_update_missing_window_with_append_creates_it(self): + resp = self.update( + "auto_created", [{"type": "text", "content": "made by append"}], append=True + ) + self.assertEqual(resp.code, 200) + self.assertTrue(self.win_exists("auto_created")) + + +class TestWindowOrdering(VisdomHTTPTestCase): + def test_windows_are_indexed_in_creation_order(self): + wins = [self.create_text_window(content=str(n)) for n in range(3)] + panes = self.panes() + self.assertEqual([panes[win]["i"] for win in wins], [0, 1, 2]) + + def test_recreating_a_window_keeps_its_index(self): + """Posting the same win id twice replaces the pane instead of adding one.""" + created_id = self.create_text_window(win="stable", content="v1") + recreated_id = self.create_text_window(win="stable", content="v2") + + # both calls hand back the id they asked for, so only one pane exists + self.assertEqual(created_id, recreated_id) + self.assertEqual(len(self.panes()), 1) + + pane = self.get_win_data("stable") + self.assertEqual(pane["i"], 0) + self.assertEqual(pane["content"], "v2") + + def test_an_index_is_never_reused_after_a_close(self): + wins = [self.create_text_window(content=str(n)) for n in range(3)] + self.close_window(wins[1]) + + added = self.create_text_window(content="after the close") + + panes = self.panes() + self.assertEqual(sorted(pane["i"] for pane in panes.values()), [0, 2, 3]) + self.assertEqual(panes[added]["i"], 3) + + def test_indices_stay_unique_while_windows_churn(self): + live = [self.create_text_window(content=str(n)) for n in range(4)] + + for round_ in range(4): + self.close_window(live.pop(0)) + live.append(self.create_text_window(content="round {}".format(round_))) + + indices = [pane["i"] for pane in self.panes().values()] + self.assertEqual(len(set(indices)), len(indices), indices) + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/integration/window_types.py b/py/tests/integration/window_types.py new file mode 100644 index 000000000..11da0ce32 --- /dev/null +++ b/py/tests/integration/window_types.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Every pane type created through ``POST /events``. + +``window()`` dispatches on ``data[0]["type"]``: anything it does not recognise +as a visdom-native pane becomes a generic ``plot`` carrying the traces +untouched. Both halves of that split are asserted here -- the native panes for +the extra keys the server synthesises (``selected``, ``old_content``, +``showEdgeLabels``, ...), and the plot panes for the traces surviving the round +trip intact. +""" + +import unittest + +import pytest + +from testutils.http import VisdomHTTPTestCase +from testutils.payloads import content_args, window_args + +pytestmark = pytest.mark.integration + + +class WindowTypeTestCase(VisdomHTTPTestCase): + """Adds a create-and-read-back helper shared by every case below.""" + + def create(self, args, eid="main"): + """POST an args dict built by ``testutils.payloads``, return the pane.""" + resp = self.post_json("/events", dict(args, eid=eid)) + self.assertEqual(resp.code, 200, resp.body) + win_id = resp.body.decode() + pane = self.get_win_data(win_id, eid=eid) + self.assertEqual(pane["id"], win_id) + return pane + + def assert_plot_round_trip(self, trace, title): + pane = self.create(window_args(data=[trace], layout={"title": title})) + self.assertEqual(pane["type"], "plot") + self.assertEqual(pane["content"]["data"], [trace]) + self.assertEqual(pane["content"]["layout"]["title"], title) + return pane + + +class TestPlotPanes(WindowTypeTestCase): + def test_scatter_survives_the_round_trip(self): + self.assert_plot_round_trip( + {"type": "scatter", "x": [1, 2, 3], "y": [4, 5, 6], "name": "t1"}, "scatter" + ) + + def test_scatter3d_survives_the_round_trip(self): + self.assert_plot_round_trip( + {"type": "scatter3d", "x": [1], "y": [2], "z": [3], "name": "3d"}, + "scatter3d", + ) + + def test_heatmap_survives_the_round_trip(self): + self.assert_plot_round_trip( + { + "type": "heatmap", + "z": [[1, 2], [3, 4]], + "x": ["a", "b"], + "y": ["c", "d"], + }, + "heatmap", + ) + + def test_bar_survives_the_round_trip(self): + self.assert_plot_round_trip( + {"type": "bar", "x": ["a", "b"], "y": [10, 20], "name": "bars"}, "bar" + ) + + def test_parcoords_dimensions_are_preserved(self): + trace = { + "type": "parcoords", + "dimensions": [ + {"label": "Learning Rate", "values": [0.01, 0.05, 0.1]}, + {"label": "Batch Size", "values": [16, 32, 64]}, + {"label": "Accuracy", "values": [85.0, 90.5, 78.2]}, + ], + "line": { + "color": [85.0, 90.5, 78.2], + "colorscale": "Viridis", + "showscale": True, + }, + } + pane = self.assert_plot_round_trip(trace, "parallel coords") + dimensions = pane["content"]["data"][0]["dimensions"] + self.assertEqual( + [d["label"] for d in dimensions], + ["Learning Rate", "Batch Size", "Accuracy"], + ) + + +class TestTextPane(WindowTypeTestCase): + def _assert_text_round_trip(self, content): + pane = self.create(content_args("text", content)) + self.assertEqual(pane["type"], "text") + self.assertEqual(pane["content"], content) + self.assertEqual(pane["command"], "window") + + def test_plain_text_is_stored_verbatim(self): + self._assert_text_round_trip("hello") + + def test_html_in_text_is_stored_verbatim(self): + self._assert_text_round_trip("bold & italic") + + +class TestMediaPanes(WindowTypeTestCase): + def test_image_pane_stores_the_data_uri(self): + pane = self.create(content_args("image", "data:image/png;base64,AAAA")) + self.assertEqual(pane["type"], "image") + self.assertEqual(pane["content"], "data:image/png;base64,AAAA") + + def test_image_history_pane_starts_a_one_entry_history(self): + frame = {"src": "data:image/png;base64,BBB", "caption": "img1"} + pane = self.create(content_args("image_history", frame)) + self.assertEqual(pane["type"], "image_history") + self.assertEqual(pane["content"], [frame]) + self.assertEqual(pane["selected"], 0) + self.assertTrue(pane["show_slider"]) + + +class TestEmbeddingsPane(WindowTypeTestCase): + def test_embeddings_pane_starts_with_no_previous_state(self): + content = { + "data": [[1, 2], [3, 4], [5, 6]], + "labels": ["a", "b", "c"], + "selected": None, + } + pane = self.create(content_args("embeddings", content)) + self.assertEqual(pane["type"], "embeddings") + self.assertEqual(pane["content"]["data"], content["data"]) + self.assertEqual(pane["content"]["labels"], content["labels"]) + self.assertFalse(pane["content"]["has_previous"]) + self.assertEqual(pane["old_content"], []) + + +class TestNetworkPane(WindowTypeTestCase): + def test_network_pane_takes_its_flags_from_opts(self): + content = { + "nodes": [{"id": 1, "label": "A"}, {"id": 2, "label": "B"}], + "links": [{"source": 1, "target": 2}], + } + pane = self.create(content_args("network", content, opts={"directed": True})) + self.assertEqual(pane["type"], "network") + self.assertTrue(pane["directed"]) + self.assertEqual(pane["showEdgeLabels"], "hover") + self.assertEqual(pane["showVertexLabels"], "hover") + self.assertEqual(pane["content"], content) + + +class TestPropertiesPane(WindowTypeTestCase): + def test_properties_pane_keeps_the_row_order(self): + rows = [ + {"type": "text", "name": "prop1", "value": "val1"}, + {"type": "number", "name": "prop2", "value": 42}, + {"type": "button", "name": "prop3", "value": "click"}, + ] + pane = self.create(content_args("properties", rows)) + self.assertEqual(pane["type"], "properties") + self.assertEqual(pane["content"], rows) + + +class TestOptsAndPlacement(WindowTypeTestCase): + def _assert_opt_flattened(self, key, value): + pane = self.create(content_args("text", "opts", opts={key: value})) + self.assertEqual(pane[key], value) + + def test_title_opt_is_flattened_onto_the_pane(self): + self._assert_opt_flattened("title", "My Title") + + def test_width_opt_is_flattened_onto_the_pane(self): + self._assert_opt_flattened("width", 400) + + def test_height_opt_is_flattened_onto_the_pane(self): + self._assert_opt_flattened("height", 300) + + def test_pane_is_created_in_the_named_env(self): + pane = self.create(content_args("text", "new env"), eid="brand_new_env") + self.assertEqual(pane["content"], "new env") + self.assertIn("brand_new_env", self.get_envs()) + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/testutils/__init__.py b/py/tests/testutils/__init__.py new file mode 100644 index 000000000..b68a3c39c --- /dev/null +++ b/py/tests/testutils/__init__.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Shared helpers for the visdom test suite. + +Importable as ``testutils`` because ``py/tests`` is on ``pythonpath`` (see +``pyproject.toml``). Note that ``py/tests`` deliberately has no ``__init__.py``: +``setup.py`` runs ``find_packages(where="py")``, and a package there would be +shipped to users as a top-level ``tests`` distribution. +""" + +from testutils.fakes import FakeHandler, FakeSocket, SpyStore +from testutils.http import VisdomHTTPTestCase +from testutils.payloads import content_args, env_payload, plot_data, window_args +from testutils.sockets import ( + commands, + last, + open_source, + open_sub, + sent, + socket_double, +) + +__all__ = [ + "FakeHandler", + "FakeSocket", + "SpyStore", + "VisdomHTTPTestCase", + "commands", + "content_args", + "env_payload", + "last", + "open_source", + "open_sub", + "plot_data", + "sent", + "socket_double", + "window_args", +] diff --git a/py/tests/testutils/fakes.py b/py/tests/testutils/fakes.py new file mode 100644 index 000000000..b835eafd7 --- /dev/null +++ b/py/tests/testutils/fakes.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Test doubles for handler, socket and storage collaborators. + +The server's handler entry points (``wrap_func``, ``on_message``) read a fixed +set of attributes off the handler rather than off ``self.app`` -- see the +``initialize()`` contract in ``visdom/server/handlers/base_handlers.py``. That +makes them drivable without a Tornado request, which is what ``FakeHandler`` +exploits. +""" + +import json + +from visdom.data_model.json_store import JSONStore + + +class FakeSocket: + """Stand-in for a subscriber or source socket. + + Records every message written to it. ``messages`` holds the raw arguments + passed to ``write_message``; ``sent`` decodes the JSON ones for assertions. + """ + + def __init__(self, sid="sid_0", eid="main"): + self.sid = sid + self.eid = eid + self.messages = [] + self.closed = False + + def write_message(self, msg): + self.messages.append(msg) + + def close(self): + self.closed = True + + @property + def sent(self): + decoded = [] + for msg in self.messages: + decoded.append(json.loads(msg) if isinstance(msg, str) else msg) + return decoded + + def commands(self): + """Every ``command`` value seen, in order.""" + return [m.get("command") for m in self.sent if isinstance(m, dict)] + + def last(self, command=None): + """Most recent decoded message, optionally filtered by ``command``.""" + for msg in reversed(self.sent): + if command is None or ( + isinstance(msg, dict) and msg.get("command") == command + ): + return msg + return None + + +class FakeHandler: + """Duck-typed handler carrying the attributes the server functions read. + + Mirrors ``_WEB_APP_ATTRIBUTES`` and ``_SOCKET_APP_ATTRIBUTES`` so the same + object can drive both ``web_handlers`` wrap functions and socket + ``on_message`` dispatch. ``write`` captures HTTP response bodies and + ``set_status`` captures the status code. + """ + + def __init__( + self, + state=None, + storage=None, + subs=None, + sources=None, + readonly=False, + login_enabled=False, + env_path=None, + port=8097, + max_text_lines=500, + max_old_content=50, + max_image_history=4, + max_plot_history=4, + ): + self.state = {} if state is None else state + self.storage = JSONStore(env_path) if storage is None else storage + self.subs = {} if subs is None else subs + self.sources = {} if sources is None else sources + self.readonly = readonly + self.login_enabled = login_enabled + self.env_path = env_path + self.port = port + self.max_text_lines = max_text_lines + self.max_old_content = max_old_content + self.max_image_history = max_image_history + self.max_plot_history = max_plot_history + + self.written = [] + self.status = None + self.eid = "main" + self.sid = "sid_handler" + self.broadcasted = [] + + def write(self, chunk): + self.written.append(chunk) + + def set_status(self, code, reason=None): + self.status = code + + def write_message(self, msg): + self.broadcasted.append(msg) + + @property + def body(self): + """Concatenated response body as a string.""" + return "".join( + c.decode() if isinstance(c, bytes) else str(c) for c in self.written + ) + + def json_body(self): + return json.loads(self.body) + + def add_sub(self, sid="sub_0", eid="main"): + sub = FakeSocket(sid=sid, eid=eid) + self.subs[sid] = sub + return sub + + def add_source(self, sid="src_0", eid="main"): + source = FakeSocket(sid=sid, eid=eid) + self.sources[sid] = source + return source + + +class SpyStore(JSONStore): + """JSONStore that counts backend calls while delegating to the real impl. + + Used to prove the server reaches persistence through the DataStore + abstraction instead of touching ``env_path`` directly. + """ + + def __init__(self, env_path): + super().__init__(env_path) + self.calls = { + "list_envs": 0, + "load_env": [], + "save_env": [], + "delete_env": [], + "save_undo": [], + } + + def list_envs(self): + self.calls["list_envs"] += 1 + return super().list_envs() + + def load_env(self, eid): + self.calls["load_env"].append(eid) + return super().load_env(eid) + + def save_env(self, eid, env_data): + self.calls["save_env"].append(eid) + return super().save_env(eid, env_data) + + def delete_env(self, eid): + self.calls["delete_env"].append(eid) + return super().delete_env(eid) + + def save_undo(self, eid, stack): + self.calls["save_undo"].append(eid) + return super().save_undo(eid, stack) diff --git a/py/tests/testutils/http.py b/py/tests/testutils/http.py new file mode 100644 index 000000000..d14c40bec --- /dev/null +++ b/py/tests/testutils/http.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Base class for tests that drive a real Application over HTTP. + +``AsyncHTTPTestCase`` starts the Tornado app in-process on an ephemeral port, so +these stay hermetic: no externally launched server, nothing bound to 8097, and +therefore no ``server`` marker. +""" + +import json +import shutil +import tempfile + +import tornado.testing + +from visdom.server.app import Application + + +class VisdomHTTPTestCase(tornado.testing.AsyncHTTPTestCase): + """Application-backed HTTP fixture with a disposable ``env_path``. + + Subclasses override ``app_kwargs`` to vary server configuration (for + example ``{"readonly": True}``) without reimplementing ``get_app``. + """ + + app_kwargs = {} + + def setUp(self): + self.env_path = tempfile.mkdtemp(prefix="visdom_test_") + super().setUp() + + def tearDown(self): + super().tearDown() + shutil.rmtree(self.env_path, ignore_errors=True) + + def get_app(self): + return Application( + port=self.get_http_port(), env_path=self.env_path, **self.app_kwargs + ) + + def post_json(self, path, body): + return self.fetch( + path, + method="POST", + body=json.dumps(body), + headers={"Content-Type": "application/json"}, + ) + + # -- Window helpers ------------------------------------------------------- + + def create_window(self, data, eid="main", win=None, opts=None, layout=None): + """POST to ``/events`` and return the assigned window id.""" + payload = {"data": data, "eid": eid, "layout": {} if layout is None else layout} + if win is not None: + payload["win"] = win + if opts is not None: + payload["opts"] = opts + return self.post_json("/events", payload).body.decode() + + def create_text_window(self, eid="main", content="test", win=None, opts=None): + return self.create_window( + [{"type": "text", "content": content}], eid=eid, win=win, opts=opts + ) + + def update(self, win, data, eid="main", **extra): + payload = {"win": win, "eid": eid, "data": data} + payload.update(extra) + return self.post_json("/update", payload) + + def close_window(self, win, eid="main"): + return self.post_json("/close", {"win": win, "eid": eid}) + + def win_exists(self, win, eid="main"): + return self.post_json("/win_exists", {"eid": eid, "win": win}).body == b"true" + + def get_win_data(self, win=None, eid="main"): + """Raw pane JSON for one window, or the whole env when ``win`` is None.""" + return json.loads(self.post_json("/win_data", {"eid": eid, "win": win}).body) + + # -- Environment helpers -------------------------------------------------- + + def get_envs(self): + return json.loads(self.post_json("/env_state", {}).body) + + def save(self, eids): + return self.post_json("/save", {"data": eids}) + + def panes(self, eid="main"): + """Live pane dict for ``eid`` straight off the application state.""" + return self._app.state[eid]["jsons"] diff --git a/py/tests/testutils/payloads.py b/py/tests/testutils/payloads.py new file mode 100644 index 000000000..421f43045 --- /dev/null +++ b/py/tests/testutils/payloads.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Builders for the request and storage payloads the server consumes. + +Keeping these in one place stops each test file from reinventing a slightly +different pane shape, which is how the previous copies drifted apart. +""" + + +def env_payload(win_id="win_0", jsons=None, reload=None): + """Environment as persisted by the DataStore: ``jsons`` plus ``reload``.""" + if jsons is None: + jsons = {win_id: {"id": win_id}} + return {"jsons": jsons, "reload": {} if reload is None else reload} + + +def plot_data(trace_type="scatter", x=None, y=None, name=None): + """A single plotly trace as the client sends it.""" + trace = { + "type": trace_type, + "x": [1, 2, 3] if x is None else x, + "y": [4, 5, 6] if y is None else y, + "mode": "lines", + } + if name is not None: + trace["name"] = name + return trace + + +def window_args( + data=None, + layout=None, + opts=None, + win=None, + eid=None, + version=None, +): + """Args dict accepted by ``server_utils.window`` and the ``/events`` route. + + ``data`` defaults to a single scatter trace, producing a generic ``plot`` + pane. Pass ``[{"type": "text", "content": "..."}]`` for a visdom-native + pane type instead. + """ + args = { + "data": [plot_data()] if data is None else data, + "layout": {} if layout is None else layout, + } + if opts is not None: + args["opts"] = opts + if win is not None: + args["win"] = win + if eid is not None: + args["eid"] = eid + if version is not None: + args["version"] = version + return args + + +def content_args(ptype, content, opts=None, win=None, eid=None): + """Args for a visdom-native pane (``text``, ``image``, ``embeddings``, ...). + + These are distinguished from generic plots by carrying a ``content`` key + inside ``data[0]``. + """ + return window_args( + data=[{"type": ptype, "content": content}], + opts=opts, + win=win, + eid=eid, + ) diff --git a/py/tests/testutils/sockets.py b/py/tests/testutils/sockets.py new file mode 100644 index 000000000..b8b29c2f1 --- /dev/null +++ b/py/tests/testutils/sockets.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Real socket handlers, built without a WebSocket connection. + +``FakeHandler`` is enough to drive ``AnySocketHandlerOrWrapper.on_message`` as +an unbound function, and ``storage_wiring.py`` does exactly that. It runs +out of road for three things this suite needs: + +* ``save_layouts`` calls ``self.broadcast_layouts()`` and reads ``self.app`` — + neither exists on a duck type. +* ``VisSocketHandlerOrWrapper.on_message`` ends in a zero-argument ``super()`` + call, which raises ``TypeError`` unless ``self`` really is an instance. +* ``open`` / ``on_close`` are the subject of the lifecycle tests, so they have + to be the real methods on a real object. + +The polling wrappers already exist for a related reason: ``SocketWrapper`` and +``VisSocketWrapper`` deliberately skip Tornado's ``__init__`` so the server can +mint one per polling client. That makes them constructible in a test, and their +``write_message`` records to a ``deque`` instead of a socket, so assertions read +straight off ``messages``. + +``socket_double`` finishes the object by hand rather than calling +``initialize``: the wrapper's own ``initialize`` starts a 15-second +``PeriodicCallback`` reaper and needs a running ``IOLoop``, which is polling's +business and belongs with the polling tests. +""" + +import json +import time +import types +from collections import deque + +from visdom.server.handlers.base_handlers import BaseWebSocketHandler +from visdom.server.handlers.socket_handlers import SocketWrapper, VisSocketWrapper + + +def socket_double(cls, app, remote_ip="127.0.0.1"): + """Build a ``cls`` bound to ``app`` with no network underneath it. + + ``cls`` is :class:`SocketWrapper` (a subscriber) or + :class:`VisSocketWrapper` (a source). The returned object is not registered + yet — call ``open()`` to exercise the registration path under test. + + ``request`` is a stub carrying only ``remote_ip``, which is all ``open()`` + reads off it. The server assigns the real one the same way, in + ``WrapSocketWrapper``'s GET route. + """ + sock = cls() + sock.request = types.SimpleNamespace(remote_ip=remote_ip) + sock.messages = deque() + sock.last_read_time = time.time() + BaseWebSocketHandler.initialize(sock, app) + return sock + + +def open_sub(app, remote_ip="127.0.0.1"): + """An opened subscriber socket, registered in ``app.subs``.""" + sub = socket_double(SocketWrapper, app, remote_ip) + sub.open() + return sub + + +def open_source(app, remote_ip="127.0.0.1"): + """An opened source socket, registered in ``app.sources``.""" + source = socket_double(VisSocketWrapper, app, remote_ip) + source.open() + return source + + +def sent(sock): + """Every message written to ``sock``, decoded, oldest first.""" + return [json.loads(m) if isinstance(m, str) else m for m in sock.messages] + + +def commands(sock): + """The ``command`` value of every dict message written to ``sock``.""" + return [m.get("command") for m in sent(sock) if isinstance(m, dict)] + + +def last(sock, command=None): + """Most recent decoded message, optionally filtered by ``command``.""" + for msg in reversed(sent(sock)): + if command is None or (isinstance(msg, dict) and msg.get("command") == command): + return msg + return None diff --git a/py/tests/test_data_model.py b/py/tests/unit/data_model.py similarity index 53% rename from py/tests/test_data_model.py rename to py/tests/unit/data_model.py index d1ea0255e..7e74c22b2 100644 --- a/py/tests/test_data_model.py +++ b/py/tests/unit/data_model.py @@ -12,14 +12,22 @@ running visdom server is needed, so these run under ``pytest -m "not server"``. """ +import copy +import errno +import hashlib import json import os import tempfile import unittest +import pytest + from visdom.data_model import JSONStore, DataStore +from visdom.data_model import json_store as json_store_module from visdom.utils.server_utils import LazyEnvData +pytestmark = pytest.mark.unit + def _env(win_id="win_0"): """Build a minimal environment payload (one window) for use in tests.""" @@ -285,5 +293,308 @@ def test_clear_undo_is_noop(self): self.assertIsNone(self.backend.clear_undo("expt")) +# -- Durability of the env write -------------------------------------------- +# +# Written as plain functions on the shared fixtures rather than as methods on +# the classes above: the conversion of those two classes belongs to a later +# clean-up, and fixtures cannot reach unittest.TestCase methods. + + +def _fail_after(monkeypatch, prefix_bytes): + """Make writes inside json_store emit ``prefix_bytes`` then die. + + Stands in for a process killed part-way through a save. ``open`` is looked + up in the module's globals before the builtins, so setting it on the module + intercepts only json_store's own writes. + """ + real_open = open + + class _DyingFile: + def __init__(self, handle): + self._handle = handle + + def write(self, data): + self._handle.write(data[:prefix_bytes]) + raise OSError(errno.EIO, "interrupted mid-write") + + def __enter__(self): + return self + + def __exit__(self, *exc_info): + self._handle.close() + return False + + def dying_open(path, mode="r", *args, **kwargs): + if "w" not in mode: + return real_open(path, mode, *args, **kwargs) + return _DyingFile(real_open(path, mode, *args, **kwargs)) + + monkeypatch.setattr(json_store_module, "open", dying_open, raising=False) + + +def test_interrupted_save_keeps_the_previous_env(store, monkeypatch): + """A save killed mid-write leaves the environment that was already there. + + Without the staging file the real env is truncated on open and load_env's + ValueError handler then reports the environment as empty — data loss with + no signal at all. + """ + store.save_env("main", _env("win_original")) + + _fail_after(monkeypatch, prefix_bytes=8) + with pytest.raises(OSError): + store.save_env("main", _env("win_replacement")) + + assert store.load_env("main") == _env("win_original") + + +def test_interrupted_save_leaves_a_usable_env(store, monkeypatch): + """The environment is still discoverable *and* readable after a failure. + + A truncated file keeps its name, so listing and existence alone would pass + even with the env destroyed; the reload is what makes this meaningful. + """ + store.save_env("main", _env()) + + _fail_after(monkeypatch, prefix_bytes=8) + with pytest.raises(OSError): + store.save_env("main", _env("win_replacement")) + + assert store.list_envs() == ["main"] + assert store.env_exists("main") + assert JSONStore(store.env_path).load_env("main") == _env() + + +def test_stranded_staging_file_is_not_an_env(store, env_path, monkeypatch): + """A leftover .tmp is on disk but never mistaken for an environment. + + list_envs only considers names ending in .json, and the staging file is + named .json.tmp precisely so that stays true. + """ + store.save_env("main", _env()) + + _fail_after(monkeypatch, prefix_bytes=8) + with pytest.raises(OSError): + store.save_env("main", _env("win_replacement")) + + assert "main.json.tmp" in os.listdir(env_path) + assert store.list_envs() == ["main"] + + +def test_successful_save_leaves_no_staging_file(store, env_path): + """The staging file is renamed away, not left behind.""" + store.save_env("main", _env()) + assert [n for n in os.listdir(env_path) if n.endswith(".tmp")] == [] + + +def test_failed_rename_keeps_the_previous_env(store, env_path, monkeypatch): + """If the rename itself fails, the old env is still the one on disk.""" + store.save_env("main", _env("win_original")) + + def boom(src, dst): + raise OSError(errno.EIO, "rename failed") + + monkeypatch.setattr(json_store_module.os, "replace", boom) + with pytest.raises(OSError): + store.save_env("main", _env("win_replacement")) + + monkeypatch.undo() + assert store.load_env("main") == _env("win_original") + + +def test_long_name_fallback_leaves_no_staging_file(store, env_path): + """The hash fallback path stages and renames too.""" + long_eid = "e" * 5000 + assert store.save_env(long_eid, _env()) + assert [n for n in os.listdir(env_path) if n.endswith(".tmp")] == [] + assert store.load_env(long_eid)["jsons"] == _env()["jsons"] + + +def test_interrupted_undo_save_keeps_the_previous_stack(store, monkeypatch): + """The undo stack was already written this way; it stays that way.""" + store.save_undo("main", [["win_0", {"id": "win_0"}]]) + + _fail_after(monkeypatch, prefix_bytes=4) + with pytest.raises(OSError): + store.save_undo("main", [["win_1", {"id": "win_1"}]]) + + assert store.load_undo("main") == [["win_0", {"id": "win_0"}]] + + +# -- Name collisions --------------------------------------------------------- + + +def _hashed_name(eid): + """The hash-fallback filename JSONStore would pick for ``eid``.""" + digest = hashlib.sha256(eid.encode("utf-8")).hexdigest() + return "hash_{0}.json".format(digest) + + +def test_env_named_like_a_hash_file_is_dropped_from_the_listing(store): + """An env whose own name matches hash_<64 hex> disappears from list_envs. + + HASHED_ENV_RE matches on the filename alone, so the file is read as a hash + fallback and skipped when the ``name`` field it expects is absent. The + environment is still saved, and still loads by id — only the listing loses + it. Documented here rather than fixed: changing the rule would break the + long-name files already on users' disks. + """ + colliding_eid = "hash_" + "a" * 64 + store.save_env(colliding_eid, _env()) + store.save_env("main", _env()) + + assert store.env_exists(colliding_eid) + assert store.load_env(colliding_eid) == _env() + assert store.list_envs() == ["main"] + + +def test_hash_fallback_file_reports_the_real_id(store, env_path): + """A genuine long-name fallback file does carry its id and is listed.""" + long_eid = "e" * 5000 + store.save_env(long_eid, _env()) + + assert _hashed_name(long_eid) in os.listdir(env_path) + assert store.list_envs() == [long_eid] + + +# -- LazyEnvData ------------------------------------------------------------- + + +def test_lazy_env_defers_the_read(spy_store): + """Constructing a LazyEnvData touches the backend not at all.""" + spy_store.save_env("main", _env()) + + LazyEnvData(spy_store, "main") + + assert spy_store.calls["load_env"] == [] + + +def test_lazy_env_reads_once_and_caches(spy_store): + """Repeated access hits the backend exactly once.""" + spy_store.save_env("main", _env()) + lazy = LazyEnvData(spy_store, "main") + + lazy["jsons"] + lazy["reload"] + len(lazy) + list(lazy) + + assert spy_store.calls["load_env"] == ["main"] + + +def test_lazy_env_satisfies_the_mapping_contract(store): + """It behaves like the dict it stands in for.""" + store.save_env("main", _env()) + lazy = LazyEnvData(store, "main") + + assert sorted(lazy.keys()) == ["jsons", "reload"] + assert lazy["jsons"] == _env()["jsons"] + assert lazy.get("reload") == {} + assert lazy.get("absent") is None + assert lazy.get("absent", "fallback") == "fallback" + assert "jsons" in lazy + assert "absent" not in lazy + assert len(lazy) == 2 + assert dict(lazy.items()) == _env() + assert lazy == _env() + + +def test_lazy_env_setitem_materialises_first(store): + """Writing a key loads the env, so the write lands on real data.""" + store.save_env("main", _env()) + lazy = LazyEnvData(store, "main") + + lazy["reload"] = {"win_0": [0, 0, 3, 3]} + + assert lazy["jsons"] == _env()["jsons"] + assert lazy["reload"] == {"win_0": [0, 0, 3, 3]} + + +def test_lazy_env_raises_value_error_for_a_malformed_env(store, env_path): + """A file that is not a valid env becomes a ValueError naming the id.""" + with open(os.path.join(env_path, "broken.json"), "w") as fn: + fn.write("{not valid json") + + lazy = LazyEnvData(store, "broken") + with pytest.raises(ValueError, match="broken"): + lazy.lazy_load_data() + + +def test_lazy_env_raises_value_error_for_a_missing_env(store): + """A never-saved env is malformed too: load_env returns {} with no jsons.""" + with pytest.raises(ValueError, match="Failed loading environment json"): + LazyEnvData(store, "ghost").lazy_load_data() + + +def test_lazy_env_keeps_the_experiment_blob(store): + """Metadata beyond jsons/reload survives the lazy load.""" + payload = dict(_env(), experiment={"name": "run-1"}) + store.save_env("main", payload) + + assert LazyEnvData(store, "main")["experiment"] == {"name": "run-1"} + + +# -- Deep copies of environment state ---------------------------------------- +# +# ForkEnvHandler (web_handlers.py) and the socket "save" command both fork an +# environment with copy.deepcopy. When the source is a LazyEnvData that means +# the store reference is deep-copied alongside the data. + + +def test_deep_copy_of_a_lazy_env_is_independent(store): + """Mutating the fork leaves the source environment alone.""" + store.save_env("main", _env()) + source = LazyEnvData(store, "main") + source.lazy_load_data() + + fork = copy.deepcopy(source) + fork["jsons"]["win_1"] = {"id": "win_1"} + + assert "win_1" not in source["jsons"] + assert sorted(fork["jsons"]) == ["win_0", "win_1"] + + +def test_deep_copy_clones_the_store_reference(store): + """The fork gets its own JSONStore, not the one the source holds. + + Harmless today because JSONStore's only state is the path, but it means the + fork does not observe a later swap of the source's backend. + """ + store.save_env("main", _env()) + source = LazyEnvData(store, "main") + source.lazy_load_data() + + fork = copy.deepcopy(source) + + assert fork._store is not source._store + assert fork._store.env_path == source._store.env_path + assert fork._store.load_env("main") == _env() + + +def test_deep_copy_of_an_unmaterialised_lazy_env_still_loads(store): + """A fork taken before the first read resolves against the copied store.""" + store.save_env("main", _env()) + source = LazyEnvData(store, "main") + + fork = copy.deepcopy(source) + + assert fork["jsons"] == _env()["jsons"] + assert source._raw_dict is None + + +def test_forked_env_persists_under_its_own_id(store): + """Saving a deep copy under a new id writes a second, separate file.""" + store.save_env("main", _env()) + source = LazyEnvData(store, "main") + source.lazy_load_data() + + fork = copy.deepcopy(source) + fork["jsons"]["win_1"] = {"id": "win_1"} + store.save_env("forked", fork) + + assert store.load_env("main") == _env() + assert sorted(store.load_env("forked")["jsons"]) == ["win_0", "win_1"] + + if __name__ == "__main__": unittest.main() diff --git a/py/tests/test_experiment_store.py b/py/tests/unit/experiment_store.py similarity index 100% rename from py/tests/test_experiment_store.py rename to py/tests/unit/experiment_store.py diff --git a/py/tests/test_image_slider.py b/py/tests/unit/image_slider.py similarity index 98% rename from py/tests/test_image_slider.py rename to py/tests/unit/image_slider.py index a2e9f1cd0..5d4603007 100644 --- a/py/tests/test_image_slider.py +++ b/py/tests/unit/image_slider.py @@ -35,7 +35,7 @@ def _plot_pane(self): } def _update(self, pane, args): - return UpdateHandler.update(pane, args, 500, 50, 4) + return UpdateHandler.update(pane, args, 500, 50, 4, 4) def test_sets_index(self): p = self._update(self._pane(), self._args(3)) @@ -141,6 +141,7 @@ def __init__(self, pane): self.max_text_lines = 500 self.max_old_content = 50 self.max_image_history = 4 + self.max_plot_history = 4 def set_status(self, code): self.status = code diff --git a/py/tests/unit/memory_caps.py b/py/tests/unit/memory_caps.py new file mode 100644 index 000000000..eddb6cfe3 --- /dev/null +++ b/py/tests/unit/memory_caps.py @@ -0,0 +1,343 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""In-memory growth caps on the panes that append rather than replace. + +A long-running job appending to the same pane used to grow the server's state +without bound (#1320). Four pane fields are capped -- text content, embedding +undo history, image history and plot history -- and each keeps the *newest* +entries, since those are what the frontend shows. The caps are passed in rather +than read from ``defaults``, so these drive ``UpdateHandler`` directly with +small limits instead of appending five hundred times. +""" + +from unittest.mock import MagicMock + +import pytest + +from visdom.server.defaults import ( + DEFAULT_MAX_IMAGE_HISTORY, + DEFAULT_MAX_OLD_CONTENT, + DEFAULT_MAX_PLOT_HISTORY, + DEFAULT_MAX_TEXT_LINES, +) +from visdom.server.handlers.base_handlers import BaseHandler +from visdom.server.handlers.web_handlers import UpdateHandler +from visdom.utils.shared_utils import get_rand_id + +pytestmark = pytest.mark.unit + + +def _pane(ptype, **extra): + pane = { + "command": "window", + "version": 1, + "id": "win_{}".format(ptype), + "title": ptype, + "inflate": True, + "width": None, + "height": None, + "contentID": get_rand_id(), + "type": ptype, + "i": 0, + } + pane.update(extra) + return pane + + +def _text_pane(content="line0"): + return _pane("text", content=content) + + +def _embeddings_pane(old_content=None): + return _pane( + "embeddings", + content={ + "data": [[1, 2], [3, 4]], + "labels": ["a", "b"], + "selected": None, + "has_previous": False, + }, + old_content=[] if old_content is None else old_content, + ) + + +def _image_history_pane(images=None): + if images is None: + images = [{"src": "data:image/png;base64,AAA", "caption": "img0"}] + return _pane( + "image_history", + content=list(images), + selected=0, + show_slider=True, + ) + + +def _update( + p, + args, + max_text=DEFAULT_MAX_TEXT_LINES, + max_old=DEFAULT_MAX_OLD_CONTENT, + max_img=DEFAULT_MAX_IMAGE_HISTORY, + max_plot=DEFAULT_MAX_PLOT_HISTORY, +): + """Call ``UpdateHandler.update`` with explicit caps.""" + return UpdateHandler.update(p, args, max_text, max_old, max_img, max_plot) + + +def _update_embeddings(p, args, max_old=DEFAULT_MAX_OLD_CONTENT): + """Call the embeddings update path with an explicit ``old_content`` cap. + + Embeddings never reach ``UpdateHandler.update``: the handler routes them to + ``update_embeddings_packet`` first, which mutates the window in place and + returns a JSON patch rather than the window. + """ + UpdateHandler.update_embeddings_packet(p, args, max_old) + return p + + +def _append_lines(count, cap, start="line0"): + """Append ``count`` numbered lines to a text pane held at ``cap`` lines.""" + p = _text_pane(start) + for i in range(1, count + 1): + p = _update(p, {"data": [{"content": "line{}".format(i)}]}, max_text=cap) + return p["content"].split("
") + + +def _append_images(count, cap): + """Append ``count`` numbered frames to a history pane held at ``cap``.""" + p = _image_history_pane() + for i in range(1, count + 1): + args = { + "data": [ + { + "type": "image_history", + "content": { + "src": "data:image/png;base64,img{}".format(i), + "caption": "img{}".format(i), + }, + } + ] + } + p = _update(p, args, max_img=cap) + return p + + +def _plot_history_pane(frames=None): + if frames is None: + frames = [{"data": [], "layout": {}, "caption": "frame0"}] + return _pane( + "plot_history", + content=list(frames), + selected=0, + show_slider=True, + ) + + +def _append_frames(count, cap): + """Append ``count`` numbered frames to a plot history held at ``cap``.""" + p = _plot_history_pane() + for i in range(1, count + 1): + args = { + "data": [ + { + "type": "plot_history", + "content": { + "data": [{"type": "scatter", "x": [i], "y": [i]}], + "layout": {}, + "caption": "frame{}".format(i), + }, + } + ] + } + p = _update(p, args, max_plot=cap) + return p + + +def _region_select(points): + return {"data": {"update_type": "RegionSelected", "points": points}} + + +# -- Text ------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "appends, cap, expected", + [(1, 5, 2), (5, 6, 6), (4, 3, 3), (199, 10, 10)], + ids=["under-cap", "at-cap", "over-cap", "long-run"], +) +def test_text_content_is_capped(appends, cap, expected): + assert len(_append_lines(appends, cap)) == expected + + +def test_text_appends_join_with_a_line_break(): + assert _append_lines(1, 5, start="hello") == ["hello", "line1"] + + +def test_text_truncation_keeps_the_newest_line(): + assert _append_lines(4, 3)[-1] == "line4" + + +def test_text_truncation_drops_the_oldest_lines(): + lines = _append_lines(4, 3) + assert "line0" not in lines + assert "line1" not in lines + + +def test_default_text_cap(): + assert DEFAULT_MAX_TEXT_LINES == 500 + + +# -- Embeddings undo history ------------------------------------------------- + + +def test_region_select_stacks_the_replaced_points(): + p = _embeddings_pane() + original = p["content"]["data"] + p = _update_embeddings(p, _region_select([[10, 20]]), max_old=5) + assert p["old_content"] == [original] + assert p["content"]["has_previous"] + + +def test_old_content_under_the_cap_is_kept_whole(): + p = _embeddings_pane(old_content=[[[1, 2]], [[3, 4]]]) + p = _update_embeddings(p, _region_select([[5, 6]]), max_old=5) + assert len(p["old_content"]) == 3 + + +def test_old_content_truncation_keeps_the_newest_entry(): + p = _embeddings_pane(old_content=[[[i, i]] for i in range(3)]) + p = _update_embeddings(p, _region_select([[99, 99]]), max_old=3) + assert len(p["old_content"]) == 3 + assert p["old_content"][-1] == [[1, 2], [3, 4]] + + +def test_repeated_region_selects_stay_bounded(): + p = _embeddings_pane() + for i in range(50): + p = _update_embeddings(p, _region_select([[i, i]]), max_old=5) + assert len(p["old_content"]) == 5 + + +def test_entity_select_does_not_grow_old_content(): + p = _embeddings_pane() + args = {"data": {"update_type": "EntitySelected", "selected": 0}} + p = _update_embeddings(p, args, max_old=5) + assert p["old_content"] == [] + + +def test_default_old_content_cap(): + assert DEFAULT_MAX_OLD_CONTENT == 50 + + +# -- Image history ----------------------------------------------------------- + + +@pytest.mark.parametrize( + "appends, cap, expected", + [(1, 4, 2), (2, 4, 3), (5, 3, 3), (199, 4, 4)], + ids=["under-cap", "still-under-cap", "over-cap", "long-run"], +) +def test_image_history_is_capped(appends, cap, expected): + assert len(_append_images(appends, cap)["content"]) == expected + + +def test_image_history_truncation_keeps_the_newest_frame(): + assert _append_images(5, 3)["content"][-1]["caption"] == "img5" + + +def test_image_history_truncation_drops_the_oldest_frames(): + captions = [img["caption"] for img in _append_images(5, 3)["content"]] + assert "img0" not in captions + assert "img1" not in captions + + +@pytest.mark.parametrize("cap", [1, 2, 4]) +def test_selected_stays_a_valid_index_after_truncation(cap): + p = _append_images(9, cap) + assert 0 <= p["selected"] < len(p["content"]) + + +def test_explicit_selection_is_unaffected_by_the_cap(): + images = [ + {"src": "data:image/png;base64,A", "caption": "img{}".format(i)} + for i in range(3) + ] + p = _image_history_pane(images) + p["selected"] = 2 + args = {"data": [{"type": "image_update_selected", "selected": 0}]} + p = _update(p, args, max_img=4) + assert p["selected"] == 0 + assert len(p["content"]) == 3 + + +def test_default_image_history_cap(): + assert DEFAULT_MAX_IMAGE_HISTORY == 4 + + +# -- Plot history ------------------------------------------------------------ + + +@pytest.mark.parametrize( + "appends, cap, expected", + [(1, 4, 2), (2, 4, 3), (5, 3, 3), (999, 4, 4)], + ids=["under-cap", "still-under-cap", "over-cap", "long-run"], +) +def test_plot_history_is_capped(appends, cap, expected): + assert len(_append_frames(appends, cap)["content"]) == expected + + +def test_plot_history_truncation_keeps_the_newest_frame(): + assert _append_frames(5, 3)["content"][-1]["caption"] == "frame5" + + +def test_plot_history_truncation_drops_the_oldest_frames(): + captions = [frame["caption"] for frame in _append_frames(5, 3)["content"]] + assert "frame0" not in captions + assert "frame1" not in captions + + +@pytest.mark.parametrize("cap", [1, 2, 4]) +def test_plot_selected_stays_a_valid_index_after_truncation(cap): + p = _append_frames(9, cap) + assert 0 <= p["selected"] < len(p["content"]) + + +def test_explicit_frame_selection_is_unaffected_by_the_cap(): + frames = [ + {"data": [], "layout": {}, "caption": "frame{}".format(i)} for i in range(3) + ] + p = _plot_history_pane(frames) + p["selected"] = 2 + args = {"data": [{"type": "plot_update_selected", "selected": 0}]} + p = _update(p, args, max_plot=4) + assert p["selected"] == 0 + assert len(p["content"]) == 3 + + +def test_default_plot_history_cap(): + assert DEFAULT_MAX_PLOT_HISTORY == 4 + + +# -- Handler wiring ---------------------------------------------------------- + + +def test_handler_copies_the_caps_off_the_application(): + app = MagicMock() + app.max_text_lines = 100 + app.max_old_content = 20 + app.max_image_history = 8 + app.max_plot_history = 6 + + handler = MagicMock(spec=BaseHandler) + BaseHandler.initialize(handler, app=app) + + assert handler.max_text_lines == 100 + assert handler.max_old_content == 20 + assert handler.max_image_history == 8 + assert handler.max_plot_history == 6 diff --git a/py/tests/test_plots.py b/py/tests/unit/plots.py similarity index 100% rename from py/tests/test_plots.py rename to py/tests/unit/plots.py diff --git a/py/tests/unit/server_utils.py b/py/tests/unit/server_utils.py new file mode 100644 index 000000000..a81ee15e1 --- /dev/null +++ b/py/tests/unit/server_utils.py @@ -0,0 +1,162 @@ +""" +Unit tests for server utility functions. +No server needed — these test pure functions directly. +""" + +import unittest + +from visdom.utils.server_utils import ( + escape_eid, + extract_eid, + hash_password, + stringify, + recursive_order, +) + + +class TestEscapeEid(unittest.TestCase): + """Tests for escape_eid() — sanitizes environment IDs.""" + + def test_forward_slash_replaced(self): + self.assertEqual(escape_eid("a/b"), "a_b") + + def test_backslash_replaced(self): + self.assertEqual(escape_eid("a\\b"), "a_b") + + def test_newline_replaced(self): + self.assertEqual(escape_eid("a\nb"), "a-b") + + def test_carriage_return_replaced(self): + self.assertEqual(escape_eid("a\rb"), "a-b") + + def test_multiple_special_chars(self): + self.assertEqual(escape_eid("a/b\\c\nd\re"), "a_b_c-d-e") + + def test_normal_string_unchanged(self): + self.assertEqual(escape_eid("my_environment"), "my_environment") + + def test_unicode_preserved(self): + self.assertEqual(escape_eid("env_éàü"), "env_éàü") + + def test_empty_string(self): + self.assertEqual(escape_eid(""), "") + + +class TestExtractEid(unittest.TestCase): + """Tests for extract_eid() — extracts and escapes eid from args dict.""" + + def test_default_is_main(self): + self.assertEqual(extract_eid({}), "main") + + def test_none_value_returns_main(self): + self.assertEqual(extract_eid({"eid": None}), "main") + + def test_with_value(self): + self.assertEqual(extract_eid({"eid": "test"}), "test") + + def test_escapes_value(self): + self.assertEqual(extract_eid({"eid": "a/b"}), "a_b") + + +class TestHashPassword(unittest.TestCase): + """Tests for hash_password() — PBKDF2-HMAC-SHA256 with salt.""" + + def test_same_password_same_salt_matches(self): + h1 = hash_password("secret") + salt = h1.split("$")[0] + h2 = hash_password("secret", salt=salt) + self.assertEqual(h1, h2) + + def test_same_password_different_salt_differs(self): + h1 = hash_password("secret") + h2 = hash_password("secret") + self.assertNotEqual(h1, h2) + + def test_different_passwords_same_salt_differs(self): + h1 = hash_password("password1") + salt = h1.split("$")[0] + h2 = hash_password("password2", salt=salt) + self.assertNotEqual(h1, h2) + + def test_output_format(self): + h = hash_password("test") + parts = h.split("$") + self.assertEqual(len(parts), 2) + salt_hex, dk_hex = parts + self.assertEqual(len(salt_hex), 64) + self.assertEqual(len(dk_hex), 64) + self.assertTrue(all(c in "0123456789abcdef" for c in salt_hex)) + self.assertTrue(all(c in "0123456789abcdef" for c in dk_hex)) + + def test_full_login_flow(self): + import hashlib as hl + + raw_password = "admin123" + client_hash = hl.sha256(raw_password.encode("utf-8")).hexdigest() + stored = hash_password(client_hash) + salt = stored.split("$")[0] + login_hash = hash_password(client_hash, salt=salt) + self.assertEqual(stored, login_hash) + + +class TestStringify(unittest.TestCase): + """Tests for stringify() — deterministic JSON serialization.""" + + def test_orders_keys(self): + result = stringify({"b": 1, "a": 2}) + self.assertLess(result.index('"a"'), result.index('"b"')) + + def test_converts_integer_floats(self): + result = stringify({"x": 1.0}) + self.assertIn(":1", result) + self.assertNotIn("1.0", result) + + def test_preserves_non_integer_floats(self): + result = stringify({"x": 1.5}) + self.assertIn("1.5", result) + + def test_nested_key_ordering(self): + result = stringify({"z": {"b": 1, "a": 2}, "a": 3}) + # "a":3 should come before "z" + self.assertLess(result.index('"a":3'), result.index('"z"')) + + def test_minimal_separators(self): + result = stringify({"a": 1, "b": 2}) + # No spaces around : or , + self.assertNotIn(": ", result) + self.assertNotIn(", ", result) + + +class TestRecursiveOrder(unittest.TestCase): + """Tests for recursive_order() — used by stringify.""" + + def test_orders_dict_keys(self): + result = recursive_order({"c": 1, "a": 2, "b": 3}) + self.assertEqual(list(result.keys()), ["a", "b", "c"]) + + def test_handles_list(self): + result = recursive_order([3, 1, 2]) + self.assertEqual(result, [3, 1, 2]) # Lists not sorted, just traversed + + def test_handles_nested(self): + result = recursive_order({"b": {"d": 1, "c": 2}, "a": 3}) + self.assertEqual(list(result.keys()), ["a", "b"]) + self.assertEqual(list(result["b"].keys()), ["c", "d"]) + + def test_integer_float_conversion(self): + self.assertEqual(recursive_order(3.0), 3) + self.assertIsInstance(recursive_order(3.0), int) + + def test_non_integer_float_preserved(self): + self.assertEqual(recursive_order(3.5), 3.5) + self.assertIsInstance(recursive_order(3.5), float) + + def test_string_unchanged(self): + self.assertEqual(recursive_order("hello"), "hello") + + def test_bytes_unchanged(self): + self.assertEqual(recursive_order(b"hello"), b"hello") + + +if __name__ == "__main__": + unittest.main() diff --git a/py/tests/unit/shared_utils.py b/py/tests/unit/shared_utils.py new file mode 100644 index 000000000..18c1ab169 --- /dev/null +++ b/py/tests/unit/shared_utils.py @@ -0,0 +1,331 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Unit tests for ``visdom.utils.shared_utils``. + +These helpers sit under every JSON response the server writes and every plot +payload the client builds, but nothing in the suite exercised them directly. +Two of them are subtle enough to be worth pinning down explicitly: + +* ``_sanitize_nans`` changes *shape* as well as values — a tuple comes back as + a list — because JSON has no tuple. +* ``warn_once`` dedupes against a module-level set, so a warning raised by one + test silently suppresses the same warning in another. ``conftest.py`` carries + an autouse fixture for that; the paired tests below are what prove it works. +""" + +import io +import json +import math +import os +import uuid +import warnings + +import numpy as np +import pytest + +from visdom.utils import shared_utils +from visdom.utils.shared_utils import ( + NanSafeEncoder, + _coerce_image_slider_index, + _is_missing_value, + _sanitize_nans, + ensure_dir_exists, + get_new_window_id, + get_rand_id, + get_visdom_path, + warn_once, +) + +pytestmark = pytest.mark.unit + + +# -- _sanitize_nans ---------------------------------------------------------- + + +@pytest.mark.parametrize( + "value", + [ + float("nan"), + float("inf"), + float("-inf"), + np.float32("nan"), + np.float64("inf"), + np.float64("-inf"), + ], +) +def test_sanitize_replaces_non_finite_floats(value): + """NaN and both infinities become None, whatever float type carries them.""" + assert _sanitize_nans(value) is None + + +@pytest.mark.parametrize( + "value", + [0, -1, 3.5, np.float64(2.5), "text", "", None, True, False, b"bytes"], +) +def test_sanitize_leaves_everything_else_alone(value): + """Finite numbers, strings, booleans and None pass through untouched.""" + assert _sanitize_nans(value) == value + + +def test_sanitize_recurses_through_nested_containers(): + """Non-finite values are replaced at any depth, inside dicts and lists.""" + payload = { + "data": [{"y": [1.0, float("nan"), 3.0]}], + "layout": {"yaxis": {"range": [float("-inf"), float("inf")]}}, + "title": "keep me", + } + assert _sanitize_nans(payload) == { + "data": [{"y": [1.0, None, 3.0]}], + "layout": {"yaxis": {"range": [None, None]}}, + "title": "keep me", + } + + +def test_sanitize_coerces_tuples_to_lists(): + """Tuples come back as lists — JSON has no tuple, so the shape changes.""" + result = _sanitize_nans((1.0, float("nan"))) + assert result == [1.0, None] + assert isinstance(result, list) + + +def test_sanitize_coerces_nested_tuples(): + """The tuple coercion applies at every level, including dict values.""" + result = _sanitize_nans({"pairs": [(1, 2), (3, 4)]}) + assert result == {"pairs": [[1, 2], [3, 4]]} + + +def test_sanitize_leaves_ndarrays_untouched(): + """An ndarray is not a list, so it is returned as-is for the encoder to reject.""" + array = np.array([1.0, np.nan]) + assert _sanitize_nans(array) is array + + +# -- NanSafeEncoder ---------------------------------------------------------- + + +def test_encoder_writes_null_not_nan(): + """dumps() emits JSON null, never the non-standard NaN/Infinity tokens.""" + payload = {"y": [1.0, float("nan"), float("inf"), float("-inf")]} + encoded = json.dumps(payload, cls=NanSafeEncoder) + assert "NaN" not in encoded + assert "Infinity" not in encoded + assert json.loads(encoded) == {"y": [1.0, None, None, None]} + + +def test_encoder_covers_the_streaming_path(): + """dump() to a stream goes through iterencode, which is patched separately.""" + stream = io.StringIO() + json.dump({"y": [float("nan")]}, stream, cls=NanSafeEncoder) + assert json.loads(stream.getvalue()) == {"y": [None]} + + +def test_encoder_round_trips_ordinary_payloads(): + """A payload with nothing to sanitise survives unchanged.""" + payload = {"data": [{"x": [1, 2], "y": [3.5, 4.5], "type": "scatter"}]} + assert json.loads(json.dumps(payload, cls=NanSafeEncoder)) == payload + + +def test_default_encoder_still_emits_nan(): + """The plain encoder does not do this, which is why NanSafeEncoder exists.""" + assert "NaN" in json.dumps({"y": [float("nan")]}) + + +# -- _is_missing_value ------------------------------------------------------- + + +@pytest.mark.parametrize( + "value", + [None, float("nan"), float("inf"), float("-inf"), np.float32("nan")], +) +def test_missing_value_detects_gaps(value): + """Clients mark a gap in a series with None, NaN or Inf.""" + assert _is_missing_value(value) is True + + +@pytest.mark.parametrize( + "value", + [0, 0.0, -1, "", "cat", "2026-01-01", [], {}, np.int64(3), np.float64(1.5), False], +) +def test_missing_value_treats_everything_else_as_present(value): + """Categorical strings and falsy-but-real values are present, not gaps. + + Passing a string to ``math.isnan`` raises ``TypeError``; guarding on the + numeric types instead is what keeps a categorical x-axis from 500ing. + """ + assert _is_missing_value(value) is False + + +# -- warn_once --------------------------------------------------------------- + + +def _prime(message, warningtype=None): + """Record ``message`` as already seen, without leaking it to the report.""" + with warnings.catch_warnings(record=True): + warnings.simplefilter("always") + warn_once(message, warningtype) + + +def test_warn_once_warns_the_first_time(): + """The first call with a given message raises the warning.""" + with pytest.warns(UserWarning, match="first sighting"): + warn_once("first sighting") + + +def test_warn_once_is_silent_the_second_time(): + """A repeat of the same message is suppressed.""" + _prime("repeated message") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + warn_once("repeated message") + assert caught == [] + + +def test_warn_once_still_warns_for_a_different_message(): + """Deduplication is per message, not global.""" + _prime("message A") + with pytest.warns(UserWarning, match="message B"): + warn_once("message B") + + +def test_warn_once_honours_the_warning_type(): + """The requested category reaches the warnings machinery.""" + with pytest.warns(DeprecationWarning, match="going away"): + warn_once("going away", DeprecationWarning) + + +def test_reset_fixture_isolates_a_message_part_one(): + """Raise a warning that the next test raises again (see part two).""" + with pytest.warns(UserWarning, match="shared between tests"): + warn_once("shared between tests") + + +def test_reset_fixture_isolates_a_message_part_two(): + """The same message must warn again: reset_warn_once cleared the set. + + Without the autouse fixture in ``conftest.py`` this fails, and only when the + two tests run in this order — the failure mode the fixture exists to stop. + """ + with pytest.warns(UserWarning, match="shared between tests"): + warn_once("shared between tests") + + +def test_reset_fixture_restores_preexisting_entries(): + """Messages seen before the suite started are put back after each test.""" + assert isinstance(shared_utils._seen_warnings, set) + _prime("recorded in the module global") + assert "recorded in the module global" in shared_utils._seen_warnings + + +# -- path helpers ------------------------------------------------------------ + + +def test_visdom_path_is_the_package_directory(): + """With no argument the package's own directory is returned.""" + import visdom + + assert get_visdom_path() == os.path.dirname(visdom.__file__) + + +def test_visdom_path_joins_a_relative_asset(): + """A filename is joined onto the package directory.""" + assert get_visdom_path("static") == os.path.join(get_visdom_path(), "static") + + +def test_visdom_path_resolves_a_real_asset(): + """The join points at something that actually ships with the package.""" + assert os.path.isdir(get_visdom_path("static")) + + +def test_ensure_dir_creates_missing_parents(tmp_path): + """Intermediate directories are created, not just the leaf.""" + target = tmp_path / "envs" / "view" / "nested" + ensure_dir_exists(str(target)) + assert target.is_dir() + + +def test_ensure_dir_is_idempotent(tmp_path): + """Calling it on an existing directory is a no-op, not an error.""" + ensure_dir_exists(str(tmp_path)) + ensure_dir_exists(str(tmp_path)) + assert tmp_path.is_dir() + + +def test_ensure_dir_keeps_existing_contents(tmp_path): + """An existing directory is left alone, contents included.""" + (tmp_path / "main.json").write_text("{}") + ensure_dir_exists(str(tmp_path)) + assert (tmp_path / "main.json").read_text() == "{}" + + +# -- id helpers -------------------------------------------------------------- + + +def test_rand_id_is_a_uuid(): + """get_rand_id returns a parseable UUID string.""" + uuid.UUID(get_rand_id()) + + +def test_rand_ids_are_distinct(): + """Two calls never collide.""" + assert get_rand_id() != get_rand_id() + + +def test_new_window_id_is_prefixed(): + """Window ids carry the window_ prefix over a UUID.""" + win = get_new_window_id() + assert win.startswith("window_") + uuid.UUID(win[len("window_") :]) + + +# -- _coerce_image_slider_index ---------------------------------------------- + + +@pytest.mark.parametrize( + "index,expected", + [ + (3, 3), + (0, 0), + (-2, -2), + (2.0, 2), + (np.int32(4), 4), + (np.float64(5.0), 5), + (np.array([6]), 6), + ], +) +def test_slider_index_normalises_to_int(index, expected): + """Whole numbers of any numeric flavour become a plain int.""" + result = _coerce_image_slider_index(index) + assert result == expected + assert type(result) is int + + +@pytest.mark.parametrize("index", [True, False, "3", None, [3]]) +def test_slider_index_rejects_non_numbers(index): + """Booleans and non-numerics are a TypeError, not a silent int().""" + with pytest.raises(TypeError): + _coerce_image_slider_index(index) + + +@pytest.mark.parametrize("index", [2.5, float("nan"), float("inf")]) +def test_slider_index_rejects_unusable_floats(index): + """A fractional or non-finite float cannot name a frame.""" + with pytest.raises(ValueError): + _coerce_image_slider_index(index) + + +def test_slider_index_rejects_multi_element_arrays(): + """An array has to hold exactly one value to be an index.""" + with pytest.raises(TypeError): + _coerce_image_slider_index(np.array([1, 2])) + + +def test_math_isnan_would_raise_on_a_string(): + """Documents why _is_missing_value guards on type before calling isnan.""" + with pytest.raises(TypeError): + math.isnan("cat") diff --git a/py/tests/test_smoke.py b/py/tests/unit/smoke.py similarity index 100% rename from py/tests/test_smoke.py rename to py/tests/unit/smoke.py diff --git a/py/tests/test_socket_setup.py b/py/tests/unit/socket_setup.py similarity index 100% rename from py/tests/test_socket_setup.py rename to py/tests/unit/socket_setup.py diff --git a/py/tests/unit/tsne.py b/py/tests/unit/tsne.py new file mode 100644 index 000000000..1cdcf1af4 --- /dev/null +++ b/py/tests/unit/tsne.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 + +# Copyright 2017-present, The Visdom Authors +# All rights reserved. +# +# This source code is licensed under the license found in the +# LICENSE file in the root directory of this source tree. + +"""Unit tests for the t-SNE helpers and backend selection. + +``do_tsne`` is bound at import time by a try/except ladder over openTSNE and +bhtsne (``visdom/__init__.py:77-111``), so the only way to exercise the ladder +is to reload the module with the candidate backends faked out. That mutates +global state for the rest of the session, so every reload goes through +``reloaded_visdom``, which restores the real module even when the test fails. +""" + +import importlib +import sys +import types +from unittest.mock import MagicMock + +import numpy as np +import pytest + +import visdom +from visdom import _get_perplexity, _normalize_tsne + +pytestmark = pytest.mark.unit + +_MISSING = object() + + +@pytest.fixture +def reloaded_visdom(): + """Reload ``visdom`` with a patched ``sys.modules``, then put it back. + + Yields a callable taking the ``sys.modules`` overrides to apply; it returns + the freshly reloaded module. The teardown reload runs unconditionally, so a + failing assertion cannot leave the rest of the suite running against a + mocked backend. + """ + saved = {} + + def reload_with(overrides): + for name, module in overrides.items(): + saved[name] = sys.modules.get(name, _MISSING) + sys.modules[name] = module + return importlib.reload(visdom) + + try: + yield reload_with + finally: + for name, module in saved.items(): + if module is _MISSING: + sys.modules.pop(name, None) + else: + sys.modules[name] = module + importlib.reload(visdom) + + +def _fake_opentsne(embedding): + """An ``openTSNE`` stand-in whose ``TSNE(...).fit(X)`` returns ``embedding``.""" + instance = MagicMock() + instance.fit.return_value = np.asarray(embedding) + module = types.ModuleType("openTSNE") + module.TSNE = MagicMock(return_value=instance) + return module + + +def _fake_bhtsne(embedding): + """A ``visdom.extra_deps.bhtsne.bhtsne`` stand-in for ``run_bh_tsne``. + + Returns the ``sys.modules`` overrides plus the leaf module, so a test can + assert on the call. Every level of the dotted path needs an entry: the + import machinery stops descending as soon as a name resolves from + ``sys.modules``, so a gap in the chain leaves the parent package without the + attribute the ``import ... as`` binding then looks for. + """ + leaf = types.ModuleType("visdom.extra_deps.bhtsne.bhtsne") + leaf.run_bh_tsne = MagicMock(return_value=np.asarray(embedding)) + package = types.ModuleType("visdom.extra_deps.bhtsne") + package.bhtsne = leaf + extra_deps = types.ModuleType("visdom.extra_deps") + extra_deps.bhtsne = package + overrides = { + "openTSNE": None, + "visdom.extra_deps": extra_deps, + "visdom.extra_deps.bhtsne": package, + "visdom.extra_deps.bhtsne.bhtsne": leaf, + } + return overrides, leaf + + +SQUARE = [[0.0, 0.0], [1.0, 1.0], [2.0, 2.0]] + + +# -- _get_perplexity --------------------------------------------------------- + + +@pytest.mark.parametrize( + "num_entities,expected", + [ + (500, 50), # large: capped at the base of 50 + (200, 50), + (150, 49), # the (n - 1) // 3 clamp bites just below 151 + (90, 29), + (60, 19), + (21, 6), + (20, 6), # below 21 the base is 7, but the clamp still wins + (10, 3), + (5, 1), + (1, 1), # never drops below 1, however small the input + (0, 1), + ], +) +def test_perplexity_table(num_entities, expected): + """Perplexity follows the size bands, then the (n - 1) // 3 clamp.""" + assert _get_perplexity(num_entities) == expected + + +@pytest.mark.parametrize("num_entities", [0, 1, 2, 7, 25, 100, 1000]) +def test_perplexity_stays_usable(num_entities): + """Perplexity is always at least 1 and never exceeds (n - 1) // 3.""" + perplexity = _get_perplexity(num_entities) + assert perplexity >= 1 + assert perplexity <= max(1, (num_entities - 1) // 3) + + +# -- _normalize_tsne --------------------------------------------------------- + + +def test_normalize_maps_extremes_to_the_unit_box(): + """The min point lands on (-1, -1) and the max on (1, 1).""" + result = _normalize_tsne(np.array(SQUARE)) + assert len(result) == 3 + assert result[0] == pytest.approx((-1.0, -1.0)) + assert result[2] == pytest.approx((1.0, 1.0)) + + +def test_normalize_bounds_both_axes(): + """Both axes span exactly [-1, 1] regardless of the input scale.""" + xs, ys = zip(*_normalize_tsne(np.random.rand(50, 2) * 100)) + assert min(xs) == pytest.approx(-1.0) + assert max(xs) == pytest.approx(1.0) + assert min(ys) == pytest.approx(-1.0) + assert max(ys) == pytest.approx(1.0) + + +def test_normalize_handles_a_degenerate_axis(): + """A zero-range axis collapses to 0, not to a division-by-zero NaN.""" + result = _normalize_tsne(np.array([[5.0, 0.0], [5.0, 1.0], [5.0, 2.0]])) + xs, ys = zip(*result) + assert xs == pytest.approx((0.0, 0.0, 0.0)) + assert ys == pytest.approx((-1.0, 0.0, 1.0)) + + +def test_normalize_accepts_a_plain_list(): + """A nested list is coerced, so callers need not pass an ndarray.""" + assert _normalize_tsne(SQUARE)[0] == pytest.approx((-1.0, -1.0)) + + +# -- backend selection ------------------------------------------------------- + + +def test_opentsne_is_preferred(reloaded_visdom): + """With openTSNE importable it is used, and it drives the perplexity.""" + fake = _fake_opentsne(SQUARE) + module = reloaded_visdom({"openTSNE": fake}) + + result = module.do_tsne(np.random.rand(30, 10).astype(np.float32)) + + fake.TSNE.assert_called_once() + assert fake.TSNE.call_args.kwargs["n_components"] == 2 + assert fake.TSNE.call_args.kwargs["perplexity"] == _get_perplexity(30) + fake.TSNE.return_value.fit.assert_called_once() + assert result[0] == pytest.approx((-1.0, -1.0)) + + +def test_falls_back_to_bhtsne(reloaded_visdom): + """With openTSNE missing, bhtsne is used and receives the input dimensions.""" + overrides, leaf = _fake_bhtsne(SQUARE) + module = reloaded_visdom(overrides) + + X = np.random.rand(30, 10).astype(np.float32) + result = module.do_tsne(X) + + leaf.run_bh_tsne.assert_called_once() + assert leaf.run_bh_tsne.call_args.kwargs["initial_dims"] == 10 + assert leaf.run_bh_tsne.call_args.kwargs["perplexity"] == _get_perplexity(30) + assert result[2] == pytest.approx((1.0, 1.0)) + + +def test_error_names_both_backends(reloaded_visdom): + """With neither backend importable, the error points at both of them.""" + module = reloaded_visdom( + { + "openTSNE": None, + "visdom.extra_deps": None, + "visdom.extra_deps.bhtsne": None, + "visdom.extra_deps.bhtsne.bhtsne": None, + } + ) + + with pytest.raises(Exception) as excinfo: + module.do_tsne(np.random.rand(10, 5)) + + message = str(excinfo.value) + assert "openTSNE" in message + assert "bhtsne" in message diff --git a/py/tests/unit/window_builder.py b/py/tests/unit/window_builder.py new file mode 100644 index 000000000..a0b2d5038 --- /dev/null +++ b/py/tests/unit/window_builder.py @@ -0,0 +1,327 @@ +"""Unit tests for the pane construction helpers in ``server_utils``. + +``window()`` is the single dispatch point that turns an ``/events`` payload into +the pane dict the frontend renders, and ``update_window()`` is how every +subsequent ``/update`` mutates it. Both are pure, and both were previously +covered only indirectly through HTTP tests. +""" + +import pytest + +from visdom.utils.server_utils import update_window, window + +from testutils.payloads import content_args, plot_data, window_args + + +def test_command_is_window(): + assert window(window_args())["command"] == "window" + + +def test_version_defaults_to_one(): + assert window(window_args())["version"] == 1 + + +def test_version_is_taken_from_args(): + assert window(window_args(version=7))["version"] == 7 + + +def test_supplied_win_is_used_as_id(): + assert window(window_args(win="my_win"))["id"] == "my_win" + + +def test_id_is_stringified(): + assert window(window_args(win=42))["id"] == "42" + + +@pytest.mark.parametrize( + "set_win_to_none", [False, True], ids=["absent", "explicit-none"] +) +def test_missing_win_generates_an_id(set_win_to_none): + args = window_args() + if set_win_to_none: + args["win"] = None + assert window(args)["id"].startswith("window_") + + +def test_generated_ids_are_unique(): + assert window(window_args())["id"] != window(window_args())["id"] + + +def test_content_id_distinguishes_rebuilds(): + args = window_args(win="stable") + assert window(args)["contentID"] != window(args)["contentID"] + + +def test_opts_populate_presentation_fields(): + p = window( + window_args( + opts={ + "title": "Loss", + "width": 300, + "height": 200, + "comment": "run 3", + "inflate": False, + } + ) + ) + assert p["title"] == "Loss" + assert p["width"] == 300 + assert p["height"] == 200 + assert p["comment"] == "run 3" + assert p["inflate"] is False + + +@pytest.mark.parametrize( + "field,expected", + [ + ("title", ""), + ("comment", ""), + ("inflate", True), + ("width", None), + ("height", None), + ], +) +def test_presentation_defaults(field, expected): + assert window(window_args())[field] == expected + + +def test_type_is_plot(): + assert window(window_args())["type"] == "plot" + + +def test_data_and_layout_are_nested_under_content(): + traces = [plot_data(), plot_data(trace_type="bar")] + layout = {"title": "t", "showlegend": True} + p = window(window_args(data=traces, layout=layout)) + assert p["content"]["data"] == traces + assert p["content"]["layout"] == layout + + +def test_caption_defaults_to_none(): + assert window(window_args())["content"]["caption"] is None + + +def test_caption_comes_from_opts(): + assert ( + window(window_args(opts={"caption": "fig 1"}))["content"]["caption"] == "fig 1" + ) + + +def test_unknown_trace_type_is_still_a_plot(): + assert window(window_args(data=[{"type": "sunburst"}]))["type"] == "plot" + + +def test_named_type_without_content_is_not_special_cased(): + """``is_visdom_type`` keys off ``content``, not off the type name.""" + p = window(window_args(data=[{"type": "text"}])) + assert p["type"] == "plot" + assert p["content"]["data"] == [{"type": "text"}] + + +@pytest.mark.parametrize( + "ptype,content", + [ + ("image_history", "img_0"), + ("plot_history", {"data": [plot_data()], "layout": {}}), + ], +) +def test_history_content_is_wrapped_in_a_slider_list(ptype, content): + p = window(content_args(ptype, content)) + assert p["type"] == ptype + assert p["content"] == [content] + assert p["selected"] == 0 + + +def test_slider_shown_by_default(): + assert window(content_args("image_history", "i"))["show_slider"] is True + + +def test_slider_can_be_hidden(): + p = window(content_args("image_history", "i", opts={"show_slider": False})) + assert p["show_slider"] is False + + +@pytest.mark.parametrize( + "ptype,content", + [ + ("text", "hello"), + ("text", "bold & italic"), + ("image", "data:image/png;base64,AAA"), + ("properties", [{"type": "number", "name": "lr", "value": "0.1"}]), + ], +) +def test_simple_content_is_stored_verbatim(ptype, content): + p = window(content_args(ptype, content)) + assert p["type"] == ptype + assert p["content"] == content + + +def test_no_slider_keys_on_simple_panes(): + p = window(content_args("text", "hello")) + assert "selected" not in p + assert "show_slider" not in p + + +@pytest.fixture +def network_content(): + return {"nodes": [{"id": 0}], "edges": []} + + +def test_network_type_and_content(network_content): + p = window(content_args("network", network_content)) + assert p["type"] == "network" + assert p["content"] == network_content + + +def test_network_label_and_direction_defaults(network_content): + p = window(content_args("network", network_content)) + assert p["directed"] is False + assert p["showEdgeLabels"] == "hover" + assert p["showVertexLabels"] == "hover" + + +def test_network_opts_override_defaults(network_content): + p = window( + content_args( + "network", + network_content, + opts={ + "directed": True, + "showEdgeLabels": True, + "showVertexLabels": False, + }, + ) + ) + assert p["directed"] is True + assert p["showEdgeLabels"] is True + assert p["showVertexLabels"] is False + + +@pytest.fixture +def embeddings_content(): + return {"data": [[0.0, 1.0]], "labels": ["a"]} + + +def test_embeddings_type_and_content(embeddings_content): + p = window(content_args("embeddings", embeddings_content)) + assert p["type"] == "embeddings" + assert p["content"]["data"] == [[0.0, 1.0]] + + +def test_embeddings_history_starts_empty(embeddings_content): + assert window(content_args("embeddings", embeddings_content))["old_content"] == [] + + +@pytest.mark.parametrize("stale_flag", [False, True]) +def test_embeddings_has_previous_is_always_reset(embeddings_content, stale_flag): + """A stale flag from the client must not survive pane construction.""" + content = dict(embeddings_content, has_previous=stale_flag) + assert ( + window(content_args("embeddings", content))["content"]["has_previous"] is False + ) + + +@pytest.fixture +def plot_pane(): + return window(window_args(layout={"title": "old", "showlegend": False})) + + +def test_layout_keys_are_merged(plot_pane): + update_window(plot_pane, {"layout": {"title": "new"}}) + assert plot_pane["content"]["layout"]["title"] == "new" + + +def test_unmentioned_layout_keys_survive(plot_pane): + update_window(plot_pane, {"layout": {"title": "new"}}) + assert plot_pane["content"]["layout"]["showlegend"] is False + + +def test_new_layout_keys_are_added(plot_pane): + update_window(plot_pane, {"layout": {"xaxis": {"type": "log"}}}) + assert plot_pane["content"]["layout"]["xaxis"] == {"type": "log"} + + +@pytest.mark.parametrize("args", [{"layout": {"title": None}}, {}]) +def test_layout_is_left_alone(plot_pane, args): + update_window(plot_pane, args) + assert plot_pane["content"]["layout"]["title"] == "old" + + +@pytest.fixture +def titled_pane(): + return window(window_args(opts={"title": "old"})) + + +def test_opts_are_written_to_the_pane_root(titled_pane): + update_window(titled_pane, {"opts": {"title": "new", "width": 500}}) + assert titled_pane["title"] == "new" + assert titled_pane["width"] == 500 + + +def test_none_opt_values_are_ignored(titled_pane): + update_window(titled_pane, {"opts": {"title": None}}) + assert titled_pane["title"] == "old" + + +def test_caption_is_routed_into_content(titled_pane): + """``caption`` lives beside the plot data, not at the pane root.""" + update_window(titled_pane, {"opts": {"caption": "fig 2"}}) + assert titled_pane["content"]["caption"] == "fig 2" + assert "caption" not in {k: v for k, v in titled_pane.items() if k != "content"} + + +def test_caption_is_skipped_when_content_is_not_a_dict(): + text_pane = window(content_args("text", "hello")) + update_window(text_pane, {"opts": {"caption": "ignored"}}) + assert text_pane["content"] == "hello" + + +def test_version_is_bumped_once_per_update(titled_pane): + update_window(titled_pane, {"opts": {"title": "a"}}) + assert titled_pane["version"] == 2 + update_window(titled_pane, {"opts": {"title": "b"}}) + assert titled_pane["version"] == 3 + + +def test_version_is_bumped_even_for_an_empty_update(titled_pane): + update_window(titled_pane, {}) + assert titled_pane["version"] == 2 + + +def test_returns_the_same_pane_object(titled_pane): + assert update_window(titled_pane, {}) is titled_pane + + +@pytest.fixture +def two_trace_pane(): + return window(window_args(data=[plot_data(name="train"), plot_data(name="val")])) + + +def trace_names(pane): + return [d.get("name") for d in pane["content"]["data"]] + + +@pytest.mark.parametrize( + "legend,expected", + [ + (["a", "b"], ["a", "b"]), + (["a"], ["a", "val"]), + (["a", "b", "c"], ["a", "b"]), + ], +) +def test_legend_renames_traces_positionally(two_trace_pane, legend, expected): + update_window(two_trace_pane, {"opts": {"legend": legend}}) + assert trace_names(two_trace_pane) == expected + + +@pytest.mark.parametrize( + "name,legend,expected", + [ + ("val", ["renamed"], ["train", "renamed"]), + ("val", [], ["train", "val"]), + ("missing", ["x"], ["train", "val"]), + ], +) +def test_named_update_targets_one_trace(two_trace_pane, name, legend, expected): + update_window(two_trace_pane, {"name": name, "opts": {"legend": legend}}) + assert trace_names(two_trace_pane) == expected diff --git a/py/visdom/data_model/json_store.py b/py/visdom/data_model/json_store.py index 75aec6f46..d751c5f46 100644 --- a/py/visdom/data_model/json_store.py +++ b/py/visdom/data_model/json_store.py @@ -98,6 +98,21 @@ def save_envs(self, state, eids): written.append(eid) return written + def _atomic_write(self, path, payload): + """Write ``payload`` to ``path`` via a temporary file and one rename. + + Writing straight to ``path`` truncates the previous contents before the + new ones are complete, so an interrupted write leaves a half-file that + :meth:`load_env` cannot parse and silently reports as an empty + environment. Staging into ``.tmp`` and calling :func:`os.replace` + keeps the old file readable until the new one is whole. This mirrors + :meth:`save_undo`. + """ + tmp = path + ".tmp" + with open(tmp, "w") as fn: + fn.write(payload) + os.replace(tmp, path) + def serialize_env(self, eid, env_data): """Write one environment to disk; return ``True`` if written. @@ -108,6 +123,11 @@ def serialize_env(self, eid, env_data): :meth:`_hash_path` so it agrees with load/delete/exists on the file a given ``eid`` maps to; over-long ids fall back to ``hash_.json`` with the real id kept in a ``name`` field. + + The write itself is atomic (see :meth:`_atomic_write`), so a crash + part-way through cannot destroy the environment already on disk. The + staging file carries a ``.tmp`` suffix rather than ``.json``, so a + stranded one is never mistaken for an environment by :meth:`list_envs`. """ if isinstance(env_data, LazyEnvData): if env_data._raw_dict is None: @@ -121,15 +141,15 @@ def serialize_env(self, eid, env_data): try: if primary is None: raise OSError(errno.ENAMETOOLONG, "env id maps outside env_path") - with open(primary, "w") as fn: - fn.write(json.dumps(payload, cls=NanSafeEncoder)) + self._atomic_write(primary, json.dumps(payload, cls=NanSafeEncoder)) except OSError as e: if e.errno != errno.ENAMETOOLONG and getattr(e, "winerror", None) != 206: raise data_to_save = copy.deepcopy(payload) data_to_save["name"] = self._safe_eid(eid) - with open(self._hash_path(eid), "w") as fn: - fn.write(json.dumps(data_to_save, cls=NanSafeEncoder)) + self._atomic_write( + self._hash_path(eid), json.dumps(data_to_save, cls=NanSafeEncoder) + ) return True def save_all(self, state): @@ -269,18 +289,12 @@ def save_undo(self, eid, stack): payload = json.dumps(stack, cls=NanSafeEncoder) try: target = plain - tmp = plain + ".tmp" - with open(tmp, "w") as fn: - fn.write(payload) - os.replace(tmp, plain) + self._atomic_write(plain, payload) except OSError as e: if e.errno != errno.ENAMETOOLONG and getattr(e, "winerror", None) != 206: raise target = hashed - tmp = hashed + ".tmp" - with open(tmp, "w") as fn: - fn.write(payload) - os.replace(tmp, hashed) + self._atomic_write(hashed, payload) return target def clear_undo(self, eid): diff --git a/py/visdom/server/app.py b/py/visdom/server/app.py index 1bf32b717..16afd7b73 100644 --- a/py/visdom/server/app.py +++ b/py/visdom/server/app.py @@ -52,6 +52,7 @@ DEFAULT_HOSTNAME, DEFAULT_MAX_IMAGE_HISTORY, DEFAULT_MAX_OLD_CONTENT, + DEFAULT_MAX_PLOT_HISTORY, DEFAULT_MAX_TEXT_LINES, DEFAULT_PORT, ) @@ -80,6 +81,7 @@ def __init__( self.eager_data_loading = eager_data_loading self.max_image_history = DEFAULT_MAX_IMAGE_HISTORY self.max_old_content = DEFAULT_MAX_OLD_CONTENT + self.max_plot_history = DEFAULT_MAX_PLOT_HISTORY self.max_text_lines = DEFAULT_MAX_TEXT_LINES self.env_path = env_path self.storage = JSONStore(env_path) @@ -102,7 +104,14 @@ def __init__( tornado_settings["cookie_secret"] = fn.read() tornado_settings["static_url_prefix"] = self.base_url + "/static/" - tornado_settings["debug"] = True + # A traceback and the raw request are debugging aids, not something to + # hand to whoever provoked the error. `debug` was forced on for every + # server, which put both on the 500 page -- and, being tornado's debug + # flag, also turned on autoreload. Follow the operator's logging level + # instead, and keep the two concerns separate. + tornado_settings["show_error_details"] = logging.getLogger().isEnabledFor( + logging.DEBUG + ) handlers = [ (r"%s/events" % self.base_url, PostHandler, {"app": self}), (r"%s/update" % self.base_url, UpdateHandler, {"app": self}), diff --git a/py/visdom/server/defaults.py b/py/visdom/server/defaults.py index f131acc6d..dc31add10 100644 --- a/py/visdom/server/defaults.py +++ b/py/visdom/server/defaults.py @@ -16,6 +16,7 @@ MAX_SOCKET_WAIT = 15 DEFAULT_MAX_IMAGE_HISTORY = 4 DEFAULT_MAX_OLD_CONTENT = 50 +DEFAULT_MAX_PLOT_HISTORY = 4 DEFAULT_MAX_TEXT_LINES = 500 DEFAULT_MAX_UNDO_HISTORY = 4 UNDO_DIRNAME = ".undo" diff --git a/py/visdom/server/handlers/base_handlers.py b/py/visdom/server/handlers/base_handlers.py index 310797048..ab9ce3abb 100644 --- a/py/visdom/server/handlers/base_handlers.py +++ b/py/visdom/server/handlers/base_handlers.py @@ -30,9 +30,11 @@ ) _WEB_APP_ATTRIBUTES = _COMMON_APP_ATTRIBUTES + ( + "readonly", "max_text_lines", "max_old_content", "max_image_history", + "max_plot_history", ) _SOCKET_APP_ATTRIBUTES = _COMMON_APP_ATTRIBUTES + ("readonly",) @@ -119,7 +121,7 @@ def write_error(self, status_code, **kwargs): logging.info( "Traceback: {}".format(traceback.format_exception(*kwargs["exc_info"])) ) - debug = self.settings.get("debug") + show_details = self.settings.get("show_error_details", False) title = http.client.responses.get(status_code, "Unknown Error") logging.error("rendering error page") exc_info = kwargs["exc_info"] @@ -129,11 +131,11 @@ def write_error(self, status_code, **kwargs): # 3. The traceback object try: params = { - "error": exc_info[1] if debug else None, + "error": exc_info[1] if show_details else None, "trace_info": ( - traceback.format_exception(*exc_info) if debug else None + traceback.format_exception(*exc_info) if show_details else None ), - "request": self.request.__dict__ if debug else None, + "request": self.request.__dict__ if show_details else None, "status_code": status_code, "title": title, } diff --git a/py/visdom/server/handlers/socket_handlers.py b/py/visdom/server/handlers/socket_handlers.py index 1447d087b..f5153f5fc 100644 --- a/py/visdom/server/handlers/socket_handlers.py +++ b/py/visdom/server/handlers/socket_handlers.py @@ -76,8 +76,24 @@ def open(self, register_to="sources"): self.eid = "main" register_list[self.sid] = self - def broadcast_layouts(self): - raise ValueError("Should be replaced in child class") + def broadcast_layouts(self, target_subs=None): + """Push the saved layouts to subscribers. + + Lives on the base class because ``save_layouts`` is handled here, for + every kind of socket: a source connection sending it used to reach an + override that only subscriber sockets had, and raise ``ValueError`` out + of the message loop. Layouts are a view concern either way, so the + recipients are always the subscribers. + """ + if target_subs is None: + target_subs = self.subs.values() + for sub in target_subs: + sub.write_message( + json.dumps( + {"command": "layout_update", "data": self.app.layouts}, + cls=NanSafeEncoder, + ) + ) def on_message(self, message): logging.info(f"from visdom client: {message}") @@ -93,13 +109,14 @@ def on_message(self, message): eid = escape_eid(msg["eid"]) if eid not in self.state: return + # One pop, under the escaped id. Popping a second time under the + # raw id used to blank out p_data before the event was built, so + # sources always saw pane_data: None -- and when the raw id was + # not itself a key in state, the lookup returned None and the + # close was never announced at all. p_data = self.state[eid]["jsons"].pop(msg["data"], None) if p_data is not None: push_deleted(self.storage, eid, msg["data"], p_data) - env = self.state.get(msg["eid"]) - if env is None: - return - p_data = env["jsons"].pop(msg["data"], None) event = { "event_type": "close", "target": msg["data"], @@ -249,9 +266,13 @@ def on_message(self, message): if p.get("type") == "plot_history": content_list = p.get("content") + # The range check has to come after a type check: a string or + # list frame from a client otherwise raises TypeError out of the + # message loop. bool is excluded because True would index as 1. if ( not isinstance(content_list, list) - or frame is None + or not isinstance(frame, int) + or isinstance(frame, bool) or not (0 <= frame < len(content_list)) ): logging.warning( @@ -351,9 +372,19 @@ def on_message(self, message): ) return p = env["jsons"][win] + old_content = p.get("old_content") + if not old_content: + # Nothing left to drill back to. Popping regardless raised + # IndexError (or KeyError, for a pane that never had a history) + # straight out of the socket's message callback. + logging.warning( + f"pop_embeddings_pane: pane {win!r} in env {eid!r} has no" + f" previous content, dropping event" + ) + return p["content"]["selected"] = None - p["content"]["data"] = p["old_content"].pop() - if len(p["old_content"]) == 0: + p["content"]["data"] = old_content.pop() + if len(old_content) == 0: p["content"]["has_previous"] = False p["contentID"] = get_rand_id() # Attach eid so the frontend can filter stale messages after env switch. @@ -488,17 +519,6 @@ def open(self): self.broadcast_layouts([self]) broadcast_envs(self, [self]) - def broadcast_layouts(self, target_subs=None): - if target_subs is None: - target_subs = self.subs.values() - for sub in target_subs: - sub.write_message( - json.dumps( - {"command": "layout_update", "data": self.app.layouts}, - cls=NanSafeEncoder, - ) - ) - def initialize(self, app): super().initialize(app) self.broadcast_layouts() @@ -570,6 +590,11 @@ def post(self): if BaseWrapper == VisSocketWrapper and sid is None: new_sub = VisSocketWrapper() + # open() logs the peer it is mocking a socket for, so the + # wrapper needs a request the same way the subscriber path + # below gives it one. Without it every polling client raised + # AttributeError here and never got a sid back. + new_sub.request = self.request new_sub.initialize(self.app) self.write(json.dumps({"success": True, "sid": new_sub.sid})) return diff --git a/py/visdom/server/handlers/web_handlers.py b/py/visdom/server/handlers/web_handlers.py index e2845a7e7..96c881fce 100644 --- a/py/visdom/server/handlers/web_handlers.py +++ b/py/visdom/server/handlers/web_handlers.py @@ -15,10 +15,10 @@ import copy import getpass +import hmac import json import jsonpatch import logging -import math import os import uuid from collections import OrderedDict @@ -29,10 +29,13 @@ from visdom.utils.shared_utils import ( get_rand_id, _coerce_image_slider_index, + _is_missing_value, NanSafeEncoder, ) from visdom.utils.server_utils import ( check_auth, + check_readonly, + reject_readonly, extract_eid, window, register_window, @@ -67,6 +70,7 @@ class PostHandler(BaseHandler): @check_auth + @check_readonly def post(self): req = tornado.escape.json_decode( tornado.escape.to_basestring(self.request.body) @@ -107,7 +111,9 @@ def post(self): class UpdateHandler(BaseHandler): @staticmethod - def update_packet(p, args, max_text_lines, max_old_content, max_image_history): + def update_packet( + p, args, max_text_lines, max_old_content, max_image_history, max_plot_history + ): # Shallow copy the packet to dynamically capture changes to top-level keys. old_p = p.copy() @@ -118,7 +124,12 @@ def update_packet(p, args, max_text_lines, max_old_content, max_image_history): old_p["old_content"] = copy.deepcopy(p["old_content"]) p = UpdateHandler.update( - p, args, max_text_lines, max_old_content, max_image_history + p, + args, + max_text_lines, + max_old_content, + max_image_history, + max_plot_history, ) p["contentID"] = get_rand_id() @@ -159,7 +170,9 @@ def update_embeddings_packet(p, args, max_old_content): return [] @staticmethod - def update(p, args, max_text_lines, max_old_content, max_image_history): + def update( + p, args, max_text_lines, max_old_content, max_image_history, max_plot_history + ): # Update text in window, separated by a line break if p["type"] == "text": p["content"] += "
" + args["data"][0]["content"] @@ -185,6 +198,8 @@ def update(p, args, max_text_lines, max_old_content, max_image_history): utype = args["data"][0]["type"] if utype == "plot_history": p["content"].append(args["data"][0]["content"]) + if len(p["content"]) > max_plot_history: + p["content"] = p["content"][-max_plot_history:] p["selected"] = len(p["content"]) - 1 elif utype == "plot_update_selected": selected = args["data"][0]["selected"] @@ -198,18 +213,24 @@ def update(p, args, max_text_lines, max_old_content, max_image_history): new_data = args.get("data") p = update_window(p, args) name = args.get("name") - if name is None and new_data is None: + delete = args.get("delete") + # a heatmap removal names no trace and carries no data, so ask about the + # deletion before reading either as "nothing to do" + if not delete and name is None and not new_data: return p # we only updated the opts or layout append = args.get("append") idxs = list(range(len(pdata))) if name is not None: - assert len(new_data) == 1 or args.get("delete") + if not delete and len(new_data) != 1: + raise tornado.web.HTTPError( + 400, reason="a named trace update takes exactly one data entry" + ) idxs = [i for i in idxs if pdata[i]["name"] == name] # Delete a trace - if args.get("delete"): + if delete: idxs_set = set(idxs) p["content"]["data"] = [e for i, e in enumerate(pdata) if i not in idxs_set] return p @@ -303,23 +324,17 @@ def update(p, args, max_text_lines, max_old_content, max_image_history): return p - # inject new trace + # inject new trace; the plot may hold none at all if every trace of it + # has been deleted if len(idxs) == 0: - idx = len(pdata) - pdata.append(dict(pdata[0])) # plot is not empty, clone an entry - idxs = [idx] - append = False - pdata[idx] = new_data[0] - for k, v in new_data[0].items(): - pdata[idx][k] = v - pdata[idx]["name"] = name + trace = dict(new_data[0]) + trace["name"] = name + pdata.append(trace) return p - # Update traces - for n, idx in enumerate(idxs): - if all( - i is None or math.isnan(i) or math.isinf(i) for i in new_data[n]["x"] - ): + # Update traces, as far as the update supplies them + for idx, new_trace in zip(idxs, new_data): + if all(_is_missing_value(i) for i in new_trace["x"]): continue # handle data for plotting axes = ["x", "y"] @@ -327,26 +342,24 @@ def update(p, args, max_text_lines, max_old_content, max_image_history): axes.append("z") for axis in axes: pdata[idx][axis] = ( - (pdata[idx][axis] + new_data[n][axis]) - if append - else new_data[n][axis] + (pdata[idx][axis] + new_trace[axis]) if append else new_trace[axis] ) # handle marker properties - if "marker" not in new_data[n]: + if "marker" not in new_trace: continue if "marker" not in pdata[idx]: pdata[idx]["marker"] = {} pdata_marker = pdata[idx]["marker"] for marker_prop in ["color"]: - if marker_prop not in new_data[n]["marker"]: + if marker_prop not in new_trace["marker"]: continue if marker_prop not in pdata[idx]["marker"]: pdata[idx]["marker"][marker_prop] = [] pdata_marker[marker_prop] = ( - (pdata_marker[marker_prop] + new_data[n]["marker"][marker_prop]) + (pdata_marker[marker_prop] + new_trace["marker"][marker_prop]) if append - else new_data[n]["marker"][marker_prop] + else new_trace["marker"][marker_prop] ) return p @@ -436,6 +449,7 @@ def wrap_func(handler, args): handler.max_text_lines, handler.max_old_content, handler.max_image_history, + handler.max_plot_history, ) except (TypeError, ValueError) as exc: if is_image_slider_update: @@ -453,6 +467,7 @@ def wrap_func(handler, args): handler.write(p["id"]) @check_auth + @check_readonly def post(self): if self.login_enabled and not self.current_user: self.set_status(400) @@ -477,6 +492,7 @@ def wrap_func(handler, args): broadcast(handler, json.dumps({"command": "close", "data": win}), eid) @check_auth + @check_readonly def post(self): args = tornado.escape.json_decode( tornado.escape.to_basestring(self.request.body) @@ -498,6 +514,7 @@ def wrap_func(handler, args): broadcast_envs(handler) @check_auth + @check_readonly def post(self): args = tornado.escape.json_decode( tornado.escape.to_basestring(self.request.body) @@ -534,7 +551,10 @@ def wrap_func(handler, args): prev_eid = escape_eid(args.get("prev_eid")) eid = escape_eid(args.get("eid")) - assert prev_eid in handler.state, "env to be forked doesn't exist" + if prev_eid not in handler.state: + # the eid stays out of the reason: it is echoed on the status line, + # which is latin-1 only, and eids are free-form unicode. + raise tornado.web.HTTPError(400, reason="env to be forked doesn't exist") handler.state[eid] = copy.deepcopy(handler.state[prev_eid]) handler.storage.save_env(eid, handler.state[eid]) @@ -543,6 +563,7 @@ def wrap_func(handler, args): handler.write(eid) @check_auth + @check_readonly def post(self): args = tornado.escape.json_decode( tornado.escape.to_basestring(self.request.body) @@ -647,6 +668,7 @@ def wrap_func(handler, args): handler.write(json.dumps(ret)) @check_auth + @check_readonly def post(self): args = tornado.escape.json_decode( tornado.escape.to_basestring(self.request.body) @@ -660,7 +682,12 @@ def wrap_func(handler, args): eid = extract_eid(args) if "data" in args: - # Load data from client + # Load data from client. This is the one write behind an endpoint + # that also reads, so it cannot be refused by the decorator. + if handler.readonly: + reject_readonly(handler) + return + data = json.loads(args["data"]) if eid not in handler.state: @@ -679,9 +706,10 @@ def wrap_func(handler, args): json.dumps(handler.state[eid]["jsons"], cls=NanSafeEncoder) ) else: - assert ( - args["win"] in handler.state[eid]["jsons"] - ), "Window {} doesn't exist in env {}".format(args["win"], eid) + if args["win"] not in handler.state[eid]["jsons"]: + raise tornado.web.HTTPError( + 400, reason="window doesn't exist in this env" + ) handler.write( json.dumps( handler.state[eid]["jsons"][args["win"]], cls=NanSafeEncoder @@ -734,7 +762,19 @@ def post(self, arg, **kwargs): salt = stored.split("$")[0] password = hash_password(json_obj["password"], salt=salt) - if (username == self.user_credential["username"]) and (password == stored): + # Constant-time comparison: `==` on the derived key returns as soon as + # two characters differ, so response timing tells an attacker how much + # of a guess was right. Both halves are always compared, which keeps a + # wrong username costing the same as a wrong password. + username_ok = hmac.compare_digest( + str(username).encode("utf-8"), + self.user_credential["username"].encode("utf-8"), + ) + password_ok = hmac.compare_digest( + password.encode("utf-8"), stored.encode("utf-8") + ) + + if username_ok and password_ok: self.set_secure_cookie("user_password", username + password) else: self.set_status(400) @@ -766,10 +806,6 @@ def get(self, text): class UploadEnvHandler(BaseHandler): - def initialize(self, app): - super().initialize(app) - self.readonly = app.readonly - @check_auth def post(self): # 100mb file size limit @@ -864,10 +900,6 @@ class ExperimentLogHandler(BaseHandler): VALID_ACTIONS = ("log", "metrics", "finish") - def initialize(self, app): - super().initialize(app) - self.readonly = app.readonly - @staticmethod def _require_mapping(args, field): """Return ``args[field]`` if it is a mapping (or absent); else raise 400.""" diff --git a/py/visdom/utils/server_utils.py b/py/visdom/utils/server_utils.py index 150a2c9d0..5e30ebec8 100644 --- a/py/visdom/utils/server_utils.py +++ b/py/visdom/utils/server_utils.py @@ -57,6 +57,32 @@ def _check_auth(handler, *args, **kwargs): return _check_auth +def reject_readonly(handler): + """Answer 403 for a write attempted against a readonly server.""" + handler.set_status(403) + handler.write({"success": False, "error": "The server is running in readonly mode"}) + + +def check_readonly(f): + """ + Wrapper for handler methods that change server state, so a server + started with ``-readonly`` refuses them instead of applying them. + + Sockets are already short-circuited wholesale in + ``AnySocketHandlerOrWrapper.on_message``; this is the HTTP half of the + same rule. Stack it under ``check_auth`` so an unauthenticated request + still answers 401 rather than 403. + """ + + def _check_readonly(handler, *args, **kwargs): + if handler.readonly: + reject_readonly(handler) + return + f(handler, *args, **kwargs) + + return _check_readonly + + def set_cookie(value=None): """Create cookie secret key for authentication""" if value is not None: @@ -585,7 +611,9 @@ def register_window(self, p, eid): p["i"] = env[p["id"]]["i"] p["comment"] = env[p["id"]].get("comment", p.get("comment", "")) else: - p["i"] = len(env) + # not len(env): closing any window but the last would hand the next + # one an index that is still in use. Same rule as the undo path. + p["i"] = max((w.get("i", -1) for w in env.values()), default=-1) + 1 env[p["id"]] = p diff --git a/py/visdom/utils/shared_utils.py b/py/visdom/utils/shared_utils.py index f40aa5f67..454dbb559 100644 --- a/py/visdom/utils/shared_utils.py +++ b/py/visdom/utils/shared_utils.py @@ -100,6 +100,21 @@ def _sanitize_nans(obj): return obj +def _is_missing_value(value): + """Whether a plotted coordinate carries no value. + + Clients mark gaps in a series with None, NaN or Inf. Only numbers can be + NaN or Inf: a categorical axis carries strings, and passing one to + ``math.isnan`` raises ``TypeError``, so anything non-numeric is present by + definition. + """ + if value is None: + return True + if isinstance(value, (float, int, np.floating, np.integer)): + return math.isnan(value) or math.isinf(value) + return False + + class NanSafeEncoder(json.JSONEncoder): """JSON encoder that converts NaN and Inf float values to None. diff --git a/pyproject.toml b/pyproject.toml index 4ef73e474..0ddef0065 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,12 +3,22 @@ # scripts in the repo root (test_*.py, test/) are ignored. testpaths = ["py/tests"] # The visdom package lives under py/ (see setup.py package_dir). Adding it to the -# path lets `import visdom` resolve without an editable install. -pythonpath = ["py"] -python_files = ["test_*.py"] +# path lets `import visdom` resolve without an editable install. py/tests is on +# the path so tests can `import testutils`; it deliberately has no __init__.py, +# because setup.py runs find_packages(where="py") and would otherwise ship a +# top-level `tests` package to users. +pythonpath = ["py", "py/tests"] +# Test modules live under py/tests/{unit,integration}, so the directory already +# says "test" and the files do not repeat it. Shared helpers are importable, not +# collectable, so testutils is kept out of discovery. +python_files = ["*.py"] +norecursedirs = ["testutils", "__pycache__", ".*", "build", "dist", "*.egg"] python_classes = ["Test*"] python_functions = ["test_*"] addopts = "-ra -q" markers = [ - "server: test requires a running visdom server (deselect with -m 'not server')", + "server: test requires an externally launched visdom server (deselect with -m 'not server')", + "unit: pure logic, no Application and no I/O beyond tmp_path", + "integration: in-process Application, real HTTP or handler dispatch", + "slow: takes more than a couple of seconds", ] diff --git a/setup.py b/setup.py index d46c0f546..20ed39c18 100644 --- a/setup.py +++ b/setup.py @@ -65,7 +65,7 @@ def get_dist(pkgname): license="Apache-2.0", python_requires=">=3.12", # Package info - packages=find_packages(where="py"), + packages=find_packages(where="py", exclude=["tests", "tests.*"]), package_dir={"": "py"}, package_data={"visdom": ["static/*.*", "static/**/*", "py.typed", "*.pyi"]}, include_package_data=True,