Refactor the reference section of the docs #4311

tybug · 2025-03-18T07:08:45Z

Still a large pr but there's no sane way to split most of this without causing more work for me, thanks to all the inter-references. I could split the redirect extension off is probably the biggest thing

There's a minor amount of added or removed content here, but most of it is moving around existing stuff into their own page

hypothesis-python/docs/reference/django.rst

tybug · 2025-03-18T07:13:19Z

hypothesis-python/docs/reference/reproducing-failures.rst

@@ -1,19 +1,8 @@
-====================
 Reproducing failures
 ====================


My heading ordering is single =, single -, single ~ (and if you need to go beyond that you're probably doomed anyway). I try to avoid double headers for anything, but I think I might be the odd one out here, because e.g. the python rst guide recommends double headers for chapters. In defense of single headers, the python docs have way more structure than we do!

hypothesis-python/docs/reference/strategies.rst

jobh · 2025-03-18T09:21:37Z

hypothesis-python/docs/reference/targeted.rst

+      ab_c = (a + b) + c
+      a_bc = a + (b + c)
+      difference = abs(ab_c - a_bc)
+      target(difference)  # Without this, the test almost always passes


As fas as I can tell, target makes little difference here. Around 40% discover rate with or without it.

same..I'm not going to follow up on this right now, but I've added it as a TODO to #4309

Thanks! A comment for that time (ignore for this PR) is that target likely doesn't work well here because difference is constant for large regions of the input space.

We might want to comment that the target should be fairly smooth, not discrete-valued, and not added willy-nilly as it may actually reduce bug-finding power with the default example budget (as indicated below).

For background, the optimiser seems to get more stuck than I'd expect in these flat regions. E.g.,

@given(st.floats()) @settings(database=None, phases=[Phase.generate, Phase.target]) def test_target(f): target(f > 1e5) print(f)

doesn't vary f very much inside the True region. Fair enough, but perhaps worth noting.

This is great. I think target used to be better at this on the bytestring since a continuous in bytes could easily lead to discontinuous change in the output, which is less likely on the typed choice sequence. We might have to reintroduce a simulated annealing-style approach for each of the 5 primitive types.

Zac-HD · 2025-03-19T04:35:19Z

Quick thoughts again:

Yeah, this is going to be about at our size limit, but I agree that chopping it up more wouldn't really be a net improvement.
IMO, searchability is really important for API reference docs. "land on page, cmd-f, done" is a key usecase that e.g. pytest serves well these days.
- If that's way too large, one page for all of the strategies, and one for everything else?
- Some of the stuff currently in reference docs will presumably move to explanations or how-tos later.
Makes sense to switch from .. automodule:: ..., but we should then add a test that we document every function in the strategies module (or perhaps everything listed in any __all__ attribute?
Haven't run through all the detail yet, since I'm expecting files to move around again.

tybug · 2025-03-19T05:25:49Z

yeah it's a fair point, I'm on board with one big page for the api reference. It'll be a bit awkward until we move the more tutorial-style things out from the api reference, but still an improvement on the status quo.

tybug · 2025-03-20T03:57:54Z

so I know we said just one or two reference pages, but I was working through this and I think there's a valid distinction between "api/coding reference" and "'meta' reference", like observability or the pytest plugin, which I've kept in separate pages for the moment. They don't feel right to me in an api reference page, because they don't have an in-library method exposed by Hypothesis.

tybug · 2025-03-20T04:00:13Z

hypothesis-python/docs/typing.rst

I think we will eventually want to move this page somewhere better ("how to write type hints for your strategies"?), or remove it entirely, but top level is fine during the transition

Agree on both counts. We could start it out as e.g./how-to/type-annotate-strategies so that we can keep the URL longer term; I think it can still be top-level in the TOC regardless of location.

tybug · 2025-03-21T00:13:35Z

I've added a test to check that all exported names are documented, as recommended. This turned up a bug in the current docs where hypothesis.extra.numpy.boolean_dtypes is not documented. ~~I couldn't figure out why, and since it's not a new issue I've added it as a todo item in the tracking list.~~ NVM, immediately after sending this I realized it was because boolean_dtypes didn't have a docstring attached. I've fixed that too here.

Zac-HD

More scattered thoughts, and appreciation for the ongoing work 🙏

hypothesis-python/docs/reference/control.rst

hypothesis-python/docs/reference/pytest-plugin.rst

Zac-HD · 2025-03-20T19:23:58Z

hypothesis-python/docs/typing.rst

Agree on both counts. We could start it out as e.g./how-to/type-annotate-strategies so that we can keep the URL longer term; I think it can still be top-level in the TOC regardless of location.

hypothesis-python/docs/conf.py

hypothesis-python/docs/reference/strategies.rst

Zac-HD · 2025-03-21T06:15:16Z

hypothesis-python/docs/reference/strategies.rst

+.. automodule:: xps
+   :members:
+        from_dtype,


Oooh... does your excellent new test check that this is complete? It's not an importable module, exactly...

hypothesis-python/docs/reference/index.rst

hypothesis-python/docs/reference/integrations.rst

Zac-HD · 2025-03-22T00:28:18Z

hypothesis-python/docs/reference/strategies.rst

+Strategy objects tell Hypothesis what types of inputs to generate. For instance, passing ``st.lists(st.integers(), min_size=1)`` to |@given| tells Hypothesis to generate lists of integers with at least one element.
+
+Strategies can be combined using :ref:`combinator strategies <combinators>`, or modified using |strategy.filter|, |strategy.map|, or |strategy.flatmap|.


Suggested change

Strategy objects tell Hypothesis what types of inputs to generate. For instance, passing ``st.lists(st.integers(), min_size=1)`` to |@given| tells Hypothesis to generate lists of integers with at least one element.

Strategies can be combined using :ref:`combinator strategies <combinators>`, or modified using |strategy.filter|, |strategy.map|, or |strategy.flatmap|.

Strategies are the way we describe values for |@given| to generate. For instance, ``st.lists(st.integers(), min_size=1)`` tells Hypothesis to generate lists of integers with at least one element.

This reference page lists all of Hypothesis' first-party functions which return a strategy; there are also many provided by <third-party libraries>(xref), and you can define your own too. Note that people often say "strategy" when they mean "function returning a strategy"; it's usually clear from context which they meant.

Strategies can be passed to many strategy functions as arguments, combined using :ref:`combinator strategies <combinators>`, and modified using |strategy.filter|, |strategy.map|, or |strategy.flatmap|.

Strategies can be passed to many strategy functions as arguments

I get the distinction you're making, and I think it's worth making, but this is a confusing sentence to parse imo. Bite the technical incorrectness and say "strategies can be passed to other strategies as arguments"?

I've applied a variant of this, but happy to tweak further

I do think we need to call out both the distinction and the common sloppy usage in this explanatory header, but don't care exactly how we do that.

hypothesis-python/docs/reference/strategies.rst

hypothesis-python/docs/reference/api.rst

Zac-HD

I think we're done with this part - obviously there are plenty of followups, but I feel good about shipping this as a single cohesive change for our users 😁

tybug commented Mar 18, 2025

View reviewed changes

hypothesis-python/docs/reference/strategies.rst Outdated Show resolved Hide resolved

tybug mentioned this pull request Mar 18, 2025

Documentation overhaul #4309

Open

36 tasks

jobh reviewed Mar 18, 2025

View reviewed changes

tybug force-pushed the docs-reference branch from 310cbbd to f63f0d3 Compare March 20, 2025 03:42

restructure reference pages

9f6998f

tybug force-pushed the docs-reference branch from f63f0d3 to 9f6998f Compare March 20, 2025 03:43

tybug commented Mar 20, 2025

View reviewed changes

add test for all exported names being documented

e9eefd6

tybug added 2 commits March 20, 2025 20:15

add a docstring to boolean_dtypes so it gets documented

eb1b899

add changelog

d4cee19

Zac-HD reviewed Mar 21, 2025

View reviewed changes

tybug added 2 commits March 21, 2025 20:01

address review comments

9b7203e

typo

50076b3

Zac-HD reviewed Mar 22, 2025

View reviewed changes

tybug added 6 commits March 21, 2025 22:47

address review

950e81b

wording tweaks

9468bda

update django docs

4a1b792

move provisional strategies to strings

fdf8173

format

5881fdf

move extras around

fbdddae

Zac-HD approved these changes Mar 22, 2025

View reviewed changes

Zac-HD enabled auto-merge March 22, 2025 08:16

Zac-HD force-pushed the docs-reference branch from 0390c6f to e8aa4fe Compare March 22, 2025 08:29

fix: artifact poisoning

d6ad796

Zac-HD force-pushed the docs-reference branch from e8aa4fe to fa7d27c Compare March 22, 2025 08:31

Zac-HD force-pushed the docs-reference branch 4 times, most recently from e6cfb7a to eb7aeb9 Compare March 22, 2025 09:38

tweak package config

f219633

Zac-HD force-pushed the docs-reference branch from eb7aeb9 to f219633 Compare March 22, 2025 09:47

Zac-HD merged commit ccb8ec2 into HypothesisWorks:master Mar 22, 2025
50 of 51 checks passed

tybug deleted the docs-reference branch March 22, 2025 16:40

		Strategy objects tell Hypothesis what types of inputs to generate. For instance, passing ``st.lists(st.integers(), min_size=1)`` to \|@given\| tells Hypothesis to generate lists of integers with at least one element.

		Strategies can be combined using :ref:`combinator strategies <combinators>`, or modified using \|strategy.filter\|, \|strategy.map\|, or \|strategy.flatmap\|.

-Strategy objects tell Hypothesis what types of inputs to generate. For instance, passing ``st.lists(st.integers(), min_size=1)`` to |@given| tells Hypothesis to generate lists of integers with at least one element.
-Strategies can be combined using :ref:`combinator strategies <combinators>`, or modified using |strategy.filter|, |strategy.map|, or |strategy.flatmap|.
+Strategies are the way we describe values for |@given| to generate.  For instance, ``st.lists(st.integers(), min_size=1)`` tells Hypothesis to generate lists of integers with at least one element.
+This reference page lists all of Hypothesis' first-party functions which return a strategy; there are also many provided by <third-party libraries>(xref), and you can define your own too.  Note that people often say "strategy" when they mean "function returning a strategy"; it's usually clear from context which they meant.
+Strategies can be passed to many strategy functions as arguments, combined using :ref:`combinator strategies <combinators>`, and modified using |strategy.filter|, |strategy.map|, or |strategy.flatmap|.

Refactor the reference section of the docs #4311

Refactor the reference section of the docs #4311

Uh oh!

Conversation

tybug commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jobh Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zac-HD commented Mar 19, 2025

Uh oh!

tybug commented Mar 19, 2025

Uh oh!

tybug commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tybug commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zac-HD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Zac-HD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tybug commented Mar 18, 2025 •

edited

Loading

jobh Mar 20, 2025 •

edited

Loading

tybug commented Mar 20, 2025 •

edited

Loading

tybug commented Mar 21, 2025 •

edited

Loading