Add beautiful soup 4 dependency with example test

[sbland]

Description

Implementing beautiful soup to allow HTML testing of views.

Fixes #593

Type of change

• Documentation (non-breaking change that adds or improves the documentation)
• New feature (non-breaking change which adds functionality)
• Optimization (non-breaking, back-end change that speeds up the code)
• Technical work (non-breaking, change which is work as part of a new feature)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (whatever its nature)

Key checklist

• All tests pass (eg. python -m pytest)
• The documentation builds and looks OK (eg. mkdocs serve)
• Pre-commit hooks run successfully (eg. pre-commit run --all-files)

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added or an issue has been opened to tackle that in the future. (Indicate issue here: # (issue))

[sbland]

One issue I'm having is beautiful soup not being able to find elements unless we strip the text. See util function implemented below to get around this:

def find_stripped_text(doc, tag, text):
    """Helper function to find the stripped text of a tag."""
    return next(
        a
        for a in doc.find_all(tag)
        if re.search(rf"{text}", a.get_text(" ", strip=True))
    )

[sbland]

@AdrianDAlessandro I've added some tests here trying out user driven testing but I need to dig into how the other tests are working for mocking the db etc.

[AdrianDAlessandro]

The DB is currently mocked using the pytest fixtures in main/conftest.py. This might change if we move to using data migrations to populate the DB (see #598). You also need to mark the tests that are going to use the DB with @pytest.mark.django_db.

What might be the best approach here is to use a class-based test that has a setup method that spins up and populates the database and then mark that class as using the DB, then all subsequent tests inherit from that class.

[AdrianDAlessandro]

I like the idea of implementing some user-driven testing, although I think maybe it should be in a separate PR to the unit tests on individual views. It is also lower priority, particularly because I think we might be changing some of the user workflows (like the self-assessment form, etc).

(FYI, I'm going to mark this PR as a draft for now)

[AdrianDAlessandro]

The DB is currently mocked using the pytest fixtures in main/conftest.py. This might change if we move to using data migrations to populate the DB (see #598). You also need to mark the tests that are going to use the DB with @pytest.mark.django_db.

What might be the best approach here is to use a class-based test that has a setup method that spins up and populates the database and then mark that class as using the DB, then all subsequent tests inherit from that class.

[AdrianDAlessandro]

I presume this print is temporary for while debugging?

[sbland]

Yes sorry! I'll clean up.

[AdrianDAlessandro]

I think the point of using BS4 is to not need to do any regex. Looking at the example PR I linked in the issue, they needed to put IDs on many of the html objects in order to test them. So maybe that gets around this?

[sbland]

Using Ids unfortunately moves away from the tests resembling how a user would interact with the app.

This view is coming from using React Testing Library. It might be that BS4 is just not set up to be able to do this but it would be a shame to have to add Ids everywhere just to make the tests work.

I'll ask around and do some digging to see if there is a better solution.