Improve pagination in web ui #1625

Rishi-source · 2024-10-19T07:51:30Z

Add pagination to VCIO & SCIO #1616 #1617 #1618

This change aligns Package Search functionality and Vulnerability Search, improving consistency across the application and enhancing user control over result display.

For Package Search:

For Vulnerability Search:

Signed-off-by: Rishi Garg [email protected]

Rishi-source · 2024-10-19T11:58:22Z

Hi @pombredanne @TG1999 @keshav-space @Hritik14 , Can you please review this PR It has passed all workflow tests and was also working fine on my local server. Thank you

pombredanne · 2024-10-25T10:26:30Z

@TG1999 ping .... @tdruez input welcomed too, as eventually we want to use the same approach across all projects.

vulnerabilities/templates/vulnerabilities.html

vulnerabilities/views.py

vulnerabilities/templates/packages.html

Rishi-source · 2024-11-02T06:52:25Z

@tdruez please see the changes now workflow tests are also successful .

Rishi-source · 2024-11-12T02:34:31Z

@tdruez I would request you to please review the PR I have made changes In the code after you requested some changes after first review.

tdruez

See the various comments.
Generally, this implementation seems too complex and not "reusable" enough at it requires to many moving parts.

tdruez · 2024-11-12T07:13:08Z

vulnerabilities/forms.py

@@ -3,7 +3,7 @@
 # VulnerableCode is a trademark of nexB Inc.
 # SPDX-License-Identifier: Apache-2.0
 # See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
-# See https://github.com/aboutcode-org/vulnerablecode for support or download.


Unwanted changes

tdruez · 2024-11-12T07:13:24Z

vulnerabilities/forms.py

@@ -12,26 +12,87 @@

 from vulnerabilities.models import ApiUser

+from .models import *


Avoid * imports.

tdruez · 2024-11-12T07:14:14Z

vulnerabilities/forms.py

+        widget=forms.Select(
+            attrs={
+                "class": "select is-small",
+                "onchange": "handlePageSizeChange(this.value)",


Avoid inline JavaScript.

tdruez · 2024-11-12T07:14:33Z

vulnerabilities/forms.py

+    def clean_search(self):
+        return self.cleaned_data.get("search", "")


What is the purpose of this?

tdruez · 2024-11-12T07:15:10Z

vulnerabilities/forms.py

+    def get_queryset(self, query=None):
+        """
+        Get queryset with search/filter/ordering applied.
+        Args:
+            query (str, optional): Direct query for testing
+        """
+        if query is not None:
+            return self._search(query)
+
+        if not self.is_valid():
+            return self.model.objects.none()
+
+        return self._search(self.clean_search())


What's the need for this logic?

tdruez · 2024-11-12T07:15:34Z

vulnerabilities/forms.py

+    def _search(self, query):
+        """Execute package-specific search logic."""
+        return (
+            self.model.objects.search(query)
+            .with_vulnerability_counts()
+            .prefetch_related()
+            .order_by("package_url")
+        )


This is not following the Django forms conventions.

tdruez · 2024-11-12T07:16:04Z

vulnerabilities/forms.py

+    def _search(self, query):
+        """Execute vulnerability-specific search logic."""
+        return self.model.objects.search(query=query).with_package_counts()


Same comment as above.

tdruez · 2024-11-12T07:17:15Z

vulnerabilities/templates/includes/pagination.html

+        <a href="?page={{ page_obj.previous_page_number }}&search={{ search|urlencode }}&page_size={{ page_obj.paginator.per_page }}" 
+           class="pagination-previous">Previous</a>


What happens if other filters are active, those are lost when using the pagination?

tdruez · 2024-11-12T07:20:12Z

vulnerabilities/templates/includes/pagination.html

All the complexity of those templates should likely be handled in Python.

tdruez · 2024-11-12T07:22:49Z

vulnerabilities/views.py

-    model = models.Package
-    template_name = "packages.html"
-    ordering = ["type", "namespace", "name", "version"]
+class BaseSearchView(ListView):


Could we use the django_filters.views.FilterView instead of re-implementing it?

Rishi-source · 2024-11-19T08:44:36Z

See the various comments. Generally, this implementation seems too complex and not "reusable" enough at it requires to many moving parts.

@tdruez what do you mean by "reusable" here. I would code the implementation again to make it more simpler and satisfy the required needs.

tdruez · 2024-11-20T05:59:57Z

@tdruez what do you mean by "reusable" here. I would code the implementation again to make it more simpler and satisfy the required needs.

@pombredanne as eventually we want to use the same approach across all projects.

@Rishi-source Let's say I want to add this pagination on another Django view, could you document the few steps I need to do with this current implementation? This would help to figure out where we can improve the "reusability".

Rishi-source · 2024-12-08T06:44:32Z

vulnerabilities/pagination_mixin.py

+        {"value": 100, "label": "100 per page"},
+    ]
+
+    def get_paginate_by(self, queryset=None):


This function handles page size from URL parameters. Sets default to 20 items, max of 100. Validates to ensure positive integers only. Returns default if invalid values provided.

Rishi-source · 2024-12-08T06:46:34Z

vulnerabilities/pagination_mixin.py

+        except (ValueError, TypeError):
+            return self.paginate_by
+
+    def get_context_data(self, **kwargs):


It adds pagination data to template context - current page size, size choices (can be handled by altering PAGE_SIZE_CHOICES) , total count and page range. it also preserves search params across pagination.

Rishi-source · 2024-12-08T06:48:06Z

vulnerabilities/pagination_mixin.py

+
+        return context
+
+    def _get_page_range(self, paginator, current_page, window=2):


it creates page number sequence. Shows all numbers for small sets (<5 pages). For larger sets, shows first/last pages around current page with (...) for gaps.

Rishi-source · 2024-12-08T06:49:46Z

vulnerabilities/views.py

@@ -36,6 +36,8 @@
 from vulnerabilities.utils import get_severity_range
 from vulnerablecode.settings import env

+from .pagination_mixin import PaginatedListViewMixin


Import the Mixin and the PaginatedListViewMixin class.

Rishi-source · 2024-12-08T06:51:19Z

vulnerabilities/views.py

@@ -62,25 +64,21 @@ def get_purl_version_class(purl: models.Package):
    return purl_version_class


-class PackageSearch(ListView):
+class PackageSearch(PaginatedListViewMixin, ListView):


Add the PaginatedListViewMixin in it and this would convert any ListView into paginated class.

Rishi-source · 2024-12-08T06:56:00Z

Hi @tdruez , I have altered code to improve the reusability for that I have made a file pagination_mixin.py which contains PaginatedListViewMixin class . This logic can convert and ListView into a paginated class and can improve the reusability as now you would have to just add this class to you views.

Rishi-source · 2024-12-08T07:08:49Z

vulnerabilities/templates/includes/pagination.html

+    window.location.href = newUrl;
+}
+</script>


This js function manages URL parameters when changing pagination size. It update the page size parameter while preserving search terms, and remove the current page parameter to reset pagination.

is there any risk of XSS there?

tdruez · 2024-12-09T05:03:48Z

@Rishi-source Thanks but looking at multiple comments is hard to follow. Could you also provide a combined documentation of the few steps required to reuse this pagination in another app?

Rishi-source · 2024-12-09T09:40:14Z

Pagination Implementation Documentation

Core Components of Code

Pagination Mixin (in pagination_mixins.py):
Pagination Template (in templates/includes/pagination.html):

Implementation Steps

Copy the Mixin
- Create pagination_mixins.py in your app directory
- Copy the PaginatedListViewMixin class code into it
Create the Template
- Create pagination.html in your templates directory
- Copy the pagination template code into it
Use PaginatedListViewMixin in Your View

from django.views.generic import ListView
from .pagination_mixins import PaginatedListViewMixin # Make sure to import the mixin

class MyListView(PaginatedListViewMixin, ListView): # Please make sure to put "PaginatedListViewMixin" Above all the mixins in case if there are 2 or more mixins are already their as position  of mixin will matter here
    model = MyModel
    template_name = "my_template.html"
    
    def get_queryset(self):
        return super().get_queryset().order_by('id')

Include in Your Pagination Template

{% extends "base.html" %}

{% block content %}

    {# Like This #}
    {% if is_paginated %}
        {% include 'pagination.html' with page_obj=page_obj %}
    {% endif %}

{% endblock %}

Customization Options

Change Page Sizes

class MyListView(PaginatedListViewMixin, ListView):
    PAGE_SIZE_CHOICES = [
        {"value": 10, "label": "10 per page"},
        {"value": 25, "label": "25 per page"},
        {"value": 50, "label": "50 per page"},
    ]
    paginate_by = 20  # Set the Default size

Adjust Window Size

def get_context_data(self, **kwargs):
    # Change window variable to show more/fewer pages around current page
    context = super().get_context_data(**kwargs)
    context['page_range'] = self._get_page_range(
        context['paginator'], 
        context['page_obj'].number,
        window=3  # Show 3 pages on each side i.e. left and right   
    return context

Notes

JavaScript is embedded directly in the template via the onchange handler
All pagination state is managed through URL parameters
All Filter terms are preserved across pagination
The mixin automatically handles invalid page numbers and sizes

Common Issues and Solutions

Missing Page Numbers
- Check that page_range is being passed correctly in context
URL Parameters Not Preserved
- Check template URLs include all necessary parameters
- Verify parameter names match in views and templates
Page Size Not Working
- Verify page_size parameter is being passed correctly
- Check get_paginate_by implementation in mixin
Styling Issues
- Ensure Bulma CSS is included if using default styling
- Or customize template classes to match your CSS framework

Remember that this implementation assumes:

You're using Django's class-based views
You want URL parameter-based pagination
You're handling search through GET parameters

Rishi-source · 2024-12-09T09:42:52Z

@tdruez Here’s the documentation on implementing pagination using this approach. Please review and let me know if any changes are required.

Rishi-source · 2024-12-10T14:30:32Z

Hi @tdruez and @pombredanne , I have implemented this pagination code in both VCIO and SCIO and it is working nice.It preserved all the filter and add a option for the user to select the items per page. I have already shared the screenshot for the VCIO pagination view here are the screenshot for SCIO pagination.

@tdruez you just have to follow the steps which I have provided in the Documentation for implementing this pagination.

Rishi-source · 2024-12-25T04:03:41Z

Hi @TG1999 @tdruez , I request you to please review this PR I have completed the code .

pombredanne

Thanks
Here is some feedback!
The PR title should be using imperative style:
Not Pagination Updated - Items per page option added

But rather:
"Improve pagination in web ui"
then add details in the body.

Could you also add tests? PaginatedListViewMixin is not trivial code

Also as @tdruez mentioned in late November, the template should have little to no logic and instead the logic should be in the Python code.

pombredanne · 2025-01-06T14:02:08Z

vulnerabilities/pagination_mixin.py

+        Get the number of items to paginate by from the request.
+        """
+        try:
+            page_size = int(self.request.GET.get("page_size", self.paginate_by))


Is this the standard approach to get a request arg?

pombredanne · 2025-01-06T14:06:31Z

vulnerabilities/pagination_mixin.py

+
+    def get_pagination_context(self, paginator, page_obj):
+        """
+        Generate pagination-related context data, preserving filters.


Suggested change

Generate pagination-related context data, preserving filters.

Return a mapping of pagination-related context data, preserving filters.

pombredanne · 2025-01-06T14:10:15Z

vulnerabilities/pagination_mixin.py

+        query_params = self.request.GET.copy()
+        query_params.pop("page", None)
+
+        base_query_string = query_params.urlencode()


Is this the right way to generate a URL query string? Is there a better way?

pombredanne · 2025-01-06T14:11:02Z

vulnerabilities/templates/includes/pagination.html

+    window.location.href = newUrl;
+}
+</script>


is there any risk of XSS there?

pombredanne · 2025-01-06T14:11:42Z

vulnerabilities/pagination_mixin.py

@@ -0,0 +1,87 @@
+class PaginatedListViewMixin:
+    """
+    A mixin that adds pagination functionality to ListView-based views.


I would call the file pagination.py instead

Signed-off-by: Rishi Garg <[email protected]>

tdruez · 2025-03-17T12:38:47Z

vulnerabilities/pagination.py

+
+    paginate_by = 20
+    max_page_size = 100
+    PAGE_SIZE_CHOICES = [


Why this option is uppercase?

This option contains the choices for the total number of items to be displayed on the page. Since the values are hardcoded, I highlighted them in uppercase to make it more noticeable in the code.

tdruez · 2025-03-17T12:39:37Z

vulnerabilities/pagination.py

+            queryset = super().get_queryset()
+            if not queryset:
+                queryset = self.model.objects.all()
+            if not isinstance(queryset, QuerySet):
+                queryset = self.model.objects.all()
+            return queryset


You should only have 1 instruction in your try block wrapping the thing you are catching the exception for.

Also, what's the purpose of overriding this method?

tdruez · 2025-03-17T12:41:41Z

vulnerabilities/pagination.py

+        try:
+            page_size = int(clean_page_size) if clean_page_size else self.paginate_by
+            valid_sizes = {choice["value"] for choice in self.PAGE_SIZE_CHOICES}
+            if page_size not in valid_sizes:
+                logger.warning(f"Attempted to use unauthorized page size: {page_size}")
+                return self.paginate_by
+            return page_size


Same as above, do not put all you logic in a try block, just the code that may raise the exception.

tdruez · 2025-03-17T12:42:43Z

vulnerabilities/pagination.py

+    def get_page_range(self, paginator, page_obj):
+        """
+        Generate a list of page numbers for navigation
+        """
+        num_pages = paginator.num_pages
+        current_page = page_obj.number
+        if num_pages <= 7:
+            return list(range(1, num_pages + 1))
+        pages = []
+        pages.append(1)
+        if current_page > 4:
+            pages.append("...")
+        start = max(2, current_page - 2)
+        end = min(num_pages - 1, current_page + 2)
+        pages.extend(range(start, end + 1))
+        if current_page < num_pages - 3:
+            pages.append("...")
+        if num_pages > 1:
+            pages.append(num_pages)


The values should not be hardcoded but configurable.

This part shows numbers and ellipses (1 2 3 4 ... 10 11 12 13) based on hardcoded values. I can replace these with configurable variables to make the display pattern easily adjustable in the future but it will still be hardcoded but in a variable form.

tdruez · 2025-03-17T12:45:23Z

vulnerabilities/pagination.py

+        try:
+            if not queryset or queryset.count() == 0:
+                queryset = self.model.objects.all()
+            paginator = Paginator(queryset, page_size)
+            page_params = self.request.GET.getlist("page")
+            page_number = page_params[-1] if page_params else "1"
+            try:
+                page_number = int(re.sub(r"\D", "", str(page_number)))
+                if not page_number:
+                    page_number = 1
+            except (ValueError, TypeError):
+                page_number = 1
+            page_number = max(1, min(page_number, paginator.num_pages))
+            page = paginator.page(page_number)
+            return (paginator, page, page.object_list, page.has_other_pages())
+        except Exception as e:
+            logger.error(f"Pagination error: {e}")
+            queryset = self.model.objects.all()
+            paginator = Paginator(queryset, page_size)
+            page = paginator.page(1)
+            return (paginator, page, page.object_list, page.has_other_pages())


Inefficient Queryset Handling

queryset.count() == 0 performs an unnecessary database query.

Suggested fix: Use queryset.exists() instead.

Overcomplicated Page Number Extraction

re.sub(r"\D", "", str(page_number)) is unnecessary.

Suggested fix: Use page_number = int(self.request.GET.get("page", "1")).

Nested Try-Except Blocks

The current structure makes debugging harder.

Suggested fix: Use a single try-except block.

Broad Exception Handling

Catching Exception hides real bugs.

Suggested fix: Handle EmptyPage and InvalidPage explicitly.

Redundant Queryset Reassignment in Exception Handling

Fetching self.model.objects.all() again is unnecessary.

Suggested fix: Log the error and return a fallback page.

tdruez · 2025-03-17T12:47:08Z

vulnerabilities/templates/includes/pagination.html

+function handlePageSizeChange(value) {
+  const url = new URL(window.location.href);
+  const params = new URLSearchParams(url.search);
+  params.set('page_size', value);
+  params.delete('page');
+
+  const search = params.get('search');
+  if (search) {
+      params.set('search', search);
+  }
+
+  const newUrl = `${window.location.pathname}?${params.toString()}`;
+  window.location.href = newUrl;
+}
+</script>


A no-JS approach would work better.

Sure I will remove this JS code and handle is via form tag

Signed-off-by: RISHI GARG <[email protected]>

Rishi-source · 2025-03-25T12:31:47Z

Hello @tdruez, please review this pull request I have made all the changes required.

Rishi-source · 2025-04-25T16:03:18Z

Hi @pombredanne please review this PR I have added tests and also addressed all the given review points.

tdruez requested changes Oct 29, 2024

View reviewed changes

Rishi-source force-pushed the Page-Updated branch 4 times, most recently from d8068aa to c28765a Compare October 29, 2024 18:44

Rishi-source requested a review from tdruez October 29, 2024 18:50

Rishi-source force-pushed the Page-Updated branch 3 times, most recently from 8d177c8 to 6ca25ee Compare November 2, 2024 05:47

tdruez requested changes Nov 12, 2024

View reviewed changes

Rishi-source force-pushed the Page-Updated branch from 170df8c to 1302bed Compare December 8, 2024 06:34

Rishi-source commented Dec 8, 2024

View reviewed changes

Rishi-source requested a review from tdruez December 8, 2024 06:56

Rishi-source commented Dec 8, 2024

View reviewed changes

Rishi-source closed this Dec 26, 2024

Rishi-source reopened this Dec 26, 2024

pombredanne requested changes Jan 6, 2025

View reviewed changes

Rishi-source closed this Jan 25, 2025

Rishi-source force-pushed the Page-Updated branch from caab3c0 to fabe035 Compare January 25, 2025 09:10

Rishi-garg03 added 2 commits January 25, 2025 15:13

Feat: Pagination improved in the UI

e671c13

Signed-off-by: Rishi Garg <[email protected]>

Fix : Workflow checks error fixed

d981356

Signed-off-by: Rishi Garg <[email protected]>

Rishi-source mentioned this pull request Jan 25, 2025

Pagination improved in the UI #1761

Closed

Rishi-source reopened this Mar 2, 2025

Rishi-source added 2 commits March 2, 2025 08:40

Merge branch 'main' into Page-Updated

d2c13a1

Merge branch 'main' into Page-Updated

936c5bb

tdruez requested changes Mar 17, 2025

View reviewed changes

Rishi-source added 3 commits March 18, 2025 10:56

Pagination Mixin optimisation and JS code removed

c75a134

Signed-off-by: RISHI GARG <[email protected]>

Tests are added and code is reformatted via make valid

7c2a138

Signed-off-by: RISHI GARG <[email protected]>

Test fixed for pagination

e5caa29

Signed-off-by: RISHI GARG <[email protected]>

Rishi-source requested a review from tdruez March 18, 2025 06:09

Rishi-source changed the title ~~Pagination Updated - Items per page option added~~ Pagination improved in the UI Mar 18, 2025

Rishi-source requested a review from pombredanne March 18, 2025 06:59

Merge branch 'main' into Page-Updated

09ca590

Rishi-source added 2 commits March 27, 2025 11:12

Merge branch 'main' into Page-Updated

d9ae3ff

Merge branch 'main' into Page-Updated

1c40cdb

Rishi-source changed the title ~~Pagination improved in the UI~~ Improve pagination in web ui Apr 6, 2025

Rishi-source added 2 commits April 16, 2025 20:49

Merge branch 'main' into Page-Updated

b8aa68e

Merge branch 'main' into Page-Updated

788e202

		@@ -12,26 +12,87 @@

		from vulnerabilities.models import ApiUser

		from .models import *

		def clean_search(self):
		return self.cleaned_data.get("search", "")

		<a href="?page={{ page_obj.previous_page_number }}&search={{ search\|urlencode }}&page_size={{ page_obj.paginator.per_page }}"
		class="pagination-previous">Previous</a>


		return context

		def _get_page_range(self, paginator, current_page, window=2):

	Generate pagination-related context data, preserving filters.
	Return a mapping of pagination-related context data, preserving filters.

Improve pagination in web ui #1625

Are you sure you want to change the base?

Improve pagination in web ui #1625

Conversation

Rishi-source commented Oct 19, 2024 • edited Loading

For Package Search:

For Vulnerability Search:

Rishi-source commented Oct 19, 2024

pombredanne commented Oct 25, 2024

Rishi-source commented Nov 2, 2024

Rishi-source commented Nov 12, 2024

tdruez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishi-source commented Nov 19, 2024

tdruez commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishi-source commented Dec 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdruez commented Dec 9, 2024

Rishi-source commented Dec 9, 2024 • edited Loading

Pagination Implementation Documentation

Core Components of Code

Implementation Steps

Customization Options

Notes

Common Issues and Solutions

Rishi-source commented Dec 9, 2024

Rishi-source commented Dec 10, 2024

Rishi-source commented Dec 25, 2024

pombredanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishi-source commented Mar 25, 2025

Rishi-source commented Apr 25, 2025

Rishi-source commented Oct 19, 2024 •

edited

Loading

Rishi-source commented Dec 9, 2024 •

edited

Loading