Add Native Support for Paginated API Responses

[shalev007]

Problem:
Many APIs return data in pages and require follow-up requests to fetch additional results. Currently, RestBook playbooks must handle pagination manually (via store + iterate), which is repetitive and fragile.

We should add first-class support for pagination.

---

✅ What to Implement

• Add a paginate: block under request
• Support cursor-based pagination via JQ path (e.g., .next_cursor)
• Support offset/page pagination via counter increments
• Loop the request automatically until:
  • Cursor is null
  • Response body is empty
  • Custom Jinja condition evaluates to false

---

🔧 Example Syntax

Cursor-based Pagination

request:
  method: GET
  endpoint: /users
  params:
    limit: 100
  paginate:
    method: cursor
    cursor_path: '.response_metadata.next_cursor'
    until: '{{ next_cursor is none }}'

Offset-based Pagination

request:
  method: GET
  endpoint: /users
  params:
    limit: 100
    offset: '{{ offset }}'
  paginate:
    method: offset
    start: 0
    step: 100
    until: '{{ response.body | length == 0 }}'

💡 Why It Matters

• Makes data ingestion workflows much easier and more declarative
• Reduces boilerplate and errors in looping logic
• Enables RestBook to handle common API patterns out-of-the-box:
  • GitHub: Link headers or next_url
  • Slack: .response_metadata.next_cursor
  • Notion, Stripe, Datadog, etc.
• Better support for long-running, data-heavy workflows
• Lets backend/data engineers focus on the logic, not API quirks

---

📎 Related Ideas

• Support storing all paginated responses into a list via store.append: true
• Combine with incremental: to support resume-from-last-page
• Integrate with retry logic and rate-limiting across page requests
• Add --max-pages or --max-items limit for debugging/testing
• Allow cursor to be extracted from headers (e.g., GitHub's Link)