[SIP-156] Proposal for a new Apache Superset Developer Portal

[rusackas]

[SIP-156] Proposal for a new Apache Superset Developer Portal

Motivation

Apache Superset is evolving to support a extension-based architecture as part of SIP-151. With this change, there is a need for comprehensive and well-structured documentation tailored to developers building extensions, integrations, and customizations. The current documentation is fragmented across multiple sources, making it difficult for contributors and developers to find relevant information efficiently. It gives no distinction between, or single entry to, documentation created for users, developers, and administrators.

Proposed Change

This SIP proposes the creation of a Superset Developer Portal, which will serve as a single, authoritative source of documentation for the Superset SDK and extension ecosystem. Additionally, it will lay the foundation for an eventual overhaul of Superset’s broader documentation, restructuring content into dedicated sections for Developers, Users, Contributors, and Installation/Deployment.

A key goal of this initiative is to grow the Superset community by making it easier for both Superset and Superset extension developers to contribute, and by empowering users and administrators to easily find all the documentation they need to successfully run and use Superset.

This SIP aims to align on the requirements and select the best tooling for implementing a modern, centralized Developer Portal for Apache Superset. After doing some tool comparison (see grid below), the tool we’re proposing is Docusaurus. While by stroke of luck that happens to be the tool we’re using on our current docs site, it also seems to be the only one to meet the broad requirements we have.

Key requirements of the Superset Developer Portal include:

• Providing well-organized, text and code-snippet-based documentation, inspired by the VS Code extension SDK docs.
• Serving as a single repository for documentation, consolidating:
  • GitHub Wiki
  • Current documentation site
  • Various README files
  • …and more!
• Supporting MDX (Markdown with JSX) to embed interactive components in documentation, reducing reliance on screenshots.
• Allowing embedding or replacement of React Storybook, enabling developers to interact with UI components directly.
• Integration with automated testing (E2E tests) to capture and embed static screenshots, ensuring documentation stays up to date.
• Implement multi-instance versioning to allow users to reference the correct documentation relative to:
  • Superset core versions
  • Monorepo package versions (e.g., components, themes, SDK, etc.)
• Site search, or preferably an AI-powered search tool (if ASF guidelines permit) for efficient navigation. [side note, Kapa.ai now has a DPA and features in place to accomplish this]
• An improved contributor experience by providing:
  • A way to contribute via PRs to markdown/MDX files.
  • A mechanism for users to highlight and suggest changes interactively.
• Automated documentation generation using JSDoc/docstrings, and sidecar scripts, and integrate an API playground similar to Swagger UI.
• Support for required tooling (GHA deployment, Matomo analytics, etc)
• Typescript/React support wherever warranted
• i18n support wherever feasible.
• Support for Superset Themes and AntDesign components (WIP) wherever relevant

New or Changed Public Interfaces

• Introduction of a new Developer Portal website.
• Potential deprecation of Superset’s GitHub Wiki in favor of structured documentation.
• Inclusion of interactive documentation components using MDX and potentially replacing Storybook.
• Automated API documentation generation from JSDoc/docstrings.

New dependencies

• Docusaurus 3.0 (based on tool evaluation below)
• MDX support for embedding interactive components.
• Search indexing tool, potentially AI-powered (Kapa.ai based on recent ASF approvals).
• Automated screenshot and API doc generation tools (TBD).

Migration Plan and Compatibility

• The Developer Portal will initially launch alongside existing documentation. Depending on the tool selected, this new portal may be included with the current documentation deployment, or require its own publication lifecycle (e.g. GitHub actions) and may be hosted on a new subdomain (subject to ASF Infra guidance and approvals). It may be plausible to host the new site at https://superset.apache.org/docs/, which is not used by the current documentation site (https://superset.apache.org/docs/intro/ is the current entry point).
• The Developer Portal will initially be populated with documentation around the new Extensions Architecture, related SDKs, and contribution guidelines for the Developer Portal itself.
• Documentation from the GitHub Wiki, Readme files, and the legacy Docs site will later be migrated to the new platform, and thus deprecating/removing those other resources, adding redirects where warranted.
• Versioning strategies will be determined to ensure backward-compatibility and that users are consuming docs relevant to the version they have installed. We will focus early efforts on versioning component parts of the docs relative to their NPM/Python package counterparts (e.g. extensions, plugins and packages as we know them today, SDK versions).
• Contributions will be continue to allow markdown/MDX-based updates via GitHub PRs, but we will assess any/all alternative means of gathering documentation suggestions/feedback/contribution

Aproximate information architecture will be laid out like so:

graph TD
    A[Apache Superset Developer Portal] --> B[Core Docusaurus Setup]
    B --> C[Multi-instance Configuration]
    C --> D1[User Docs]
    C --> D2[Admin Docs]
    C --> D3[Contributor Docs]
    C --> D4[Community Resources]
    C --> D5[Developer/SDK Docs]
    C --> D6[Package Docs]
    
    B --> E[Shared Components]
    E --> E1[Navigation]
    E --> E2[Search]
    E --> E3[Theme]
    E --> E4[i18n Support]
    
    B --> F[Plugins & Integrations]
    F --> F1[OpenAPI/Swagger Integration]
    F --> F2[Interactive Component Playground]
    F --> F3[TypeDoc Integration]
    F --> F4[Matomo Analytics]
    F --> F5[Feedback Mechanisms]

Loading

We'll use a multi-instance approach in Docusaurus that flows approximately like so:

graph TD
    A[Root Docusaurus Project] --> B[Shared Components & Config]
    A --> C[Content Instances]
    C --> D1[User Docs Instance]
    C --> D2[Admin Docs Instance]
    C --> D3[Contributor Docs Instance]
    C --> D4[Community Resources Instance]
    C --> D5[Developer/SDK Docs Instance]
    C --> D6[Package Docs Instance]
    
    B --> E1[Theme]
    B --> E2[Navigation]
    B --> E3[Search]
    B --> E4[i18n]
    
    D1 --> F1[User Docs Versioning]
    D2 --> F2[Admin Docs Versioning]
    D3 --> F3[Contributor Docs Versioning]
    D4 --> F4[Community Resources Versioning]
    D5 --> F5[Developer/SDK Docs Versioning]
    D6 --> F6[Package Docs Versioning]

Loading

Rejected Alternatives

• Keeping the current documentation structure: This was deemed insufficient due to fragmentation and difficulty in navigation.
• Using only React Storybook for interactive documentation: Storybook alone does not address broader documentation needs.
• Maintaining multiple sources of truth: A single Developer Portal will be more maintainable and user-friendly.
• All other tools seen in this table besides Docusaurus.

Feature/Aspect	Docusaurus 3.0	Nextra (NextJS)	GitBook	VitePress	MkDocs + Material	VuePress	Starlight	Docz
MDX Support	✅ First-class support	✅ Full support	❌ No	❌ No (uses Vue)	❌ No	✅ Yes	✅ Yes	✅ Yes
React/JSX Components	✅ Yes	✅ Yes	❌ No (iframes only)	❌ Vue only	❌ No	✅ Yes	✅ Yes	✅ Yes
Versioning	✅ Built-in	⚠️ Manual implementation needed	❌ No	⚠️ Limited	⚠️ via deprecated plugin	❌ No	❌ No	❌ No
Multi-instance versioning (plugins, SDKs, etc)	✅ via "multi-instance" setup	⚠️ Complex	❌ No	❌ No	⚠️ via deprecated plugin	❌ No	❌ No	❌ No
Search	✅ Algolia + Local search	⚠️ Needs Algolia integration	✅ Built-in	✅ Built-in	✅ Lunr	✅ Yes	✅ Yes	⚠️ Limited
API Docs Integration	✅ Via plugins	✅ Flexible	❌ No	⚠️ Limited	✅ Supports Swagger/OpenAPI	✅ OpenAPI/Swagger/GraphQL	⚠️ Limited	⚠️ Limited
Interactive API Playground (Swagger, GraphQL, etc.)	✅ Via plugins	✅ Yes	❌ No (iframes only)	⚠️ Custom/plugins	✅ Yes	🚧 Limited	✅ Yes	✅ Yes
Interactive Components	✅ React-based	✅ Full React support	⚠️ Limited	❌ Vue-based	❌ No native MDX or JSX support	❌ Vue only	✅ Yes	✅ Yes
Interactive Component Coding	✅ can use react-live or Sandpack	⚠️ Complex (uses SSR)	❌ No	❌ Vue only	❌ No	❌ Vue only	⚠️ Limited	✅ Built-in live editing
PR-based Workflow	✅ Native GitHub integration	✅ Full support	⚠️ Limited	✅ Supported	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Storybook Integration	✅ Via MDX	✅ Full control	❌ Not supported	⚠️ Complex	❌ No	⚠️ Difficult	⚠️ Limited	✅ Yes
Customization	⚠️ Moderate	✅ Maximum	❌ Limited	⚠️ Moderate	⚠️ Moderate	❌ Vue based	⚠️ Moderate	✅ High
Setup Complexity	✅ Low	⚠️ High	✅ Low	✅ Low	✅ Low	⚠️ Moderate	⚠️ Moderate	⚠️ Moderate
Maintenance Burden	✅ Low	⚠️ High	✅ Low	⚠️ Moderate	✅ Low	⚠️ Moderate	✅ Low	✅ Low
ASF Compatibility	✅ Proven	⚠️ Needs verification	⚠️ Needs verification	✅ Compatible	❌ No	🚧 Difficult	⚠️ Needs verification	⚠️ Needs verification
i18n Support	✅ Built-in	⚠️ Manual setup	✅ Built-in	✅ Built-in	✅ via mkdocs-static-i18n	✅ Yes	✅ Built-in	❌ No
Cost	✅ Free/Open Source	✅ Free/Open Source	❌ Commercial	✅ Free/Open Source	✅ Free/Open Source	✅ Free/Open Source	✅ Free/Open Source	✅ Free/Open Source
Community Size	✅ Large	✅ Large	✅ Large	⚠️ Moderate	⚠️ Moderate	✅ Large	⚠️ Growing	⚠️ Moderate
AI-Generated Docs	⚠️ Via plugins (AI-generated summaries)	❌ No	✅ Built-in AI-powered documentation generation	❌ No	🚧 Requires custom scripts (OpenAI, LLMs)	❌ No	❌ No	❌ No
Auto-generate from Python Docstrings	❌ No	❌ No	❌ No	❌ No	✅ Yes (mkdocstrings)	❌ No	❌ No	❌ No
Auto-generate from JSDoc/TS	✅ Via typedoc plugin	✅ Full integration possible with Next.js	❌ No	❌ No	❌ No	❌ No	❌ No	✅ Via typedoc plugin