Merge remote-tracking branch 'origin/completeness' into iformredirect-compat

Author: wardi

.github/workflows/test.yml

@@ -24,6 +24,7 @@ jobs:
     runs-on: ubuntu-latest
     container:
       image: ckan/ckan-dev:${{ matrix.ckan-version }}
+      options: --user root
     services:
       solr:
         image: ckan/ckan-solr:${{ matrix.ckan-version }}-solr9

CHANGELOG.md

@@ -103,3 +103,26 @@
 * fix for applying default org/group types
 * sync example dataset schemas, presets and templates with upstream ckan
   changes
+
+## 3.1.0
+
+2024-03-27
+
+* This version drops support for CKAN 2.8, 2.9 and adds support for 2.11
+* Pass dataset name to resource fields snippets [#437](https://github.com/ckan/ckanext-scheming/pull/354)
+* Allow literal parameters in validator string [#372](https://github.com/ckan/ckanext-scheming/pull/372)
+* Fix delete URL for custom organizations [#374](https://github.com/ckan/ckanext-scheming/pull/374)
+* Group Form Required Message Position [#376](https://github.com/ckan/ckanext-scheming/pull/376)
+* Resource Form Errors Super Fallback [#380](https://github.com/ckan/ckanext-scheming/pull/380)
+* Use form_snippets/help_text.html in repeating_subfields.html [#387](https://github.com/ckan/ckanext-scheming/pull/397)
+* Add form-select CSS class to select elements [#399](https://github.com/ckan/ckanext-scheming/pull/399)
+* Fix ckan version comparison [#406](https://github.com/ckan/ckanext-scheming/pull/406)
+* Add number and file_size snippets [#412](https://github.com/ckan/ckanext-scheming/pull/412)
+* Before and After Validators for Groups [#428](https://github.com/ckan/ckanext-scheming/pull/428)
+* Drop ckantoolkit requirement [#432](https://github.com/ckan/ckanext-scheming/pull/432)
+* Fix `is_organization` for custom `organization_type` [#437](https://github.com/ckan/ckanext-scheming/pull/437), [#431](https://github.com/ckan/ckanext-scheming/pull/431)
+* fix: Move toggle-more rows to bottom of resource table [#420](https://github.com/ckan/ckanext-scheming/pull/420)
+* fix: Add default text to update/edit buttons in group_form [#421](https://github.com/ckan/ckanext-scheming/pull/421)
+* fix: False is also a value for radio and select snippets [#417](https://github.com/ckan/ckanext-scheming/pull/417)
+* Remove presets parsing cache check [#425](https://github.com/ckan/ckanext-scheming/pull/425)
+* fix: optional multiple_text saved as empty string if missing on create

README.md

@@ -5,7 +5,8 @@ This CKAN extension provides a way to configure and share metadata schemas using
 YAML or JSON schema description. Custom validation and template snippets for editing
 and display are supported.
 
-[![Tests](https://github.com/ckan/ckanext-scheming/workflows/Tests/badge.svg?branch=master)](https://github.com/ckan/ckanext-scheming/actions)
+[![Tests](https://github.com/ckan/ckanext-scheming/actions/workflows/test.yml/badge.svg)](https://github.com/ckan/ckanext-scheming/actions)
+
 
 Table of contents:
 
@@ -14,19 +15,18 @@ Table of contents:
 3. [Configuration](#configuration)
    - [Schema Types](#schema-types)
    - [Example Schemas](#example-schemas)
-   - [Storing non-string data](#storing-non-string-data)
    - [Common Schema Keys](#common-schema-keys)
      - [`scheming_version`](#scheming_version)
      - [`about_url`](#about_url)
+     - [`before_validators`, `after_validators`](#before_validators-after_validators)
    - [Dataset Schema Keys](#dataset-schema-keys)
      - [`dataset_type`](#dataset_type)
      - [`dataset_fields`, `resource_fields`](#dataset_fields-resource_fields)
-     - [`before_validators`, `after_validators`](#before_validators-after_validators)
+     - [`draft_fields_required`](#draft_fields_required)
    - [Group / Organization Schema Keys](#group--organization-schema-keys)
      - [`group_type`](#group_type)
      - [`organization_type`](#organization_type)
      - [`fields`](#fields)
-     - [`before_validators`, `after_validators`](#before_validators-after_validators-1)
    - [Field Keys](#field-keys)
      - [`field_name`](#field_name)
      - [`label`](#label)
@@ -113,6 +113,7 @@ Dataset schemas:
 * [camel photos schema](ckanext/scheming/camel_photos.yaml)
 * [subfields schema](ckanext/scheming/subfields.yaml)
 * [form pages schema](ckanext/scheming/ckan_formpages.yaml)
+* [form pages schema with `draft_fields_required: false`](ckanext/scheming/ckan_formpages_draft.yaml)
 
 These schemas are included in ckanext-scheming and may be enabled
 with e.g: `scheming.dataset_schemas = ckanext.scheming:camel_photos.yaml`
@@ -196,7 +197,25 @@ before_validators: validator_name
 after_validators: validator_name
 ```
 
-Runs validator functions before and after the `dataset_type` package is created/updated.
+Runs validator functions before and after the `dataset_type` package or `organization_type`/`group_type` group is created/updated.
+
+See [`validators`](#validators) for a description of the values accepted.
+
+### `draft_fields_required`
+
+Enforce required fields even if a dataset is still in draft state.
+Default: `true`
+
+```yaml
+draft_fields_required: false
+```
+
+> [!NOTE]
+> Setting `draft_fields_required: false` is only supported by **_CKAN 2.12+_** or with https://github.com/ckan/ckan/pull/8309 .
+
+Set to `false` to allow saving dataset and resource metadata even
+if some *required fields* have not been provided as long as the dataset
+is still in *draft* state (has not been published).
 
 
 ## Group / Organization Schema Keys
@@ -241,21 +260,9 @@ fields:
 
 ...
 ```
-A single `fields` list replaces the `dataset_fields` and `resource_fields` schema properties doin dataset schemas.
+A single `fields` list replaces the `dataset_fields` and `resource_fields` schema properties in dataset schemas.
 
 
-### `before_validators`, `after_validators`
-
-```yaml
-before_validators: validator_name
-
-after_validators: validator_name
-```
-
-Runs validator functions before and after the `organization_type`/`group_type` group is created/updated.
-
-
-----------------
 
 ## Field Keys
 ### `field_name`
@@ -304,7 +311,8 @@ When using a plain string translations will be provided with gettext:
 This field is the parent of group of repeating subfields. The value is
 a list of fields entered the same way as normal fields.
 
-> **_NOTE:_** CKAN needs an IPackageController plugin with `before_index` to
+> [!NOTE]
+> CKAN needs an IPackageController plugin with `before_index` to
 > convert repeating subfields to formats that can be indexed by solr. For
 > testing you may use the included `scheming_nerf_index` plugin to encode
 > all repeating fields as JSON strings to prevent solr errors.
@@ -333,9 +341,12 @@ for each group.
 ### `start_form_page`
 
 Dataset fields may be divided into separate form pages for creation
-and editing. **_CKAN 2.9+ only_**. Form pages for `dataset` type
-only supported by **_CKAN 2.10+_** or with https://github.com/ckan/ckan/pull/7032
-. Adding `start_form_page` to a field marks this field as the start of a
+and editing.
+
+> [!NOTE]
+> Form pages for `dataset` type is only supported by **_CKAN 2.10+_** or with https://github.com/ckan/ckan/pull/7032 .
+
+Adding `start_form_page` to a field marks this field as the start of a
 new page of fields.
 
 
@@ -364,13 +375,7 @@ Use for fields that must be included. Set to `false` or
 don't include this key for fields that are optional.
 
 Setting to `true` will mark the field as required in the editing form
-and include `not_empty` in the default validators that will be applied
-when `validators` is not specified.
-
-> **_NOTE:_** To honor this settings with custom validators include `scheming_required`
-> as the first validator. `scheming_required` will check the required
-> setting for this field and apply either the `not_empty` or `ignore_missing`
-> validator.
+and affects the behavior of the `scheming_required` field validator.
 
 
 ### `choices`
@@ -587,40 +592,50 @@ Set a `property` attribute on dataset fields displayed as "Additional Info", use
 ### `validators`
 
 The `validators` value is a space-separated string of validator and converter
-functions to use for this field when creating or updating data.  When a
-validator name is followed by parenthesis the function is called passing the
-comma-separated values within and the result is used as the
+functions to use for this field when creating or updating data. For core fields
+the validators default to the ones defined in the CKAN core schemas.
+
+> [!WARNING]
+> Use `scheming_required` as the first validator
+> or the [`required`](#required) field setting will be ignored.
+> `scheming_required` applies either the `not_empty` or `ignore_missing`
+> validator based on the `required` field setting.
+
+When a validator name is followed by parenthesis the function is called
+passing the comma-separated values within and the result is used as the
 validator/converter.
 
 ```yaml
-  validators: if_empty_same_as(name) unicode_safe
+  validators: scheming_required if_empty_same_as(name) unicode_safe
 ```
 
-is the same as a plugin using the validators:
+is the same as a form plugin using the validators:
 
 ```python
-[get_validator('if_empty_same_as')("name"), unicode_safe]
+[
+    get_validator('scheming_required'),
+    get_validator('if_empty_same_as')("name"),
+    get_validator('unicode_safe')
+]
 ```
 
-If parameters can be parsed as a valid python literals, they are passed with
-original type. If not, all parameters passed as strings. In addition, space
-character is not allowed in argument position. Use its HEX code instead `\\x20`.
+If validator parameters can be parsed as a valid python literals,
+they are passed with the original type. If not, all parameters passed
+as strings. In addition, space character is not allowed in argument
+position. Use its HEX code instead `\\x20`.
 
 ```yaml
-  validators: xxx(hello,world)    # xxx("hello", "world")
-  validators: xxx(hello,1)        # xxx("hello", "1")
-  validators: xxx("hello",1,None) # xxx("hello", 1, None)
+  validators: xxx(hello,world)       # xxx("hello", "world")
+  validators: xxx(hello,1)           # xxx("hello", "1")
+  validators: xxx("hello",1,None)    # xxx("hello", 1, None)
   validators: xxx("hello\\x20world") # xxx("hello world")
 ```
 
-This string does not contain arbitrary python code to be executed,
-you may only use registered validator functions, optionally calling
-them with static string values provided.
-
-> **_NOTE:_** ckanext-scheming automatically adds calls to `convert_to_extras`
+> [!NOTE]
+> ckanext-scheming automatically adds a call to `convert_to_extras`
 > for extra fields when required.
 
-New validators and converters may be added using the
+New validators and converters may be registered using the
 [IValidators plugin interface](http://docs.ckan.org/en/latest/extensions/plugin-interfaces.html?highlight=ivalidator#ckan.plugins.interfaces.IValidators).
 
 Validators that need access to other values in this schema (e.g.
@@ -657,7 +672,8 @@ retrieving values from the database instead of when saving them.
 These validators may be used to transform the data before it is
 sent to the user.
 
-> **_NOTE:_** ckanext-scheming automatically adds calls to `convert_from_extras`
+> [!NOTE]
+> ckanext-scheming automatically adds calls to `convert_from_extras`
 > for extra fields when required.
 
 ### `create_validators`

ckanext/scheming/ckan_formpages_draft.yaml

@@ -0,0 +1,115 @@
+scheming_version: 2
+dataset_type: formpages
+about: The default CKAN dataset schema with form split across multiple pages
+about_url: http://github.com/ckan/ckanext-scheming
+
+draft_fields_required: false
+
+dataset_fields:
+
+- start_form_page:
+    title: Basic Info
+    description: Required and core dataset fields
+
+  field_name: title
+  label: Title
+  preset: title
+  form_placeholder: eg. A descriptive title
+
+- field_name: name
+  label: URL
+  preset: dataset_slug
+  form_placeholder: eg. my-dataset
+
+- field_name: notes
+  label: Description
+  form_snippet: markdown.html
+  form_placeholder: eg. Some useful notes about the data
+  required: true
+  validators: scheming_required unicode_safe
+
+- field_name: owner_org
+  label: Organization
+  preset: dataset_organization
+
+- start_form_page:
+    title: Detailed Info
+    description:
+      These fields improve search and give users important links
+
+  field_name: tag_string
+  label: Tags
+  preset: tag_string_autocomplete
+  form_placeholder: eg. economy, mental health, government
+
+- field_name: license_id
+  label: License
+  form_snippet: license.html
+  help_text: License definitions and additional information can be found at http://opendefinition.org/
+
+- field_name: url
+  label: Source
+  form_placeholder: http://example.com/dataset.json
+  display_property: foaf:homepage
+  display_snippet: link.html
+
+- field_name: version
+  label: Version
+  validators: ignore_missing unicode_safe package_version_validator
+  form_placeholder: '1.0'
+  required: true
+  validators: scheming_required unicode_safe package_version_validator
+
+- start_form_page:
+    title: Contact Info
+    description: Names and email addresses for this dataset
+
+  field_name: author
+  label: Author
+  form_placeholder: Joe Bloggs
+  display_property: dc:creator
+  required: true
+  validators: scheming_required unicode_safe
+
+- field_name: author_email
+  label: Author Email
+  form_placeholder: [email protected]
+  display_property: dc:creator
+  display_snippet: email.html
+  display_email_name_field: author
+  validators: ignore_missing unicode_safe strip_value email_validator
+
+- field_name: maintainer
+  label: Maintainer
+  form_placeholder: Joe Bloggs
+  display_property: dc:contributor
+
+- field_name: maintainer_email
+  label: Maintainer Email
+  form_placeholder: [email protected]
+  display_property: dc:contributor
+  display_snippet: email.html
+  display_email_name_field: maintainer
+  validators: ignore_missing unicode_safe strip_value email_validator
+
+
+resource_fields:
+
+- field_name: url
+  label: URL
+  preset: resource_url_upload
+
+- field_name: name
+  label: Name
+  form_placeholder: eg. January 2011 Gold Prices
+  required: true
+  validators: scheming_required unicode_safe
+
+- field_name: description
+  label: Description
+  form_snippet: markdown.html
+  form_placeholder: Some useful notes about the data
+
+- field_name: format
+  label: Format
+  preset: resource_format_autocomplete

ckanext/scheming/helpers.py

@@ -441,3 +441,21 @@ def scheming_flatten_subfield(subfield, data):
         for k in record:
             flat[prefix + k] = record[k]
     return flat
+
+
+@helper
+def scheming_missing_required_fields(pages, data=None, package_id=None):
+    if package_id:
+        try:
+            data = LocalCKAN().action.package_show(id=package_id)
+        except (NotFound, NotAuthorized):
+            pass
+    if data is None:
+        data = {}
+    missing = []
+    for p in pages:
+        missing.append([
+            f['field_name'] for f in p['fields']
+            if f.get('required') and not data.get(f['field_name'])
+        ])
+    return missing