From 592fa3976fc7c0944d3482348617c63bd50e5908 Mon Sep 17 00:00:00 2001 From: Pete Brink Date: Thu, 13 Mar 2025 06:52:28 -0700 Subject: [PATCH 1/7] First Draft ELISA Criteria Evaluation doc This (incomplete) document is intended to establish both a process and criteria for the evaluation of documentation submitted to ELISA. Signed-off-by: Pete Brink --- .../Criteria-Documentation-Evaluation.md | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 Contributions/Criteria-Documentation-Evaluation.md diff --git a/Contributions/Criteria-Documentation-Evaluation.md b/Contributions/Criteria-Documentation-Evaluation.md new file mode 100644 index 0000000..066bb0e --- /dev/null +++ b/Contributions/Criteria-Documentation-Evaluation.md @@ -0,0 +1,84 @@ +# Criteria for documentation evaluation + +| Desirable Property | Explanation | +| :---- | :---- | +| Clear | Documents must be unambiguously clear to the intended readers (not to anyone, just to the relevant people) | +| Comprehensible | A document is comprehensible if the stakeholders and consumers of the document understand its meaning | +| Detail | Documents must detail complex concepts as a set of elementary measurable concepts | +| Scale | Documents must specify a scale of measure to define the concept | +| Quantity | Given a Scale, documents must specify at least two points of reference on the defined ‘Scale’ to define relative terms (e.g. ‘higher’.) These can be called ‘benchmark’ and ‘target’ in the specification. | +| Qualify | ‘Target’ must specify ‘when’ a performance level is available. Other qualifiers such as ‘when’ and ‘if’ should be explicit if not elsewhere specified. | +| Complete | A complete document is clear without further elaboration because it sufficiently describes the capability and characteristics it is intended to cover. | +| Correct | A document is correct when it is either free from error or it is accurate. | +| Consistent | A document is consistent if it contains no contradictions. | +| Verifiable | A document is verifiable if it is possible to demonstrate that all assertions in the document can be reproduced. There are no hypotheses in the document. The assertions can be verified by either demonstration, observation or review. | +| Reproducible | Reproducibility is specifically about what a document is referencing. When a document describes a project sufficiently that it can be recreated, the document is considered reproducible. | +| Compliant | A document is compliant when it is being used to satisfy a standard. The standard can be from a public source, such as IEC or ISO, or it can be a regulation as from a government, such as FMVSS. | +| Maintainable | A document is maintainable when it can be modified or extended, e.g. by the introduction of new versions or by adding/removing sections. | +| Grammatical | A document is grammatical when it is well formed. The document is in accordance with the productive rules of the grammar of a language. Because there is variation in what is expected, so the grammatical rules followed for a given document should be specified before the document is created. For example, a requirements document could be specified to conform to the syntactic rules of EARS (Easy Approach to Requirement Syntax.) | + +### File metadata + +| Metadata | Explanation | +| :---- | :---- | +| Major Version | X(.0) | +| Minor Version | Example: X(.1 to .6) – Minor version can be an indicator of the maturity of the version X.1 – Initial version 🡪 Send for peer review X.2 – Peer review feedback received 🡪 Mark as X.2, incorporate feedback X.3 – Feedback incorporated 🡪 Mark as X.3, send for expert review X.4 – Expert feedback received 🡪 Mark as X.4, incorporate feedback X.5 – Feedback incorporated 🡪 Mark as X.5, send for acceptance review X.6 – Acceptance review received 🡪 Mark as X.6, incorporate feedback (X+1).0 – Feedback incorporated 🡪 Mark as next Major version (.0), release | +| ID | GUID? | +| Status | (See below) | + +### Status (Example only) + +| Status | +| :---- | +| Proposed | +| In Progress | +| Under Review | +| Verified | +| Released | + +### File header metadata (Example only) + +| Metadata | Explanation | +| :---- | :---- | +| Title | Title of the document | +| Author | Author of the document Can be multiple authors | +| Lead Reviewer | Person responsible for reviewing, gathering feedback and providing feedback to the author | +| Reviewers | Reviewer of the document, should have relevant experience | +| Date | Date of the current version of the document Note that this can differ from the file date | + +## Document review process: + +Entry Conditions: + +* Documents should be numbered either using headings or line numbers enabled for ease of feedback references. +* Documents should have change tracking enabled +* Author requests the review + * Author can use the review criteria to determine if the document is mature enough to be reviewed + * The Author fills out the file metadata + * The Author fills out the file header metadata +* Review lead is selected from the approved list +* Reviewers are selected by the review lead +* Relevant documents are made available + * Perform review + * Violations should be noted as: Critical (C), Severe (S), Minor (M), Observation (O), Question (Q) + * Mark feedback according to criteria + * Feedback should be included as comments unless the change is minor + * Feedback should be assigned to the author to address + * Note to author: + * The reviewers are not directing their feedback at you or your ability. The purpose of this is to find potential bugs and minimize systematic error in the documentation + +Exit Condition: + +* Fewer remaining major defects per page than the agreed exit standard +* Recommendation is one (1) Severe (or higher) defect per page as an initial exit criterion + +## Review Checklist + +| Review ID | Criterion | Reference | Rating | Comments | +| :---- | :---- | :---- | :---- | :---- | +| ID\_01 | Is the document unambiguously clear to the target audience? | Location(s) of the violation of the criterion | C, S, M, O, Q | Feedback on what the problem is and a suggested correction. | +| ID\_02 | Do the stakeholders and consumers of the document understand its meaning? | | | | +| ID\_03 | Is there sufficient detail? | | | | +| ID\_04 | Are complex concepts described as measurable elementary concepts? | | | | +| ID\_05 | Is there a scale defined for any measurable concepts? | | | | +| ID\_06 | … | | | | From 8b58358c5da18fac6046b0d5970c4f234de9a9dc Mon Sep 17 00:00:00 2001 From: Pete Brink Date: Thu, 13 Mar 2025 09:40:08 -0700 Subject: [PATCH 2/7] Updated with feedback Added changes to criteria and grammatical fixes from PR. Updated Version update section for readability Updated document review entry condition criteria. Signed-off-by: Pete Brink --- .../Criteria-Documentation-Evaluation.md | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/Contributions/Criteria-Documentation-Evaluation.md b/Contributions/Criteria-Documentation-Evaluation.md index 066bb0e..166b21f 100644 --- a/Contributions/Criteria-Documentation-Evaluation.md +++ b/Contributions/Criteria-Documentation-Evaluation.md @@ -2,7 +2,7 @@ | Desirable Property | Explanation | | :---- | :---- | -| Clear | Documents must be unambiguously clear to the intended readers (not to anyone, just to the relevant people) | +| Clear | Documents must be unambiguously clear to the intended readers (not to anyone, just to the people active on the project) | | Comprehensible | A document is comprehensible if the stakeholders and consumers of the document understand its meaning | | Detail | Documents must detail complex concepts as a set of elementary measurable concepts | | Scale | Documents must specify a scale of measure to define the concept | @@ -15,14 +15,21 @@ | Reproducible | Reproducibility is specifically about what a document is referencing. When a document describes a project sufficiently that it can be recreated, the document is considered reproducible. | | Compliant | A document is compliant when it is being used to satisfy a standard. The standard can be from a public source, such as IEC or ISO, or it can be a regulation as from a government, such as FMVSS. | | Maintainable | A document is maintainable when it can be modified or extended, e.g. by the introduction of new versions or by adding/removing sections. | -| Grammatical | A document is grammatical when it is well formed. The document is in accordance with the productive rules of the grammar of a language. Because there is variation in what is expected, so the grammatical rules followed for a given document should be specified before the document is created. For example, a requirements document could be specified to conform to the syntactic rules of EARS (Easy Approach to Requirement Syntax.) | +| Grammatical | A document is grammatical when it is well formed. The document is in accordance with the productive rules of the grammar of a language. Because there is variation in what is expected, the grammatical rules followed for a given document should be specified before the document is created. For example, a requirements document could be specified to conform to the syntactic rules of EARS (Easy Approach to Requirement Syntax.) | ### File metadata | Metadata | Explanation | | :---- | :---- | | Major Version | X(.0) | -| Minor Version | Example: X(.1 to .6) – Minor version can be an indicator of the maturity of the version X.1 – Initial version 🡪 Send for peer review X.2 – Peer review feedback received 🡪 Mark as X.2, incorporate feedback X.3 – Feedback incorporated 🡪 Mark as X.3, send for expert review X.4 – Expert feedback received 🡪 Mark as X.4, incorporate feedback X.5 – Feedback incorporated 🡪 Mark as X.5, send for acceptance review X.6 – Acceptance review received 🡪 Mark as X.6, incorporate feedback (X+1).0 – Feedback incorporated 🡪 Mark as next Major version (.0), release | +| Minor Version | Example: X(.1 to .6) – Minor version can be an indicator of the maturity of the version | +|| X.1 – Initial version 🡪 Send for peer review | +|| X.2 – Peer review feedback received 🡪 Mark as X.2, incorporate feedback | +|| X.3 – Feedback incorporated 🡪 Mark as X.3, send for expert review | +|| X.4 – Expert feedback received 🡪 Mark as X.4, incorporate feedback | +|| X.5 – Feedback incorporated 🡪 Mark as X.5, send for acceptance review | +|| X.6 – Acceptance review received 🡪 Mark as X.6, incorporate feedback | +|| (X+1).0 – Feedback incorporated 🡪 Mark as next Major version (.0), release | | ID | GUID? | | Status | (See below) | @@ -41,17 +48,17 @@ | Metadata | Explanation | | :---- | :---- | | Title | Title of the document | -| Author | Author of the document Can be multiple authors | -| Lead Reviewer | Person responsible for reviewing, gathering feedback and providing feedback to the author | -| Reviewers | Reviewer of the document, should have relevant experience | -| Date | Date of the current version of the document Note that this can differ from the file date | +| Author | Author of the document. Can be multiple authors. | +| Lead Reviewer | Person responsible for reviewing, gathering feedback and providing feedback to the author. | +| Reviewers | Reviewer of the document, should have relevant experience. | +| Date | Date of the current version of the document. Note that this can differ from the file date. | ## Document review process: Entry Conditions: -* Documents should be numbered either using headings or line numbers enabled for ease of feedback references. -* Documents should have change tracking enabled +* An enumeration method (e.g. headings, line numbers, etc.) should be available for ease of review document feedback references. +* A mechanism should be available for tracking changes to a review document. * Author requests the review * Author can use the review criteria to determine if the document is mature enough to be reviewed * The Author fills out the file metadata From 71c1db9091c6d68213ebcb88dee59fd28837dce7 Mon Sep 17 00:00:00 2001 From: Pete Brink Date: Mon, 17 Mar 2025 14:55:44 -0700 Subject: [PATCH 3/7] Added Review Ratings section with ratings and explanations. Signed-off-by: Pete Brink --- Contributions/Criteria-Documentation-Evaluation.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/Contributions/Criteria-Documentation-Evaluation.md b/Contributions/Criteria-Documentation-Evaluation.md index 166b21f..92b669f 100644 --- a/Contributions/Criteria-Documentation-Evaluation.md +++ b/Contributions/Criteria-Documentation-Evaluation.md @@ -53,6 +53,16 @@ | Reviewers | Reviewer of the document, should have relevant experience. | | Date | Date of the current version of the document. Note that this can differ from the file date. | +### Review Ratings (Example only) + +| Rating | Explanation | +| :---- | :---- | +| Critical | Showstopper, the document is fundamentally incorrect in its current state and should be reworked. Must be fixed immediately. | +| Severe | Major problem with the document. Must be addressed or fixed before the document can be considered for further review. | +| Minor | Problem with the document, should be corrected, but review can continue. | +| Observation | Comment from the reviewer to the author about how something might be improved. Not a problem with the document. | +| Question | Could be considered as a minor also. This is a question from the reviewer to the author to elaborate or clarify something that is unclear. Could also be considered as a violation of the Clear, Comprehensible, or Detail property. | + ## Document review process: Entry Conditions: From 4d10faab84e9764f938a4833a9afcbb1b1a017bf Mon Sep 17 00:00:00 2001 From: Pete Brink Date: Wed, 19 Mar 2025 13:37:37 -0700 Subject: [PATCH 4/7] Added NA criterion, Added further checklist questions to cover all criteria. Signed-off-by: Pete Brink --- .../Criteria-Documentation-Evaluation.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/Contributions/Criteria-Documentation-Evaluation.md b/Contributions/Criteria-Documentation-Evaluation.md index 92b669f..1a7acb8 100644 --- a/Contributions/Criteria-Documentation-Evaluation.md +++ b/Contributions/Criteria-Documentation-Evaluation.md @@ -62,6 +62,7 @@ | Minor | Problem with the document, should be corrected, but review can continue. | | Observation | Comment from the reviewer to the author about how something might be improved. Not a problem with the document. | | Question | Could be considered as a minor also. This is a question from the reviewer to the author to elaborate or clarify something that is unclear. Could also be considered as a violation of the Clear, Comprehensible, or Detail property. | +| Not Applicable | It does not make sense to use this criterion for this evaluation. | ## Document review process: @@ -77,7 +78,7 @@ Entry Conditions: * Reviewers are selected by the review lead * Relevant documents are made available * Perform review - * Violations should be noted as: Critical (C), Severe (S), Minor (M), Observation (O), Question (Q) + * Violations should be noted as: Critical (C), Severe (S), Minor (M), Observation (O), Question (Q), Not Applicable (N) * Mark feedback according to criteria * Feedback should be included as comments unless the change is minor * Feedback should be assigned to the author to address @@ -93,9 +94,21 @@ Exit Condition: | Review ID | Criterion | Reference | Rating | Comments | | :---- | :---- | :---- | :---- | :---- | -| ID\_01 | Is the document unambiguously clear to the target audience? | Location(s) of the violation of the criterion | C, S, M, O, Q | Feedback on what the problem is and a suggested correction. | +| ID\_01 | Is the document unambiguously clear to the target audience? | Location(s) of the violation of the criterion | C, S, M, O, Q, N | Feedback on what the problem is and a suggested correction. | | ID\_02 | Do the stakeholders and consumers of the document understand its meaning? | | | | | ID\_03 | Is there sufficient detail? | | | | | ID\_04 | Are complex concepts described as measurable elementary concepts? | | | | | ID\_05 | Is there a scale defined for any measurable concepts? | | | | -| ID\_06 | … | | | | +| ID\_06 | Are there benchmarks defined for any quantitative references? | | | | +| ID\_07 | Are there targets defined for any quantitative references? | | | | +| ID\_08 | If there are any performance ratings, does the document specify when those ratings are applicable? | | | | +| ID\_09 | Does the document sufficiently describe the capabilities and characteristics it is intended to cover? | | | | +| ID\_10 | Is the document accurate? | | | | +| ID\_11 | Is the document free from error? | | | | +| ID\_12 | Does the document contain any contradictions? | | | | +| ID\_13 | Can all of any assertions in the document be reproduced? | | | | +| ID\_14 | Are there any hypotheses in the document? | | | | +| ID\_15 | Does the document specify the project sufficiently such that it can be reproduced? | | | | +| ID\_16 | Is the document compliant with any applicable standards? | | | | +| ID\_17 | Is the document organized such that it can be modified or extended? | | | | +| ID\_18 | Is the document compliant with the intended target language? | | | | From 901e0da2db28762033deeb8bd882a8a6a8b15310 Mon Sep 17 00:00:00 2001 From: Pete Brink Date: Thu, 17 Apr 2025 10:40:19 -0700 Subject: [PATCH 5/7] Updated with feedback from review. Signed-off-by: Pete Brink --- Contributions/Criteria-Documentation-Evaluation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Contributions/Criteria-Documentation-Evaluation.md b/Contributions/Criteria-Documentation-Evaluation.md index 1a7acb8..543015f 100644 --- a/Contributions/Criteria-Documentation-Evaluation.md +++ b/Contributions/Criteria-Documentation-Evaluation.md @@ -97,7 +97,7 @@ Exit Condition: | ID\_01 | Is the document unambiguously clear to the target audience? | Location(s) of the violation of the criterion | C, S, M, O, Q, N | Feedback on what the problem is and a suggested correction. | | ID\_02 | Do the stakeholders and consumers of the document understand its meaning? | | | | | ID\_03 | Is there sufficient detail? | | | | -| ID\_04 | Are complex concepts described as measurable elementary concepts? | | | | +| ID\_04 | Are complex concepts described as quantifiable elementary concepts? | | | | | ID\_05 | Is there a scale defined for any measurable concepts? | | | | | ID\_06 | Are there benchmarks defined for any quantitative references? | | | | | ID\_07 | Are there targets defined for any quantitative references? | | | | @@ -107,8 +107,8 @@ Exit Condition: | ID\_11 | Is the document free from error? | | | | | ID\_12 | Does the document contain any contradictions? | | | | | ID\_13 | Can all of any assertions in the document be reproduced? | | | | -| ID\_14 | Are there any hypotheses in the document? | | | | +| ID\_14 | Are there any unsupported hypotheses in the document? | | | | | ID\_15 | Does the document specify the project sufficiently such that it can be reproduced? | | | | -| ID\_16 | Is the document compliant with any applicable standards? | | | | +| ID\_16 | Is the document compliant with any applied standards? | | | | | ID\_17 | Is the document organized such that it can be modified or extended? | | | | | ID\_18 | Is the document compliant with the intended target language? | | | | From ccc8d3155a831a3d81d8abfa56f8daa2000c1c67 Mon Sep 17 00:00:00 2001 From: Pete Brink Date: Thu, 17 Apr 2025 13:09:59 -0700 Subject: [PATCH 6/7] Final updates for first draft. Signed-off-by: Pete Brink --- Contributions/Criteria-Documentation-Evaluation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Contributions/Criteria-Documentation-Evaluation.md b/Contributions/Criteria-Documentation-Evaluation.md index 543015f..1f6c82d 100644 --- a/Contributions/Criteria-Documentation-Evaluation.md +++ b/Contributions/Criteria-Documentation-Evaluation.md @@ -11,7 +11,7 @@ | Complete | A complete document is clear without further elaboration because it sufficiently describes the capability and characteristics it is intended to cover. | | Correct | A document is correct when it is either free from error or it is accurate. | | Consistent | A document is consistent if it contains no contradictions. | -| Verifiable | A document is verifiable if it is possible to demonstrate that all assertions in the document can be reproduced. There are no hypotheses in the document. The assertions can be verified by either demonstration, observation or review. | +| Verifiable | A document is verifiable if it is possible to demonstrate that all assertions in the document can be reproduced. There are no unsupported hypotheses in the document. The assertions can be verified by either demonstration, observation or review. Claims or hypotheses must be backed by references to the associated source code or published documentation (ideally in the public domain), with specific, reproducible tests to demonstrate or confirm the claim. | | Reproducible | Reproducibility is specifically about what a document is referencing. When a document describes a project sufficiently that it can be recreated, the document is considered reproducible. | | Compliant | A document is compliant when it is being used to satisfy a standard. The standard can be from a public source, such as IEC or ISO, or it can be a regulation as from a government, such as FMVSS. | | Maintainable | A document is maintainable when it can be modified or extended, e.g. by the introduction of new versions or by adding/removing sections. | @@ -30,7 +30,7 @@ || X.5 – Feedback incorporated 🡪 Mark as X.5, send for acceptance review | || X.6 – Acceptance review received 🡪 Mark as X.6, incorporate feedback | || (X+1).0 – Feedback incorporated 🡪 Mark as next Major version (.0), release | -| ID | GUID? | +| ID | UUID | | Status | (See below) | ### Status (Example only) @@ -111,4 +111,4 @@ Exit Condition: | ID\_15 | Does the document specify the project sufficiently such that it can be reproduced? | | | | | ID\_16 | Is the document compliant with any applied standards? | | | | | ID\_17 | Is the document organized such that it can be modified or extended? | | | | -| ID\_18 | Is the document compliant with the intended target language? | | | | +| ID\_18 | Is the document compliant with the applicable rules of syntax or grammar? (e.g. The Trustable Software Framework rules about Statements.) | | | | From d36a0721b6b00ae1f15eefdf27874fc3a264339f Mon Sep 17 00:00:00 2001 From: Pete Brink Date: Fri, 18 Apr 2025 15:53:48 -0700 Subject: [PATCH 7/7] Added license. Signed-off-by: Pete Brink --- Contributions/Criteria-Documentation-Evaluation.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Contributions/Criteria-Documentation-Evaluation.md b/Contributions/Criteria-Documentation-Evaluation.md index 1f6c82d..6ffff6a 100644 --- a/Contributions/Criteria-Documentation-Evaluation.md +++ b/Contributions/Criteria-Documentation-Evaluation.md @@ -1,4 +1,4 @@ -# Criteria for documentation evaluation +# Criteria for Open-Source Documentation Evaluation | Desirable Property | Explanation | | :---- | :---- | @@ -112,3 +112,5 @@ Exit Condition: | ID\_16 | Is the document compliant with any applied standards? | | | | | ID\_17 | Is the document organized such that it can be modified or extended? | | | | | ID\_18 | Is the document compliant with the applicable rules of syntax or grammar? (e.g. The Trustable Software Framework rules about Statements.) | | | | + +This work is licensed under CC BY 4.0 \ No newline at end of file