Skip to content

Commit 45b4681

Browse files
localjolocaljo
authored
Add spelling check (#1)
* Add spelling check * Add option to add word to ignore list with -i flag * Add workaround for number ranges and dimensions * Make spelling errors fatal * Add silent mode * Add custom dictionary support
1 parent 8419e02 commit 45b4681

5 files changed

Lines changed: 214 additions & 34 deletions

File tree

‎README.md‎

Lines changed: 32 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# quality-docs
1+
# `quality-docs`
22

33
A CLI tool to check the quality of writing in your project's markdown or plain text documentation.
44

@@ -20,6 +20,10 @@ quality-docs {,**/}*.md
2020

2121
### Options
2222

23+
#### Silent mode
24+
25+
The `-s`, `--silent` flag enables silent mode which mutes warnings. Fatal errors are displayed in silent mode.
26+
2327
#### Rules
2428

2529
The `-r`, `--rules` flag passes in a JSON file to override default linting rules.
@@ -32,12 +36,16 @@ The override uses this format (without comments):
3236

3337
```js
3438
{
39+
"customDictionary": "sample.dic",
3540
"lint": { // Options for remark-lint rules
3641
"list-item-indent": false
3742
},
3843
"readability": { // Options for remark-readability
3944
"minWords": 7
4045
},
46+
"units": [ // Acceptable units on the end of numbers or ranges
47+
"GB", "MB", "KB", "K", "am", "pm", "in", "ft"
48+
],
4149
"ignore": [ // Words and phrases to ignore
4250
"address",
4351
"function",
@@ -52,31 +60,44 @@ The override uses this format (without comments):
5260
}
5361
```
5462

55-
See [sample-rules-override.json](./sample-rules-override.json) for an example.
63+
See [`sample-rules-override.json`](./sample-rules-override.json) for an example.
64+
65+
#### Ignore
66+
67+
When used along with the rules flag, the `-i`, `--ignore` flag adds a word to the rules file's ignore list. Example;
68+
69+
```bash
70+
$ quality-docs {,**/}*.md --rules sample-rules-override.json --ignore irregardless
71+
Added 'irregardless' to ignore list. Don't forget to commit the changes to sample-rules-override.json.
72+
```
5673
5774
### Reports
5875
59-
The tool prints a report of writing quality issues with their location and description. For example, a warning in README.md from line 23 column 9 to line 23 column 16 prints;
76+
The tool prints a report of writing quality issues with their location and description. For example, a warning in `README.md` from line 23 column 9 to line 23 column 16 prints;
6077
6178
```bash
6279
README.md
6380
23:9-23:16 warning Replace “utilize” with “use”
6481
```
6582
83+
### Custom Dictionary
84+
85+
By default, `quality-docs` spell checks documents against [a US English dictionary](https://github.com/wooorm/dictionaries/dictionaries/en_US). To extend the built in dictionary with custom English terms related to your project(s), add a [hunspell format](http://linux.die.net/man/4/hunspell) `.dic` file to your project, and reference it with the `customDictionary` key in the rules override JSON file. See [`sample.dic`](./sample.dic) for an example. (Note: `quality-docs` uses the [US English affix file](https://github.com/wooorm/dictionaries/blob/master/dictionaries/en_US/index.aff) to check for valid variants of dictionary words. Non-English characters or prefix/suffix rules are not supported.)
86+
6687
### Changing Default Rules
6788
6889
`quality-docs` ships with an opinionated set of rules to improve your writing and we recommend you use the defaults. If you want to make an exception, you have three options;
6990
7091
1. [Exclude documentation files from the glob argument](http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm#STANDARD-WILDCARDS).
71-
2. Use the `--rules` flag to pass in a JSON file with overrides. See [sample-rules-override.json](./sample-rules-override.json).
72-
3. Use [remark-message-control marks](https://github.com/wooorm/remark-message-control) to turn on/off specific rules for individual documents or text nodes.
92+
2. Use the `--rules` flag to pass in a JSON file with overrides. See [`sample-rules-override.json`](./sample-rules-override.json).
93+
3. Use [`remark-message-control` marks](https://github.com/wooorm/remark-message-control) to turn on/off specific rules for individual documents or text nodes.
7394
7495
## Other Notes
7596
76-
This tool uses [retext](https://github.com/wooorm/retext) to check the quality of writing in your project's documentation using these plugins;
97+
This tool uses [`retext`](https://github.com/wooorm/retext) to check the quality of writing in your project's documentation using these plugins;
7798

78-
* [remark-lint](https://github.com/wooorm/remark-lint) checks for proper markdown formatting.
79-
* [retext-readability](https://github.com/wooorm/retext-readability) checks the reading level of the whole document.
80-
* [retext-simplify](https://github.com/wooorm/retext-simplify) warns on over-complicated phrases.
81-
* [retext-equality](https://github.com/wooorm/retext-equality) warns on insensitive, inconsiderate language.
82-
* [retext-intensify](https://github.com/wooorm/retext-intensify) warns on filler, weasel and hedge words.
99+
* [`remark-lint`](https://github.com/wooorm/remark-lint) checks for proper markdown formatting.
100+
* [`retext-readability`](https://github.com/wooorm/retext-readability) checks the reading level of the whole document.
101+
* [`retext-simplify`](https://github.com/wooorm/retext-simplify) warns on complicated phrases.
102+
* [`retext-equality`](https://github.com/wooorm/retext-equality) warns on insensitive, inconsiderate language.
103+
* [`retext-intensify`](https://github.com/wooorm/retext-intensify) warns on filler, weasel and hedge words.

‎index.js‎

Lines changed: 91 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -3,34 +3,45 @@ const _ = require('lodash');
33
const argv = require('minimist')(process.argv.slice(2));
44
const concise = require('retext-intensify');
55
const control = require('remark-message-control');
6+
const en_US = require('dictionary-en-us');
67
const equality = require('retext-equality');
78
const fs = require('fs');
89
const lint = require('remark-lint');
910
const map = require("async/map");
1011
const meow = require('meow');
12+
const path = require('path');
1113
const readability = require('retext-readability');
1214
const remark = require('remark');
1315
const remark2retext = require('remark-retext');
1416
const report = require('vfile-reporter');
1517
const retext = require('retext');
1618
const simplify = require('retext-simplify');
19+
const spell = require('retext-spell');
20+
const toString = require('nlcst-to-string');
1721
const toVFile = require('to-vfile');
22+
const visit = require('unist-util-visit');
1823

1924
const cli = meow(`
2025
Usage
2126
$ quality-docs <glob>
2227
2328
Options
2429
-r, --rules A JSON file to override default linting rules.
30+
-i, --ignore A word or phrase to ignore and add to the rules file's list.
31+
-s, --silent Silent mode. Mutes warnings and only shows fatal errors.
2532
2633
Examples
2734
$ quality-docs --rules docStyle.json
2835
`, {
2936
alias: {
30-
r: 'rules'
37+
r: 'rules',
38+
i: 'ignore',
39+
s: 'silent'
3140
}
3241
});
3342

43+
var silent = cli.flags.silent || false;
44+
3445
// Build array of files that match input glob
3546
var docFiles = [];
3647
cli.input.forEach((file) => { if (!file.includes('*')) docFiles.push(file); });
@@ -44,18 +55,90 @@ var rules = cli.flags.rules ? JSON.parse(
4455
fs.readFileSync(cli.flags.rules, 'utf8')
4556
) : {};
4657

58+
var ignore = cli.flags.ignore;
59+
60+
// If --rules and --ignore are specified, update the rules with new ignore
61+
if (rules.ignore && ignore) {
62+
var isValidString = /^[ A-Za-z0-9_@./#&+-]*$/.test(ignore);
63+
var isUnique = !_.includes(rules.ignore, ignore);
64+
if (isValidString && isUnique) {
65+
rules.ignore.push(ignore);
66+
rules.ignore.sort();
67+
fs.writeFile(cli.flags.rules, JSON.stringify(rules, null, 2), function(err) {
68+
if(err) {
69+
return console.log(err);
70+
}
71+
console.log('Added \'' + ignore + '\' to ignore list. Don\'t forget to commit the changes to ' + cli.flags.rules + '.');
72+
});
73+
} else {
74+
console.log('Could not add \'' + ignore + '\' to ignore list. Please add it manually.');
75+
}
76+
}
77+
78+
var dictionary = en_US;
79+
if (rules.customDictionary && rules.customDictionary.length >= 1) {
80+
dictionary = function (cb) {
81+
en_US(function(err, primary) {
82+
fs.readFile(path.join(process.cwd(), rules.customDictionary), function (err, customDic) {
83+
cb(err, !err && {aff: primary.aff, dic: Buffer.concat([primary.dic, customDic])});
84+
});
85+
});
86+
}
87+
}
88+
4789
map(docFiles, toVFile.read, function(err, files){
48-
var allResults = [];
4990
var hasErrors = false;
5091

51-
files.forEach((file) => {
92+
map(files, checkFile, function(err, results) {
93+
console.log(report(err || results, {silent: silent}));
94+
95+
// Check for errors and exit with error code if found
96+
results.forEach((result) => {
97+
result.messages.forEach((message) => {
98+
if (message.fatal) hasErrors = true;
99+
});
100+
});
101+
if (hasErrors) process.exit(1);
102+
103+
})
104+
105+
function checkFile(file, cb) {
52106
remark()
53107
.use(lint, rules.lint || {})
54108
.use(remark2retext, retext() // Convert markdown to plain text
55109
.use(readability, rules.readability || {})
56110
.use(simplify, {ignore: rules.ignore || []})
57111
.use(equality, {ignore: rules.ignore || []})
58112
.use(concise, {ignore: rules.ignore || []})
113+
.use(function () {
114+
return function (tree) {
115+
visit(tree, 'WordNode', function (node, index, parent) {
116+
var word = toString(node);
117+
118+
var unitArr = rules.units || ['GB', 'MB', 'KB', 'K', 'am', 'pm', 'in', 'ft'];
119+
unitArr = unitArr.concat(['-', 'x']); // Add ranges and dimensions to RegExp
120+
var units = unitArr.join('|');
121+
122+
// Ignore email addresses and the following types of non-words:
123+
// 500GB, 8am-6pm, 10-11am, 1024x768, 3x5in, etc
124+
var unitFilter = new RegExp('^\\d+(' + units + ')+\\d*(' + units + ')*$','i');
125+
var emailFilter = new RegExp('^[-a-z0-9~!$%^&*_=+}{\'?]+(\.[-a-z0-9~!$%^&*_=+}{\'?]+)*@([a-z0-9_][-a-z0-9_]*(\.[-a-z0-9_]+)*\.(aero|arpa|biz|com|coop|edu|gov|info|int|mil|museum|name|net|org|pro|travel|mobi|[a-z][a-z])|([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}))(:[0-9]{1,5})?$', 'i');
126+
if (emailFilter.test(word) || unitFilter.test(word)) {
127+
parent.children[index] = {
128+
type: 'SourceNode',
129+
value: word,
130+
position: node.position
131+
};
132+
}
133+
134+
});
135+
};
136+
})
137+
.use(spell, {
138+
dictionary: dictionary,
139+
ignore: rules.ignore || [],
140+
ignoreLiteral: true
141+
})
59142
)
60143
.use(control, {name: 'quality-docs', source: [
61144
'remark-lint',
@@ -67,28 +150,15 @@ map(docFiles, toVFile.read, function(err, files){
67150
.process(file, function (err, results) {
68151
var filteredMessages = [];
69152
results.messages.forEach((message) => {
70-
// Make equality and simplify rules easier to flag as fatal
71-
if (/(equality|simplify)/.test(message.source)) {
72-
message.ruleId = message.source;
73-
}
74-
if (rules.fatal && _.includes(rules.fatal, message.ruleId)) {
153+
var hasFatalRuleId = _.includes(rules.fatal, message.ruleId);
154+
var hasFatalSource = _.includes(rules.fatal, message.source);
155+
if (rules.fatal && (hasFatalRuleId || hasFatalSource)) {
75156
message.fatal = true;
76157
}
77158
filteredMessages.push(message);
78159
});
79160
results.messages = filteredMessages;
80-
81-
allResults.push(results);
161+
cb(null, results);
82162
});
83-
});
84-
85-
console.log(report(err || allResults));
86-
87-
allResults.forEach((result) => {
88-
result.messages.forEach((message) => {
89-
if (message.fatal) hasErrors = true;
90-
});
91-
});
92-
93-
if (hasErrors) process.exit(1);
163+
}
94164
});

‎package.json‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,11 @@
2727
"homepage": "https://github.com/SparkartGroupInc/quality-docs#readme",
2828
"dependencies": {
2929
"async": "^2.0.1",
30+
"dictionary-en-us": "^1.0.2",
3031
"lodash": "^4.14.0",
3132
"meow": "^3.7.0",
3233
"minimist": "^1.2.0",
34+
"nlcst-to-string": "^2.0.0",
3335
"remark": "^5.0.1",
3436
"remark-lint": "^4.0.2",
3537
"remark-message-control": "^2.0.2",
@@ -38,8 +40,10 @@
3840
"retext-equality": "^2.3.2",
3941
"retext-intensify": "^2.0.0",
4042
"retext-readability": "^3.0.0",
41-
"retext-simplify": "^2.0.0",
43+
"retext-simplify": "^3.0.0",
44+
"retext-spell": "wooorm/retext-spell#9e2c9a9",
4245
"to-vfile": "^1.0.0",
46+
"unist-util-visit": "^1.1.0",
4347
"vfile-reporter": "^2.0.1"
4448
}
4549
}

‎sample-rules-override.json‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
{
2+
"customDictionary": "sample.dic",
23
"lint": {
34
"list-item-indent": false,
45
"maximum-line-length": false,
@@ -11,6 +12,16 @@
1112
"age": 18,
1213
"minWords": 7
1314
},
15+
"units": [
16+
"GB",
17+
"MB",
18+
"KB",
19+
"K",
20+
"am",
21+
"pm",
22+
"in",
23+
"ft"
24+
],
1425
"ignore": [
1526
"address",
1627
"attempt",
@@ -28,9 +39,9 @@
2839
"multiple",
2940
"option",
3041
"previous",
42+
"request",
3143
"require",
3244
"requires",
33-
"request",
3445
"submit",
3546
"transmit",
3647
"try",
@@ -72,6 +83,7 @@
7283
"retext-readability",
7384
"retext-simplify",
7485
"rule-style",
86+
"spelling",
7587
"table-pipe-alignment",
7688
"table-pipes",
7789
"unordered-list-marker-style"

0 commit comments

Comments
 (0)