RFC: Escaping doc comments inside doc comments

[sindresorhus]

It's useful to users to include code examples in documentation comments. Sometimes you need to illustrate what the output of different code examples are, so you put it in a comment below, and if it's multiple lines, you use a block comment. The problem is that TSDoc doesn't correctly parse this:

/**
 * ```
 * getAverage()
 * /* 'some output' */
 * ```
 */
function getAverage(x, y) {
    return (x + y) / 2.0;
}

I think it should handle block comments when they're inside a code block in the documentation comment.

[Gerrit0]

This isn't a TSDoc problem. It's a JavaScript problem. JavaScript (and therefore TypeScript) does not support nested block comments, so the whole comment ends at 'some output' */

[octogonz]

We could address this by providing an escaping mechanism like *\/. We need to formalize escaping in TSDoc but I've been procrastinating it because the problem is very thorny due to the onion skin of JSDoc tags, Markdown, HTML, and /* */. HTML is the only one of those four things that provides sane escaping mechanisms, but it would be */... (This is the problem with de facto standards heheh.)

TSDoc escaping will need to depend heavily on context. This works against our goal that the rendered result should be predictable by a person who doesn't have access to an interactive renderer. For example, the most convenient rule is probably: "Inside a markdown code fence, every character is preserved literally except for \/ which encodes a slash to support nested comments." But this rule is nearly impossible to infer. You would have to read the TSDoc spec and memorize it. So instead we would much prefer a rule like "Inside a markdown code fence, the \ character escapes any printable character that follows it." That's easy to infer and remember. But the quoted code itself may rely heavily on backslash escapes, which would cause nasty double-encodings. Perhaps / is not so bad!

[octogonz]

This problem is fairly difficult. Users have a very strong expectation that a fenced code block is reproduced literally. For example, suppose we choose \ as our escape character. We get something like this:

/**
 * ```ts
 * /**
 *  * Matches "abc\/def" or "*\\/abc" but not "C:\\foo\\bar"
 *  *\/
 * function foo(path) {
 *   return /[a-z*]+\/[a-z]+$/.test(path);
 * }
 * ```
 */

If we allow \/ to be escaped by the /* */ framing, it needs to be just the / character. It would be wildly unacceptable to expand every single \ escape before TSDoc starts parsing the input, because \ is a standard escape character used by Markdown, JSDoc, and also source code.

But this means \/ gets expanded before \\. So the second layer of the onion skin will look like this:

```ts
/**
 * Matches "abc/def" or "*\/abc" but not "C:\\foo\\bar"
 */
function foo(path) {
  return /[a-z*]+/[a-z]+$/.test(path);
}
```

It's highly confusing why the / in the RegExp got expanded but not the "C:\\, but I suppose someone could memorize that / is special. It's particularly counterintuitive with "*\\/abc" where \\/ gets processed from right-to-left, whereas a casual reader would assume the opposite.

The */ might be more acceptable merely because people don't use HTML escapes as much.

[octogonz]

I wonder if *//* could be used as an escape character. 🤭 That at least doesn't conflict with any possible expression inside a comment.

[octogonz]

I thought of a possibly better solution to this problem!

Suppose we have an API like this:

function isDocComment(s: string): boolean {
    return /^\/\*.*\*\/$/.test(s);
}

...and we want to show docs like this:

---

isDocComment()

function isDocComment(s: string): boolean;

Returns true if s is enclosed in /* and */ comment characters.

Example

// This prints "true":
console.log(isDocComment("/** @public */"));

---

When we try to write TSDoc like this...

/**
 * Returns `true` if `s` is enclosed in `/*` and `*/`  <=== ERROR
 * comment characters.
 * 
 * @example
 * ```ts
 * // This prints "true":
 * console.log(isDocComment("/** @public */"));        <=== ERROR
 * ```
 */

....we encounter a syntax error because JavaScript provides no mechanism for */ to appear inside a /** */ doc comment. But in well-formed doc comments, the * framing always has a space after it. Idea: What if we adopted a convention that *+ continues the previous line? Then we could escape it like this:

/**
 * Returns `true` if `s` is enclosed in `/*` and `*
 *+/`
 * comment characters.
 * 
 * @example
 * ```ts
 * // This prints "true":
 * console.log(isDocComment("/** @public *
 *+/"));
 * ```
 */

The TSDoc parser would discard the newline and *+ when it extracts the content from the /** */ framing, and then the emitter could inject these escapes as needed.

If you want to line it up horizontally for readability, maybe we could also allow multiple + characters, like this:

/**
 * Returns `true` if `s` is enclosed in `/*` and `*
 *+++++++++++++++++++++++++++++++++++++++++++++++++/`
 * comment characters.
 * 
 * @example
 * ```ts
 * // This prints "true":
 * console.log(isDocComment("/** @public *
 *++++++++++++++++++++++++++++++++++++++++/"));
 * ```
 */

(The delimiter doesn't have to be + . It could be any symbol other than * or /, however we should avoid characters that already have special meaning for Markdown or HTML. That eliminates a lot of options, leaving us mostly with +, -, or ^.)

What do you think? This seems better than the earlier proposals.

@rbuckton @iansan5653 @EisenbergEffect FYI

[iansan5653]

I read your comment first, and then had to go back and read all the previous comments to be able to to understand what that escaping is actually doing. Of course, I'm not complaining about having to read, but I think this would be very difficult to understand. It's one of those things that comes up enough that it would need to be used every so often, but not often enough that anyone would have any idea what they're looking at when they see it.

Is the only use case when you would ever use + for continuation when you have */ in a comment? If it's really only this one case, maybe it would be better to just reserve a special character sequence that always (no matter where it is in a TSDoc comment) resolves to */. Something like &COMMENT_CLOSE; maybe?

[octogonz]

maybe it would be better to just reserve a special character sequence that always (no matter where it is in a TSDoc comment) resolves to */. Something like &COMMENT_CLOSE; maybe?

How would your approach represent this case? 🤓

/**
 * Here is an example of how to escape `/
 *+*` using TSDoc:
 *
 * ```ts
 * /**
 *  * Returns `true` if `s` is enclosed in `/*` and `*
 *  *+/`
 *  * comment characters.
 *  * 
 *  * @example
 *  * ```ts
 *  * // This prints "true":
 *  * console.log(isDocComment("/** @public *
 *  *+/"));
 *  * ```
 *  */
 * ```
 */

[octogonz]

Is the only use case when you would ever use + for continuation when you have */ in a comment?

It doesn't have to be a comment though; e.g. it could be a glob or RegExp with those characters.

BTW this issue might also arise if someone is using DocComment.emitAsTsdoc() to generate comments containing arbitrary strings. If */ isn't escaped properly, the output would have an actual JavaScript syntax error.

[octogonz]

This feature could also be used to wrap long lines, e.g. to satisfy an ESLint rule that limits line length.

/**
 * For more info please see https://github.com/micro
 *+soft/tsdoc/issues/166
 */

[octogonz]

Here's a poll where you can vote on the syntax: 🙂

https://twitter.com/octogonz_/status/1204589348525527040?s=19

[rbuckton]

@DanielRosenwasser, @sandersn: What do you think of this? If *+ essentially becomes a JSDoc/TSDoc line continuation character, we might want to update our parser/language service to support it for quick info.

[iansan5653]

How would your approach represent this case?

I wrote a comment before but deleted it because I misunderstood what you're asking.

I don't know if we necessarily need to support this scenario - someone could just link to the spec page. Your escaping method does support this, but at the expense of being pretty difficult to read and write. Even your own example has a syntax error in it - it should be:

/**
 * Here is an example of how to escape `/
 *+*` using TSDoc:
 *
 * ```ts
 * /**
 *  * Returns `true` if `s` is enclosed in `/*` and `*
 *  *+/`
 *  * comment characters.
 *  * 
 *  * @example
 *  * ```ts
 *  * // This prints "true":
 *  * console.log(isDocComment("/** @public *
 *  *+/"));
 *  * ```
 *  *
 *+/
 * ```
 */