What is the behavior of TokenizerContext.previous

[gSarciotto]

Hey, so I have been writing an extension to parse the following: [[some-alphanumeric-value]]. I want to check that the first [ is not escaped (\[) and if it is then I want to nok

The problem is that I am finding the behavior of the previous key in the tokenizer context pretty hard to understand.,The following entries have the previous with the following values in the starting state:

• a \[[b]] c => 32 (space)
• a
  [[b]] c => 32

In the first case I expected the value 92 (backlash) while on the second one I expected either one of the line endings or null. I looked around to see how other extensions dealt with this case , however this doesn't work for me, the values in the events array are all of type data with previous value of 32 as well :/

For reference, this is the code for the starting state of the tokenizer:

function tokenizer(
    this: TokenizeContext,
    effects: Effects,
    ok: State,
    nok: State
): ReturnType<Tokenizer> {
    const tokenizeContext = this;
    return start;

    function start(code: Code): State | undefined {
        assert(
            code === micromarkCodes.leftSquareBracket,
            "expected '['"
        );
        console.log("previous", tokenizeContext.previous);
        const previousCharacter = tokenizeContext.previous;
        const isPreviousCharacterValid =
            previousCharacter === micromarkCodes.leftSquareBracket || // the leftSquareBracket is so that it doesnt throw on the second [ if it fails to create token on the first [
            previousCharacter === null ||
            unicodeWhitespace(previousCharacter);
        assert(
            isPreviousCharacterValid,
            ` previous character ${tokenizeContext.previous} is not valid`
        );

        effects.enter(micromarkTypes.obsidianWikilink);
        effects.enter(micromarkTypes.obsidianWikilinkMarker);
        effects.consume(code);
        console.log("start end");

        return parseSecondLeftSquareBracket;
    }

[wooorm]

Starting a tokenizer, as what? Typically they are hooked into looking for a particular character? Or are a particular kind? Can you maybe share some more code?

The example you found is how it is indeed almost always handled. Say, a : cannot be allowed literally, or it can be allowed by only when it is an escape?
So from your original point, you’d probably want to do exactly the same?

But, do note that [ is a crowded space! Many things can start with that!

[gSarciotto]

The tokenizer is being hooked on left square brackets:

export function micromarkObsidianWikilink(): Extension {
    const obsidianWikilinkConstruct: Construct = {
        name: "obsidianWikilink",
        tokenize: obsidianWikilinkTokenizer
    };

    return {
        text: { [micromarkCodes.leftSquareBracket]: obsidianWikilinkConstruct }
    };
}

On my first example, I think I managed to understand why previous was empty space, I have to escape it twice since the input was a JS string (a \\[[b]] c), so I think that since there was only one \ at first it just ignored since [ doesnt mean anything and the previous character was a space. Using the code from the directive extension I managed to make it work with escaped [.
However I still dont understand on why the second input example (with the linebreak), why previous has value of 32 (space) instead of one of the linebreaks values (-5, -4 or -3)

[wooorm]

Can you paste the second example in markdown code? As you see, escapes are tough. It is unclear to me what you are doing

[gSarciotto]

What i want to test is: the only valid characters before the first [ are null, end lines, spaces or tabs

this is the first line
[[this is inside the brackets]] this is after the brackets

So for the markdown above, I expected the character just before [ to be one of the line endings, however it is space (code 32) for some reason

[wooorm]

I made code that runs based on the things you’ve shared. example.js:

/**
 * @typedef {import('micromark-util-types').Code} Code
 * @typedef {import('micromark-util-types').Construct} Construct
 * @typedef {import('micromark-util-types').Effects} Effects
 * @typedef {import('micromark-util-types').Extension} Extension
 * @typedef {import('micromark-util-types').State} State
 * @typedef {import('micromark-util-types').TokenizeContext} TokenizeContext
 * @typedef {import('micromark-util-types').Tokenizer} Tokenizer
 */

import fs from 'node:fs/promises'
import {ok as assert} from 'devlop'
import {micromark} from 'micromark'
import {codes} from 'micromark-util-symbol'

const file = String(await fs.readFile('example.md'))
const result = micromark(file, {extensions: [micromarkObsidianWikilink()]})

console.log(result)

/**
 * @returns {Extension}
 */
function micromarkObsidianWikilink() {
  /** @type {Construct} */
  const obsidianWikilinkConstruct = {
    name: 'obsidianWikilink',
    tokenize: obsidianWikilinkTokenizer
  }

  return {
    text: {[codes.leftSquareBracket]: obsidianWikilinkConstruct}
  }
}

/**
 * @this {TokenizeContext}
 * @type {Tokenizer}
 */
function obsidianWikilinkTokenizer(effects, ok, nok) {
  const self = this
  return start

  /**
   * @param {Code} code
   * @returns {State | undefined}
   */
  function start(code) {
    console.log('p:', self.previous)
    assert(code === codes.leftSquareBracket, "expected '['")
    return nok(code)
  }
}

example.md:

this is the first line
[[this is inside the brackets]] this is after the brackets

Now running that yields:

p: -4
p: 91
<p>this is the first line
[[this is inside the brackets]] this is after the brackets</p>

This makes sense to me.

Can you please explain what you want differently, or how to reproduce your question?