Skip to content

Question: elegantly streaming an object composed of arrays #174

@deam00n

Description

@deam00n

Hi,

I'm trying to create a custom streamer to parse the following type of file

{
    "attr1": [
        {"a": "data", "id": 1 },
        {"a": "data", "id": 2 }
    ],
    "attr2": [
        {"b": "data", "id": 1 },
        {"b": "data", "id": 2 }
    ]
}

The way I would like it to be chucked is by each element of each array. I understand that streamArray() can do this. However, I also would like to keep the key updated and not as a counter.
Desire output

{key: "attr1", value: {a: "data", id: 1}}
{key: "attr1", value: {a: "data", id: 2}}
{key: "attr2", value: {b: "data", id: 1}}
{key: "attr2", value: {b: "data", id: 2}}

I came up with this solution

class StreamObjectArrays extends StreamBase {
  private _level: number;
  private _lastKey: string | null;

  static make(options?: StreamBase.StreamOptions) {
    return new StreamObjectArrays(options);
  }

  static withParser(options: FilterOptions) {
    return withParser(StreamObjectArrays.make, options);
  }

  constructor(options?: StreamBase.StreamOptions) {
    super(options);
    this._level = 2;
    this._lastKey = null;
  }

  _wait(chunk: any, _: BufferEncoding, callback: TransformCallback) {
    if (chunk.name !== 'startObject') {
      return callback(new Error('Top-level object should be an object.'));
    }
    this._transform = this._filter;
    return this._transform(chunk, _, callback);
  }

  _push(discard: boolean) {
    if (
      this._assembler.stack.length === 2 &&
      this._assembler.stack[1] !== this._lastKey
    ) {
      this._lastKey = String(this._assembler.stack[1]);
    }
  _push(discard: boolean) {
    if (
      this._assembler.stack.length === 2 &&
      this._assembler.stack[1] !== this._lastKey
    ) {
      this._lastKey = String(this._assembler.stack[1]);
    }
    if (this._lastKey !== null) {
        if (this._assembler.current.length) {
          this.push({
            key: this._lastKey,
            value: this._assembler.current.pop(),
          });
        }
    }
  }
}

I had a previous version that was getting the key from _assembler.key in the same way as done in streamObject and updating the level. But I was unable to find the right way to reset the level and the key as once set to level 2 I was never able to get back the top level key from assembler anymore.

    if (this._lastKey === null) {
      this._lastKey = this._assembler.key;
      if (this._assembler.key !== null) {
        this._level = 2;
      }
    } else {
      ...
    }

I feel unsettle of using the assembler stack to retrieve the parent key. I wonder if their is not a better & cleaner solution instead

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions