Skip to content

Use an array for building up mappings in the sourcemap generator. #43785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Use an array for building up mappings in the sourcemap generator. #43785

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4c86404
Use an array for building up mappings in the sourcemap generator.
DanielRosenwasser Apr 23, 2021
312ce16
Ugh.
DanielRosenwasser Apr 23, 2021
799bbbe
Just calculate the result at the end of commitPendingMapping.
DanielRosenwasser Apr 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions src/compiler/sourcemap.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ namespace ts {
const names: string[] = [];
let nameToNameIndexMap: ESMap<string, number> | undefined;
let mappings = "";
let mappingSegmentsBuffer: string[] = [];

// Last recorded and encoded mappings
let lastGeneratedLine = 0;
Expand Down Expand Up @@ -221,7 +222,7 @@ namespace ts {
if (lastGeneratedLine < pendingGeneratedLine) {
// Emit line delimiters
do {
mappings += ";";
mappingSegmentsBuffer.push(";");
lastGeneratedLine++;
lastGeneratedCharacter = 0;
}
Expand All @@ -231,34 +232,37 @@ namespace ts {
Debug.assertEqual(lastGeneratedLine, pendingGeneratedLine, "generatedLine cannot backtrack");
// Emit comma to separate the entry
if (hasLast) {
mappings += ",";
mappingSegmentsBuffer.push(",");
}
}

// 1. Relative generated character
mappings += base64VLQFormatEncode(pendingGeneratedCharacter - lastGeneratedCharacter);
mappingSegmentsBuffer.push(base64VLQFormatEncode(pendingGeneratedCharacter - lastGeneratedCharacter));
lastGeneratedCharacter = pendingGeneratedCharacter;

if (hasPendingSource) {
// 2. Relative sourceIndex
mappings += base64VLQFormatEncode(pendingSourceIndex - lastSourceIndex);
mappingSegmentsBuffer.push(base64VLQFormatEncode(pendingSourceIndex - lastSourceIndex));
lastSourceIndex = pendingSourceIndex;

// 3. Relative source line
mappings += base64VLQFormatEncode(pendingSourceLine - lastSourceLine);
mappingSegmentsBuffer.push(base64VLQFormatEncode(pendingSourceLine - lastSourceLine));
lastSourceLine = pendingSourceLine;

// 4. Relative source character
mappings += base64VLQFormatEncode(pendingSourceCharacter - lastSourceCharacter);
mappingSegmentsBuffer.push(base64VLQFormatEncode(pendingSourceCharacter - lastSourceCharacter));
lastSourceCharacter = pendingSourceCharacter;

if (hasPendingName) {
// 5. Relative nameIndex
mappings += base64VLQFormatEncode(pendingNameIndex - lastNameIndex);
mappingSegmentsBuffer.push(base64VLQFormatEncode(pendingNameIndex - lastNameIndex));
lastNameIndex = pendingNameIndex;
}
}

mappings = mappingSegmentsBuffer.join("");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In local experimentation, I've found that I can get about a 40% speed up by batching the char code conversion:

function base64VLQFormatEncode2(value) {
  if (value < 0) {
      value = ((-value) << 1) + 1;
  }
  else {
      value = value << 1;
  }

  // Encode 5 bits at a time starting from least significant bits
  const result = [];
  do {
      let currentDigit = value & 31; // 11111
      value = value >> 5;
      if (value > 0) {
          // There are still more digits to decode, set the msb (6th bit)
          currentDigit = currentDigit | 32;
      }
      result.push(base64FormatEncode(currentDigit));
  } while (value > 0);

  return result;
}

function buildMappingStr() {
  const charCodes = [];
  // Mock representation of adding a segment, for benchmarking
  for (let i = 0; i < target; i ++) {
    charCodes.push(...base64VLQFormatEncode2(i));
  }
  let str = '';
  const segmentLength = 4096;
  for (let i = 0, len = charCodes.length; i < len; i += segmentLength) {
    const segment = String.fromCharCode.apply(null, charCodes.slice(i, i + segmentLength));
    str += segment;
  }
  return str;
}

Copy link
Contributor

@dmichon-msft dmichon-msft May 6, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are encoding a very large number of mappings (>100000), it gets faster to use an encoder with its own buffer:

class SourceMapEncoder {
  constructor() {
    this._encoded = '';
    this._buffer = new Uint8Array(1024);
    this._bufferPos = 0;
  }

  get encoded() {
    if (this._bufferPos > 0) {
      this._commit(); 
    }
    return this._encoded;
  }

  /**
   * @param {number} code
   */
  appendCharCode(code) {
    this._buffer[this._bufferPos] = code;
    this._bufferPos++;
    
    if (this._bufferPos === this._buffer.length) {
      this._commit();
    }
  }

  /**
   * @param {number} value
   */
  appendBase64VLQ(value) {
    if (value < 0) {
        value = ((-value) << 1) + 1;
    }
    else {
        value = value << 1;
    }
  
    // Encode 5 bits at a time starting from least significant bits
    do {
        let currentDigit = value & 31; // 11111
        value = value >> 5;
        if (value > 0) {
            // There are still more digits to decode, set the msb (6th bit)
            currentDigit = currentDigit | 32;
        }
        this.appendCharCode(formatArr[currentDigit]);
    } while (value > 0);
  }

  reset() {
    this._encoded = '';
    this._bufferPos = 0;
  }

  _commit() {
    this._encoded += String.fromCharCode.apply(null, this._buffer.slice(0, this._bufferPos));
    this._bufferPos = 0;
  }
}

Call .appendBase64VLQ(value) for each mapping to append, then retrieve .encoded at the end.
Still trying to figure out why the overhead is so much higher for small counts

Copy link
Member Author

@DanielRosenwasser DanielRosenwasser May 6, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually tried something in the same spirit with a Uint8Array - check out #43987

mappingSegmentsBuffer.length = 1;
mappingSegmentsBuffer[0] = mappings;
hasLast = true;
exit();
}
Expand Down