Progress has encoding issues

[passsy]

I noticed that some characters from flutter_tools cli , captured with Progress.capture() and then printed to stdout are encoded wrong. Here is a full sample

import 'dart:convert';
import 'dart:io';
import 'package:dcli/dcli.dart' as dcli;

/// Minimal reproduction case for dcli UTF-8 corruption issue
///
/// This demonstrates that dcli.Progress corrupts UTF-8 multi-byte sequences
/// when capturing process output that contains Unicode box drawing characters.
void main() async {
  print('=== DCLI UTF-8 Bug Reproduction ===\n');

  // Create a simple script that outputs UTF-8 box drawing characters
  final scriptContent = '''
#!/usr/bin/env dart
import 'dart:io';
void main() {
  // Output UTF-8 box drawing characters (as used by Flutter test framework)
  stdout.write('══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════\\n');
  stdout.write('The following error occurred:\\n');
  stdout.write('══════════════════════════════════════════════════════════════════════════════════════════════════════\\n');
  // Don't exit with error, just complete normally
}
''';

  final scriptFile = File('temp_utf8_test.dart');
  await scriptFile.writeAsString(scriptContent);

  try {
    print('1. Direct process execution (correct UTF-8):');
    final directResult = await Process.run('dart', ['temp_utf8_test.dart']);
    print('Direct stdout: ${directResult.stdout}');

    print('\n2. Using dcli.Progress.capture() (corrupted UTF-8):');
    final capture = dcli.Progress.capture();
    final dcliResult = await Process.start('dart', ['temp_utf8_test.dart']);
    
    // This is what causes the corruption
    dcliResult.stdout.listen((data) {
      (capture as dynamic).addToStdout(data);
    });

    await dcliResult.exitCode;

    print('Captured lines: ${(capture as dynamic).lines}');
    print('Captured output: ${(capture as dynamic).lines.join('\\n')}');

    print('\n3. Raw bytes comparison:');
    final correctBytes = utf8.encode(
      '══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════',
    );
    print('Correct UTF-8 bytes: $correctBytes');

    // Show what dcli is actually storing
    final capturedText = (capture as dynamic).lines.join('\\n') as String;
    final capturedBytes = utf8.encode(capturedText);
    print('DCLI captured bytes: $capturedBytes');
  } finally {
    // Cleanup
    if (await scriptFile.exists()) {
      await scriptFile.delete();
    }
  }
}

The output is

$ dart dcli_utf8_bug_demo.dart

=== DCLI UTF-8 Bug Reproduction ===

1. Direct process execution (correct UTF-8):
Direct stdout: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following error occurred:
══════════════════════════════════════════════════════════════════════════════════════════════════════


2. Using dcli.Progress.capture() (corrupted UTF-8):
Captured lines: [��� EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK �����������������������������������������������������, The following error occurred:, ������������������������������������������������������������������������������������������������������]
Captured output: ��� EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK �����������������������������������������������������\nThe following error occurred:\n������������������������������������������������������������������������������������������������������

3. Raw bytes comparison:
Correct UTF-8 bytes: [226, 149, 144, 226, 149, 144, 226, 149, 161, 32, 69, 88, 67, 69, 80, 84, 73, 79, 78, 32, 67, 65, 85, 71, 72, 84, 32, 66, 89, 32, 70, 76, 85, 84, 84, 69, 82, 32, 84, 69, 83, 84, 32, 70, 82, 65, 77, 69, 87, 79, 82, 75, 32, 226, 149, 158, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144, 226, 149, 144]
DCLI captured bytes: [195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 161, 32, 69, 88, 67, 69, 80, 84, 73, 79, 78, 32, 67, 65, 85, 71, 72, 84, 32, 66, 89, 32, 70, 76, 85, 84, 84, 69, 82, 32, 84, 69, 83, 84, 32, 70, 82, 65, 77, 69, 87, 79, 82, 75, 32, 195, 162, 194, 149, 194, 158, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 92, 110, 84, 104, 101, 32, 102, 111, 108, 108, 111, 119, 105, 110, 103, 32, 101, 114, 114, 111, 114, 32, 111, 99, 99, 117, 114, 114, 101, 100, 58, 92, 110, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144, 195, 162, 194, 149, 194, 144]

Process finished with exit code 0

I could not find out where in dcli this happens.

[bsutton]

so the issue is that we are processing the data as a binary stream without regard for string encoding. So the isolate passes the data as a series of int arrays.

 final message = Message.stdout(data as Uint8List);

The primary isolate then receives those bytes:

   final messageData =
            channel.toPrimaryIsolate.take(timeout: const Duration(seconds: 2));
        processLogger(
            () => 'take returned with data: len(${messageData.length - 1})');
        response = MessageResponse.fromData(messageData)
          ..onStdout((payload) {
            progress.addToStdout(payload);
          })
          ..onStderr((payload) {
            progress.addToStderr(payload);
          })

and just adds them to the progress.

The original assumption was that we may (in some instances) want to process the data as binary rather than strings.

So the engine basically passes all data as uint8 lists. This may cause breaks in the middle of a multi-byte character.

If we are to processing the data as a string then we need to use something like a StringConversionSink to rebuild the original utf-8 encoded strings, allowing for the fact that a single character may be split across two Message's coming up from the isolate.

Looking at each of the existing Progress implementations, they all assume that the data is treated as text (except the base ProgressImpl abstraction).

So maybe we can just change all of the Progress implementations to rebuild the utf-8 encodings without causing any side effects?

[passsy]

I was able to parse it correctly!

    final capture = dcli.Progress.capture();
    dcli.startFromArgs('dart', ['temp_utf8_test.dart'], progress: capture);
    final utf8String = capture.toList().map((bytes) {
      return utf8.decoder.convert(bytes.codeUnits);
    });

I think it is completely fine to send List<int> between the isolates. I like that it is possible to send plain bytes via stdout.

I don't think we need StringConversionSink, we should assume that the executed programs stdout is correct and does not split characters across messages. (Assuming the data between the isolates doesn't split messages).

The only problem with the current Progress API is that some methods return String.

• List<String> get lines;
• String? get firstLine;
• List<String> toList();
• void forEach(void Function(String line) print);
• String toParagraph();

Those methods currently assume a certain encoding (Dart uses UTF-16 for strings, e.g. in .codeUnits), which is often the wrong one. From my research I learned, most CLIs output in utf-8. My Terminal also uses $ echo $LANG en_US.UTF-8.

Ideally, all of those APIs would be methods with an optional encoding parameter and APIs to read the plain bytes.