Proposal: write tests that exercise the interaction of expression evaluation with new language features

[derekxu16]

This issue tracks a proposal that is planned to be discussed in next week's meeting.

The VM team would like the co19 team to write tests that exercise the interaction of expression evaluation with new language features, when new language features are added. Our plan is for the VM team to write a package that makes it easy to write tests similar to the ones in package:vm_service; that is, ones that start a testee VM and make VM Service RPC calls on that testee. For example, evaluate_uninitialized_late_variable_test. Then co19 would use that package to write expression evaluation tests.

[sgrekhov]

@derekxu16 Could you please provide some more details? As far as I understand, there will be a new package that should be used for future testing. What exactly do you want to test using this new package?

[derekxu16]

For new language features, co19 already writes tests that cover the CFE and VM implementations of the features. The purpose of the tests written using the new package will be to cover the communication that happens between the VM and the CFE during expression evaluation. That is, to ensure that the VM sends the context needed by the CFE to evaluate the expression, that the CFE returns correct kernel data, and that the VM loads and executes the kernel data correctly. So, for each new language feature, there should be a test performing a few evaluations of expressions that use the new language feature. Only a few evaluations will need to be performed because, as mentioned earlier, the purpose of these tests will be to cover communication between the VM and CFE, not implementation within the CFE and VM themselves.

Here’s an example I just wrote for dot shorthands: https://github.com/dart-lang/sdk/blob/fe46d2089f05536666bd96de955af798b09a6165/pkg/vm_service/test/dot_shorthands_test.dart.

[sgrekhov]

Thanks for the example! Do I understand correctly that the requested co19 tests should do roughly the same as the function "// Test interaction of expression evaluation with dot shorthands."? Also, what role is the new package supposed to play? Why not use the developer.dart package?

[derekxu16]

Do I understand correctly that the requested co19 tests should do roughly the same as the function "// Test interaction of expression evaluation with dot shorthands."?

Yes, that's correct.

Also, what role is the new package supposed to play? Why not use the developer.dart package?

The package will provide utilities that make it easier to start a testee VM and make VM Service RPC calls on that testee. One of the most important utilities that it will provide is an equivalent of runIsolateTests. Very few developers will need to use these utilities, so they are not a good fit for dart:developer. Perhaps a new package does not need to be made for them, and they can just be exposed from package:vm_service. I'll discuss this with the team.

[sgrekhov]

Thank you! Please let me know when the package is ready for use.

[derekxu16]

Actually, since dependencies are resolved according to the Dart SDK's package_config.json when co19 tests are run, do you think it would be ok if your tests just import the utilities from 'package:vm_service/test/common/test_helper.dart', or would you prefer for the utilities to be made public APIs?

[sgrekhov]

co19 tests are not part of the Dart SDK.
@athomas, is there a way to import package:vm_service/test/common/test_helper.dart in a file that is not part of the SDK?
If not, then I prefer using the public API (since the only alternative would be to make co19 part of the Dart SDK).

[athomas]

@sgrekhov what would be the practical implication of importing package:vm_service for you? From most of the Dart team's perspective, co19 is part of the SDK because gclient magic manifests it inside the SDK checkout.

[sgrekhov]

There is no problem to import package:vm_service. But there is no way to import package:vm_service/test/common/test_helper.dart because this path is not inside the lib directory.

[derekxu16]

If the tests only ever run in the context of the Dart SDK, then it is possible to import the utilities using a relative path. For example, tests/co19/src/LanguageFeatures/Static-access-shorthand/constant_expression_A01_t01.dart still works when import '../../../../../pkg/vm_service/test/common/test_helper.dart'; is added to it. If this seems too hacky to you, we can make the utilities public APIs; that won't be a big deal.

[sgrekhov]

Ok. I'll use a relative path.