Skip to content

[Android] Run CoreCLR library tests on Android #114148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
88aacc6
Enable running library tests
kotlarmilos Apr 2, 2025
42028b6
Add jobs to extra-platforms
kotlarmilos Apr 4, 2025
f45a428
Fix System.Runtime tests
kotlarmilos Apr 7, 2025
84a9855
Fix arm64 android coreclr tests
kotlarmilos Apr 10, 2025
a78d0ce
Add TestUtilities project reference
kotlarmilos Apr 10, 2025
3187139
Add support for local android test execution on windows
steveisok Apr 10, 2025
71db4dd
Merge branch 'main' into improvement/coreclr-android-library-tests
kotlarmilos Apr 18, 2025
8d9964c
Disable failing tests on Android devices
kotlarmilos Apr 18, 2025
6250953
Disable failing tests on Android devices
kotlarmilos Apr 18, 2025
56eea59
Disable failing tests
kotlarmilos Apr 19, 2025
00a94e1
Remove unnecessary using System directive
kotlarmilos Apr 19, 2025
e3871b2
Disable failing tests
kotlarmilos Apr 20, 2025
33a0703
Disable failing tests
kotlarmilos Apr 21, 2025
2dcc580
Fix typo
kotlarmilos Apr 22, 2025
973d81f
Disable failing tests
kotlarmilos Apr 22, 2025
b91da90
Disable ConfigSwitchIsHonored test
kotlarmilos Apr 22, 2025
c6c3696
Disable failing tests
kotlarmilos Apr 23, 2025
92a4570
Fix typo
kotlarmilos Apr 23, 2025
fd82f71
Enable System.Runtime.Loader.Tests tests
kotlarmilos Apr 23, 2025
eba4f6c
Enable System.Text.RegularExpressions tests
kotlarmilos Apr 23, 2025
e10afcc
Disable failing tests
kotlarmilos Apr 24, 2025
7ad54e8
Try run disabled tests
kotlarmilos Apr 24, 2025
c2612e5
Fix build
kotlarmilos Apr 24, 2025
e061e9d
Disable failing tests
kotlarmilos Apr 24, 2025
10de077
Enable tests that are passing
kotlarmilos Apr 24, 2025
2bf63ac
Fix typo
kotlarmilos Apr 24, 2025
da3343a
Fix ActiveIssue annotation
kotlarmilos Apr 25, 2025
2111a6e
Disable failing tests
kotlarmilos Apr 25, 2025
250dfca
Add comment
kotlarmilos Apr 25, 2025
876aa62
Disable failing tests
kotlarmilos Apr 25, 2025
7eb456e
Fix test annotations
kotlarmilos Apr 28, 2025
945560d
Update ProjectExclusions tracking issue
kotlarmilos Apr 28, 2025
b918e27
Disable failing tests on Android platform
kotlarmilos Apr 29, 2025
b0f3952
Skip MutexTests on CoreCLR Android
kotlarmilos Apr 30, 2025
8be70f0
Disable tests on CoreCLR Android
kotlarmilos Apr 30, 2025
917fffa
Fix ActiveIssue annotation
kotlarmilos Apr 30, 2025
8cae98e
Disable failing tests
kotlarmilos May 1, 2025
2dc7c73
Disable failing tests
kotlarmilos May 6, 2025
cf2209f
Exclude System.Net.Security.Tests from project builds
kotlarmilos May 7, 2025
9c36604
Exclude System.Net.Security.Tests from project builds
kotlarmilos May 7, 2025
39e9601
Merge branch 'main' into improvement/coreclr-android-library-tests
kotlarmilos May 9, 2025
c2dc647
Disable CoreCLR tests on Mono
kotlarmilos May 9, 2025
adc6b76
Merge branch 'main' into improvement/coreclr-android-library-tests
kotlarmilos May 12, 2025
046e100
Disable System.Net.WebSockets.Tests due to disk space limitation
kotlarmilos May 14, 2025
e29ad26
Exclude debug symbols
kotlarmilos May 14, 2025
a81d29c
Update StripDebugSymbols property
kotlarmilos May 14, 2025
a2700f7
Disable failing tests
kotlarmilos May 14, 2025
2c48518
Fix build
kotlarmilos May 15, 2025
da61933
Disable tests
kotlarmilos May 15, 2025
ee7512a
Disable failing tests
kotlarmilos May 15, 2025
f2d056b
Test run
kotlarmilos May 16, 2025
c1eb68e
Use Debug config to ensure android:debuggable=true
kotlarmilos May 17, 2025
00fa70b
Fix StripDebugSymbols
kotlarmilos May 19, 2025
a54f678
Fix formatting
kotlarmilos May 19, 2025
2052db2
Merge branch 'main' into improvement/coreclr-android-library-tests
kotlarmilos May 21, 2025
83e6860
Test run
kotlarmilos May 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions eng/pipelines/extra-platforms/runtime-extra-platforms-android.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,34 @@ jobs:
parameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Android devices
# Build the whole product using CoreCLR and run libraries tests
#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
buildConfig: Release
runtimeFlavor: coreclr
platforms:
- android_arm64
variables:
# map dependencies variables to local variables
- name: librariesContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
- name: coreclrContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_coreclr.containsChange'] ]
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_CoreCLR
isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }}
buildArgs: -s clr.runtime+clr.alljits+clr.corelib+clr.nativecorelib+clr.tools+clr.packages+libs+libs.tests+host+packs -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg)
timeoutInMinutes: 480
# extra steps, run tests
postBuildSteps:
- template: /eng/pipelines/libraries/helix.yml
parameters:
creator: dotnet-bot
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,34 @@ jobs:
parameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Android emulators
# Build the whole product using CoreCLR and run libraries tests
#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
buildConfig: Release
runtimeFlavor: coreclr
platforms:
- android_x64
variables:
# map dependencies variables to local variables
- name: librariesContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
- name: coreclrContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_coreclr.containsChange'] ]
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_CoreCLR
isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }}
buildArgs: -s clr.runtime+clr.alljits+clr.corelib+clr.nativecorelib+clr.tools+clr.packages+libs+libs.tests+host+packs -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg)
timeoutInMinutes: 240
# extra steps, run tests
postBuildSteps:
- template: /eng/pipelines/libraries/helix.yml
parameters:
creator: dotnet-bot
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)
2 changes: 1 addition & 1 deletion eng/pipelines/runtime.yml
Original file line number Diff line number Diff line change
Expand Up @@ -974,7 +974,7 @@ extends:
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_CoreCLR
buildArgs: -s clr.runtime+clr.alljits+clr.corelib+clr.nativecorelib+clr.tools+clr.packages+libs++libs.tests+host+packs -c $(_BuildConfig) /p:ArchiveTests=true /p:TestAssemblies=false
buildArgs: -s clr.runtime+clr.alljits+clr.corelib+clr.nativecorelib+clr.tools+clr.packages+libs+libs.tests+host+packs -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=true
timeoutInMinutes: 480
condition: >-
or(
Expand Down
47 changes: 47 additions & 0 deletions eng/testing/AndroidRunnerTemplate.cmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
@ECHO OFF
setlocal enabledelayedexpansion

SET EXECUTION_DIR=%~dp0
SET ASSEMBLY_NAME=%1
SET TARGET_ARCH=%2
SET TARGET_OS=%3
SET TEST_NAME=%4
SET REPO_ROOT=%5

:Arg_Loop
if "%6" == "" goto ArgsDone
set "__AdditionalArgs=!__AdditionalArgs! %6"&shift&goto Arg_Loop
:ArgsDone

SET "XHARNESS_OUT=%EXECUTION_DIR%xharness-output"

cd %EXECUTION_DIR%

:lock
MKDIR androidtests.lock 2>NUL
IF "%errorlevel%" NEQ "0" (
ping -n 6 127.0.0.1 >NUL
GOTO :lock
)

IF [%XHARNESS_CLI_PATH%] NEQ [] (
:: When running in CI, we only have the .NET runtime available
:: We need to call the XHarness CLI DLL directly via dotnet exec
SET HARNESS_RUNNER=%REPO_ROOT%dotnet.cmd exec "%XHARNESS_CLI_PATH%"
) ELSE (
SET HARNESS_RUNNER=%REPO_ROOT%dotnet.cmd xharness
)

%HARNESS_RUNNER% android test --instrumentation="net.dot.MonoRunner" --package-name="net.dot.%ASSEMBLY_NAME%" --app="%EXECUTION_DIR%bin\%TEST_NAME%.apk" --output-directory="%XHARNESS_OUT%" --timeout=1800 %__AdditionalArgs%

SET EXIT_CODE=%ERRORLEVEL%

ECHO XHarness artifacts: %XHARNESS_OUT%

RMDIR /Q androidtests.lock 2>NUL
EXIT /B %EXIT_CODE%

:: ========== FUNCTIONS ==========
:NORMALIZEPATH
SET RETVAL=%~f1
EXIT /B
12 changes: 8 additions & 4 deletions eng/testing/tests.android.targets
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<BundleTestAppTargets>$(BundleTestAppTargets);BundleTestAndroidApp</BundleTestAppTargets>
</PropertyGroup>

<Import Project="$(MonoProjectRoot)\msbuild\common\LibraryBuilder.props" />
<Import Project="$(MonoProjectRoot)\msbuild\common\LibraryBuilder.props" />
<Import Project="$(MonoProjectRoot)\msbuild\android\build\AndroidBuild.props" />
<Import Project="$(MonoProjectRoot)\msbuild\android\build\AndroidBuild.InTree.targets" />

Expand All @@ -12,6 +12,10 @@
<BundleTestAndroidAppDependsOn>AndroidBuild</BundleTestAndroidAppDependsOn>
</PropertyGroup>

<PropertyGroup Condition="'$(RuntimeFlavor)' == 'CoreCLR'">
<DefineConstants>$(DefineConstants);SINGLE_FILE_TEST_RUNNER</DefineConstants>
</PropertyGroup>

<!-- Target that kicks off the whole test build and run flow -->
<Target Name="BundleTestAndroidApp" DependsOnTargets="$(BundleTestAndroidAppDependsOn)" />

Expand All @@ -22,7 +26,7 @@

<PropertyGroup>
<MainLibraryFileName Condition="'$(MainLibraryFileName)' == ''">AndroidTestRunner.dll</MainLibraryFileName>

<AndroidBuildDir>$(PublishDir)</AndroidBuildDir>
<AndroidBundleDir>$(BundleDir)</AndroidBundleDir>
</PropertyGroup>
Expand All @@ -44,7 +48,7 @@
<_InternalForceInterpret>true</_InternalForceInterpret>
<_IsNative>true</_IsNative>
</AndroidAssembliesToBundle>

<_PublishAssemblies Include="$(PublishDir)\**\*.dll" Exclude="$(PublishDir)\**\*.resources.dll" />
<_SatelliteAssemblies Include="$(PublishDir)\**\*.resources.dll" />

Expand Down Expand Up @@ -73,4 +77,4 @@
<RemoveDir Condition="'$(ArchiveTests)' == 'true' and '$(AndroidGenerateAppBundle)' == 'true'" Directories="$(OutDir)" />
</Target>

</Project>
</Project>
10 changes: 6 additions & 4 deletions eng/testing/tests.targets
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
<Project>
<PropertyGroup>
<RunScriptWindowsCmd Condition="'$(TargetOS)' == 'windows' and '$(RunScriptWindowsCmd)' == ''">true</RunScriptWindowsCmd>
<RunScriptWindowsCmd Condition="'$(TargetOS)' != 'windows' and '$(RunScriptWindowsCmd)' == ''">false</RunScriptWindowsCmd>
<RunScriptWindowsCmd Condition="'$(OS)' == 'Windows_NT' and '$(RunScriptWindowsCmd)' == ''">true</RunScriptWindowsCmd>
<RunScriptWindowsCmd Condition="'$(OS)' != 'Windows_NT' and '$(RunScriptWindowsCmd)' == ''">false</RunScriptWindowsCmd>
</PropertyGroup>
<PropertyGroup Condition="'$(RunScriptInputName)' == ''">
<RunScriptInputName Condition="'$(RunScriptWindowsCmd)' == 'true'">RunnerTemplate.cmd</RunScriptInputName>
<RunScriptInputName Condition="'$(RunScriptWindowsCmd)' != 'true'">RunnerTemplate.sh</RunScriptInputName>
<RunScriptInputName Condition="'$(BuildTestsOnHelix)' == 'true' and '$(TargetsAppleMobile)' == 'true'">AppleHelixRunnerTemplate.sh</RunScriptInputName>
<RunScriptInputName Condition="'$(BuildTestsOnHelix)' != 'true' and '$(TargetsAppleMobile)' == 'true'">AppleRunnerTemplate.sh</RunScriptInputName>
<RunScriptInputName Condition="'$(TargetOS)' == 'android'">AndroidRunnerTemplate.sh</RunScriptInputName>
<RunScriptInputName Condition="'$(TargetOS)' == 'android' and '$(OS)' != 'Windows_NT'">AndroidRunnerTemplate.sh</RunScriptInputName>
<RunScriptInputName Condition="'$(TargetOS)' == 'android' and '$(OS)' == 'Windows_NT'">AndroidRunnerTemplate.cmd</RunScriptInputName>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the AndroidRunnerTemplate.cmd needed now when we enable CoreCLR and we didn't need it before for Mono?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script is used to run tests from Windows, it generating a batch script that runs and invokes xharness.

/cc: @steveisok

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it is mostly for local testing? We have Mono Android runs which use Windows hosts and invoke xharness to run tests on connected Android devices.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is for local testing. What we use on helix is different

<RunScriptInputName Condition="'$(TargetOS)' == 'wasi' and '$(OS)' != 'Windows_NT'">WasiRunnerTemplate.sh</RunScriptInputName>
<RunScriptInputName Condition="'$(TargetOS)' == 'wasi' and '$(OS)' == 'Windows_NT'">WasiRunnerTemplate.cmd</RunScriptInputName>
<RunScriptInputName Condition="'$(TargetOS)' == 'browser' and '$(OS)' != 'Windows_NT'">WasmRunnerTemplate.sh</RunScriptInputName>
Expand Down Expand Up @@ -172,7 +173,8 @@ TEST_ARCH=$(_AndroidArchitecture)
<PropertyGroup Condition="'$(TargetsMobile)' == 'true'">
<RunTestsCommand>"$(RunScriptOutputPath)" $(AssemblyName) $(TargetArchitecture) $(TargetOS) $(TestProjectName)</RunTestsCommand>
<RunTestsCommand Condition="'$(TargetsAppleMobile)' == 'true'">$(RunTestsCommand) $(Configuration) $(AdditionalXHarnessArguments)</RunTestsCommand>
<RunTestsCommand Condition="'$(TargetOS)' == 'android'">$(RunTestsCommand) $(AdditionalXHarnessArguments)</RunTestsCommand>
<RunTestsCommand Condition="'$(TargetOS)' == 'android' and '$(OS)' != 'Windows_NT'">$(RunTestsCommand) $(AdditionalXHarnessArguments)</RunTestsCommand>
<RunTestsCommand Condition="'$(TargetOS)' == 'android' and '$(OS)' == 'Windows_NT'">$(RunTestsCommand) $(RepoRoot) $(AdditionalXHarnessArguments)</RunTestsCommand>
<RunTestsCommand Condition="'$(TargetOS)' == 'browser'">"$(RunScriptOutputPath)" $(JSEngine) $(AssemblyName).dll $(Scenario)</RunTestsCommand>
</PropertyGroup>

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Diagnostics;
Expand All @@ -17,7 +18,7 @@

namespace Microsoft.Extensions.SourceGeneration.Configuration.Binder.Tests
{
[ActiveIssue("https://github.com/dotnet/runtime/issues/52062", TestPlatforms.Browser)]
[ConditionalClass(typeof(PlatformDetection), nameof(PlatformDetection.HasAssemblyFiles))]
public partial class ConfigurationBindingGeneratorTests : ConfigurationBinderTestsBase
{
internal sealed class ConfigBindingGenTestDriver
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ public async Task Configure_T_NamedParameters_OutOfOrder(string row)
using System.Collections.Generic;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

public class Program
{
public static void Main()
Expand All @@ -84,7 +84,7 @@ public static void Main()

{{row}}
}

public class MyClass
{
public string MyString { get; set; }
Expand All @@ -109,7 +109,7 @@ public async Task Bind_T_NamedParameters_OutOfOrder(string row)
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;

public class Program
{
public static void Main()
Expand All @@ -121,7 +121,7 @@ public static void Main()

{{row}}
}

public class MyClass
{
public string MyString { get; set; }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Threading.Tasks;
using Microsoft.CodeAnalysis;
using Microsoft.Extensions.Configuration.Binder.SourceGeneration;
Expand All @@ -11,7 +12,7 @@ namespace Microsoft.Extensions.SourceGeneration.Configuration.Binder.Tests
{
public partial class ConfigurationBindingGeneratorTests : ConfigurationBinderTestsBase
{
[ActiveIssue("https://github.com/dotnet/runtime/issues/52062", TestPlatforms.Browser)]
[ConditionalClass(typeof(PlatformDetection), nameof(PlatformDetection.HasAssemblyFiles))]
public sealed class IncrementalTests
{
[Fact]
Expand Down
Loading
Loading