Pipeline parameters with '=' character in string value raise parsing exceptions

[jarandaf]

I have encountered this issue in different components (pipeline parameters, Databricks spark config values, etc.), the underlying key/value parsing mechanism seems not to support string values with '=' character:

[2021-09-14 09:37:34Z] Parsed interface. Will be submitting job to Azure Databricks adb-***.azuredatabricks.net
[2021-09-14 09:37:35Z] Failed to submit job because of error: Invalid base_parameters parameter some_parameter=AT_PW=14_FW=60_batch, stack_trace:    at Microsoft.Aether.Clouds.CloudCommon.ParameterParser.TryGetDictionary(Dictionary`2 parameters, String name) in d:\dbs\sh\l0hi\0902_133310\cmd\2h\src\aether\platform\backendV2\Clouds\Common\CloudCommon\ParameterParser.cs:line 41
   at Microsoft.Aether.DataBricksCloud.K8sService.Parser.ValidateWorkloadSetup(DataBricksJobMetadata job, Dictionary`2 parameters) in d:\dbs\sh\l0hi\0903_131236\cmd\i\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\Parser.cs:line 427
   at Microsoft.Aether.DataBricksCloud.K8sService.Parser.GetSubmitRunDto(DataBricksJobMetadata job, Dictionary`2 metadata, Dictionary`2 parameters) in d:\dbs\sh\l0hi\0903_131236\cmd\i\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\Parser.cs:line 31
   at Microsoft.Aether.DataBricksCloud.K8sService.JobProcessor.StartNewJobAsync(DataBricksJobMetadata job) in d:\dbs\sh\l0hi\0903_131236\cmd\i\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\JobProcessor.cs:line 264
[2021-09-14 09:37:35Z] Exception while running job, error: Invalid base_parameters parameter some_parameter=AT_PW=14_FW=60_batch, stack_trace:    at Microsoft.Aether.Clouds.CloudCommon.ParameterParser.TryGetDictionary(Dictionary`2 parameters, String name) in d:\dbs\sh\l0hi\0902_133310\cmd\2h\src\aether\platform\backendV2\Clouds\Common\CloudCommon\ParameterParser.cs:line 41
   at Microsoft.Aether.DataBricksCloud.K8sService.Parser.ValidateWorkloadSetup(DataBricksJobMetadata job, Dictionary`2 parameters) in d:\dbs\sh\l0hi\0903_131236\cmd\i\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\Parser.cs:line 427
   at Microsoft.Aether.DataBricksCloud.K8sService.Parser.GetSubmitRunDto(DataBricksJobMetadata job, Dictionary`2 metadata, Dictionary`2 parameters) in d:\dbs\sh\l0hi\0903_131236\cmd\i\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\Parser.cs:line 31
   at Microsoft.Aether.DataBricksCloud.K8sService.JobProcessor.StartNewJobAsync(DataBricksJobMetadata job) in d:\dbs\sh\l0hi\0903_131236\cmd\i\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\JobProcessor.cs:line 264
   at Microsoft.Aether.DataBricksCloud.K8sService.JobProcessor.ProcessJobAsync(DataBricksJobMetadata job) in d:\dbs\sh\l0hi\0903_131236\cmd\i\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\JobProcessor.cs:line 188

[weedqian]

Hello, @jarandaf.

Thanks for reporting this issue and sorry for the late response.
Could you share more details about your failed pipeline please? Could you share your submission script/notebook with us for repro?

[cardoner1993]

I'm seeing the same behaviour when passing any environment variable having an '=' character:
For example when using the spark_env_variables if the value includes an equal character the output error is as follows:

System.ArgumentException: Invalid spark_env_vars parameter INSTRUMENTATION_KEY=***;LOG_ANALYTICS_WORKSPACE_ID=***;LOG_ANALYTICS_WORKSPACE_KEY=***== at Microsoft.Aether.Clouds.CloudCommon.ParameterParser.TryGetDictionary(Dictionary2 parameters, String name) in D:\a_work\1\s\src\aether\platform\backendV2\Clouds\Common\CloudCommon\ParameterParser.cs:line 55
at Microsoft.Aether.DataBricksCloud.K8sService.Parser.GetSubmitRunDto(DataBricksJobMetadata job, Dictionary2 metadata, Dictionary2 parameters) in D:\a_work\1\s\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\Parser.cs:line 29
at Microsoft.Aether.DataBricksCloud.K8sService.JobProcessor.StartNewJobAsync(DataBricksJobMetadata job) in D:\a_work\1\s\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\JobProcessor.cs:line 298
at Microsoft.Aether.DataBricksCloud.K8sService.JobProcessor.ProcessJobAsync(DataBricksJobMetadata job) in D:\a_work\1\s\src\aether\platform\backendV2\BlueBox\Clouds\DataBricksK8s\Service\JobProcessor.cs:line 200
`
The last to equal signs are parsed as new variables when they are just part of the string.