[QUESTION]: Streaming example takes a long time to run

[el-genius]

Describe the bug
I try to run that example
spark/examples/Microsoft.Spark.CSharp.Examples/Sql/Streaming/

good news:
it's run
bad news:
it takes much more time to represent data on the console
I didn't know what is the reason
So, if you've any expecting reason for that please share with me

[elvaliuliuliu]

@el-genius , thanks for your question. Can you please provide the command line you use and the full log?

[imback82]

@el-genius You can try with --master local[*] as part of spark-submit or you can set spark.sql.shuffle.partitions config to a low number like 3. That should speed things up.

[el-genius]

@elvaliuliuliu
this is command line
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.1\microsoft-spark-2.4.x-0.10.0.jar dotnet bin\Debug\netcoreapp3.1\NetCatStream.dll

I'll share full log on Sunday morning.

[el-genius]

@imback82
I set --master local as you see in command line I used.
so, try to explain how to play with config is it setting in code?

[elvaliuliuliu]

@el-genius: You can set the config in the code or in the command like spark-submit --conf spark.sql.shuffle.partitions=3

Or try use --master local[*] instead of --master local, It will run Spark locally with as many worker threads as logical cores on your machine.

[el-genius]

good, I'll try that them inform you that make things run better or not.
thank you for your help

[el-genius]

I tried --master local[*] and spark-submit --conf spark.sql.shuffle.partitions=3
It did better but when I used the same example using python, I got the result immediately without any configuration could you tell me the reason behind that?

[imback82]

Can you share the python script? And did you run it thru pyspark or spark-submit?

[el-genius]

using pyspark the code almost the same in C# like python both using netcat by the sameway and perform the same calculation on words

[imback82]

Can you paste the script here? Thanks.