[FEATURE REQUEST]: How to query from Hadoop and deploy in production?

[mii9000]

Is your feature request related to a problem? Please describe.
Yes. Dont know how to write a Spark job using .NET for:

1. Query from Hadoop
2. Deploy to production

Describe the solution you'd like
Documentation or sample project showing how to query from Hadoop and deploy to production

Describe alternatives you've considered
Searched online for samples and resources but could not find many

Additional context
Add any other context or screenshots about the feature request here.

[imback82]

@ibrahim-islam, can you be more specific to what problems you are encountering? (e.g., what steps did you take? what errors do you see?)

[mii9000]

@ibrahim-islam, can you be more specific to what problems you are encountering? (e.g., what steps did you take? what errors do you see?)

@imback82 Usually any Microsoft product comes with good documentation and examples. I have to write a spark job recently for production so I wanted to understand how I can do that using Spark for .NET.

[elvaliuliuliu]

@ibrahim-islam, here are some great resources that can help you get started using .NET for Apache Spark.

• Tutorial on the .NET website that helps you write and run your first .NET for Spark app in just 10 minutes.
• Articles on our official docs page that explain various big data and .NET for Spark concepts.
• Microsoft.Spark API reference.
• Introductory 101 video series introducing .NET for Spark and creating/running C# Spark sample apps
• Official .NET for Spark GitHub repo, where you can view the .NET for Spark source code and sample C# and F# Spark apps.

[mii9000]

@elvaliuliuliu Any examples that show getting data from Hadoop?

[elvaliuliuliu]

@elvaliuliuliu Any examples that show getting data from Hadoop?

@ibrahim-islam: Can you please specify the scenarios you are working on? (e.g. are you running your application locally or in the cluster?)

[mii9000]

@ibrahim-islam: Can you please specify the scenarios you are working on? (e.g. are you running your application locally or in the cluster?)

Currently running locally only for testing purposes. If this framework results to be useful then we can think about deploying to production.

[imback82]

Any examples that show getting data from Hadoop?

@ibrahim-islam do you mean to read data from wasb or abfs?

[mii9000]

Any examples that show getting data from Hadoop?

@ibrahim-islam do you mean to read data from wasb or abfs?

We have our own HDFS and would like to read from that

[imback82]

Did you try hdfs protocol like "hdfs://localhost:9000/user/hduser/data/testfile"?