[Test]: Test the performance of MCP with Spark events logs that ran for 4+ hours

[vara-bonthu]

Description

We want to test the Spark History Server MCP's ability to handle large-scale Spark event logs from long-running jobs (4+ hours) with gigabytes of event data. This will validate the MCP server's performance and scalability when processing enterprise-scale Spark applications.

Test Objectives

• Validate MCP server performance with large event logs (GB-scale data)
• Test response times for complex queries on long-running Spark jobs (4+ hours)
• Identify potential bottlenecks in MCP server processing
• Ensure memory efficiency when handling large event datasets

Test Plan

Phase 1: Data Generation

• Use Spark Benchmarking Kit to run 8-hour benchmark job to generate Spark event logs in S3 buckets
• Duplicate the same logs with different spark app id - 50 to 100 copies

Load Testing

• Test with local MCP server pointing to spark history server (with 50+ jobs Spark events)
• Simulate realistic query patterns:
  • Application overview requests
  • Stage-level performance analysis
  • Task-level bottleneck identification
  • Job comparison operations
• Measure response times and resource utilization

[nabuskey]

I do have a prototype version written in kotlin which should give us more performance if required. I switched to python because I think it's easier to use and get more contributions.

[vara-bonthu]

I would keep this in Python for now and publish the benchmarks.

[andrewdkim47]

Would be interested in implementing ways to reduce context window size for certain tools. For example, we could implement a "summarize" parameter which would help reduce the context window size of the results for certain tools such as get_stages