Increase max file size to 2 MB

[melodell]

To support EECS 485 P5 -- The HTML Dataset changes use full Wikipedia pages. We want to make sure that each file gets read in its entirety in a MapReduce job so that students can use the full document content in the pipeline. Some Wikipedia pages are larger than 1 MB, so let's bump the limit so they don't get split.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.31%. Comparing base (5408a7d) to head (e02df80).
Report is 6 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop      #74   +/-   ##
========================================
  Coverage    96.31%   96.31%           
========================================
  Files            4        4           
  Lines          217      217           
========================================
  Hits           209      209           
  Misses           8        8           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[awdeorio]

I'd happy to make the change, but it would be helpful to memorialize a short explanation here.

[noah-weingarden]

I think this is redundant because of #70 too?

[melodell]

I'd happy to make the change, but it would be helpful to memorialize a short explanation here.

Was short on time earlier. Updated now!

[awdeorio]

I think this is redundant because of #70 too?

There are some changes coming down the pipeline (pun intended) to EECS 485 Project 5. The first stage of the MapReduce pipeline will process pages in parallel map tasks. Currently, the first mapper is a single task which processes all input files (each file is a wiki page).

[awdeorio]

LGTM

[noah-weingarden]

There are some changes coming down the pipeline (pun intended) to EECS 485 Project 5. The first stage of the MapReduce pipeline will process pages in parallel map tasks. Currently, the first mapper is a single task which processes all input files (each file is a wiki page).

That sounds promising; is it related to this comment of mine?

[awdeorio]

It's different. HTML parsing will be part of the first pipeline stage (stage 0). There will be one mapper per file, which means one execution per HTML document.

[noah-weingarden]

Mappers will run per file instead of per line? That sounds like we're not going to be using a streaming interface anymore.