Merge remote-tracking branch 'origin/develop'

Author: awdeorio

.github/workflows/continuous_integration.yml

@@ -4,7 +4,7 @@ name: CI
 # Define conditions for when to run this action
 on:
   pull_request: # Run on all pull requests
-  push: # Run on all pushes to main
+  push: # Run on all pushes to main or develop
     branches:
       - main
       - develop
@@ -49,12 +49,14 @@ jobs:
       # https://github.com/ymyzk/tox-gh-actions#workflow-configuration
       - name: Run tests
         run: tox
-      # - name: Combine coverage
-      #   run: coverage xml
+
+      # Combine coverage data from all test executions
+      - name: Combine coverage
+        run: coverage xml
 
       # Upload coverage report
       # https://github.com/codecov/codecov-action
-      # - name: Upload coverage report
-      #   uses: codecov/codecov-action@v1
-      #   with:
-      #     fail_ci_if_error: true
+      - name: Upload coverage report
+        uses: codecov/codecov-action@v1
+        with:
+          fail_ci_if_error: true

.gitignore

@@ -2,6 +2,9 @@
 *.pyc
 __pycache__
 
+# Example input
+/example/
+
 # Python virtual environment 
 /env/
 /env2/
@@ -32,6 +35,3 @@ build/
 
 # macOS system files
 *.DS_Store
-
-# Output directories
-/output

CONTRIBUTING.md

@@ -78,3 +78,11 @@ $ git describe
 X.Y.Z
 $ git push --tags origin main
 ```
+
+Create a release on GitHub using the "Auto-generate release notes" feature. https://github.com/eecs485staff/madoop/releases/new
+
+Upload to PyPI
+```console
+$ python3 setup.py sdist bdist_wheel
+$ twine upload --sign dist/*
+```

MANIFEST.in

@@ -1,10 +1,11 @@
 include LICENSE
 include MANIFEST.in
 include README.md
+include README_Hadoop_Streaming.md
 include CONTRIBUTING.md
 include .pylintrc
 graft tests
-graft example
+graft madoop/example
 
 # Avoid dev and and binary files
 exclude tox.ini

README.md

@@ -1,73 +1,50 @@
 Madoop: Michigan Hadoop
 =======================
 
-Michigan Hadoop (`madoop`) is a light weight MapReduce framework for education.  Madoop implements the [Hadoop Streaming](https://hadoop.apache.org/docs/r1.2.1/streaming.html) interface.  Madoop is implemented in Python and runs on a single machine.
+[![PyPI](https://img.shields.io/pypi/v/madoop.svg)](https://pypi.org/project/madoop/)
+[![CI main](https://github.com/eecs485staff/madoop/workflows/CI/badge.svg?branch=develop)](https://github.com/eecs485staff/madoop/actions?query=branch%3Adevelop)
+[![codecov](https://codecov.io/gh/eecs485staff/madoop/branch/develop/graph/badge.svg)](https://codecov.io/gh/eecs485staff/madoop)
 
-## Quick start
-Install and run an example word count MapReduce program.
-```console
-$ pip install madoop
-$ madoop \
-  -input example/input \
-  -output output \
-  -mapper example/map.py \
-  -reducer example/reduce.py
-$ cat output/part-*
-autograder	2
-world	1
-eecs485	1
-goodbye	1
-hello	3
-```
+Michigan Hadoop (`madoop`) is a light weight MapReduce framework for education.  Madoop implements the [Hadoop Streaming](https://hadoop.apache.org/docs/r1.2.1/streaming.html) interface.  Madoop is implemented in Python and runs on a single machine.
 
+For an in-depth explanation of how to write MapReduce programs in Python for Hadoop Streaming, see our [Hadoop Streaming tutorial](README_hadoop_streaming.md).
 
-## Example
-We'll walk through the example in the Quick Start again, providing more detail.  For an in-depth explanation of the map and reduce code, see the [Hadoop Streaming tutorial](https://eecs485staff.github.io/p5-search-engine/hadoop_streaming.html).
 
-## Install
-Install Madoop.  Your version might be different.
+## Quick start
+Install Madoop.
 ```console
 $ pip install madoop
-$ madoop --version
-Madoop 0.1.0
 ```
 
-### Input
-We've provided two small input files.
+Create example MapReduce program with input files.
 ```console
-$ cat example/input/input01.txt
-hello world
-hello eecs485
-$ cat example/input/input02.txt
-goodbye autograder
-hello autograder
+$ madoop --example
+$ tree example
+example
+├── input
+│   ├── input01.txt
+│   └── input02.txt
+├── map.py
+└── reduce.py
 ```
 
-### Run
-Run a MapReduce word count job.  By default, there will be one mapper for each input file.  Large input files maybe segmented and processed by multiple mappers.
-- `-input DIRECTORY` input directory
-- `-output DIRECTORY` output directory
-- `-mapper FILE` mapper executable
-- `-reducer FILE` reducer executable
+Run example word count MapReduce program.
 ```console
 $ madoop \
-    -input example/input \
-    -output output \
-    -mapper example/map.py \
-    -reducer example/reduce.py
+  -input example/input \
+  -output example/output \
+  -mapper example/map.py \
+  -reducer example/reduce.py
 ```
 
-### Output
-Concatenate and print output.  The concatenation of multiple output files may not be sorted.
+Concatenate and print the output.
 ```console
-$ ls output
-part-00000  part-00001  part-00002  part-00003
-$ cat output/part-*
-autograder	2
-world	1
-eecs485	1
-goodbye	1
-hello	3
+$ cat example/output/part-*
+Goodbye 1
+Bye 1
+Hadoop 2
+World 2
+Hello 2
 ```
 
 ## Comparison with Apache Hadoop and CLI