Skip to content

Commit e279e8f

Browse files
authored
Documented using the Prism runner (#3656)
There is now a portable runner for local beam jobs written natively in Go. This means we don't have to set any python stuff up any more! It also enforces that functions are properly serializable and registered, which makes it better than the old direct runner.
1 parent 9650741 commit e279e8f

File tree

2 files changed

+3
-27
lines changed

2 files changed

+3
-27
lines changed

experimental/batchmap/README.md

Lines changed: 2 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -54,40 +54,15 @@ are identified.
5454
### Setup
5555

5656
The map uses Apache Beam to assemble and run a pipeline of tasks to construct the map.
57-
The following instructions show how to run this code locally using the Python portable runner.
58-
Comprehensive documentation for setting up a Python environment is beyond the scope of this demo, but instructions can be found at [Beam Python Tips](https://cwiki.apache.org/confluence/display/BEAM/Python+Tips).
59-
Steps that worked for me using python 3.10.12 on a Debian machine are:
60-
61-
```shell
62-
python3 --version # Confirm 3.10.12
63-
sudo apt install python3-venv
64-
65-
# Create a virtual python env for configuring the runner and deps
66-
python3 -m venv ~/Virtualenvs/beampy31012
67-
source ~/Virtualenvs/beampy31012/bin/activate
68-
python3 -m pip install --upgrade pip
69-
pip install wheel
70-
pip install apache-beam
71-
72-
# Now run the portable runner, and leave it listening for work on 8099
73-
python -m apache_beam.runners.portability.local_job_service_main --port=8099
74-
```
75-
76-
Alternate steps that may work from checking out the source repo:
77-
78-
1. Check out `github.com/apache/beam` (tested at branch `release-2.24.0`)
79-
2. `cd sdks/python` within that repository
80-
3. `python -m apache_beam.runners.portability.local_job_service_main --port=8099`
81-
57+
Fortunately there is now a Go local portable runner, so you don't need to set up Python any more!
8258

8359
### Building the map
8460

8561
These instructions are for Linux/MacOS but can likely be adapted for Windows.
86-
Ensure that you have a python portable runner listening on port 8099, as documented above.
8762

8863
In another terminal:
8964
1. Check out this repository and `cd` to the folder containing this README
90-
2. `go run ./cmd/build/mapdemo.go --output=/tmp/mapv1 --runner=universal --endpoint=localhost:8099 --environment_type=LOOPBACK`
65+
2. `go run ./cmd/build/mapdemo.go --output=/tmp/mapv1 --runner=PrismRunner`
9166

9267
The pipeline should run and generate files under the ouput directory, each of which contains a tile from the map.
9368
Note that a new file will be constructed for each tile output, which can get very large

experimental/batchmap/cmd/build/mapdemo.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ func main() {
8989
if err := beamx.Run(context.Background(), p); err != nil {
9090
klog.Fatalf("Failed to execute job: %v", err)
9191
}
92+
klog.Infof("Pipeline completed successfully! Output data is at %s", output)
9293
}
9394

9495
// mapEntryFn is a Beam ParDo function that generates a key/value from an int64 input.

0 commit comments

Comments
 (0)