|
| 1 | +Comparison with other python libraries |
| 2 | +====================================== |
| 3 | + |
| 4 | +There's lots of way to skin the threading cat! |
| 5 | + |
| 6 | +### When to use *pylateral* |
| 7 | + |
| 8 | +- Your workload is network-bound and/or IO-bound (e.g., API calls, database queries, read/write to FTP, read/write to files). |
| 9 | + |
| 10 | +- Your workload can be run [embarrassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel). |
| 11 | + |
| 12 | +- You are writing a script or prototype that isn't very large nor complex. |
| 13 | + |
| 14 | +### When not to use *pylateral* |
| 15 | + |
| 16 | +- Your workload is CPU-bound and blocked by the [Global Interpreter Lock](https://en.wikipedia.org/wiki/CPython#Design). *python* threading will not help speed up your workload, consider using [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) or [concurrent.futures.ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) instead. |
| 17 | + |
| 18 | +- The complexity of your program would benefit from thinking about it in terms of [futures and promises](https://en.wikipedia.org/wiki/Futures_and_promises). Consider using [asyncio](https://docs.python.org/3/library/asyncio.html) or [concurrent.futures.ThreadPoolExecutors](https://docs.python.org/3/library/concurrent.futures.html) instead. |
| 19 | + |
| 20 | +- When you want to have tighter controls around the lifecycle of your thread. Consider using [threading](https://docs.python.org/3/library/threading.html) instead. |
| 21 | + |
| 22 | +- For larger workloads, consider using [dask.distributed](https://distributed.dask.org/en/latest/#), [Airflow](https://airflow.apache.org/), [Dagster](https://github.com/dagster-io/dagster/) or [Prefect](https://www.prefect.io/) to perform work across many nodes. |
| 23 | + |
| 24 | +- You would benefit from a web UI for viewing and interacting with your tasks. For that, consider using [Airflow](https://airflow.apache.org/) or [Prefect](https://www.prefect.io/). |
| 25 | + |
| 26 | +Feature comparison |
| 27 | +------------------ |
| 28 | + |
| 29 | +| Feature | pylateral | [asyncio](https://docs.python.org/3/library/asyncio.html) | [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) | [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) | [threading](https://docs.python.org/3/library/threading.html) | |
| 30 | +| ---------------------------------- | --------- | ------- | ------------------------ | --------------- | --------- | |
| 31 | +| Easy to adapt single-threaded code | ✅ | ❌ | ❌ | ❌ | ❌ | |
| 32 | +| [Simple nested tasks](usage.md#working-with-nested-tasks) | ✅ | ✅ | ❌ | ❌ | ❌ | |
| 33 | +| Concurrent IO-bound workloads | ✅ | ✅ | ✅ | ✅ | ✅ | |
| 34 | +| Concurrent CPU-bound workloads | ❌ | ❌ | ✅ (Process Pool) | ✅ | ❌ | |
| 35 | +| Flexibility in using return values | ❌ | ✅ | ✅ | ❌ | ❌ | |
| 36 | + |
| 37 | +Code comparison |
| 38 | +---------- |
| 39 | + |
| 40 | +[PEP-3148 -- futures - execute computations asynchronously](https://www.python.org/dev/peps/pep-3148/#id13) introduces `concurrent.futures` and illustrates it by example. Here I show that example in *pylateral*, stacked up against the main threading libraries offered in python. |
| 41 | + |
| 42 | +### `asyncio` |
| 43 | + |
| 44 | +```python |
| 45 | +import aiohttp |
| 46 | +import asyncio |
| 47 | +import sqlite3 |
| 48 | + |
| 49 | +URLS = [ |
| 50 | + 'http://www.foxnews.com/', |
| 51 | + 'http://www.cnn.com/', |
| 52 | + 'http://europe.wsj.com/', |
| 53 | + 'http://www.bbc.co.uk/', |
| 54 | + 'http://some-made-up-domain.com/', |
| 55 | +] |
| 56 | + |
| 57 | +async def extract_and_load(url, timeout=30): |
| 58 | + try: |
| 59 | + async with aiohttp.ClientSession() as session: |
| 60 | + async with session.get(url, timeout=timeout) as response: |
| 61 | + web_result = await response.text() |
| 62 | + print(f"{url} is {len(web_result)} bytes") |
| 63 | + |
| 64 | + with sqlite3.connect('example.db') as conn, conn as cursor: |
| 65 | + cursor.execute('CREATE TABLE IF NOT EXISTS web_results (url text, length int);') |
| 66 | + cursor.execute('INSERT INTO web_results VALUES (?, ?)', (url, len(web_result))) |
| 67 | + except Exception as e: |
| 68 | + print(f"{url} generated an exception: {e}") |
| 69 | + return False |
| 70 | + else: |
| 71 | + return True |
| 72 | + |
| 73 | +async def main(): |
| 74 | + succeeded = await asyncio.gather(*[ |
| 75 | + extract_and_load(url) |
| 76 | + for url in URLS |
| 77 | + ]) |
| 78 | + |
| 79 | + print(f"Successfully completed {sum(1 for result in succeeded if result)}") |
| 80 | + |
| 81 | +asyncio.run(main()) |
| 82 | +``` |
| 83 | + |
| 84 | +### `concurrent.futures.ThreadPoolExecutor` |
| 85 | + |
| 86 | +```python |
| 87 | +import concurrent.futures |
| 88 | +import requests |
| 89 | +import sqlite3 |
| 90 | + |
| 91 | +URLS = [ |
| 92 | + 'http://www.foxnews.com/', |
| 93 | + 'http://www.cnn.com/', |
| 94 | + 'http://europe.wsj.com/', |
| 95 | + 'http://www.bbc.co.uk/', |
| 96 | + 'http://some-made-up-domain.com/', |
| 97 | +] |
| 98 | + |
| 99 | +def extract_and_load(url, timeout=30): |
| 100 | + try: |
| 101 | + web_result = requests.get(url, timeout=timeout).text |
| 102 | + print(f"{url} is {len(web_result)} bytes") |
| 103 | + |
| 104 | + with sqlite3.connect('example.db') as conn, conn as cursor: |
| 105 | + cursor.execute('CREATE TABLE IF NOT EXISTS web_results (url text, length int);') |
| 106 | + cursor.execute('INSERT INTO web_results VALUES (?, ?)', (url, len(web_result))) |
| 107 | + except Exception as e: |
| 108 | + print(f"{url} generated an exception: {e}") |
| 109 | + return False |
| 110 | + else: |
| 111 | + return True |
| 112 | + |
| 113 | +succeeded = [] |
| 114 | + |
| 115 | +with concurrent.futures.ThreadPoolExecutor() as executor: |
| 116 | + future_to_url = dict( |
| 117 | + (executor.submit(extract_and_load, url), url) |
| 118 | + for url in URLS |
| 119 | + ) |
| 120 | + |
| 121 | + for future in concurrent.futures.as_completed(future_to_url): |
| 122 | + succeeded.append(future.result()) |
| 123 | + |
| 124 | +print(f"Successfully completed {sum(1 for result in succeeded if result)}") |
| 125 | +``` |
| 126 | + |
| 127 | +### `pylateral` |
| 128 | + |
| 129 | +```python |
| 130 | +import requests |
| 131 | +import sqlite3 |
| 132 | + |
| 133 | +import pylateral |
| 134 | + |
| 135 | +URLS = [ |
| 136 | + 'http://www.foxnews.com/', |
| 137 | + 'http://www.cnn.com/', |
| 138 | + 'http://europe.wsj.com/', |
| 139 | + 'http://www.bbc.co.uk/', |
| 140 | + 'http://some-made-up-domain.com/', |
| 141 | +] |
| 142 | + |
| 143 | +@pylateral.task(has_return_value=True) |
| 144 | +def extract_and_load(url, timeout=30): |
| 145 | + try: |
| 146 | + web_result = requests.get(url, timeout=timeout).text |
| 147 | + print(f"{url} is {len(web_result)} bytes") |
| 148 | + |
| 149 | + with sqlite3.connect('example.db') as conn, conn as cursor: |
| 150 | + cursor.execute('CREATE TABLE IF NOT EXISTS web_results (url text, length int);') |
| 151 | + cursor.execute('INSERT INTO web_results VALUES (?, ?)', (url, len(web_result))) |
| 152 | + except Exception as e: |
| 153 | + print(f"{url} generated an exception: {e}") |
| 154 | + return False |
| 155 | + else: |
| 156 | + return True |
| 157 | + |
| 158 | +with pylateral.task_pool() as pool: |
| 159 | + for url in URLS: |
| 160 | + extract_and_load(url) |
| 161 | + |
| 162 | +succeeded = pool.results |
| 163 | + |
| 164 | +print(f"Successfully completed {sum(1 for result in succeeded if result)}") |
| 165 | +``` |
| 166 | + |
| 167 | +### Unthreaded |
| 168 | + |
| 169 | +```python |
| 170 | +import requests |
| 171 | +import sqlite3 |
| 172 | + |
| 173 | +URLS = [ |
| 174 | + 'http://www.foxnews.com/', |
| 175 | + 'http://www.cnn.com/', |
| 176 | + 'http://europe.wsj.com/', |
| 177 | + 'http://www.bbc.co.uk/', |
| 178 | + 'http://some-made-up-domain.com/', |
| 179 | +] |
| 180 | + |
| 181 | +def extract_and_load(url, timeout=30): |
| 182 | + try: |
| 183 | + web_result = requests.get(url, timeout=timeout).text |
| 184 | + print(f"{url} is {len(web_result)} bytes") |
| 185 | + |
| 186 | + with sqlite3.connect('example.db') as conn, conn as cursor: |
| 187 | + cursor.execute('CREATE TABLE IF NOT EXISTS web_results (url text, length int);') |
| 188 | + cursor.execute('INSERT INTO web_results VALUES (?, ?)', (url, len(web_result))) |
| 189 | + except Exception as e: |
| 190 | + print(f"{url} generated an exception: {e}") |
| 191 | + return False |
| 192 | + else: |
| 193 | + return True |
| 194 | + |
| 195 | +succeeded = [ |
| 196 | + extract_and_load(url) |
| 197 | + for url in URLs |
| 198 | +] |
| 199 | + |
| 200 | +print(f"Successfully completed {sum(1 for result in succeeded if result)}") |
| 201 | +``` |
0 commit comments