add lazy data doc for comments#4793
Conversation
|
Great, will take a look! |
|
Thanks! I'll take a closer look later but at first glance, this looks like a good summary of what 'lazy' means and what 'compute' and 'persist' are about. Of course a user guide will need to have some concrete, executable examples but this looks like a great start! |
| "source": [ | ||
| "### Lazy Data\n", | ||
| "\n", | ||
| "Lazy data, or lazy loading of data, is a technique used in which data is not actually loaded until it is needed. If used effectively, this allows for a more efficient use of system resources and performance. Additionally, a user can build up a list of tasks, or build a task graph, that uses the data, but doesn't actually run anything until told to do so. The *expression* is stored instead of the result.\n", |
There was a problem hiding this comment.
Maybe emphasize that lazy evaluation lets you separate describing what computations need to be done from actually doing them, and then have a bulleted or numbered list of why capturing such a pipeline lazily can be important:
- Lets you select different compute engines at the point the computation is done (e.g. to parallelize the computation, run it on GPUs, run it on a remote server, etc.)
- Lets you describe a series of operations where some of them might drastically reduce the size of the problem before computation begins, e.g. by sampling/selecting/slicing a large dataset for a case when only a small portion is actually used
- Lets you run the operation "out of core", i.e., processing only a chunk at a time rather than loading all of the data at once, when the same operation must be done on many datapoints that together would not fit into memory
etc.
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "As explained here: https://holoviz.org/tutorial/Large_Data.html, HoloViews can accept Dask dataframes just as well as Pandas dataframes. It has the infostructure to accept the lazy data objects, and will evaluate the objects as needed in order to display the relevant information. \n", |
There was a problem hiding this comment.
I think infostructure counts as jargon here, and should be replaced with a more specific description of what you mean. computational infrastructure?
| "source": [ | ||
| "### When would you want to use .persist() when using HoloViews?\n", | ||
| "\n", | ||
| "If you are going to be doing any computations before plotting (aggregating or reducing your data, for example), calling .persist() on these computations will avoid evaluating that portion of the task graph several times. These computations will be stored in memory on the cluster nodes, ready to be used when needed for your various plots. \n", |
There was a problem hiding this comment.
| "If you are going to be doing any computations before plotting (aggregating or reducing your data, for example), calling .persist() on these computations will avoid evaluating that portion of the task graph several times. These computations will be stored in memory on the cluster nodes, ready to be used when needed for your various plots. \n", | |
| "If you are going to be doing any computations before plotting (aggregating or reducing your data, for example), calling .persist() on these computations will avoid evaluating that portion of the task graph repeatedly, e.g. as a user zooms or pans on the plot. These computations will be stored in memory on the cluster nodes, ready to be used when needed for your various plots. \n", |
| "\n", | ||
| "### When do you use .compute()? \n", | ||
| "\n", | ||
| "Some methods you may use will not support data being stored on multiple nodes. You may need to perform a sort on the entire dataset or get an index location with with `.iloc()`, and these require putting all the data on one node. In this case, it will be best to do as much as you can to reduce or aggregate your data on each node, and then use `.compute()` to bring everything to a single node. You can then use the methods that do not work across nodes to obtain your desired result. " |
There was a problem hiding this comment.
I'd say something about ideally .compute() would return a reduced version of the original data -- aggregated, sliced, selected, etc., so that the resulting object is reasonable to fit in a single machine's memory even for a large distributed dataset.
|
One general thought, it might be worth giving a really basic and fundamental overview of what 'lazy' means. The easiest way to do this in Python is to compare lists to generators: eager = [i for i in range(1000)] # versus
lazy = (i for i in range(1000000000000000000000000000000000)) # This is ok but don't try it in the list case! |
|
Good point, but maybe use an example where the result could actually be used for something. E.g. an infinite generator for something like a Fibonacci sequence, but then you ask for the first 10 terms. Something lazy, followed by actually evaluating it for something reasonable, and pointing out that the eager version computes all 1000 then gives you the first 10, while the lazy one only ever does the first 10. |
|
In that case you would use |
|
@jbednar @jlstevens @philippjfr |
4 participants
I have started a document for lazy data that @philippjfr and @jlstevens and I discussed awhile back. I included a cell with Philipp's notes on what to include. I think I need to include some examples, but I wanted to get feedback on what I have so far before I proceed.