Paper: On the Path to Seamless Python Serverless Data Analytics - #1107
Conversation
…edings_2025 into lithops-dataplug
…ings_2025 into enrique_molina
|
Curvenote Preview
|
rowanc1
added
paper
|
Hello - My name is Praveen. Thanks for inviting me over email to review this paper. I’m glad to contribute and look forward to providing helpful feedback. |
|
Inviting reviewer: @nitinappiah@gmail.com |
|
Hi, I’m Nitin Appiah. I’ll be reviewing your paper and will share detailed feedback soon. Looking forward to working with the SciPy team and providing reviews for the authors. |
spraveengupta
left a comment
spraveengupta
left a comment
There was a problem hiding this comment.
This paper is a great read and highlights some important yet relatively unexplored tools for analytics. The following comments are intended to help make the work even stronger and more impactful.
Thanks for the opportunity to review this submission. I’ll take one more pass again to ensure completeness.
|
|
||
| The rise of the serverless paradigm introduced a radically different model, centered on functions with automatic scaling and pay-per-use billing. To enable data analytics on serverless architectures, Lithops emerges as a powerful Python framework that abstracts away infrastructure complexities, empowering seamless parallel execution through serverless functions and allowing users to focus purely on their code. | ||
|
|
||
| Another persistent challenge in serverless data analytics is the efficient partitioning of data, a crucial factor for maximizing parallelism and minimizing latency in data transfer from storage to compute resources. Dataplug addresses this by enabling on-the-fly partitioning for standard data formats, optimizing data distribution among workers. |
There was a problem hiding this comment.
Consider aligning the phrasing in this section with the conclusion that has more clarity. For instance, you might reframe the sentence as
Another Persistent challenge in serverless data analytics is efficient partitioning of complex scientific datasets that are not originally designed for cloud environments.
|
|
||
| The rise of the serverless paradigm introduced a radically different model, centered on functions with automatic scaling and pay-per-use billing. To enable data analytics on serverless architectures, Lithops emerges as a powerful Python framework that abstracts away infrastructure complexities, empowering seamless parallel execution through serverless functions and allowing users to focus purely on their code. | ||
|
|
||
| Another persistent challenge in serverless data analytics is the efficient partitioning of data, a crucial factor for maximizing parallelism and minimizing latency in data transfer from storage to compute resources. Dataplug addresses this by enabling on-the-fly partitioning for standard data formats, optimizing data distribution among workers. |
There was a problem hiding this comment.
Similarly, replacing the word standard in the second statement with something along the lines of datasets not designed for cloud environments can provide extreme transparency
|
|
||
| Another persistent challenge in serverless data analytics is the efficient partitioning of data, a crucial factor for maximizing parallelism and minimizing latency in data transfer from storage to compute resources. Dataplug addresses this by enabling on-the-fly partitioning for standard data formats, optimizing data distribution among workers. | ||
|
|
||
| To further streamline the scientific development experience, PyRun tackles the steep learning curve posed by configuring and managing serverless environments for users without DevOps expertise. By providing a shared, remote browser-based environment seamlessly connected to the cloud provider, PyRun drastically reduces the cognitive burden of setup — delivering a ready-to-execute environment with zero configuration required. |
There was a problem hiding this comment.
It would be helpful to clarify that PyRun is an IDE, similar to other existing tools (e.g., [provide examples such as JupyterLab, VS Code, etc.]).
Highlighting this comparison, along with its added capability to manage underlying infrastructure, can help readers better understand its unique value proposition.
| print("Estimated Pi: {}".format(pi)) | ||
| \end{lstlisting} | ||
|
|
||
| The code snippet above illustrates a straightforward example of using the Lithops framework to estimate the value of $\pi$. First, an instance of \texttt{lithops.FunctionExecutor()} is created. The \texttt{map()} method then launches multiple serverless functions that execute the \texttt{is_inside()} function concurrently, each processing a portion of the input data. Once all tasks complete, \texttt{get_result()} collects the results from the distributed execution and receives them locally, enabling the user to obtain the final estimation seamlessly. |
There was a problem hiding this comment.
Consider briefly explaining how this function estimates the value of Pi. A concise clarification would help readers unfamiliar with the method understand the underlying logic and significance of this simulations approach.
|
|
||
| \begin{figure}[htbp] | ||
| \centering | ||
| \includegraphics[width=0.7\textwidth]{assets/cdf_activation.pdf}\label{fig:cdf} |
There was a problem hiding this comment.
I believe “CDF activation” in the plot here refers to the cumulative distribution function. To improve clarity for a broader audience, consider rephrasing it in simpler terms, such as “percentage of functions activated”, if that accurately reflects the intended meaning.
|
|
||
| METASPACE initially relied on a Spark cluster for its computational backend. However, as user numbers grew and workloads became more demanding, the limitations of this setup became increasingly evident. Spark’s static resource allocation and rigid execution model made it difficult to handle large, diverse datasets efficiently. This often led to resource underutilization or performance bottlenecks, restricting the system’s ability to scale effectively. | ||
|
|
||
| To address these issues, METASPACE transitioned to a serverless architecture, aiming for greater flexibility and ease of management. By adopting Lithops, the platform gained the ability to scale computing resources dynamically based on workload demands. This move significantly reduced operational overhead while improving performance and cost efficiency. The switch to a serverless model allowed METASPACE to overcome the constraints of its Spark-based infrastructure and provided a more responsive and scalable environment for data processing. In Figure~\ref{fig:wallclock} we can see one the execution timeline of one METASPACE metabolomics pipeline sample, which is a complex workflow that includes multiple steps of parallel computations: the variability in the demand for resources is evident between stages, with some steps requiring much more compute power than others. This variability is a key driver for the adoption of serverless computing, aspect in which Lithops excels, as it allows for dynamic scaling---by providing just the right amount of resources---based on the specific needs of each step in the pipeline. |
There was a problem hiding this comment.
There may be a small typo in the phrase we can see one the execution timeline. Should one be on in this context? Clarifying this would improve readability.
|
Hello @spraveengupta , I'm Germán, a co-author on this paper, and I found your comments both constructive and valuable. I have addressed your feedback and have incorporated your in-text suggestions. The renaming of the plot axis labels is the only pending item, which I will address shortly. I have added references for JupyterLab and Dask, along with a URL for Google Colab, as a formal DOI was unavailable (as far as I know). I have revised the final paragraph on the metabolomics pipeline to improve readability. Thank you for your time and helpful feedback. |
|
|
||
| Also in the metabolomics field, it is common to work with large datasets that are not cloud-optimized, such as imzML~\citep{imzml} files. The imzML format is a standard for storing mass spectrometry imaging data, but it does not natively support cloud-aware access patterns. To address this challenge, the Dataplug framework implements the required plugin for imzML files, enabling efficient data on-the-fly partitioning. This allows the scientific community to process large imzML files without the need for statically partitioning them into smaller chunks. Dataplug framework extracts the necessary metadata from the imzML files, enabling dynamic partitioning strategies that can adapt to the specific needs of each analysis. This approach significantly reduces the preprocessing time and storage costs associated with traditional static partitioning methods, as shown in Figure~\ref{fig:dataplug_preprocess}. Dynamic partitioning also paves the way for researchers to experiment with different partitioning strategies without the need to rewrite the original data, providing greater flexibility and significant ingestion time savings due to parallel data access, that are shown in Figure~\ref{fig:dataplug_downloading}. | ||
|
|
||
| A common issue in metabolomics involves processing large, non-cloud-optimized datasets, such as files in the standard imzML format~\citep{imzml}. While widely used for mass spectrometry imaging, imzML lacks native support for cloud-aware access patterns. The Dataplug framework overcomes this limitation with a plugin that enables efficient, on-the-fly partitioning of imzML files. By extracting key metadata, Dataplug facilitates dynamic partitioning strategies tailored to the specific need of each workload, eliminating the need to create static, pre-processed data chunks. This approach yields significant advantages: it reduces preprocessing time and storage costs (Figure~\ref{fig:dataplug_preprocess}) and enables parallel data access, which drastically cuts data ingestion times (Figure~\ref{fig:dataplug_downloading}). Furthermore, it offers researchers the flexibility to experiment with different partitioning schemes without modifying the original dataset. |
There was a problem hiding this comment.
I refer to this paragraph about the metabolomics pipeline, which I just rephrased for readability
|
We adapted the text, addressing your last suggestions and observations - this plot remains to be corrected. This week (hopefully on thursday) I will add:
Thank you |
|
Hi folk, I will serve as the editor for this paper when reviews are complete. Editor: @ameyxd |
|
Hi, I added a comparison with PySpark, and the cost savings given different workload frequencies in a testbed similar to the real METASPACE scenario (also took workload frequencies close to the real ones). During the next days I will upload the IO-performance evaluations. Thanks once again for your reviews German |
|
@ameyxd - @GEizaguirre has addressed my feedback and it is already in a great shape. I am wondering what's the next step? Do you want me to approve the changes from the files changed tab? (wanted to make sure that it is not blocked by me.) |
|
Hi, This time I added an I/O-intensive workload on different cloud providers and storage backends, answering point 1 (this review). I linked it to performance & transparency in the explanation. For now, that should be the last evaluation I am updating the paper with, if not asked to add further experiments. Thank you |
|
@GEizaguirre, thanks for updating the results based on my comments. I have reviewed the additions and I approve them. |
|
@kikemolina3 are you satisfied with the reviews and edits? |
|
Yes! Everything is okay. Thank you very much for your reviews. |
|
Hello, We just noticed that while the web-based version of the paper looks fine, the generated PDF has some plots incorrectly formatted (the sizes appear too small). Would it be possible for us to proceed with this change? Thank you very much in advance, and apologies for the inconvenience. |
Paging @fwkoch. |
|
Besides the request/rec from @fwkoch, the content and quality of the work looks good me. @kikemolina3 - good job. Approved. |
|
@kikemolina3 - Yes, as long is this PR is still open, feel free to make changes to image sizes (or any other formatting changes that do not impact the content). On publish, we will also make a final editorial pass to ensure both the web version and the PDF look good, so if there is anything you are having trouble fixing just let us know. |
|
Thanks @fwkoch. Approved. |
|
Minor style changes done! |
Co-authored-by: German Telmo Eizaguirre Suarez <germantelmo.eizaguirre@urv.cat> Co-authored-by: German <germantelmo.eizaguirre@urv.cat>
c786ef9 to
fb296a5
Compare
fwkoch
added
approved
9 participants
This is a submission of a paper evolving the next presentations:
that will be presented on Scipy Conference 2025.