Framework for data analysis in Python

[ynikitenko]

Dear CCCL Team,

It is great to see NVIDIA focusing more on Python. Following our exchange with @aterrel and @shwina, I was encouraged to open a discussion here. What I have seen in cuda.compute is very similar to what I have been developing in my architectural framework for data analysis. Your team has captured important features of modern data analysis:

• lazy evaluation. This is especially relevant for large-scale data processing, since monolithic memory allocation of NumPy arrays can exhaust available memory,
• iterators. They have been proven to be useful abstractions since the introduction of the STL in C++.

Building on this, it is also possible to structure an entire analysis framework. A functional design greatly facilitates modularity – enabling clear structure, loose coupling, and code reuse. All of that is glued by Python, a very high-level and flexible language.

Apart from the features mentioned above, the framework also supports:

• Metadata. This is essential for modern analysis. The framework frees the analyst from tracing that manually.
• Composability. Analyses can be combined into larger pipelines without restructuring existing components.
• High-level optimization. Identical computational blocks can be detected and optimized.

Please refer to ReadTheDocs for a quick overview or watch a video of my presentation at the conference of Python developers in High-Energy Physics. The framework is under development, but the core features are stable, as they are based on solid design principles, my rigorous mathematical (MIPT, IUM) and software background (Debian/Ubuntu), and validated in real analysis workflows. Feel free to ask any relevant questions (test coverage, code maturity, etc.) here.

Lena was developed while solving real-world data analysis problems. A framework implies an initial entry threshold. On the other hand, having started, a scientist could use the same framework for decades. A common framework would greatly improve code quality and sharing – consider the Django ecosystem. The computational community and German funding agencies are steadily increasing requirements for exchangeable algorithms.

Why it could be interesting for NVIDIA:

• No existing framework for scientific data analysis. Django/Flask are frameworks that guide application structure. NumPy/pandas are libraries upon which users build analyses themselves.
• A natural extension of CUDA Python. An instrument to build an entire analysis is more valuable to a scientist than a computational optimization. For a high-level framework, low-level optimization is, in turn, a strong advantage.
• Performance. While NVIDIA excels at low-level optimizations, its recent shift to higher-level technologies enables substantial performance gains.

What are your thoughts on the strategic alignment of our goals? I would also be interested in discussing this with others working on long-term CUDA Python architecture.

[shwina]

Hi @ynikitenko - thanks for following up and sharing the lena project here. I tried it out to get an understanding of the functionality and features. Really nice work here!

My impression is that Lena operates at the much higher level of Python generators, while cuda.compute is ultimately bound by what can execute on the GPU (thus, calls to the Python runtime would be out of the question).

That being said, I do see some analogies. An interesting experiment would be the ability to execute certain lena expressions (Sequences) on the GPU via cuda.compute (by mapping flows to iterators). Admittedly I don't think it would be very trivial and nor would it just work for any arbitrary Sequence you can build up with lena today. Still, a proof of concept would be a good place to start.

Ultimately, cuda.compute is quite a low-level library. We would love for higher level libraries like lena to build on it to enable users to leverage GPUs. We're eager to learn from library developers like you as to what we can provide to help make that easier.

[ynikitenko]

Hi @shwina - thanks for looking into the library and for your kind words!

Pipelines can operate at all levels. Let me show you several examples.

1. Lena sequence.

This is the first example from the tutorial you’ve improved:

s = Sequence(
    lambda i: pow(-1, i) * (2 * i + 1),
)
spi = Source(
    CountFrom(0),
    Slice(10**6),
    s,  # reused sequence
    lambda x: 4./x,
    Sum(),  # defined elsewhere
)

2. CUDA beginner code.

Unfortunately, I could not launch a notebook yet, so I apologize for possible errors.

first_item = 0
num_items = 1000000

dtype = np.float64

# Create the counting iterator.
first_it = CountingIterator(np.int32(first_item))

def make_transform_iterator(op):
    return lambda underlying: TransformIterator(underlying, op)

# Is a TransformIterator allowed after a CountingIterator, or is array allocation needed? Unclear from docs
s_in = first_it
# (1.0 if i % 2 == 0 else -1.0) would probably be faster than pow
s_out = make_transform_iterator(lambda i: pow(-1, i) * (2 * i + 1))(s_in)

# int changes to double
l_out = make_transform_iterator(lambda x: 4./x)(s_out)

# Prepare the initial value for the reduction.
h_init = np.array([0.0], dtype=dtype)

# Prepare the output array.
d_result = cp.empty(1, dtype=dtype)

# Perform the reduction.
# Could use num_items only here; no slicing iterator available yet
cuda.compute.reduce_into(l_out, d_result, OpKind.PLUS, num_items, h_init)
result_host = d_result.get()
assert math.isclose(result_host[0], math.pi, abs_tol=1e-3)

This code is straightforward and in imperative style, though rather low-level.

3. “CUDA framework.”

User code:

first_item = 0
num_items = 1000000

dtype = np.float64

first_it = CountingIterator(np.int32(first_item))

# Create the computational sequence.
transform_it = make_source_it(
    first_it,
    lambda i: pow(-1, i) * (2 * i + 1),
    lambda x: 4./x
)
h_init = np.array([0.0], dtype=dtype)

# Prepare the output array.
d_result = cp.empty(1, dtype=dtype)

# similar to previous
cuda.compute.reduce_into(transform_it, d_result, OpKind.PLUS, num_items, h_init)
result_host = d_result.get()

Library code:

def make_source_it(start, *seq):
    s_in = start
    for el in seq:
        # for composability, it would be better to first create sequences, and then iterators; merge in one function now
        s_in = make_transform_iterator(el)(s_in)
    return s_in

Here, the user’s code, even though it could still be improved, is already more compact and structured.

4. GPU: 90% of analysis possible.

As I’ve shown in example 3, it is straightforward to construct analysis chains already based on your implemented libraries. You have exposed good interfaces, which allow computation on-the-fly and abstract compositions of transformation iterators.

The core feature of Data Analysis is that it uses mostly a standard set of functions: averages, standard deviations, quantiles, which are defined through simple mathematical operations and can be calculated on a GPU. Even when it is not enough, it is still necessary to calculate them in most cases. The fact that you’ve added histograms to the library makes it fit for 90% of analysis workflows.

5. 10% cases: fault tolerance.

As you correctly mentioned, not every calculation can be performed on a GPU. Some tasks are inherently unparallelizable; many tasks do not require high performance, and often analysts don’t have GPUs (for example, I’m writing from a laptop with an integrated graphics card). Analysts often don’t have the required skills, which creates an entry barrier for them.

Here a general framework would largely help. The reasons could be:

1. Lower entry threshold. There are estimates that about 90% or more of data analysts use Python. I believe this is also connected with the simplicity of the language. In that sense approaches 2-3 are still fairly technical and error-prone for many users,
2. Wider range of algorithms. Say we do a data analysis and realize that we need a complicated function involved. If we write the previous code for the GPU, we may have to throw it away (and probably won’t be eager to use that for future tasks). In any case that remains a constant risk (or a burden if we have to implement everything at once on a GPU).
3. Development. Wider range of architectures. We can develop and test our algorithms also without access to a powerful GPU cluster. We can develop programs that run on different (hybrid) architectures via framework backends. We don’t have a risk of a vendor lock-in for our code.

The features 2-3 can be called fault tolerance: If our program stops working on GPU, we lose performance, but can still execute that. We can also introduce optimizations only when needed and employ rapid development and prototyping. This largely improves the value of a general NVIDIA framework.

Even though an exclusive “CUDA framework” would simplify some tasks for the users (section 3), I believe that because of the limitations of GPU programming it won’t be general and useful enough: You’ve made a right decision to support only a library.

6. Unified developer interface.

You correctly noticed that Lena uses high-level iterators as well. Probably you meant another example of a “real analysis”:

s = Sequence(
        ReadData(),
        lambda dt: (dt[0][0], dt[1]),
        Histogram(mesh((-10, 10), 10)),
        ToCSV(),
        …
    )

Indeed, after reduction (producing a histogram from our data) we operate on higher-level objects. Though we don’t need a GPU performance for a handful of histograms or tables, we still use one sequence for low-level and high-level analysis parts, since it is easier for the user to think in terms of the same sequence of operations (no context switching). A unified design improves clarity and usability of our interface.

7*. Example from DALI.

@pipeline_def(num_threads=4, device_id=0)
def get_dali_pipeline():
    images, labels = fn.readers.file(
        file_root=images_dir, random_shuffle=True, name="Reader")
    images = fn.decoders.image_random_crop(
        images, device="mixed", output_type=types.RGB)
    images = fn.resize(images, resize_x=256, resize_y=256)
    images = fn.crop_mirror_normalize(images, ...)
    return images, labels

This is a shortened example from the NVIDIA Data Loading Library. It is clear that images variable can be omitted in the pipeline (compare with example 1). Moreover, the interface (images, labels) is exactly a special case of what is called (data, context) in Lena.

8. What CUDA developers can provide.

As I wrote, you have already implemented the core data analysis tools, which allows users to efficiently perform most standard analysis workflows. There are of course certain things to improve usability: For example, many tools are scattered across different CUDA libraries (like reading data and processing that).

We would love for higher level libraries to build on CUDA to enable users to leverage GPUs.

I’m curious about your perspective here. Why do you prefer users to rely on third-party libraries instead of building more directly on CUDA? Do you see the entry barrier for CUDA GPU programming as relatively high? Is it mainly due to the risk of subtle mistakes that can affect results, or has user interest in GPU programming only increased recently? What do you think are the main reasons?

If this feels like a long discussion, I’ve sent you a connection request on LinkedIn and we could discuss it in a short call.

[shwina]

Thanks @ynikitenko . I hear you about functionality being scattered across various libraries.

I’m curious about your perspective here. Why do you prefer users to rely on third-party libraries instead of building more directly on CUDA? Do you see the entry barrier for CUDA GPU programming as relatively high? Is it mainly due to the risk of subtle mistakes that can affect results, or has user interest in GPU programming only increased recently? What do you think are the main reasons?

As you noted earlier, we are only providing a library here (not a framework). Frameworks have all the advantages you mentioned in your initial comment in this discussion. Additionally, frameworks can unify functionality from the various libraries we provide, hiding all that complexity from its users.

If this feels like a long discussion, I’ve sent you a connection request on LinkedIn and we could discuss it in a short call.

Thanks! Are there specific challenges related to lena you would like to discuss that would benefit from a synchronous call?

[ynikitenko]

Thanks @shwina.

You write that it'd be easier for users to have a framework; however, NVIDIA supports only libraries. Is it programmers' team decision or a managerial one? What do you think could be the reasons for that?

Building upon NVIDIA Python libraries.
When will cuda.compute be considered stable? NVIDIA claims leadership also in software. If I build a framework on NVIDIA's libraries, can the company meanwhile create their own framework?

Do I understand right that NVIDIA is interested in scientific and data analysis applications? What do you think as a developer, if more people use CUDA libraries, will it generally improve its quality?

[danielfrg]

Hi @ynikitenko,

This is a really good discussion! NVIDIA is very interested in accelerating all scientific and data analysis applications and for sure we benefit from more people using CUDA libraries as we can increase the quality of the CUDA libraries and interfaces like cuda.compute.

In general we provide as many tools for developers and end users to be successful in as many areas of high performance compute as we can. So lower level libs like cuda.compute target developers who want to build on top of them but we also build and support higher level libraries that are more targeted to data scientists/analysts, for example cupy, cudf.pandas but nothing like the framework you describe today, in general I have seen some attempts to do similar stuff but nothing that was wildly (this is subjective) adopted by the community at large.

From the CCCL side as @shwina said our idea is to provide the building blocks for domain experts to build/accelerate their libraries. In general our approach is to accelerate the libraries and frameworks that our users need and not to be super prescriptive on what libraries they should use.

When will cuda.compute be considered stable?

We are working towards a 1.0 release soon and that would make the APIs stable.

And about the other question we are interesting in working with the community on building these tools, we cant promise resources but our approach is to work with our users and therefore the permissive opensource licenses in our libraries.

[ynikitenko]

Hi @danielfrg - glad that you found it interesting! Let me explain what I understand under frameworks.

Imagine your company is a renowned producer of bricks, windows and (high-level) components like walls and kitchens. Every consumer goes to one factory to buy bricks, then to another one to buy windows, and somehow assembles their own house as best they can. The question is: Why doesn't the company produce houses?

Evidently, a ready product would be easier for most users – but we don’t know that in advance. Even if the potential market is huge, we still risk resources (that is essentially venture investment). Then I would ask the following questions: How much can you invest into an innovative product? How much can you invest to improve your infrastructure? How much can you invest to pay your technical debt?

You mentioned that your first version would be 1.0.0 – which means you are preparing your product until it is ideal. In open source we often don’t make an ideal product, but first create a Minimal Viable Product (MVP) with version 0.1. That helps with scarce resources. Feedback from users then greatly helps with improving for the major release.

I was thinking why a framework for data analysis could fail:

• Not general enough. Indeed, it is of little use if implementing a different analysis requires switching frameworks.
• Not developed enough. One of Django’s major advantages was the ‘batteries included’ philosophy – the users didn’t have to code much themselves.
• Difficult to use.

In fact, there also existed Web frameworks before Django (and I also met some for data analysis), but it’s easy to make a mistake when creating them. Their failures, however, did not preclude the success of Django.

For users, as you have noted, a framework is a limitation. It takes control from the user and makes decisions for them. However, Braess's paradox shows that adding limitations can sometimes improve the system. And those are unimportant decisions: The user cares about their data and algorithm, not about how exactly data files are read and processed, or how memory is allocated.

A framework by NVIDIA could help with the following:

• Structured interface. Say I want to use CUDA for data analysis and I visit the CUDA Python documentation. I would like to know how I can read data directly into GPU memory… And cannot find that. Also, it’s not possible to guess from that page that a histogram could be found in the cuda.compute library. A framework could be a more gentle guideline into various existing libraries.
• Hide complexity. For example, to get a sum of numbers, one wouldn’t need to allocate memory because the framework would handle that. In fact, memory management is not Pythonic: The language usually manages that automatically.
• Safety/correctness. It’s easy to get wrong results by e.g. violating strict weak ordering. A framework in a “safe mode” could identify fragile places, look out for NaNs, etc.
• Performance. You have observed that kernel fusion improves speed. Lazy evaluation and computational pipelines facilitate kernel fusion. If users structure their analyses into pipelines, they can get more performance. My framework has demonstrated that an entire analysis can be structured into pipelines.

My personal reasons for a framework were:

• Extensible architecture.
• Common interfaces.
• Structured (clean) code.

Those were also the reasons why my framework did not gain popularity: They are not a priority of the market, they don’t solve its daily problems. Many scientists, even those specialized on programming, do not understand what a framework is and how it could be useful, they’ve never used Django. Most data-analysis tasks do not involve complex systems. Iterators and lazy evaluation are a new foundation for computation and reasoning compared to arrays. The framework, however, proved indispensable for my advanced analysis in experimental physics. I also believe that those features are strategically important in the age of poorly-written AI code, since structured code allows easier reasoning and better testing. And I believe those features could be most valuable for NVIDIA.

I’ve asked AI about major challenges of NVIDIA Python libraries, and these are its answers: 1) Fragmentation. There is no unified Python GPU layer. 2) Interoperability issues resulting in data copies between frameworks and hidden performance penalties, 3) Python control overhead: kernel launch overhead, many small GPU calls, synchronization issues, 4) GPU memory management. Better global memory orchestration would help. 5) Interface complexity. Structured GPU programming is missing, 6) User base imbalance. There is little tooling bridging the worlds of AI and HPC users.
Then the AI proposes declarative operations: “Users describe what they want. The framework decides how to run them.”

Developer team – please correct me if the AI is wrong. It mostly says what I wrote, but in different words. Probably you imagine a framework as a large product with many prepared decisions; but you may also think about it as an interface. If you don’t release a framework, you can miss an opportunity. If you wait with the interface, you accumulate technical debt, which grows invisibly and dangerously.

An important concern you mentioned is the ecosystem. A pipeline interface is orthogonal to the existing software. Data in a pipeline can still be NumPy arrays, so there is no competition. As I’ve shown, the already implemented transform iterators from cuda.compute can be naturally combined into pipelines. That is so trivial, that probably no library would do that: That belongs more to NVIDIA. In 2021 you released a blog post, “To date, access to CUDA and NVIDIA GPUs through Python could only be accomplished by means of third-party software... Each wrote its own interoperability layer between the CUDA API and Python.” An interface is what the company should define itself. In 2021, to write a kernel, one still had to do so in C++. Since that time your developers have greatly improved the interface for the users. You create great resources like accelerated computing notebooks; however, as I tried to run them, it seems that few people outside NVIDIA actually use them, which is a pity. I can not guarantee how many new users it will bring, but I guarantee, that the existing interface can be further improved and simplified.

In the end, success of a framework depends on the concrete framework and on how well your existing code base supports that (being 90% ready is a good sign).
That is why I’m asking you for a technical assessment of my argumentation.

I hope I could explain what a framework could be. In my opinion, you have all bricks to build a good framework, which could help users, third-party and CUDA developers themselves. Data analysis is important and also foundational for AI. If this direction is interesting for you, I would be glad to discuss it further and see whether my experience could be useful.

[jrhemstad]

@ynikitenko I often think of what we provide as bricks as well.

There are many examples of projects who use our building blocks to build their own custom frameworks.

One example of that would be RAPIDS cuDF. They use many of the building blocks in CCCL to build a more domain specific, data analytics framework/library.

Another example would be PyTorch who also uses CCCL algorithms.

In CCCL our focus is in providing general purpose building blocks that empower other teams and projects to make it easier to build CUDA accelerated libraries.

[ynikitenko]

@jrhemstad - thanks for joining this discussion! Yes, the foundational bricks are crucial for the architecture.

There are lots of NVIDIA libraries; today Jensen mentioned about 100 of them. It's hard for me to orient in them yet. What I understood is that CCCL provides exactly the building blocks for other NVIDIA (and not only) libraries. This is exactly its goal.

But then there is a question of how you decide: If you consider only power users (library developers), then they have to use whatever you provide. It does not give you an independent view of how good your interface was.

As I understood, recently you've switched to lazy evaluation (what I see from the library). Of course, other libraries are going to use that interface internally. Do you know whether they are going to change their interfaces based on that (I honestly doubt)? Are you going to expand, develop that interface further in CCCL?

Do I understand right that the top priority of CCCL is speed? (as I said, power developers can adapt to any interface) With that user base and goal there is no reason to include a data analysis framework into CCCL, of course. Even though data analysis is foundational to AI, it is based on more primitive computations.

[ynikitenko]

Dear CCCL Team,

I’ve recently been thinking about global optimizations that a framework could provide. Let us take a simple example:

(
    CountingIterator(from 1 to N),
    x * const,
    sum
)

This is a sum of numbers from 1 to N, multiplied by a constant. It is based on the benchmark from Delivering the Missing Building Blocks for NVIDIA CUDA Kernel Fusion in Python published last year.

In the benchmark, as well as in many CUDA Python documentation examples, the transformation is typically a Python function:

lambda x: x * const

I decided to try a symbolic wrapper (also called an Expression Template in C++). In the code above, x = SymbolicWrapper(). Such a wrapper stores operations applied to it in a tree, to be modified and calculated later. That would allow Ahead-of-Time optimizations, when the const could be moved outside the sum. It is straightforward to implement, and in the comparison below I assume it has already been done.

Performance

First, I tried more complicated examples using the library operations from OpKind. It turns out that a TransformIterator supports only functions (it was recently updated but does not seem to be available in Google Colab yet). In principle, there are some resources spent on introspecting a lambda, but they look tiny in the benchmark. That is why I focus on a single multiplication by a constant.

For many floating-point constants the benchmark is very close between a lambda and an expression tree. This is probably because of fused multiply-add, when an intermediate multiplication does not require additional processor cycles. However, I found a more complex example, where the constant is defined via an array and a lambda:

    const_arr = cupy.array([1.57], dtype=cupy.float32)
    # const_ = sin(const_arr[0])
    parallel_result, parallel_times = benchmark_parallel_approach(size, transform_op=lambda x: sin(const_arr[0]) * x)

Here are the benchmark results and the benchmark code I used. “Parallel” stands for the standard example with lambda.

=======================================================================================================================
BENCHMARK RESULTS SUMMARY
========================================================================================================================
N         Approach                       Mean (ms)    Std (ms)     Min (ms)     Max (ms)    
------------------------------------------------------------------------------------------------------------------------

Size: 1,000
------------------------------------------------------------------------------------------------------------------------
             Symbolic Wrapper               0.037        0.008        0.033        0.071       
             Parallel                       0.043        0.005        0.039        0.077       

Size: 1,000,000
------------------------------------------------------------------------------------------------------------------------
             Symbolic Wrapper               0.112        0.005        0.108        0.138       
             Parallel                       0.171        0.006        0.165        0.194       

Size: 10,000,000
------------------------------------------------------------------------------------------------------------------------
             Symbolic Wrapper               0.569        0.006        0.563        0.586       
             Parallel                       0.822        0.213        0.598        1.078       

Size: 100,000,000
------------------------------------------------------------------------------------------------------------------------
             Symbolic Wrapper               2.309        0.452        1.849        2.898       
             Parallel                       3.548        0.028        3.525        3.650       
========================================================================================================================
Note: All times are in milliseconds. Lower values indicate better performance.

The optimized algorithm would be about 50% faster than the standard one.

This is most likely due to the complicated runtime of Python. NVCC optimizes instructions quite well, but cannot optimize everything. One could argue that the example is somewhat exaggerated, but in our experiment we also use slightly different constants for each of the three detectors. An array of constants is realistic; in C++ we simply define that as constant, but standard Python does not support this directly. Here a framework, which would have a global view of the computations, could be of help.

Another framework advantage would be a horizontal fusion, that is applying several algorithms to the same data within a single kernel. In data analysis we often want to calculate both the mean and the standard deviation of the same variable. Do I understand correctly that cuda.compute does not support horizontal fusion yet? Are there plans to implement it?

Precision

For the simple problem at hand, we have a known answer. Moving a constant outside the sum improves accuracy. In our case of integer summation the results are exact; and when we don’t optimize lambdas, the results are wrong at the 8th digit. Naturally, it could get worse with more constants and computations. It is clear that for GPU algorithms the order of operations matters; however, for data analysis there is usually only one answer based on mathematics.

To sum up, a global view of the computations would provide more optimizations both for the speed and for the accuracy of the computations. In my framework I used global optimizations for high-level operations such as reading data files only once, but it turns out they can be useful also for micro-calculations, which can be still too complicated for low-level optimizers. On the human level, thinking about optimizations of single vs combined operations is also different: For the former, a lambda would be ideal; the benefits of a symbolic wrapper appear only for the latter.

Thanks to the CCCL Team for supporting the computational resources and for sharing the code benchmark.