899 extend, test and benchmark data transfer between Python and DAPHNE#949
Merged
pdamme merged 26 commits intodaphne-project:mainfrom May 21, 2025
Merged
899 extend, test and benchmark data transfer between Python and DAPHNE#949pdamme merged 26 commits intodaphne-project:mainfrom
pdamme merged 26 commits intodaphne-project:mainfrom
Conversation
…pytorch 899 benchmarks tensorflow pytorch
update fork repository
irhox
changed the title
Update Fork
# Conflicts: # src/api/python/daphne/operator/nodes/matrix.py
add previous lost commits
…ransfer from python lists to daphne
Contributor
Author
|
Since creating this PR, there have been a couple updates:
|
Contributor
Author
|
Update:
|
Contributor
Author
|
Update:
|
…tion and addition using numpy, pandas, pytorch, tensorflow and daphnelib
…erent python libraries
Contributor
Author
|
Final updates:
|
irhox
changed the title
pdamme
added
DaphneLib
pdamme
force-pushed
the
main
branch
4 times, most recently
from
a24cae3 to
a6614de
Compare
- Removed experimental results (not needed in the code base). - Fixed minor issues in the documentation and the example script. - Cleaned up the DaphneLib source code: - Merged from_numpy()/from_numpy_numeric() and from_pandas()/from_pandas_numeric(); these variants partly duplicated code and unnecessarily complicated the API. - Fixed the string data transfer via files (in code_line(), it stored the matrix as a JSON file, which cannot work correctly as JSON data cannot be read as CSV on the other side; the fix uses appropriate formatting for np.savetxt() depending on the dtype). - Removed some unused or irrelevant code. - Reintroduced various code lines that were removed without clear reason, e.g., in script.py. - Undid pure whitespace changes. - Various other little things. - Restructured the new DaphneLib test cases. - Removed the unrelated "function_*" test cases. - Removed script files that were not used in any test case, e.g., "data_transfer_numpy.py" and "data_transfer_pandas.py". - More systematic naming scheme for the newly added test script files. - Reduced the number of newly introduced test script files by testing special float values (nan, inf, large, small, etc.) all in the same script instead of separate scripts. - Simplified the test scripts. - Undid the renaming of some existing test script files. - Removed the specialization of the receiveFromNumpy-kernel for string value type, since string data transfer via shared memory is anyway not supported in this PR.
pdamme
approved these changes
May 21, 2025
Collaborator
pdamme
left a comment
pdamme
left a comment
There was a problem hiding this comment.
Thanks @irhox and @mariakrzywnicka for improving the data transfer between Python and DAPHNE by (a) adding support for more Python data structures (e.g., Python lists, string-valued numpy arrays, 1-dimensional numpy arrays), (b) fixing some bugs related to the data transfer, and (c) adding numerous additional test cases. This contribution is highly welcome.
I tidied up the code in this PR thoroughly (see the commit I added for details) and now it's ready to be merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
This PR contains additions, bug fixes, new tests and benchmarks on the data transfer features of the daphnelib library
This PR addresses the concerns in issue: #899 and was done by @mariakrzywnicka and @irhox
First update: We extended the tests regarding data transfer between python and daphne and found two bugs in the data transfer with numpy and pandas. One of them was fixed immediately, the other was reported.