[Experimental] Integrate ONNX Passes Front End #116
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds the same constant amount to the estimate for the stitched IP verification simulation which was already present for the RTL simulation performance measurement, but also adds the option to overwrite this via the LIVENESS_THRESHOLD in both cases.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
📋 Docstring Check Report Checked files:
❌ Docstring check failed! Missing Docstrings Details:📄 src/finn/custom_op/fpgadataflow/hls/pool_hls.py:
📄 src/finn/custom_op/fpgadataflow/pool.py:
Total missing docstrings: 20 How to Fix:Please add docstrings to the missing functions, classes, and modules listed above. Docstring Guidelines:
Raw output from docstring checker |
Note: This is mostly a workaround for missing datatype inference for Reduce operators in QONNX. Inferring these as pooling enables type inference via the FINN custom-op. Eventually QONNX should handle this.
Note: Inferring Reshape is included as a default hardware conversion transformation, meaning any Reshape remaining to the end of hardware conversion will be turned into the Reshape pass-through operator.
Note: Previously this assumed identical input and output shape for PE-ops, which might not always be the case, e.g., Reshape. As PE refers to the output parallelism, the fallback should be derived from the output shape to get valid folding.
|
📋 Docstring Check Report Checked files:
❌ Docstring check failed! Missing Docstrings Details:📄 src/finn/transformation/fpgadataflow/set_folding.py:
Total missing docstrings: 6 How to Fix:Please add docstrings to the missing functions, classes, and modules listed above. Docstring Guidelines:
Raw output from docstring checker |
|
📋 Docstring Check Report Checked files:
❌ Docstring check failed! Missing Docstrings Details:📄 src/finn/transformation/streamline/round_thresholds.py:
Total missing docstrings: 2 How to Fix:Please add docstrings to the missing functions, classes, and modules listed above. Docstring Guidelines:
Raw output from docstring checker |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR explores integration of a new ONNX IR and ONNX Script based front end mostly covering streamlining and graph-level optimization transformations. Most of the added steps comprises glue to apply passes from the pass library developed at ONNX Passes and for converting back to the FINN-supported format afterwards. Most of these could eventually be reduced to a bare minimum once transitioning more infrastructure to ONNX IR and ONNX Script, making the formats directly compatible.
Contributions in Detail
TODO: There might be even more? Split/Concat, Im2Col, Pooling is mostly done, needs testing though...
Where to go from here?
Seems like
step_qonnx_to_finnand (with some extra effort)step_tidy_upcould be immediately deprecated as everything done by those is covered by the new transformations or could at least be recreated without too much effort. Due to the mismatch between QONNX and ONNX IR representation, we probably still want to maintain astep_tidy_upusing QONNX transformations with aModelWrapperto be able to apply this at any point within a step sequence, at least for some transitional period.With more and more transformations being recreated as ONNX Passes, or even as part of the ONNX Script rewriter, we can probably gradually phase out the
step_streamline, eventually moving any remaining FINN specifics into the hardware conversion step or creating new, more explicitly named steps.The new infrastructure could be used to overhaul almost all of the transformations and steps, but I am not sure this is actually worth the effort, as beyond
step_convert_to_hwmost transformations merely set/update annotations and node attributes or call into simulation or synthesis tools. I don't see any problems here which could be solved by new infrastructure or model representation.TODO: Propose and discuss a strategy for when/how to use the new infrastructure vs. the old QONNX Transformation base when creating new or reworking existing transformations... probably best to discuss this offline for the start?
TODO: Discuss how to extend the tight verification and ONNX Runtime execution (for verification) to the rest of the flow - probably this is infeasible due to complexity and runtime of C++ and RTL simulation, but some aspects, such as metric-based verification, could be incorporated?
Layout Annotation and Conversion
Now also contributes a new experimental approach to the ongoing data layout issues (see #53, #84, #86, #87, #91, #101, fastmachinelearning/qonnx/issues/173). For details refer to the convolution demo at ONNX Passes.
This aims at replicating the core idea of the "data layout" concept of FINN and QONNX without the baggage of layout propagation: Layouts are annotated as (for now unenforced) assumptions and matching transposes are inserted which try to propagate the information via simple (compared to layout propagation) streamlining.
This might require further extension of streamlining transpose operators and modifications to the
MultiThresholdoperator in QONNX and the conversion of these to FINN'sThresholdingoperator.Im2Coland transposes to not block the flow of layout information in front of padded convolution.Other Contributions
Includes PRs #127 and #128 for implementing Reshape operations remaining in the graph after streamlining as some of the architectures to be enabled by this new front end have "real" reshape operations as part of the graph.