Question/Problem about calling FINN build inside files instead of terminal

[moviaso1]

Hello,

I have a question/problem when trying to use the FINN+ build inside of a file compared to calling it in the terminal. I am currently implementing an AutoML tool (Archai) that generates DNNs and I want to create FINN accelerators from these DNN directly after the tool found them. For this, I want to use the FINN building steps inside a file instead of calling the build in the terminal afterwards. However, when I try to run the FINN building steps I get an error for the set_fifo_depths step. The error message looks like this:

Before I run the building steps I call prepare_finn() to set the needed dependencies, etc. like this:

I don't really know, what causes this error, because when try to run the FINN build using the terminal for the found models, it works, meaning the models themselves are not the problem. The way I set up FINN+ was by setting up a virtual environment, cloning the Github repository and execute poetry install. I also tried it using the pip install route but in both cases the error occurred.
I hope someone could maybe help me with this problem.

Thank you in advance,
Marvin

[bwintermann]

Thanks for bringing this to our attention!
This error happens because FINN can't find the finn_xsi package responsible for simulation during the FIFO Sizing step. Could you try to print(os.environ["PYTHONPATH"]) in your build script (after prepare_finn)? prepare_finn is supposed to setup the package and add its path to PYTHONPATH. Normally, we would recommend to run finn run or finn build, so we never tested a call to prepare_finn outside of FINN+ itself. In the meanwhile I'll try to reproduce your issue locally and will let you know if I find something.

[bwintermann]

Thanks! So far I wasn't able to reproduce it on my side. Could you send the entire script that you are executing?
Also, are you running finn run <script> or python3 <script>, using the environments interpreter directly?

[moviaso1]

The complete script looks like this:

from overrides import overrides
from typing import List
import torch
import torch.nn as nn
import numpy as np
import json
from random import Random
from tqdm import tqdm
import math
import os
from pathlib import Path

from archai.discrete_search.api import ArchaiModel, EvolutionarySearchSpace, BayesOptSearchSpace
from archai.api.dataset_provider import DatasetProvider
from archai.discrete_search.api import ModelEvaluator
from archai.discrete_search.evaluators import RayParallelEvaluator, EvaluationFunction
from archai.datasets.cv.mnist_dataset_provider import MnistDatasetProvider
from archai.discrete_search.api import SearchObjectives
from archai.discrete_search.evaluators import AvgOnnxLatency, TorchFlops, TorchNumParameters, TorchLatency
from archai.discrete_search.algos import EvolutionParetoSearch, RandomSearch, MoBananasSearch

from archai_testing.archai_quant_complete import QuantModel
from archai_testing.archai_quant_complete import QNNSearchSpace, QNNSearchSpaceExt

from brevitas.export import export_qonnx

from qonnx.custom_op.registry import getCustomOp
import finn.builder.build_dataflow as build
import finn.builder.build_dataflow_config as build_cfg
import finn.util.data_packing as dpk
from finn.interface.run_finn import prepare_finn
from finn.interface.run_finn import build as finn_build

#finn_build_dir = os.environ["FINN_BUILD_DIR"]


class PartialTrainingValAccuracy(ModelEvaluator):
    """Evaluates a model by training it for a partial number of epochs and returning validation accuracy."""

    def __init__(self, dataset: DatasetProvider, training_epochs: float  = 1.0, lr: float = 1e-4,
                 device: str = 'cpu', progress_bar: bool = False):
        self.dataset_provider = dataset_provider
        self.training_epochs = training_epochs
        self.lr = lr
        self.device = device
        self.progress_bar = progress_bar

    @overrides
    def evaluate(self, model, budget = None) -> float:
        train_data = self.dataset_provider.get_test_dataset()
        val_data = self.dataset_provider.get_val_dataset()

        train_dl = torch.utils.data.DataLoader(train_data, batch_size=16, shuffle=True, num_workers=4)
        val_dl = torch.utils.data.DataLoader(val_data, batch_size=16, shuffle=False, num_workers=4)

        optimizer = torch.optim.Adam(model.arch.parameters(), lr=self.lr)
        criterion = nn.CrossEntropyLoss()

        model.arch.train()
        model.arch.to(self.device)

        epoch_iter = range(math.ceil(self.training_epochs))
        if self.progress_bar:
            epoch_iter = tqdm(epoch_iter, desc=f"Training model {model.archid}")

        for epoch_nb in epoch_iter:
            early_step = len(train_dl) + 1
            if 0 < (self.training_epochs - epoch_nb) < 1:
                early_step = int((self.training_epochs - epoch_nb) * len(train_dl))

            for i, (x, y) in enumerate(train_dl):
                if i >= early_step:
                    break

                optimizer.zero_grad()

                pred = model.arch(x.to(self.device))
                loss = criterion(pred, y.to(self.device))

                loss.backward()
                optimizer.step()

        model.arch.eval()

        with torch.no_grad():
            val_pred, val_target = [], []

            for x, y, in val_dl:
                val_pred.append(model.arch(x.to(self.device)).argmax(axis=1).to("cpu"))
                val_target.append(y.to("cpu"))

            val_pred, val_target = torch.cat(val_pred, axis=0), torch.cat(val_target, axis=0)
            val_acc = (val_pred.squeeze() == val_target.squeeze()).numpy().mean()

        model.arch.cpu()

        filename = "/home/student/finn_new_env/finn-plus/archai_testing/quant_random/onnx_models/"+str(model.archid)+".onnx"
        input_shape = np.zeros(shape=(1, 1, 28, 28)) #input shape for ONNX export
        input_shape = torch.tensor(input_shape, dtype=torch.float32)
        export_qonnx(model.arch.cpu(), input_t=torch.randn(input_shape.shape), export_path=filename)

        return val_acc
    

class FINN_LUT_Estimation(ModelEvaluator):
    """Evaluates a model by creating FINN resource estimation reports and returning estimated LUTs."""

    def __init__(self):
        super(ModelEvaluator, self).__init__()

    @overrides
    def evaluate(self, model, budget = None):
        #try:

        model_name = model.arch.get_archid()
        model_file = "/home/student/finn_new_env/finn-plus/archai_testing/quant_random/onnx_models/"+str(model_name)+".onnx"
        platform_name = "fpga"

        prepare_finn(deps=Path("/home/student/.finn/deps").expanduser(),
                     flow_config=Path("/home/student/finn_new_env/finn-plus/finn_config.yaml").expanduser(), 
                     build_dir=Path("/home/student/finn_new_env/finn-plus/FINN_TMP").expanduser(), 
                     num_workers=24,
                     is_test_run=True)
        
        print("PYTHONPATH = ",os.environ["PYTHONPATH"])

        build_steps = build.default_build_dataflow_steps
        #build_steps = build_cfg.estimate_only_dataflow_steps

        
        cfg = build.DataflowBuildConfig(
            steps=build_steps,
            board="ZCU104",
            output_dir="/home/student/finn_new_env/finn-plus/archai_testing/quant_random/FINN_builds/output_%s_ZCU104" % (model_name),
            #mvau_wwidth_max = 80,
            target_fps=1000000,
            synth_clk_period_ns=10.0,
            #folding_config_file="folding_config.json",
            #fpga_part="xc7z020clg400-1",
            shell_flow_type=build_cfg.ShellFlowType.VIVADO_ZYNQ,
            #stitched_ip_gen_dcp=False,
            generate_outputs=[
                build_cfg.DataflowOutputType.ESTIMATE_REPORTS,
                build_cfg.DataflowOutputType.STITCHED_IP,
                build_cfg.DataflowOutputType.RTLSIM_PERFORMANCE,
                #build_cfg.DataflowOutputType.OOC_SYNTH,
                build_cfg.DataflowOutputType.BITFILE,
                build_cfg.DataflowOutputType.PYNQ_DRIVER,
                build_cfg.DataflowOutputType.DEPLOYMENT_PACKAGE
            ],
            #verify_steps=[
            #    build_cfg.VerificationStepType.TIDY_UP_PYTHON,
            #    build_cfg.VerificationStepType.STREAMLINED_PYTHON,
            #    build_cfg.VerificationStepType.FOLDED_HLS_CPPSIM,
            #    build_cfg.VerificationStepType.STITCHED_IP_RTLSIM,
            #],
            save_intermediate_models=True,
        )
        
        
        build.build_dataflow_cfg(model_file, cfg)
        
                
        try:
            with open("/home/student/finn_new_env/finn-plus/archai_testing/quant_random/FINN_builds/output_%s_ZCU104/report/estimate_layer_resources.json" % (model_name)) as f:
                data = json.load(f)
                
            LUTs = data["total"]["LUT"]
            print(LUTs)
            return int(LUTs)
        except Exception as e:
            print("Build did not complete, cannot find estimated layer resources")
            return 0


class FINN_FPS_Estimation(ModelEvaluator):
    """Evaluates a model by reading out the previously created FINN resource estimation reports and returning estimated FPS."""

    def __init__(self):
        super(ModelEvaluator, self).__init__()

    @overrides
    def evaluate(self, model, budget = None):
        
        model_name = model.arch.get_archid()
                
        try:
            with open("/home/student/finn_new_env/finn-plus/archai_testing/quant_random/FINN_builds/output_%s_ZCU104/report/estimate_network_performance.json" % (model_name)) as f:
                data = json.load(f)
                
            fps = data["estimated_throughput_fps"]
            print(fps)
            return int(fps)
        except Exception as e:
            print("Build did not complete, cannot find estimated network performance")
            return 0       


search_space = QNNSearchSpaceExt(max_layers=4, kernel_list=[3, 5], hidden_list=[7, 14, 28], bit_width_list=[4, 6, 8])

dataset_provider = MnistDatasetProvider()
tr_d = dataset_provider.get_train_dataset()

objectives = SearchObjectives()
objectives.add_objective(name="ONNX Latency (ms)", model_evaluator=AvgOnnxLatency(input_shape=(1, 1, 28, 28), num_trials=3),
                         higher_is_better=False, compute_intensive=False)
objectives.add_objective(name='Partial taining Validation Accuracy (1 Epochs)', 
                         model_evaluator=PartialTrainingValAccuracy(dataset_provider, training_epochs=1, device="cuda"),
                         higher_is_better=True, compute_intensive=True)
objectives.add_objective(name="Number of parameters", model_evaluator=TorchNumParameters(), higher_is_better=False, constraint=(0.0, 1e6), compute_intensive=False)
objectives.add_objective(name="Estimated LUTs", model_evaluator=FINN_LUT_Estimation(), higher_is_better=False, compute_intensive=True)
objectives.add_objective(name="Estimated FPS", model_evaluator=FINN_FPS_Estimation(), higher_is_better=True, compute_intensive=True)


algo = RandomSearch(search_space=search_space, search_objectives=objectives, output_dir="/home/student/finn_new_env/finn-plus/archai_testing/quant_random",
                             num_iters=3, save_pareto_model_weights=True, seed=42, clear_evaluated_models=False)
search_result = algo.search()

In short, it imports the search space for the AutoML search and then implements evaluators that decide, which found models perform the best. The Partial_Accuracy evaluator first trains the models and also creates ONNX files of them that are later save in a folder. The implemented FINN evaluator then uses these ONNX files and originally only generates the estimation reports and then reads the resources from them. But now I want to extent it by creating the complete accelerator and not only the estimation reports. The FINN build inside the evaluators are called when running the search algorithms search_result = algo.search().

The way I run the script inside the virtual environment is by calling it via python:
/home/student/finn_new_env/bin/python /home/student/finn_new_env/finn-plus/archai_testing/complete_algo_quant.py

[bwintermann]

I am relatively sure that the problem is in the import order. You import finn.builder.... at the top of the file, however prepare_finn gets called afterwards. So the function correctly modifies the path, but at this point, since finn.builder... imports the entire framework indirectly, the finn_xsi was already unsucessfully loaded, setting finnxsi = None, which is why you got that error. The fix would be to import the finn modules only after having called prepare_finn. This is also what we do in src/finn/interface/run_finn.py, which does not immediately import the rest of FINN, but only after the environment is prepared.

Using FINN+ this way seems quite useful, so we'll discuss implementing a fix for this in our next developers meeting.

[moviaso1]

Hi, running prefare_finn() before the finn imports solved it. Thank you very much!

[fpjentzsch]

@bwintermann Maybe this will help once we pull it in from finn/dev? Xilinx#1437