Kubeflow v2.0 issue "'NoneType' object has no attribute 'path'"

[Mohit-Verma-1688]

Hello Team,

This is my first time I am writing any question on this platform. Please forgive me if my question is dumb. I am trying to learn kubeflow , and I am kinda stuck indefinitely on one issue. I want to use how Input/Output or intputPath/outputPath functionality in my demo pipeline. I am trying to make an artifact as output[Dataset], which I can use further in the next step in my pipeline. However even after 50 attempts, I am not getting how to do that. May be this is a issue in my understanding of the concept. Below is what I read

https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#using-output-artifacts

=> environment is running on macbook with kind cluster

kind v0.21.0 go1.21.6 darwin/arm64

installation via -> kubectl apply -k 'github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic?timeout=120&ref=2.2.0'

kfp                       2.7.0
kfp-kubernetes            1.2.0
kfp-pipeline-spec         0.3.0
kfp-server-api            2.0.5

~/Doc/S/d/s/python-containerized ❯ tree src                                                   

src
└── components
    ├── data_download
    │   ├── Dockerfile
    │   ├── __pycache__
    │   │   └── data_download_component.cpython-310.pyc
    │   ├── component_metadata
    │   │   └── dataset_download.yaml
    │   ├── data_download_component.py
    │   ├── kfp_config.ini
    │   └── runtime-requirements.txt
    └── model_train_cnn
        ├── Dockerfile
        ├── __pycache__
        │   └── model_train_component.cpython-310.pyc
        ├── component_metadata
        │   └── model_train.yaml
        ├── data_setup.py
        ├── engine.py
        ├── kfp_config.ini
        ├── model_builder.py
        ├── model_train_component.py
        ├── runtime-requirements.txt
        └── utils.py

=> Below is what my component looks like, In this step I want to create an output artifact which is a dataset

@dsl.component(base_image='mohitverma1688/model_train_component:v0.22',
               target_image='mohitverma1688/model_train_component:v0.33',
               packages_to_install=['pandas']
               )

def model_train(num_epochs:int, 
                batch_size:int, 
                hidden_units:int,
                learning_rate: float,
                train_dir: str,
                test_dir: str,
                model_name: str,
                model_dir: str,
                output_data: Output[Dataset],   --> Here
                export_bucket: str = "modelbucket",        
               )  :

            import os
            import json
            import pandas as pd
            import torch
            import data_setup, engine, model_builder, utils

            from torchvision import transforms
         
            

            # Setup hyperparameters
            NUM_EPOCHS = num_epochs
            BATCH_SIZE = batch_size
            HIDDEN_UNITS = hidden_units
            LEARNING_RATE = learning_rate
            MODEL_NAME = model_name
            MODEL_DIR = model_dir
            EXPORT_BUCKET = export_bucket

    

            # Setup directories
            TRAIN_DIR = train_dir
            TEST_DIR = test_dir

            # Setup target device
            device = "cuda" if torch.cuda.is_available() else "cpu"

            # Create transforms
            data_transform = transforms.Compose([
              transforms.Resize((64, 64)),
              transforms.ToTensor()
            ])

            # Create DataLoaders with help from data_setup.py
            train_dataloader, test_dataloader, class_names = data_setup.create_dataloaders(
                train_dir=TRAIN_DIR,
                test_dir=TEST_DIR,
                transform=data_transform,
                batch_size=BATCH_SIZE
            )

            # Create model with help from model_builder.py
            model = model_builder.TinyVGG(
                input_shape=3,
                hidden_units=HIDDEN_UNITS,
                output_shape=len(class_names)
            ).to(device)

            # Set loss and optimizer
            loss_fn = torch.nn.CrossEntropyLoss()
            optimizer = torch.optim.Adam(model.parameters(),
                                         lr=LEARNING_RATE)

            # Start training with help from engine.py
            result = engine.train(model=model,
                         train_dataloader=train_dataloader,
                         test_dataloader=test_dataloader,
                         loss_fn=loss_fn,
                         optimizer=optimizer,
                         epochs=NUM_EPOCHS,
                         device=device)

            df = pd.DataFrame(result)     ---> here I am saving the dataset. 
            with open(output_data.path, "w") as file:
              file.write(df.to_csv(index=False))
                
                
            # Save the model with help from utils.py
            utils.save_model(model=model,
                             model_dir=MODEL_DIR,
                             model_name=MODEL_NAME + ".pth",
                             export_bucket=EXPORT_BUCKET)

=> This dataset is the resulted dict which I am converting to a dataframe and what I want as my output artifact.

import pandas as pd
model_0_df = pd.DataFrame(x)
model_0_df

    test_acc    test_loss   train_acc   train_loss
0   0.197917    1.099857    0.316406    1.100338
1   0.541667    1.090166    0.253906    1.104582
2   0.541667    1.090699    0.281250    1.101027
3   0.260417    1.093742    0.277344    1.097989
4   0.260417    1.089232    0.304688    1.097009
5   0.541667    1.083633    0.414062    1.094837
6   0.541667    1.078700    0.281250    1.110045
7   0.541667    1.081974    0.281250    1.100708
8   0.541667    1.087431    0.402344    1.096542
9   0.541667    1.086511    0.281250    1.097733

=> This is how the pipeline looks.

@dsl.pipeline(name='CNN-TinyVG-Demo',
              description='This pipeline is a demo for training,evaluating and deploying Convutional Neural network',
              display_name='Kubeflow-MlFLow-Demo')



def kubeflow_pipeline(base_path: str = BASE_PATH,
                     url:str = URL,
                     batch_size:int = BATCH_SIZE,
                     train_dir:str = TRAIN_DIR,
                     test_dir:str = TEST_DIR,
                     input_bucket:str = INPUT_BUCKET,
                     num_epochs: int = NUM_EPOCHS,
                     hidden_units:int = HIDDEN_UNITS,
                     learning_rate:float = LEARNING_RATE,
                     model_name: str = MODEL_NAME,
                     model_dir: str = MODEL_DIR,
                     export_bucket: str = EXPORT_BUCKET
                     ):
    pvc1 = kubernetes.CreatePVC(
        # can also use pvc_name instead of pvc_name_suffix to use a pre-existing PVC
        pvc_name='kubeflow-pvc4',
        access_modes=['ReadWriteOnce'],
        size='500Mi',
        storage_class_name='standard',
    )
    task1 = dataset_download(base_path=base_path,
                            url=url,
                            input_bucket=input_bucket)
    task1.set_caching_options(True)
    task2 = model_train(batch_size=batch_size,
                        num_epochs=num_epochs,
                        train_dir=train_dir,
                        test_dir=test_dir,
                        hidden_units=hidden_units,
                        learning_rate=learning_rate,
                        model_name=model_name,
                        model_dir=model_dir,
                        export_bucket=export_bucket,
                        ).after(task1)
    task2.set_caching_options(False)
 #   task3 = model_inference(data_dir=DATA_DIR).after(task2)
 #   task3.set_caching_options(False)
    kubernetes.mount_pvc(
        task1,
        pvc_name=pvc1.outputs['name'],
        mount_path='/data',
    )
    kubernetes.mount_pvc(
        task2,
        pvc_name=pvc1.outputs['name'],
        mount_path='/data',
    )

=> This is what I get in the error

[Mohit-Verma-1688]

Please ignore, I got my mistake :D , The mistake was in my imports. I was using notebook to create the pipeline yaml and the imports from component were not reflecting. I switched to terminal and created pipeline from correct folder and component changes reflected.

[rimolive]

Can we consider this discussion answered? If so, can you mark as ansrewed?