Fix the workflows (#883)

* fix batch_inference_v2:infer() endpoint_id docstring

* new structure

* fix workflow

* trigger function test

* trigger function test

* modify workflow to work on push event

* modify workflow to work on push event

* test if its a matter with the grep

* fix the matrix creation step

* mock the functions directoy (instead of with path we used)

* correct the location of requirements.txt

* correct the location of requirements.txt

* upgrade mlrun version due to cython bug

* remove dummy-category

* re-structure cli

* correct functions_cli path

* ci.yaml fix attempt

* ci.yaml fix attempt #2

* ci.yaml fix attempt #3

* align the fix to test-all workflow

* fix command flag (long option must start with --)

* one centralized cli directory

* cli import path fix attempt

* adjusting test-all to the new structure

* investigate why there appears to be 0 passed tests

* investigate why there appears to be 0 passed tests #2

* add a print to function name

* fix attempt to functions discovery

* fix describe_dask cython bug by upgrading the mlrun version

* fix sklearn_calssifier cython bug by upgrading the mlrun version

* fix cython bug by upgrading all mlrun versions to be >= 1.7.0

* Revert "fix cython bug by upgrading all mlrun versions to be >= 1.7.0"

This reverts commit 61bcd26.

* fix cython bug failures by upgrading mlrun versions to be >= 1.7.0

* delete logs

* adds logs to debug describe_dask stuck test

* adds flag to view logs to debug describe_dask stuck test

* remove the debug prints

* test workflow optimization trick

* test workflow optimization trick (fix)

* test workflow optimization trick (test-all)

* test new matrix building method

* test new matrix building method (identation fix)

* test new matrix building method (another fix attempt)

* test new matrix building method (another fix)

* test new matrix building method (another fix attempt)

* remove redundant logging

* test new matrix building method also for test-all

* invalidate the sklearn_classifier test

* remove extra logging

* invalidate test due to mlrun-varsion required update (no module named mlrun.mlrutils)

* add dummy module test

* add modules/README and dummy_module/item.yaml so the test will be identified

* test build-marketplace new command

* test build-marketplace new command (remove needs)

* fix the project-root path (after moving helpers in utils sub-directory)

* fix worng cli path

* add print of temp_docs to inspect the module not found error

* attempt to fix module not found by inserting their path to sys.path

* attempt to fix module not found by inserting their path to sys.path #2

* test notebooks internal links fix

* attempt to upload the result as artifact to github

* another fix attempt to notebooks' broken links

* another fix attempt to notebooks' broken links

* test build-marketplace on modules

* add assumed fields and files to dummy module for testing

* fic dummy_module filename

* check is the function.yaml exist before rendering it

* READMEs auto-update

* READMEs auto-update (add install requirements step)

* chore(readme): auto-update asset tables [skip ci]

* remove the old functions/readme section

* replace the deprecated set-output command

* remove the debug mode from the workflows

* don't mention asset-types by name in the command

* fix and modify the check for function.yaml file

* remove todo

* fixes + adapt to upcoming repo rename

* add the branch to the trigger top test workflow

* uncomment the conditions trigger top test workflow

* revert the debug mode

* remove the dummy module

* remove todo

* remove mlrun/functions mentions

* post review fixes

* revert debug mode

* remove redundant character

* keep empty modules/src

* remove unwanted changes from last commit

* Delete .DS_Store

* Delete modules/.DS_Store

* Delete modules/src/.DS_Store

* avoid including non-directories in the tests matrix

* removed .ds_store files

* skip testing if matrix is empty

* skip testing if matrix is empty (#2)

* debug why matrix is empty

* debug why matrix is empty by removing filter step

* debug why matrix is empty by removing filter step

* rename changed-files to all-files to match this workflow

* missing $

* extra checks to exclude the root from the matrix

* extra checks to exclude the root from the matrix

* debug the empty path by logging candidates

* add grep to remove empty lines from json

* align ci to the fix as well

* align ci to the fix as well

* align ci to the fix as well

* align ci to the fix as well

* reset to development and start again debug process

* debug mode

* grep to remove empty lined from metrix json

* align ci.yaml to the fix and remove debug mode

* new-line delimiter

---------

Co-authored-by: iguazio-cicd <iguaziocicd@gmail.com>

Author: 2 people

.github/workflows/ci.yaml

@@ -63,6 +63,7 @@ jobs:
             matrix_json='{"package":[]}'
           else
             matrix_json=$(printf '%s\n' "$packages" \
+              | grep -v '^$' \
               | jq -R . \
               | jq -s '{package: .}' \
               | jq -c)   # <-- compact

.github/workflows/test-all.yaml

@@ -36,7 +36,7 @@ jobs:
           all_files="$(git ls-files || true)"
           # Collect candidate package paths from diff
           candidates=$(
-            printf '%s\n' "$changed_files" \
+            printf '%s\n' "$all_files" \
             | awk -F'/' '
                 /^functions\/src\// {print $1"/"$2"/"$3}
                 /^modules\/src\//  {print $1"/"$2"/"$3}
@@ -55,7 +55,7 @@ jobs:
           if [[ -z "$packages" ]]; then
             matrix_json='{"package":[]}'
           else
-            matrix_json=$(printf '%s\n' "$packages" | jq -R . | jq -s '{package: .}' | jq -c)
+            matrix_json=$(printf '%s\n' "$packages" | grep -v '^$' | jq -R . | jq -s '{package: .}' | jq -c)
           fi
           echo "matrix=$matrix_json" >> "$GITHUB_OUTPUT"
     outputs: