misc: Update advisor with oneAPI 2025 #2533

georgebisbas · 2025-02-10T18:38:45Z

Some more changes are needed, but finally works

review-notebook-app · 2025-02-10T18:38:51Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

devito/arch/compiler.py

mloubout · 2025-02-10T18:48:12Z

devito/arch/compiler.py

+            # Locate the Intel Advisor installation and
+            # add the necessary flags to the compiler
+            from devito.operator.profiling import locate_intel_advisor
+            path = locate_intel_advisor()


Why aren't all those set by the setvars.sh that's what it's supposed to be for

Unfortunately, this was true with Intel studios, but now they are not by default. You have to link explicitly. At least, this is what I see.

And this is actually why it stopped working at some point

codecov · 2025-02-10T18:57:41Z

Codecov Report

Attention: Patch coverage is 41.66667% with 28 lines in your changes missing coverage. Please review.

Project coverage is 87.55%. Comparing base (e1dcb86) to head (fd5df3f).
Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
devito/operator/profiling.py	31.57%	12 Missing and 1 partial ⚠️
devito/arch/archinfo.py	25.00%	9 Missing ⚠️
tests/test_benchmark.py	40.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2533      +/-   ##
==========================================
+ Coverage   87.52%   87.55%   +0.03%     
==========================================
  Files         238      238              
  Lines       47034    47032       -2     
  Branches     4144     4143       -1     
==========================================
+ Hits        41165    41178      +13     
+ Misses       5176     5161      -15     
  Partials      693      693

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

benchmarks/user/advisor/README.md

EdCaunt · 2025-02-11T09:50:06Z

benchmarks/user/advisor/roofline.py

    project = advisor.open_project(str(project))

    if not project:
-        err('Could not open project %s.' % project)
+        err(f'Could not open project {project}.')


What does open_project return if it fails to open the project? This error message might return something cryptic since project gets overwritten

I was sure I had replied in this, not sure I cannot see my message. This is part of the advisor API, it should already be there

https://www.intel.com/content/www/us/en/developer/articles/training/how-to-use-the-intel-advisor-python-api.html

benchmarks/user/advisor/roofline.py

georgebisbas · 2025-02-11T15:35:10Z

examples/performance/02_advisor_roofline.ipynb

@@ -267,86 +272,6 @@
    "As you can see from this roofline graph, the main point is different from the single point of the previous graph. Moreover, each point is labelled with 'Time' and 'Incidence' indicators. These represent the total execution time of each loop's main body and their percentage incidence on the total execution time of the main time loop."
   ]
  },
-  {


All this is redundant in my opinion to be treated as-a-worthy to mention API

FabioLuporini · 2025-02-13T12:56:16Z

devito/operator/profiling.py

-            compiler.add_library_dirs(libdir)
-            compiler.add_ldflags('-Wl,-rpath,%s' % libdir)
+        super(AdvisorProfiler, self).__init__(name)
+        return


AFAICT, this is just yet another empty constructor? droppable?

devito/operator/profiling.py

FabioLuporini · 2025-02-13T12:58:53Z

devito/arch/compiler.py

@@ -771,6 +771,18 @@ def __init_finalize__(self, **kwargs):
            self.__init_intel_mpi__()
            self.__init_intel_mpi_flags__()

+        if configuration['profiling'] == 'advisor':


see comment below, but basically since "profiling" requires third-party tools (as opposed to MPI which is an actual API) I have a preference for this bit of logic to be kept in profiling

conftest.py

devito/arch/archinfo.py

devito/operator/profiling.py

FabioLuporini · 2025-02-14T07:56:35Z

devito/operator/operator.py

@@ -219,6 +219,12 @@ def _build(cls, expressions, **kwargs):

        # Required for the jit-compilation
        op._compiler = kwargs['compiler']
+
+        # Add any metadata from the profiler


nitpicking, I wouldn't call it "metadata"

hope is better

devito/arch/archinfo.py

tests/test_benchmark.py

FabioLuporini · 2025-02-14T09:02:58Z

conftest.py

@@ -80,7 +80,7 @@ def skipif(items, whole_module=False):
            skipit = "`icx+cpu64` won't work with this test"
            break
        # Skip if icx or advisor are not available
-        if i not in 'only-advisor' or \
+        if i not in ('noadvisor') or \


no this is wrong as it's not a tuple

omg, forgot the comma

FabioLuporini · 2025-02-14T09:03:34Z

devito/operator/profiling.py

+            libdir = path.joinpath('lib64').as_posix()
+            self._lib_dirs = [libdir]
+        else:
+            warning("Intel Advisor not found; reverting to `advanced`")


is it really reverting?

good catch, indeed it is reverting, but there is another mini-bug. It was reverting to basic, now it is reverting to advanced, and dropped a double-emmited warning due to this

georgebisbas · 2025-02-14T09:21:16Z

devito/operator/profiling.py

-        warning("Couldn't set up `%s` profiler; reverting to `advanced`" % level)
-        profiler = profiler_registry['basic'](name)


Here, it was reverting to basic

ok but no need for that extra var imho (nitpicking)

well this was only for a better fstring...but ok

FabioLuporini · 2025-02-14T12:56:43Z

devito/operator/profiling.py

+        if path:
+            self._include_dirs = [path.joinpath('include').as_posix()]
+            libdir = path.joinpath('lib64').as_posix()
+            self._lib_dirs = [libdir]


no, instances shouldn't change class attributes like that....

and btw why is it even a class attribute ?

What should it have been?

FabioLuporini · 2025-02-14T12:58:16Z

devito/operator/profiling.py

-        warning("Couldn't set up `%s` profiler; reverting to `advanced`" % level)
-        profiler = profiler_registry['basic'](name)


ok but no need for that extra var imho (nitpicking)

FabioLuporini · 2025-02-26T03:59:07Z

devito/operator/profiling.py

+
+        path = get_advisor_path()
+        if path:
+            self.add_include_dir(path.joinpath('include').as_posix())


as explained above, this should just be self._include_dirs.append(...)

FabioLuporini · 2025-02-26T14:39:21Z

conftest.py

@@ -79,6 +79,12 @@ def skipif(items, whole_module=False):
           isinstance(configuration['platform'], Cpu64):
            skipit = "`icx+cpu64` won't work with this test"
            break
+        # Skip if icx or advisor are not available
+        if i == 'noadvisor' and \
+           not isinstance(configuration['compiler'], IntelCompiler) or \


this is wrong again, I think, because of and/or operator precedence...

The fact we got this wrong multiple times and nothing breaks on CI suggests it's not very useful maybe

FabioLuporini · 2025-03-03T15:49:32Z

devito/operator/profiling.py

+        if path:
+            self._include_dirs.append(path.joinpath('include').as_posix())
+            self._lib_dirs.append(path.joinpath('lib64').as_posix())
+            self._attempted_init = True


maybe for another day, but these three lines

self._attempted_init = True else: self._attempted_init = False

could become a single line

self._attempted_init = bool(path)

but anyway, nitpicking... maybe you can do it in another PR

but why append then if path is None? If the compiler reverts to advanced in the case of advisor's absence, the paths will be linked right?

FabioLuporini · 2025-03-03T15:50:14Z

examples/performance/02_advisor_roofline.ipynb

@@ -158,9 +155,13 @@
     "name": "stdout",
     "output_type": "stream",
     "text": [
-      "\u001b[1;37;32mOpening project...\u001b[0m\n",
+      "\u001b[1;37;32mOpening project /home/gb4018/workspace/devitocodes/devito/examples/performance/profilings/JupyterProfiling...\u001b[0m\n",


This is not OK, I don't like personal dirs in logs (ie gb4018)

FabioLuporini · 2025-03-03T15:50:19Z

examples/performance/02_advisor_roofline.ipynb

@@ -218,9 +219,13 @@
     "name": "stdout",
     "output_type": "stream",
     "text": [
-      "\u001b[1;37;32mOpening project...\u001b[0m\n",
+      "\u001b[1;37;32mOpening project /home/gb4018/workspace/devitocodes/devito/examples/performance/profilings/JupyterProfiling...\u001b[0m\n",


mloubout · 2025-03-03T15:54:28Z

devito/operator/operator.py

@@ -219,6 +219,12 @@ def _build(cls, expressions, **kwargs):

        # Required for the jit-compilation
        op._compiler = kwargs['compiler']
+
+        # Required for compilation by the profiler


Why aren't those part of the profller init?

It was before, and this was the reason it was not working, as a different compiler was build after doing the profiler init. Related discussion here: https://devitocodes.slack.com/archives/C7JMLMSG0/p1739458707616429?thread_ts=1739458681.588539&cid=C7JMLMSG0

JDBetteridge

Nitpicks. But otherwise LGTM

JDBetteridge · 2025-03-06T11:45:22Z

benchmarks/user/advisor/roofline.py

-    # saving the chart in PNG format
-    plt.savefig('%s.png' % name, bbox_extra_artists=(legend,), bbox_inches='tight')
+    # saving the chart in PDF format
+    plt.savefig('%s.pdf' % name, bbox_extra_artists=(legend,), bbox_inches='tight')


Suggested change

plt.savefig('%s.pdf' % name, bbox_extra_artists=(legend,), bbox_inches='tight')

plt.savefig(f'{name}.pdf', bbox_extra_artists=(legend,), bbox_inches='tight')

JDBetteridge · 2025-03-06T11:47:30Z

benchmarks/user/advisor/README.md

  mode. This analysis, together with the `survey` analysis, is necessary to
  generate a roofline. Both are run by `run_advisor.py`.
-* Requires python3, untested in earlier versions of python and conda environments
-* Currently requires download of repository and running `pip3 install .`, the scripts
+* Requires Python3, untested in conda environments


Suggested change

* Requires Python3, untested in conda environments

* Requires Python 3.9 or later, untested in conda environments

Let's not use unsupported Python versions!

JDBetteridge · 2025-03-06T11:49:12Z

benchmarks/user/advisor/roofline.py

    figpath = os.path.realpath(__file__).split(os.path.basename(__file__))[0]
-    log('Figure saved in %s%s.png.' % (figpath, name))
+    log('\nFigure saved in %s%s.pdf.' % (figpath, name))


Suggested change

log('\nFigure saved in %s%s.pdf.' % (figpath, name))

log(f'\nFigure saved in {figpath}{name}.pdf.')

JDBetteridge · 2025-03-06T11:49:35Z

benchmarks/user/advisor/roofline.py


    # Save the JSON file
    with open('%s.json' % name, 'w') as f:
        f.write(json.dumps(roofline_data))

-    log('JSON file saved as %s.json.' % name)
+    log('\nJSON file saved as %s.json.' % name)


Suggested change

log('\nJSON file saved as %s.json.' % name)

log(f'\nJSON file saved as {name}.json.')

Co-authored-by: Jack Betteridge <[email protected]>

FabioLuporini · 2025-03-07T08:26:15Z

Merged, thanks

georgebisbas added the documentation label Feb 10, 2025

georgebisbas requested review from FabioLuporini, mloubout, EdCaunt and JDBetteridge February 10, 2025 18:38

georgebisbas self-assigned this Feb 10, 2025

mloubout reviewed Feb 10, 2025

View reviewed changes

EdCaunt reviewed Feb 11, 2025

View reviewed changes

georgebisbas commented Feb 11, 2025

View reviewed changes

FabioLuporini reviewed Feb 13, 2025

View reviewed changes

devito/operator/profiling.py Show resolved Hide resolved

FabioLuporini reviewed Feb 13, 2025

View reviewed changes

georgebisbas force-pushed the advisor_refresh_II branch 3 times, most recently from 4c1f058 to ed910e5 Compare February 14, 2025 07:48