Benchmarks

Author: probably-not

README.md

@@ -18,6 +18,16 @@ Wrap your untrusted NIFs so that they can never crash your node.
 > For right now, it can be used to wrap any function or MFA that might cause some sort of crash on the BEAM node in order to keep that function safe and isolated.
 > It currently carries performance penalties of peer node startup and code loading, however warm node pooling is in development to optimize this performance penalty.
 
+## Benchmarks
+
+Benchmarks can be found in the [`bench`](./bench/) directory. As of v0.1.0, SafeNIF has not implemented pooling of peer nodes.
+This means that it currently incurs the high cost of starting up a peer node for every call, which can take anywhere from 100ms to over a second,
+depending on how much code needs to be loaded onto the peer node. You can see from the benchmarks that out of the three methods benchmarked (CLI+Port, NIF, SafeNIF),
+SafeNIF is currently the slowest due to this incurred cost.
+
+Adding pooling will be implemented in v0.2.0 and should make this far more efficient as we will only need to incur the cost once per node created.
+It should be noted that pooling will incur different costs - namely memory and CPU since it spins up a node on the same machine.
+
 **The following information was generated by Claude and Reviewed by @probably-not. If issues in this README are found, feel free to open up a PR to fix them!**
 
 ## The Problem

bench/README.md

@@ -0,0 +1,68 @@
+# SafeNIF Benchmarks
+
+Most of the code in this directory was generated by Claude and reviewed by @probably-not.
+
+This directory contains benchmarks comparing SafeNIF's performance characteristics.
+
+## Running Benchmarks
+
+Ensure dependencies are installed and NIF/Port are compiled:
+
+```bash
+mix deps.get
+mix compile
+```
+
+Then run individual benchmarks (must use `--sname` for distribution):
+
+```bash
+# Main comparison: NIF vs SafeNIF vs Port
+elixir --sname bench -S mix run bench/nif_vs_port.exs
+
+# Payload size impact
+elixir --sname bench -S mix run bench/payload_sizes.exs
+```
+
+## Benchmark Files
+
+| File                | Description                                       |
+| ------------------- | ------------------------------------------------- |
+| `nif_vs_port.exs`   | Compares Direct NIF, SafeNIF, and Port approaches |
+| `payload_sizes.exs` | Tests how data size affects overhead              |
+
+## Support Modules
+
+| File              | Description                                     |
+| ----------------- | ----------------------------------------------- |
+| `helpers.ex`      | Shared setup/teardown and Benchee configuration |
+| `port_wrapper.ex` | GenServer wrapping the Port executable          |
+
+## Results
+
+Results are saved to `bench/results/` as Markdown files. These are gitignored
+by default, but you may want to commit significant baseline results.
+
+## Understanding the Results
+
+### Direct NIF (Unsafe)
+- **Fastest** - No isolation overhead
+- **Dangerous** - Crashes take down the entire node
+- Use as baseline for "best possible" performance
+
+### Port
+- **Traditional safe approach** - Isolation via OS process
+- **Overhead**: Process spawn + stdio + serialization
+- Good for: Stateless operations, moderate throughput needs
+
+### SafeNIF
+- **Peer node isolation** - Each call spawns a BEAM node
+- **Overhead**: BEAM startup + code loading + distribution
+- Good for: Untrusted code, crashy NIFs, safety > speed
+
+### When to Use What
+
+| Approach   | Use When                                                          |
+| ---------- | ----------------------------------------------------------------- |
+| Direct NIF | Trusted code, high performance needed, crashes acceptable         |
+| Port       | Moderate isolation needs, stateless ops, reasonable throughput    |
+| SafeNIF    | Untrusted code, must not crash main node, throughput not critical |

bench/helpers.ex

@@ -0,0 +1,68 @@
+defmodule SafeNIF.Bench.Helpers do
+  @moduledoc """
+  Shared helpers for SafeNIF benchmarks.
+
+  Generated by Claude and reviewed by @probably-not
+  """
+
+  alias SafeNIF.Bench.PortWrapper
+  alias SafeNIFTest.TestNIF
+
+  @doc """
+  Ensures all benchmark dependencies are ready.
+  Returns {:ok, context} or {:error, reason}.
+  """
+  def setup do
+    with :ok <- ensure_distributed(),
+         :ok <- ensure_nif_loaded(),
+         :ok <- ensure_port_exists() do
+      {:ok, %{}}
+    end
+  end
+
+  @doc """
+  Cleans up benchmark resources.
+  """
+  def teardown(_ctx), do: :ok
+
+  defp ensure_distributed do
+    if Node.alive?() do
+      :ok
+    else
+      {:error, :not_distributed}
+    end
+  end
+
+  defp ensure_nif_loaded do
+    case TestNIF.load_nif() do
+      :ok -> :ok
+      {:error, reason} -> {:error, {:nif_not_loaded, reason}}
+    end
+  end
+
+  defp ensure_port_exists do
+    if File.exists?(PortWrapper.port_path()) do
+      :ok
+    else
+      {:error, {:port_not_found, PortWrapper.port_path()}}
+    end
+  end
+
+  @doc """
+  Standard Benchee configuration.
+  """
+  def benchee_config(opts \\ []) do
+    markdown_file = Keyword.get(opts, :markdown_file, "bench/results/latest.md")
+
+    [
+      warmup: 2,
+      time: 10,
+      memory_time: 2,
+      reduction_time: 2,
+      formatters: [
+        Benchee.Formatters.Console,
+        {Benchee.Formatters.Markdown, file: markdown_file}
+      ]
+    ]
+  end
+end

bench/nif_vs_port.exs

@@ -0,0 +1,104 @@
+# SafeNIF Benchmark: NIF vs SafeNIF vs CLI Port
+#
+# Compares three approaches to running native code:
+#   1. Direct NIF call - fastest, but crashes take down the node
+#   2. SafeNIF wrapped - isolated via peer nodes (new BEAM per call)
+#   3. CLI Port - isolated via OS process (new process per call)
+#
+# Run with:
+#   elixir --sname bench -S mix run bench/nif_vs_port.exs
+#   Generated by Claude and reviewed by @probably-not
+
+alias SafeNIF.Bench.Helpers
+alias SafeNIF.Bench.PortWrapper
+alias SafeNIFTest.TestNIF
+
+IO.puts("""
+================================================================================
+SafeNIF Benchmark: NIF vs SafeNIF vs CLI Port
+================================================================================
+
+This benchmark compares execution overhead for three isolation strategies:
+
+  • Direct NIF:  No isolation - crashes kill the node
+  • CLI Port:    OS process isolation - spawns new process per call
+  • SafeNIF:     Peer node isolation - spawns new BEAM node per call
+
+Both CLI Port and SafeNIF provide true isolation (fresh process per call).
+
+""")
+
+# Setup
+case Helpers.setup() do
+  {:ok, _ctx} ->
+    IO.puts("✓ Setup complete\n")
+
+    # Verify everything works before benchmarking
+    IO.puts("Verifying implementations...")
+    :ok = TestNIF.safe_noop()
+    :ok = PortWrapper.noop()
+    {:ok, :ok} = SafeNIF.wrap({TestNIF, :safe_noop, []})
+
+    42 = TestNIF.safe_add(17, 25)
+    42 = PortWrapper.add(17, 25)
+    {:ok, 42} = SafeNIF.wrap({TestNIF, :safe_add, [17, 25]})
+
+    IO.puts("✓ All implementations working\n")
+
+    # Run benchmarks
+    IO.puts("Running benchmarks...\n")
+
+    Benchee.run(
+      %{
+        "Direct NIF (unsafe)" => fn -> TestNIF.safe_noop() end,
+        "CLI Port" => fn -> PortWrapper.noop() end,
+        "SafeNIF" => fn -> SafeNIF.wrap({TestNIF, :safe_noop, []}) end
+      },
+      Helpers.benchee_config(markdown_file: "bench/results/noop_comparison.md")
+    )
+
+    IO.puts("\n" <> String.duplicate("=", 80) <> "\n")
+
+    Benchee.run(
+      %{
+        "Direct NIF (unsafe)" => fn -> TestNIF.safe_add(17, 25) end,
+        "CLI Port" => fn -> PortWrapper.add(17, 25) end,
+        "SafeNIF" => fn -> SafeNIF.wrap({TestNIF, :safe_add, [17, 25]}) end
+      },
+      Helpers.benchee_config(markdown_file: "bench/results/add_comparison.md")
+    )
+
+    IO.puts("""
+
+    ================================================================================
+    Benchmark complete!
+
+    Results saved to:
+      • bench/results/noop_comparison.md
+      • bench/results/add_comparison.md
+
+    Key insights:
+      • Direct NIF shows the baseline (no isolation overhead)
+      • CLI Port shows OS process spawn overhead (~1-10ms typical)
+      • SafeNIF shows BEAM node spawn overhead (~500-2000ms typical)
+
+    Note: SafeNIF is designed for SAFETY, not speed. Use it when you need
+    BEAM-level isolation (crash protection, code loading, distribution).
+    ================================================================================
+    """)
+
+  {:error, :not_distributed} ->
+    IO.puts("""
+    ❌ Node is not running in distributed mode.
+
+    Run benchmarks with:
+      elixir --sname bench -S mix run bench/nif_vs_port.exs
+    """)
+
+    System.halt(1)
+
+  {:error, reason} ->
+    IO.puts("❌ Setup failed: #{inspect(reason)}")
+    IO.puts("\nMake sure to run 'mix compile' first to build the NIF and Port.")
+    System.halt(1)
+end

bench/payload_sizes.exs

@@ -0,0 +1,99 @@
+# SafeNIF Benchmark: Payload Size Impact
+#
+# Tests how different payload sizes affect overhead for each approach.
+# Important for understanding when SafeNIF is appropriate.
+#
+# Run with:
+#   elixir --sname bench -S mix run bench/payload_sizes.exs
+#   Generated by Claude and reviewed by @probably-not
+
+alias SafeNIF.Bench.Helpers
+alias SafeNIF.Bench.PortWrapper
+alias SafeNIFTest.TestNIF
+
+IO.puts("""
+================================================================================
+SafeNIF Benchmark: Payload Size Impact
+================================================================================
+
+Tests how data size affects overhead for NIF vs CLI Port vs SafeNIF.
+All three use the same echo operation for fair comparison.
+
+""")
+
+case Helpers.setup() do
+  {:ok, _ctx} ->
+    IO.puts("✓ Setup complete\n")
+
+    # Test payloads
+    payloads = %{
+      "tiny (10 B)" => :crypto.strong_rand_bytes(10),
+      "small (100 B)" => :crypto.strong_rand_bytes(100),
+      "medium (1 KB)" => :crypto.strong_rand_bytes(1_000),
+      "large (10 KB)" => :crypto.strong_rand_bytes(10_000),
+      "xlarge (100 KB)" => :crypto.strong_rand_bytes(100_000)
+    }
+
+    # Verify all implementations work with all payload sizes
+    IO.puts("Verifying echo implementations...")
+
+    for {name, data} <- payloads do
+      ^data = TestNIF.safe_echo(data)
+      ^data = PortWrapper.echo(data)
+      {:ok, ^data} = SafeNIF.wrap({TestNIF, :safe_echo, [data]})
+      IO.puts("  ✓ #{name}")
+    end
+
+    IO.puts("")
+
+    # Benchmark each payload size
+    for {size_name, data} <- payloads do
+      IO.puts("Benchmarking #{size_name}...\n")
+
+      safe_filename =
+        size_name
+        |> String.downcase()
+        |> String.replace(~r/[^a-z0-9]/, "_")
+        |> String.replace(~r/_+/, "_")
+        |> String.trim("_")
+
+      Benchee.run(
+        %{
+          "Direct NIF" => fn -> TestNIF.safe_echo(data) end,
+          "CLI Port" => fn -> PortWrapper.echo(data) end,
+          "SafeNIF" => fn -> SafeNIF.wrap({TestNIF, :safe_echo, [data]}) end
+        },
+        Helpers.benchee_config(markdown_file: "bench/results/payload_#{safe_filename}.md")
+      )
+
+      IO.puts("\n" <> String.duplicate("-", 80) <> "\n")
+    end
+
+    IO.puts("""
+
+    ================================================================================
+    Payload size benchmark complete!
+
+    Results saved to bench/results/payload_*.md
+
+    Key insights:
+      • Direct NIF is constant time (just returns the term)
+      • CLI Port overhead scales with payload (data through stdio + process spawn)
+      • SafeNIF overhead is dominated by node startup, payload is minor factor
+    ================================================================================
+    """)
+
+  {:error, :not_distributed} ->
+    IO.puts("""
+    ❌ Node is not running in distributed mode.
+
+    Run benchmarks with:
+      elixir --sname bench -S mix run bench/payload_sizes.exs
+    """)
+
+    System.halt(1)
+
+  {:error, reason} ->
+    IO.puts("❌ Setup failed: #{inspect(reason)}")
+    System.halt(1)
+end