Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand benchmarks #136

Merged
merged 2 commits into from
May 1, 2024
Merged

Expand benchmarks #136

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a518f61
Expand benchmarks
giarc3 Apr 18, 2024
cab8505
Add notes to the benchmark about Logdriver
giarc3 Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@ This directory contains a benchmark suite for the Java version of the Tenant Sec
To build and run the benchmark, just execute the following commands in this directory:

To show single threaded latency of a roundtrip (encrypt/decrypt):

```
mvn clean install
java -Xms1024m -Xmx1024m -jar target/benchmarks.jar -f0 -wi 1
```

A variation that will show throughput using key leasing:

```
mvn clean install
TENANT_ID=tenant-gcp-l java -Xms1024m -Xmx1024m -jar target/benchmarks.jar -f 0 -wi 1 -bm thrpt -tu s
Expand All @@ -18,17 +20,13 @@ TENANT_ID=tenant-gcp-l java -Xms1024m -Xmx1024m -jar target/benchmarks.jar -f 0
You have to benchmark an actual version of the TSC, though this can be a `SNAPSHOT` version published locally.
Update the `pom.xml` to whatever version you'd like to test.


## Tenant Security Proxy

In order to run the benchmarks, the TSC needs to connect to a _Tenant Security Proxy (TSP)_.
This service is provided as a Docker container, so it is easy to run the proxy on any computer that has Docker
installed. IronCore Labs hosts the Docker container on a publicly accessible container registry, so you can pull
the image from there and run it locally.

The TSP has a companion Docker container, the _Tenant Security Logdriver_ (LD) that runs alongside it in your environment. It is also hosted on the same publicly accessible container registry.


In addition to the Docker containers, you need a configuration file that specifies how the TSP should communicate
with the IronCore Labs Configuration Broker and Data Control Platform, which work together to enable the end-to-end
encryption that keeps all of the tenant KMS configuration information secure. To simplify the process of running
Expand All @@ -40,9 +38,11 @@ have included this configuration in the repository as a convenience. Also note t
created in IronCore's staging infrastructure.

The following `docker-compose` command will get a TSP+LD running on your computer with the provided configuration:

giarc3 marked this conversation as resolved.
Show resolved Hide resolved
```
docker-compose -f docker-compose.yml up
```

This starts the TSP locally listening on port 32804. The benchmark expects to connect to the TSP at that address.

To connect with and use the TSP, you need to supply a couple more configuration values:
Expand Down Expand Up @@ -78,6 +78,6 @@ before running the benchmark.

Since TSC-java is a library that interacts with a back-end service (TSP), the benchmark results are not always straightforward to interpret. Most API calls in the TSC make a round-trip to the TSP, and the TSP also does some computation. If testing on a single machine, it is good to monitor the CPU/RAM usage of the TSP processes in addition to the Java benchmark process to make sure you aren't resource constrained.

In general, operation latency is a function of latency to the TSP + latency to the tenant's KMS (if key-leasing is disabled).
In general, operation latency is a function of latency to the TSP + latency to the tenant's KMS (if key-leasing is disabled).

The TSP's tenant logging mechanism has some tunable limits. By default, a TSP should be able to sustain 500 ops/sec/tenant, with the ability to burst higher for a limited time. The benchmark is using a single tenant, and (depending on your machine and benchmark config) can easily be executing a few thousand ops/sec. If you run a benchmark long enough you will overwhelm the TSP. In a real application, you would scale-out the TSP at this point. See [the TSP documentation](https://ironcorelabs.com/docs/saas-shield/tenant-security-proxy/overview/) for more details.
3 changes: 2 additions & 1 deletion benchmarks/demo-tsp.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ SERVICE_SIGNING_PRIVATE_KEY=9lAJ3a1Cb4xTL9jD/qdt53pBSG6m9ZNWWFs2WsMN2Mpp+huxRRRM
SERVICE_SEGMENT_ID=698
API_KEY=0WUaXesNgbTAuLwn
IRONCORE_ENV=stage
RUST_LOG=info
RUST_LOG=warn
TSP_ENABLE_LOGDRIVER_INTEGRATION=false
198 changes: 81 additions & 117 deletions benchmarks/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,124 +1,88 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.ironcorelabs</groupId>
<artifactId>benchmarks</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<groupId>com.ironcorelabs</groupId>
<artifactId>benchmarks</artifactId>
<version>1.0</version>
<packaging>jar</packaging>

<name>Tenant Security Proxy Benchmarks</name>
<name>Tenant Security Proxy Benchmarks</name>

<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.ironcorelabs</groupId>
<artifactId>tenant-security-java</artifactId>
<version>4.0.0</version>
</dependency>
</dependencies>
<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.ironcorelabs</groupId>
<artifactId>tenant-security-java</artifactId>
<version>7.2.0</version>
</dependency>
</dependencies>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<jmh.version>1.23</jmh.version>
<javac.target>1.8</javac.target>
<uberjar.name>benchmarks</uberjar.name>
</properties>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<jmh.version>1.37</jmh.version>
<javac.target>17</javac.target>
<uberjar.name>benchmarks</uberjar.name>
</properties>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<compilerVersion>${javac.target}</compilerVersion>
<source>${javac.target}</source>
<target>${javac.target}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${uberjar.name}</finalName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>2.5</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.1</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.6</version>
</plugin>
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.3</version>
</plugin>
<plugin>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
</plugin>
</plugins>
</pluginManagement>
</build>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.13.0</version>
<configuration>
<compilerVersion>${javac.target}</compilerVersion>
<source>${javac.target}</source>
<target>${javac.target}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.5.2</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${uberjar.name}</finalName>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

</project>
Loading
Loading