Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ bin
project.properties
out

# Un-ignore the wrapper script shipped in the distribution
!shark/shark-cli/src/dist/bin/
!shark/shark-cli/src/dist/bin/shark-ai-investigate

# Finder
.DS_Store

Expand All @@ -42,4 +46,7 @@ site
docs/api

# Python virtual env
venv
venv

# Claude Code
.claude
1 change: 1 addition & 0 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ buildscript {
dependencies {
classpath(libs.gradlePlugin.android)
classpath(libs.gradlePlugin.kotlin)
classpath(libs.gradlePlugin.kotlinSerialization)
classpath(libs.gradlePlugin.dokka)
classpath(libs.gradlePlugin.mavenPublish)
classpath(libs.gradlePlugin.detekt)
Expand Down
10 changes: 10 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@

Please thank our [contributors](https://github.com/square/leakcanary/graphs/contributors) 🙏 🙏 🙏.

## Unreleased

`shark-cli` gains a new **AI-driven leak investigation skill**: run `shark-ai-investigate --hprof <file>` to load a heap dump and start a session. A shell wrapper prints a session shortcode, then starts the `ai-investigate` daemon in the background. An AI agent (or a human) can then send commands to the daemon via `ai-investigate-cmd <shortcode> <command>` — all over named pipes. Commands include `trace` (leak traces as structured JSON), `fields` / `string` / `array` (inspect objects), `mark-leaking` / `mark-not-leaking` (supply context), `retained-size` (memory retained by an object), and `human-readable-trace` (plain-text summary). The skill embeds a full algorithm guide in its `--help` output so an AI agent can drive the investigation autonomously.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this design, but I think the name ai-investigate is misleading, even if it is how this tool is likely to be used. I'd suggest shark-daemon start --hprof <file> and shark-daemon stop --shortcode ...


New in `shark`:

* **`SingleObjectRetainedSizeCalculator`**: computes the retained size of a single heap object (the total memory freed if it were GC'd) using the provably-correct exclude-and-reach two-BFS algorithm.
* **`HeapClass.readInstanceFieldCount()`**: reads the declared instance field count as a single unsigned short from the class fields buffer, without allocating field records.
* **`ShallowSizeCalculator` accuracy improvements**: array objects now include the 12–16 byte ART object header that HPROF omits from array records; class object size now uses static field value sizes plus `ArtField` metadata (16 B × field count in `LinearAlloc`) instead of the raw HPROF record size.

## Version 3.0 Alpha 8 (2024-06-04)

* Added support for proper hprof handling on heap growth detection failures. Also inlined some public collaborators to achieve that. We now have a single class that's a bit larger but also a lot more obvious.
Expand Down
2 changes: 2 additions & 0 deletions gradle/libs.versions.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ androidCompileSdk = "34"
[libraries]
gradlePlugin-android = { module = "com.android.tools.build:gradle", version = "8.0.0" }
gradlePlugin-kotlin = { module = "org.jetbrains.kotlin:kotlin-gradle-plugin", version.ref = "kotlin" }
gradlePlugin-kotlinSerialization = { module = "org.jetbrains.kotlin:kotlin-serialization", version.ref = "kotlin" }
gradlePlugin-dokka = { module = "org.jetbrains.dokka:dokka-gradle-plugin", version = "1.8.10" }
gradlePlugin-binaryCompatibility = { module = "org.jetbrains.kotlinx:binary-compatibility-validator", version = "0.13.1" }
gradlePlugin-mavenPublish = { module = "com.vanniktech:gradle-maven-publish-plugin", version = "0.25.2" }
Expand All @@ -36,6 +37,7 @@ gradlePlugin-keeper = { module = "com.slack.keeper:keeper", version = "0.7.0" }
gradlePlugin-sqldelight = { module = "app.cash.sqldelight:gradle-plugin", version = "2.0.0-alpha05" }

coroutines-core = { module = "org.jetbrains.kotlinx:kotlinx-coroutines-core", version.ref = "coroutines" }
kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serialization-json", version = "1.5.1" }

kotlin-stdlib = { module = "org.jetbrains.kotlin:kotlin-stdlib", version.ref = "kotlin" }
kotlin-reflect = { module = "org.jetbrains.kotlin:kotlin-reflect", version.ref = "kotlin" }
Expand Down
2 changes: 2 additions & 0 deletions shark-ai-investigate.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
./gradlew --quiet --no-configuration-cache :shark:shark-cli:installDist
./shark/shark-cli/build/install/shark-cli/bin/shark-ai-investigate "$@"
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ class HprofRetainedHeapPerfTest {

val retained = analysisRetained - baselineHeap.retainedHeap(ANALYSIS_THREAD).first

assertThat(retained).isEqualTo(4.5 MB +-5 % margin)
assertThat(retained).isEqualTo(4.88 MB +-5 % margin)
}

@Test fun `freeze retained memory when indexing leak_asynctask_m`() {
Expand All @@ -70,7 +70,7 @@ class HprofRetainedHeapPerfTest {

val retained = analysisRetained - baselineHeap.retainedHeap(ANALYSIS_THREAD).first

assertThat(retained).isEqualTo(4.4 MB +-5 % margin)
assertThat(retained).isEqualTo(4.74 MB +-5 % margin)
}

@Test fun `freeze retained memory through analysis steps of leak_asynctask_o`() {
Expand Down Expand Up @@ -110,14 +110,14 @@ class HprofRetainedHeapPerfTest {
retainedPair.first - retainedBeforeAnalysis to retainedPair.second
}

assertThat(retained after PARSING_HEAP_DUMP).isEqualTo(5.01 MB +-5 % margin)
assertThat(retained after EXTRACTING_METADATA).isEqualTo(5.06 MB +-5 % margin)
assertThat(retained after FINDING_RETAINED_OBJECTS).isEqualTo(5.16 MB +-5 % margin)
assertThat(retained after FINDING_PATHS_TO_RETAINED_OBJECTS).isEqualTo(6.56 MB +-5 % margin)
assertThat(retained after FINDING_DOMINATORS).isEqualTo(6.56 MB +-5 % margin)
assertThat(retained after INSPECTING_OBJECTS).isEqualTo(6.57 MB +-5 % margin)
assertThat(retained after COMPUTING_NATIVE_RETAINED_SIZE).isEqualTo(6.57 MB +-5 % margin)
assertThat(retained after COMPUTING_RETAINED_SIZE).isEqualTo(5.49 MB +-5 % margin)
assertThat(retained after PARSING_HEAP_DUMP).isEqualTo(5.38 MB +-5 % margin)
assertThat(retained after EXTRACTING_METADATA).isEqualTo(5.63 MB +-5 % margin)
assertThat(retained after FINDING_RETAINED_OBJECTS).isEqualTo(5.68 MB +-5 % margin)
assertThat(retained after FINDING_PATHS_TO_RETAINED_OBJECTS).isEqualTo(6.96 MB +-5 % margin)
assertThat(retained after FINDING_DOMINATORS).isEqualTo(6.96 MB +-5 % margin)
assertThat(retained after INSPECTING_OBJECTS).isEqualTo(6.96 MB +-5 % margin)
assertThat(retained after COMPUTING_NATIVE_RETAINED_SIZE).isEqualTo(6.96 MB +-5 % margin)
assertThat(retained after COMPUTING_RETAINED_SIZE).isEqualTo(5.92 MB +-5 % margin)
}

private fun indexRecordsOf(hprofFile: File): HprofIndex {
Expand Down
8 changes: 4 additions & 4 deletions shark/shark-android/src/test/java/shark/LegacyHprofTest.kt
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ class LegacyHprofTest {
"LeakCanary version" to "Unknown"
)
)
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(193431)
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(195007)
}

@Test fun androidM() {
Expand All @@ -41,14 +41,14 @@ class LegacyHprofTest {
val leak = analysis.applicationLeaks[0].leakTraces.first()
assertThat(leak.leakingObject.className).isEqualTo("com.example.leakcanary.MainActivity")
assertThat(leak.gcRootType).isEqualTo(GcRootType.STICKY_CLASS)
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(49584)
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(52824)
}

@Test fun gcRootReferencesUnknownObject() {
val analysis = analyzeHprof("gcroot_unknown_object.hprof")

assertThat(analysis.applicationLeaks).hasSize(2)
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(5306218)
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(5339418)
}

@Test fun androidMStripped() {
Expand Down Expand Up @@ -77,7 +77,7 @@ class LegacyHprofTest {
assertThat(analysis.applicationLeaks).hasSize(1)
val leak = analysis.applicationLeaks[0].leakTraces.first()
assertThat(leak.leakingObject.className).isEqualTo("com.example.leakcanary.MainActivity")
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(211038)
assertThat(analysis.allLeaks.sumBy { it.totalRetainedHeapByteSize!! }).isEqualTo(214410)
}

private enum class WRAPS_ACTIVITY {
Expand Down
7 changes: 7 additions & 0 deletions shark/shark-cli/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ plugins {
id("com.vanniktech.maven.publish")
}

apply(plugin = "kotlinx-serialization")

java {
sourceCompatibility = JavaVersion.VERSION_1_8
targetCompatibility = JavaVersion.VERSION_1_8
Expand All @@ -21,10 +23,15 @@ tasks.withType<KotlinCompile> {
dependencies {
api(projects.shark.sharkAndroid)

implementation(libs.kotlinx.serialization.json)
implementation(libs.clikt)
implementation(libs.neo4j)
implementation(libs.jline)
implementation(libs.kotlin.stdlib)

testImplementation(libs.junit)
testImplementation(libs.assertjCore)
testImplementation(projects.shark.sharkHprofTest)
}

application {
Expand Down
95 changes: 95 additions & 0 deletions shark/shark-cli/src/dist/bin/shark-ai-investigate
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
#!/bin/sh
#
# Wrapper for 'shark-cli ... ai-investigate'.
#
# 1. Generates a unique session shortcode.
# 2. Optionally prints the investigation instructions (--no-help skips this).
# 3. Starts the ai-investigate daemon in the background with that shortcode.
# 4. Waits until the daemon has created its named pipes (happens after heap
# analysis, so may take a minute on large dumps).
# 5. Prints SHORT_CODE=<shortcode> — the AI agent uses this in every command.
#
# Usage:
# shark-ai-investigate --hprof <file>
# shark-ai-investigate --hprof <file> --obfuscation-mapping <map>
# shark-ai-investigate --hprof <file> --no-help # skip instructions
#

# Resolve the directory that contains this script, following symlinks.
app_path=$0
while [ -h "$app_path" ]; do
ls=$(ls -ld "$app_path")
link=${ls#*' -> '}
case $link in
/*) app_path=$link ;;
*) app_path=$(dirname "$app_path")/$link ;;
esac
done
SCRIPT_DIR=$(cd "$(dirname "$app_path")" && pwd -P)
SHARK_CLI="$SCRIPT_DIR/shark-cli"

# If the first argument is "cmd", forward the rest to ai-investigate-cmd and exit.
# shark-ai-investigate cmd <shortcode> <command...>
if [ "$1" = "cmd" ]; then
shift
exec "$SHARK_CLI" ai-investigate-cmd "$@"
fi

# Generate a unique session shortcode.
shortcode=$(uuidgen | tr -d '-' | tr '[:upper:]' '[:lower:]' | cut -c1-8)

# Check whether --no-help was passed; if so skip the instructions step.
no_help=false
for arg in "$@"; do
case "$arg" in
--no-help) no_help=true ;;
esac
done

# --no-help is consumed by this wrapper and must not be forwarded to shark-cli.
# Rebuild $@ without it using a POSIX-safe eval loop.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really need POSIX compat? Ancient bash (like the version that ships with Mac) supports arrays, so you can write much simpler code:

new_args=()
for arg in "$@"; do
  [[ $arg == --no-help ]] || new_args+=("$arg")
done
set -- "${new_args[@]}"

orig_count=$#
i=1
while [ $i -le $orig_count ]; do
eval "arg=\$$i"
case "$arg" in
--no-help) ;;
*) set -- "$@" "$arg" ;;
esac
i=$((i + 1))
done
shift "$orig_count"
# $@ now contains only the shark-cli global options (--hprof, --device, etc.)

if ! "$no_help"; then
# Print instructions (blocking — agent reads them before the session opens).
"$SHARK_CLI" "$@" ai-investigate --instructions-only
fi

# Start the daemon in the background with the session shortcode.
# Redirect stdout/stderr to /dev/null so the daemon does not inherit this
# script's stdout pipe. Without this, any process that waits for stdout EOF
# (e.g. a task runner) would block until the daemon exits, because the daemon
# holds the write end of the pipe open. The daemon communicates via named
# pipes, not stdout, so nothing useful is lost.
"$SHARK_CLI" "$@" ai-investigate --session "$shortcode" >/dev/null 2>&1 &
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the daemon will still be tied to the session - if you close the window within which you ran this, the daemon will die too. You can fix this with nohup if desired.


# Wait until the daemon creates its named pipes.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any code to capture the pid of the daemon - how to shut it down after?

# Pipes are created after heap analysis completes, so this may take a while.
printf 'Running heap analysis and creating pipes\n'
fifo="/tmp/shark-$shortcode.in"
i=0
while [ ! -p "$fifo" ] && [ "$i" -lt 600 ]; do
sleep 0.5
i=$((i + 1))
done
if [ ! -p "$fifo" ]; then
printf '{"error":"Daemon did not start within 5 minutes."}\n'
exit 1
fi

# Print a separator, then the shortcode.
# The agent passes the shortcode as the first argument to every command:
# shark-ai-investigate cmd <shortcode> <command>
printf '\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n'
printf 'SHORT_CODE=%s\n' "$shortcode"
96 changes: 96 additions & 0 deletions shark/shark-cli/src/main/java/shark/AiInvestigateCmdCommand.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
package shark

import com.github.ajalt.clikt.core.CliktCommand
import com.github.ajalt.clikt.core.context
import com.github.ajalt.clikt.parameters.arguments.argument
import com.github.ajalt.clikt.parameters.arguments.multiple
import java.io.File
import java.io.FileOutputStream
import java.util.concurrent.CompletableFuture
import java.util.concurrent.TimeUnit
import java.util.concurrent.TimeoutException
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.jsonObject
import kotlinx.serialization.json.jsonPrimitive
import shark.SharkCliCommand.Companion.echo

/**
* Thin client that sends a single command to a running [AiInvestigateCommand] daemon and prints
* the JSON response. The daemon and this command communicate via two named pipes identified by
* the session shortcode printed by shark-ai-investigate.
*
* Usage (start the session first with shark-ai-investigate, then send commands):
* shark-ai-investigate --hprof <file> # prints SHORT_CODE=<shortcode>
* shark-ai-investigate cmd <shortcode> trace
* shark-ai-investigate cmd <shortcode> fields 3
* shark-ai-investigate cmd <shortcode> mark-leaking 3 mDestroyed is true
* shark-ai-investigate cmd <shortcode> close-session
*/
class AiInvestigateCmdCommand : CliktCommand(
name = "ai-investigate-cmd",
help = "Send a command to a running ai-investigate daemon. First argument is the session shortcode."
) {

init {
context {
// Disable @-file expansion so `fields @<objectId>` isn't treated as "load args from file".
expandArgumentFiles = false
// Stop option-parsing after the first positional arg (session shortcode).
// Clikt 2.3.0 treats ANY token containing '=' as a long-option attempt, so a reason like
// "size=1" would otherwise trigger "no such option: ..." before even reaching the daemon.
allowInterspersedArgs = false
}
}

private val session by argument(help = "Session shortcode printed by shark-ai-investigate (SHORT_CODE=...)")

private val commandParts by argument(help = "Command and arguments to send to the daemon")
.multiple(required = true)

override fun run() {
val inPath = "/tmp/shark-$session.in"
val outPath = "/tmp/shark-$session.out"

if (!File(inPath).exists() || !File(outPath).exists()) {
echo("""{"error":"No active session '$session'. Run: shark-ai-investigate --hprof <file>"}""")
return
}

val cmd = commandParts.joinToString(" ")

// Opening a FIFO blocks until both ends are open. Start both opens concurrently so
// the client is never waiting sequentially while the daemon waits on the other FIFO.
// supplyAsync uses the common fork-join pool (daemon threads), matching the daemon side.
val writerFuture = CompletableFuture.supplyAsync { FileOutputStream(inPath).bufferedWriter() }
val readerFuture = CompletableFuture.supplyAsync { File(outPath).bufferedReader() }

try {
CompletableFuture.allOf(writerFuture, readerFuture).get(10_000L, TimeUnit.MILLISECONDS)
} catch (e: TimeoutException) {
echo("""{"error":"Session '$session' is not responding. Daemon may have exited."}""")
return
}

val writer = writerFuture.get()
val reader = readerFuture.get()

writer.write(cmd + "\n")
writer.flush()
writer.close()

val response = reader.readLine()
reader.close()

if (response != null) {
if (commandParts.firstOrNull() == "human-readable-trace") {
// Daemon sends a single JSON line {"trace":"..."}; extract and print plain text.
val trace = runCatching {
Json.parseToJsonElement(response).jsonObject["trace"]?.jsonPrimitive?.content
}.getOrNull()
echo(trace ?: response)
} else {
echo(response)
}
}
}
}
Loading
Loading