Free weekly email: the week's new posts, videos and one tip from the books, every Sunday. Get it free →
Homelab the Right Way

Home › Posts

Trap: Using Debug Builds for Performance Benchmarks

You might measure performance using debug builds, but these contain overhead that skews results. Learn why you must use profileable release builds and the exact Gradle configuration to get accurate data.

· 4 min read · androidbenchmarkingprofilinggradleperformancemacrobenchmark

Trap: Using Debug Builds for Performance Benchmarks

Most developers pick the tool they know first rather than the question they need answered. This leads to a specific, expensive trap: measuring performance using debug builds. Debug builds contain instrumentation and overhead that distort timing data. When you run benchmarks on these builds, you are not measuring how your app performs for users; you are measuring how fast it runs with extra diagnostic baggage attached. The result is misleading data that looks good in testing but fails in production.

The chapter "The Tools" in the book Fast by Default provides a map of which tool answers which question. It explicitly states that modern Android Studio versions can profile either a debuggable build with complete data or a profileable release build with low overhead. The rule is strict: use the low-overhead mode for anything about time; use complete data only when you need detail a profileable build cannot give, such as a heap dump with full allocation tracking. If you are measuring startup time, frame durations, or scrolling performance, you must use a release-like build that includes the `` tag.

The trap: benchmarking debug builds

The easiest way to start a benchmark is to run a test against your current build configuration. However, if you do not explicitly configure a separate benchmark build type, Android Studio may default to running tests against a debuggable variant. Debug builds include extensive logging and instrumentation that slow down execution. This overhead masks the true performance characteristics of your code. You might see acceptable frame times in development but encounter jank in production because the release build lacks the safety nets and includes optimizations that change timing behavior.

The book warns that Layout Inspector shows recomposition counts for debuggable builds, so it is a diagnostic tool, not a measuring tool. You should use it to find what recomposes, then confirm the effect on frame times with Macrobenchmark on a release build. This distinction applies to all timing measurements. If you want to know how long startup takes or if scrolling drops frames, you must use Macrobenchmark against a release-like build.

The fix: configure the benchmark module correctly

To avoid this trap, you need to set up a separate com.android.test module that targets your app module. This module handles the installation, control, and execution of benchmarks against a real release-like build. The book provides the exact Gradle configuration required for this setup. You must define the plugin IDs, namespace, and dependencies in your macrobenchmark module's build.gradle.kts file.

// macrobenchmark/build.gradle.kts
plugins {
    id("com.android.test")
    id("org.jetbrains.kotlin.android")
}
android {
    namespace = "com.example.fieldnotes.macrobenchmark"
    compileSdk = 36

    defaultConfig {
        minSdk = 28
        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
    }
    targetProjectPath = ":app"
    experimentalProperties["android.experimental.self-instrumenting"] = true
    buildTypes {
        create("benchmark") {
            isDebuggable = true
            signingConfig = signingConfigs.getByName("debug")
            matchingFallbacks += listOf("release")
        }
    }
}
dependencies {
    implementation("androidx.test.ext:junit:1.3.0")
    implementation("androidx.benchmark:benchmark-macro-junit4:1.5.0")
}

Notice that the benchmark module is debuggable, not the app. This allows the test runner to instrument the benchmark code while keeping the target app close to a real release state. The book also notes that you should take whatever the module template generates for your Android Studio version rather than copying test library versions from a book, as these move independently.

The fix: configure the app build type

The next step is to ensure the target app module includes a matching benchmark build type. This build type must be a copy of the release build but signed with the debug key. This allows it to install on your test device without requiring your upload key, while still retaining the optimizations and lack of debug instrumentation found in the release variant.

Add the following configuration to your app module's build.gradle.kts:

// app/build.gradle.kts
android {
    buildTypes {
        release {
            isMinifyEnabled = true
            isShrinkResources = true
            proguardFiles(
                getDefaultProguardFile("proguard-android-optimize.txt"),
                "proguard-rules.pro")
        }
        create("benchmark") {
            initWith(getByName("release"))
            signingConfig = signingConfigs.getByName("debug")
            matchingFallbacks += listOf("release")
        }
    }
}

This configuration ensures that when you run Macrobenchmark, it installs the app using the benchmark build type. This variant inherits isMinifyEnabled and isShrinkResources from release, meaning code is obfuscated and resources are shrunk. It also includes the `` tag, which Android Studio's Benchmark module template creates automatically. This tag tells the system to expose performance data without adding the overhead of full debug instrumentation.

The fix: use Macrobenchmark for user experience metrics

Once your build types are configured, you can use Jetpack Macrobenchmark to measure what users actually experience. Macrobenchmark drives interactions like app startup, scrolling, animations, and navigation from a separate test module. It installs the app, controls compilation, kills and restarts it, and runs your interaction many times. It reports metrics such as time to initial display and frame durations, plus a Perfetto trace of every iteration.

The book provides a table that maps questions to tools. For "How long does startup take, really?" and "Does scrolling drop frames?", the correct tool is Macrobenchmark combined with StartupTimingMetric or FrameTimingMetric. For "Where did the time go?", you should use a Perfetto trace generated by Macrobenchmark or the profiler. If you are asking "Is this one function slow?", you might consider Microbenchmark, but only after a trace has shown that a specific function matters. Most startup and jank problems are not about one hot function, so Macrobenchmark is the right first tool for release-like builds.

Checklist

  • Verify your macrobenchmark module uses the com.android.test plugin.
  • Ensure the app module has a benchmark build type that inherits from release.
  • Confirm the benchmark build type uses the debug signing config.
  • Check that the app includes the `` tag via the Android Studio template.
  • Use Macrobenchmark for startup, scrolling, and animation metrics on release-like builds.
  • Reserve Layout Inspector for finding recomposition issues in debug builds only.
  • Use Microbenchmark only after traces identify specific slow functions.
  • Take dependency versions from the module template rather than copying fixed versions.

More articles