So a collegue comes over to me one day to ask what changes we did since the last release, reason being: “The app is very slow on Pixel 7 running on <cloud-based emulator service>“.
As I, too, am addicted to being useful, I get curious and sympathetic to their call for help. I suggest we should try a debug build instead to get a profile trace. But instead, we get… a crash? Cannot locate symbol _ZTVNSt6__ndk118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE in libc++_shared.so.
After a while of looking through the logs and trying bumping different versions without having any effect on the issue we read something at the start of the log that really has been staring us in the face this whole time… the emulator is running on x86_64. Suddenly a lot of things start to click. The emulated “Pixel 7” device isn’t really anything like the actual physical device. Now we have some actual technical reason that could begin to give an explanation to the symptoms of the bug.
Looking into the actual APK file, we can see that the only provided .so files are for ARM-v7 and ARM64. This is because we recently worked on optimizing the APK file size and removed support for architectures other than ARM. So I figured all problems would be solved by just re-enabling x86, but no luck - the crash was still there even though we now have included specific libs for x86_64. Maybe there was some difference between them? But regardless from all this: how did the app even launch at all if the native files for the architecture was missing?
Together with Claude we made a shell script that inspects the symbol table inside libc++_shared.so artifacts and searches for the specific symbol name that was missing. All intermediary files matche, but some of our vendors provide prebuilt .so files inside of .aar (Android library archives) files - let’s check those as well… And bingo, we finally found where the outdated libc++_shared.so file comes from!
During the APK assembly step, there can only be one of each native artifacts, and for reasons unbeknownst to man, gradle consistently picks the prebuilt one that is outdated instead of using any of the freshly built ones. I didn’t have any luck configuring gradle to skip this one or to prefer any other artifact, and another teammate informed me that our vendor fixed this in a later release by letting us providing libc++_shared.so and not embedding it within the archive.
So finally we are back to debugging the original issue, the slowness. But when we start the app again, not only is there no crash at startup — the app is fast! …what?
Somewhere in this debugging rabbit hole we actually solved the original bug? This is where we go though and cherry-pick all the changes one by one to a new branch, and it turns out that the relevant change was… enabling x86_64 support?
Claude helped me to connect the dots, I gave it all the symptoms and details of the debugging session this was the short answer:
Android emulators use ARM-to-x86 translation technology (sometimes referred to as libhoudini in the community) that allows ARM apps to run on x86 hardware, though with significant performance overhead
Finally we had a solution to both and which also answered how the app even worked on x86 to begin with: The app was emulated through an additional translation layer. I also found this blogpost by Google that mentions this being a thing.
All in all, sometimes it really pays off to dive deep into issues that only happens on an esoteric configuration of a Pixel 7 device. I learned a lot on this journey where what seemed to be a bug was actually a feature.