Use Variables for Versions of Dependencies on Multi Module Android Projects, Please.

Motive

Recently I was tasked, by myself, to update all the dependencies of project containing submodules in counts of tens, to the newest version.

This was required in order to add a new dependency that itself depended on a newer version of some of the existing dependencies, think, appcompat-v7 for example.

The pain

Now that I had lousily when creating new modules on the project just let every single project get its default version of appcompat-v7 and when shared dependencies arised, just copied from existing gradle files, there was literally hundreds of dependencies that would have needed updated.

Solution

First I thought, this should be simple, right? Use the Android Studios replace all within project feature to perform the update on regex, for appcompat-v7 I would have target regex of

["']com\.android\.support:appcompat-v7:.*["']

no biggie, right? Okay, to replace with regex, the replacement must be regex, or some similar format too such as

"com\.android\.support:appcompat-v7:\$appcompatVersion"

next step would be to add this variable $appcompatVersion to the ext section of projects build.gradle, easypeasy!

Now imagine doing this for all dependencies! I for one have at least five different dependencies that are depended upon by at least ten modules in the project, such hassle!

Automation to the rescue

I don’t remember who was the wise man of internet who once said that you could automate something, even if it took longer time, because it would consume less mental energy, it would still be a win-win situation, but that’s what I have been living upon, and it has been much happier life.

I botched up a quick Java application to automate the tasks, it could and maybe should have been a python script, as its pretty simple but man, where does zebra get from their stripes?!

The application has a simple job!

  1. Take a path from comamand line arguments
  2. For that path go through each sub-folder that contains file build.gradle
    1. For this build.gradle file go through each dependency and keep track of each depency by their group, name and version
    2. Make sure versions don’t differ across project, if they do, throw exception
  3. Now we’ve got list of all dependencies and their versions!
  4. Go through all collected dependencies and create a variable name by using the group and name of the dependency, also keep track of the value of this variable, so it can be saved later on
  5. Go again through each sub-folder that contains file build.gradle
    1. This time replace all dependencies version with variable we created before
    2. backup this build.gradle and replace it with the newly created
  6. Add variable to projects build.gradle, copy and replace it
  7. Cross fingers that everything went well!

For me, this ended up creating 39 badly named variable in my projects build.gradle and much boring work was saved.

        comAndroidSupportRecyclerviewVVersion = '27.0.2'
        comAnjlabAndroidIabVLibraryVersion = '1.0.44'
        ioRequeryRequeryProcessorVersion = '1.5.0'
        comGoogleCodeGsonGsonVersion = '2.8.2'
        comFlaviofariaKenburnsviewVersion = '1.0.7'

Hmm, could this naming scheme be improved?

Success!

Now every dependencys version is defined in the project, awesome! This could and maybe should have been done by beginning of the time, but, oh well, humans ain’t perfect!

Best of luck to my fellow Android Devs and if you’re interested in this little piece of code, here’s the GitHub link!(be aware, Microsoft inside)

How I automated taking a screenshot of every single Pokémon in Pokémon GO

Hello guys, hows it going, huh?

Today I am going to share you an extremely wild story about how I automated taking a screenshot of every single Pokémon in game!

The journey starts with a problem of not having every Pokémon in game. I mean.. Who does?

After all.. I am not going to ‘catch ’em all’, now am I? No! I am a programmer, not a Pokémon trainer..

First step was of course to get renders of all Pokémons.. Shouldn’t be too hard, right? Just manipulate, or more precisely.. Man in the middle the connection between my phone and the server, fake that the server sent every Pokémon in game and.. there we go, right?

After quick research I found this implementation in GitHub called Pokemon Go MITM Proxy by a nickname rastapasta, so I tried to setup it in Windows environment.. No luck.. obviously.. Its also documented in the repo that its hard to compile..

After recovering from the disappointment I cloned the repo on my Linux server and ran it there, installation went smoothly and I was ready to go!

Unfortunately there’s still need for root or removal of certificate pinning, but gladly from my last post I had a patch ready to go! So no worries! I installed the certificate as documented from the server and ran an example script that would replace all my pokémons, and I was good to go!

pokemon-list

When I ran Pokémon Go after this, I ended up getting a list like  this, however it wasn’t enough.. The example would only replace the Pokémons I had with others, but what I really needed is it to clear my Pokémons and add all Pokémons in increasing ids..

So its time to edit the script.. So I opened example.replacePokemons.coffee but coffeescript wasn’t too familiar with me.. Nor was the structure of packets, so first of all I had to lookup coffeescripts syntax before anything. It wasn’t too bad knowing Javascript, but required some effort to get through, but I found also this js2coffee compiler online I could use when stuck.

So the smallest modification I could get through with is remove all the code touching the pokemons, and craft a piece of code that replaces whole inventory with inventory cotaining an item with PlayerStats and all available Pokémons.

As funny as it sounds, PlayerStast too are passed as an InventoryItem.. Smart..

The minimal implementation I was left with is as follows:

PokemonGoMITM = require './lib/pokemon-go-mitm'
fs = require 'fs'

server = new PokemonGoMITM port: 8081
	.addResponseHandler "GetInventory", (data) ->
		if data.inventory_delta
			data.inventory_delta.inventory_items = [
				{
					inventory_item_data: {
						player_stats: {
							level: 99,
							experience: 999999,
							prev_level_xp: 1,
							next_level_xp: 1,
							km_walked: 0,
							pokemons_encountered: 2,
							unique_pokedex_entries: 1,
							pokemons_captured: 2,
							poke_stop_visits: 0,
							pokeballs_throws: 0,
							pokemon_caught_by_type: Int32Array.from([]),
							small_rattata_caught: 0
						}
					}
				}
			];

			for i in [1..151]
				data.inventory_delta.inventory_items.push({
						inventory_item_data: {
							pokemon_data: {
								id: i,
								pokemon_id: i,
								cp: 10,
								stamina: 10,
								stamina_max: 10,
								move_1: "QUICK_ATTACK_FAST",
								move_2: "AIR_CUTTER",
								height_m: 0.29903167486190796,
								weight_kg: 1.878556489944458,
								individual_attack: 1,
								individual_defense: 14,
								individual_stamina: 15,
								cp_multiplier: 0.09399999678134918,
								pokeball: "ITEM_POKE_BALL",
								captured_cell_id: "5082605749092220928",
								creation_time_ms: "1468915294161"
							}
						}
					});
		data

I am not even sure if all of this is required.. But its pretty straightforward.. Replace inventory_items with array containing just PlayerStats, then add 151 Pokemons with ids 1-151 and be fone with it!

The end result is.. well the same as the previous as I was too lazy to run the initial version of replacePokemons to get the real initial result sooo.. Now we’re left with just taking a screenshot of every Pokémon.. Should be easy.. right!

For our good you can scroll the list of Pokémons by swiping so capturing screenshot should be easy as taking screenshot with Pokémon id as name, then swipe predefined amount, sleep 10 seconds to let the model load, and repeat!

A little Windows shell script I made is as follows:

FOR /L %%G IN (1,1,151) DO (
	adb screencap -p /sdcard/%%G.png
	adb pull /sdcard/%%G.png
	adb shell input touchscreen swipe 400 350 200 350
	ping 127.0.0.1 -n 11 > nul
)

pause

Its pretty simple!

  • Repeat from 1 to 151
    • Take a screenshot with naming it with looper number
    • Pull the screenshot from the sdcard to the working directory
    • Evaluate a swipe, from right to left, changing the Pokémon on the screen
    • Sleep for ten seconds, because this is not implemented in a simple sleep fashion, ping 11 times and get approximately a 10 second delay

After running this for.. let me calculate.. approx 30 minutes.. we’re done, and have a folder full of images!

pokimons

As you can see, we’re good, and as far as this glimpse tells me.. we did fine! Now its time to get on with these pictures.. but that’s a topic for another blog.. Thanks for reading!

Automating patching of Pokemon Go – and other applications..

Hello guys!

Last time I posted about patching Pokemon Go without touching the native side, posted it on reddit and gained some good upvotes in /r/pokemongodev, and initial thought in the post was, should this be automated, and given the popularity, I’d say hell yeah!

That’s what I have been up for previous couple of days, writing a fool proof command line based application that lets users patch APK’s with any set of changes, currently it only supports complete file replacements or additions, but that’s easily expandable if required.

The basic task the tool was geared towards is to use APK Tool, as runnable Jar to unpack the APK, then apply some kind of patches to the unpacked folder and repack, sign and align it.

Requirements of running the tool is to have Android SDK, APK Tool and a JDK Installation, which are pretty common, if you’re into this kind of stuff!

The setup is quite simple, when you run the released jar from command line, the program looks for ANDROID_HOME environment variable, JVM parameter and the configuration file, created after the first run to see if its been set, and if it hasn’t it asks user to input it.

ANDROID_HOME here is the root folder of Android SDK, which could automatically be detected to some extent, but as I have other things to do.. for now.. It just looks for the environment variable and command line flag.. You can see where Android SDK is located from Android studio, but at least in Windows environment its usually located in

C:\Users\PasiMatalamaki\AppData\Local\Android\Sdk

Where PasiMatalamaki is replaced with your epic us3rname.

Aftere setting up ANDROID_HOME, we’re not done yet.. The next thing to set is path to APK Tool.jar which can be downloaded from here. Just place it anywhere and copy the path to the jar and paste it in.. after that is the final step, before getting to the real deal! The most important part, which hopefully you’ve already set! Setting JAVA_HOME environment variable or just inputting it..

As written in the first Google result, If you didn’t change the path during installation, it will be something like this: C:\Program Files\Java\jdk1.8.0_65

And that is the final step before getting into the tool, after that you will be asked for build tools version you’d like to use, if you’ve got multiple.

All should work, but I am myself using the newest one, this is other thing, that could be modified to automatically pick the newest one, but man.. the effort!

Choose your buildToolsDir
0: 19.1.0
1: 21.1.2
2: 23.0.1
3: 23.0.3
4: 24.0.0
5: 24.0.1

So I input 5 and I’ll be asked for the apk to be used, or if the folder you’re working in, as in the folder you ran the jar in, contains none, you will be asked for a directory where the apk is, or just a full path to the apk.

Choose APK to be used
0: com.nianticlabs.pokemongo(version: 0.35.0, name: pokemon_original.apk)

For example here’s only one apk, which is the pokemon go, as used in this example, so we select 0.

After that the application will list available patches in the given directory, which by default is the /patches folder, which by default will look like this

Toggle selected patches, or press enter to proceed patching..
0: [ ] frida_gadget
Add Frida Gadget

Adds Frida Gadget to libs and loads it, easy!
1: [ ] remove_pinning
Remove Certificate Pinning

These are the two directories inside the /patches folder, which contains the following structure:

  • patch_name:
    • details.txt
    • replaced files…

as example there two patches in the release. The patching simply replaces the files inside the apk by the files inside the patch, simple and efficient!

You can toggle the patches by their version number, for example, here we want to apply remove pinning patch, we just input 1, and after that we can press enter to proceed with patching.

After that we’ll see a long output of information of whats happening, but after its all done, we’ll see output file

build_output_release.apk

which is result of debuilding, patching, rebuilding, signing and aligning, which is ready to be installed on an Android device! Simply run

adb install build_output_release.apk

And enjoy your Pokemon Go without certificate pinning!

Thanks for reading and for those interested, be sure to see the project in GitHub!

Removing Certificate Pinning from Pokemon Go Without Going Native

What is up?

Today we’re removing certificate pinning from Pokemon Go to man-in-the-middle the connection between my phone and the server, which is now required.

Main difference from similar posts, such as Eaton’s method where he removes it from the native code, is that, we’re not stepping to the dark side, and the main reason for that, for me, its much easier to patch the applications Dalvik code than the native ARM code, and I don’t feel too comfortable poking around in the native code, also I don’t have the tool required, Ida Pro with ARM..

Anyway! Lets get to the real deal, right!

First step, we need to take apart the APK, which is simple! Just use your favourite APK Extractor from Google Play, dump the APK on your machine and use APK Tools to unpack it.

java -jar apktool_2.1.1.jar d pokemon_original.apk -o pokemon

Where apktool_2.1.1.jar is your flavour of APK tool and pokemon_original.apk is the APK dumped from your device.

After that we have a folder containing all the files, good to modify!

Logically we make our way through niantic packages, com/nianticlabs/nia/network sounds good, right?

In this package we find a NianticTrustManager.smali, so its time to fire up your favorite text editor, you can use any, but please don’t use Notepad, its pain in the ass to read..

Given the name too, its clear that this Class implements X509TrustManager, which is as the name says, responsible for the trust management. So its time to lose the trust, right?

Because I’ve poked around with this before, I know how to overcome this slight problem, we’ll have just to return a null from getAcceptedIssuers() and do nothing in checkClientTrusted() and checkServerTrusted()

We take roughly our handle of the latter two methods and remove all the code, leaving behind this kind of structure.

# virtual methods
.method public checkClientTrusted([Ljava/security/cert/X509Certificate;Ljava/lang/String;)V
    .locals 2
    .param p1, "chain"    # [Ljava/security/cert/X509Certificate;
    .param p2, "authType"    # Ljava/lang/String;
    .annotation system Ldalvik/annotation/Throws;
        value = {
            Ljava/security/cert/CertificateException;
        }
    .end annotation

    .prologue
    return-void
.end method

.method public checkServerTrusted([Ljava/security/cert/X509Certificate;Ljava/lang/String;)V
    .locals 2
    .param p1, "chain"    # [Ljava/security/cert/X509Certificate;
    .param p2, "authType"    # Ljava/lang/String;
    .annotation system Ldalvik/annotation/Throws;
        value = {
            Ljava/security/cert/CertificateException;
        }
    .end annotation

    .prologue
    return-void
.end method

And in the getAcceptedIssuers() we have to actually write some code, meh.. Shouldn’t be too hard!

.method public getAcceptedIssuers()[Ljava/security/cert/X509Certificate;
    .locals 2

    .prologue
    .line 44
    const/4 v0, 0x0

    return-object v0
.end method

So we just assign v0 a null(0) and return it with no further ado

Okay, I think we are ready.. Lets cross our fingers and put together this goodness!

First we build the apk from the folder that contains all the files, meaning the pokemon folder!

java -jar apktool_2.1.1.jar b pokemon -o pokemon.apk

Then we need to get a keystore from somewhere, which will be used for signing, creating that is quite simple, just run the keytool!

keytool -genkey -v -keystore key -alias pokego -keyalg RSA -keysize 2048 -validity 10000

After that we’re left with a key that is used to sign the actual apk using jarsigner!

jarsigner -verbose -keystore key -signedjar "pokemon_ua.apk" "pokemon.apk" pokego -storepass SecretPokeGoPassXD

This creates a pokemon_ua.apk file. ua stands for Un-Aligned, if you’re into Android Dev, you might now that the apks needs to be aligned before shipping! Which is a matter of running a zipalign on them, which is also described in the manual signing process article.

zipalign -f 4 "pokemon_ua.apk" "pokemon_release.apk"

And finally we got the final ‘release’ build! Lets shove that into our device! Make sure you uninstall previous installations of the app!

adb install pokemon_release.apk

Run the app on the device, throw in your age, sign in aaand… In like a flynn!

Open up your favorite MITM proxy, such as Charles, which is what I am going to use, and configure your device to connect to the proxy instead, continue using the app and you will see what will make angels hum.

result

 

 

 

There we go, unencrypted version of the packets! However this is in a undecoded protocol buffer, which descriptor you can probably find online.. In Github or somewhere..!

Reversing Bytecode That Your Avarage Decompiler Can’t Handle By Hand

Hello guys, long time no crocodile, or how did it go, huh? Meaning, haven’t written here for too long.. Enough excuses!

Recently I was facing a problem, regarding taking a screenshot in Android programmatically, without falling back on unsupported API’s meaning, using the MediaProjection API’s introduced in API level 21, however, the actual problem I had, following their official examples or those of third party ones is a subject for another blog post!

Today we are covering, yet again, reverse engineering of a proven working application, this time, not for abuse of virtual benefits, but to copy a working functionality from a third party app, not in line by line manner, but in a overall concept, to avoid any copyright related takedowns.

scales-36417_640

IANAL but not that these kind of small companies have any power to pursue such or take down every app that uses similar technology to compare the code, but you can never be too sure or secure, or have too much moral to secure nightsleep.

Fun fact, the app I broke down seems to actually have a class leeched from rival application.

I wouldn’t have noticed, if they hadn’t kept the exact same package name as the rival application is released under.. Smart..

Initial steps I took to figure what I was doing wrong with the MediaProjection, to only receive the black screen, was ofcourse to find one application that does screencapture from service or such, that’s available in Google Play currently, and there’s plenty, not to name any, not that I am going to release a screenshotting application for now..

hacker-1446193_640

When I first found one I obviously put it through my favorite set of Android reverse engineering tools, which of easiest and quickest is bytecode viewer, big shootout for them, the UX is crap, but its for me, the easiest way to drop an APK in and receive both bytecode and human readable version of the code inside, and not even talking about the search tools, which are hard to use, but useful!

To my disappointment after finally finding the reference to the API I knew the app was using, due to the Android’s forced dialogue of giving app permission of capturing the screen, the decompiler were unable to reverse the bytecode due to the complicated branching or whatever.

After that I looked at the bytecode, that spanned across maybe 200 lines and ran it through in my head, couldn’t initially figure the difference between it and my rough implementation that I had partially scrapped from the API example, but for one, there was 500 ms sleep in added in the code, I tried adding it between same API calls, but no luck.. We had to go deeper!

Before digging deeper and spending all my time just to find out I am looking at wrong piece of code, as the application had what, 20 ways of capturing a screenshot, to support some legacy phones that I wasn’t looking to support, I copied baldly the .smali files, containing the dalvik code, to my own project, after ‘baksmaling’ it using apktool, and added a quick call to the function, which worked flawlessy and resulted in good looking screenshot, I was ready to proceed with the reverse engineering efforts!

I tried another decompiler or two, with no luck, hard to tell what it is, that breaks the decompiler, and I am not too much into those anyway, I started writing the code by hand, following the Dalvik code on my other screen.

If you’re not a robot or a machine in any other way, it will be pretty hard to go through so many lines of bytecode and getting the grasp of what’s happening without spending countless of hours on it.

brain-1295128_640

I found its much easier to implement the logic in other language or even in pseudo code, for the ease of my brains, which has the most working paths of understanding Java, I did it in java, which is good, as it is the final language I am going the feature to implement in, even tough, implementing all the branches which are implemented in dalvik using labels, gotos and conditionals might get pretty tedious, after having it all written down, it will be easy to refactor it for real world use!

Lets look at an example!

const-string v1, "screenshot"

new-instance v2, Ljava/lang/StringBuilder;

invoke-direct {v2}, Ljava/lang/StringBuilder;->()V

invoke-virtual {v2, p2}, Ljava/lang/StringBuilder;->append(Ljava/lang/String;)Ljava/lang/StringBuilder;

move-result-object v2

const-string v3, " "

invoke-virtual {v2, v3}, Ljava/lang/StringBuilder;->append(Ljava/lang/String;)Ljava/lang/StringBuilder;

move-result-object v2

invoke-virtual {v2, p3}, Ljava/lang/StringBuilder;->append(Ljava/lang/String;)Ljava/lang/StringBuilder;

move-result-object v2

const-string v3, " "

invoke-virtual {v2, v3}, Ljava/lang/StringBuilder;->append(Ljava/lang/String;)Ljava/lang/StringBuilder;

move-result-object v2

invoke-virtual {v2, p4}, Ljava/lang/StringBuilder;->append(Ljava/lang/String;)Ljava/lang/StringBuilder;

move-result-object v2

invoke-virtual {v2}, Ljava/lang/StringBuilder;->toString()Ljava/lang/String;

move-result-object v2

invoke-static {v1, v2}, Landroid/util/Log;->d(Ljava/lang/String;Ljava/lang/String;)I

This isn’t too complicated, but if you have to stitch tens of this size patches of code in your mind, it gets pretty hard..

A quick rundown of above code in pseudo code for comparison

takeScreenShot(a, b, c, d)
 log("screenshot", b + " " + c + " " + d)

This is not(IIRC) not part of the applications code, but part of the code I used to find out which parameters the reverse engineered app passes to the function

As you can see, a small piece of java goodness turns into crazy amount of bytecode, and without even including the conditionals with all the labels, don’t even let me get into that part!

applause-297115_640
Round of applause for the findings

TL/DR
When reverse engineering bytecode your decompiler or other tool can’t handle, and its too overwhelming, just write it down in pseudo code or in your favorite language to give your brains a break!

Losing dreams for SMS_RECEIVED broadcast receiver

Hello guys,

It’s been a long time, but I am back with yet another Android packed problem I was recently solving!

I was implementing an SMS gateway for Android, which sound like a quite solved task, but apparently is not!

I tried the first Google Play results but couldn’t get any of the listed apps work.

Some of the apps required setting them to default sms app which made messenger not work, and for untold reasons I had to run this on my primary phone!

There’s an open market, go ahead and make one of your own, that actually works on Android 6.0!

So the problem sounds simple, I’ll create a BroadcastReceiver which listens for a action SMS_RECEIVED, which as the name says, listens for sms receiving broadcasts.

It seems like an easy task to hook into the sms system, unfortunately there have been couple of bigger changes after the original API has been updated(or at least all the documentation for it?) and that code, which blindly listens for SMS_RECEIVED with the RECEIVE_SMS permission, works on device thats running Android version lower than kitkat, which the device I originally tested on was.

Unfortunately when the code was ready for its prime time and I uploaded it on my primary phone(Nexus 6P <3) and as I received a test message, nothing happened, I was stumbled, but after a quick google I remember, the phone of mine had the new fancy runtime permissions, easy beasy, right?

After setting the sms reading permission, or whatever it is called on the runtime permission management UI and fired another test message,  but as I look into the screen of my trusty Nexus 6P, still nothing appeared, I was getting frustrated!

The testing was getting quite costy, what are those messages here, 8 cents a pop? I have to apply for a loan before the app is finished, I guess..

Then I kept on googling and actually one of the SMS Gateway app creators have blogged about a similar problem on post kitkat devices, and I was getting hopeful, but after declaring the BROADCAST_SMS permission in my manifest, which document for reads as:

Allows an application to broadcast an SMS receipt notification.

Not for use by third-party applications.

This doesn’t sound right, but can a blog post be wrong, mine isn’t so lets try!

Unfortunately it was still no help, no broadcasts were received by my app whatsoever! Not to say that this permission could solve some kitkat problem, I don’t know, I haven’t tried, but lets figure out whats wrong with my receiver!

Time to look into some working apps! I am actually using one of those sms receive listening apps, namely ‘Fonecta Caller‘, probably not familiar for my foreign readers, but its basically a call ID app which shows the caller name if you don’t have it on the address book, so whats better way to find out what I am doing wrong than to decompile one of these apps, right?

I looked into the sources of Fonecta Callers manifest and source but couldn’t find anything exciting, and I was stumbled, but then it hit me, maybe there’s some hidden ‘protection feature’ to annoy us, developers, that one needs to declare an activity in order to be able to receive sms broadcasts, it surely would be weird, but that’s really the only difference between mine and the caller apps source, the receiver wise, and to my surprise it worked!

And after all this hard work, spending couple of hours working on the stupidest problem, I decided to create minimal implementation with a working broadcast receiver of which requirements I am listing below

  • RECEIVE_SMS permission
  • Launcher activity with action ACTION_MAIN  and category LAUNCHER (other combinations might work, I am not sure which is the exact requirement, but without a home visible activity I couldn’t receive any broadcasts)
  • As of Android 6.0 a runtime permission request for RECEIVE_SMS or manually setting the permission on

TLDR:

To receive SMS_RECEIVED broadcasts you must have an activity with action ACTION_MAIN and category LAUNCHER combination, RECEIVE_SMS permission declared in manifest and a runtime permission if you’re running Android 6.0 or newer!

Final result: (logcat)

D/SMSReceiver: onReceive

Implementing Sticky RecyclerView Grid Headers on Android

Hello guys!

I am working on an application that needs to display certain amount of items in a a scrollable view that have headers on top of every one of them, so as the title says, RecyclerView.

However, my data that is coming from SQLite backed database didn’t fit too well on the mould of just using different kind of view types, because the data was coming from SQLite on demand and if I were to use view types to support headers I would have to process the whole database before showing it and add some kind of layer that provides me with the headers.

After couple of quick googles I found couple of promising libraries , but unfortunately for my need, they didn’t have support for GridLayout or required same kind of view type mapping as regular headers, but I wouldn’t give up now, as I knew RecyclerView provides lovely way of adding custom decorations, which were used to implement timeshops sticky headers, and after studying the code of other implementations I was ready to go!

But first I needed to make my own SpanSizeLookUp implementation to separate different sections of the grid.

Sectioned SpanSizeLookUp sketch
Sectioned SpanSizeLookUp sketch

So the basic idea is that last item of each section is given with a span that makes it fill the rest of the row, so that the next sections items are dropped to the next row.

First I needed the information of which item belongs to which section, which I made my StickyGridHeadersAdapter in similar fashion as in timeshops implementation.

So the primary function of the adapter is to provide the header id for each position and its corresponding view.

So now that I got the information I am ready to extend the SpanSizeLookUp!

SpanSizeLookUp has one important function of determining each items span size, which is done by implementing getSpanSize method that takes the position of them item as parameter.

My implementation of getSpanSize first determines whether the item in question is the last item before next header, and if so, it calculates the remaining amount of columns in that row, that it needs to fill in order to bump the following item to next row.

So first we implement the last item before next header check, which is quite simple, take current and next item, and if their ids are different, they belong to different categories, meaning that the item in question is actually the last item before next header! Pretty simple huh?

    /**
     * checks if item is last before next header
     * <p/>
     * aka if next items id is different than current
     *
     * @param itemPosition
     * @return
     */
    private boolean isLastBeforeNextHeader(int itemPosition) {
        //header id of item in question
        long headerId = adapter.getHeaderId(itemPosition);

        //next header id, -1 if out of bounds, aka the rest of the row must be filled!
        long nextHeaderId = -1;

        //next item position, aka next item in question
        int nextItemPosition = itemPosition + 1;

        //checking if is within the bounds of the adapter
        if (nextItemPosition >= 0 && nextItemPosition < adapter.getItemCount()) {
            nextHeaderId = adapter.getHeaderId(nextItemPosition);
        }

        //checking if the ids different
        return headerId != nextHeaderId;
    }

Then we need to implement the actual span size lookup, it needs a little bit more logic!

    @Override
    public int getSpanSize(int position) {
        if (isLastBeforeNextHeader(position)) {

            //gets the number of items before this particular position in range of 0..spancount - 1
            int categoryOffset = getNumberOfItemsBeforePositionInCategory(position);

            //gets column index in range of 0..spancount - 1
            int columnIndex = (categoryOffset % spanCount);

            //gets number of extra columns in range of 0..spancount - 1
            int extraColumns = spanCount - (columnIndex + 1);

            return 1 + extraColumns;
        } else {
            //is just any ordinary item, takes one column width..
            return 1;
        }
    }

So first we use our lastly implemented method and check if it actually is the last item within the section, and if not, just return one.

But if the case is different, aka if it is the last item, we first calculate the offset of this item in question within the section, which is needed to calculate the remaining columns within the row, but for that we need to implement a way to calculate those preceding items.

Fortunately for us, its quite simple chore, given the fact that we can query the header id of each item, so basically what we do is we loop through each item preceding the current item til we find a change in header id!

    private int getNumberOfItemsBeforePositionInCategory(int position) {
        long categoryId = adapter.getHeaderId(position);

        for(int i = 1; i < position; i++) {
            if(adapter.getHeaderId(position - i) != categoryId) {
                return i - 1;
            }
        }

        return position;
    }

And to account for the case that this happens to be within the first category, we just return the given position, as it is the offset to the start of the list.

So with that info we can now calculate the column index..

//gets column index in range of 0..spancount - 1
int columnIndex = (categoryOffset % spanCount);

which is just taking modulus of the offset within category and the span count

and to get the amount of items there are left on this row after the particular item, we just substract the column index + 1 from the span count, accounting for the the range of 0.. span count – 1 of the columnIndex instead of 1 – span count

//gets number of extra columns in range of 0..spancount - 1
int extraColumns = spanCount - (columnIndex + 1);

so then we can just return the amount of columns that this is supposed to fill, meaning one plus the amount of extra columns

return 1 + extraColumns;

So now if we run the code, what we can see, is that each category is withing its own boundaries, cool!

This may not be obvious from the preview above, but if you look closely the last item within each section(read: color) is spanning to fill the rest of the row.

Now we only need to add the headers for each section, let the fun begin!

So we need to implement RecyclerView.ItemDecoration, which has two important methods, onDrawOver to draw over the RecyclerView, which handles the drawing of the header view, and getItemOffsets, which provides the offsets to the views, giving enough space for the header in this case, but the possibilities are limitless!

So first lets give enough space for the headers by implementing getItemOffsets!

    @Override
    public void getItemOffsets(Rect outRect, View view, RecyclerView parent, RecyclerView.State state) {
        super.getItemOffsets(outRect, view, parent, state);

        int itemPosition = parent.getChildAdapterPosition(view);

        if (itemPosition == RecyclerView.NO_POSITION) {
            return;
        }

        boolean underHeader = isUnderHeader(itemPosition);

        if (underHeader) {

            View header = getHeaderView(parent, itemPosition);

            outRect.top = header.getHeight();
        }
    }

This method in my case only checks if the item is under header, and if so, gives it top padding from the header height

To determine if item is under header we use isUnderHeader method which is implemented as follows:

    private boolean isUnderHeader(int itemPosition) {
        return isUnderHeader(itemPosition, spanCount);
    }

    /**
     * checks if item is "under header"
     * <p/>
     * Items is under header if any of the following conditions are true:
     * <p/>
     * a) within spanCount the header id has changed once
     *
     * @param itemPosition
     * @return
     */
    private boolean isUnderHeader(int itemPosition, int spanCount) {
        if (itemPosition == 0) {
            return true;
        }

        //get current items header id
        long headerId = adapter.getHeaderId(itemPosition);

        //loop through each item within spancount
        for (int i = 1; i < spanCount + 1; i++) {
            long previousHeaderId = -1;

            int previousItemPosition = itemPosition - i;

            //gets previous items headerId
            if (previousItemPosition >= 0 && previousItemPosition < adapter.getItemCount()) {
                previousHeaderId = adapter.getHeaderId(previousItemPosition);
            }

            //checks if header id at given position is different from previous header id and if so, returns true to indicate this item belongs under the header
            if (headerId != previousHeaderId) {
                return true;
            }
        }

        return false;
    }

We just loop the range of span count and check if there’s any item that has different header id than the given position, and if so, we know that this item belongs under header and needs to be given space!

now we can implement the drawing of the header, this function is a bit more exotic to handle the stickiness of headers, meaning moving the topmost header upwards when it is pushed by the following header.

    @Override
    public void onDrawOver(Canvas c, RecyclerView parent, RecyclerView.State state) {
        super.onDrawOver(c, parent, state);

        final int childCount = parent.getChildCount();

        //checks if there's any childs, aka can we even have any header?
        if (childCount <= 0 || adapter.getItemCount() <= 0) {
            return;
        }

        //stores the "highest" seen top value of any header to perform the pusheroo of topmost header
        int highestTop = Integer.MAX_VALUE;

        //loops through childs in the recyclerview on reverse order to perform the pushing of uppermost header faster, because before it, there is the next headers top stored to highestTop
        for (int i = childCount - 1; i >= 0; i--) {
            View itemView = parent.getChildAt(i);

            //fetches the position within adapter
            int position = parent.getChildAdapterPosition(itemView);

            if (position == RecyclerView.NO_POSITION) {
                continue;
            }

            //only draw if is the first withing recyclerview, aka is the first view in whole tree or if the item in question is the first under its category(or header..)
            if (i == 0 || isFirstUnderHeader(position)) {
                //fetches the header from header provider, which is basically just call to adapters getHeader/bindHeader
                View header = headerProvider.getHeader(parent, position);

                //calculates the translations of the header within view, which is on top of the give item
                int translationX = parent.getLeft();
                int translationY = Math.max(itemView.getTop() - header.getHeight(), 0);

                tempRect.set(translationX, translationY, translationX + header.getWidth(),
                        translationY + header.getHeight());

                //moves the header so it is pushed by the following header upwards
                if(tempRect.bottom > highestTop) {
                    tempRect.offset(0, highestTop - tempRect.bottom);
                }

                //draws the actual header
                drawHeader(parent, c, header, tempRect);

                //stores top of the header to help with the pushing of topmost header
                highestTop = tempRect.top;
            }
        }
    }

so first we perform couple of sanity checks to avoid useless drawing, we check if there actually is any items within the RecyclerView and if not, we just return to avoid useless work

        final int childCount = parent.getChildCount();

        //checks if there's any childs, aka can we even have any header?
        if (childCount <= 0 || adapter.getItemCount() <= 0) {
            return;
        }

then we initialize the value to store the preceeding headers top value to perform the pusheroo of topmost header, as commented

//stores the "highest" seen top value of any header to perform the pusheroo of topmost header
int highestTop = Integer.MAX_VALUE;

then for each item in backward order we do the following:

check if header should be drawn, by checking if is first item, which always should show header, making the sticky header effect, or if is first under any header

if (i == 0 || isFirstUnderHeader(position)) {

and the checking if the item is last header is quite simple, we just check if the preceding item has different header than current

    private boolean isFirstUnderHeader(int position) {
        return position == 0 || adapter.getHeaderId(position) != adapter.getHeaderId(position - 1);
    }

and the item is first in the whole RecyclerView or first within its category, we now know that we can draw the header for this particular item, so we do that!

first we get the header item

//fetches the header from header provider, which is basically just call to adapters getHeader/bindHeader
View header = headerProvider.getHeader(parent, position);

which works in similar fashion as in timeshops implementation, we just call adapters get/bind header methods to obtain the header bound for this header id

then we just calculate the translations for this header, to draw it on top of the item in question

//calculates the translations of the header within view, which is on top of the give item
int translationX = parent.getLeft();
int translationY = Math.max(itemView.getTop() - header.getHeight(), 0);

then we set these values for a rect to pass them around and to more easily apply moving when translation topmost header to give room for the next

tempRect.set(translationX, translationY, translationX + header.getWidth(),
                        translationY + header.getHeight());

Then we adjust the rect position if it is pushed by the following header, aka if the current headers bottom is smaller than highest top, meaning the previously processed headers top

//moves the header so it is pushed by the following header upwards
if(tempRect.bottom > highestTop) {
    tempRect.offset(0, highestTop - tempRect.bottom);
}

then we actually draw the header

//draws the actual header
drawHeader(parent, c, header, tempRect);

and what this method call does is just draws the view on the given canvas within the rect

    public void drawHeader(RecyclerView recyclerView, Canvas canvas, View header, Rect offset) {
        canvas.save();

        canvas.translate(offset.left, offset.top);

        header.draw(canvas);

        canvas.restore();
    }

and then just set the highestTop to current headers top

//stores top of the header to help with the pushing of topmost header
highestTop = tempRect.top;

and I think we’re about to be done here!

After a quick run we can see it works as expected!

However this is just a simple and quick implementation and we might be missing something here, but for my simple use case it works, and if I find out anything, I’ll update here!

Should of probably used thicker header on the preview but… meh!

Thanks for reading anyway!

Implementing Android autofill form

Today I was implementing an autofilling login form for Android Application which had the following requirements:

  • When user logs in he can check “Remember credentials” box to make the application remember the previous credentials
  • When user would log in next time if he had saved single username before he would be straightly offered with the previously saved credentials
  • If the user had previously entered multiple credentials a drop down will be shown that will let the user decide which of the credentials to use

An obvious choice to perform this task is to use AutoCompleteTextView, but unfortunately it has couple of limitations, which I posted about before, when I created InstantAutoCompleteTextView.

InstantAutoCompleteTextView solved some of these problems, but after that I still had couple of them left to overcome, and most important of them, the AutoCompleteTextView wouldn’t offer auto completion if you haven’t inputted any characters, or that after filling field completely, it would still show the user as hint.

I was using ArrayAdapter as AutoCompleteTextViews username adapter, to list matching usernames, but ArrayAdapters filter uses start of each word for matching prefixes, but in my case, as I was matching for usernames, I would need to match whole username, so basically I had to extend the ArrayAdapter to perform this operation.

Unfortunately ArrayAdapter stores the objects that it has as private field, so my extending adapter wouldn’t have access to them, without reimplementing most of its logic, so what I did, was open the ArrayAdapters source and reimplement most of its business, and reimplement the Filter it provides.

How I changed the Filter is that instead of matching for each word, I matched the whole string, and if the inputted text was exact to the adapters object, it wouldn’t be included, as when user had an exact match, it wouldn’t be needed to be shown anymore.

    private class ExactArrayFilter extends Filter {


        @Override
        protected FilterResults performFiltering(CharSequence prefix) {
            FilterResults results = new FilterResults();

            if(originalObjects == null) {
                originalObjects = Collections.synchronizedList(objects);
            }

            if (prefix == null || prefix.length() == 0) {
                List<T> list = new ArrayList<>(originalObjects);

                results.values = list;
                results.count = list.size();
            } else {
                String prefixString = prefix.toString().toLowerCase();

                final List<T> values = Collections.synchronizedList(originalObjects);

                final int count = values.size();
                final ArrayList<T> newValues = new ArrayList<>();

                for (int i = 0; i < count; i++) {
                    final T value = values.get(i);
                    final String valueText = value.toString().toLowerCase();

                    // Match against the whole with prefix and disinclude exact matches
                    if (valueText.startsWith(prefixString) && !valueText.equals(prefixString)) {
                        newValues.add(value);
                    }
                }

                results.values = newValues;
                results.count = newValues.size();
            }

            return results;
        }

        @Override
        protected void publishResults(CharSequence constraint, FilterResults results) {
            objects = (List<T>) results.values;
            if (results.count > 0) {
                notifyDataSetChanged();
            } else {
                notifyDataSetInvalidated();
            }
        }
    }

Now I have my Filter that makes the popup disappear after matching input is typed into.

Then I just have to implement the piece of code that handles the password autofill, and how I did that, is I stored set of user credentials as plain text(very secure) in JSON string to preferences, and now I could just when the AutoCompleteTextViews value changes, look up if there’s a already saved password and use that.

So what that basically meant is to add an TextWatcher and on afterTextChanged perform this check and set a fake password on the password field and store the selected password, or “override password” on other field to be used on login, so the potentially blackhat user can’t use the number of characters in the password to gain access to the password more easily.

Also now that I knew if the password was “fake” and “overriden” I could add a TextWatcher and another afterTextChanged callback that would reset the password if override was present, to imitate usual autofill behaviour.

Below you can see the final result

Thanks for reading!

Patching Android Application for fun and.. fun!

Hello guys! Its reverse engineering time!

But first quick disclaimer..

ALL THIS MATERIAL IS PROVIDED FOR STUDYING PURPOSES ONLY AND MAY NOT USED TO PERFORM ANYTHING ILLEGAL

Yesterday my friend told me about a certain Android application that had a voucher for one lunch place, so as software engineers, a question quickly popped up – how would one work around it, and reuse the voucher.

Its surely connecting to a API of some kind to fetch the details of available vouchers, and it must somehow mark the voucher used after the cashier presses the “Use” button.

So working around it should be simple, right? We could use a proxy and just filter the consume requests? That would require extensive amount of work to do inside the phone and possibly rooting.

What about changing the devices DNS or setting the proxy to external server which would do the processing for us? Unfortunately as it turns out after setting a simple proxy, as the application happens to use certificate pinning, which basically means that doing a Man In Middle Attach is slightly harder, we would need to first change the bound certificate withing the application and then make our proxy sign the requests with the certificate inside the application.

So the next thought as a hobbyist reverse engineer is to export the APK to PC, decompile it and patch the request so it is not actually never sent to the server, but the completion listener is fired and the application thinks that we have actually received a positive answer from the server.

So first I had to figure out how hard is it to patch applications? Exporting the APK is simple as downloading an app from Google Play and fetching it over adb to PC.

Working with the APK itself is slightly harder because the APKs have compiled the Java source to different format than usual Java classes, meaning the dex file.

There is a classes.dex file in the root of the .apk file after you open it with your favourite zip tool(WinRAR anyone?)

apk-root

So we take the classes.dex out from the APK and go ahead and figure what can we do with it?

We need to use some kind of tool to turn this into class files to better work with it.

So as I have previously worked with dex files I knew there was a tool called dex2jar(I like the naming btw) that would let me convert the classes.dex file(or the whole apk) to a .jar package containing the classes, but I had never before repackaged the outcoming classes into classes.dex, so this would be something new.

So I downloaded the dex2jar from its github page and unpackaged the binaries and saw what tools I now had at my disposal.

  • d2j-dex2jar.bat
  • d2j-jar2dex.bat

which meant that possibly I could do the unpacking and repacking with just these tools, right?

So I transferred the classes.dex to the containing folder and fired couple of commands to turn the classes.dex into classes-dex2jar.jar which I could then work with

d2j-dex2jar.bat classes.dex

As it turns out after a minute, I had the .jar file within the same folder and was ready to get my hands dirty!

After unpackaging the jar I needed a tool I could work on the .class files with, I had recently bought DJ Java Decompiler but I thought I could try something new this time, as I have recently stumbled upon a Bytecode Viewer, and decided to give it a go!

The tool was supposed to have inline bytecode edition, but I couldn’t get it working, but I could find what I was looking for, the place where the HTTP request is made for the item to be consumed.

It have been a while since I have last time got my hands dirty with Java Bytecode or with decompiled code.

This piece of code is a anonymous class of anonymous class, which you can see from the naming of the class, Java adds postfix to class names that have been compiled from anonymous classes(classes within classes).

But from this code I can see that it is calling some method consumeCoupon, which after checking from its definition sends the HTTP request, and also it takes a OnCouponConsumedCallback as a parameter, so we know it is also passed here, even if its hard to judge from the decompiled source.

So what we can do now, is see what methods does this callback have and see if we can make this method call the completion callback straight, instead of going to the HTTP Server first, which ultimately means the coupon is never actually consumed in the server side.

The interface has two methods, but we only care about the other, onCouponConsumed, it takes a int, which possibly is the position within some list, or somekind of id, and a CouponItem.

The server request passes also a position along in the request, so we can try to plug that in and see what happens, unfortunately at this moment I have to change tool, because I am unable to edit the bytecode within this tool.

So its time to fire up jbe(Java Bytecode Editor) which is also new tool to me

 

After looking at this beauty for a while I was ready to perform some patching

First of all I had to remind myself a little how the bytecode worked and in which order things had to be in order for this to work

So first I removed the isAdded check, because it is not needed, and I don’t even know what it checks for, which was matter of just trimming the code down a little

Then i started the real work, first I had to load the object into the stack, which I am calling the method on, so I had to figure out how is the OnCouponConsumedCallback added to the stack, which was relatively easy, I just tracked it down by the knowledge of the argument order.

I moved it to the top of the bytecode list, so it was first loaded on stack, as it is the object I am calling the method on.

Then I needed the parameters, the method I am calling is taking first the int and then the CouponItem as parameter, so I needed to figure out exactly how those were loaded to the older function call and move them after the loadage of listener.

Now I only need to call the method with the given pameters, so its time to stretch my fingers and check up the opcode names

Ultimately I ended up with the following code

And to check that the code is valid before getting it back to the apk and trying it on my device I first open the saved class file on the ByteCode Viewer

And boom! Its just like I expected

Now its time to get this goodness into the apk and actually be able to install it into the device! So first I added this class back to the .jar that the dex2jar outputted originally

Then I ran jar2dex on it

And there I had it, a shiny new classes.dex ready to be loaded into my precious devices memory!

So I moved the new classes.dex back to the apk, and now it was only matter of resigning the apk and installing it

So I had to create a new keystore for this purpose, so I fired up keytool

 

And ran it with -genkey -alias pasi -keystore key arguments

and set unknown values to all the questions asked, and was ready to go!

Now its time to sign the apk using jarsigner, so I ran the following command

C:\Users\PasiMatalamaki\Desktop>jarsigner -verbose -keystore key -signedjar “app_ua.apk” “app.apk” pasi -storepass helloworld

 

and that ultimately generates a signed versio of the “app.apk” apk file signed with keystore with name key(the one we just created) using the alias pasi and password helloworld

Now I have to align it using zipalign to make it work with android devices so I ran

zipalign -f 4 “app_ua.apk” “app_release.apk”

decompiledmethod8

And ended up with app_release.apk that can now finally be installed on my phone and after a quick try, it worked like a charm!

Of course this is not to be used in real world, because that would be illegal, this is just proof of concept.

But thanks for reading and hit me up in comments if you’re interested in more of this kind of material.

Android AutoCompleteTextView with drop-down always visible or how to figure your way with internal Android APIs

Hello guys!

Today I was stuck with yet another Android related problem, I was creating a edit text with auto completion that always shows the drop down to help the user remember the last choices.

So the ultimate requirement is to on activity open show list of all items in the auto completion adapter in a drop-down, which should be simple right?

The obvious choice is to use AutoCompleteTextView, which provides the feature of listing possible choices after inputting certain amount of characters, however I knew that it wouldn’t show the listing before inputting certain amount of characters.

First thing to do was of course to open the documentation which didn’t appear to contain any helpful information, however I threw together the activity with one of these view with a dummy adapter and tested how it works normally.

Normally it takes two matching characters to get the drop down to appear, and there is actually a option to set the threshold, so I decided to set it to 0 and checked what would happen if I focused the field

Judging from the documentation of setThreshold I know that if the threshold is set under one, the threshold is actually being set to minimum of one.

When threshold is less than or equals 0, a threshold of 1 is applied.

source: developer.android.com

No luck.. You have to input at least one character to make the popup appear. I think its time to browse through the source..

First thing I noticed was that the AutoCompleteTextView has reference to the ListPopupWindow which is ultimately used to show the actual popup that lists the possible options

I looked for the conditions which needs to be fulfilled in order for the popup to be shown and found out that there is actually a flag on the ListPopupWindow, isDropDownAlwaysVisible, which also has a setter, setDropDownAlwaysVisible, so I was sure I was onto something!

Unfortunately the setDropDownAlwaysVisible setter had a @hide javadoc annotation, which removes the method from public with the following comment

Only used by AutoCompleteTextView under special conditions.

So I decided to find out those conditions, but there was only single method call to this setter within AutoCompleteTextView which was just a delegate setter which too had a @hide javadoc annotation, but this time the comment declared that it was pending API council approval, which meant this was a dead end.

This basically meant that I needed to hook on the onFocusChanged by either subclassing or setting a listener and within that hook make the popup visible.

Luckily there was public showDropDown method within AutoCompleteTextView which meant that subclassing wouldn’t be necessary and this could be a matter of a simple View.OnFocusChangeListener, the day was saved!

So I created a View.OnFocusChangeListener and implemented its onFocusChange callback that calls showDropDown on focus

Unfortunately it turns out that the onFocusChange may be called even if the window containing isn’t necessarily visible, so this is fixed with matter of checking the fact, right?

So I added a check if getWindowVisiblity of the view that the onFocusChange listener passes as parameter is visible, but as it turns out after orientation change the View.OnFocusChangeListener is triggered before the window is visible, okay.. we have a problem here..

I thought to myself, okay.. so if we add a callback to the view so I’ll know when the containing view is visible, right? Unfortunately the View doesn’t provide an API or a mechanism to do attach a callback to the onAttachedToWindow method, so I had to scrape that idea..

After swallowing my pride that this couldn’t be implemented using a listener, I decided to give it a try to subclass the AutoCompleteTextView and override nessaccary methods such as onFocusChanged and onAttachedToWindow

I ended up creating a little helper method showDropDownIfFocused that checks whether enoughToFilter, isFocused and getWindowVisibility matches the conditions and if so, call the showDropDown method of AutoCompleteTextView and calling this from onFocusChanged  and onAttachedToWindow

I also override enoughToFilter method to make sure that I could set minimum threshold to zero using a special flag of my new InstantAutoCompleteTextView

Below you can see the final result of the progress.

 

The final source code can be obtained as gist here

If you have better ideas of how the problem could be solved, feel free to comment below.

Thanks for reading