Functions whitelist

[jpalanco]

I have been playing with cosa-nostra and I found very interesting information, it works pretty well. I tried with a dataset of more than 10k samples. I realized that in some cases, some samples dont belong to the sample family, but they re-use small pieces of code (maybe a statically compiled library or copy/paste code from internet).

I think it may be very useful to have a whitelist of this common functions to avoid merging clusters.

Regards

[jpalanco]

I attach my dataset. For example the clusters with id 3 and 60 are a good examples

db.sqlite.zip

[joxeankoret]

This is the problem of this approach: collisions. Let me explain why a white-list is most likely a bad idea. In some cases, I have had false positives with Cosa Nostra where 2 DLLs, totally different, only differ in the DllMain procedure but have SQLite, OpenSSL and other libraries statically compiled in. Naturally, the number of functions shared between them is very high.

We know such functions must be, somehow, ignored. So, what's the obvious solution? White-listing, right? OK. Let's compile statically all the most used open-source libraries and C runtimes out there to extract signatures of functions to ignore.

When you start doing this (as I did in the past), you will realise you are never done adding open source libraries (Curl, SQLite, OpenSSL, BigNumbers, Zlib, Boost, etc are just some on top of my mind) and, even when you believe that you're "mostly done", another problem arises: you need to build them with various compilers and runtimes (GCC, CLang, MingW, Microsoft C++ 2010/2012/2013/2015, ...). And, if you happen to continue after this part, you will realise there is still one more problems: you need to build the libraries to extract signatures with all the possible compiler -O flags (-Os, -O0, -O1, -O2 and -O3) because the code can be pretty-pretty different.

And, supposing you continue with this idea after this point, you have yet another problem: how to handle and make use of that massive database of functions to ignore without making the whole process incredibly slow.

[joxeankoret]

BTW, this is why I did start another open source project with a similar idea on mind. Check this: https://github.com/joxeankoret/maltindex

[jpalanco]

Thank you Joxean, when I said function whitelist I didn't mean to hashing the functions but identifying for example with yara.

BTW maltindex looks awesome, I'm fond of your projects :D

[joxeankoret]

Thank you! :)

BTW, using Yara or IDA FLIRT signatures or any other way is actually doing the same. I wouldn't go for that approach.