Experimental Vue.JS GUI

README.md

@@ -25,7 +25,9 @@ On non-Hermes based setups of React Native, this file usually contains minified/
 
 ## Installation
 
-The application itself only relies on the Python 3.x standard library for now.
+The command-line application itself only relies on the Python 3.x standard library for now.
+
+The experimental GUI also relies on the a few external modules (listed in the `pyproject.toml` file below) to work.
 
 You can install the tool through one the following commands:
 

docs/GUI index database format.md

@@ -0,0 +1,71 @@
+# GUI index database format
+
+This file summarizes and describes the data that should be indexed in order to speed up/allow for checking cross-refernces/dumping decompiled code over the `hermes-dec` GUI.
+
+## Summary of use/dataset pairs
+
+XX
+
+In order to *implement viewing cross-references for function cross-references (both from the side pane of the application and the decompiled code) of the `hermes-dec` web UI*, we should implement the following functionality:
+- XX
+
+In order to *implement viewing cross-references for raw strings in the HBC file(both from the side pane of the application and the decompiled code) of the `hermes-dec` web UI*, we should implement the following functionality:
+- XX
+
+In order to *implement searching for function names and offsets from the left side pane of the `hermes-dec` web UI*, we should implement the following functionality:
+- XX
+
+In order to *implement searching for raw strings in the HBC file from the left side pane of the `hermes-dec` web UI*, we should implement the following functionality:
+- XX
+
+In order to *implement viewing decompiled code with cross-references in the code browser of the `hermes-dec` web UI*, we should implement the following functionality:
+- XX
+
+
+## Cross-reference database
+
+### Code common to launching the indexer subprocess
+
+- In `server.py` entry point -> `WebsocketServer.handle_ws_client`
+  - `server.py` -> `WebsocketClient.create_file`
+    - `project_meta.py` -> `ProjectInstanceManager.new_with_name`
+      - `self.subdir_path` = `join(user_data_dir('HermesDec', 'P1Security'), self.gen_unique_dirname(name))`
+        - Creates: `self.subdir_path` = `/home/marin/.local/share/HermesDec/by-date/2023-09-06T20:53:55-samplehbc/' + '{index.android.bundle,metadata.json}`
+  - `server.py` -> `WebsocketClient.parse_file`
+    - `project_meta.py` -> `ProjectInstanceManager.save_to_disk`
+      - `project_meta.py` -> `ProjectInstanceManager.write_or_update_metadata` (writes `metadata.json` in the project directory)
+    - `ProjectInstanceManager.write_or_update_metadata` also called here for updating the bytecode version
+  - `server.py` -> `WebsocketClient.spawn_indexer`
+    - (xxx: code to write with ensure_future(asyncio.spawn_subprocess XX) + launching subprocess as described in the GUI index worker pipe IPC.md" file + watching the output and doing communication with the Websocket and local object state when needed)
+
+### Functions/function and function/built-in cross-references (SQLite DB)
+
+The application should maintain, in an SQLite database, a repository of cross-refences between HBC functions and the string IDs, function IDs, and built-in IDs they reference.
+
+This information should be referenced into the SQLite/SQLAlchemy defined in "`src/gui/index_worker/index_db.py`".
+
+In order to perform clean paralellization and to avoid daemon thread-related or GIL-related issues, indexing should be done into a subprocess that frequently reports its state of indexing through a `stdout`-based pipe mechanism, along with other indexing tasks described in the present document, as described in the `GUI index worker pipe IPC.md` document present in the current directory.
+
+XX Implementation notes
+XX
+
+Call chain:
+
+
+
+
+### Function strings cross-references (SQLite DB)
+
+XX
+
+## Functions list (free-text search/SQLite DB?)
+
+XX
+
+## Strings list (free-text search/SQLite DB?)
+
+XX
+
+## Pre-generated decompiled code with inline x-refs (SQLite DB/custom markup?)
+
+XX

docs/GUI index worker pipe IPC.md

@@ -0,0 +1,57 @@
+This document presents the interprocess communication used for the decompiler/index worker to communicate with the main Websocket server worker over the `hermes-dec` GUI.
+
+Communication is done over unidirectional unbuffered process pipe (stdout) communication, and passing CLI arguments + watching the present of an otherwise input pipe (stdin) for cancelling cleanly on the other side whenever the parent process gets terminated.
+
+## Subprocess args
+
+The subprocess shall have the following arguments:
+
+```
+orig_argv[0]: Python interpreter
+orig_argv[1]: Worker script full path
+orig_argv[2]: Data folder path
+orig_argv[3]: Raw .HBC file path
+```
+
+## Cancel signal (parent->child)
+
+Just a SIGINT/simulated CTRL+C should be used for ending out the subprocess when needed (for example when the parent process gets interrupted itself)
+
+The child watches the presence of an otherwise unused stdin pipe in a background/daemon thread, and interrupts its main thread whenever the stdin pipe gets broken.
+
+## Child end signal (child->parent)
+
+The subprocess should terminate when the whole document has been indexed (or was already indexed and the indexing data can be loaded immediately).
+
+## Line-delimited JSON subprocess pipe communication spec (child->parent)
+
+The following message types shall be implemented, and serialized with a `\n` endline through stdout:
+
+- `indexing_state`: inform the Websocket worker process about the state of the indexing (immediately after he index has been loaded from disk if it was alreay generated earlier, or at every step of the data generation process) along with an estimated process rate ranging from 0 to 100 when finished
+Example: ```json
+{
+    "type": "indexing_state",
+    "state": "indexing_functions" (Later: | "decompiling_code") | "fully_indexed" (Later: | "XXX"),
+    (Later: "process_percent": 95)
+}
+```
+
+- (Later: ) `indexing_status_log`: transmit a readable log string about the indexing status of the document
+Example: ``json
+{
+    "type": "indexing_status_log",
+    "message": "Text line to log"
+}
+```
+
+- ``
+
+## Fetching the indexed and text chunk-decompiled data
+
+Fetching the indexed and text chunk-decompiled data is done in the process of the main worker, thanks to the shared SQLAlchemy ORM code with the main process. (Once all the code has been decompiled, the secondary/index worker emits its status and closes)
+
+## TODO also
+
+Check whether the pass1b/1c of the decompilation pipeline are still required when just decompiling and not rendering the UI graph. (=> Yes, it will be used to optionally reconstruct contron flow later)
+
+(Also obviously send their text status comments regularly to the new decompilation pipeline processing console at the bottom pane of the web UI in order to be able to debug where the background decompilation process takes times, freezes, crashes, etc.)

docs/GUI server websocket protocol.md

@@ -0,0 +1,101 @@
+# GUI Websocket protocol
+
+The browser/Websocket-based GUI of `hermes-dec` should implement the following JSON messages:
+
+```
+S->C {"type": "recent_files", recent_files: [
+    {
+        "file_hash": ...,
+        "orig_name": ...
+    }, ...
+]} (saved files w/ context, etc.)
+
+C->S {"type": "open_file_by_hash", "file_hash": "<sha256 HEX>"} (load saved context)
+
+S->C {"type": "file_hash_unknown", "file_hash": "<sha256 HEX>"}
+
+// (Implement first:)
+C->S {"type": "begin_tranfer", "file_name": "x.index.bundle OR apk", "file_hash": "<sha256 HEX>"}
+(Raw binary buffer websocket transmission...)
+C->S {"type": "end_transfer"}
+
+// (Maybe implement later): We should both prompt the user using our own file shell (in order to have the full path) and copy the file name + metadata + size + origin path + create a parameters file in ~/.hermes-dec:
+C->S {"type": "open_file", "disk_path": "(...)/x.index.bundle"}
+
+S->C {"type": "pop_message", "icon": "error", "message_html": "Could not open file:<br><br><pre>XXX</pre>"}
+
+S->C {
+    "type": "file_opened",
+    "file_metadata": {
+        "bytecode_version": 76,
+        "db_created_time": "2023-04-07T08:33:57.200578",
+        "db_updated_time": "2023-07-18T13:53:13.827344",
+        "dir_disk_path": "/home/marin/.local/share/HermesDec/by-date/2023-04-07T08:33:56-indexandroidbundle",
+        "file_hash": "b2b514abac3dedbc83619d0e782ae61a0d958038a432f01a11b22d09537d7090",
+        "file_size": 3482132,
+        "orig_name": "index.android.bundle",
+        "raw_disk_path": "/home/marin/.local/share/HermesDec/by-date/2023-04-07T08:33:56-indexandroidbundle/index.android.bundle"
+    }
+}
+
+S->C {
+    "type": "console_log",
+    "data": "Raw text\n"
+}
+
+
+S->C {
+    "type": "console_error_log",
+    "data": "Raw text\n"
+}
+
+C->S
+{
+    "type": "get_table_data",
+    "table": "functions",
+    "text_filter": "xx" or null,
+    "current_row": 21,
+    "page": 42
+}
+
+S->C
+{
+    "type": "table_data",
+    "table": "functions",
+    "result_count": 452,
+    "pages": 12,
+    "current_page": 1,
+    "model": {
+        "has_search_bar": true,
+        "has_pagination": true,
+        "pagination_thresold": 350,
+        "columns": [
+            'Name', 'Offset'
+        ],
+        "has_visible_headers": true,
+        "has_selectable_rows": true
+    },
+    "displayed_rows": [{
+        "id": 42,
+        "cells": ["fun_000001", '%08x', '42']
+    }, ...]}
+
+C->S {"type": "analyze_function", "function_id": 49}
+// Will return disassembly graph + cross-references + decompiled code with clickable expandable "..." buttons for nested closures for a given function
+
+S->C {"type": "analyzed_function", "function_id": 49, "blocks": [{
+    "grid_x": 1, // Used for rendering with the CSS grid layout
+    "grid_y": 2,
+    "text": "Assembly...\n",
+    "child_nodes": [index, index...],
+    "child_error_nodes": [index, index...],
+    "parent_nodes": [index, index...],
+    "parent_error_nodes": [index, index...]
+}]}
+
+(+ a progress bar upstream packet?)
+
+(+ a search feature?)
+
+(+ a code export button?)
+```