From 9cc206b3a7b434cf40eecfa4be20b495af234188 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Sat, 25 Jul 2026 01:49:14 +0300 Subject: [PATCH 01/11] Make the docs consistent with week 6 code change --- docs/test_env/README.md | 2 +- .../architecture/03-database-schema.md | 214 +++--------------- 2 files changed, 28 insertions(+), 188 deletions(-) diff --git a/docs/test_env/README.md b/docs/test_env/README.md index 791959861e40f..d5aaa189fea5a 100644 --- a/docs/test_env/README.md +++ b/docs/test_env/README.md @@ -8,7 +8,7 @@ This directory contains the architecture and workflow design for the `test_env` |----------|-------------| | [01-command-dispatcher.md](https://github.com/Nayeraneru/metasploit-framework/blob/vulnenv-week1/docs/test_env/architecture/01-command-dispatcher.md) | How `test_env` is added to msfconsole via plugin dispatcher | | [02-module-metadata.md](https://github.com/Nayeraneru/metasploit-framework/blob/vulnenv-week1/docs/test_env/architecture/02-module-metadata.md) | How modules expose `VulnEnv` metadata and how the plugin reads it | -| [03-database-schema.md](https://github.com/Nayeraneru/metasploit-framework/blob/vulnenv-week1/docs/test_env/architecture/03-database-schema.md) | Registry persistence: in-memory Phase 1, PostgreSQL Phase 2 | +| [03-database-schema.md](https://github.com/Nayeraneru/metasploit-framework/blob/vulnenv-week1/docs/test_env/architecture/03-database-schema.md) | Registry persistence: in-memory Phase 1, YAML Persistence with ActiveModel Phase 2 | | [04-environment-schema.md](https://github.com/Nayeraneru/metasploit-framework/blob/vulnenv-week1/docs/test_env/architecture/04-environment-schema.md) | YAML schema for shared environment definitions in `data/vuln_envs/` | | [05-runtime-adapter.md](https://github.com/Nayeraneru/metasploit-framework/blob/vulnenv-week1/docs/architecture/05-runtime-adapter.md) | Docker/Podman abstraction, port allocation, container labels | diff --git a/docs/test_env/architecture/03-database-schema.md b/docs/test_env/architecture/03-database-schema.md index 1abc45bf73426..ec225524d3049 100644 --- a/docs/test_env/architecture/03-database-schema.md +++ b/docs/test_env/architecture/03-database-schema.md @@ -182,204 +182,44 @@ docker run -d \ | Schema evolution | Adding a new label is simpler than versioning a JSON schema | | No encoding/decoding complexity | No Base64, no JSON parsing errors | -### State Reconstruction From Labels (Future Enhancement) -```ruby -def reconstruct_from_labels(runtime) - # Step 1: Discover all framework-managed containers via native filter - containers = runtime.list(filters: { 'label' => 'msf.vulnenv.managed_by=test_env' }) - - containers.each do |container| - labels = container['Config']['Labels'] || {} - - # Step 2: Skip containers from other msfconsole instances - instance_id = labels['msf.vulnenv.instance_id'] - next unless instance_id == current_instance_id - - # Step 3: Extract minimal dynamic data from labels - module_fullname = labels['msf.vulnenv.module'] - env_id = labels['msf.vulnenv.env_id'].to_i - version = labels['msf.vulnenv.version'] - # Parse port mapping: "8081:8080,9090:61616" -> {8080=>8081, 61616=>9090} - ports = parse_port_label(labels['msf.vulnenv.ports']) +## Phase 2: YAML Persistence with ActiveModel (Week 6+) - # Step 4: Load module and resolve its VulnerableEnvironment definition - mod = framework.modules.create(module_fullname) - next unless mod +Phase 2 replaces purely in-memory storage with **ActiveModel-backed YAML persistence** in `~/.msf4/test_env_registry.yml`. This provides cross-session state sharing without requiring any framework database changes. - vuln_env_meta = mod.send(:module_info)['VulnerableEnvironment'] - next unless vuln_env_meta - - definition_name = vuln_env_meta['definition'] - profile = vuln_env_meta['profile'] || 'default' - overrides = vuln_env_meta['overrides'] || {} - - # Step 5: Resolve environment config from YAML - loader = EnvironmentDefinitionLoader.new(Msf::Config.data_directory) - config = loader.resolve(definition_name, version, profile, overrides) - - # Step 6: Build datastore from port_mapping + allocated ports - datastore = { 'RHOSTS' => '127.0.0.1' } - vuln_env_meta['port_mapping'].each do |container_port, ds_option| - datastore[ds_option] = ports[container_port] - end +### Why ActiveModel + YAML? - # Step 7: Reconstruct registry entry - @environments[env_id] = { - local_id: env_id, - container_id: container['Id'], - module_fullname: module_fullname, - env_version: version, - rhost: '127.0.0.1', - rport: ports.values.first, - runtime: runtime.name, - image_ref: container['Config']['Image'], - status: container['State']['Status'], - exploit_command: build_exploit_command(datastore), - datastore: datastore, - created_at: Time.parse(container['Created']), - started_at: Time.parse(container['State']['StartedAt']) - } - - @next_id = [@next_id, env_id + 1].max - end -end - -# Parse "8081:8080,9090:61616" into {8080=>8081, 61616=>9090} -def parse_port_label(label_value) - return {} unless label_value - - label_value.split(',').each_with_object({}) do |pair, hash| - host_port, container_port = pair.split(':') - hash[container_port.to_i] = host_port.to_i - end -end +| Concern | Resolution | +|---------|-----------| +| **Framework coupling** | No dependency on `framework.db.active` or PostgreSQL | +| **Upstream friction** | No migration files, no `metasploit-data-models` coordination | +| **Portability** | Works in any msfconsole session, even without `msfdb` | +| **Concurrency** | `flock` file locking enables safe multi-process access | +| **Security** | `Psych.safe_load` with permitted classes prevents arbitrary code execution | -# Build exploit command from datastore -def build_exploit_command(datastore) - cmds = datastore.map { |k, v| "set #{k} #{v}" } - cmds.join('; ') + '; exploit' -end -``` +### Design Overview +- **`VulnTarget`** — `ActiveModel::Model` representing a single provisioned container +- **`VulnEnvironment`** — `ActiveModel::Validations` collection of all targets +- **`VulnEnvironmentStore`** — Atomic load/save via `File::LOCK_SH` / `File::LOCK_EX` -## Phase 2: Database Integration (Week 6+) +### Cross-Session Behavior -When adding PostgreSQL persistence: +The YAML file acts as a **single shared registry** across all msfconsole instances: -### Migration File -```ruby -# db/migrate/20240624000001_create_vuln_environments.rb -class CreateVulnEnvironments < ActiveRecord::Migration[8.0] - def change - create_table :vuln_environments, id: :serial do |t| - t.string :container_id, null: false - t.string :image_ref, null: false - t.string :module_fullname, null: false - t.string :env_version - t.string :rhost, default: '127.0.0.1' - t.integer :rport, null: false - t.text :datastore - t.string :runtime, default: 'docker', null: false - t.string :msf_instance_id - t.string :status, null: false, default: 'running' - t.text :exploit_command - t.timestamps - t.datetime :started_at - t.datetime :stopped_at - t.datetime :removed_at - end - - add_index :vuln_environments, :module_fullname - add_index :vuln_environments, :status - add_index :vuln_environments, :container_id, unique: true - add_index :vuln_environments, :msf_instance_id - add_index :vuln_environments, [:status, :module_fullname] - end -end -``` +- **Global sequential IDs**: All sessions reload the file before allocating, so IDs are sequential globally (1, 2, 3...) regardless of which terminal created them +- **Compaction on removal**: When an environment is removed, IDs are renumbered to close gaps (1, 2, 4 → 1, 2, 3) +- **Pruning on startup**: Dead entries (containers that no longer exist) are automatically removed and IDs compacted before new allocations -### ActiveRecord Model -```ruby -class VulnEnvironment < ActiveRecord::Base - self.table_name = 'vuln_environments' - serialize :datastore, JSON - - scope :active, -> { where(status: ['running', 'stopped']) } - scope :running, -> { where(status: 'running') } - scope :by_module, ->(name) { where(module_fullname: name) } - - validates :container_id, presence: true, uniqueness: true - validates :module_fullname, presence: true - validates :rport, presence: true, numericality: { only_integer: true } - validates :status, inclusion: { in: %w[running stopped removed orphaned error] } -end -``` - -### Integration With In-Memory Registry - -```ruby -class BuiltEnvironmentRegistry - def initialize(framework) - @framework = framework - @environments = {} - @next_id = 1 - load_from_database if database_available? - end - - private - - def database_available? - framework.db.active && defined?(VulnEnvironment) - end - - def load_from_database - VulnEnvironment.active.each do |db_env| - @environments[@next_id] = { - local_id: @next_id, - db_id: db_env.id, - container_id: db_env.container_id, - # ... map all fields ... - } - @next_id += 1 - end - end - - def persist_to_database(record) - VulnEnvironment.create!(...) - end -end -``` - -## Reference: sessions Table Pattern - -From `db/schema.rb`: -```ruby -create_table "sessions", id: :serial, force: :cascade do |t| - t.integer "host_id" - t.string "stype" - t.string "via_exploit" # Module association - t.string "via_payload" - t.string "desc" - t.integer "port" - t.string "platform" - t.text "datastore" # Serialized hash - t.datetime "opened_at", precision: nil, null: false - t.datetime "closed_at", precision: nil - t.string "close_reason" - t.integer "local_id" # In-memory mapping - t.datetime "last_seen", precision: nil - t.integer "module_run_id" - t.index ["module_run_id"], name: "index_sessions_on_module_run_id" -end -``` +### State Reconstruction -My `vuln_environments` table follows this exact pattern: -- `id: :serial` primary key -- `module_fullname` like `via_exploit` -- `datastore` serialized text -- `local_id` equivalent via `env_id` label -- Lifecycle timestamps (`created_at`, `started_at`, `stopped_at`, `removed_at`) +On startup, the plugin: +1. Loads the shared YAML registry +2. Queries the runtime for all containers with `label=msf.vulnenv.managed_by=test_env` +3. Prunes registry entries whose `container_id` no longer exists +4. Compacts IDs after pruning +5. Reconstructs any running containers missing from the registry by reading their labels and assigning the next sequential ID +**Important:** Reconstruction uses `container_id` as the ground truth for matching, not the `env_id` label. The `env_id` label becomes stale after ID compaction and is only used as a historical reference. \ No newline at end of file From ab3c1b31382d5e2f08d5bb5f97d9872b07fe8c71 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Sat, 25 Jul 2026 18:49:51 +0300 Subject: [PATCH 02/11] Environment Management Commands --- plugins/test_env.rb | 301 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 281 insertions(+), 20 deletions(-) diff --git a/plugins/test_env.rb b/plugins/test_env.rb index ba2caf3388ebc..0764977088d48 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -11,7 +11,6 @@ require 'timeout' require 'net/http' - module Msf class Plugin::TestEnv < Msf::Plugin # ===================================================================== @@ -853,6 +852,12 @@ def initialize(framework) end + # Docker run returns full 64-char IDs, but docker ps returns short 12-char IDs. + # Normalize to short form so registry lookups and runtime queries are consistent. + def normalize_container_id(id) + id.to_s[0, 12] + end + def register(container_id:, module_fullname:, env_version: nil, runtime:, image_ref:, datastore: {}, allocated_ports: {}, temp_dirs: []) @mutex.synchronize do @@ -860,7 +865,7 @@ def register(container_id:, module_fullname:, env_version: nil, target = VulnTarget.new( local_id: id, - container_id: container_id, + container_id: normalize_container_id(container_id), module_fullname: module_fullname, env_version: env_version, runtime: runtime, @@ -883,7 +888,7 @@ def register_with_id(env_id:, container_id:, module_fullname:, env_version: nil, @mutex.synchronize do target = VulnTarget.new( local_id: env_id, - container_id: container_id, + container_id: normalize_container_id(container_id), module_fullname: module_fullname, env_version: env_version, runtime: runtime, @@ -940,7 +945,7 @@ def remove(id) end @vuln_env.remove_target(id) - compact_ids! + #compact_ids! @store.save(@vuln_env) end @@ -963,12 +968,13 @@ def prune_and_compact(runtime) # PRUNE: remove registry entries for containers that no longer exist alive_ids = runtime.list(filters: { 'label' => 'msf.vulnenv.managed_by=test_env' }).map { |c| c['ID'] } @vuln_env.targets.each do |target| - # docker ps returns short IDs (12 chars), but registry stores full IDs (64 chars) - unless alive_ids.any? { |id| target.container_id.start_with?(id) } + unless alive_ids.any? { |id| target.container_id == id } @vuln_env.remove_target(target.local_id) end end - compact_ids! # Renumber after pruning dead entries + # NOTE: We do NOT compact IDs here. Sparse IDs are stable references. + # Compaction during batch operations causes identity shift bugs. + #compact_ids! # Renumber after pruning dead entries @store.save(@vuln_env) end @@ -1063,12 +1069,12 @@ def running? private - def compact_ids! - sorted = @vuln_env.targets.sort_by(&:local_id) - sorted.each_with_index do |target, idx| - target.local_id = idx + 1 - end - end + #def compact_ids! + #sorted = @vuln_env.targets.sort_by(&:local_id) + #sorted.each_with_index do |target, idx| + #target.local_id = idx + 1 + #end + #end def decode_port_label(label_value) return {} unless label_value @@ -1397,17 +1403,17 @@ def cmd_test_env(*args) when 'build' cmd_test_env_build(args) when 'list' - print_status("TODO: test_env list") + cmd_test_env_list(args) when 'modules' cmd_test_env_modules(args) when 'stop' - print_status("TODO: test_env stop") + cmd_test_env_stop(args) when 'start' - print_status("TODO: test_env start") + cmd_test_env_start(args) when 'remove' - print_status("TODO: test_env remove") + cmd_test_env_remove(args) when 'remove-all' - print_status("TODO: test_env remove-all") + cmd_test_env_remove_all(args) when 'exec' print_status("TODO: test_env exec") when 'status' @@ -1741,6 +1747,242 @@ def cmd_test_env_status(args) rescue => e print_error("Status check failed: #{e.message}") end + def cmd_test_env_list(_args = []) + targets = self.class.registry.list + + if targets.empty? + print_status("No environments currently tracked.") + return + end + + # Build rows first so fallback mode can reuse them + rows = targets.map do |t| + [ + t.local_id.to_s, + t.container_id.to_s[0..11], + t.module_fullname.to_s, + t.rhost || '127.0.0.1', + t.rport.to_s, + t.status.to_s, + t.env_version.to_s + ] + end + + begin + tbl = Rex::Ui::Text::Table.new( + 'Header' => 'Test Environments', + 'Indent' => 1, + 'Columns' => ['ID', 'Container', 'Module', 'RHOST', 'RPORT', 'Status', 'Version'] + ) + + rows.each { |r| tbl << r } + print_line(tbl.to_s) + rescue NameError + # Rex::Ui::Text::Table not yet loaded in this msfconsole context + print_status("Test Environments") + print_status("=" * 100) + print_status( + ['ID', 'Container', 'Module', 'RHOST', 'RPORT', 'Status', 'Version'] + .map { |h| h.ljust(12) }.join + ) + print_status("-" * 100) + rows.each do |r| + print_status(r.map { |cell| cell.to_s.ljust(12) }.join) + end + end + + print_status("#{targets.length} environment(s) tracked.") + end + + def cmd_test_env_start(args) + if args.empty? + print_error("Usage: test_env start ") + return + end + + # start accepts single ID for safety (per your Week 1 spec) + id = args.first.to_i + target = self.class.registry.get(id) + + unless target + print_error("Environment #{id} not found.") + return + end + + unless target.stopped? + print_warning("Environment #{id} is already #{target.status}.") + return + end + + runtime = self.class.runtime + unless runtime + print_error("No container runtime available.") + return + end + + begin + unless runtime.start(target.container_id) + print_error("Failed to start container for environment #{id}.") + return + end + + # Reconstruct health check configuration + mod = framework.modules.create(target.module_fullname) rescue nil + if mod + raw = mod.send(:module_info)['VulnerableEnvironment'] rescue nil + if raw + env_meta = VulnerableEnvironment.new(raw) + loader = EnvironmentDefinitionLoader.new(Msf::Config.data_directory) + config = loader.resolve(env_meta.definition, target.env_version, + env_meta.profile, env_meta.overrides) rescue nil + + if config && config['health_check'] + # Determine which host port to health-check + primary_port = target.allocated_ports[env_meta.port_mapping.key('RPORT')] + health_port = primary_port || target.allocated_ports.values.first + + begin + HealthManager.new(runtime, target.container_id, + config['health_check'], health_port, self).wait + rescue => e + print_warning("Health check warning after start: #{e.message}") + # Container started; we still mark running but warn user + end + end + end + end + + self.class.registry.update_status(id, :running) + print_good("Environment #{id} started. RPORT=#{target.rport}") + rescue => e + print_error("Error starting environment #{id}: #{e.message}") + end + end + + def cmd_test_env_stop(args) + if args.empty? + print_error("Usage: test_env stop ") + return + end + + ids = parse_id_range(args.first) + if ids.empty? + print_error("No valid IDs provided.") + return + end + + runtime = self.class.runtime + unless runtime + print_error("No container runtime available.") + return + end + + ids.each do |id| + target = self.class.registry.get(id) + unless target + print_error("Environment #{id} not found.") + next + end + + unless target.running? + print_warning("Environment #{id} is already #{target.status}.") + next + end + + begin + if runtime.stop(target.container_id) + self.class.registry.update_status(id, :stopped) + print_good("Environment #{id} stopped.") + else + print_error("Failed to stop container for environment #{id}.") + end + rescue => e + print_error("Error stopping environment #{id}: #{e.message}") + end + end + end + + def cmd_test_env_remove(args) + if args.empty? + print_error("Usage: test_env remove ") + return + end + + ids = parse_id_range(args.first) + if ids.empty? + print_error("No valid IDs provided.") + return + end + + runtime = self.class.runtime + unless runtime + print_error("No container runtime available.") + return + end + + # Resolve all targets FIRST before any mutation + targets = ids.map { |id| [id, self.class.registry.get(id)] }.to_h + + # Validate all exist before attempting any removal + missing = targets.select { |_id, t| t.nil? }.keys + missing.each { |id| print_error("Environment #{id} not found.") } + + valid_ids = targets.reject { |_id, t| t.nil? }.keys + + valid_ids.each do |id| + target = targets[id] + + begin + runtime.stop(target.container_id) if target.running? + rescue => e + print_warning("Stop warning for #{id}: #{e.message}") + end + + begin + runtime.remove(target.container_id) + rescue => e + print_error("Failed to remove container for environment #{id}: #{e.message}") + next # DO NOT purge registry if runtime removal failed + end + + self.class.registry.remove(id) + print_good("Environment #{id} removed.") + end + end + + def cmd_test_env_remove_all(_args = []) + runtime = self.class.runtime + unless runtime + print_error("No container runtime available.") + return + end + + targets = self.class.registry.list + if targets.empty? + print_status("No environments to remove.") + return + end + + print_status("Tearing down #{targets.length} environment(s)...") + + targets.each do |target| + begin + runtime.stop(target.container_id) if target.running? + rescue => e + print_warning("Stop warning for #{target.local_id}: #{e.message}") + end + + begin + runtime.remove(target.container_id) + rescue => e + print_warning("Remove warning for #{target.local_id}: #{e.message}") + end + end + + # Atomic registry reset + self.class.registry.remove_all + print_good("All environments removed.") + end def cmd_test_env_modules(args) print_status("Scanning framework modules for test_env support...") @@ -1848,7 +2090,7 @@ def cmd_test_env_tabs(str, words) if words.length == 2 case words[0] when 'stop', 'start', 'remove', 'exec' - return [] + return self.class.registry.list.map(&:local_id).map(&:to_s) end end @@ -1888,8 +2130,27 @@ def parse_build_args(args) end options end - end + # Parses "1", "1-3", "1,3,5", "1-3,5,7-9" into [1, 2, 3, 5, 7, 8, 9] + def parse_id_range(range_str) + return [] if range_str.nil? || range_str.empty? + + ids = [] + range_str.split(',').each do |part| + part.strip! + if part.include?('-') + start_id, end_id = part.split('-', 2).map(&:to_i) + ids.concat((start_id..end_id).to_a) + else + ids << part.to_i + end + end + ids.uniq.sort + rescue => e + print_error("Invalid ID range format: #{range_str}") + [] + end + end # ===================================================================== # Plugin Lifecycle From 17144ee345f428b320978e26b5aee1169eca3c84 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Sat, 25 Jul 2026 18:56:23 +0300 Subject: [PATCH 03/11] minor fix --- plugins/test_env.rb | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/plugins/test_env.rb b/plugins/test_env.rb index 0764977088d48..63d9a22fbd235 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -962,7 +962,7 @@ def remove_all @store.save(@vuln_env) end - def prune_and_compact(runtime) + def prune(runtime) return unless runtime # PRUNE: remove registry entries for containers that no longer exist @@ -972,16 +972,13 @@ def prune_and_compact(runtime) @vuln_env.remove_target(target.local_id) end end - # NOTE: We do NOT compact IDs here. Sparse IDs are stable references. - # Compaction during batch operations causes identity shift bugs. - #compact_ids! # Renumber after pruning dead entries @store.save(@vuln_env) end def reconstruct_state(runtime) return unless runtime - prune_and_compact(runtime) + prune(runtime) # RECONSTRUCT: add containers found by labels but missing from registry containers = runtime.list(filters: { 'label' => 'msf.vulnenv.managed_by=test_env' }) @@ -1069,13 +1066,6 @@ def running? private - #def compact_ids! - #sorted = @vuln_env.targets.sort_by(&:local_id) - #sorted.each_with_index do |target, idx| - #target.local_id = idx + 1 - #end - #end - def decode_port_label(label_value) return {} unless label_value @@ -1518,7 +1508,7 @@ def cmd_test_env_build(args) end # 10. Prune manually-removed containers and compact IDs - self.class.registry.prune_and_compact(runtime) + self.class.registry.prune(runtime) # Build container labels for cross-session identification instance_id = "msf-#{Socket.gethostname}-#{Process.pid}" From d422d027c1a66833722d511ad8c2720ed405acfb Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Sat, 25 Jul 2026 19:05:05 +0300 Subject: [PATCH 04/11] doc refactor --- docs/test_env/architecture/03-database-schema.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/test_env/architecture/03-database-schema.md b/docs/test_env/architecture/03-database-schema.md index ec225524d3049..5892a09253fbc 100644 --- a/docs/test_env/architecture/03-database-schema.md +++ b/docs/test_env/architecture/03-database-schema.md @@ -208,9 +208,8 @@ Phase 2 replaces purely in-memory storage with **ActiveModel-backed YAML persist The YAML file acts as a **single shared registry** across all msfconsole instances: -- **Global sequential IDs**: All sessions reload the file before allocating, so IDs are sequential globally (1, 2, 3...) regardless of which terminal created them -- **Compaction on removal**: When an environment is removed, IDs are renumbered to close gaps (1, 2, 4 → 1, 2, 3) -- **Pruning on startup**: Dead entries (containers that no longer exist) are automatically removed and IDs compacted before new allocations +- **Sparse monotonic IDs**: IDs are allocated sequentially but never renumbered. Removing environment 2 from `[1, 2, 3]` yields `[1, 3]`, not `[1, 2]`. This matches the Metasploit `sessions` table behavior and prevents identity-shift bugs during batch operations. +- **Pruning on startup**: Dead entries (containers that no longer exist) are automatically removed. IDs are **not** compacted after pruning — sparse IDs remain stable. ### State Reconstruction @@ -219,7 +218,6 @@ On startup, the plugin: 1. Loads the shared YAML registry 2. Queries the runtime for all containers with `label=msf.vulnenv.managed_by=test_env` 3. Prunes registry entries whose `container_id` no longer exists -4. Compacts IDs after pruning -5. Reconstructs any running containers missing from the registry by reading their labels and assigning the next sequential ID +4. Reconstructs any running containers missing from the registry by reading their labels and assigning the next monotonic ID -**Important:** Reconstruction uses `container_id` as the ground truth for matching, not the `env_id` label. The `env_id` label becomes stale after ID compaction and is only used as a historical reference. \ No newline at end of file +**Important:** Reconstruction uses `container_id` as the ground truth for matching, not the `env_id` label. Because IDs are never compacted, `env_id` labels remain stable references, but `container_id` is still the authoritative identifier. \ No newline at end of file From 3e015953233eeb42ef254371118b0d6eeeffc2e5 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Sat, 25 Jul 2026 20:32:01 +0300 Subject: [PATCH 05/11] minor refactor --- plugins/test_env.rb | 1 - 1 file changed, 1 deletion(-) diff --git a/plugins/test_env.rb b/plugins/test_env.rb index 63d9a22fbd235..57d5dec073930 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -945,7 +945,6 @@ def remove(id) end @vuln_env.remove_target(id) - #compact_ids! @store.save(@vuln_env) end From 707044c3a82b518b534ae8686ed9ef7e2304184b Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Sun, 26 Jul 2026 15:18:01 +0300 Subject: [PATCH 06/11] code refactor --- plugins/test_env.rb | 495 ++++++++++++++++++++++++-------------------- 1 file changed, 267 insertions(+), 228 deletions(-) diff --git a/plugins/test_env.rb b/plugins/test_env.rb index 57d5dec073930..5b93febc8d9b8 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -1415,270 +1415,309 @@ def cmd_test_env(*args) end end - def cmd_test_env_build(args) - begin - container_id = nil # Track for cleanup - registered = false # Track whether registration succeeded - - # 1. Preconditions - mod = driver.active_module - unless mod - print_error("No active module. Use 'use ' first.") - return - end +def cmd_test_env_build(args) + container_id = nil + registered = false + runtime = nil + + begin + # Steps 1-3: validate module, metadata, and user arguments + mod, env, options = build_validate_prerequisites(args) + return unless mod && env + + # Steps 4-7: resolve definition, variant, profile, and config + definition_name, variant, profile, config, port_mapping = build_resolve_environment(mod, env, options) + return unless definition_name + + # Step 6: runtime availability + runtime = self.class.runtime + unless runtime + print_error("No container runtime available. Install Docker or Podman.") + return + end - # 2. Read and validate module metadata - env = vulnerable_environment(mod) - unless env - print_error("Module does not define a vulnerable environment configuration.") - return - end + # Step 8: pull image + return unless build_pull_image(runtime, config) - # 3. Parse user arguments - options = parse_build_args(args) + # Step 9: allocate host ports + allocated_ports = build_allocate_ports(runtime, port_mapping, options) - # 4. Extract references from validated Struct - definition_name = env.definition - default_variant = env.default_variant - port_mapping = env.port_mapping + # Steps 10-12: prune registry, build labels, prepare volumes + labels, run_ports, volumes, temp_dirs, env_id = build_prepare_resources( + runtime, config, definition_name, variant, mod, allocated_ports + ) - # 5. Determine variant and profile - variant = options['VARIANT'].to_s.empty? ? default_variant : options['VARIANT'] - profile = options['PROFILE'].to_s.empty? ? env.profile : options['PROFILE'] + # Step 13: launch container + container_id = build_launch_container(runtime, config, run_ports, labels, volumes, definition_name, variant) - unless variant - print_error("No variant specified and module has no default_variant.") - return - end + # verify container actually started + container_info = runtime.inspect(container_id) + unless container_info + print_error("Container started but inspect failed immediately.") + runtime.remove(container_id) + return + end - # 6. Check runtime availability - runtime = self.class.runtime - unless runtime - print_error("No container runtime available. Install Docker or Podman.") - return - end + # check if running (Docker/Podman both use State.Status) + status = container_info.dig('State', 'Status') || container_info.dig('State', 'Running') + if status != 'running' && status != true + error_msg = container_info.dig('State', 'Error') || 'unknown' + print_error("Container failed to start. Status: #{status.inspect}, Error: #{error_msg}") + runtime.remove(container_id) + return + end - # 7. Load and resolve environment definition - loader = EnvironmentDefinitionLoader.new(Msf::Config.data_directory) - config = loader.resolve(definition_name, variant, profile, env.overrides) - - # Validate: every port in module's port_mapping must exist in the environment's exposed ports - resolved_ports = config.fetch('ports', {}).values.map(&:to_i) - port_mapping.keys.each do |container_port| - port_int = container_port.to_i - unless resolved_ports.include?(port_int) - available = resolved_ports.join(', ') - raise "Port mapping mismatch: module maps port #{port_int} but environment '#{definition_name}' (variant '#{variant}', profile '#{profile}') only exposes ports: #{available}" - end - end + print_good("Container started: #{container_id[0..11]}") - print_status("Resolving environment for #{mod.fullname}...") - print_status("Definition: #{definition_name} | Variant: #{variant} | Profile: #{profile}") - print_status("Image: #{config['image']}") + # Step 14: health check BEFORE registering + return unless build_wait_for_health(runtime, container_id, config, port_mapping, allocated_ports) - # 8. Pull the container image - print_status("Pulling image #{config['image']}...") - unless runtime.pull(config['image']) - print_error("Failed to pull image: #{config['image']}") - return - end - print_good("Image pulled successfully.") - - # 9. Allocate ports using PortAllocator - # Pass the runtime so PortAllocator can scan Docker/Podman for - # ports already bound by orphaned containers from previous sessions. - allocator = PortAllocator.new(runtime, self.class.registry.used_ports) - allocated_ports = {} # {container_port => host_port} - user_rport = options['RPORT'] ? options['RPORT'].to_i : nil - # Resolve which container port the user actually wants to override. - # this ensures RPORT=8081 always targets the port mapped to the - # 'RPORT' datastore key, regardless of Ruby hash insertion order. - target_container_port = user_rport ? port_mapping.key('RPORT') : nil - - port_mapping.each do |container_port, ds_option| - preferred = (container_port == target_container_port) ? user_rport : nil - host_port = allocator.allocate(preferred) - - if preferred && host_port != preferred - print_status("Requested port #{preferred} unavailable. Using dynamically allocated port #{host_port}.") - end + # Steps 15-16: register environment in registry + datastore = build_register_environment( + runtime, container_id, mod, variant, config, allocated_ports, port_mapping, temp_dirs, env_id + ) + registered = true - allocated_ports[container_port] = host_port - end + # Step 17: apply datastore to the active module + datastore.each do |key, value| + mod.datastore[key] = value + end - # 10. Prune manually-removed containers and compact IDs - self.class.registry.prune(runtime) - - # Build container labels for cross-session identification - instance_id = "msf-#{Socket.gethostname}-#{Process.pid}" - # reserve the ID first, before starting the container - env_id = self.class.registry.reserve_id - - # now build labels with the GUARANTEED ID - labels = runtime.build_labels( - instance_id: instance_id, - module_fullname: mod.fullname, - env_id: env_id, - version: variant, - ports: allocated_ports - ) + # Step 18: display results to user + build_display_results(env_id, config, datastore) + + rescue PortAllocator::NoPortsAvailable => e + print_error("No available ports: #{e.message}") + rescue => e + # Clean up orphaned container if we created one but failed to register it + if container_id && !registered + begin + runtime.stop(container_id) rescue nil + runtime.remove(container_id) + print_status("Cleaned up orphaned container #{container_id[0..11]}") + rescue => cleanup_err + elog("Failed to cleanup orphaned container: #{cleanup_err.message}") + end + end - # 11. Prepare port mappings for docker run - # Format: {container_port => host_port} for runtime.run - run_ports = {} - allocated_ports.each do |container_port, host_port| - run_ports[container_port] = host_port - end + print_error("test_env build failed: #{e.message}") + elog("test_env build error: #{e.class} - #{e.message}") + elog(e.backtrace.join("\n")) + end +end - # 12. Prepare volumes from config - volumes = {} - temp_dirs = [] +# ------------------------------------------------------------------------- +# Build Phase Helpers +# ------------------------------------------------------------------------- + +# Steps 1-3: Preconditions, metadata validation, and argument parsing. +# Returns [mod, env, options] or [nil, nil, nil] on failure. +def build_validate_prerequisites(args) + mod = driver.active_module + unless mod + print_error("No active module. Use 'use ' first.") + return [nil, nil, nil] + end - if config['volumes'] - config['volumes'].each do |name, vol_cfg| - host_path = vol_cfg['host_path'] - unless host_path # <-- CHANGED: split the || into two lines - host_path = Dir.mktmpdir("test_env_#{name}_") - temp_dirs << host_path # <-- ADD THIS LINE - end - volumes[host_path] = vol_cfg['container_path'] - end - end + env = vulnerable_environment(mod) + unless env + print_error("Module does not define a vulnerable environment configuration.") + return [nil, nil, nil] + end - # 13. Launch the container - print_status("Starting container...") - container_name = "msf-vulnenv-#{definition_name}-#{variant}-#{Time.now.to_f.to_s.delete('.')}" - container_id = runtime.run( - image: config['image'], - ports: run_ports, - labels: labels, - volumes: volumes, - name: container_name - ) + options = parse_build_args(args) + [mod, env, options] +end - # verify container actually started - container_info = runtime.inspect(container_id) - unless container_info - print_error("Container started but inspect failed immediately.") - runtime.remove(container_id) - return - end +# Steps 4-7: Resolve environment definition and validate port mappings. +# Returns [definition_name, variant, profile, config, port_mapping] +# or [nil, nil, nil, nil, nil] when variant is missing. +def build_resolve_environment(mod, env, options) + definition_name = env.definition + default_variant = env.default_variant + port_mapping = env.port_mapping - # check if running (Docker/Podman both use State.Status) - status = container_info.dig('State', 'Status') || container_info.dig('State', 'Running') - if status != 'running' && status != true - # Try to get the error reason - error_msg = container_info.dig('State', 'Error') || 'unknown' - print_error("Container failed to start. Status: #{status.inspect}, Error: #{error_msg}") + variant = options['VARIANT'].to_s.empty? ? default_variant : options['VARIANT'] + profile = options['PROFILE'].to_s.empty? ? env.profile : options['PROFILE'] - # clean up the dead container - runtime.remove(container_id) - return - end + unless variant + print_error("No variant specified and module has no default_variant.") + return [nil, nil, nil, nil, nil] + end - print_good("Container started: #{container_id[0..11]}") + loader = EnvironmentDefinitionLoader.new(Msf::Config.data_directory) + config = loader.resolve(definition_name, variant, profile, env.overrides) - # Determine the host port for health checks. - # If the module maps a port to 'RPORT', use that. Otherwise fall back - # to the first allocated port so health checks don't crash on modules - # that use a different datastore key (e.g., auxiliary scanners). - primary_container_port = port_mapping.key('RPORT') - health_host_port = allocated_ports[primary_container_port] || allocated_ports.values.first + # Validate: every port in module's port_mapping must exist in the environment's exposed ports + resolved_ports = config.fetch('ports', {}).values.map(&:to_i) + port_mapping.keys.each do |container_port| + port_int = container_port.to_i + unless resolved_ports.include?(port_int) + available = resolved_ports.join(', ') + raise "Port mapping mismatch: module maps port #{port_int} but environment '#{definition_name}' (variant '#{variant}', profile '#{profile}') only exposes ports: #{available}" + end + end - # 14. wait for health check BEFORE registering the environment - # If this fails, the container is cleaned up and the environment is NOT tracked - health = config['health_check'] - begin - HealthManager.new(runtime, container_id, health, health_host_port, self).wait - rescue => e - print_error("Health check failed: #{e.message}") + print_status("Resolving environment for #{mod.fullname}...") + print_status("Definition: #{definition_name} | Variant: #{variant} | Profile: #{profile}") + print_status("Image: #{config['image']}") - # stop and remove the unhealthy container so it doesn't leak - begin - runtime.stop(container_id) rescue nil - runtime.remove(container_id) rescue nil - rescue => cleanup_err - elog("Failed to cleanup unhealthy container: #{cleanup_err.message}") - end - return - end + [definition_name, variant, profile, config, port_mapping] +end - # 15. Build datastore from allocated ports and config defaults - datastore = { 'RHOSTS' => '127.0.0.1' } - allocated_ports.each do |container_port, host_port| - ds_option = port_mapping[container_port] - datastore[ds_option] = host_port if ds_option - end +# Step 8: Pull the container image. Returns true on success, false on failure. +def build_pull_image(runtime, config) + print_status("Pulling image #{config['image']}...") + unless runtime.pull(config['image']) + print_error("Failed to pull image: #{config['image']}") + return false + end + print_good("Image pulled successfully.") + true +end - # Apply datastore_defaults from environment definition - if config['datastore_defaults'] - config['datastore_defaults'].each do |key, value| - datastore[key] = value unless datastore.key?(key) # Don't override port mappings - end - end +# Step 9: Allocate free host ports for container bindings. +def build_allocate_ports(runtime, port_mapping, options) + allocator = PortAllocator.new(runtime, self.class.registry.used_ports) + allocated_ports = {} + user_rport = options['RPORT'] ? options['RPORT'].to_i : nil + target_container_port = user_rport ? port_mapping.key('RPORT') : nil - # TODO(Week 8): If module requires payload, auto-set PAYLOAD, LHOST, LPORT + port_mapping.each do |container_port, ds_option| + preferred = (container_port == target_container_port) ? user_rport : nil + host_port = allocator.allocate(preferred) + if preferred && host_port != preferred + print_status("Requested port #{preferred} unavailable. Using dynamically allocated port #{host_port}.") + end - self.class.registry.register_with_id( - env_id: env_id, - container_id: container_id, - module_fullname: mod.fullname, - env_version: variant, - runtime: runtime.name, - image_ref: config['image'], - datastore: datastore, - allocated_ports: allocated_ports, - temp_dirs: temp_dirs - ) - # Labels already contain correct env_id from reserve_id above + allocated_ports[container_port] = host_port + end - registered = true + allocated_ports +end - # 17. Apply datastore to the active module - datastore.each do |key, value| - mod.datastore[key] = value - end +# Steps 10-12: Prune registry, build labels, map ports for runtime, and prepare volumes. +# Returns [labels, run_ports, volumes, temp_dirs, env_id]. +def build_prepare_resources(runtime, config, definition_name, variant, mod, allocated_ports) + self.class.registry.prune(runtime) + + instance_id = "msf-#{Socket.gethostname}-#{Process.pid}" + env_id = self.class.registry.reserve_id + + labels = runtime.build_labels( + instance_id: instance_id, + module_fullname: mod.fullname, + env_id: env_id, + version: variant, + ports: allocated_ports + ) + + run_ports = {} + allocated_ports.each do |container_port, host_port| + run_ports[container_port] = host_port + end + volumes = {} + temp_dirs = [] - # 18. Display results to user - print_good("Environment ready.") - print_status("Environment ID: #{env_id}") - datastore.each do |key, value| - print_status(" #{key.ljust(12)} => #{value}") - end + if config['volumes'] + config['volumes'].each do |name, vol_cfg| + host_path = vol_cfg['host_path'] + unless host_path + host_path = Dir.mktmpdir("test_env_#{name}_") + temp_dirs << host_path + end + volumes[host_path] = vol_cfg['container_path'] + end + end - if config['credentials'] && config['credentials']['default'] - creds = config['credentials']['default'] - print_status(" #{'USERNAME'.ljust(12)} => #{creds['username']}") - print_status(" #{'PASSWORD'.ljust(12)} => #{creds['password']}") - end + [labels, run_ports, volumes, temp_dirs, env_id] +end - env = self.class.registry.get(env_id) - print_status("Suggested: #{env.exploit_command}") +# Step 13: Launch the container. Returns the container_id. +def build_launch_container(runtime, config, run_ports, labels, volumes, definition_name, variant) + print_status("Starting container...") + container_name = "msf-vulnenv-#{definition_name}-#{variant}-#{Time.now.to_f.to_s.delete('.')}" + + runtime.run( + image: config['image'], + ports: run_ports, + labels: labels, + volumes: volumes, + name: container_name + ) +end - rescue PortAllocator::NoPortsAvailable => e - print_error("No available ports: #{e.message}") - rescue => e - # Clean up orphaned container if we created one but failed to register it - if container_id && !registered - begin - # A running container cannot be removed without -f - # we stop first to ensure clean removal - runtime.stop(container_id) rescue nil - runtime.remove(container_id) - print_status("Cleaned up orphaned container #{container_id[0..11]}") - rescue => cleanup_err - elog("Failed to cleanup orphaned container: #{cleanup_err.message}") - end - end +# Step 14: Wait for health check. Returns true if healthy, false otherwise +# (container is already cleaned up on failure). +def build_wait_for_health(runtime, container_id, config, port_mapping, allocated_ports) + primary_container_port = port_mapping.key('RPORT') + health_host_port = allocated_ports[primary_container_port] || allocated_ports.values.first + + health = config['health_check'] + begin + HealthManager.new(runtime, container_id, health, health_host_port, self).wait + rescue => e + print_error("Health check failed: #{e.message}") + begin + runtime.stop(container_id) rescue nil + runtime.remove(container_id) rescue nil + rescue => cleanup_err + elog("Failed to cleanup unhealthy container: #{cleanup_err.message}") + end + return false + end + true +end - print_error("test_env build failed: #{e.message}") - elog("test_env build error: #{e.class} - #{e.message}") - elog(e.backtrace.join("\n")) - end - end +# Steps 15-16: Build datastore, register with registry. +# Returns the constructed datastore hash. +def build_register_environment(runtime, container_id, mod, variant, config, allocated_ports, port_mapping, temp_dirs, env_id) + datastore = { 'RHOSTS' => '127.0.0.1' } + allocated_ports.each do |container_port, host_port| + ds_option = port_mapping[container_port] + datastore[ds_option] = host_port if ds_option + end + + if config['datastore_defaults'] + config['datastore_defaults'].each do |key, value| + datastore[key] = value unless datastore.key?(key) + end + end + + self.class.registry.register_with_id( + env_id: env_id, + container_id: container_id, + module_fullname: mod.fullname, + env_version: variant, + runtime: runtime.name, + image_ref: config['image'], + datastore: datastore, + allocated_ports: allocated_ports, + temp_dirs: temp_dirs + ) + + datastore +end +# Step 18: Display build results and suggested exploit command. +def build_display_results(env_id, config, datastore) + print_good("Environment ready.") + print_status("Environment ID: #{env_id}") + datastore.each do |key, value| + print_status(" #{key.ljust(12)} => #{value}") + end + + if config['credentials'] && config['credentials']['default'] + creds = config['credentials']['default'] + print_status(" #{'USERNAME'.ljust(12)} => #{creds['username']}") + print_status(" #{'PASSWORD'.ljust(12)} => #{creds['password']}") + end + + env = self.class.registry.get(env_id) + print_status("Suggested: #{env.exploit_command}") +end def cmd_test_env_help print_line("Usage: test_env ") print_line From 2d6615fc4bf2cb14bb251ab366d202c7deeb2d21 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Wed, 29 Jul 2026 20:05:58 +0300 Subject: [PATCH 07/11] Inject credentials into datastore --- plugins/test_env.rb | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/plugins/test_env.rb b/plugins/test_env.rb index 5b93febc8d9b8..228ec41e38afd 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -1686,6 +1686,15 @@ def build_register_environment(runtime, container_id, mod, variant, config, allo end end + # Merge credentials into datastore so they are applied to the module, + # stored in the registry, and included in the suggested exploit command. + if config['credentials'] && config['credentials']['default'] + config['credentials']['default'].each do |key, value| + ds_key = key.to_s.upcase + datastore[ds_key] = value unless datastore.key?(ds_key) + end + end + self.class.registry.register_with_id( env_id: env_id, container_id: container_id, @@ -1709,11 +1718,6 @@ def build_display_results(env_id, config, datastore) print_status(" #{key.ljust(12)} => #{value}") end - if config['credentials'] && config['credentials']['default'] - creds = config['credentials']['default'] - print_status(" #{'USERNAME'.ljust(12)} => #{creds['username']}") - print_status(" #{'PASSWORD'.ljust(12)} => #{creds['password']}") - end env = self.class.registry.get(env_id) print_status("Suggested: #{env.exploit_command}") From 2f44f564128f6f58b538420e216e022061aae7e3 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Thu, 30 Jul 2026 00:44:34 +0300 Subject: [PATCH 08/11] fix Podman behavior --- plugins/test_env.rb | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/plugins/test_env.rb b/plugins/test_env.rb index 228ec41e38afd..3f208e74c2541 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -965,9 +965,10 @@ def prune(runtime) return unless runtime # PRUNE: remove registry entries for containers that no longer exist - alive_ids = runtime.list(filters: { 'label' => 'msf.vulnenv.managed_by=test_env' }).map { |c| c['ID'] } + alive_ids = runtime.list(filters: { 'label' => 'msf.vulnenv.managed_by=test_env' }).map { |c| normalize_container_id(c['ID'] || c['Id']) } + @vuln_env = @store.load @vuln_env.targets.each do |target| - unless alive_ids.any? { |id| target.container_id == id } + unless alive_ids.include?(target.container_id) @vuln_env.remove_target(target.local_id) end end @@ -984,10 +985,11 @@ def reconstruct_state(runtime) containers.each do |container| labels = container.dig('Config', 'Labels') || container['Labels'] || {} + container_id = normalize_container_id(container['ID'] || container['Id']) # Identify by container_id, NOT by env_id label (labels are immutable # and become stale after ID compaction) - next if @vuln_env.find_by_container(container['ID']) + next if @vuln_env.find_by_container(container_id) module_fullname = labels['msf.vulnenv.module'] version = labels['msf.vulnenv.version'] @@ -1024,7 +1026,7 @@ def reconstruct_state(runtime) assigned_id = @vuln_env.next_id target = VulnTarget.new( local_id: assigned_id, - container_id: container['ID'], + container_id: container_id, module_fullname: module_fullname, env_version: version, runtime: runtime.name, From 875ab85a11919a45a92efd29874cdb73a8bd6364 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Thu, 30 Jul 2026 03:16:13 +0300 Subject: [PATCH 09/11] Resolved table formatting --- plugins/test_env.rb | 73 ++++++++++----------------------------------- 1 file changed, 16 insertions(+), 57 deletions(-) diff --git a/plugins/test_env.rb b/plugins/test_env.rb index 3f208e74c2541..31cda483d22ce 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -10,6 +10,7 @@ require 'fileutils' require 'timeout' require 'net/http' +require 'rex/text/table' module Msf class Plugin::TestEnv < Msf::Plugin @@ -1789,7 +1790,6 @@ def cmd_test_env_list(_args = []) return end - # Build rows first so fallback mode can reuse them rows = targets.map do |t| [ t.local_id.to_s, @@ -1802,28 +1802,14 @@ def cmd_test_env_list(_args = []) ] end - begin - tbl = Rex::Ui::Text::Table.new( - 'Header' => 'Test Environments', - 'Indent' => 1, - 'Columns' => ['ID', 'Container', 'Module', 'RHOST', 'RPORT', 'Status', 'Version'] - ) + tbl = Rex::Text::Table.new( + 'Header' => 'Test Environments', + 'Indent' => 1, + 'Columns' => ['ID', 'Container', 'Module', 'RHOST', 'RPORT', 'Status', 'Version'] + ) - rows.each { |r| tbl << r } - print_line(tbl.to_s) - rescue NameError - # Rex::Ui::Text::Table not yet loaded in this msfconsole context - print_status("Test Environments") - print_status("=" * 100) - print_status( - ['ID', 'Container', 'Module', 'RHOST', 'RPORT', 'Status', 'Version'] - .map { |h| h.ljust(12) }.join - ) - print_status("-" * 100) - rows.each do |r| - print_status(r.map { |cell| cell.to_s.ljust(12) }.join) - end - end + rows.each { |r| tbl << r } + print_line(tbl.to_s) print_status("#{targets.length} environment(s) tracked.") end @@ -2071,44 +2057,17 @@ def cmd_test_env_modules(args) return end - # Try formatted table output first; fall back to plain text if - # Rex::Ui::Text::Table isn't loaded yet (common in -q mode). - begin - tbl = Rex::Ui::Text::Table.new( - 'Header' => 'Modules with test_env Support', - 'Columns' => ['Module', 'Definition', 'Variant', 'Profile', 'Ports', 'Image'] - ) - - matches.sort_by { |m| m[:fullname] }.each do |m| - tbl << [m[:fullname], m[:definition], m[:variant], m[:profile], m[:ports], m[:image]] - end + tbl = Rex::Text::Table.new( + 'Header' => 'Modules with test_env Support', + 'Columns' => ['Module', 'Definition', 'Variant', 'Profile', 'Ports', 'Image'] + ) - print_line(tbl.to_s) - rescue NameError - print_status("Modules with test_env Support") - print_status("=" * 110) - print_status( - "Module".ljust(50) + - "Definition".ljust(12) + - "Variant".ljust(10) + - "Profile".ljust(10) + - "Ports".ljust(12) + - "Image" - ) - print_status("-" * 110) - - matches.sort_by { |m| m[:fullname] }.each do |m| - print_status( - m[:fullname].ljust(50) + - m[:definition].ljust(12) + - m[:variant].ljust(10) + - m[:profile].ljust(10) + - m[:ports].ljust(12) + - m[:image] - ) - end + matches.sort_by { |m| m[:fullname] }.each do |m| + tbl << [m[:fullname], m[:definition], m[:variant], m[:profile], m[:ports], m[:image]] end + print_line(tbl.to_s) + print_status("Found #{matches.length} module(s) with test_env support (scanned #{scanned} total).") end From 502a661cc22ffaa627bc355b7c7bfd0dfa7f76d0 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Thu, 30 Jul 2026 23:18:44 +0300 Subject: [PATCH 10/11] Automatic provisioning if installation required --- data/vuln_envs/wordpress.yml | 36 +++++++- plugins/test_env.rb | 166 ++++++++++++++++++++++++++++++++--- 2 files changed, 190 insertions(+), 12 deletions(-) diff --git a/data/vuln_envs/wordpress.yml b/data/vuln_envs/wordpress.yml index b983a80e74a3e..e2e5e2d9727bf 100644 --- a/data/vuln_envs/wordpress.yml +++ b/data/vuln_envs/wordpress.yml @@ -19,12 +19,46 @@ shared: datastore_defaults: TARGETURI: / + # Liveness only: confirms Apache/PHP are up and responding. + # Does NOT confirm WordPress is installed - a redirect to + # /wp-admin/install.php is a valid 302 here and is expected + # on first boot, since this image ships with wp-config.php + # pointed at a database but never runs the install step itself. health_check: type: http path: / expected_status: 302 interval: 3 timeout: 2 + retries: 15 + + # One-time setup: drives the WordPress install wizard + # (creates the DB schema + admin account) using the + # credentials defined above. Runs once, after health_check + # passes and before the environment is considered ready. + provision: + type: http_post + path: /wp-admin/install.php?step=2 + body: + weblog_title: "Vulnerable WP" + user_name: "{{ credentials.default.username }}" + admin_password: "{{ credentials.default.password }}" + admin_password2: "{{ credentials.default.password }}" + pw_weak: 1 + admin_email: "admin@example.com" + blog_public: 0 + Submit: "Install WordPress" + run_once: true + + # Confirms provisioning actually succeeded: the login form + # (not the installer) must now be served. + verify: + type: http + path: /wp-login.php + expected_status: 200 + match: "user_login" + interval: 3 + timeout: 2 retries: 10 ci: @@ -41,4 +75,4 @@ shared: profiles: default: - description: Standard vulnerable WordPress + description: Standard vulnerable WordPress \ No newline at end of file diff --git a/plugins/test_env.rb b/plugins/test_env.rb index 31cda483d22ce..53883191fcf08 100644 --- a/plugins/test_env.rb +++ b/plugins/test_env.rb @@ -1180,13 +1180,19 @@ def check_http response = http.request(request) actual_status = response.code.to_i - if actual_status == expected_status - true - else + unless actual_status == expected_status # Diagnostic output: tell the user what actually came back print_status(" Health check returned #{actual_status}, expected #{expected_status}") - false + return false end + + expected_match = @config['match'] + if expected_match && !response.body.to_s.include?(expected_match) + print_status(" Health check got status #{actual_status} but response did not contain expected content") + return false + end + + true rescue Errno::ECONNRESET # TCP connection accepted but HTTP server not yet initialized. # This is a transient "not ready" signal — retry on next attempt. @@ -1228,6 +1234,98 @@ def check_command end end + # ===================================================================== + # Provisioner + # ===================================================================== + # Runs a one-time setup action against a container after it passes its + # health check, but before the environment is considered ready. Needed + # for images (like vulnerable WordPress) that boot with an unconfigured + # app: the process is up and answering HTTP, but there's no database + # schema or admin account yet, so nothing is actually exploitable until + # this runs. + class Provisioner + def initialize(runtime, container_id, provision_config, host_port, dispatcher = nil) + @runtime = runtime + @container_id = container_id + @config = provision_config || {} + @host_port = host_port + @dispatcher = dispatcher + end + + # Returns true if there's nothing to do, or if provisioning succeeds. + # Returns false (does not raise) on failure so the caller decides + # whether that's fatal. + def run(datastore = {}) + return true if @config.empty? + + type = @config['type'] + unless type == 'http_post' + raise "Unknown provision type: #{type.inspect}" + end + + print_status("Provisioning environment...") + + Timeout.timeout(@config['timeout'] || 10) do + post_http(datastore) + end + + print_good("Provisioning request sent.") + true + rescue => e + print_error("Provisioning failed: #{e.class} - #{e.message}") + false + end + + def print_status(msg) + @dispatcher&.print_status(msg) + end + + def print_good(msg) + @dispatcher&.print_good(msg) + end + + def print_error(msg) + @dispatcher&.print_error(msg) + end + + private + + def post_http(datastore) + path = @config['path'] || '/' + uri = URI("http://127.0.0.1:#{@host_port}#{path}") + + http = Net::HTTP.new(uri.host, uri.port) + http.open_timeout = 5 + http.read_timeout = 10 + + request = Net::HTTP::Post.new(uri) + body = resolve_template(@config['body'] || {}, datastore) + request.set_form_data(body) + + response = http.request(request) + + unless response.code.to_i.between?(200, 399) + raise "provision request to #{path} returned #{response.code}" + end + end + + # Resolves "{{ credentials.default.username }}"-style tokens in the + # provision body against the datastore already built for this + # environment (USERNAME/PASSWORD etc. are merged in from + # config['credentials']['default'] before this runs). + def resolve_template(body, datastore) + body.each_with_object({}) do |(key, value), out| + out[key] = value.is_a?(String) ? substitute(value, datastore) : value + end + end + + def substitute(str, datastore) + str.gsub(/\{\{\s*credentials\.default\.(\w+)\s*\}\}/) do + datastore[Regexp.last_match(1).upcase].to_s + end + end + end + # ===================================================================== # Environment Definition Loader # ===================================================================== @@ -1475,9 +1573,21 @@ def cmd_test_env_build(args) # Step 14: health check BEFORE registering return unless build_wait_for_health(runtime, container_id, config, port_mapping, allocated_ports) - # Steps 15-16: register environment in registry - datastore = build_register_environment( - runtime, container_id, mod, variant, config, allocated_ports, port_mapping, temp_dirs, env_id + # Step 15: build the datastore (RHOSTS/RPORT/credentials/etc.) + datastore = build_construct_datastore(config, allocated_ports, port_mapping) + + # Step 15.5: run optional one-time provisioning (e.g. WordPress install + # wizard), then re-verify, before the environment is registered as ready + unless build_provision_environment(runtime, container_id, config, port_mapping, allocated_ports, datastore) + print_error("Provisioning failed; tearing down container.") + runtime.stop(container_id) rescue nil + runtime.remove(container_id) rescue nil + return + end + + # Step 16: register environment in registry + build_register_environment( + runtime, container_id, mod, variant, config, allocated_ports, temp_dirs, env_id, datastore ) registered = true @@ -1674,9 +1784,10 @@ def build_wait_for_health(runtime, container_id, config, port_mapping, allocated true end -# Steps 15-16: Build datastore, register with registry. -# Returns the constructed datastore hash. -def build_register_environment(runtime, container_id, mod, variant, config, allocated_ports, port_mapping, temp_dirs, env_id) +# Step 15: Build the datastore hash (RHOSTS/RPORT/TARGETURI/credentials/etc.) +# Split out from registration so provisioning (below) has USERNAME/PASSWORD +# and the resolved ports available before the environment is registered. +def build_construct_datastore(config, allocated_ports, port_mapping) datastore = { 'RHOSTS' => '127.0.0.1' } allocated_ports.each do |container_port, host_port| ds_option = port_mapping[container_port] @@ -1698,6 +1809,39 @@ def build_register_environment(runtime, container_id, mod, variant, config, allo end end + datastore +end + +# Step 15.5: Run the optional one-time provisioning step (e.g. driving the +# WordPress install wizard) against the primary port, then re-verify with +# an optional 'verify' block. No-op (returns true) if the definition has no +# 'provision' config. On failure, the caller is responsible for cleanup. +def build_provision_environment(runtime, container_id, config, port_mapping, allocated_ports, datastore) + provision = config['provision'] + return true unless provision.is_a?(Hash) && !provision.empty? + + primary_container_port = port_mapping.key('RPORT') + host_port = allocated_ports[primary_container_port] || allocated_ports.values.first + + unless Provisioner.new(runtime, container_id, provision, host_port, self).run(datastore) + return false + end + + verify = config['verify'] + return true unless verify.is_a?(Hash) && !verify.empty? + + begin + HealthManager.new(runtime, container_id, verify, host_port, self).wait + rescue => e + print_error("Post-provision verification failed: #{e.message}") + return false + end + + true +end + +# Step 16: Register the built, provisioned environment with the registry. +def build_register_environment(runtime, container_id, mod, variant, config, allocated_ports, temp_dirs, env_id, datastore) self.class.registry.register_with_id( env_id: env_id, container_id: container_id, @@ -2212,4 +2356,4 @@ def desc 'Automated vulnerable environment provisioning' end end -end +end \ No newline at end of file From 41578c65a92c9f04d46093e706e8063a82e34511 Mon Sep 17 00:00:00 2001 From: Nayeraneru Date: Sat, 1 Aug 2026 17:52:15 +0300 Subject: [PATCH 11/11] update docs --- .../architecture/04-environment-schema.md | 77 ++++++++++++++++++- docs/test_env/reference_modules.md | 7 +- 2 files changed, 80 insertions(+), 4 deletions(-) diff --git a/docs/test_env/architecture/04-environment-schema.md b/docs/test_env/architecture/04-environment-schema.md index 8e9dceeee5199..ca0bd05126b32 100644 --- a/docs/test_env/architecture/04-environment-schema.md +++ b/docs/test_env/architecture/04-environment-schema.md @@ -23,7 +23,7 @@ Ports: {"http"=>8080} Health check type: http ``` -## Directory Structure +## Directory Structure Example ``` data/ @@ -71,6 +71,7 @@ This gives maximum reusability while allowing precise per-module customization. | Different health check endpoint or expected status | **Module override** | Same profile, module overrides `health_check.path` | | Different datastore default | **Module override** | Same profile, module overrides `datastore_defaults.TARGETURI` | | Different credentials | **Module override** | Same profile, module overrides `credentials.default` | +| Service boots unconfigured and needs one-time setup before it's exploitable | **`shared.provision` (+ `verify`)** | WordPress image with no DB schema/admin account until the install wizard is submitted | ## Schema @@ -149,6 +150,7 @@ shared: | `type` | String | Yes | `http`, `tcp`, or `command` | | `path` | String | If type=http | HTTP path to check | | `expected_status` | Integer | No | Default: 200 | +| `match` | String | No | If type=http. Substring the response body must contain. Only checked once `expected_status` matches; use to distinguish "server responding" from "app actually ready" (e.g. an install wizard and a working login page can both return the same status) | | `command` | String | If type=command | Command to execute | | `expected_output` | String | If type=command | Substring to match | | `interval` | Integer | No | Seconds between checks. Default: 5 | @@ -171,6 +173,73 @@ shared: TARGETURI: /script ``` +#### provision (Optional) + +Some images boot with the target process running but the application itself +not yet usable — e.g. a fresh WordPress container serves HTTP but has no +database schema or admin account until the install wizard is submitted. +`provision` describes a one-time setup action to run after `health_check` +passes and before the environment is registered as ready. + +```yaml +shared: + provision: + type: http_post + path: /wp-admin/install.php?step=2 + body: + weblog_title: "Vulnerable WP" + user_name: "{{ credentials.default.username }}" + admin_password: "{{ credentials.default.password }}" + admin_password2: "{{ credentials.default.password }}" + admin_email: "admin@example.com" + blog_public: 0 + Submit: "Install WordPress" + timeout: 10 +``` + +| Key | Type | Required | Description | +|-----|------|----------|-------------| +| `type` | String | Yes | Currently only `http_post` is supported | +| `path` | String | Yes | Request path, sent to the primary mapped port (the container port mapped to `RPORT` in `port_mapping`) | +| `body` | Hash | No | Form fields, sent as `application/x-www-form-urlencoded`. Values may reference `{{ credentials.default. }}`, which is resolved against the built datastore (e.g. `USERNAME`/`PASSWORD`) | +| `timeout` | Integer | No | Seconds to wait for the request. Default: 10 | + +A response status in the `200`–`399` range is treated as success. Anything +else (including a request error or timeout) fails the build; the container +is stopped and removed, matching the existing cleanup behavior for a failed +health check. + +**Architectural decisions:** single stateless request only — no multi-step +flows (e.g. a form load to fetch a CSRF token before the real submit), no +session/cookie carryover between requests, and no non-HTTP provisioning +(e.g. running a setup command inside the container via `runtime.exec`, the +way `health_check`'s `command` type does). Extend `type` as new provisioning +shapes come up rather than building these speculatively. + +#### verify (Optional) + +Re-checks the environment after `provision` runs, confirming the setup +action actually took effect rather than just assuming success from the +HTTP status code. Uses the same shape and checker as `health_check` +(including the `match` field above), so it's commonly used to look for +content that only appears once setup is complete. + +```yaml +shared: + verify: + type: http + path: /wp-login.php + expected_status: 200 + match: "user_login" + interval: 3 + timeout: 2 + retries: 10 +``` + +If `provision` is not defined, `verify` is ignored. If `provision` is +defined but `verify` is not, the environment is registered as ready as +soon as `provision` returns a `200`–`399` response, with no further check. + #### volumes (Optional) ```yaml shared: @@ -255,6 +324,12 @@ When `test_env build` resolves an environment definition, the loader performs a 6. Deep-merge the module's `VulnerableEnvironment['overrides']` (if any). 7. Attach the variant-specific `image`, `version`, and `build_args` from the matching variant in the `variants` list. +If the resolved config includes `provision`, it runs once the container +passes `health_check` and before the environment is registered. If +`verify` is also present, it runs immediately after `provision` succeeds. +A failure at either step aborts the build and tears down the container, +the same as a failed `health_check`. + ### Error Cases | Condition | Error | diff --git a/docs/test_env/reference_modules.md b/docs/test_env/reference_modules.md index 7553c375dfa69..b72280be997c8 100644 --- a/docs/test_env/reference_modules.md +++ b/docs/test_env/reference_modules.md @@ -27,9 +27,10 @@ - **Path:** `exploit/unix/webapp/wp_admin_shell_upload` - **Type:** Web application / CMS - **Port:** 80 -- **Health Check:** HTTP GET `/` expecting 200 -- **Why:** Pre-built vulnerable image exists (`eystsen/vulnerablewordpress`), starts immediately without setup wizard, widely used in security testing +- **Health Check:** Two-stage — see **Provisioning** below. Base `health_check` is HTTP GET `/` expecting **[200, 302]** (the container returns `302` to `/wp-admin/install.php` on first boot; this is a valid "server is up" signal, not a failure). A separate `verify` check confirms the app is actually usable after provisioning: HTTP GET `/wp-login.php` expecting `200` and containing `user_login`. +- **Why:** Pre-built vulnerable image exists (`eystsen/vulnerablewordpress`), widely used in security testing, self-contained (bundles its own MySQL, no external DB link required). - **VulnerableEnvironment Definition:** `wordpress` - **Docker Image:** `eystsen/vulnerablewordpress` - **Credentials:** admin / admin -- **Exploit Context:** Authenticated admin access; uploads PHP shell via theme/plugin editor \ No newline at end of file +- **Exploit Context:** Authenticated admin access; uploads PHP shell via theme/plugin editor +- **Provisioning:** This image does **not** start ready-to-use. The Dockerfile configures `wp-config.php` to point at a `wordpress` database but never creates the schema or an admin account — WordPress boots straight into the install wizard (`/wp-admin/install.php`), and stays there indefinitely with no admin/admin login until the wizard is submitted. `wordpress.yml` now defines a `provision` step (`type: http_post`, submits `install.php?step=2` with the credentials from `credentials.default`) that runs once the base health check passes, followed by the `verify` check above before the environment is registered as ready. See `04-environment-schema.md` for the general `provision`/`verify` schema this relies on. \ No newline at end of file