Add provider-agnostic training observability with optional W&B backend

[iamdarshg]

Motivation

• Provide explicit run lifecycle hooks and per-step observability for training so core metrics can be exported to external systems.
• Make W&B optional so local/test runs remain dependency-free and safe to run without external telemetry.
• Collect basic GPU telemetry and core training metrics (loss, LR, grad norm, throughput, step time, eval metrics) in a fault-tolerant way for improved debugging and monitoring.

Description

• Added a new provider-agnostic adapter module at better_ai/monitoring/observability.py implementing BaseObservabilityBackend, NoOpObservabilityBackend, WandBObservabilityBackend, ObservabilityConfig, ObservabilityAdapter, and a collect_gpu_stats() helper.
• Implemented an optional W&B backend that activates only when configured (via config or env) and otherwise falls back to a no-op backend so logging becomes a safe no-op.
• Wired observability into EnhancedMoETrainer by calling ObservabilityAdapter.from_config(config) in initialization, invoking start_run() at training start, logging per-step metrics (loss, aux loss, lr, grad_norm, tokens_per_sec, step_time_sec, epoch) augmented with collect_gpu_stats(), and calling finish_run() on interrupted/failed/completed flows; also emit eval/train_loss from the HTSR monitor step.
• Exported observability symbols from better_ai.monitoring and added README documentation with environment variable and config examples for BETTER_AI_OBSERVABILITY_BACKEND, BETTER_AI_RUN_NAME, WANDB_PROJECT, WANDB_ENTITY, and WANDB_MODE.

Testing

• Added better_ai/tests/test_observability.py which verifies that the adapter defaults to the no-op backend when disabled and that collect_gpu_stats() contains the expected keys.
• Ran python -m pytest better_ai/tests/test_observability.py -q and the test suite passed (3 passed).

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 452941bc79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Remove per-step nvidia-smi subprocess in hot path

train() now calls collect_gpu_stats() on every optimization step, and that helper shells out to nvidia-smi each time with a 1s timeout. On GPU runs this adds a process spawn in the tight training loop (and can stall for the timeout when nvidia-smi is slow/unavailable), which can materially reduce training throughput and distort step-time metrics; this should be sampled less frequently or collected asynchronously/cached.

Useful? React with 👍 / 👎.