|
| 1 | +# Manual tracking |
| 2 | + |
| 3 | +Use manual tracking when your framework is not covered by a WildEdge integration, or when you need richer metadata than auto-instrumentation provides. |
| 4 | + |
| 5 | +## When to use it |
| 6 | + |
| 7 | +- Your model class is custom (e.g. a `torch.nn.Module` subclass not loaded via `timm` or `transformers`) |
| 8 | +- You are calling a remote API not yet covered by an integration |
| 9 | +- You want to attach input/output metadata (token counts, image dimensions, confidence scores, etc.) |
| 10 | +- You want to record user feedback tied to a specific inference |
| 11 | + |
| 12 | +## Register your model |
| 13 | + |
| 14 | +Every model needs a handle before you can track events against it. Pass the model object and an explicit `model_id`: |
| 15 | + |
| 16 | +```python |
| 17 | +import wildedge |
| 18 | + |
| 19 | +client = wildedge.WildEdge() # set WILDEDGE_DSN env var |
| 20 | + |
| 21 | +handle = client.register_model( |
| 22 | + my_model, |
| 23 | + model_id="my-org/my-model", |
| 24 | + source="local", # where the weights came from |
| 25 | + family="resnet", # optional model family |
| 26 | + version="1.0.0", # optional version string |
| 27 | + quantization="int8", # optional quantization label |
| 28 | +) |
| 29 | +``` |
| 30 | + |
| 31 | +For remote APIs with no local object to inspect, pass a placeholder: |
| 32 | + |
| 33 | +```python |
| 34 | +handle = client.register_model( |
| 35 | + object(), |
| 36 | + model_id="openai/gpt-4o", |
| 37 | + source="https://api.openai.com", |
| 38 | + family="gpt-4o", |
| 39 | + version="2024-08-06", |
| 40 | +) |
| 41 | +``` |
| 42 | + |
| 43 | +`register_model` is idempotent - calling it twice with the same `model_id` returns the existing handle. |
| 44 | + |
| 45 | +## Track inference |
| 46 | + |
| 47 | +### Decorator |
| 48 | + |
| 49 | +WildEdge times the function and records success or error automatically: |
| 50 | + |
| 51 | +```python |
| 52 | +@wildedge.track(handle, input_type="text", output_type="text") |
| 53 | +def generate(prompt: str) -> str: |
| 54 | + return my_model(prompt) |
| 55 | +``` |
| 56 | + |
| 57 | +### Context manager |
| 58 | + |
| 59 | +Use this when you need access to the result before emitting the event, or when the tracked block is not a standalone function: |
| 60 | + |
| 61 | +```python |
| 62 | +with wildedge.track(handle, input_type="image", output_type="structured"): |
| 63 | + result = my_model(image_tensor) |
| 64 | +``` |
| 65 | + |
| 66 | +### Direct call |
| 67 | + |
| 68 | +For full control over metadata - token counts, confidence scores, generation config, etc.: |
| 69 | + |
| 70 | +```python |
| 71 | +from wildedge import GenerationConfig, GenerationOutputMeta, TextInputMeta |
| 72 | +from wildedge.timing import Timer |
| 73 | + |
| 74 | +with Timer() as t: |
| 75 | + result = my_model(prompt) |
| 76 | + |
| 77 | +inference_id = handle.track_inference( |
| 78 | + duration_ms=t.elapsed_ms, |
| 79 | + input_modality="text", |
| 80 | + output_modality="generation", |
| 81 | + success=True, |
| 82 | + input_meta=TextInputMeta( |
| 83 | + token_count=len(prompt.split()), |
| 84 | + prompt_type="chat", |
| 85 | + ), |
| 86 | + output_meta=GenerationOutputMeta( |
| 87 | + tokens_in=input_tokens, |
| 88 | + tokens_out=output_tokens, |
| 89 | + tokens_per_second=round(output_tokens / t.elapsed_ms * 1000, 1), |
| 90 | + stop_reason="stop", |
| 91 | + ), |
| 92 | + generation_config=GenerationConfig( |
| 93 | + temperature=0.7, |
| 94 | + max_tokens=512, |
| 95 | + ), |
| 96 | +) |
| 97 | +``` |
| 98 | + |
| 99 | +`track_inference` returns an `inference_id` string you can use to attach feedback later. |
| 100 | + |
| 101 | +## Input and output metadata |
| 102 | + |
| 103 | +All fields are optional. |
| 104 | + |
| 105 | +### Text input (`TextInputMeta`) |
| 106 | + |
| 107 | +| Field | Type | Description | |
| 108 | +|---|---|---| |
| 109 | +| `char_count` | `int` | Character count | |
| 110 | +| `word_count` | `int` | Word count | |
| 111 | +| `token_count` | `int` | Token count | |
| 112 | +| `language` | `str` | BCP-47 language code | |
| 113 | +| `prompt_type` | `str` | e.g. `"chat"`, `"completion"`, `"instruct"` | |
| 114 | +| `turn_index` | `int` | Position in a multi-turn conversation | |
| 115 | +| `contains_code` | `bool` | Whether the input contains code | |
| 116 | + |
| 117 | +### Image input (`ImageInputMeta`) |
| 118 | + |
| 119 | +| Field | Type | Description | |
| 120 | +|---|---|---| |
| 121 | +| `width` | `int` | Pixel width | |
| 122 | +| `height` | `int` | Pixel height | |
| 123 | +| `channels` | `int` | Channel count | |
| 124 | +| `format` | `str` | e.g. `"jpeg"`, `"png"`, `"rgb"` | |
| 125 | +| `source` | `str` | e.g. `"camera"`, `"file"`, `"stream"` | |
| 126 | + |
| 127 | +### Audio input (`AudioInputMeta`) |
| 128 | + |
| 129 | +| Field | Type | Description | |
| 130 | +|---|---|---| |
| 131 | +| `duration_ms` | `int` | Audio length | |
| 132 | +| `sample_rate` | `int` | Hz | |
| 133 | +| `channels` | `int` | Channel count | |
| 134 | +| `format` | `str` | e.g. `"wav"`, `"mp3"` | |
| 135 | +| `is_streaming` | `bool` | Live stream vs pre-recorded | |
| 136 | + |
| 137 | +### Generation output (`GenerationOutputMeta`) |
| 138 | + |
| 139 | +| Field | Type | Description | |
| 140 | +|---|---|---| |
| 141 | +| `tokens_in` | `int` | Prompt token count | |
| 142 | +| `tokens_out` | `int` | Completion token count | |
| 143 | +| `tokens_per_second` | `float` | Generation throughput | |
| 144 | +| `cached_input_tokens` | `int` | Tokens served from KV cache | |
| 145 | +| `reasoning_tokens_out` | `int` | Reasoning/thinking tokens (e.g. o1) | |
| 146 | +| `time_to_first_token_ms` | `int` | TTFT latency | |
| 147 | +| `stop_reason` | `str` | e.g. `"stop"`, `"length"`, `"content_filter"` | |
| 148 | + |
| 149 | +### Classification output (`ClassificationOutputMeta`) |
| 150 | + |
| 151 | +| Field | Type | Description | |
| 152 | +|---|---|---| |
| 153 | +| `num_predictions` | `int` | Number of predictions returned | |
| 154 | +| `avg_confidence` | `float` | Mean confidence across predictions | |
| 155 | +| `top_k` | `list[TopKPrediction]` | Top-k labels with confidence scores | |
| 156 | + |
| 157 | +### Detection output (`DetectionOutputMeta`) |
| 158 | + |
| 159 | +| Field | Type | Description | |
| 160 | +|---|---|---| |
| 161 | +| `num_predictions` | `int` | Number of detections | |
| 162 | +| `avg_confidence` | `float` | Mean confidence | |
| 163 | +| `top_k` | `list[TopKPrediction]` | Top detections with bounding boxes | |
| 164 | + |
| 165 | +### Embedding output (`EmbeddingOutputMeta`) |
| 166 | + |
| 167 | +| Field | Type | Description | |
| 168 | +|---|---|---| |
| 169 | +| `dimensions` | `int` | Embedding vector size | |
| 170 | + |
| 171 | +## Track errors |
| 172 | + |
| 173 | +```python |
| 174 | +try: |
| 175 | + result = my_model(input) |
| 176 | +except Exception as exc: |
| 177 | + handle.track_error( |
| 178 | + error_code="UNKNOWN", |
| 179 | + error_message=str(exc), |
| 180 | + ) |
| 181 | + raise |
| 182 | +``` |
| 183 | + |
| 184 | +`wildedge.track` (decorator and context manager) captures errors automatically when `capture_errors=True` (the default). |
| 185 | + |
| 186 | +## Track feedback |
| 187 | + |
| 188 | +Link user or automated feedback to a specific inference via the `inference_id` returned by `track_inference`: |
| 189 | + |
| 190 | +```python |
| 191 | +from wildedge import FeedbackType |
| 192 | + |
| 193 | +inference_id = handle.track_inference(duration_ms=..., ...) |
| 194 | + |
| 195 | +# Later, when feedback is available |
| 196 | +handle.track_feedback(inference_id, FeedbackType.THUMBS_UP) |
| 197 | +``` |
| 198 | + |
| 199 | +If you always want to attach feedback to the most recent inference on a handle, use the shorthand: |
| 200 | + |
| 201 | +```python |
| 202 | +handle.feedback(FeedbackType.THUMBS_DOWN) |
| 203 | +``` |
| 204 | + |
| 205 | +`FeedbackType` values: `THUMBS_UP`, `THUMBS_DOWN`. |
0 commit comments