diff --git a/Weaviate_monitoring/Readme.md b/Weaviate_monitoring/Readme.md new file mode 100644 index 00000000..c0194fc1 --- /dev/null +++ b/Weaviate_monitoring/Readme.md @@ -0,0 +1,168 @@ +# Weaviate Monitoring +Weaviate is an open-source vector database that stores and searches embeddings for AI applications, such as semantic search and Retrieval-Augmented Generation (RAG). It exposes both a REST/GraphQL API and Prometheus-style metrics that this plugin uses to report on cluster health, storage, and vector index performance. + +## Prerequisites + +- Download and install the latest version of the [Site24x7 Server Monitoring agent](https://www.site24x7.com/help/admin/adding-a-monitor/linux-server-monitoring.html#add-linux-server-monitor) on the server where you plan to run the plugin. +- Weaviate must be running with Prometheus monitoring enabled (`PROMETHEUS_MONITORING_ENABLED: "true"` in its environment/docker-compose configuration), so the metrics endpoint (default port `2112`) is available. + +## Authentication Setup + +Unlike RabbitMQ, Weaviate does not ship with a built-in user/permission system for its REST API. Depending on how your Weaviate instance is secured, choose one of the following: + +**Anonymous Access (default for local/dev setups)** + +If your Weaviate instance has `AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true"` set, no credentials are required. Leave `api_key` blank in the plugin configuration. + +**API Key Authentication** + +If your Weaviate instance has API key authentication enabled, generate or obtain an API key with read access and provide it as `api_key`. The plugin sends it as a `Bearer` token in the `Authorization` header. + +In both cases, the monitoring credentials only require **read-only** access. The plugin only retrieves metrics and metadata from Weaviate and never creates, modifies, or deletes any data. + +## Plugin Installation + +#### Linux + +- Create a directory named `weaviate`. + + mkdir weaviate + cd weaviate/ + +- Place the following files under the `weaviate` directory: + + Weaviate_Monitoring.py + Weaviate_Monitoring.cfg + +- Execute the below command with appropriate arguments to check for the valid json output: + + python3 Weaviate_Monitoring.py --host 'localhost' --port '8080' --metrics_port '2112' --api_key '' --ssl 'false' --ssl_verify 'true' + +- Provide your Weaviate configurations in the `Weaviate_Monitoring.cfg` file: + + [global_configurations] + use_agent_python=1 + + [weaviate] + host = "localhost" + port = "8080" + metrics_port = "2112" + api_key = "" + ssl = "false" + ssl_verify = "true" + +- Move the directory `weaviate` under the Site24x7 Linux Agent plugin directory: + + mv weaviate /opt/site24x7/monagent/plugins/ + +#### Windows + +- Create a directory named `weaviate`. + +- Place the files `Weaviate_Monitoring.py` and `Weaviate_Monitoring.cfg` under the `weaviate` directory. + +- Since it's a Python plugin, to run the plugin in a Windows server please follow the steps in [this link](https://support.site24x7.com/portal/en/kb/articles/run-python-plugin-scripts-in-windows-servers). + +- Install the required Python packages: + + pip install requests + +- Execute the below command with appropriate arguments in cmd to check for the valid json output: + + python Weaviate_Monitoring.py --host 'localhost' --port '8080' --metrics_port '2112' --api_key '' --ssl 'false' --ssl_verify 'true' + +- Provide your Weaviate configurations in the `Weaviate_Monitoring.cfg` file: + + [weaviate] + host = "localhost" + port = "8080" + metrics_port = "2112" + api_key = "" + ssl = "false" + ssl_verify = "true" + +- Move the folder `weaviate` under the Site24x7 Windows Agent plugin directory: + + C:\Program Files (x86)\Site24x7\WinAgent\monitoring\Plugins + +The agent will automatically execute the plugin within five minutes and user can see the plugin monitor under Site24x7 > Plugins > Plugin Integrations. + +## Supported Metrics + +### Summary + +| Name | Description | Impact on Weaviate | +|------|-------------|--------------------| +| Batch Duration | Cumulative time spent processing batch insert operations | Higher values indicate slower bulk data ingestion. | +| Batch Delete Duration | Cumulative time spent processing batch delete operations | High values may indicate expensive delete operations or storage cleanup overhead. | +| Object Duration | Cumulative time spent processing single-object operations | Reflects the performance of individual object insert, update, and delete requests. | +| Async Operations Running | Number of asynchronous background operations currently running | A consistently high value may indicate background processing backlog. | +| Startup Duration | Time taken for the Weaviate node to complete startup | Longer startup times may indicate large datasets or slower storage initialization. | +| Response Time | Time taken for the REST readiness check (`/v1/.well-known/ready`) to respond | Higher response times indicate slower API availability and can impact application responsiveness. | +| GraphQL Response Time | Time taken for a GraphQL introspection query against `/v1/graphql` to respond | Higher values indicate slower query execution and may affect search performance. | +| CPU Percent | Current CPU utilization of the Weaviate process as a percentage of the available CPU cores | Sustained high CPU usage can increase query latency and reduce overall throughput. | +| Memory Usage | Resident memory currently used by the Weaviate process | High memory usage may lead to swapping, reduced performance, or out-of-memory failures. | +| Requests Total | Total number of API requests handled by this Weaviate instance | Reflects workload on the server and helps identify traffic spikes. | +| Open File Descriptors | Number of file descriptors currently open by the Weaviate process | Values approaching the operating system limit may prevent new files or network connections from being opened. | +| Total Shard Count | Total number of shards across all nodes in the cluster | Indicates how data is partitioned for scalability and load distribution. | + +### Vector Index Performance + +| Name | Description | Impact on Weaviate | +|------|-------------|--------------------| +| Vector Index Size | Number of vectors currently stored in the vector index | Larger indexes require more memory and may increase search time. | +| Vector Index Operations | Total number of operations performed against the vector index | High values indicate increased indexing or search activity. | +| Vector Index Duration | Cumulative time spent performing vector index operations | Rapid growth may indicate slower indexing or vector search performance. | +| Vector Index Maintenance | Cumulative time spent on background vector index maintenance tasks | Increased maintenance activity can temporarily consume CPU and I/O resources. | +| Vector Index Tombstones | Number of tombstoned (deleted but not yet cleaned up) entries in the vector index | Excessive tombstones can reduce search efficiency until cleanup is completed. | +| Vector Index Tombstone Threads | Number of background threads currently cleaning up index tombstones | Indicates background cleanup activity after delete operations. | + +### LSM(Log-Structured-Merge) Storage + +| Name | Description | Impact on Weaviate | +|------|-------------|--------------------| +| LSM Active Segments | Number of currently active segments in the LSM storage engine | A large number of active segments can increase disk reads and query latency. | +| LSM Segment Count | Total number of segments in the LSM store | High segment counts may indicate pending compaction and additional storage overhead. | +| LSM Segment Size | Total on-disk size of LSM segments | Reflects disk space consumed by stored data. | +| LSM Bloom Filter Duration | Cumulative time spent evaluating bloom filters during LSM lookups | Higher values may indicate increased storage lookup overhead. | + +### Cluster Topology + +### Collection,Object and NOde count + +| Name | Description | Impact on Weaviate | +|------|-------------|--------------------| +| Collection Count | Number of collections (classes) defined in the Weaviate schema | Indicates the number of datasets managed by Weaviate. | +| Object Count | Total number of objects stored across all nodes | Reflects database size and storage growth over time. | +| Node Count | Number of nodes in the Weaviate cluster | Helps monitor cluster availability and scalability. | + +#### Collection Details + +| Name | Description | Impact on Weaviate | +|------|-------------|--------------------| +| Name | Name of the collection (class) in the Weaviate schema | Identifies the collection being monitored and used for storing related objects. | +| Vectorizer | Vectorizer module configured for the collection (`none` if vectorization is handled externally) | Determines how vectors are generated and affects indexing and semantic search behavior. | + +#### Node Details + +| Name | Description | Impact on Weaviate | +|------|-------------|--------------------| +| Name | Unique identifier for the node in the Weaviate cluster | Identifies the node for monitoring, diagnostics, and cluster management. | +| Shard Count | Number of shards hosted on the node | Indicates how data is distributed and balanced across cluster nodes. | +| Object Count | Total number of objects stored on the node | Helps monitor data distribution and identify storage imbalance. | + +#### Shard Details + +| Name | Description | Impact on Weaviate | +|------|-------------|--------------------| +| Name | Unique identifier of the shard | Identifies the specific shard being monitored. | +| Class | Collection (class) to which the shard belongs | Indicates which collection's data is stored in the shard. | +| Node | Node currently hosting the shard | Shows the physical location of the shard within the cluster. | +| Object Count | Number of objects stored in the shard | Helps monitor shard utilization and detect uneven data distribution. | + +## Sample Images +image +image + + + diff --git a/Weaviate_monitoring/Weaviate_monitoring.cfg b/Weaviate_monitoring/Weaviate_monitoring.cfg new file mode 100644 index 00000000..870af784 --- /dev/null +++ b/Weaviate_monitoring/Weaviate_monitoring.cfg @@ -0,0 +1,10 @@ +[global_configurations] +redirect_output=1 + +[instance] +host="localhost" +port="8080" +metrics_port="2112" +api_key="" +ssl="false" +ssl_verify="true" \ No newline at end of file diff --git a/Weaviate_monitoring/Weaviate_monitoring.py b/Weaviate_monitoring/Weaviate_monitoring.py new file mode 100644 index 00000000..bc21f071 --- /dev/null +++ b/Weaviate_monitoring/Weaviate_monitoring.py @@ -0,0 +1,326 @@ +import json +import time +import sys +import os +import tempfile + + + + +# Initialize variables securely +SSL_VERIFY = True + +PLUGIN_VERSION = 1 +HEARTBEAT = "true" + +# Base units definition to ensure Site24x7 always has a schema to read +UNITS = { + "Response Time": "ms", + "GraphQL Response Time": "ms", + "CPU Percent": "%", + "Memory Usage": "MB", + "Batch Duration": "ms", + "Batch Delete Duration": "ms", + "Object Duration": "ms", + "Segment Size": "MB", + "Bloom Filter Duration": "ms", + "Duration": "ms", + "Maintenance": "ms", + "Startup Duration": "ms", +} + +TABS = { + "Vector Index": { + "order": 1, + "tablist": [ + "Operations", + "Size", + "Duration", + "Maintenance", + "Tombstones", + "Tombstone Threads", + ] + }, + "LSM Storage": { + "order": 2, + "tablist": [ + "Active Segments", + "Segment Count", + "Segment Size", + "Bloom Filter Duration", + ] + }, + "Cluster Topology": { + "order": 3, + "tablist": [ + "Collection Count", + "Object Count", + "Collection Details", + "Node Details", + "Total Shard Count", + "Shard Details" + ] + } +} + +def get_error_payload(error_message): + """Returns a valid JSON payload even if the script crashes completely.""" + return { + "plugin_version": PLUGIN_VERSION, + "heartbeat_required": HEARTBEAT, + "status": 0, + "msg": error_message, + "units": UNITS + } + +# Safely attempt to import requests to catch Local System environment issues +try: + import requests +except ImportError: + print(json.dumps(get_error_payload("Python 'requests' module not found. Install globally using Admin PowerShell: python -m pip install requests"))) + sys.exit(0) + +REQUEST_TIMEOUT = 5 + +def auth_headers(): + if WEAVIATE_API_KEY: + return {"Authorization": f"Bearer {WEAVIATE_API_KEY}"} + return {} + +def fetch_json(url): + try: + response = requests.get(url, headers=auth_headers(), verify=globals().get("SSL_VERIFY", True), timeout=REQUEST_TIMEOUT) + response.raise_for_status() + return response.json() + except (requests.exceptions.RequestException, ValueError): + return None + +def fetch_text(url): + try: + response = requests.get(url, headers=auth_headers(), verify=globals().get("SSL_VERIFY", True), timeout=REQUEST_TIMEOUT) + response.raise_for_status() + return response.text + except requests.exceptions.RequestException: + return None + +def timed_get(url): + try: + start = time.time() + response = requests.get(url, headers=auth_headers(), verify=globals().get("SSL_VERIFY", True), timeout=REQUEST_TIMEOUT) + elapsed_ms = round((time.time() - start) * 1000, 2) + return response.status_code == 200, elapsed_ms + except requests.exceptions.RequestException: + return False, 0 + +def timed_post(url, body): + try: + start = time.time() + response = requests.post(url, json=body, headers=auth_headers(), verify=globals().get("SSL_VERIFY", True), timeout=REQUEST_TIMEOUT) + elapsed_ms = round((time.time() - start) * 1000, 2) + return response.status_code == 200, elapsed_ms + except requests.exceptions.RequestException: + return False, 0 + +def parse_prometheus(raw_text): + totals = {} + if not raw_text: + return totals + for line in raw_text.splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + parts = line.rsplit(" ", 1) + if len(parts) != 2: + continue + name_with_labels, value_str = parts + name = name_with_labels.split("{")[0] + if name.endswith("_bucket"): + continue + try: + value = float(value_str) + except ValueError: + continue + totals[name] = totals.get(name, 0.0) + value + return totals + +def get_metric(totals, exact_name): + return totals.get(exact_name, 0.0) + +def bytes_to_mb(value): + if not value: return 0 + return round(value / (1024 * 1024), 2) + +def to_number(value): + return value if isinstance(value, (int, float)) else 0 + +def build_collection_details(schema): + classes = (schema or {}).get("classes", []) or [] + details = [] + for cls in classes: + cls_name = cls.get("class") + if cls_name: + vectorizer = cls.get("vectorizer", "-") + # Site24x7 tables only allow string values in 'name' + details.append({ + "name": f"{cls_name} (vectorizer: {vectorizer})" + }) + return details + +def build_node_and_shard_details(nodes_data): + nodes = (nodes_data or {}).get("nodes", []) or [] + node_details, shard_details = [], [] + for node in nodes: + stats = node.get("stats") or {} + node_name = node.get("name") + if not node_name: + continue + + node_details.append({ + "name": node_name, + "Shard_Count": to_number(stats.get("shardCount")), + "Object_Count": to_number(stats.get("objectCount")) + }) + + for shard in node.get("shards", []) or []: + shard_name = shard.get("name") + if not shard_name: + continue + + cls_name = shard.get("class", "Unknown") + # Combine class, shard name, and node into the 'name' field + shard_details.append({ + "name": f"{cls_name} / {shard_name} ({node_name})", + "Object_Count": to_number(shard.get("objectCount")) + }) + + return node_details, shard_details +CPU_STATE_FILE = os.path.join(tempfile.gettempdir(), "weaviate_cpu_state.json") + +def get_cpu_utilization_percent(current_cpu_seconds, cores=1): + """ + Converts a cumulative Prometheus counter (process_cpu_seconds_total) + into an instantaneous CPU utilization percentage, by comparing it + against the value from the previous plugin run. + """ + now = time.time() + previous_time = None + previous_cpu = None + + try: + with open(CPU_STATE_FILE, "r") as f: + state = json.load(f) + previous_time = state.get("timestamp") + previous_cpu = state.get("cpu_seconds") + except (FileNotFoundError, ValueError, OSError): + pass + + try: + with open(CPU_STATE_FILE, "w") as f: + json.dump({"timestamp": now, "cpu_seconds": current_cpu_seconds}, f) + except OSError: + pass + + if previous_time is None or previous_cpu is None: + return 0 # first run — no prior data point to diff against + + delta_time = now - previous_time + delta_cpu = current_cpu_seconds - previous_cpu + + if delta_cpu < 0 or delta_time <= 0: + return 0 # counter reset (process restarted) or no time elapsed + + cores = cores if cores and cores > 0 else 1 + return round(((delta_cpu / delta_time) * 100) / cores, 2) + +def metricCollector(): + ready_ok, response_time_ms = timed_get(f"{WEAVIATE_SERVER}/v1/.well-known/ready") + _, graphql_time_ms = timed_post(f"{WEAVIATE_SERVER}/v1/graphql", {"query": "{ __schema { queryType { name } } }"}) + raw_metrics = fetch_text(WEAVIATE_METRICS_URL) + totals = parse_prometheus(raw_metrics) + cpu_seconds = get_metric(totals, "process_cpu_seconds_total") + cpu_percent = get_cpu_utilization_percent(cpu_seconds, cores=os.cpu_count() or 1) + schema = fetch_json(f"{WEAVIATE_SERVER}/v1/schema") + nodes_data = fetch_json(f"{WEAVIATE_SERVER}/v1/nodes?output=verbose") + nodes = (nodes_data or {}).get("nodes", []) or [] + + object_count = sum(to_number((n.get("stats") or {}).get("objectCount")) for n in nodes) + shard_count = sum(to_number((n.get("stats") or {}).get("shardCount")) for n in nodes) + collection_count = len((schema or {}).get("classes", []) or []) + node_details, shard_details = build_node_and_shard_details(nodes_data) + collection_details = build_collection_details(schema) + + return { + "plugin_version": PLUGIN_VERSION, + "heartbeat_required": HEARTBEAT, + "status": 1 if ready_ok else 0, + "Response Time": response_time_ms, + "GraphQL Response Time": graphql_time_ms, + "CPU Percent": cpu_percent, + "Memory Usage": bytes_to_mb(get_metric(totals, "process_resident_memory_bytes")), + "Requests Total": get_metric(totals, "requests_total"), + "Open File Descriptors": get_metric(totals, "process_open_fds"), + "Collection Count": collection_count, + "Object Count": object_count, + "Node Count": len(nodes), + "Total Shard Count": shard_count, + "Batch Duration": get_metric(totals, "batch_durations_ms_sum"), + "Batch Delete Duration": get_metric(totals, "batch_delete_durations_ms_sum"), + "Object Duration": get_metric(totals, "objects_durations_ms_sum"), + #LSM params + "Active Segments": get_metric(totals, "lsm_active_segments"), + "Segment Count": get_metric(totals, "lsm_segment_count"), + "Segment Size": bytes_to_mb(get_metric(totals, "lsm_segment_size")), + "Bloom Filter Duration": get_metric(totals, "lsm_bloom_filter_duration_ms_sum"), + #Vector Index parameters + "Size": get_metric(totals, "vector_index_size"), + "Tombstones": get_metric(totals, "vector_index_tombstones"), + "Tombstone Threads": get_metric(totals, "vector_index_tombstone_cleanup_threads"), + "Operations": get_metric(totals, "vector_index_operations"), + "Duration": get_metric(totals, "vector_index_durations_ms_sum"), + "Maintenance": get_metric(totals, "vector_index_maintenance_durations_ms_sum"), + #Ends here + "Async Operations Running": get_metric(totals, "async_operations_running"), + "Startup Duration": get_metric(totals, "startup_durations_ms_sum"), + "Collection Details": collection_details, + "Node Details": node_details, + "Shard Details": shard_details, + "units": UNITS, + "tabs": TABS, + } + +def run(param): + global WEAVIATE_HOST, WEAVIATE_PORT, WEAVIATE_METRICS_PORT, WEAVIATE_API_KEY, WEAVIATE_SERVER, WEAVIATE_METRICS_URL + global SSL_VERIFY + + try: + WEAVIATE_HOST = param.get("host", "localhost") + WEAVIATE_PORT = param.get("port", "8080") + WEAVIATE_METRICS_PORT = param.get("metrics_port", "2112") + WEAVIATE_API_KEY = param.get("api_key", "") + ssl_enabled = str(param.get("ssl", "false")).lower() == "true" + + verify_val = param.get("ssl_verify", "true") + if verify_val.lower() == "true": SSL_VERIFY = True + elif verify_val.lower() == "false": SSL_VERIFY = False + else: SSL_VERIFY = verify_val + + protocol = "https" if ssl_enabled else "http" + WEAVIATE_SERVER = f"{protocol}://{WEAVIATE_HOST}:{WEAVIATE_PORT}" + WEAVIATE_METRICS_URL = f"http://{WEAVIATE_HOST}:{WEAVIATE_METRICS_PORT}/metrics" + + return metricCollector() + except Exception as e: + return get_error_payload(f"Plugin Execution Error: {str(e)}") + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--host', help="Host Name", nargs='?', default='localhost') + parser.add_argument('--port', help="REST API Port", nargs='?', default="8080") + parser.add_argument('--metrics_port', help="Prometheus Metrics Port", nargs='?', default="2112") + parser.add_argument('--api_key', help="API Key (if authentication is enabled)", default="") + parser.add_argument('--ssl', help="Use SSL/HTTPS", default="false") + parser.add_argument('--ssl_verify', help="SSL Verification", default="true") + args = parser.parse_args() + + print(json.dumps(run(vars(args)))) \ No newline at end of file diff --git a/nginx/README.md b/nginx/README.md index 305afcc2..e5fcb8d1 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -2,45 +2,70 @@ ## Nginx plugin installer -On Linux servers, execute the command below in the terminal to run an installer that checks the prerequisites and installs the plugin. +### On Windows Servers +Open a PowerShell terminal as Administrator and execute the commands below to set up your directory layout, download the script files, and configure prerequisites. +```powershell +New-Item -ItemType Directory -Force -Path "C:\nginx-1.30.4\plugins\nginx" +cd "C:\nginx-1.30.4\plugins\nginx" +``` +### On Linux Servers +Execute the command below in the terminal to run an installer that checks the prerequisites and installs the plugin. ```bash wget https://raw.githubusercontent.com/site24x7/plugins/master/nginx/installer/Site24x7NginxPluginInstaller.sh && sudo bash Site24x7NginxPluginInstaller.sh ``` ## Prerequisites -- Download and install the latest version of the [Site24x7 Linux agent](https://www.site24x7.com/app/client#/admin/inventory/add-monitor) in the server where you plan to run the plugin. +- Download and install the latest version of the [Site24x7 Windows Agent](https://site24x7.com) or [Site24x7 Linux Agent](https://site24x7.com) on the server where you plan to run the plugin. - Python version 3 or higher. -#### Enable nginx_status to get metrics - +#### Enable nginx_status and Advanced Logging to get metrics - -1. Open terminal and run the following command to open NGINX server configuration file. +1. Open your terminal and run your text editor to modify the NGINX configuration file. + **Windows:** + ```powershell + notepad C:\nginx-1.30.4\conf\nginx.conf + ``` + **Linux:** ```bash sudo vi /etc/nginx/nginx.conf ``` -2. Add the following code inside the server block which is present in the "/etc/nginx/nginx.conf" file. +2. Define the custom tracking tokens in the text layout format and ensure `stub_status` is active with an `allow all` rule inside the HTTPS server block. ```nginxconf + log_format advanced_text '\(remote_addr -\)remote_user [\(time_local] "\)request" ' + '\(status\)body_bytes_sent "\(http_referer" "\)http_user_agent" ' + 'rt=\$request_time urt=\(upstream_response_time cs=\)upstream_cache_status'; + + access_log logs/access.log advanced_text; + location /nginx_status { stub_status; + allow all; } ``` 3. Save and close the /etc/nginx/nginx.conf file. -4. Now reload the nginx to apply the changes : +4. Now reload NGINX to apply your tracking changes: + **Windows:** + ```powershell + cd C:\nginx-1.30.4 + .\nginx.exe -s reload + ``` + **Linux:** ```bash sudo systemctl reload nginx ``` -5. Test the nginx status URL; it should return a response without error. +5. Test the secure nginx status URL; it should return a raw response without error. For Example - ``` - curl http://localhost/nginx_status + ```bash + curl https://localhost/nginx_status -Insecure ``` Response of the command should be similar to the below output. @@ -51,60 +76,72 @@ wget https://raw.githubusercontent.com/site24x7/plugins/master/nginx/installer/S Reading: 0 Writing: 1 Waiting: 1 ``` **Note :** - The nginx status URL used was the default one. If you have assigned a domain, please use that in the URL accordingly. + The nginx status URL uses HTTPS and ignores self-signed certificate constraints. If you have assigned a domain, please update the URL parameters accordingly. ## Plugin Installation - Once the agent is installed on the server, create a directory named `nginx`. + **Windows:** + ```powershell + mkdir nginx + cd nginx/ + ``` + **Linux:** ```bash mkdir nginx cd nginx/ ``` -- Download all the files and place it under the `nginx` directory. - +- Place your updated monitoring script files (`nginx_monitoring.py` and `nginx.cfg`) inside this `nginx` directory. On Linux servers, execute the following command to configure the Python path: ```bash - wget https://raw.githubusercontent.com/site24x7/plugins/master/nginx/nginx.py && sed -i "1s|^.*|#! $(which python3)|" nginx.py - wget https://raw.githubusercontent.com/site24x7/plugins/master/nginx/nginx.cfg + sed -i "1s|^.*|#! \$(which python3)|" nginx_monitoring.py ``` - -- Execute the below command with the appropriate arguments to check for valid JSON output. +- Execute the below command with the appropriate arguments to verify a clean metric JSON output string. ```bash python3 nginx.py --nginx_status_url "http://localhost/nginx_status" --username "nginx username" --password "nginx password" ``` -- After executing the above command and receiving valid JSON, provide the command argument as configurations in the `nginx.cfg` file. +- After executing the above command and receiving valid JSON, provide the configuration arguments inside the `nginx.cfg` file. - ```bash + ```ini [nginx] plugin_version=1 heartbeat=true - nginx_status_url="http://localhost/nginx_status" + nginx_status_url="https://localhost/nginx_status" username="None" password="None" timeout=60 - logs_enabled = "true" + logs_enabled = "false" log_type_name = "Nginx Logs" - log_file_path = "/var/log/nginx/access*" - ``` + log_file_path = "C:\nginx-1.30.4\logs\access.log" # Use /var/log/nginx/access.log for Linux + ``` -- Once the configuration done, move the `nginx` directory under the Site24x7 Linux Agent plugin directory: +- Once the configuration is done, move the `nginx` directory under the Site24x7 Agent plugin directory: - mv nginx /opt/site24x7/monagent/plugins/ + **Windows:** + ```powershell + mv nginx "C:\Program Files\Site24x7\WinAgent\plugins\" + ``` + **Linux:** + ```bash + sudo mv nginx /opt/site24x7/monagent/plugins/ + ``` The agent will automatically execute the plugin within five minutes and user can see the plugin monitor under Site24x7 > Plugins > Plugin Integrations. -If you need to run this nginx plugin on a Windows server, please follow the steps provided in the link below. -https://support.site24x7.com/portal/en/kb/articles/run-python-plugin-scripts-in-windows-servers +If you need to run this nginx plugin on a Linux server, please follow the steps provided in the link below. +https://site24x7.com ## Supported Metrics +### Core Connection Metrics + Name | Description --- | --- Currently active client connections | Current active client connections including waiting connections. @@ -114,3 +151,19 @@ Number of idle client connections waiting for a request | The current number Count of client requests | Client request count in nginx. Count of successful client connections | Successful client connection in nginx. Count of dropped connections | Dropped connections count. + +### Advanced Metrics (Log-Parsed) + +Name | Description +--- | --- +HTTP Status 2xx Count | Total count of successful status 200-299 responses processed. +HTTP Status 3xx Count | Total count of redirect status 300-399 responses processed. +HTTP Status 4xx Count | Total count of client error status 400-499 responses processed. +HTTP Status 5xx Count | Total count of server error status 500-599 responses processed. +Traffic Throughput Bytes | Total volume of data bandwidth sent back to clients in bytes. +Cache Hit Count | Total number of requests served directly from local disk cache. +Cache Miss Count | Total number of requests that missed cache and hit backend. +Cache Bypass Count | Total number of requests explicitly configured to bypass cache. +Cache Expired Count | Total number of requests where cached content was expired. +Avg Request Time ms | Average end-to-end client request round-trip time in milliseconds. +Avg Upstream Response Time ms | Average response execution latency of backend applications in milliseconds. \ No newline at end of file diff --git a/nginx/nginx.py b/nginx/nginx.py index 6921fffd..855d8018 100755 --- a/nginx/nginx.py +++ b/nginx/nginx.py @@ -1,9 +1,28 @@ #!/usr/bin/python +import os import sys import re import json +UNITS = { + "Avg Request Time": "ms", + "Avg Upstream Response Time": "ms", + "Traffic Throughput": "MB", +} + +TABS = { + "HTTP Responses": { + "order": 1, + "tablist": [ + "HTTP Status 2xx", + "HTTP Status 3xx", + "HTTP Status 4xx", + "HTTP Status 5xx", + ] + }, +} + class NginxServerMonitoring(): def __init__(self, config_data) : self.data = {} @@ -22,6 +41,9 @@ def _get_request_data_(self): try: url = self._config_data_['url'] + import ssl + ctx = ssl._create_unverified_context() + if self._config_data_['username'] and self._config_data_['password']: password_mgr = urlconnection.HTTPPasswordMgrWithDefaultRealm() password_mgr.add_password(None, url, self._config_data_['username'], self._config_data_['password']) @@ -30,15 +52,13 @@ def _get_request_data_(self): opener = urlconnection.build_opener(auth_handler, proxy_support) urlconnection.install_opener(opener) - response = urlconnection.urlopen(url, timeout=self._config_data_['timeout']) + response = urlconnection.urlopen(url, timeout=self._config_data_['timeout'], context=ctx) return response.read() except Exception as e: self.data['status'] = 0 #self.data['msg'] = str(e.code) + " " + str(e.reason) self.data['msg'] = str(e) - - - + def _collect_metrics_(self): output = self._get_request_data_() if output == None : return self.data @@ -68,6 +88,75 @@ def _collect_metrics_(self): self.data['Count of client requests'] = requests self.data['Count of successful client connections']= handled self.data['Count of dropped connections '] = (conn - handled) + + log_path = r"C:\nginx-1.30.4\logs\access.log" + if os.path.exists(log_path): + status_2xx = status_3xx = status_4xx = status_5xx = 0 + cache_hit = cache_miss = cache_bypass = cache_expired = 0 + request_times = [] + upstream_times = [] + total_bytes = 0 + + # Exact matching patterns for your advanced_text format layout + STATUS_REGEX = re.compile(r']\s+"[^"\\]*(?:\\.[^"\\]*)*"\s+([1-5]\d{2})') + BYTES_REGEX = re.compile(r']\s+"[^"\\]*(?:\\.[^"\\]*)*"\s+[1-5]\d{2}\s+(\d+)') + RT_REGEX = re.compile(r'\brt=([0-9.]+|-)灯?') + URT_REGEX = re.compile(r'\burt=([0-9., -]+)') + CS_REGEX = re.compile(r'\bcs=([A-Za-z_-]+|-)') + + with open(log_path, "r", errors="ignore") as f: + for line in f: + clean_line = line.strip() + if not clean_line: continue + + # 1. Parse Status Codes + st_m = STATUS_REGEX.search(clean_line) + if st_m: + code = st_m.group(1) + if code.startswith('2'): status_2xx += 1 + elif code.startswith('3'): status_3xx += 1 + elif code.startswith('4'): status_4xx += 1 + elif code.startswith('5'): status_5xx += 1 + + # 2. Parse Throughput Bandwidth + by_m = BYTES_REGEX.search(clean_line) + if by_m: total_bytes += int(by_m.group(1)) + + # 3. Parse End-to-End Request Time + rt_m = RT_REGEX.search(clean_line) + if rt_m and rt_m.group(1) != '-': + request_times.append(float(rt_m.group(1))) + + # 4. Parse Backend Upstream Time + urt_m = URT_REGEX.search(clean_line) + if urt_m and urt_m.group(1).strip() != '-': + for val in urt_m.group(1).replace(" ", "").split(','): + try: upstream_times.append(float(val)) + except ValueError: pass + + # 5. Parse Cache Status + cs_m = CS_REGEX.search(clean_line) + if cs_m: + state = cs_m.group(1).upper() + if 'HIT' in state: cache_hit += 1 + elif 'MISS' in state: cache_miss += 1 + elif 'BYPASS' in state: cache_bypass += 1 + elif 'EXPIRED' in state: cache_expired += 1 + + # Populate metrics to JSON structure + self.data['HTTP Status 2xx'] = status_2xx + self.data['HTTP Status 3xx'] = status_3xx + self.data['HTTP Status 4xx'] = status_4xx + self.data['HTTP Status 5xx'] = status_5xx + self.data['Traffic Throughput'] = round(total_bytes/(1024*1024),2) + self.data['Cache Hit Count'] = cache_hit + self.data['Cache Miss Count'] = cache_miss + self.data['Cache Bypass Count'] = cache_bypass + self.data['Cache Expired Count'] = cache_expired + self.data['Avg Request Time'] = round((sum(request_times)/len(request_times))*1000, 2) if request_times else 0.0 + self.data['Avg Upstream Response Time'] = round((sum(upstream_times)/len(upstream_times))*1000, 2) if upstream_times else 0.0 + self.data['units'] = UNITS + self.data['tabs']=TABS return self.data @@ -119,4 +208,4 @@ def _load_args_(): nginx = NginxServerMonitoring(_config_data_) result = nginx._collect_metrics_() print(json.dumps(result, indent=4, sort_keys=True)) - + \ No newline at end of file