Assessment performance issue with many servers

[Thommil]

Hi,

I'm currently working with @PierreRustOrange on LCAaC through Cloud Assess and I'm facing a critical performance issue.

Using this inventory containing about 300 servers, when accessing /virtual_machines/assess API endpoint:

• Request for 1 VM:

~/workspace/cloud-assess$ curl -s -w "time_total: %{time_total}s\n" --header "Content-Type: application/json" -o /dev/null --request POST --data @vms-1.json http://localhost:8080/virtual_machines/assess
time_total: 2.553245s

• Request for 10 VMs:

~/workspace/cloud-assess$ curl -s -w "time_total: %{time_total}s\n" --header "Content-Type: application/json" -o /dev/null --request POST --data @vms-10.json http://localhost:8080/virtual_machines/assess
time_total: 25.126882s

• Request for 100 VMs:

~/workspace/cloud-assess$ curl -s -w "time_total: %{time_total}s\n" --header "Content-Type: application/json" -o /dev/null --request POST --data @vms-100.json http://localhost:8080/virtual_machines/assess
time_total: 267.101148s

Response time which is initialy pretty slow for 1 VM (>2s) increases linearly with number of VMs and seems tidly bound to the number of servers in inventory.

My need is for thousands of VMs each hour (more than 4000) and based on this inventory and my testing setup, we can't hardly handle 1500 VMs per hour and potential errors/timeouts will also become fatal issues.

Based on logs and the linear increase, I suspect that for each VM, many redundants processes are done.

Is there any way to introduce caching mechanism to improve response time?

[pevab]

Hi,

Thanks for reaching out.

• Which version of Cloud Assess are you using?
• Can you share the lcaac config file lcaac.yaml you are using?

There is already a caching mechanism put in place. Maybe just a quick info: for now, the cache is in-memory, so the cache is not persisted over restarts of the cloud assess server (container).

[Thommil]

The version is 1.8.2 and the lcaac.yaml is the default one:

name: Cloud Assess Trusted Library
description: Reference models for Cloud Assess

# -------------------------------------- #
#               Generic
# -------------------------------------- #

connectors:
  - name: csv
    cache:
      enabled: true
    options:
      directory: data

[PierreRustOrange]

To add some context : we suspect that the performance issue is a lcaac level, we are able to reproduce the problem when using lcaac directly (bypassing the API from cloud assess entirely).
When we calculate the footprint of a set a workloads, a large part of the calculation is the same for all VM (building, server and equipment pools footprint, etc.) but it seems that this part is re-computed sequentially for each workload.