Skip to content

Tests/datakrypto benchmarking tests #807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
snova-rodrigom wants to merge 17 commits into
base: main
Choose a base branch
from
Draft

Tests/datakrypto benchmarking tests #807

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
57488d7
necessary changes for datakrypto test
snova-rodrigom Feb 19, 2026
de7ae48
small change
snova-rodrigom Feb 20, 2026
3228bd7
datakrypto config tests
snova-rodrigom Feb 23, 2026
d533c34
last tests
snova-rodrigom Feb 23, 2026
583f4c2
prompt change
snova-rodrigom Feb 23, 2026
9af414b
additional changes for metrics
snova-rodrigom Feb 23, 2026
1ba3bdd
update configs
jhpiedrahitao Feb 23, 2026
47e6ded
minor change
jhpiedrahitao Feb 23, 2026
7a6b0a9
add encrypted config
jhpiedrahitao Feb 23, 2026
d4761e6
fix typo
jhpiedrahitao Feb 23, 2026
9adccab
added datakrypto metrics to consolidated report
snova-rodrigom Feb 23, 2026
302ab97
update configs
jhpiedrahitao Feb 25, 2026
f41abc7
update bundle reports
jhpiedrahitao Feb 28, 2026
ec9a0d3
model configs for complete runs
snova-rodrigom May 7, 2026
7de9684
last run
snova-rodrigom May 8, 2026
12e6697
updated notebook with break up by input size
snova-rodrigom May 8, 2026
556e4dd
last run using data from vm and ss up to 16k
snova-rodrigom May 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions benchmarking/benchmarking_bundles/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
model_configs_path: '<PATH TO AISK REPO HERE>/benchmarking/benchmarking_bundles/model_configs_example.csv'
model_configs_path: '../benchmarking/benchmarking_bundles/model_configs_example.csv'
llm_api: 'sncloud'
output_files_dir: '<PATH TO AISK REPO HERE>/benchmarking/data/bundle_tests/output_files'
consolidated_results_dir: '<PATH TO AISK REPO HERE>/benchmarking/data/bundle_tests/consolidated_results'
output_files_dir: '../benchmarking/data/datakrypto/high_ss/speed_bench_test_encrypted_multiple'
consolidated_results_dir: '../benchmarking/data/datakrypto/high_ss/speed_bench_test_encrypted_multiple/consolidated_results'
timeout: 3600
time_delay: 0
# Row-level concurrency
Expand Down
17 changes: 17 additions & 0 deletions benchmarking/benchmarking_bundles/datakrypto-complete-encrypted-model_configs.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-xLAM-2-8b-fc-r-encrypted,3900,100,10,1,,,
Llama-xLAM-2-8b-fc-r-encrypted,3900,100,30,3,,,
Llama-xLAM-2-8b-fc-r-encrypted,3900,100,50,5,,,
Llama-xLAM-2-8b-fc-r-encrypted,3900,100,80,8,,,
Llama-xLAM-2-8b-fc-r-encrypted,3900,100,160,16,,,
Llama-xLAM-2-8b-fc-r-encrypted,3900,100,320,32,,,
Llama-xLAM-2-8b-fc-r-encrypted,8000,100,10,1,,,
Llama-xLAM-2-8b-fc-r-encrypted,8000,100,30,3,,,
Llama-xLAM-2-8b-fc-r-encrypted,8000,100,50,5,,,
Llama-xLAM-2-8b-fc-r-encrypted,8000,100,80,8,,,
Llama-xLAM-2-8b-fc-r-encrypted,8000,100,160,16,,,
Llama-xLAM-2-8b-fc-r-encrypted,8000,100,320,32,,,
Llama-xLAM-2-8b-fc-r-encrypted,16000,100,10,1,,,
Llama-xLAM-2-8b-fc-r-encrypted,16000,100,30,3,,,
Llama-xLAM-2-8b-fc-r-encrypted,16000,100,50,5,,,
Llama-xLAM-2-8b-fc-r-encrypted,16000,100,80,8,,,
17 changes: 17 additions & 0 deletions benchmarking/benchmarking_bundles/datakrypto-complete-non_encrypted-model_configs.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-xLAM-2-8b-fc-r,3900,100,10,1,,,
Llama-xLAM-2-8b-fc-r,3900,100,30,3,,,
Llama-xLAM-2-8b-fc-r,3900,100,50,5,,,
Llama-xLAM-2-8b-fc-r,3900,100,80,8,,,
Llama-xLAM-2-8b-fc-r,3900,100,160,16,,,
Llama-xLAM-2-8b-fc-r,3900,100,320,32,,,
Llama-xLAM-2-8b-fc-r,8000,100,10,1,,,
Llama-xLAM-2-8b-fc-r,8000,100,30,3,,,
Llama-xLAM-2-8b-fc-r,8000,100,50,5,,,
Llama-xLAM-2-8b-fc-r,8000,100,80,8,,,
Llama-xLAM-2-8b-fc-r,8000,100,160,16,,,
Llama-xLAM-2-8b-fc-r,8000,100,320,32,,,
Llama-xLAM-2-8b-fc-r,16000,100,10,1,,,
Llama-xLAM-2-8b-fc-r,16000,100,30,3,,,
Llama-xLAM-2-8b-fc-r,16000,100,50,5,,,
Llama-xLAM-2-8b-fc-r,16000,100,80,8,,,
Binary file added benchmarking/benchmarking_bundles/datakrypto_tee_performance_report.pdf
View file
Open in desktop
Binary file not shown.
979 changes: 979 additions & 0 deletions benchmarking/benchmarking_bundles/datakrypto_tee_report.py
View file
Open in desktop

Large diffs are not rendered by default.

22 changes: 16 additions & 6 deletions benchmarking/benchmarking_bundles/model_configs_example.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,17 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-4-Maverick-17B-128E-Instruct,128,128,1,1,,,
Llama-4-Maverick-17B-128E-Instruct,128,128,2,2,,,
Llama-4-Maverick-17B-128E-Instruct,128,128,4,4,,,
Llama-4-Maverick-17B-128E-Instruct,128,128,8,8,,,
Llama-4-Maverick-17B-128E-Instruct,128,128,16,16,,,
Llama-4-Maverick-17B-128E-Instruct,128,128,32,32,,,
Llama-xLAM-2-8b-fc-r,3900,100,10,1,,,
Llama-xLAM-2-8b-fc-r,3900,100,30,3,,,
Llama-xLAM-2-8b-fc-r,3900,100,50,5,,,
Llama-xLAM-2-8b-fc-r,3900,100,80,8,,,
Llama-xLAM-2-8b-fc-r,3900,100,160,16,,,
Llama-xLAM-2-8b-fc-r,3900,100,320,32,,,
Llama-xLAM-2-8b-fc-r,8000,100,10,1,,,
Llama-xLAM-2-8b-fc-r,8000,100,30,3,,,
Llama-xLAM-2-8b-fc-r,8000,100,50,5,,,
Llama-xLAM-2-8b-fc-r,8000,100,80,8,,,
Llama-xLAM-2-8b-fc-r,8000,100,160,16,,,
Llama-xLAM-2-8b-fc-r,8000,100,320,32,,,
Llama-xLAM-2-8b-fc-r,16000,100,10,1,,,
Llama-xLAM-2-8b-fc-r,16000,100,30,3,,,
Llama-xLAM-2-8b-fc-r,16000,100,50,5,,,
Llama-xLAM-2-8b-fc-r,16000,100,80,8,,,
13 changes: 13 additions & 0 deletions benchmarking/benchmarking_bundles/model_configs_example_datakrypto_encrypted.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,1,1,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,2,2,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,4,4,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,8,8,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,16,16,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,32,32,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,1,1,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,2,2,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,4,4,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,8,8,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,16,16,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,32,32,,,
13 changes: 13 additions & 0 deletions benchmarking/benchmarking_bundles/model_configs_example_datakrypto_encrypted_multiple.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,10,1,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,20,2,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,40,4,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,80,8,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,160,16,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,100,320,32,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,10,1,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,20,2,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,40,4,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,80,8,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,160,16,,,
Llama-xLAM-2-8b-fc-r-encrypted,2000,1024,320,32,,,
13 changes: 13 additions & 0 deletions benchmarking/benchmarking_bundles/model_configs_example_datakrypto_non_encrypted.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-xLAM-2-8b-fc-r,2000,100,1,1,,,
Llama-xLAM-2-8b-fc-r,2000,100,2,2,,,
Llama-xLAM-2-8b-fc-r,2000,100,4,4,,,
Llama-xLAM-2-8b-fc-r,2000,100,8,8,,,
Llama-xLAM-2-8b-fc-r,2000,100,16,16,,,
Llama-xLAM-2-8b-fc-r,2000,100,32,32,,,
Llama-xLAM-2-8b-fc-r,2000,1024,1,1,,,
Llama-xLAM-2-8b-fc-r,2000,1024,2,2,,,
Llama-xLAM-2-8b-fc-r,2000,1024,4,4,,,
Llama-xLAM-2-8b-fc-r,2000,1024,8,8,,,
Llama-xLAM-2-8b-fc-r,2000,1024,16,16,,,
Llama-xLAM-2-8b-fc-r,2000,1024,32,32,,,
13 changes: 13 additions & 0 deletions ...hmarking/benchmarking_bundles/model_configs_example_datakrypto_non_encrypted_multiple.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-xLAM-2-8b-fc-r,2000,100,10,1,,,
Llama-xLAM-2-8b-fc-r,2000,100,20,2,,,
Llama-xLAM-2-8b-fc-r,2000,100,40,4,,,
Llama-xLAM-2-8b-fc-r,2000,100,80,8,,,
Llama-xLAM-2-8b-fc-r,2000,100,160,16,,,
Llama-xLAM-2-8b-fc-r,2000,100,320,32,,,
Llama-xLAM-2-8b-fc-r,2000,1024,10,1,,,
Llama-xLAM-2-8b-fc-r,2000,1024,20,2,,,
Llama-xLAM-2-8b-fc-r,2000,1024,40,4,,,
Llama-xLAM-2-8b-fc-r,2000,1024,80,8,,,
Llama-xLAM-2-8b-fc-r,2000,1024,160,16,,,
Llama-xLAM-2-8b-fc-r,2000,1024,320,32,,,
9 changes: 9 additions & 0 deletions benchmarking/benchmarking_bundles/model_configs_example_datakrypto_only.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
model_name,input_tokens,output_tokens,num_requests,concurrent_requests,qps,qps_distribution,multimodal_img_size
Llama-xLAM-2-8b-fc-r,2000,100,1,1,,,
Llama-xLAM-2-8b-fc-r,2000,2000,1,1,,,
Llama-xLAM-2-8b-fc-r,3900,100,1,1,,,
Llama-xLAM-2-8b-fc-r,3900,3900,1,1,,,
Llama-xLAM-2-8b-fc-r,8000,100,1,1,,,
Llama-xLAM-2-8b-fc-r,8000,8000,1,1,,,
Llama-xLAM-2-8b-fc-r,16000,100,1,1,,,
Llama-xLAM-2-8b-fc-r,16000,16000,1,1,,,
Loading