[Windows] Support CPU shared memory (Client/Frontend) - #7048

Open

fpetrini15 wants to merge 3302 commits into

mainfrom

fpetrini-win-cpu-shm

This pull request is big! We're only showing the most recent 250 commits

Commits on Jul 24, 2023

Fix default instance name (#6097)
kthui
authored
a58246f

Commits on Jul 25, 2023

Removing unused tests (#6085)
oandreeva-nv
authored
1d02ea6

Commits on Jul 28, 2023

Update post-23.07 release (#6103)
mc-nv
authored
4b3c278
Improve asyncio testing (#6122)
Tabrizian
authored
46859db

Commits on Aug 1, 2023

Reduce instance count to 1 for python bls model loading test (#6130)
krishung5
authored
4de6d8a

Commits on Aug 4, 2023

Commits on Aug 7, 2023

Adding the support tracing of child models invoked from a BLS model (#6063)

oandreeva-nv
and
rmccorm4
authored
a129063

Commits on Aug 8, 2023

Commits on Aug 9, 2023

Fix the versions in the doc (#6164)
tanmayv25
authored
1370ec7
Update docs with NVAIE messaging (#6162)
dzier
authored
39fafff

Commits on Aug 10, 2023

Commits on Aug 14, 2023

Commits on Aug 15, 2023

Add end-to-end CI test for decoupled model support (#6131) (#6184)
matthewkotila
authored
1bb0131
Test preserve_ordering for oldest strategy sequence batcher (#6185)
rmccorm4
authored
d8fd125
added debugging guide (#5924)

tanayvarshney
and
David Yastremsky
authored
c968045
Add deadlock gdb section to debug guide (#6193)
rmccorm4
authored
ecfe017

Commits on Aug 16, 2023

Fix character escape in model repository documentation (#6197)
GuanLuo
authored
8cca875
Fix docs test (#6192)
matthewkotila
authored
5d2ea14

Commits on Aug 17, 2023

Commits on Aug 18, 2023

Check for log error on failing to find explicit load model (#6204)
rmccorm4
authored
d8931b0

Commits on Aug 22, 2023

Set default shm size to 1MB for Python backend (#6209)
Tabrizian
authored
1423260

Commits on Aug 23, 2023

Trace Model Name Validation (#6199)
fpetrini15
authored
abd53c9
Fix the check for error message (#6226)
krishung5
authored
e43aa5d
Fix copyright for debugging guide (#6225)
krishung5
authored
2007bd8

Commits on Aug 28, 2023

Add watts units to GPU power metric descriptions (#6242)
rmccorm4
authored
1f9db58

Commits on Aug 30, 2023

Update post-23.08 release (#6234)

authored
6ec125b
Add tests/docs for queue size (pending request count) metric (#6233)
rmccorm4
authored
4b45fa6
Adding safe string to number conversions (#6173)
oandreeva-nv
authored
dd696b1

Commits on Sep 2, 2023

Test Python execute() to return Triton error code (#6228)
kthui
authored
38bdc53
Add docs on IPv6 (#6262)
rmccorm4
authored
ed76ffb

Commits on Sep 5, 2023

Add test for TensorRT version-compatible model support (#6255)
the-david-oy
authored
1ea7765

Commits on Sep 6, 2023

Commits on Sep 8, 2023

Update 'main' to track development of 2.39.0 / 23.10 (#6277)
mc-nv
authored
a3cb238

Commits on Sep 11, 2023

Apply UCX workaround (#6254)
Tabrizian
authored
edb95cb
Add ensemble parameter forwarding test (#6284)
rmccorm4
authored
d622ed4
Exclude extra TRT version-compatible models from tests (#6294)

the-david-oy
and
rmccorm4
authored
38a1eab

Commits on Sep 12, 2023

Adding installation of docker and docker-buildx (#6299)
mc-nv
authored
0f576d2

Commits on Sep 13, 2023

Use targetmodel from header as model name in SageMaker (#6147)
nikhil-sk
authored
f5371a6

Commits on Sep 14, 2023

Add more error messages, return codes, and refactor HTTP server (#6297)
rmccorm4
authored
4e6d7be

Commits on Sep 18, 2023

Fix typo (#6318)
krishung5
authored
bce13c8
Update the request re-use example (#6283)
Tabrizian
authored
f8e83f9

Commits on Sep 19, 2023

Commits on Sep 20, 2023

Tests and docs for ability to specify subdirectory to download for LocalizePath (#6308)
oandreeva-nv
authored
d329df4

Commits on Sep 21, 2023

Refactor HandleInfer into more readable chunks (#6332)
rmccorm4
authored
605c15b

Commits on Sep 25, 2023

Refactor model generation scripts (#6336)
Tabrizian
authored
2275845

Commits on Sep 26, 2023

Add testing for scalar I/O in ORT backend (#6343)
Tabrizian
authored
f88b4fb

Commits on Sep 29, 2023

Update post-23.09 release (#6367)
mc-nv
authored
60d6690

Commits on Oct 3, 2023

Add security policy (#6376)
the-david-oy
authored
e7ec69d

Commits on Oct 4, 2023

Adding client-side request cancellation support and testing (#6383)

authored
cfa5f72
Add missing copyright (#6388)
tanmayv25
authored
9a8bcf4

Commits on Oct 6, 2023

Add basic generate endpoints for LLM tasks (#6366)

authored
dd5efac
Add Python backend request cancellation test (#6364)
kthui
authored
d67b725

Commits on Oct 7, 2023

Add TRT-LLM backend build to Triton (#6365) (#6392)
krishung5
authored
6959419

Commits on Oct 8, 2023

Commits on Oct 9, 2023

Support parameters object in generate route
nnshah1
authored
93da69d
Update 'main' to track development of 2.40.0 / 23.11 (#6400)
mc-nv
authored
f54ddd6

Commits on Oct 11, 2023

Fix L0_sdk (#6387)
tanmayv25
authored
2ffbf85
Add documentation on request cancellation (#6403)

authored
f8ba585
Fixes in request cancellation doc (#6409)
tanmayv25
authored
9fe8569

Commits on Oct 12, 2023

Document generate HTTP endpoint (#6412)
GuanLuo
authored
1992892
Update SECURITY.md to not display commented copyright (#6426)
the-david-oy
authored
f849724
Fix missing library in L0_data_compression (#6424)
GuanLuo
authored
9bf792d

Commits on Oct 13, 2023

Commits on Oct 16, 2023

Fix L0_implicit_state (#6427)
Tabrizian
authored
ccbae03
Fixing vllm build (#6433)
oandreeva-nv
authored
c112666

Commits on Oct 17, 2023

Switch Jetson model TensorRT models generation to container (#6378)
mc-nv
authored
3aba5f4

Commits on Oct 18, 2023

Bumped vllm version (#6444)
oandreeva-nv
authored
46f93e9
Adjust test_concurrent_same_model_load_unload_stress (#6436)
kthui
authored
cf85998

Commits on Oct 19, 2023

Commits on Oct 20, 2023

Update TRT-LLM backend url (#6455)
krishung5
authored
566facd
Added docs on python based backends (#6429)

oandreeva-nv
and
nnshah1
authored
7e7ee88

Commits on Oct 24, 2023

L0_model_config Fix (#6472)
fpetrini15
authored
383850d
Add test for Python model parameters (#6452)
kthui
authored
8c37608

Commits on Oct 25, 2023

Test Python BLS with different sizes of CUDA memory pool (#6276)
krishung5
authored
2f7f396

Commits on Oct 26, 2023

Add documentation for K8s-onprem StartupProbe (#5257)

authored
5d6a60a

Commits on Oct 27, 2023

Update `main` post-23.10 release (#6484)

authored
bd99799

Commits on Oct 28, 2023

Adding structure reference to the new document (#6493)
mc-nv
authored
9f04d6d

Commits on Oct 30, 2023

Improve L0_backend_python test stability (ensemble / gpu_tensor_lifecycle) (#6490)
rmccorm4
authored
dee479d

Commits on Nov 1, 2023

Add L0_generative_sequence test (#6475)
GuanLuo
authored
ab4d03a
Downgrade vcpkg version (#6503)
GuanLuo
authored
1f8507e

Commits on Nov 3, 2023

Fixing L0_io (#6510)
oandreeva-nv
authored
aa473f1
Add Python-based backends CI (#6466)

authored
e552ef2
Enabling option to restrict access to HTTP APIs based on header value pairs (similar to gRPC)
nnshah1
authored
e9677ec

Commits on Nov 6, 2023

Upgrade DCGM from 2.4.7 to 3.2.6 (#6515)
kthui
authored
659611c

Commits on Nov 7, 2023

Enhance GCS credentials documentations (#6526)
kthui
authored
f1465b9
Test file override outside of model directory (#6516)
rmccorm4
authored
2ad2786
Update ORT version to 1.16.2 (#6531)
mc-nv
authored
e12d06c

Commits on Nov 8, 2023

Adjusting expected error msg (#6517)
oandreeva-nv
authored
21a007a
Update 'main' to track development of 2.41.0 / 23.12 (#6543)
mc-nv
authored
e33752b
Enhance testing for pending request count (#6532)
tanmayv25
authored
0a0ea18

Commits on Nov 9, 2023

Add testing for Python backend request rescheduling (#6509)

krishung5
and
Tabrizian
authored
ce9652b
Check that the wget is installed (#6556)
mc-nv
authored
cf40588

Commits on Nov 10, 2023

secure deployment considerations guide (#6533)

nnshah1
and
nv-kmcgill53
authored
26ef299
Fix typo and change the command line order (#6557)
mc-nv
authored
705f810

Commits on Nov 13, 2023

Add error during rescheduling test to L0_generative_sequence (#6550)
GuanLuo
authored
806fed4

Commits on Nov 15, 2023

changing references to concrete instances
nnshah1
authored
1897add
Add testing for implicit state enhancements (#6524)
Tabrizian
authored
246564a
Fix copyright (#6584)
Tabrizian
authored
4be58a3

Commits on Nov 16, 2023

Commits on Nov 19, 2023

Change name generative -> iterative (#6601)
nnshah1
authored
68d5bb4

Commits on Nov 20, 2023

Add documentation on generative sequence (#6595)
GuanLuo
authored
926330e
Updated description of iterative sequences
nnshah1
authored
65bc558
Restricted HTTP API documentation

nnshah1
and
rmccorm4
authored
96933b1

Commits on Nov 22, 2023

Use compliant preprocessor (#6626)
tanmayv25
authored
17d2de7
Update README.md (#6627)
dzier
authored
e0d00e6
Extend request objects lifetime and fixes possible segmentation fault (#6620)

tanmayv25
and
nnshah1
authored
0556b7c

Commits on Nov 29, 2023

Update protobuf after python update for testing (#6638)
jbkyang-nvi
authored
b2dd98b

Commits on Dec 4, 2023

Update post-23.11 release (#6653)

mc-nv
and
krishung5
authored
7c0ed09
PYBE Timeout Tests (#6483)
fpetrini15
authored
0e721e4

Commits on Dec 5, 2023

Add note on lack of ensemble support (#6648)
rmccorm4
authored
9e7f237

Commits on Dec 6, 2023

Added request id to span attributes (#6667)
oandreeva-nv
authored
df9d6e8

Commits on Dec 8, 2023

Add test for optional internal tensor within an ensemble (#6663)
GuanLuo
authored
88b8843

Commits on Dec 10, 2023

Set CMake version to 3.27.7 (#6675)
mc-nv
authored
afaf6b0

Commits on Dec 11, 2023

restore typo (#6680)
mc-nv
authored
c329f48

Commits on Dec 12, 2023

Update 'main' to track development of 2.42.0 / 24.01 (#6673)
mc-nv
authored
d496f95
iGPU build refactor (#6684) (#6691)
kthui
authored
d191ef9

Commits on Dec 13, 2023

Mlflow Plugin Fix (#6685)
fpetrini15
authored
0b63e0d
Fix extra content-type headers in HTTP server (#6678)
rmccorm4
authored
ebd6a3e
Fix iGPU CMakeFile tags (#6695)

kthui
and
kyle
authored
6ca4764

Commits on Dec 14, 2023

adding default value for TRITON_IGPU_BUILD=OFF (#6705)

nv-kmcgill53
and
kyle
authored
57b49b4

Commits on Dec 15, 2023

Commits on Dec 19, 2023

Modify HTTP frontend to return error code reflecting Triton error. Add test (#6713)
GuanLuo
authored
a4b8162

Commits on Dec 26, 2023

Update `main` branch post-23.12 release (#6726)

authored
06df7a8

Commits on Dec 28, 2023

trtllm CI fixes (#6687)
mc-nv
authored
45278d7

Commits on Jan 2, 2024

Remove double unit test (#6714)
kthui
authored
cb0c2e5

Commits on Jan 3, 2024

Update RE2 package location (#6750)
Tabrizian
authored
14f70b6
Add testing for zero tensors in PyTorch backend (#6760)
Tabrizian
authored
6bc5625

Commits on Jan 4, 2024

Fix L0_lifecycle on insufficient hardware concurrency (#6762)
kthui
authored
fb3747a

Commits on Jan 5, 2024

Simplify cmake install command (#6725)
Tabrizian
authored
1ea633a
Add cmdline option to set model load retry. Add test (#6764)
GuanLuo
authored
b48aa57

Commits on Jan 8, 2024

Commits on Jan 11, 2024

Add Triton Inference Server In-Process Python API Tests
nnshah1
authored
310c38c
Bring back Python backend based PyTorch backend (#6518)
kthui
authored
fa72cf2
Bump min cxx standard to 17 (#6742)
oandreeva-nv
authored
2782d30

Commits on Jan 12, 2024

Add tests/docs for Pinned Memory metrics (#6754)

pskiran1
and
rmccorm4
authored
cf5fe35
Update 'main' to track development of 2.42.0 / 24.02 (#6786)
mc-nv
authored
c205451

Commits on Jan 17, 2024

Commits on Jan 18, 2024

Added docs for otel context propagation (#6804)
oandreeva-nv
authored
7b06a37
Fix typos in trace.md (#6808)
rmccorm4
authored
b6e017e

Commits on Jan 19, 2024

Fix test_model_config_overwite in L0_lifecycle (#6818)
GuanLuo
authored
3e79b2a
Improve L0_backend_python on shm reliability (#6803)

kthui
and
rmccorm4
authored
7edeb9f

Commits on Jan 22, 2024

Remove boost::filesystem (#6810)
rmccorm4
authored
3bff367

Commits on Jan 23, 2024

Generate unittest xml reports from L0_python_api (#6822)
rmccorm4
authored
bc71da0

Commits on Jan 25, 2024

Commits on Jan 27, 2024

Add test for shutdown while unloading in background (#6835)
kthui
authored
ddfdb2a

Commits on Jan 29, 2024

Handle 0 dimension output for generate endpoint (#6833)
krishung5
authored
56e4232
tensorrt-llm benchmarking test (#6771)
pskiran1
authored
d98a59c

Commits on Jan 30, 2024

Commits on Jan 31, 2024

Add test for shutdown while loading model (#6837)
kthui
authored
f92732d

Commits on Feb 1, 2024

Commits on Feb 7, 2024

Updating vllm version to 0.3.0 (#6858)
oandreeva-nv
authored
508929a

Commits on Feb 8, 2024

Python Backend Windows Support (#6830)
fpetrini15
authored
738c98f

Commits on Feb 9, 2024

Add support for Oracle Cloud in deploy (#6850)
bruno-garbaccio
authored
3d79568

Commits on Feb 13, 2024

Add link to TRTLLM metrics docs (#6874)
rmccorm4
authored
1df73dc

Commits on Feb 14, 2024

Commits on Feb 16, 2024

Update 'main' to track development of 2.44.0 / 24.03 (#6892)
mc-nv
authored
8a2a229

Commits on Feb 17, 2024

Add response statistics (#6869)

kthui
and
rmccorm4
authored
59e267f

Commits on Feb 22, 2024

Fix busyop test for L0_memory_growth (#6900)
krishung5
authored
21a7fc5

Commits on Feb 23, 2024

Add cancellation into response statistics (#6904)
kthui
authored
60872b9

Commits on Feb 24, 2024

Install required pip pkgs (#6906)
krishung5
authored
8d8b607

Commits on Feb 27, 2024

Match forward headers case insensitively. (#6889)
yinggeh
authored
adafa4f

Commits on Mar 1, 2024

Add note on --cache-config spacing and fix typos (#6929)
rmccorm4
authored
551978b

Commits on Mar 2, 2024

Remove ignore files that are not in use by repository (#6893)
mc-nv
authored
246f46c
Update README and versions for 2.43.0 / 24.02 (#6886)
mc-nv
committed
1dcf2cf
Set ONNX Runtime version 1.17.2
mc-nv
committed
9be77f1

Commits on Mar 4, 2024

Expose tritonserver args in values.yaml (#5582)
okyspace
authored
19b02a2

Commits on Mar 6, 2024

Parameterize git repository (#6934)

nv-kmcgill53
and
kyle
authored
d0f332b

Commits on Mar 8, 2024

Commits on Mar 9, 2024

Test improved input validation errors (#6933)
indrajit96
authored
b012bd0

Commits on Mar 11, 2024

Commits on Mar 13, 2024

Add AsyncIO HTTP compression test (#6975)
kthui
authored
e92abf2
Install `genai-pa` into SDK container (#6942)

mc-nv
and
debermudez
authored
8139431

Commits on Mar 15, 2024

extend existing tests with more parameters (#6951)
yf711
authored
5c6e487
Exposing trace context to python backend (#6985)
oandreeva-nv
authored
9f16eef

Commits on Mar 19, 2024

Add documentation for mapping between Triton Errors and HTTP status codes (#6992)
Tabrizian
authored
8b36aa8

Commits on Mar 21, 2024

Remove hatch version (#7009)
tgerdesnv
authored
afaa6f4
Update vLLM to 0.3.2 for gemma support (#6918)
kebe7jun
authored
fdbfb27

Commits on Mar 25, 2024

Add missing copyright for L0_trace (#6996)
oandreeva-nv
authored
2be127b
fix sphinx warnings (#7030)
yinggeh
authored
df753d7

Commits on Mar 27, 2024

Add meetup invite banner (#7049)
rmccorm4
authored
a844eda

Commits on Apr 1, 2024

Update 'main' post-24.03 (#7051)

authored
8a208d7

Commits on Apr 4, 2024

Commits on Apr 5, 2024

Commits on Apr 8, 2024

Fix html image rendering in sphinx documentation (#7084)
tanmayv25
authored
2150fc2

Commits on Apr 9, 2024

Commits on Apr 10, 2024

Updated vllm version (#7095)
oandreeva-nv
authored
5e20ef6

Commits on Apr 11, 2024

Disable Dynamic Log File (#7092)
yinggeh
authored
52f97b5
Initial commit
fpetrini15
committed
b1663e9
Fix GPU case
fpetrini15
committed
14dacb3
Validate offset
fpetrini15
committed
a8914c4
Open shm file, don't create
fpetrini15
committed
3de58c9
Intermmediate commit: Major test restructuring.
fpetrini15
committed
be0d10e
Formatting
fpetrini15
committed
6d99076
Gitbot Fixes
fpetrini15
committed
ef2c735
Major software bloat refactor. Opaque shm file handle
fpetrini15
committed
35aa5cc
Fixes for Unix and handle-agnostic get restructure
fpetrini15
committed
9df15bd
Review comments. Passing ShmFile pointer instead of void
fpetrini15
committed
76f9f2b
Validate system shared memory region size when registering a region (#7093)
rmccorm4
authored
159b060
Decoupled Async Execute (#7062)
kthui
authored
196caf0
Add trace mode and trace config entries in trace settings API (#7050)

indrajit96
and
rmccorm4
authored
5b739db
Merge remote-tracking branch 'origin/main' into fpetrini-win-cpu-shm
fpetrini15
committed
2d206e6

Commits on Apr 13, 2024

Open backing file to validate shared memory
fpetrini15
committed
06d1e6b