Skip to content

Commit aec0743

Browse files
committed
fixup! feat(exapp_development): add Kubernetes setup instructions
Signed-off-by: Edward Ly <contact@edward.ly>
1 parent 0735327 commit aec0743

4 files changed

Lines changed: 57 additions & 20 deletions

File tree

developer_manual/exapp_development/scaling/AppAPIEmulation.rst

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,8 @@ Prerequisites
4343
You can also hit the agent directly on
4444
``http://127.0.0.1:8200/...`` for debugging, but that bypasses the
4545
HAProxy/AppAPI path and may skip shared-key enforcement depending
46-
on your routing.
46+
on your routing. Moreover, the port is only accessible from within the HaRP container
47+
(as it is the internal SPOE control API).
4748

4849
1. Check if ExApp is present (k8s deployment exists)
4950
----------------------------------------------------

developer_manual/exapp_development/scaling/KEDASetup.rst

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ When the queue reduces in size, KEDA scales back down.
4242
0. GPU Setup (kind cluster)
4343
---------------------------
4444

45-
If your ExApp needs GPU (e.g. llm2), you must set up GPU passthrough in
45+
If your ExApp requires a GPU (e.g. llm2), you must set up GPU passthrough in
4646
the kind cluster.
4747

4848
0.1 Configure Docker on the host
@@ -100,7 +100,17 @@ the kind cluster.
100100
0.5 Install NVIDIA device plugin
101101
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
102102

103-
For a single GPU shared across multiple pods, use **time-slicing**.
103+
By default, if a Deploy daemon is configured with multiple GPUs,
104+
AppAPI will attach one ExApp container/pod per GPU.
105+
In order to create multiple pods that share a single GPU,
106+
**time-slicing** must be configured for the GPU device plugin.
107+
108+
.. note::
109+
110+
All ExApps registered on the same Deploy daemon will share the same GPU resources.
111+
For ExApps that require heavy use of GPUs,
112+
it is recommended to have a separate Deploy daemon (host) for each of them.
113+
104114
First create a ConfigMap with the number of replicas (virtual GPUs):
105115

106116
.. code:: bash
@@ -186,7 +196,8 @@ Then deploy the device plugin with the config:
186196
print(f'{n[\"metadata\"][\"name\"]}: nvidia.com/gpu = {gpu}')
187197
"
188198
189-
Expected: ``nvidia.com/gpu = 4`` (or your configured replicas count).
199+
Expected: ``nvidia.com/gpu = 4`` (or your configured replicas count
200+
multiplied by the number of available physical GPUs).
190201
191202
0.7 Test GPU from a pod
192203
~~~~~~~~~~~~~~~~~~~~~~~

developer_manual/exapp_development/scaling/KubernetesSetup.rst

Lines changed: 41 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,24 @@ Architecture overview
4545

4646
--------------
4747

48+
0. Environment variables (optional)
49+
-----------------------------------
50+
51+
If you have a custom Kubernetes environment, you can specify its configuration here
52+
(the values below are the current defaults).
53+
54+
.. code:: bash
55+
56+
# .env
57+
HP_K8S_NAMESPACE="nextcloud-exapps"
58+
HP_K8S_STORAGE_CLASS=""
59+
HP_K8S_DEFAULT_STORAGE_SIZE="10Gi"
60+
HP_K8S_BEARER_TOKEN_FILE="/var/run/secrets/kubernetes.io/serviceaccount/token"
61+
62+
.. code:: bash
63+
64+
source .env
65+
4866
.. _scaling-kubernetes-setup-step-1:
4967

5068
1. Create the kind Cluster
@@ -160,15 +178,16 @@ From the HaRP repository root:
160178

161179
.. code:: bash
162180
163-
cd ~/nextcloud/HaRP
181+
cd path/to/HaRP
164182
bash development/redeploy_host_k8s.sh
165183
166184
The script will:
167185

168186
1. Auto-detect the k8s API server URL
169187
2. Generate a fresh bearer token
170-
3. Build the HaRP Docker image
171-
4. Start HaRP with k8s backend enabled on host network
188+
3. Connect the Nextcloud network to the cluster network
189+
4. Build the HaRP Docker image
190+
5. Start HaRP with k8s backend enabled on host network
172191

173192
Wait for HaRP to become healthy:
174193

@@ -204,6 +223,11 @@ your container ID or name, and ``<GATEWAY_IP>`` with the gateway from
204223
--k8s_expose_type=nodeport \
205224
--set-default
206225
226+
.. note::
227+
228+
While the ``--harp_frp_address`` is a required argument for the ``docker`` command,
229+
it is not used by Kubernetes at all.
230+
207231
Verify:
208232

209233
.. code:: bash
@@ -397,15 +421,26 @@ Clean up all ExApp resources
397421
Reset everything
398422
~~~~~~~~~~~~~~~~
399423

424+
Remove the deploy daemon config:
425+
400426
.. code:: bash
401427
402-
# Remove daemon config
403428
docker exec <NC_CONTAINER> sudo -E -u www-data php occ app_api:daemon:unregister k8s_local
404429
405-
# Delete kind cluster
430+
.. note::
431+
432+
Optionally, add the ``--rm-data`` option to also delete the associated PVC (Persistent Volume Claim).
433+
434+
Delete the ``kind`` cluster:
435+
436+
.. code:: bash
437+
406438
kind delete cluster --name nc-exapps
407439
408-
# Remove HaRP container
440+
Remove the HaRP container:
441+
442+
.. code:: bash
443+
409444
docker rm -f appapi-harp
410445
411446
Then start again from :ref:`Step 1<scaling-kubernetes-setup-step-1>`.

developer_manual/exapp_development/scaling/index.rst

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,6 @@ to increase the capacity and performance.
1414
The rest of this section will explain how to setup and use Kubernetes for automated scaling.
1515
Additional instructions are also provided if you have a GPU device for GPU scaling.
1616

17-
18-
.. note::
19-
20-
Currently, if a Deploy daemon is configured with GPUs available,
21-
AppAPI will by default attach all available GPU devices to each ExApp container on this Deploy daemon.
22-
This means that these GPUs are shared between all ExApps on the same Deploy daemon.
23-
Therefore, for the ExApps that require heavy use of GPUs,
24-
it is recommended to have a separate Deploy daemon (host) for them.
25-
26-
2717
.. toctree::
2818
:maxdepth: 2
2919

0 commit comments

Comments
 (0)