From 29088bbb3d3619b2726e53516ab35b0a20013e9c Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 12:59:42 -0500 Subject: [PATCH 01/34] Create 2019-10-21-hosted-ce.md This is a WIP PR. It's reached a sufficient amount of content that I figured it would be best to get the PR up while I hammer things out. --- _posts/2019-10-21-hosted-ce.md | 198 +++++++++++++++++++++++++++++++++ 1 file changed, 198 insertions(+) create mode 100644 _posts/2019-10-21-hosted-ce.md diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md new file mode 100644 index 00000000..cb651c1e --- /dev/null +++ b/_posts/2019-10-21-hosted-ce.md @@ -0,0 +1,198 @@ +--- +title: "Deploying and testing an OSG Hosted CE via SLATE" +overview: Blog +published: false +permalink: blog/deploy-hosted-ce.html +attribution: The SLATE Team +layout: post +type: markdown +--- + +# Deploying an OSG HostedCE + +## Prerequisites +The OSG HostedCE uses SSH to submit pilot jobs that will connect back to the +OSG central pool. In order for the HostedCE to work, you'll first need to +create a local service account for OSG to submit jobs. This should be done +according to whatever process you normally use to create accounts for users. + +## Generating and storing the key +Once the account has been created, you'll want to create a new SSH key pair. +The private part of the key will be stored within SLATE, and the public part of +the key will be installed into `authorized_keys` file of the OSG user on your +cluster. To generate the key, you'll need to run the following on some machine +with OpenSSH installed: + + ssh-keygen -f osg-keypair + +Note that you will need to make this key passphraseless, as the HostedCE +software will consume this key. Once you've created the key, you'll want to +store the public part of it (osg-keypair.pub) into the `authorized_keys` file +on the OSG account for your cluster. For example, if your OSG service account +is called `osg`, you'll want to append the contents of `osg-keypair.pub` to +`/home/osg/.ssh/authorized_keys`. + +The private part of the keypair will need to be stored in SLATE for use by the +CE. To do that: + + slate secret create ce-private-key --from-file=bosco.key=osg-keypair + +Where `ce-private-key` will be the name of the secret, and `osg-keypair` is the +path to your private key (assumed to be the current working directory). + +## Configuring the CE from SLATE +You'll want to download the application configuration template: + + slate app get-conf --dev osg-hosted-ce > hosted-ce.yaml + +There are several things that you'll need to edit here. + +### Site Section +First, for the site section you'll want to put in appropriate values that will +ultimately map onto your OSG Topology entry. Let's take a look at this group +first: + + Site: + Group: OSG + Resource: IRISHEP-SSL-UCHICAGO + ResourceGroup: UCHICAGO + Sponsor: osg:100 + Contact: Lincoln Bryant + ContactEmail: lincolnb@uchicago.edu + City: Chicago + Country: US + Latitude: 41.8781 + Longitude: 87.6298 + +Group and Sponsor are safe to leave as defaults if you plan to support OSG +users with your CE. From the [OSG Resource Registration +Documentation](https://opensciencegrid.org/docs/common/registration/), here's +the definition of Resource and ResourceGroup: + +| Level | Definition | +| Resource Group | A logical grouping of resources at a site. Production and testing resources must be placed into separate Resource Groups. | +| Resource | A host belonging to a resource group that provides grid services, e.g. Compute Elements, storage endpoints, or perfSonar hosts. A resource may provide more than one service.) | + +These will ultimately get mapped into the [OSG +Topology](https://topology.opensciencegrid.org/). Following those, you'll need +to update the Contact and location information as appropriate. The contact +information will be used to reach you in case there are any problems with your +CE or site. + +### Cluster Section +Next we'll go through the Cluster section, this section defines the batch +system (e.g. HTCondor, Slurm, PBS, etc) used at your site. The `Users` field is +safe to leave as "osg" if you support primarily OSG jobs. The endpoint value +should be of the form `osg-service-account@your-login-node.site.edu`. + +You will also need to plug in your batch system type. Accepted values are +`condor`, `pbs`, `slurm`, `sge`, and `lsf`. My batch system is `condor`, and +I'll start by allowing a maximum of only 10 jobs at my site, to make sure OSG +pilots are working successfully at a small scale. + +Finally you'll need to refer to the private key we stored in SLATE previously. +I had called mine `ce-private-key`, so my configuration ends up +looking like this: + + Cluster: + Users: osg + Endpoint: osguser@slate-micro-condor.slateci.io + PrivateKeySecret: ce-private-key + Batch: condor + MaxJobs: 10 + +### Storage Section +For the Storage section, you'll want to define the location where the OSG Worker Node Client is installed and the location of temp/scratch space for your workers. The HostedCE SLATE application installs this automatically to the home directory of your service account, so you can just put that here. For temp, I have no particular directory in mind, so I will just use `/tmp`. So for me, it would be: + + Storage: + GridDir: /home/osguser/osg-wn-client + WorkerNodeTemp: /tmp + +### Subcluster Section +The subcluster section will be used for reporting information about workers at +your site. You can read more on the [OSG +Docs](https://opensciencegrid.org/docs/other/configuration-with-osg-configure/#subcluster-resource-entry). +For my cluster, I'll plug in the appropriate values for my workers: + + Subcluster: + Name: RIVER + NodeCount: 75 + Memory: 262144 + CpusPerNode: 2 + CoresPerNode: 48 + +### Squid Section +If you have a Squid service running, you can ensure that your workers will +communicate with your local squid here. I will point my cluster to the local +squid: + + Squid: + Enabled: True + Location: squid.grid.uchicago.edu + +(It's also possible to launch a Squid service through SLATE, see +[here](https://portal.slateci.io/applications/osg-frontier-squid)) + +### CustomizationScript +Your site may need some additional parameters to be added to jobs, such as +`SBATCH` parameters (Slurm) or `Requirements` arguments (HTCondor). You can +insert anything needed via the `CustomizationScript` parameter. For example, +we'll add an `+AccountingGroup` to our OSG jobs: + +``` +CustomizationScript: |+ + #!/bin/sh + echo "+AccontingGroup=osg" +``` + +## Finalizing the configuration + +Now that we've gone through the sections line-by-line, let's look at our completed configuration: + +``` +Site: + Group: OSG + Resource: IRISHEP-SSL-UCHICAGO + ResourceGroup: UCHICAGO + Sponsor: osg:100 + Contact: Lincoln Bryant + ContactEmail: lincolnb@uchicago.edu + City: Chicago + Country: US + Latitude: 41.8781 + Longitude: 87.6298 + +Cluster: + Users: osg + Endpoint: osguser@slate-micro-condor.slateci.io + PrivateKeySecret: lincolnb-bosco # maps to SLATE secret + Batch: condor + MaxJobs: 10 + +Storage: + GridDir: /home/osguser/osg-wn-client + WorkerNodeTemp: /tmp + +Subcluster: + Name: RIVER + NodeCount: 75 + Memory: 262144 + CpusPerNode: 2 + CoresPerNode: 48 + +Squid: + Enabled: True + Location: squid.grid.uchicago.edu + +CustomizationScript: |+ + #!/bin/sh + echo "+AccontingGroup=osg" +``` + +Once we're happy with the configuration, we can ask SLATE to install it: + + slate app install --cluster uchicago-prod --group slate-dev osg-hosted-ce --dev --conf hostedce.yaml + + + +## Testing the HostedCE From 5bed6f19e580a97cb40dedab37ec7a82d526d549 Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 13:01:44 -0500 Subject: [PATCH 02/34] Update 2019-10-21-hosted-ce.md --- _posts/2019-10-21-hosted-ce.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index cb651c1e..d5b93e80 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -32,10 +32,10 @@ on the OSG account for your cluster. For example, if your OSG service account is called `osg`, you'll want to append the contents of `osg-keypair.pub` to `/home/osg/.ssh/authorized_keys`. -The private part of the keypair will need to be stored in SLATE for use by the -CE. To do that: +The private part of the keypair will need to be stored on a SLATE cluster for use by the +CE. In this particular example, I'll be operating under the `slate-dev` group and using the `uchicago-prod` cluster to host a CE pointed at a HTCondor cluster at UChicago. To do that: - slate secret create ce-private-key --from-file=bosco.key=osg-keypair + slate secret create ce-private-key --from-file=bosco.key=osg-keypair --group slate-dev --cluster uchicago-prod Where `ce-private-key` will be the name of the secret, and `osg-keypair` is the path to your private key (assumed to be the current working directory). From f823cb41f8424e3415940010e8f633e4d616dd9c Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 13:06:25 -0500 Subject: [PATCH 03/34] Update 2019-10-21-hosted-ce.md --- _posts/2019-10-21-hosted-ce.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index d5b93e80..7cc02efe 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -189,10 +189,18 @@ CustomizationScript: |+ echo "+AccontingGroup=osg" ``` -Once we're happy with the configuration, we can ask SLATE to install it: +Before deploying the HostedCE, at this point you may want to test your private key and SSH endpoint before deploying the application to SLATE. You may also want to check that job submission is working as expected. For example, here I try to log in with the key (notice that it *does not* require a passphrase, this is a HostedCE requirement!), and I'm able to successfully run a job through my local batch system. - slate app install --cluster uchicago-prod --group slate-dev osg-hosted-ce --dev --conf hostedce.yaml + $ ssh -i osg-keypair osguser@condor.grid.uchicago.edu + Last login: Mon Oct 14 12:51:23 2019 from cdf38.uchicago.edu + [osguser@condor ~]$ condor_run /bin/hostname + htcondor-river-v2-586998c97f-m4vkp + +This is often the first place to start looking when your HostedCE isn't working, so I would encourage you to test this appropriately at your site. +Once we're happy with the configuration and we have tested basic access, we can ask SLATE to install it: + + slate app install --cluster uchicago-prod --group slate-dev osg-hosted-ce --dev --conf hostedce.yaml ## Testing the HostedCE From 45b14d9f1c7b37aa911064c6e15bde24031baacf Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 13:13:11 -0500 Subject: [PATCH 04/34] Update 2019-10-21-hosted-ce.md --- _posts/2019-10-21-hosted-ce.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 7cc02efe..10c2f1fd 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -164,7 +164,7 @@ Site: Cluster: Users: osg - Endpoint: osguser@slate-micro-condor.slateci.io + Endpoint: osguser@condor.grid.uchicago.edu PrivateKeySecret: lincolnb-bosco # maps to SLATE secret Batch: condor MaxJobs: 10 @@ -186,7 +186,7 @@ Squid: CustomizationScript: |+ #!/bin/sh - echo "+AccontingGroup=osg" + echo "+AccountingGroup=osg" ``` Before deploying the HostedCE, at this point you may want to test your private key and SSH endpoint before deploying the application to SLATE. You may also want to check that job submission is working as expected. For example, here I try to log in with the key (notice that it *does not* require a passphrase, this is a HostedCE requirement!), and I'm able to successfully run a job through my local batch system. From cd02167e998600d5c7df5645b9092633766104e2 Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 13:19:57 -0500 Subject: [PATCH 05/34] Update 2019-10-21-hosted-ce.md --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 10c2f1fd..d793e699 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -165,7 +165,7 @@ Site: Cluster: Users: osg Endpoint: osguser@condor.grid.uchicago.edu - PrivateKeySecret: lincolnb-bosco # maps to SLATE secret + PrivateKeySecret: ce-private-key Batch: condor MaxJobs: 10 From 84177ebf59079c3f47045f1e15aa7b84a4b1a9de Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 13:20:15 -0500 Subject: [PATCH 06/34] Update 2019-10-21-hosted-ce.md --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index d793e699..0492d548 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -96,7 +96,7 @@ looking like this: Cluster: Users: osg - Endpoint: osguser@slate-micro-condor.slateci.io + Endpoint: osguser@condor.grid.uchicago.edu PrivateKeySecret: ce-private-key Batch: condor MaxJobs: 10 From 6ca279b607f76a7a459a1aedbc1c9caf8d9ca870 Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 16:22:16 -0500 Subject: [PATCH 07/34] Update 2019-10-21-hosted-ce.md --- _posts/2019-10-21-hosted-ce.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 0492d548..3bcf6aaa 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -70,6 +70,7 @@ Documentation](https://opensciencegrid.org/docs/common/registration/), here's the definition of Resource and ResourceGroup: | Level | Definition | +| -------------- | ----------- | | Resource Group | A logical grouping of resources at a site. Production and testing resources must be placed into separate Resource Groups. | | Resource | A host belonging to a resource group that provides grid services, e.g. Compute Elements, storage endpoints, or perfSonar hosts. A resource may provide more than one service.) | From 009c3457aff33dc587cdb942b64d62a865dd9bdd Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Mon, 14 Oct 2019 16:22:45 -0500 Subject: [PATCH 08/34] Update 2019-10-21-hosted-ce.md --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 3bcf6aaa..b347bd01 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -72,7 +72,7 @@ the definition of Resource and ResourceGroup: | Level | Definition | | -------------- | ----------- | | Resource Group | A logical grouping of resources at a site. Production and testing resources must be placed into separate Resource Groups. | -| Resource | A host belonging to a resource group that provides grid services, e.g. Compute Elements, storage endpoints, or perfSonar hosts. A resource may provide more than one service.) | +| Resource | A host belonging to a resource group that provides grid services, e.g. Compute Elements, storage endpoints, or perfSonar hosts. A resource may provide more than one service.| These will ultimately get mapped into the [OSG Topology](https://topology.opensciencegrid.org/). Following those, you'll need From 2faa867940902b1ed5c4163f3e0924089aaab29d Mon Sep 17 00:00:00 2001 From: Lincoln Bryant Date: Tue, 14 Jan 2020 11:17:04 -0600 Subject: [PATCH 09/34] Update 2019-10-21-hosted-ce.md adding draft tag --- _posts/2019-10-21-hosted-ce.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index b347bd01..38e7a28f 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -1,11 +1,12 @@ --- title: "Deploying and testing an OSG Hosted CE via SLATE" overview: Blog -published: false +published: true permalink: blog/deploy-hosted-ce.html attribution: The SLATE Team layout: post type: markdown +tag: draft --- # Deploying an OSG HostedCE From 07c0a5d02a9d1915c32fef829476bb36873cbf8c Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 11:03:24 -0700 Subject: [PATCH 10/34] deprecate users endpoint batch and maxjobs values merge values from subcluster mostly moved to job router entries --- _posts/2019-10-21-hosted-ce.md | 48 +++++++++------------------------- 1 file changed, 13 insertions(+), 35 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 38e7a28f..cad0cdfc 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -82,26 +82,23 @@ information will be used to reach you in case there are any problems with your CE or site. ### Cluster Section -Next we'll go through the Cluster section, this section defines the batch -system (e.g. HTCondor, Slurm, PBS, etc) used at your site. The `Users` field is -safe to leave as "osg" if you support primarily OSG jobs. The endpoint value -should be of the form `osg-service-account@your-login-node.site.edu`. +Next we'll go through the Cluster section, this section defines some of the hardware specifications on the remote side. Memory should be the total per node memory available on the remote cluster. By default this is interpreted as megabytes, but you can format this as `256G` if you wish. You can tailor the amount of memory to the lowest common denominator if you have a remote cluster with different kinds of nodes. The same principle applies for the CoresPerNode field. MaxWallTime is the maxmimum allowed CPU walltime for the job expressed in minutes by default (in this case it is 72 hours). -You will also need to plug in your batch system type. Accepted values are -`condor`, `pbs`, `slurm`, `sge`, and `lsf`. My batch system is `condor`, and -I'll start by allowing a maximum of only 10 jobs at my site, to make sure OSG -pilots are working successfully at a small scale. +You can read more on the [OSG Docs](https://opensciencegrid.org/docs/other/configuration-with-osg-configure/#subcluster-resource-entry) + +VOs are virtual orginizations within the OSG, they correspond to different research groups. AllowedVOs let's us specify which groups will run jobs through our CE. Finally you'll need to refer to the private key we stored in SLATE previously. I had called mine `ce-private-key`, so my configuration ends up looking like this: Cluster: - Users: osg - Endpoint: osguser@condor.grid.uchicago.edu PrivateKeySecret: ce-private-key - Batch: condor - MaxJobs: 10 + Memory: 262144 + CoresPerNode: 96 + MaxWallTime: 1440 + AllowedVOs: osg, cms, atlas, glow, hcc, fermilab, ligo, virgo, sdcc, sphenix, gluex, icecube, xenon + ### Storage Section For the Storage section, you'll want to define the location where the OSG Worker Node Client is installed and the location of temp/scratch space for your workers. The HostedCE SLATE application installs this automatically to the home directory of your service account, so you can just put that here. For temp, I have no particular directory in mind, so I will just use `/tmp`. So for me, it would be: @@ -110,18 +107,6 @@ For the Storage section, you'll want to define the location where the OSG Worker GridDir: /home/osguser/osg-wn-client WorkerNodeTemp: /tmp -### Subcluster Section -The subcluster section will be used for reporting information about workers at -your site. You can read more on the [OSG -Docs](https://opensciencegrid.org/docs/other/configuration-with-osg-configure/#subcluster-resource-entry). -For my cluster, I'll plug in the appropriate values for my workers: - - Subcluster: - Name: RIVER - NodeCount: 75 - Memory: 262144 - CpusPerNode: 2 - CoresPerNode: 48 ### Squid Section If you have a Squid service running, you can ensure that your workers will @@ -165,23 +150,16 @@ Site: Longitude: 87.6298 Cluster: - Users: osg - Endpoint: osguser@condor.grid.uchicago.edu PrivateKeySecret: ce-private-key - Batch: condor - MaxJobs: 10 + Memory: 262144 + CoresPerNode: 96 + MaxWallTime: 1440 + AllowedVOs: osg, cms, atlas, glow, hcc, fermilab, ligo, virgo, sdcc, sphenix, gluex, icecube, xenon Storage: GridDir: /home/osguser/osg-wn-client WorkerNodeTemp: /tmp -Subcluster: - Name: RIVER - NodeCount: 75 - Memory: 262144 - CpusPerNode: 2 - CoresPerNode: 48 - Squid: Enabled: True Location: squid.grid.uchicago.edu From c53933cb3768dfb69dd23cc48a9d389b1f9756e3 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 11:04:14 -0700 Subject: [PATCH 11/34] deprecate enabled value for squid --- _posts/2019-10-21-hosted-ce.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index cad0cdfc..51704215 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -114,7 +114,6 @@ communicate with your local squid here. I will point my cluster to the local squid: Squid: - Enabled: True Location: squid.grid.uchicago.edu (It's also possible to launch a Squid service through SLATE, see @@ -161,7 +160,6 @@ Storage: WorkerNodeTemp: /tmp Squid: - Enabled: True Location: squid.grid.uchicago.edu CustomizationScript: |+ From 5f447c6182fd336fd8f91255974726e7f7a00ace Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 11:10:20 -0700 Subject: [PATCH 12/34] add updated values to end config TODO add explanatory sections --- _posts/2019-10-21-hosted-ce.md | 48 +++++++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 3 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 51704215..add64b38 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -162,9 +162,51 @@ Storage: Squid: Location: squid.grid.uchicago.edu -CustomizationScript: |+ - #!/bin/sh - echo "+AccountingGroup=osg" +Networking: + Hostname: "" + RequestIP: null + +HTCondorCeConfig: |+ + JOB_ROUTER_ENTRIES @=jre + [ + GridResource = "batch slurm osg-slate@lonepeak1.chpc.utah.edu"; + Requirements = (Owner == "osg-slate"); + ] + @jre + +BoscoOverrides: + Enabled: true + GitEndpoint: git@gitlab.mwt2.org:osg/hosted-ce-config.git + RepoNeedsPrivKey: true + GitKeySecret: mwtGitLabSecret + +HTTPLogger: + Enabled: true + +VomsmapOverride: |+ + "/osg/Role=NULL/Capability=NULL" osg01 + "/GLOW/Role=htpc/Capability=NULL" osg02 + "/hcc/Role=NULL/Capability=NULL" osg03 + "/cms/*" osg04 + "/fermilab/*" osg05 + "/osg/ligo/Role=NULL/Capability=NULL" osg07 + "/virgo/ligo/Role=NULL/Capability=NULL" osg07 + "/sdcc/Role=NULL/Capability=NULL" osg08 + "/sphenix/Role=NULL/Capability=NULL" osg08 + "/atlas/*" osg09 + "/Gluex/Role=NULL/Capability=NULL" osg10 + "/dune/Role=pilot/Capability=NULL" osg11 + "/icecube/Role=pilot/Capability=NULL" osg12 + "/xenon.biggrid.nl/Role=NULL/Capability=NULL" osg13 + +GridmapOverride: |+ + "/DC=foo/DC=bar/OU=Organic Units/OU=Users/CN=YourUserName" osg + +Certificate: + Secret: null + +Developer: + Enabled: false ``` Before deploying the HostedCE, at this point you may want to test your private key and SSH endpoint before deploying the application to SLATE. You may also want to check that job submission is working as expected. For example, here I try to log in with the key (notice that it *does not* require a passphrase, this is a HostedCE requirement!), and I'm able to successfully run a job through my local batch system. From 36d572e7568fd448208c7b6f42a30e613c716e3b Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 11:22:05 -0700 Subject: [PATCH 13/34] stub sections for new config options --- _posts/2019-10-21-hosted-ce.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index add64b38..d25b820c 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -119,6 +119,30 @@ squid: (It's also possible to launch a Squid service through SLATE, see [here](https://portal.slateci.io/applications/osg-frontier-squid)) +### Netowrking Sections +TODO + +### HTCondorCeConfig Section +TODO + +### BoscoOverrides Section +TODO + +### HTTPLogger Section +TODO + +### VomsmapOverride Section +TODO + +### GridmapOverride Section +TODO + +### Certficate Section +TODO + +### Developer Section +TODO + ### CustomizationScript Your site may need some additional parameters to be added to jobs, such as `SBATCH` parameters (Slurm) or `Requirements` arguments (HTCondor). You can From 38fd35fece3009b3b7cb75932380a185d12e0089 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 11:23:49 -0700 Subject: [PATCH 14/34] remove errant comma --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index d25b820c..527d2386 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -82,7 +82,7 @@ information will be used to reach you in case there are any problems with your CE or site. ### Cluster Section -Next we'll go through the Cluster section, this section defines some of the hardware specifications on the remote side. Memory should be the total per node memory available on the remote cluster. By default this is interpreted as megabytes, but you can format this as `256G` if you wish. You can tailor the amount of memory to the lowest common denominator if you have a remote cluster with different kinds of nodes. The same principle applies for the CoresPerNode field. MaxWallTime is the maxmimum allowed CPU walltime for the job expressed in minutes by default (in this case it is 72 hours). +Next we'll go through the Cluster section, this section defines some of the hardware specifications on the remote side. Memory should be the total per node memory available on the remote cluster. By default this is interpreted as megabytes but you can format this as `256G` if you wish. You can tailor the amount of memory to the lowest common denominator if you have a remote cluster with different kinds of nodes. The same principle applies for the CoresPerNode field. MaxWallTime is the maxmimum allowed CPU walltime for the job expressed in minutes by default (in this case it is 72 hours). You can read more on the [OSG Docs](https://opensciencegrid.org/docs/other/configuration-with-osg-configure/#subcluster-resource-entry) From bbe6ac9ee9b58cfc2526e9899f59da1e58efe882 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 11:27:09 -0700 Subject: [PATCH 15/34] fix job router entry for consistency --- _posts/2019-10-21-hosted-ce.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 527d2386..8ec06df0 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -193,8 +193,8 @@ Networking: HTCondorCeConfig: |+ JOB_ROUTER_ENTRIES @=jre [ - GridResource = "batch slurm osg-slate@lonepeak1.chpc.utah.edu"; - Requirements = (Owner == "osg-slate"); + GridResource = "batch condor osguser@condor.grid.uchicago.edu"; + Requirements = (Owner == "osguser"); ] @jre From a37412ea731b676de4c5547e554d64d3a8052b51 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 14:51:10 -0700 Subject: [PATCH 16/34] add networking sub section --- _posts/2019-10-21-hosted-ce.md | 161 ++++++++++++++++----------------- 1 file changed, 79 insertions(+), 82 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 8ec06df0..556cdc4c 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -34,11 +34,11 @@ is called `osg`, you'll want to append the contents of `osg-keypair.pub` to `/home/osg/.ssh/authorized_keys`. The private part of the keypair will need to be stored on a SLATE cluster for use by the -CE. In this particular example, I'll be operating under the `slate-dev` group and using the `uchicago-prod` cluster to host a CE pointed at a HTCondor cluster at UChicago. To do that: +CE. In this particular example, I'll be operating under the `slate-dev` group and using the `uutah-prod` cluster to host a CE pointed at a SLURM cluster at University of Utah. To do that: - slate secret create ce-private-key --from-file=bosco.key=osg-keypair --group slate-dev --cluster uchicago-prod + slate secret create utah-lp-hostedce-privkey --from-file=bosco.key=osg-keypair --group slate-dev --cluster uutah-prod -Where `ce-private-key` will be the name of the secret, and `osg-keypair` is the +Where `utah-lp-hostedce-privkey` will be the name of the secret, and `osg-keypair` is the path to your private key (assumed to be the current working directory). ## Configuring the CE from SLATE @@ -52,18 +52,17 @@ There are several things that you'll need to edit here. First, for the site section you'll want to put in appropriate values that will ultimately map onto your OSG Topology entry. Let's take a look at this group first: - + Site: - Group: OSG - Resource: IRISHEP-SSL-UCHICAGO - ResourceGroup: UCHICAGO - Sponsor: osg:100 - Contact: Lincoln Bryant - ContactEmail: lincolnb@uchicago.edu - City: Chicago - Country: US - Latitude: 41.8781 - Longitude: 87.6298 + Resource: SLATE_US_UUTAH_LONEPEAK + ResourceGroup: CHPC Group + Sponsor: osg:100 + Contact: Mitchell Steinman + ContactEmail: chpc-osg-support@lists.utah.edu + City: Salt Lake City + Country: United States + Latitude: 40.7608 + Longitude: -111.891 Group and Sponsor are safe to leave as defaults if you plan to support OSG users with your CE. From the [OSG Resource Registration @@ -82,45 +81,53 @@ information will be used to reach you in case there are any problems with your CE or site. ### Cluster Section -Next we'll go through the Cluster section, this section defines some of the hardware specifications on the remote side. Memory should be the total per node memory available on the remote cluster. By default this is interpreted as megabytes but you can format this as `256G` if you wish. You can tailor the amount of memory to the lowest common denominator if you have a remote cluster with different kinds of nodes. The same principle applies for the CoresPerNode field. MaxWallTime is the maxmimum allowed CPU walltime for the job expressed in minutes by default (in this case it is 72 hours). +Next we'll go through the Cluster section, this section defines some of the hardware specifications on the remote side. Memory should be the total per node memory available on the remote cluster. By default this is interpreted as megabytes but you can format this as `24G` if you wish. You can tailor the amount of memory to the lowest common denominator if you have a remote cluster with different kinds of nodes. The same principle applies for the CoresPerNode field. MaxWallTime is the maxmimum allowed CPU walltime for the job expressed in minutes by default (in this case it is 72 hours). You can read more on the [OSG Docs](https://opensciencegrid.org/docs/other/configuration-with-osg-configure/#subcluster-resource-entry) -VOs are virtual orginizations within the OSG, they correspond to different research groups. AllowedVOs let's us specify which groups will run jobs through our CE. +VOs are virtual orginizations within the OSG and they correspond to different research groups. AllowedVOs let's us specify which groups will run jobs through our CE. Finally you'll need to refer to the private key we stored in SLATE previously. -I had called mine `ce-private-key`, so my configuration ends up +I had called mine `utah-lp-hostedce-privkey`, so my configuration ends up looking like this: Cluster: - PrivateKeySecret: ce-private-key - Memory: 262144 - CoresPerNode: 96 - MaxWallTime: 1440 + PrivateKeySecret: utah-lp-hostedce-privkey # maps to SLATE secret + Memory: 24576 + CoresPerNode: 4 + MaxWallTime: 4320 AllowedVOs: osg, cms, atlas, glow, hcc, fermilab, ligo, virgo, sdcc, sphenix, gluex, icecube, xenon + ### Storage Section -For the Storage section, you'll want to define the location where the OSG Worker Node Client is installed and the location of temp/scratch space for your workers. The HostedCE SLATE application installs this automatically to the home directory of your service account, so you can just put that here. For temp, I have no particular directory in mind, so I will just use `/tmp`. So for me, it would be: +For the Storage section, you'll want to define the location where the OSG Worker Node Client is installed and the location of temp/scratch space for your workers. The OSG Worker Node Client is expected to be installed in the osguser's home directory in a subdirectory called `bosco-osg-wn-client`. You need to expand the absolute path of the user's home directory. In my case, this is under a shared file system. The HostedCE SLATE application will install this Worker Node Client directory for you. For temp, I have a specific scratch directory I want the OSG to use. `/tmp` is also a typical location to use. Storage: - GridDir: /home/osguser/osg-wn-client - WorkerNodeTemp: /tmp + GridDir: /uufs/chpc.utah.edu/common/home/osguserl/bosco-osg-wn-client + WorkerNodeTemp: /scratch/local/.osgscratch + ### Squid Section If you have a Squid service running, you can ensure that your workers will communicate with your local squid here. I will point my cluster to the local -squid: +squid I have deployed with SLATE: Squid: - Location: squid.grid.uchicago.edu + Location: sl-uu-es1.slateci.io:31726 (It's also possible to launch a Squid service through SLATE, see [here](https://portal.slateci.io/applications/osg-frontier-squid)) ### Netowrking Sections -TODO +The HostedCE application requires both forward and reverse DNS resolution to be enabled for its publicly routable IP. Most SLATE clusters come pre-configured with a handful of "LoadBalancer" IP addresses that can be allocated automatically to different applications. You must set-up the DNS records for this address so it is a good idea to request a specific address from the pool. For Utah my network config looks like this: + + + Networking: + Hostname: "sl-uu-hce2.slateci.io" + RequestIP: 155.101.6.238 + ### HTCondorCeConfig Section TODO @@ -143,94 +150,84 @@ TODO ### Developer Section TODO -### CustomizationScript -Your site may need some additional parameters to be added to jobs, such as -`SBATCH` parameters (Slurm) or `Requirements` arguments (HTCondor). You can -insert anything needed via the `CustomizationScript` parameter. For example, -we'll add an `+AccountingGroup` to our OSG jobs: - -``` -CustomizationScript: |+ - #!/bin/sh - echo "+AccontingGroup=osg" -``` - ## Finalizing the configuration Now that we've gone through the sections line-by-line, let's look at our completed configuration: ``` +Instance: "lonepeak" + Site: - Group: OSG - Resource: IRISHEP-SSL-UCHICAGO - ResourceGroup: UCHICAGO + Resource: SLATE_US_UUTAH_LONEPEAK + ResourceGroup: CHPC Group Sponsor: osg:100 - Contact: Lincoln Bryant - ContactEmail: lincolnb@uchicago.edu - City: Chicago - Country: US - Latitude: 41.8781 - Longitude: 87.6298 + Contact: Mitchell Steinman + ContactEmail: chpc-osg-support@lists.utah.edu + City: Salt Lake City + Country: United States + Latitude: 40.7608 + Longitude: -111.891 Cluster: - PrivateKeySecret: ce-private-key - Memory: 262144 - CoresPerNode: 96 - MaxWallTime: 1440 + PrivateKeySecret: utah-lp-hostedce-privkey # maps to SLATE secret + Memory: 24000 + CoresPerNode: 4 + MaxWallTime: 4320 AllowedVOs: osg, cms, atlas, glow, hcc, fermilab, ligo, virgo, sdcc, sphenix, gluex, icecube, xenon Storage: - GridDir: /home/osguser/osg-wn-client - WorkerNodeTemp: /tmp + GridDir: /uufs/chpc.utah.edu/common/home/osguserl/bosco-osg-wn-client + WorkerNodeTemp: /scratch/local/.osgscratch Squid: - Location: squid.grid.uchicago.edu + Location: sl-uu-es1.slateci.io:31726 Networking: - Hostname: "" - RequestIP: null - + Hostname: "sl-uu-hce2.slateci.io" + RequestIP: 155.101.6.238 + HTCondorCeConfig: |+ JOB_ROUTER_ENTRIES @=jre [ - GridResource = "batch condor osguser@condor.grid.uchicago.edu"; - Requirements = (Owner == "osguser"); + GridResource = "batch slurm osguserl@lonepeak1.chpc.utah.edu"; + Requirements = (Owner == "osguserl"); ] @jre - + BoscoOverrides: Enabled: true - GitEndpoint: git@gitlab.mwt2.org:osg/hosted-ce-config.git - RepoNeedsPrivKey: true - GitKeySecret: mwtGitLabSecret - + GitEndpoint: "https://github.com/slateci/utah-bosco.git" + RepoNeedsPrivKey: false + GitKeySecret: none + HTTPLogger: Enabled: true - + VomsmapOverride: |+ - "/osg/Role=NULL/Capability=NULL" osg01 - "/GLOW/Role=htpc/Capability=NULL" osg02 - "/hcc/Role=NULL/Capability=NULL" osg03 - "/cms/*" osg04 - "/fermilab/*" osg05 - "/osg/ligo/Role=NULL/Capability=NULL" osg07 - "/virgo/ligo/Role=NULL/Capability=NULL" osg07 - "/sdcc/Role=NULL/Capability=NULL" osg08 - "/sphenix/Role=NULL/Capability=NULL" osg08 - "/atlas/*" osg09 - "/Gluex/Role=NULL/Capability=NULL" osg10 - "/dune/Role=pilot/Capability=NULL" osg11 - "/icecube/Role=pilot/Capability=NULL" osg12 - "/xenon.biggrid.nl/Role=NULL/Capability=NULL" osg13 - + "/osg/Role=NULL/Capability=NULL" osguserl + "/GLOW/Role=htpc/Capability=NULL" osguserl + "/hcc/Role=NULL/Capability=NULL" osguserl + "/cms/*" osguserl + "/fermilab/*" osguserl + "/osg/ligo/Role=NULL/Capability=NULL" osguserl + "/virgo/ligo/Role=NULL/Capability=NULL" osguserl + "/sdcc/Role=NULL/Capability=NULL" osguserl + "/sphenix/Role=NULL/Capability=NULL" osguserl + "/atlas/*" osguserl + "/Gluex/Role=NULL/Capability=NULL" osguserl + "/dune/Role=pilot/Capability=NULL" osguserl + "/icecube/Role=pilot/Capability=NULL" osguserl + "/xenon.biggrid.nl/Role=NULL/Capability=NULL" osguserl + GridmapOverride: |+ "/DC=foo/DC=bar/OU=Organic Units/OU=Users/CN=YourUserName" osg - + Certificate: Secret: null Developer: Enabled: false + ``` Before deploying the HostedCE, at this point you may want to test your private key and SSH endpoint before deploying the application to SLATE. You may also want to check that job submission is working as expected. For example, here I try to log in with the key (notice that it *does not* require a passphrase, this is a HostedCE requirement!), and I'm able to successfully run a job through my local batch system. From 5d236d4e651ddc562b3a5e42ac9e401a18e47c8b Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 14:58:38 -0700 Subject: [PATCH 17/34] add job router entry configuration --- _posts/2019-10-21-hosted-ce.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 556cdc4c..da6fedf2 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -130,7 +130,26 @@ The HostedCE application requires both forward and reverse DNS resolution to be ### HTCondorCeConfig Section -TODO +The HTCondorCeConfig file contains additional configuration for the CE itself. Most importantly the required JOB_ROUTER_ENTRIES. This is the configuration that allows the CE to route jobs to your remote cluster. It has the following format: + + JOB_ROUTER_ENTRIES @=jre + [ + GridResource = "batch @"; + Requirements = (Owner == ""); + ] + @jre + +This is my CE configuration: + + JOB_ROUTER_ENTRIES @=jre + [ + GridResource = "batch slurm osguserl@lonepeak1.chpc.utah.edu"; + Requirements = (Owner == "osguserl"); + ] + @jre + +My remote user is `osguserl` and jobs will be routed to the `lonepeak1.chpc.utah.edu` endpoint. Finally the remote cluster is running SLURM. + ### BoscoOverrides Section TODO From 8534086bd2c6f2e67b3597d6ccf36b5fb186d953 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 15:32:31 -0700 Subject: [PATCH 18/34] add narrative for the rest of the new configs --- _posts/2019-10-21-hosted-ce.md | 66 ++++++++++++++++++++++++++++++---- 1 file changed, 59 insertions(+), 7 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index da6fedf2..d7340bde 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -150,24 +150,76 @@ This is my CE configuration: My remote user is `osguserl` and jobs will be routed to the `lonepeak1.chpc.utah.edu` endpoint. Finally the remote cluster is running SLURM. +There is more extensive documentation on the [Job Router](https://opensciencegrid.org/docs/compute-element/job-router-recipes/) ### BoscoOverrides Section -TODO +The BoscoOverrides section provides a mechanism to override default configuration placed on the remote cluster for the OSG Worker Node Client. This can include things like the local path to the batch system's executables and additional configuration options for the batch system. + +This will vary depending on your batch system, but I did need to override the slurm_binpath and add some SBATCH parameters. All the overrides are expected to be placed in a git repository with a subdirectory format that matches `/bosco-override` in my case `SLATE_US_UUTAH_LONEPEAK/bosco-override`. + +It may take some trial and error to get the correct overrides in place. The general proccess for this is to deploy the CE, then check the logs on the applications HTTP Log exporter to see what must be changed. Finally redpeploy with the updated overrides. + +You can use a private git repo, and provide the key to the application as a SLATE secret. My configuration with a public git repo is: + + BoscoOverrides: + Enabled: true + GitEndpoint: "https://github.com/slateci/utah-bosco.git" + RepoNeedsPrivKey: false + GitKeySecret: none ### HTTPLogger Section -TODO +This allows you to turn the HTTP logging side car on and off. This will allow you to view the CE logs from your browser. + + HTTPLogger: + Enabled: true + +You can get the endpoint for your logger by running `slate instance info `, and the randomly generated credentials will be written to the sidecar's logs. + +`slate instance logs ` ### VomsmapOverride Section -TODO +Each VO that is enabled must be mapped to a user on the remote cluster. It is standard to create a user for each VO you intend to support. It is possible to map each VO to the same remote user, which I have done here: + + VomsmapOverride: |+ + "/osg/Role=NULL/Capability=NULL" osguserl + "/GLOW/Role=htpc/Capability=NULL" osguserl + "/hcc/Role=NULL/Capability=NULL" osguserl + "/cms/*" osguserl + "/fermilab/*" osguserl + "/osg/ligo/Role=NULL/Capability=NULL" osguserl + "/virgo/ligo/Role=NULL/Capability=NULL" osguserl + "/sdcc/Role=NULL/Capability=NULL" osguserl + "/sphenix/Role=NULL/Capability=NULL" osguserl + "/atlas/*" osguserl + "/Gluex/Role=NULL/Capability=NULL" osguserl + "/dune/Role=pilot/Capability=NULL" osguserl + "/icecube/Role=pilot/Capability=NULL" osguserl + "/xenon.biggrid.nl/Role=NULL/Capability=NULL" osguserl + ### GridmapOverride Section -TODO +The GridmapOverride will allow you to add your own personal grid proxy to the CE. This is for the purpose of testing basic job submission. + +You can obtain one with your institutional credential at [cilogon.org](cilogon.org) + + GridmapOverride: |+ + "/DC=org/DC=cilogon/C=US/O=University of Utah/CN=HENRY STEINMAN A14364946" osguserl ### Certficate Section -TODO +Each time the CE is deployed it requests a new certificate from Lets Encrypt, which has rate limits to prevent DOS attacks. This means that if you are redeploying a CE frequently for troubleshooting purposes, you may experience the rate limit. + +It is possible to save the certificate (hostkey.pem and hostcert.pem) and store these as a SLATE secret for re-use. This circumvents the rate limit. + +I will leave it disabled, because my configuration is stable. + + Certificate: + Secret: null ### Developer Section -TODO +Simply disable this. It is in place for the purpose of ITB Hosts. + + Developer: + Enabled: false ## Finalizing the configuration @@ -239,7 +291,7 @@ VomsmapOverride: |+ "/xenon.biggrid.nl/Role=NULL/Capability=NULL" osguserl GridmapOverride: |+ - "/DC=foo/DC=bar/OU=Organic Units/OU=Users/CN=YourUserName" osg + "/DC=org/DC=cilogon/C=US/O=University of Utah/CN=HENRY STEINMAN A14364946" osguserl Certificate: Secret: null From f73ddbbe8b6f1c3888545746f5237e3992ffb626 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Tue, 21 Jan 2020 16:05:06 -0700 Subject: [PATCH 19/34] Add testing section --- _posts/2019-10-21-hosted-ce.md | 58 +++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 12 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index d7340bde..8bf79673 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -81,7 +81,11 @@ information will be used to reach you in case there are any problems with your CE or site. ### Cluster Section -Next we'll go through the Cluster section, this section defines some of the hardware specifications on the remote side. Memory should be the total per node memory available on the remote cluster. By default this is interpreted as megabytes but you can format this as `24G` if you wish. You can tailor the amount of memory to the lowest common denominator if you have a remote cluster with different kinds of nodes. The same principle applies for the CoresPerNode field. MaxWallTime is the maxmimum allowed CPU walltime for the job expressed in minutes by default (in this case it is 72 hours). +Next we'll go through the Cluster section, this section defines some of the hardware specifications on the remote side. + +Memory should be the total per node memory available on the remote cluster. By default this is interpreted as megabytes but you can format this as `24G` if you wish. You can tailor the amount of memory to the lowest common denominator if you have a remote cluster with different kinds of nodes. The same principle applies for the CoresPerNode field. + +MaxWallTime is the maxmimum allowed walltime for the job and is expressed in minutes by default. You can read more on the [OSG Docs](https://opensciencegrid.org/docs/other/configuration-with-osg-configure/#subcluster-resource-entry) @@ -101,7 +105,11 @@ looking like this: ### Storage Section -For the Storage section, you'll want to define the location where the OSG Worker Node Client is installed and the location of temp/scratch space for your workers. The OSG Worker Node Client is expected to be installed in the osguser's home directory in a subdirectory called `bosco-osg-wn-client`. You need to expand the absolute path of the user's home directory. In my case, this is under a shared file system. The HostedCE SLATE application will install this Worker Node Client directory for you. For temp, I have a specific scratch directory I want the OSG to use. `/tmp` is also a typical location to use. +For the Storage section, you'll want to define the path where the OSG Worker Node Client is installed as well as the location of temp/scratch space for your workers. + +The OSG Worker Node Client is expected to be installed in the osguser's home directory in a subdirectory called `bosco-osg-wn-client`. You need to expand the absolute path of the user's home directory. In my case, this is under a shared file system. The HostedCE SLATE application will install this Worker Node Client directory for you. + +For temp, I have a specific scratch directory I want the OSG jobs to use. `/tmp` is also a typical location for this. Storage: GridDir: /uufs/chpc.utah.edu/common/home/osguserl/bosco-osg-wn-client @@ -117,20 +125,25 @@ squid I have deployed with SLATE: Squid: Location: sl-uu-es1.slateci.io:31726 + + (It's also possible to launch a Squid service through SLATE, see [here](https://portal.slateci.io/applications/osg-frontier-squid)) ### Netowrking Sections -The HostedCE application requires both forward and reverse DNS resolution to be enabled for its publicly routable IP. Most SLATE clusters come pre-configured with a handful of "LoadBalancer" IP addresses that can be allocated automatically to different applications. You must set-up the DNS records for this address so it is a good idea to request a specific address from the pool. For Utah my network config looks like this: +The HostedCE application requires both forward and reverse DNS resolution for its publicly routable IP. Most SLATE clusters come pre-configured with a handful of "LoadBalancer" IP addresses that can be allocated automatically to different applications. You must set-up the DNS records for this address so it is a good idea to request a specific address from the pool. This is my network configuration: Networking: Hostname: "sl-uu-hce2.slateci.io" RequestIP: 155.101.6.238 + +It simply consists of a RequestIP and corresponding hostname. + ### HTCondorCeConfig Section -The HTCondorCeConfig file contains additional configuration for the CE itself. Most importantly the required JOB_ROUTER_ENTRIES. This is the configuration that allows the CE to route jobs to your remote cluster. It has the following format: +The HTCondorCeConfig file contains additional configuration for the CE itself. Most importantly it contains the required JOB_ROUTER_ENTRIES section. This is the configuration that allows the CE to route jobs to your remote cluster. It has the following format: JOB_ROUTER_ENTRIES @=jre [ @@ -139,7 +152,7 @@ The HTCondorCeConfig file contains additional configuration for the CE itself. M ] @jre -This is my CE configuration: +My remote user is `osguserl` and jobs will be routed to the `lonepeak1.chpc.utah.edu` endpoint and the remote cluster is running SLURM. So my HTCondorCeConfig looks like this: JOB_ROUTER_ENTRIES @=jre [ @@ -147,19 +160,18 @@ This is my CE configuration: Requirements = (Owner == "osguserl"); ] @jre - -My remote user is `osguserl` and jobs will be routed to the `lonepeak1.chpc.utah.edu` endpoint. Finally the remote cluster is running SLURM. + There is more extensive documentation on the [Job Router](https://opensciencegrid.org/docs/compute-element/job-router-recipes/) ### BoscoOverrides Section -The BoscoOverrides section provides a mechanism to override default configuration placed on the remote cluster for the OSG Worker Node Client. This can include things like the local path to the batch system's executables and additional configuration options for the batch system. +The BoscoOverrides section provides a mechanism to override default configuration placed on the remote cluster for the OSG Worker Node Client. This can include things like the local path to the batch system's executables and additional job submission parameters for the batch system. -This will vary depending on your batch system, but I did need to override the slurm_binpath and add some SBATCH parameters. All the overrides are expected to be placed in a git repository with a subdirectory format that matches `/bosco-override` in my case `SLATE_US_UUTAH_LONEPEAK/bosco-override`. +This will vary depending on your batch system. I did needed to override the slurm_binpath and add some SBATCH parameters. All the overrides are expected to be placed in a git repository with a subdirectory format that matches `/bosco-override` in my case `SLATE_US_UUTAH_LONEPEAK/bosco-override`. -It may take some trial and error to get the correct overrides in place. The general proccess for this is to deploy the CE, then check the logs on the applications HTTP Log exporter to see what must be changed. Finally redpeploy with the updated overrides. +It may take some trial and error to get the correct overrides in place. The general proccess for this is to deploy the CE, then check the logs on the application's HTTP Log exporter to see what must be changed. Finally re-dpeploy with the updated overrides. -You can use a private git repo, and provide the key to the application as a SLATE secret. My configuration with a public git repo is: +You can use a private git repo and provide the key to the application as a SLATE secret. My configuration with a public git repo is: BoscoOverrides: Enabled: true @@ -168,7 +180,7 @@ You can use a private git repo, and provide the key to the application as a SLAT GitKeySecret: none ### HTTPLogger Section -This allows you to turn the HTTP logging side car on and off. This will allow you to view the CE logs from your browser. +This allows you to turn toggle HTTP logging side car. When it is enabled, it will allow you to view the CE logs from your browser. HTTPLogger: Enabled: true @@ -316,3 +328,25 @@ Once we're happy with the configuration and we have tested basic access, we can ## Testing the HostedCE + +In order to test end-to-end job submission through the CE you will need a valid grid proxy and the condor tools. First you must obtain a certificate. + +An easy way to do that is through [CILogon](cilogon.org). + +You will need to install `htcondor-ce-client` on the machine you would like to submit from. + +On EL7 enable the OSG yum repos + +`yum install https://repo.opensciencegrid.org/osg/3.5/osg-3.5-el7-release-latest.rpm && yum update` + +Then install the tools + +`yum install htcondor-ce-client` + +You should be able to use your cert to initialize your grid proxy + +`voms-proxy-init` + +If that was successful you should be able to run a job trace against your CE, which will trace the end-to-end submission of a small test job. + +`condor_ce_trace --debug sl-uu-hce2.slateci.io` From 05a8eeb2faa914f6087718b1eed86e6c626f8aae Mon Sep 17 00:00:00 2001 From: amblinalong Date: Wed, 22 Jan 2020 09:14:43 -0700 Subject: [PATCH 20/34] fixed cilogon link to point to actual website --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 8bf79673..614cdd4e 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -331,7 +331,7 @@ Once we're happy with the configuration and we have tested basic access, we can In order to test end-to-end job submission through the CE you will need a valid grid proxy and the condor tools. First you must obtain a certificate. -An easy way to do that is through [CILogon](cilogon.org). +An easy way to do that is through [CILogon](https://cilogon.org/). You will need to install `htcondor-ce-client` on the machine you would like to submit from. From fcb3513fc7dea6bc3bc1150295317fe73db9a973 Mon Sep 17 00:00:00 2001 From: amblinalong Date: Wed, 22 Jan 2020 09:20:45 -0700 Subject: [PATCH 21/34] Fixed another cilogon link --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 614cdd4e..32d8eb82 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -212,7 +212,7 @@ Each VO that is enabled must be mapped to a user on the remote cluster. It is st ### GridmapOverride Section The GridmapOverride will allow you to add your own personal grid proxy to the CE. This is for the purpose of testing basic job submission. -You can obtain one with your institutional credential at [cilogon.org](cilogon.org) +You can obtain one with your institutional credential at [cilogon.org](https://cilogon.org/) GridmapOverride: |+ "/DC=org/DC=cilogon/C=US/O=University of Utah/CN=HENRY STEINMAN A14364946" osguserl From 507f258991ea7fd6152d3f38f80d93ce3eb976bb Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 09:52:25 -0700 Subject: [PATCH 22/34] add instructions to conver the grid proxy certs --- _posts/2019-10-21-hosted-ce.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 32d8eb82..d3656829 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -333,6 +333,18 @@ In order to test end-to-end job submission through the CE you will need a valid An easy way to do that is through [CILogon](https://cilogon.org/). +Create a cert and download it. + +Next you will need to convert it to PKCS12 format for voms. + +`openssl pkcs12 -in usercred.p12 -nocerts -out hostkey.pem` + +`openssl pkcs12 -in usercred.p12 -nocerts -nodes -out hostkey.pem` + +Be sure that both files have the correct file permissions + +`chmod 600 hostkey.pem && chmod 600 hostcert.pem` + You will need to install `htcondor-ce-client` on the machine you would like to submit from. On EL7 enable the OSG yum repos @@ -341,11 +353,13 @@ On EL7 enable the OSG yum repos Then install the tools -`yum install htcondor-ce-client` +`yum clean all; yum install htcondor-ce-client` You should be able to use your cert to initialize your grid proxy -`voms-proxy-init` +`voms-proxy-init -cert hostcert.pem -key hostkey.pem --debug` + +Here I use the `--debug` flag because `voms-proxy-init` won't give us very helpful output, if it fails. If that was successful you should be able to run a job trace against your CE, which will trace the end-to-end submission of a small test job. From caff8532c96f5118f2f9cb7b0d966982d70c779d Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:04:04 -0700 Subject: [PATCH 23/34] Add conclusion --- _posts/2019-10-21-hosted-ce.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index d3656829..96104ffb 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -364,3 +364,5 @@ Here I use the `--debug` flag because `voms-proxy-init` won't give us very helpf If that was successful you should be able to run a job trace against your CE, which will trace the end-to-end submission of a small test job. `condor_ce_trace --debug sl-uu-hce2.slateci.io` + +This command will output a great deal of helpful information about the job submission, it's status and the eventual result. If the jobs sits idle on the remote cluster for too long, the command may time out. From 05d91d6c8ec4bc61f051b2b939a2e5dbe058e16a Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:15:24 -0700 Subject: [PATCH 24/34] grammatical fixes --- _posts/2019-10-21-hosted-ce.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 96104ffb..be10ccd9 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -10,8 +10,13 @@ tag: draft --- # Deploying an OSG HostedCE +An OSG Hosted Compute Element is an application that allows a site to contribute idle HPC or HTC compute resources to the Open Science Grid. The CE is responsible for recieving jobs from the grid and routing them to your local cluster(s). These are pre-emptable jobs that will run when the cluster resources would have otherwise gone idle. + +Using SLATE you can deploy the HostedCE locally at your site, and easily scale to as many instances as needed. It is reccomended to create a unique HostedCE instance for each individual cluster you intend to support. ## Prerequisites +You must have a functional batch system on a cluster that you like to support OSG with. + The OSG HostedCE uses SSH to submit pilot jobs that will connect back to the OSG central pool. In order for the HostedCE to work, you'll first need to create a local service account for OSG to submit jobs. This should be done @@ -89,7 +94,7 @@ MaxWallTime is the maxmimum allowed walltime for the job and is expressed in min You can read more on the [OSG Docs](https://opensciencegrid.org/docs/other/configuration-with-osg-configure/#subcluster-resource-entry) -VOs are virtual orginizations within the OSG and they correspond to different research groups. AllowedVOs let's us specify which groups will run jobs through our CE. +VOs are virtual orginizations within the OSG and they correspond to different research groups. AllowedVOs allows us to specify which groups will run jobs through our CE. Finally you'll need to refer to the private key we stored in SLATE previously. I had called mine `utah-lp-hostedce-privkey`, so my configuration ends up @@ -167,7 +172,7 @@ There is more extensive documentation on the [Job Router](https://opensciencegri ### BoscoOverrides Section The BoscoOverrides section provides a mechanism to override default configuration placed on the remote cluster for the OSG Worker Node Client. This can include things like the local path to the batch system's executables and additional job submission parameters for the batch system. -This will vary depending on your batch system. I did needed to override the slurm_binpath and add some SBATCH parameters. All the overrides are expected to be placed in a git repository with a subdirectory format that matches `/bosco-override` in my case `SLATE_US_UUTAH_LONEPEAK/bosco-override`. +This will vary depending on your batch system. I did need to override the slurm_binpath and add some SBATCH parameters. All the overrides are expected to be placed in a git repository with a subdirectory format that matches `/bosco-override` in my case `SLATE_US_UUTAH_LONEPEAK/bosco-override`. It may take some trial and error to get the correct overrides in place. The general proccess for this is to deploy the CE, then check the logs on the application's HTTP Log exporter to see what must be changed. Finally re-dpeploy with the updated overrides. From d4f740139d956580d9c47c85a6edba8b8b2d0840 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:16:37 -0700 Subject: [PATCH 25/34] fix wording for site section --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index be10ccd9..76858e82 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -69,7 +69,7 @@ first: Latitude: 40.7608 Longitude: -111.891 -Group and Sponsor are safe to leave as defaults if you plan to support OSG +Sponsor is safe to leave as defaults if you plan to support OSG users with your CE. From the [OSG Resource Registration Documentation](https://opensciencegrid.org/docs/common/registration/), here's the definition of Resource and ResourceGroup: From 3eb92b19b80dea9de3d17e524c96f975e82e1d1f Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:19:37 -0700 Subject: [PATCH 26/34] update testing instructions to mention cert password --- _posts/2019-10-21-hosted-ce.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 76858e82..4c9adcef 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -338,9 +338,9 @@ In order to test end-to-end job submission through the CE you will need a valid An easy way to do that is through [CILogon](https://cilogon.org/). -Create a cert and download it. +Create a cert and download it. You'll need to remember the password you set. -Next you will need to convert it to PKCS12 format for voms. +Next you will need to convert it to PKCS12 format for voms. These commands will prompt for your password. `openssl pkcs12 -in usercred.p12 -nocerts -out hostkey.pem` From 98eac54a475c33fa9d3eda89c95784abf3f61aff Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:23:11 -0700 Subject: [PATCH 27/34] elaborate on ITB host --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 4c9adcef..5df19ae8 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -233,7 +233,7 @@ I will leave it disabled, because my configuration is stable. Secret: null ### Developer Section -Simply disable this. It is in place for the purpose of ITB Hosts. +Simply disable this. It is in place for the purpose of OSG Internal Testbed hosts, and is not intended for use with production CEs. Developer: Enabled: false From aba4b31b6608c0e9eea865e49ac7570b02d88fd2 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:25:25 -0700 Subject: [PATCH 28/34] add link to opensciencegrid --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 5df19ae8..a4000990 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -10,7 +10,7 @@ tag: draft --- # Deploying an OSG HostedCE -An OSG Hosted Compute Element is an application that allows a site to contribute idle HPC or HTC compute resources to the Open Science Grid. The CE is responsible for recieving jobs from the grid and routing them to your local cluster(s). These are pre-emptable jobs that will run when the cluster resources would have otherwise gone idle. +An OSG Hosted Compute Element is an application that allows a site to contribute idle HPC or HTC compute resources to the [Open Science Grid](opensciencegrid.org). The CE is responsible for recieving jobs from the grid and routing them to your local cluster(s). These are pre-emptable jobs that will run when the cluster resources would have otherwise gone idle. Using SLATE you can deploy the HostedCE locally at your site, and easily scale to as many instances as needed. It is reccomended to create a unique HostedCE instance for each individual cluster you intend to support. From 03d028d2bbe6c74d88624bc5e6dd6a1e9e74ce3f Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:25:51 -0700 Subject: [PATCH 29/34] fix link to osg --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index a4000990..6ea75062 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -10,7 +10,7 @@ tag: draft --- # Deploying an OSG HostedCE -An OSG Hosted Compute Element is an application that allows a site to contribute idle HPC or HTC compute resources to the [Open Science Grid](opensciencegrid.org). The CE is responsible for recieving jobs from the grid and routing them to your local cluster(s). These are pre-emptable jobs that will run when the cluster resources would have otherwise gone idle. +An OSG Hosted Compute Element is an application that allows a site to contribute idle HPC or HTC compute resources to the [Open Science Grid](https://opensciencegrid.org). The CE is responsible for recieving jobs from the grid and routing them to your local cluster(s). These are pre-emptable jobs that will run when the cluster resources would have otherwise gone idle. Using SLATE you can deploy the HostedCE locally at your site, and easily scale to as many instances as needed. It is reccomended to create a unique HostedCE instance for each individual cluster you intend to support. From 9a599eba877153f76863750f7ca12da14bf9bce8 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:27:31 -0700 Subject: [PATCH 30/34] fix typo --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 6ea75062..a7285642 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -15,7 +15,7 @@ An OSG Hosted Compute Element is an application that allows a site to contribute Using SLATE you can deploy the HostedCE locally at your site, and easily scale to as many instances as needed. It is reccomended to create a unique HostedCE instance for each individual cluster you intend to support. ## Prerequisites -You must have a functional batch system on a cluster that you like to support OSG with. +You must have a functional batch system on a cluster that you would like to support OSG with. The OSG HostedCE uses SSH to submit pilot jobs that will connect back to the OSG central pool. In order for the HostedCE to work, you'll first need to From f4af7f68027683d351944a5a59e5a86b41e8cfb2 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:30:20 -0700 Subject: [PATCH 31/34] fix wording --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index a7285642..98bc7c68 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -55,7 +55,7 @@ There are several things that you'll need to edit here. ### Site Section First, for the site section you'll want to put in appropriate values that will -ultimately map onto your OSG Topology entry. Let's take a look at this group +ultimately map onto your OSG Topology entry. Let's take a look at this section first: Site: From 2b81ecb9cc83a5cfebc75e2ed691e7b5536f08ae Mon Sep 17 00:00:00 2001 From: amblinalong Date: Wed, 22 Jan 2020 11:37:24 -0700 Subject: [PATCH 32/34] fixed some spelling --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index 98bc7c68..eccbb576 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -10,7 +10,7 @@ tag: draft --- # Deploying an OSG HostedCE -An OSG Hosted Compute Element is an application that allows a site to contribute idle HPC or HTC compute resources to the [Open Science Grid](https://opensciencegrid.org). The CE is responsible for recieving jobs from the grid and routing them to your local cluster(s). These are pre-emptable jobs that will run when the cluster resources would have otherwise gone idle. +An OSG Hosted Compute Element is an application that allows a site to contribute idle HPC or HTC compute resources to the [Open Science Grid](https://opensciencegrid.org). The CE is responsible for receiving jobs from the grid and routing them to your local cluster(s). These are preemptible jobs that will run when the cluster resources would have otherwise gone idle. Using SLATE you can deploy the HostedCE locally at your site, and easily scale to as many instances as needed. It is reccomended to create a unique HostedCE instance for each individual cluster you intend to support. From 89a541e4cfddd0a97001477bc608357db65eb010 Mon Sep 17 00:00:00 2001 From: amblinalong Date: Wed, 22 Jan 2020 11:39:26 -0700 Subject: [PATCH 33/34] more spelling --- _posts/2019-10-21-hosted-ce.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index eccbb576..daf0753a 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -135,7 +135,7 @@ squid I have deployed with SLATE: (It's also possible to launch a Squid service through SLATE, see [here](https://portal.slateci.io/applications/osg-frontier-squid)) -### Netowrking Sections +### Networking Sections The HostedCE application requires both forward and reverse DNS resolution for its publicly routable IP. Most SLATE clusters come pre-configured with a handful of "LoadBalancer" IP addresses that can be allocated automatically to different applications. You must set-up the DNS records for this address so it is a good idea to request a specific address from the pool. This is my network configuration: From 3548ee46c7873b0e02b28882fa4e65d9f7b49177 Mon Sep 17 00:00:00 2001 From: Mitchell <35324660+Mansalu@users.noreply.github.com> Date: Thu, 23 Jan 2020 11:44:08 -0700 Subject: [PATCH 34/34] add bosco-override-template --- _posts/2019-10-21-hosted-ce.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/_posts/2019-10-21-hosted-ce.md b/_posts/2019-10-21-hosted-ce.md index daf0753a..42df8e2c 100644 --- a/_posts/2019-10-21-hosted-ce.md +++ b/_posts/2019-10-21-hosted-ce.md @@ -176,6 +176,12 @@ This will vary depending on your batch system. I did need to override the slurm_ It may take some trial and error to get the correct overrides in place. The general proccess for this is to deploy the CE, then check the logs on the application's HTTP Log exporter to see what must be changed. Finally re-dpeploy with the updated overrides. +There is a template bosco override repository that you can fork and tailor to your needs. + +https://github.com/slateci/bosco-override-template + +Once you've customized your fork, you can simply provide that repository as the `GitEndpoint` for this section. + You can use a private git repo and provide the key to the application as a SLATE secret. My configuration with a public git repo is: BoscoOverrides: