Fix URL signing api for URLs containing special characters

conf/keycloak/docker-compose-dev.yml

@@ -31,6 +31,7 @@ services:
       DATAVERSE_FEATURE_API_BEARER_AUTH_USE_BUILTIN_USER_ON_ID_MATCH: "1"
       DATAVERSE_MAIL_SYSTEM_EMAIL: "dataverse@localhost"
       DATAVERSE_MAIL_MTA_HOST: "smtp"
+      DATAVERSE_API_SIGNING_SECRET: "dev-only-signing-secret-change-me"
       DATAVERSE_AUTH_OIDC_ENABLED: "1"
       DATAVERSE_AUTH_OIDC_CLIENT_ID: test
       DATAVERSE_AUTH_OIDC_CLIENT_SECRET: 94XHrfNRwXsjqTqApRrwWmhDLDHpIYV8

doc/release-notes/fix-url-signing-special-characters.md

@@ -0,0 +1,59 @@
+### Signed URLs work again for URLs with special characters
+
+Requesting a signed URL (e.g. via `/api/admin/requestSignedUrl`, used by external tools, the Globus
+integration and third-party integrations such as the `rdm-integration` connector) was broken in 6.10
+for URLs whose query contained special characters — most notably persistent IDs such as
+`doi:10.5072/FK2/ABC` (which contain `:` and `/`), as well as spaces, percent-encoded values and
+non-ASCII characters. In 6.10 the signing step began re-encoding/normalizing the URL (for example
+percent-encoding `:` and `/`) before computing the signature, while the request is validated against
+the URL the caller actually presents back. The re-encoded signature no longer matched, so validation
+failed with 401 "Bad signed URL" authentication errors.
+
+Signing no longer re-encodes the URL: it is signed exactly as provided, with only the reserved
+signing parameters (`until`, `user`, `method`, `token`, `key`, `signed`) stripped out; the rest of
+the URL is left untouched, character for character.
+
+**This restores the URL-signing behavior used before 6.10, so it is compatible with older versions
+and with existing integrations.** Clients and connectors that build or consume signed URLs the way
+they did before 6.10 keep working unchanged, signatures are computed the same way as before the
+regression, and URLs containing special characters validate again. No client-side changes are
+required.
+
+Note that, as before 6.10, the `url` submitted to `/api/admin/requestSignedUrl` (and any URL consumed
+as a signed URL) must be in its URL-decoded form: the signature is computed over the URL exactly as
+provided, but the server validates it against the URL-decoded request. Passing a percent-encoded URL
+(e.g. `persistentId=doi%3A10.5072%2FFK2%2FABC` instead of `persistentId=doi:10.5072/FK2/ABC`) and then
+using it verbatim will therefore fail validation.
+
+### A signing secret is now required for signed URLs
+
+Separately from the fix above, Dataverse no longer falls back to a weak signing key when
+`dataverse.api.signing-secret` is unset. Previously, with no secret configured, signed URLs were
+signed using only the user's API token (or, for a guest, a value derived from the public URL), which
+is too weak to be a signing key. A non-empty `dataverse.api.signing-secret` is now required wherever
+URLs are signed with a key based on a user's API token:
+
+- The endpoints that issue a signed URL on request - `/api/admin/requestSignedUrl` and the `POST`
+  guestbook-response download endpoints under `/api/access` (`datafile/{id}`, `datafiles/{ids}`,
+  `dataset/{id}` and `dataset/{id}/versions/{versionId}`) - return an error instead of issuing a
+  weakly-signed URL.
+- External tool launches send their callback unsigned (with a warning logged) rather than weakly
+  signed. Such an unsigned callback only allows anonymous access to public data; it cannot be used to
+  access draft datasets or restricted files.
+- Globus transfers are effectively disabled without a signing secret: the callbacks the
+  dataverse-globus app relies on require an authenticated, signed request, so Globus upload (and
+  download of restricted or unpublished files) fails with an authorization error. Installations that
+  use Globus must set `dataverse.api.signing-secret`.
+- The permissions-history CSV download links on the permission management pages are not offered
+  (a warning is logged).
+
+Remote and Globus overlay stores are unaffected: they sign with their own per-store secret key, not
+`dataverse.api.signing-secret`.
+
+**Upgrade note:** installations that rely on signed URLs - including the `rdm-integration` connector,
+signed guestbook-response downloads, and external tools or Globus transfers that use signed callbacks -
+must set `dataverse.api.signing-secret`. See the
+[Configuration Guide](https://guides.dataverse.org/en/latest/installation/config.html#dataverse-api-signing-secret).
+Treat the value like a password. Because the signing secret is part of the signing key, setting (or
+later changing) it invalidates previously issued signed URLs: any existing signed URLs that have not
+yet expired will stop working, and clients/integrations will need to request new ones.

doc/sphinx-guides/source/api/external-tools.rst

@@ -178,6 +178,10 @@ The signed URL mechanism is more secure than exposing API tokens and therefore r
 - For tools invoked via a GET call, Dataverse will include a callback query parameter with a Base64 encoded value. The decoded value is a signed URL that can be called to retrieve a JSON response containing all of the queryParameters and allowedApiCalls specified in the manfiest.
 - For tools invoked via POST, Dataverse will send a JSON body including the requested queryParameters and allowedApiCalls. Dataverse expects the response to the POST to indicate a redirect which Dataverse will use to open the tool.
 
+.. note::
+
+   **For Dataverse site administrators:** Signing these URLs requires a non-empty :ref:`dataverse.api.signing-secret` to be configured on the Dataverse installation. Without it, the callback and ``allowedApiCalls`` URLs are sent unsigned (a warning is logged), so tools cannot use them to access draft datasets or restricted files.
+
 .. note::
 
    **For Dataverse site administrators:** When Dataverse is behind a proxy, signed URLs may not work correctly due to protocol mismatches (HTTP vs HTTPS). Please refer to the :ref:`signed-urls-forwarded-proto-header` section to ensure signed URLs work properly in proxy environments.

doc/sphinx-guides/source/api/native-api.rst

@@ -9079,7 +9079,7 @@ Signed URLs were developed to support External Tools but may be useful in other
 This API call allows a Dataverse superUser to generate a signed URL for such scenarios.
 The JSON input parameter required is an object with the following keys:
 
-- ``url`` - the exact URL to sign, including api version number and all query parameters
+- ``url`` - the exact URL to sign, including api version number and all query parameters. Provide it in its URL-decoded form (for example ``persistentId=doi:10.5072/FK2/ABC``, not ``persistentId=doi%3A10.5072%2FFK2%2FABC``): the signature is computed over the URL exactly as provided, but Dataverse validates it against the URL-decoded request, so a percent-encoded URL used verbatim will fail validation.
 - ``timeOut`` - how long in minutes the signature should be valid for, default is 10 minutes
 - ``httpMethod`` - which HTTP method is required, default is GET
 - ``user`` - the user identifier for the account associated with this signature, the default is the superuser making the call. The API call will succeed/fail based on whether the specified user has the required permissions. 
@@ -9094,8 +9094,9 @@ A curl example using allowing access to a dataset's metadata
 
   curl -H "X-Dataverse-key:$API_KEY" -H 'Content-Type:application/json' -d "$JSON" "$SERVER_URL/api/admin/requestSignedUrl"
 
-Please see :ref:`dataverse.api.signing-secret` for the configuration option to add a shared secret, enabling extra
-security.
+Note that a non-empty :ref:`dataverse.api.signing-secret` must be configured on the server for this endpoint to work:
+the key used to sign the URL is created from the user's API token plus the signing-secret, and without the secret the
+endpoint returns an error instead of issuing a weakly-signed URL.
 
 .. _send-feedback-admin:
 

doc/sphinx-guides/source/developers/globus-api.rst

@@ -78,6 +78,7 @@ Note that while Dataverse will not add files that violate the size or quota rule
 they intend to transfer before submitting a transfer request to Globus.
 
 The getDatasetMetadata and getFileListing URLs are just signed versions of the standard Dataset metadata and file listing API calls. The other two are Globus specific.
+Note that these URLs are only signed when a non-empty :ref:`dataverse.api.signing-secret` is configured on the Dataverse installation. Without it the callbacks are sent unsigned (a warning is logged), and because the Globus callback endpoints (``globusUploadParameters``, ``requestGlobusUploadPaths``, ``addGlobusFiles``, and the download equivalents for restricted or unpublished files) require an authenticated, signed request, the Globus transfer flow fails with an authorization error; only anonymous download of public files can proceed. In practice, Globus transfer support requires ``dataverse.api.signing-secret`` to be set.
 
 If called for a dataset using a store that is configured with a remote Globus endpoint(s), the return response is similar but the response includes a
 the "managed" parameter will be false, the "endpoint" parameter is replaced with a JSON array of "referenceEndpointsWithPaths" and the
@@ -112,7 +113,7 @@ Once the user identifies which files are to be added, the requestGlobusTransferP
 
   curl -H "X-Dataverse-key:$API_TOKEN" -H "Content-type:application/json" -X POST  -d "$JSON_DATA" "$SERVER_URL/api/datasets/:persistentId/requestGlobusUploadPaths?persistentId=$PERSISTENT_IDENTIFIER"
 
-Note that when using the dataverse-globus app or the return from the previous call, the URL for this call will be signed and no API_TOKEN is needed. 
+Note that when using the dataverse-globus app or the return from the previous call, the URL for this call will be signed (provided a non-empty :ref:`dataverse.api.signing-secret` is configured, as noted above) and no API_TOKEN is needed.
 
 In the managed case, the JSON body sent must include the id of the Globus user that will perform the transfer and the number of files that will be transferred:
 

doc/sphinx-guides/source/installation/config.rst

@@ -3349,15 +3349,18 @@ are time limited and only allow the action of the API call in the URL. See :ref:
 :ref:`api-native-signed-url` for more details. 
 
 The key used to sign a URL is created from the API token of the creating user plus a signing-secret provided by an administrator.
-**Using a signing-secret is highly recommended.** This setting defaults to an empty string. Using a non-empty 
-signing-secret makes it impossible for someone who knows an API token from forging signed URLs and provides extra security by 
-making the overall signing key longer.
+**A non-empty signing-secret is required to request signed URLs through the API.** If it is not configured, the
+``/api/admin/requestSignedUrl`` endpoint (see :ref:`api-native-signed-url`) and the ``POST`` guestbook-response download
+endpoints under ``/api/access`` return an error instead of issuing a weakly-signed URL, and internally generated links
+(external tool callbacks, Globus transfers, the permissions-history CSV downloads) are sent unsigned or omitted, with a
+warning logged. (The setting otherwise defaults to an empty string.) A non-empty signing-secret makes it impossible for
+someone who only knows an API token to forge signed URLs, and provides extra security by making the overall signing key longer.
 
 **WARNING**:
 *Since the signing-secret is sensitive, you should treat it like a password.*
 *See* :ref:`secure-password-storage` *to learn about ways to safeguard it.*
 
-Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_API_SIGNATURE_SECRET`` (although you shouldn't use environment variables for passwords) .
+Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_API_SIGNING_SECRET`` (although you shouldn't use environment variables for passwords) .
 
 .. _dataverse.api.allow-incomplete-metadata:
 

docker-compose-dev.yml

@@ -60,6 +60,7 @@ services:
         -Ddataverse.pid.fake.label=FakeDOIProvider
         -Ddataverse.pid.fake.authority=10.5072
         -Ddataverse.pid.fake.shoulder=FK2/
+        -Ddataverse.api.signing-secret=dev-only-signing-secret-change-me
         -Ddataverse.cors.origin=* \
         -Ddataverse.cors.methods=GET,POST,PUT,DELETE,OPTIONS \
         -Ddataverse.cors.headers.allow=range,content-type,x-dataverse-key,accept \

docker/compose/demo/compose.yml

@@ -16,6 +16,9 @@ services:
       DATAVERSE_FEATURE_API_BEARER_AUTH: "1"
       DATAVERSE_MAIL_SYSTEM_EMAIL: "Demo Dataverse <dataverse@example.org>"
       DATAVERSE_MAIL_MTA_HOST: "smtp"
+      # Required for signed URLs (external tools, Globus transfers, signed guestbook downloads,
+      # rdm-integration). Change this to a private value; see the Configuration Guide.
+      DATAVERSE_API_SIGNING_SECRET: "demo-only-signing-secret-change-me"
       JVM_ARGS: -Ddataverse.files.storage-driver-id=file1
         -Ddataverse.files.file1.type=file
         -Ddataverse.files.file1.label=Filesystem

src/main/java/edu/harvard/iq/dataverse/ManageFilePermissionsPage.java

@@ -22,7 +22,6 @@
 import edu.harvard.iq.dataverse.engine.command.exception.PermissionException;
 import edu.harvard.iq.dataverse.engine.command.impl.AssignRoleCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.RevokeRoleCommand;
-import edu.harvard.iq.dataverse.settings.JvmSettings;
 import edu.harvard.iq.dataverse.util.BundleUtil;
 import edu.harvard.iq.dataverse.util.DateUtil;
 import edu.harvard.iq.dataverse.util.JsfHelper;
@@ -640,9 +639,11 @@ public String getSignedUrlForRAHistoryCsv() {
                     key = apiToken.getTokenString();
                 }
             }
-            key = JvmSettings.API_SIGNING_SECRET.lookupOptional().orElse("") + key;
-            if(key.length() >= 36) {
-                return UrlSignerUtil.signUrl(fullApiPath, 10, userId, "GET", key);
+            if (key != null) {
+                if (UrlSignerUtil.isSigningSecretConfigured()) {
+                    return UrlSignerUtil.signUrlWithApiKey(fullApiPath, 10, userId, "GET", key);
+                }
+                logger.warning("Cannot sign the permissions-history CSV link: no signing secret configured (dataverse.api.signing-secret). The download link will not be shown.");
             }
         } catch (Exception e) {
             logger.log(Level.SEVERE, "Error generating signed URL for permissions history CSV: " + e.getMessage(), e);

src/main/java/edu/harvard/iq/dataverse/ManagePermissionsPage.java

@@ -20,7 +20,6 @@
 import edu.harvard.iq.dataverse.engine.command.impl.CreateRoleCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.RevokeRoleCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.UpdateDataverseDefaultContributorRoleCommand;
-import edu.harvard.iq.dataverse.settings.JvmSettings;
 import edu.harvard.iq.dataverse.util.BundleUtil;
 import edu.harvard.iq.dataverse.util.JsfHelper;
 import static edu.harvard.iq.dataverse.util.JsfHelper.JH;
@@ -736,9 +735,11 @@ public String getSignedUrlForRAHistoryCsv() {
                     key = apiToken.getTokenString();
                 }
             }
-            key = JvmSettings.API_SIGNING_SECRET.lookupOptional().orElse("") + key;
-            if(key.length() >= 36) {
-                return UrlSignerUtil.signUrl(fullApiPath, 10, userId, "GET", key);
+            if (key != null) {
+                if (UrlSignerUtil.isSigningSecretConfigured()) {
+                    return UrlSignerUtil.signUrlWithApiKey(fullApiPath, 10, userId, "GET", key);
+                }
+                logger.warning("Cannot sign the permissions-history CSV link: no signing secret configured (dataverse.api.signing-secret). The download link will not be shown.");
             }
         } catch (Exception e) {
             logger.log(Level.SEVERE, "Error generating signed URL for permissions history CSV: " + e.getMessage(), e);