diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..0515ff7 --- /dev/null +++ b/LICENSE @@ -0,0 +1,229 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or Derivative + Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, excluding + those notices that do not pertain to any part of the Derivative + Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one of + the following places: within a NOTICE text file distributed as + part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and do + not modify the License. You may add Your own attribution notices + within Derivative Works that You distribute, alongside or as an + addendum to the NOTICE text from the Work, provided that such + additional attribution notices cannot be construed as modifying + the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +======================================================================= +APACHE PHOENIX ADAPTERS SUBCOMPONENTS +======================================================================= + +The Apache Phoenix Adapters SOURCE distribution contains only Apache-licensed +first-party source code and requires no additional third-party license notices. + +The Apache Phoenix Adapters BINARY distribution (the -bin.tar.gz produced by the +phoenix-ddb-assembly module) bundles third-party dependencies under their own +licenses. Notable non-Apache-2.0 bundled components include: + + * ANTLR 4 Runtime (org.antlr:antlr4-runtime) -- BSD 3-Clause License + https://www.antlr.org/license.html + +All other bundled runtime dependencies (Apache Phoenix, Apache HBase, Apache +Hadoop, Apache ZooKeeper, Apache Log4j 2, AWS SDK for Java, Eclipse Jetty's +Apache-2.0-licensed artifacts, etc.) are distributed under the Apache License, +Version 2.0. + +NOTE TO THE RELEASE MANAGER: before calling a release vote, complete a full +LICENSE/NOTICE audit of every artifact packaged into the binary tarball +(mvn dependency:tree on phoenix-ddb-assembly) and expand the list above with the +exact license text/pointer for every non-Apache-2.0 component (e.g. Jetty's EPL +dual-licensed pieces, CDDL components, BSD/MIT components). See +https://www.apache.org/legal/resolved.html and +https://infra.apache.org/licensing-howto.html diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..4cadb16 --- /dev/null +++ b/NOTICE @@ -0,0 +1,15 @@ +Apache Phoenix Adapters +Copyright 2024-2026 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +----------------------------------------------------------------------------- +NOTE TO THE RELEASE MANAGER +----------------------------------------------------------------------------- +The binary distribution bundles third-party dependencies that ship their own +NOTICE files. Before calling a release vote, aggregate the required attribution +notices from every bundled dependency's NOTICE (Apache Phoenix, Apache HBase, +Apache Hadoop, Apache ZooKeeper, Apache Log4j 2, AWS SDK for Java, etc.) into +this file, per https://infra.apache.org/licensing-howto.html#mod-notice . +Only content actually required by a bundled dependency's own NOTICE belongs here. diff --git a/coverage-report/pom.xml b/coverage-report/pom.xml index 7295ddb..f4b5a21 100644 --- a/coverage-report/pom.xml +++ b/coverage-report/pom.xml @@ -15,6 +15,11 @@ Phoenix Adapters - Coverage Report Module dedicated to generating aggregated code coverage reports + + + true + + diff --git a/dev/create-release/README.txt b/dev/create-release/README.txt new file mode 100644 index 0000000..6768775 --- /dev/null +++ b/dev/create-release/README.txt @@ -0,0 +1,95 @@ +Entrance script is _do-release-docker.sh_. Requires a local docker; +for example, on macOS, Docker for Desktop installed and running. + +These scripts are adapted from the Apache Phoenix dev/create-release scripts +(which themselves came from HBase and originally Spark). They produce Apache +Phoenix Adapters release candidates. + +For usage, pass '-h': + + $ ./do-release-docker.sh -h + +./do-release-docker.sh accepts the following options + + -d [path] required. working directory. output will be written to "output" in here. + -f "force" -- actually publish this release. Unless you specify '-f', it will + default to dry run mode, which checks and does local builds, but does not upload anything. + -t [tag] tag for the phoenix-adapters-rm docker image to use for building (default: "latest"). + -j [path] path to local JDK installation to use building. By default the script will + use openjdk8 installed in the docker image. + -p [project] project to build; defaults to the PROJECT env var (phoenix-adapters). + -r [repo] git repo to use for remote git operations. defaults to ASF gitbox for the project. + -s [step] runs a single step of the process; valid steps are: tag|publish-dist|publish-release. + If none specified, runs tag, then publish-dist, and then publish-release. + 'publish-snapshot' is also an allowed, less used, option. + -h display usage information + -x debug. do less clean up. (env file, gpg forwarding on mac) + +For example, use the following command to do a full dry run build: + +./do-release-docker.sh -d /tmp/phoenix-adapters-build + +To run a build w/o invoking docker (not recommended!), use _do-release.sh_. + +Both scripts will query interactively for needed parameters and passphrases. +For explanation of the parameters, execute: + $ release-build.sh --help + +Notes specific to phoenix-adapters: + * The binary distribution tarball is produced by the "phoenix-ddb-assembly" module rather than + the conventional "${PROJECT}-assembly" name. This is handled via the ASSEMBLY_MODULE environment + variable, which do-release-docker.sh and do-release.sh default to "phoenix-ddb-assembly". + * phoenix-adapters shares the PHOENIX JIRA project. By convention its "Fix Version" values are + prefixed with "adapters-" (e.g. "adapters-1.0.0"). release-build.sh uses this to generate + CHANGES.md and RELEASENOTES.md via Apache Yetus releasedocmaker. + * Release tarballs are staged under https://dist.apache.org/repos/dist/dev/phoenix/ and the RM's + signing key must be present in https://dist.apache.org/repos/dist/release/phoenix/KEYS . + +Before starting the RC build, run a reconciliation of what is in +JIRA with what is in the commit log. Make sure they align and that +anomalies are explained up in JIRA. + +See http://hbase.apache.org/book.html#maven.release +(Even though the above documentation is for HBase, we use the same process for Phoenix +and its sub-projects.) + +Regardless of where your release build will run (locally, locally in docker, on a remote machine, +etc) you will need a local gpg-agent with access to your secret keys. A quick way to tell gpg +to clear out state and start a gpg-agent is via the following command phrase: + +$ gpgconf --kill all && gpg-connect-agent /bye + +Before starting an RC build, make sure your local gpg-agent has configs +to properly handle your credentials, especially if you want to avoid +typing the passphrase to your secret key. + +e.g. if you are going to run and step away, best to increase the TTL +on caching the unlocked secret via ~/.gnupg/gpg-agent.conf + # in seconds, e.g. a day + default-cache-ttl 86400 + max-cache-ttl 86400 + +In the current version, passphrase entry doesn't work at all, at least for Linux Docker builds. +Increasing the TTL only works if you unlock the key before starting the release script by running +gpg separately before the script. +A better way to handle passphrases without changing the TTLs is to preset the passphrase, +which avoids using pinentry mechanism completely, and will be reset on logout. + +# Find the "gpg-preset-passphrase" program. It is not on the PATH by default. +$ find / -name gpg-preset-passphrase +# Make sure you have the "allow-preset-passphrase" line in your $HOME/.gnupg/gpg-agent.conf +# Restart gpg +$ gpgconf --kill all && gpg-connect-agent /bye +# List your keys with key grip +$ gpg --with-keygrip --list-secret-keys +# Preset the passphrase for your signing key +# /gpg-preset-passphrase -P -c +# Check that the passphrase is successfully preset. There should be a '1' at the fourth position +# after the keygrip for your key in the output for the signing key +$ gpg-connect-agent 'keyinfo --list' /bye +# Run the release script (see above) +# Restart the gpg agent again to make sure it forgets the preset passphrase +$ gpgconf --kill all && gpg-connect-agent /bye + +Note that according to https://www.apache.org/legal/release-policy.html#owned-controlled-hardware +building an Apache release must be done on hardware owned and controlled by the committer. diff --git a/dev/create-release/cache-apache-project-artifact.sh b/dev/create-release/cache-apache-project-artifact.sh new file mode 120000 index 0000000..d9e4e02 --- /dev/null +++ b/dev/create-release/cache-apache-project-artifact.sh @@ -0,0 +1 @@ +../cache-apache-project-artifact.sh \ No newline at end of file diff --git a/dev/create-release/do-release-docker.sh b/dev/create-release/do-release-docker.sh new file mode 100755 index 0000000..88aafa6 --- /dev/null +++ b/dev/create-release/do-release-docker.sh @@ -0,0 +1,354 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# +# Creates a Phoenix release candidate. The script will update versions, tag the branch, +# build Phoenix binary packages and documentation, and upload maven artifacts to a staging +# repository. There is also a dry run mode where only local builds are performed, and +# nothing is uploaded to the ASF repos. +# +# Run with "-h" for options. For example, running below will do all +# steps above using the 'rm' dir under Downloads as workspace: +# +# $ ./do-release-docker.sh -d ~/Downloads/rm +# +# The scripts in this directory came originally from spark [1]. They were then +# modified to suite the hbase context, which were further adopted to Phoenix. +# These scripts supercedes the old +# ../make_rc.sh script for making release candidates because what is here is more +# comprehensive doing more steps of the RM process as well as running in a +# container so the RM build environment can be a constant. +# +# It: +# * Tags release +# * Sets version to the release version +# * Sets version to next SNAPSHOT version. +# * Builds, signs, and hashes all artifacts. +# * Pushes release tgzs to the dev dir in a apache dist. +# * Pushes to repository.apache.org staging. +# +# The entry point is here, in the do-release-docker.sh script. +# +# 1. https://github.com/apache/spark/tree/master/dev/create-release +# +set -e + +# Set this to build other phoenix repos: e.g. PROJECT=phoenix-connectors +export PROJECT="${PROJECT:-phoenix-adapters}" + +# Name of the Maven module that produces the binary distribution tarball. The Phoenix release +# scripts assume "${PROJECT}-assembly"; phoenix-adapters keeps its historical module name, so we +# make this overridable and default it here. +export ASSEMBLY_MODULE="${ASSEMBLY_MODULE:-phoenix-ddb-assembly}" + +SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=SCRIPTDIR/release-util.sh +. "$SELF/release-util.sh" +ORIG_PWD="$(pwd)" + +function usage { + local NAME + NAME="$(basename "${BASH_SOURCE[0]}")" + cat < 0 )); then + error "Arguments can only be provided with option flags, invalid args: $*" +fi +export DEBUG +if [ "$DEBUG" = "1" ]; then + set -x +fi + +if [ -z "$WORKDIR" ] || [ ! -d "$WORKDIR" ]; then + error "Work directory (-d) must be defined and exist. Run with -h for help." +fi + +if [ -d "$WORKDIR/output" ]; then + read -r -p "Output directory already exists. Overwrite and continue? [y/n] " ANSWER + if [ "$ANSWER" != "y" ]; then + error "Exiting." + fi +fi + +if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \ + [ -f "${WORKDIR}/gpg-proxy.cid" ] || \ + [ -f "${WORKDIR}/release.cid" ]; then + read -r -p "container/pid files from prior run exists. Overwrite and continue? [y/n] " ANSWER + if [ "$ANSWER" != "y" ]; then + error "Exiting." + fi +fi + +cd "$WORKDIR" +rm -rf "$WORKDIR/output" +rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid" "${WORKDIR}/release.cid" +mkdir "$WORKDIR/output" + +banner "Gathering release details." +HOST_OS="$(get_host_os)" +get_release_info + +banner "Setup" + +# Place all RM scripts and necessary data in a local directory that must be defined in the command +# line. This directory is mounted into the image. Its WORKDIR, the arg passed with -d. +for f in "$SELF"/*; do + if [ -f "$f" ]; then + cp "$f" "$WORKDIR" + fi +done + +# We need to import that public key in the container in order to use the private key via the agent. +GPG_KEY_FILE="$WORKDIR/gpg.key.public" +echo "Exporting public key for ${GPG_KEY}" +fcreate_secure "$GPG_KEY_FILE" +$GPG "${GPG_ARGS[@]}" --export "${GPG_KEY}" > "${GPG_KEY_FILE}" + +function cleanup { + local id + banner "Release Cleanup" + if is_debug; then + echo "skipping due to debug run" + return 0 + fi + echo "details in cleanup.log" + if [ -f "${ENVFILE}" ]; then + rm -f "$ENVFILE" + fi + rm -f "$GPG_KEY_FILE" + if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ]; then + id=$(cat "${WORKDIR}/gpg-proxy.ssh.pid") + echo "Stopping ssh tunnel for gpg-agent at PID ${id}" | tee -a cleanup.log + kill -9 "${id}" >>cleanup.log 2>&1 || true + rm -f "${WORKDIR}/gpg-proxy.ssh.pid" >>cleanup.log 2>&1 + fi + if [ -f "${WORKDIR}/gpg-proxy.cid" ]; then + id=$(cat "${WORKDIR}/gpg-proxy.cid") + echo "Stopping gpg-proxy container with ID ${id}" | tee -a cleanup.log + docker kill "${id}" >>cleanup.log 2>&1 || true + rm -f "${WORKDIR}/gpg-proxy.cid" >>cleanup.log 2>&1 + # TODO we should remove the gpgagent volume? + fi + if [ -f "${WORKDIR}/release.cid" ]; then + id=$(cat "${WORKDIR}/release.cid") + echo "Stopping release container with ID ${id}" | tee -a cleanup.log + docker kill "${id}" >>cleanup.log 2>&1 || true + rm -f "${WORKDIR}/release.cid" >>cleanup.log 2>&1 + fi +} + +trap cleanup EXIT + +echo "Host OS: ${HOST_OS}" +if [ "${HOST_OS}" == "DARWIN" ]; then + run_silent "Building gpg-agent-proxy image with tag ${IMGTAG}..." "docker-proxy-build.log" \ + docker build --build-arg "UID=${UID}" --build-arg "RM_USER=${USER}" \ + --tag "org.apache.phoenixadapters/gpg-agent-proxy:${IMGTAG}" "${SELF}/mac-sshd-gpg-agent" +fi + +run_silent "Building phoenix-adapters-rm image with tag $IMGTAG..." "docker-build.log" \ + docker build --tag "org.apache.phoenixadapters/phoenix-adapters-rm:$IMGTAG" --build-arg "UID=$UID" \ + --build-arg "RM_USER=${USER}" "$SELF/phoenix-adapters-rm" + +banner "Final prep for container launch." +echo "Writing out environment for container." +# Write the release information to a file with environment variables to be used when running the +# image. +ENVFILE="$WORKDIR/env.list" +fcreate_secure "$ENVFILE" + +cat > "$ENVFILE" <> "$ENVFILE" + JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dst=/opt/phoenix-java,readonly") +fi + +#TODO some debug output would be good here +GIT_REPO_MOUNT=() +if [ -n "${GIT_REPO}" ]; then + case "${GIT_REPO}" in + # skip the easy to identify remote protocols + ssh://*|git://*|http://*|https://*|ftp://*|ftps://*) ;; + # for sure local + /*) + GIT_REPO_MOUNT=(--mount "type=bind,src=${GIT_REPO},dst=/opt/phoenix-repo,consistency=delegated") + echo "HOST_GIT_REPO=${GIT_REPO}" >> "${ENVFILE}" + GIT_REPO="/opt/phoenix-repo" + ;; + # on the host but normally git wouldn't use the local optimization + file://*) + echo "[INFO] converted file:// git repo to a local path, which changes git to assume --local." + GIT_REPO_MOUNT=(--mount "type=bind,src=${GIT_REPO#file://},dst=/opt/phoenix-repo,consistency=delegated") + echo "HOST_GIT_REPO=${GIT_REPO}" >> "${ENVFILE}" + GIT_REPO="/opt/phoenix-repo" + ;; + # have to decide if it's a local path or the "scp-ish" remote + *) + declare colon_remove_prefix; + declare slash_remove_prefix; + declare local_path; + colon_remove_prefix="${GIT_REPO#*:}" + slash_remove_prefix="${GIT_REPO#*/}" + if [ "${GIT_REPO}" = "${colon_remove_prefix}" ]; then + # if there was no colon at all, we assume this must be a local path + local_path="no colon at all" + elif [ "${GIT_REPO}" != "${slash_remove_prefix}" ]; then + # if there was a colon and there is no slash, then we assume it must be scp-style host + # and a relative path + + if [ "${#colon_remove_prefix}" -lt "${#slash_remove_prefix}" ]; then + # Given the substrings made by removing everything up to the first colon and slash + # we can determine which comes first based on the longer substring length. + # if the slash is first, then we assume the colon is part of a path name and if the colon + # is first then it is the seperator between a scp-style host name and the path. + local_path="slash happened before a colon" + fi + fi + if [ -n "${local_path}" ]; then + # convert to an absolute path + GIT_REPO="$(cd "$(dirname "${ORIG_PWD}/${GIT_REPO}")"; pwd)/$(basename "${ORIG_PWD}/${GIT_REPO}")" + GIT_REPO_MOUNT=(--mount "type=bind,src=${GIT_REPO},dst=/opt/phoenix-repo,consistency=delegated") + echo "HOST_GIT_REPO=${GIT_REPO}" >> "${ENVFILE}" + GIT_REPO="/opt/phoenix-repo" + fi + ;; + esac + echo "GIT_REPO=${GIT_REPO}" >> "${ENVFILE}" +fi + +GPG_PROXY_MOUNT=() +if [ "${HOST_OS}" == "DARWIN" ]; then + GPG_PROXY_MOUNT=(--mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/") + echo "Setting up GPG agent proxy container needed on OS X." + echo " we should clean this up for you. If that fails the container ID is below and in " \ + "gpg-proxy.cid" + #TODO the key pair used should be configurable + docker run --rm -p 62222:22 \ + --detach --cidfile "${WORKDIR}/gpg-proxy.cid" \ + --mount \ + "type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly" \ + "${GPG_PROXY_MOUNT[@]}" \ + "org.apache.phoenixadapters/gpg-agent-proxy:${IMGTAG}" + # gotta trust the container host + ssh-keyscan -p 62222 localhost 2>/dev/null | sort > "${WORKDIR}/gpg-agent-proxy.ssh-keyscan" + sort "${HOME}/.ssh/known_hosts" | comm -1 -3 - "${WORKDIR}/gpg-agent-proxy.ssh-keyscan" \ + > "${WORKDIR}/gpg-agent-proxy.known_hosts" + if [ -s "${WORKDIR}/gpg-agent-proxy.known_hosts" ]; then + echo "Your ssh known_hosts does not include the entries for the gpg-agent proxy container." + echo "The following entry(ies) arre missing:" + sed -e 's/^/ /' "${WORKDIR}/gpg-agent-proxy.known_hosts" + read -r -p "Okay to add these entries to ${HOME}/.ssh/known_hosts? [y/n] " ANSWER + if [ "$ANSWER" != "y" ]; then + error "Exiting." + fi + cat "${WORKDIR}/gpg-agent-proxy.known_hosts" >> "${HOME}/.ssh/known_hosts" + fi + echo "Launching ssh reverse tunnel from the container to gpg agent." + echo " we should clean this up for you. If that fails the PID is in gpg-proxy.ssh.pid" + ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir agent-socket)" \ + -i "${HOME}/.ssh/id_rsa" -N -n localhost >gpg-proxy.ssh.log 2>&1 & + echo $! > "${WORKDIR}/gpg-proxy.ssh.pid" +else + # Note that on linux we always directly mount the gpg agent's extra socket to limit what the + # container can ask the gpg-agent to do. + # When working on a remote linux machine you should be sure to forward both the remote machine's + # agent socket and agent extra socket to your local gpg-agent's extra socket. See the README.txt + # for an example. + GPG_PROXY_MOUNT=(--mount \ + "type=bind,src=$(gpgconf --list-dir agent-socket),dst=/home/${USER}/.gnupg/S.gpg-agent") +fi + +banner "Building $RELEASE_TAG; output will be at $WORKDIR/output" +echo "We should clean the container up when we are done. If that fails then the container ID " \ + "is in release.cid" +echo +# Where possible we specifcy "consistency=delegated" when we do not need host access during the +# build run. On Mac OS X specifically this gets us a big perf improvement. +cmd=(docker run --rm -ti \ + --env-file "$ENVFILE" \ + --cidfile "${WORKDIR}/release.cid" \ + --mount "type=bind,src=${WORKDIR},dst=/home/${USER}/phoenix-adapters-rm,consistency=delegated" \ + "${JAVA_MOUNT[@]}" \ + "${GIT_REPO_MOUNT[@]}" \ + "${GPG_PROXY_MOUNT[@]}" \ + "org.apache.phoenixadapters/phoenix-adapters-rm:$IMGTAG") +echo "${cmd[*]}" +"${cmd[@]}" diff --git a/dev/create-release/do-release.sh b/dev/create-release/do-release.sh new file mode 100755 index 0000000..bde5bf7 --- /dev/null +++ b/dev/create-release/do-release.sh @@ -0,0 +1,159 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +if [ "$DEBUG" = "1" ]; then + set -x +fi + +# Use the adjacent do-release-docker.sh instead, if you can. +# Otherwise, this runs core of the release creation. +# Will ask you questions on what to build and for logins +# and passwords to use building. +export PROJECT="${PROJECT:-phoenix-adapters}" +export ASSEMBLY_MODULE="${ASSEMBLY_MODULE:-phoenix-ddb-assembly}" + +SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=SCRIPTDIR/release-util.sh +. "$SELF/release-util.sh" + +while getopts "b:fs:" opt; do + case $opt in + b) export GIT_BRANCH=$OPTARG ;; + f) export DRY_RUN=0 ;; # "force", ie actually publish this release (otherwise defaults to dry run) + s) RELEASE_STEP="$OPTARG" ;; + ?) error "Invalid option: $OPTARG" ;; + esac +done +shift $((OPTIND-1)) +if (( $# > 0 )); then + error "Arguments can only be provided with option flags, invalid args: $*" +fi + +function gpg_agent_help { + cat < gpg_test.txt +if ! "${GPG}" "${GPG_ARGS[@]}" --detach --armor --sign gpg_test.txt ; then + gpg_agent_help +fi +# In --batch mode we have to be explicit about what we are verifying +if ! "${GPG}" "${GPG_ARGS[@]}" --verify gpg_test.txt.asc gpg_test.txt ; then + gpg_agent_help +fi + +if [[ -z "$RELEASE_STEP" ]]; then + # If doing all stages, leave out 'publish-snapshot' + RELEASE_STEP="tag_publish-dist_publish-release" + # and use shared maven local repo for efficiency + export REPO="${REPO:-$(pwd)/$(mktemp -d phoenix-repo-XXXXX)}" +fi + +function should_build { + local WHAT=$1 + if [[ -z "$RELEASE_STEP" ]]; then + return 0 + elif [[ "$RELEASE_STEP" == *"$WHAT"* ]]; then + return 0 + else + return 1 + fi +} + +if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then + if [ -z "${YETUS_HOME}" ] && [ "${RUNNING_IN_DOCKER}" != "1" ]; then + declare local_yetus="/opt/apache-yetus/0.12.0/" + if [ "$(get_host_os)" = "DARWIN" ]; then + local_yetus="/usr/local/Cellar/yetus/0.12.0/" + fi + YETUS_HOME="$(read_config "YETUS_HOME not defined. Absolute path to local install of Apache Yetus" "${local_yetus}")" + export YETUS_HOME + fi + run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \ + "$SELF/release-build.sh" tag + if is_dry_run; then + export TAG_SAME_DRY_RUN="true"; + fi +else + echo "Skipping tag creation for $RELEASE_TAG." +fi + +if should_build "publish-dist"; then + run_silent "Publishing distribution packages (tarballs)" "publish-dist.log" \ + "$SELF/release-build.sh" publish-dist +else + echo "Skipping publish-dist step." +fi + +if should_build "publish-snapshot"; then + run_silent "Publishing snapshot" "publish-snapshot.log" \ + "$SELF/release-build.sh" publish-snapshot + +elif should_build "publish-release"; then + run_silent "Publishing release" "publish-release.log" \ + "$SELF/release-build.sh" publish-release +else + echo "Skipping publish-release step." +fi diff --git a/dev/create-release/mac-sshd-gpg-agent/Dockerfile b/dev/create-release/mac-sshd-gpg-agent/Dockerfile new file mode 100644 index 0000000..efa36a2 --- /dev/null +++ b/dev/create-release/mac-sshd-gpg-agent/Dockerfile @@ -0,0 +1,101 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Image for use on Mac boxes to get a gpg agent socket available +# within transient release building ocntainers. +# +# Copied directly from HBase +# +# build like: +# +# docker build --build-arg "UID=$UID" --build-arg "RM_USER=$USER" \ +# --tag org.apache.phoenix/gpg-agent-proxy mac-sshd-gpg-agent +# +# run like: +# +# docker run --rm -p 62222:22 \ +# --mount "type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly" \ +# --mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/" \ +# org.apache.phoenix/gpg-agent-proxy:latest +# +# test like: +# +# ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir agent-socket)" \ +# -i "${HOME}/.ssh/id_rsa" -N -n localhost +# +# launch a docker container to do work that shares the mount for the gpg agent +# expressly does not need to be this same image, but needs to have defined the same user +# +# docker run --rm -it \ +# --mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/" \ +# --mount "type=bind,src=${HOME}/projects/phoenix-releases/KEYS,dst=/home/${USER}/KEYS,readonly" \ +# --entrypoint /bin/bash --user "${USER}" --workdir "/home/${USER}/" \ +# org.apache.phoenix/gpg-agent-proxy:latest +# +# +# Make sure to import the public keys +# +# gpg --no-autostart --import < ${HOME}/KEYS +# Optional? +# gpg --no-autostart --edit-key ${YOUR_KEY} +# trust +# 5 +# y +# quit +#ubu +# gpg --no-autostart --armor --detach --sign foo +# gpg --no-autostart --verify foo.asc +# +# For more info see +# * gpg forwarding over ssh: https://wiki.gnupg.org/AgentForwarding +# * example docker for sshd: https://github.com/hotblac/nginx-ssh +# * why we have to bother with this: https://github.com/docker/for-mac/issues/483 +# +# If the docker image changes then the host key used by sshd will change and you will get a +# nastygram when launching ssh about host identification changing. This is expected. you should +# remove the previous host key. +# +# Tested with +# * Docker Desktop 2.2.0.5 +# * gpg 2.2.20 +# * pinentry-mac 0.9.4 +# * yubikey 5 +# +FROM ubuntu:18.04 + +# This is all in a single "RUN" command so that if anything changes, "apt update" is run to fetch +# the most current package versions (instead of potentially using old versions cached by docker). +# +# We only need gnupg2 here if we want the ability to test out the gpg-agent forwarding by sshing +# into the container rather than launching a new docker container. +RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y update \ + && DEBIAN_FRONTEND=noninteractive apt-get -qq -y install --no-install-recommends \ + openssh-server=1:7.6* gnupg2=2.2.4* && mkdir /run/sshd \ + && echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +EXPOSE 22 +# Set up our ssh user +ARG UID +ARG RM_USER +RUN groupadd sshgroup && \ + useradd --create-home --shell /bin/bash --groups sshgroup --uid $UID $RM_USER && \ + mkdir /home/$RM_USER/.ssh /home/$RM_USER/.gnupg && \ + chown -R $RM_USER:sshgroup /home/$RM_USER/ && \ + chmod -R 700 /home/$RM_USER/ +# When we run we run sshd +ENTRYPOINT ["/usr/sbin/sshd", "-D"] diff --git a/dev/create-release/phoenix-adapters-rm/Dockerfile b/dev/create-release/phoenix-adapters-rm/Dockerfile new file mode 100644 index 0000000..29f7e4a --- /dev/null +++ b/dev/create-release/phoenix-adapters-rm/Dockerfile @@ -0,0 +1,83 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Image for building Phoenix Adapters releases. Based on Ubuntu 22.04. +# +# Adapted from the Apache Phoenix phoenix-rm image. +# +# Includes: +# * Java 8 +FROM ubuntu:22.04 + +# Install extra needed repos and refresh. +# +# This is all in a single "RUN" command so that if anything changes, "apt update" is run to fetch +# the most current package versions (instead of potentially using old versions cached by docker). +RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y update \ + && DEBIAN_FRONTEND=noninteractive apt-get -qq -y install --no-install-recommends \ + curl='7.81.0-*' \ + git='1:2.34.1-*' \ + gnupg='2.2.27-*' \ + libcurl4-openssl-dev='7.81.0-*' \ + libxml2-dev='2.9.13+dfsg-*' \ + libxml2-utils='2.9.13+dfsg-*' \ + lsof='4.93.2+dfsg-*' \ + openjdk-8-jdk='8u*' \ + python3='3.10.6-1~22.04.1' \ + python2='2.7.18-3' \ + python3-pip='22.0.2+dfsg-*' \ + subversion='1.14.1-*' \ + wget='1.21.2-*' \ + patch='2.7.6-*' \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* \ + && update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java \ + && update-alternatives --install /usr/bin/python python /usr/bin/python2 1 \ + && pip3 install --no-cache-dir python-dateutil==2.8.2 \ + && pip install --no-cache-dir python-dateutil==2.8.2 +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +# Install maven +ENV MAVEN_VERSION=3.8.6 +ARG MAVEN_URL="https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz" +ARG MAVEN_SHA512="f790857f3b1f90ae8d16281f902c689e4f136ebe584aba45e4b1fa66c80cba826d3e0e52fdd04ed44b4c66f6d3fe3584a057c26dfcac544a60b301e6d0f91c26" +RUN mkdir -p /opt/maven \ + && curl -fsSL -o /tmp/apache-maven.tar.gz "${MAVEN_URL}" \ + && echo "${MAVEN_SHA512} /tmp/apache-maven.tar.gz" | sha512sum -c - \ + && tar -xzf /tmp/apache-maven.tar.gz -C /opt/maven --strip-components=1 \ + && rm -f /tmp/apache-maven.tar.gz \ + && ln -s /opt/maven/bin/mvn /usr/bin/mvn + +# Install Apache Yetus +ENV YETUS_VERSION 0.13.0 +SHELL ["/bin/bash", "-o", "pipefail", "-c"] +RUN wget -qO- "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=/yetus/${YETUS_VERSION}/apache-yetus-${YETUS_VERSION}-bin.tar.gz" | \ + tar xvz -C /opt +ENV YETUS_HOME /opt/apache-yetus-${YETUS_VERSION} + +ARG UID +ARG RM_USER +RUN groupadd phoenix-adapters-rm && \ + useradd --create-home --shell /bin/bash -p phoenix-adapters-rm -u $UID $RM_USER && \ + mkdir /home/$RM_USER/.gnupg && \ + chown -R $RM_USER:phoenix-adapters-rm /home/$RM_USER && \ + chmod -R 700 /home/$RM_USER + +USER $RM_USER:phoenix-adapters-rm +WORKDIR /home/$RM_USER/phoenix-adapters-rm/ + +ENTRYPOINT [ "./do-release.sh" ] diff --git a/dev/create-release/rebuild_hbase.sh b/dev/create-release/rebuild_hbase.sh new file mode 120000 index 0000000..53a5e6e --- /dev/null +++ b/dev/create-release/rebuild_hbase.sh @@ -0,0 +1 @@ +../rebuild_hbase.sh \ No newline at end of file diff --git a/dev/create-release/release-build.sh b/dev/create-release/release-build.sh new file mode 100755 index 0000000..ac49008 --- /dev/null +++ b/dev/create-release/release-build.sh @@ -0,0 +1,298 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +trap cleanup EXIT + +# Source in utils. +SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=SCRIPTDIR/release-util.sh +. "$SELF/release-util.sh" + +# Print usage and exit. +function exit_with_usage { + cat <<'EOF' +Usage: release-build.sh +Creates release deliverables from a tag or commit. +Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release' + tag Prepares for release on specified git branch: Set release version, + update CHANGES and RELEASENOTES, create release tag, + increment version for ongoing dev, and publish to Apache git repo. + publish-dist Build and publish distribution packages (tarballs) to Apache dist repo + publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo + publish-release Build and publish maven artifacts release to Apache release repo, and + construct vote email from template + +All other inputs are environment variables. Please use do-release-docker.sh or +do-release.sh to set up the needed environment variables. This script, release-build.sh, +is not intended to be called stand-alone, and such use is untested. The env variables used are: + +Used for 'tag' and 'publish' stages: + PROJECT - The project to build. No default. + RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2) + Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF + RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by + publish-dist as package version name in dist directory path + ASF_USERNAME - Username of ASF committer account + ASF_PASSWORD - Password of ASF committer account + DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but doesn't actually + publish anything to upstream source or object repositories. It defaults to "1", so if you want + to actually publish you have to set '-f' (force) flag in do-release.sh or do-release-docker.sh. + +Used only for 'tag': + YETUS_HOME - installation location for Apache Yetus + GIT_NAME - Name to use with git + GIT_EMAIL - E-mail address to use with git + GIT_BRANCH - Git branch on which to make release. Tag is always placed at HEAD of this branch. + +Used only for 'publish': + GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG; only need to + separately define GIT_REF if RELEASE_TAG is not actually present as a tag at publish time) + If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of master. + GPG_KEY - GPG key id (usually email addr) used to sign release artifacts + REPO - Set to full path of a directory to use as maven local repo (dependencies cache) + to avoid re-downloading dependencies for each stage. It is automatically set if you + request full sequence of stages (tag, publish-dist, publish-release) in do-release.sh. + +For example: + $ PROJECT="phoenix-connectors" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh publish-dist +EOF + exit 1 +} + +set -e + +function cleanup { + # If REPO was set, then leave things be. Otherwise if we defined a repo clean it out. + if [[ -z "${REPO}" ]] && [[ -n "${MAVEN_LOCAL_REPO}" ]]; then + echo "Cleaning up temp repo in '${MAVEN_LOCAL_REPO}'. Set REPO to reuse downloads." >&2 + rm -f "${MAVEN_SETTINGS_FILE}" &> /dev/null || true + rm -rf "${MAVEN_LOCAL_REPO}" &> /dev/null || true + fi +} + +if [ $# -ne 1 ]; then + exit_with_usage +fi + +if [[ "$*" == *"help"* ]]; then + exit_with_usage +fi + +init_locale +init_java +init_mvn +init_python +# Print out subset of perl version (used in git hooks and japi-compliance-checker) +perl --version | grep 'This is' + +rm -rf "${PROJECT}" + +if is_debug; then + set -x # detailed logging during action +fi + +if [[ "$1" == "tag" ]]; then + init_yetus + # for 'tag' stage + set -o pipefail + check_get_passwords ASF_PASSWORD + check_needed_vars PROJECT RELEASE_VERSION RELEASE_TAG GIT_EMAIL GIT_NAME GIT_BRANCH + if [ -z "${GIT_REPO}" ]; then + check_needed_vars ASF_USERNAME ASF_PASSWORD + fi + git_clone_overwrite + + # 'update_releasenotes' searches the project's Jira for issues where 'Fix Version' matches specified + # $jira_fix_version. For most projects this is same as ${RELEASE_VERSION}. However, the original 'phoenix-*' + # projects share the same PHOENIX jira name. To make this work, by convention, the PHOENIX jira "Fix Version" + # field values have the sub-project name without the phoenix- prefix pre-pended, as in "connectors-6.0.0". + # So, here we prepend the project name to the version, but only for the original phoenix sub-projects. + # phoenix-omid and phoenix-tephra have their own JIRA projects and versioning. + jira_fix_version="${RELEASE_VERSION}" + shopt -s nocasematch + if [[ "${PROJECT}" == "phoenix-queryserver" || "${PROJECT}" == "phoenix-connectors" || "${PROJECT}" == "phoenix-thirdparty" || "${PROJECT}" == "phoenix-adapters" ]]; then + # phoenix-adapters shares the PHOENIX JIRA; by convention its "Fix Version" values are + # prefixed with the sub-project name minus the "phoenix-" prefix, e.g. "adapters-1.0.0". + jira_fix_version="${PROJECT#phoenix-}-${RELEASE_VERSION}" + fi + + shopt -u nocasematch + update_releasenotes "$(pwd)/${PROJECT}" "${jira_fix_version}" + + cd "${PROJECT}" + + git config user.name "$GIT_NAME" + git config user.email "$GIT_EMAIL" + + # Create release version + maven_set_version "$RELEASE_VERSION" + git add --ignore-errors RELEASENOTES.md CHANGES.md + + git commit --allow-empty -a -m "Preparing ${PROJECT} release $RELEASE_TAG; tagging and updates to CHANGES.md and RELEASENOTES.md" + echo "Creating tag $RELEASE_TAG at the head of $GIT_BRANCH" + git tag "$RELEASE_TAG" + + if ! is_dry_run; then + # Push changes + git push origin "$RELEASE_TAG" + git push origin "HEAD:$GIT_BRANCH" + cd .. + rm -rf "${PROJECT}" + else + cd .. + mv "${PROJECT}" "${PROJECT}.tag" + echo "Dry run: Clone with version changes and tag available as ${PROJECT}.tag in the output directory." + fi + exit 0 +fi + +### Below is for 'publish-*' stages ### +check_get_passwords ASF_PASSWORD +check_needed_vars PROJECT ASF_USERNAME ASF_PASSWORD GPG_KEY + +# Commit ref to checkout when building +BASE_DIR=$(pwd) +GIT_REF=${GIT_REF:-master} +if [[ "$PROJECT" =~ ^phoenix ]]; then + RELEASE_STAGING_LOCATION="https://dist.apache.org/repos/dist/dev/phoenix" +else + RELEASE_STAGING_LOCATION="https://dist.apache.org/repos/dist/dev/${PROJECT}" +fi + +# in case of dry run, enable publish steps to chain from tag step +if is_dry_run && [[ "${TAG_SAME_DRY_RUN:-}" == "true" && -d "${PROJECT}.tag" ]]; then + ln -s "${PROJECT}.tag" "${PROJECT}" +else + git_clone_overwrite +fi +cd "${PROJECT}" +git checkout "$GIT_REF" +git_hash="$(git rev-parse --short HEAD)" +echo "Checked out ${PROJECT} at ${GIT_REF} commit $git_hash" + +if [ -z "${RELEASE_VERSION}" ]; then + RELEASE_VERSION="$(maven_get_version)" +fi + +# This is a band-aid fix to avoid the failure of Maven nightly snapshot in some Jenkins +# machines by explicitly calling /usr/sbin/lsof. Please see SPARK-22377 and the discussion +# in its pull request. +LSOF=lsof +if ! hash $LSOF 2>/dev/null; then + LSOF=/usr/sbin/lsof +fi + +package_version_name="$RELEASE_TAG" +if [ -z "$package_version_name" ]; then + package_version_name="${RELEASE_VERSION}-$(date +%Y_%m_%d_%H_%M)-${git_hash}" +fi + +git clean -d -f -x +cd .. + +if [[ "$1" == "publish-dist" ]]; then + # Source and binary tarballs + echo "Packaging release source tarballs" + make_src_release "${PROJECT}" "${RELEASE_VERSION}" + + # we do not have binary tarballs for phoenix-thirdparty, omid and tephra + if [[ "${PROJECT}" != "phoenix-thirdparty" && "${PROJECT}" != "phoenix-omid" && "${PROJECT}" != "phoenix-tephra" ]]; then + make_binary_release "${PROJECT}" "${RELEASE_VERSION}" + fi + + DEST_DIR_NAME="${PROJECT}-${package_version_name}" + svn_target="svn-${PROJECT}" + svn co --depth=empty "$RELEASE_STAGING_LOCATION" "$svn_target" + rm -rf "${svn_target:?}/${DEST_DIR_NAME}" + mkdir -p "$svn_target/${DEST_DIR_NAME}" + + echo "Copying release tarballs" + cp "${PROJECT}"-*.tar.* "$svn_target/${DEST_DIR_NAME}/" + cp "${PROJECT}/CHANGES.md" "$svn_target/${DEST_DIR_NAME}/" + cp "${PROJECT}/RELEASENOTES.md" "$svn_target/${DEST_DIR_NAME}/" + shopt -s nocasematch + # Generate api report only if project is hbase for now. - Not for Phoenix + if [ "${PROJECT}" == "hbase" ]; then + # This script usually reports an errcode along w/ the report. + generate_api_report "./${PROJECT}" "${API_DIFF_TAG}" "${GIT_REF}" || true + cp api*.html "$svn_target/${DEST_DIR_NAME}/" + fi + shopt -u nocasematch + + svn add "$svn_target/${DEST_DIR_NAME}" + + if ! is_dry_run; then + cd "$svn_target" + svn ci --username "$ASF_USERNAME" --password "$ASF_PASSWORD" -m"Apache ${PROJECT} $package_version_name" --no-auth-cache + cd .. + rm -rf "$svn_target" + else + mv "$svn_target/${DEST_DIR_NAME}" "${svn_target}_${DEST_DIR_NAME}.dist" + echo "Dry run: svn-managed 'dist' directory with release tarballs, CHANGES.md and RELEASENOTES.md available as $(pwd)/${svn_target}_${DEST_DIR_NAME}.dist" + rm -rf "$svn_target" + fi + + exit 0 +fi + +if [[ "$1" == "publish-snapshot" ]]; then + ( + cd "${PROJECT}" + mvn_log="${BASE_DIR}/mvn_deploy_snapshot.log" + echo "Publishing snapshot to nexus" + maven_deploy snapshot "$mvn_log" + if ! is_dry_run; then + echo "Snapshot artifacts successfully published to repo." + rm "$mvn_log" + else + echo "Dry run: Snapshot artifacts successfully built, but not published due to dry run." + fi + ) + exit $? +fi + +if [[ "$1" == "publish-release" ]]; then + ( + cd "${PROJECT}" + mvn_log="${BASE_DIR}/mvn_deploy_release.log" + echo "Staging release in nexus" + maven_deploy release "$mvn_log" + declare staged_repo_id="dryrun-no-repo" + if ! is_dry_run; then + staged_repo_id=$(grep -o "Closing staging repository with ID .*" "$mvn_log" \ + | sed -e 's/Closing staging repository with ID "\([^"]*\)"./\1/') + echo "Release artifacts successfully published to repo ${staged_repo_id}" + rm "$mvn_log" + else + echo "Dry run: Release artifacts successfully built, but not published due to dry run." + fi + # Dump out email to send. Where we find vote.tmpl depends + # on where this script is run from + PROJECT_TEXT="${PROJECT//-/ }" #substitute like 's/-/ /g' + export PROJECT_TEXT + eval "echo \"$(< "${SELF}/vote.tmpl")\"" |tee "${BASE_DIR}/vote.txt" + ) + exit $? +fi + +set +x # done with detailed logging +cd .. +rm -rf "${PROJECT}" +echo "ERROR: expects to be called with 'tag', 'publish-dist', 'publish-release', or 'publish-snapshot'" >&2 +exit_with_usage diff --git a/dev/create-release/release-util.sh b/dev/create-release/release-util.sh new file mode 100755 index 0000000..7116155 --- /dev/null +++ b/dev/create-release/release-util.sh @@ -0,0 +1,785 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +DRY_RUN=${DRY_RUN:-1} #default to dry run +DEBUG=${DEBUG:-0} +GPG=${GPG:-gpg} +GPG_ARGS=(--no-autostart --batch --pinentry-mode error) +if [ -n "${GPG_KEY}" ]; then + GPG_ARGS=("${GPG_ARGS[@]}" --local-user "${GPG_KEY}") +fi +# Maven Profiles for publishing snapshots and release to Maven Central and Dist +PUBLISH_PROFILES=("-P" "apache-release,release") + +set -e + +function error { + echo "Error: $*" >&2 + exit 1 +} + +function read_config { + local PROMPT="$1" + local DEFAULT="$2" + local REPLY= + + read -r -p "$PROMPT [$DEFAULT]: " REPLY + local RETVAL="${REPLY:-$DEFAULT}" + if [ -z "$RETVAL" ]; then + error "$PROMPT must be provided." + fi + echo "$RETVAL" +} + +function parse_version { + # Heuristics to skip the parent version + grep -e '.*\..*' | \ + head -n 1 | tail -n 1 | cut -d'>' -f2 | cut -d '<' -f1 +} + +function banner { + local msg="$1" + echo "========================" + echo "=== ${msg}" + echo +} + +# current number of seconds since epoch +function get_ctime { + date +"%s" +} + +function run_silent { + local BANNER="$1" + local LOG_FILE="$2" + shift 2 + local -i start_time + local -i stop_time + + banner "${BANNER}" + echo "Command: $*" + echo "Log file: $LOG_FILE" + start_time="$(get_ctime)" + + if ! "$@" 1>"$LOG_FILE" 2>&1; then + echo "Command FAILED. Check full logs for details." + tail "$LOG_FILE" + exit 1 + fi + stop_time="$(get_ctime)" + echo "=== SUCCESS ($((stop_time - start_time)) seconds)" +} + +function fcreate_secure { + local FPATH="$1" + rm -f "$FPATH" + touch "$FPATH" + chmod 600 "$FPATH" +} + +# API compare version. +function get_api_diff_version { + local version="$1" + local rev + local api_diff_tag + rev=$(echo "$version" | cut -d . -f 3) + if [ "$rev" != 0 ]; then + local short_version + short_version="$(echo "$version" | cut -d . -f 1-2)" + api_diff_tag="rel/${short_version}.$((rev - 1))" + else + local major minor + major="$(echo "$version" | cut -d . -f 1)" + minor="$(echo "$version" | cut -d . -f 2)" + if [ "$minor" != 0 ]; then + api_diff_tag="rel/${major}.$((minor - 1)).0" + else + api_diff_tag="rel/$((major - 1)).0.0" + fi + fi + api_diff_tag="$(read_config "api_diff_tag" "$api_diff_tag")" + echo "$api_diff_tag" +} + +# Get all branches that begin with 'branch-', the hbase convention for +# release branches, sort them and then pop off the most recent. +function get_release_info { + PROJECT="$(read_config "PROJECT" "$PROJECT")" + export PROJECT + + if [[ -z "${ASF_REPO}" ]]; then + ASF_REPO="https://gitbox.apache.org/repos/asf/${PROJECT}.git" + fi + if [[ -z "${ASF_REPO_WEBUI}" ]]; then + ASF_REPO_WEBUI="https://gitbox.apache.org/repos/asf?p=${PROJECT}.git" + fi + if [[ -z "${ASF_GITHUB_REPO}" ]]; then + ASF_GITHUB_REPO="https://github.com/apache/${PROJECT}" + fi + if [ -z "$GIT_BRANCH" ]; then + # If no branch is specified, find out the latest branch from the repo. + GIT_BRANCH="$(git ls-remote --heads "$ASF_REPO" | + grep refs/heads/branch- | + awk '{print $2}' | + sort -r | + head -n 1 | + cut -d/ -f3)" + fi + + GIT_BRANCH="$(read_config "GIT_BRANCH" "$GIT_BRANCH")" + export GIT_BRANCH + + # Find the current version for the branch. + # FIXME this only works with gitbox + local version + version="$(curl -L -s "$ASF_REPO_WEBUI;a=blob_plain;f=pom.xml;hb=refs/heads/$GIT_BRANCH" | + parse_version)" + echo "Current branch VERSION is $version." + + RELEASE_VERSION="" + SHORT_VERSION="$(echo "$version" | cut -d . -f 1-2)" + if [[ ! "$version" =~ .*-SNAPSHOT ]]; then + RELEASE_VERSION="$version" + else + RELEASE_VERSION="${version/-SNAPSHOT/}" + fi + + local REV + REV="$(echo "${RELEASE_VERSION}" | cut -d . -f 3)" + + # Find out what RC is being prepared. + # - If the current version is "x.y.0", then this is RC0 of the "x.y.0" release. + # - If not, need to check whether the previous version has been already released or not. + # - If it has, then we're building RC0 of the current version. + # - If it has not, we're building the next RC of the previous version. + local RC_COUNT + if [ "$REV" != 0 ]; then + local PREV_REL_REV=$((REV - 1)) + PREV_REL_TAG="rel/${SHORT_VERSION}.${PREV_REL_REV}" + if git ls-remote --tags "$ASF_REPO" "$PREV_REL_TAG" | grep -q "refs/tags/${PREV_REL_TAG}$" ; then + RC_COUNT=0 + REV=$((REV + 1)) + else + RELEASE_VERSION="${SHORT_VERSION}.${PREV_REL_REV}" + RC_COUNT="$(git ls-remote --tags "$ASF_REPO" "${RELEASE_VERSION}RC*" | wc -l)" + # This makes a 'number' of it. + RC_COUNT=$((RC_COUNT)) + fi + else + REV=$((REV + 1)) + RC_COUNT=0 + fi + + RELEASE_VERSION="$(read_config "RELEASE_VERSION" "$RELEASE_VERSION")" + export RELEASE_VERSION + + RC_COUNT="$(read_config "RC_COUNT" "$RC_COUNT")" + RELEASE_TAG="${RELEASE_VERSION}RC${RC_COUNT}" + RELEASE_TAG="$(read_config "RELEASE_TAG" "$RELEASE_TAG")" + + # Check if the RC already exists, and if re-creating the RC, skip tag creation. + SKIP_TAG=0 + if git ls-remote --tags "$ASF_REPO" "$RELEASE_TAG" | grep -q "refs/tags/${RELEASE_TAG}$" ; then + read -r -p "$RELEASE_TAG already exists. Continue anyway [y/n]? " ANSWER + if [ "$ANSWER" != "y" ]; then + echo "Exiting." + exit 1 + fi + SKIP_TAG=1 + fi + + export RELEASE_TAG SKIP_TAG + + GIT_REF="$RELEASE_TAG" + if is_dry_run; then + echo "This is a dry run. If tag does not actually exist, please confirm the ref that will be built for testing." + GIT_REF="$(read_config "GIT_REF" "$GIT_REF")" + fi + export GIT_REF + + if [ "${PROJECT}" == "hbase" ]; then + API_DIFF_TAG="$(get_api_diff_version "$RELEASE_VERSION")" + fi + + # Gather some user information. + ASF_USERNAME="$(read_config "ASF_USERNAME" "$LOGNAME")" + + GIT_NAME="$(git config user.name || echo "")" + GIT_NAME="$(read_config "GIT_NAME" "$GIT_NAME")" + + GIT_EMAIL="$ASF_USERNAME@apache.org" + GPG_KEY="$(read_config "GPG_KEY" "$GIT_EMAIL")" + if ! GPG_KEY_ID=$("${GPG}" "${GPG_ARGS[@]}" --keyid-format 0xshort --list-public-key "${GPG_KEY}" | grep "\[S\]" | grep -o "0x[0-9A-F]*") || + [ -z "${GPG_KEY_ID}" ] ; then + GPG_KEY_ID=$("${GPG}" "${GPG_ARGS[@]}" --keyid-format 0xshort --list-public-key "${GPG_KEY}" | head -n 1 | grep -o "0x[0-9A-F]*" || true) + fi + read -r -p "We think the key '${GPG_KEY}' corresponds to the key id '${GPG_KEY_ID}'. Is this correct [y/n]? " ANSWER + if [ "$ANSWER" = "y" ]; then + GPG_KEY="${GPG_KEY_ID}" + fi + export API_DIFF_TAG ASF_USERNAME GIT_NAME GIT_EMAIL GPG_KEY + + cat < 0 )) && exit_with_usage + return 0 +} + +function init_locale { + local locale_value + OS="$(uname -s)" + case "${OS}" in + Darwin*) locale_value="en_US.UTF-8";; + Linux*) locale_value="C.UTF-8";; + *) error "unknown OS";; + esac + export LC_ALL="$locale_value" + export LANG="$locale_value" +} + +# Initializes JAVA_VERSION to the version of the JVM in use. +function init_java { + if [ -z "$JAVA_HOME" ]; then + error "JAVA_HOME is not set." + fi + JAVA_VERSION=$("${JAVA_HOME}"/bin/javac -version 2>&1 | cut -d " " -f 2) + echo "java version: $JAVA_VERSION" + export JAVA_VERSION +} + +function init_python { + if ! [ -x "$(command -v python2)" ]; then + error 'python2 needed by yetus. Install or add link? E.g: sudo ln -sf /usr/bin/python2.7 /usr/local/bin/python2' + fi + echo "python version: $(python2 --version)" +} + +# Set MVN +function init_mvn { + if [ -n "$MAVEN_HOME" ]; then + MVN=("${MAVEN_HOME}/bin/mvn") + elif [ "$(type -P mvn)" ]; then + MVN=(mvn) + else + error "MAVEN_HOME is not set nor is mvn on the current path." + fi + # Add batch mode. + MVN=("${MVN[@]}" -B) + export MVN + echo -n "mvn version: " + "${MVN[@]}" --version + configure_maven +} + +function init_yetus { + declare YETUS_VERSION + if [ -z "${YETUS_HOME}" ]; then + error "Missing Apache Yetus." + fi + # Work around yetus bug by asking test-patch for the version instead of rdm. + YETUS_VERSION=$("${YETUS_HOME}/bin/test-patch" --version) + echo "Apache Yetus version ${YETUS_VERSION}" +} + +function configure_maven { + # Add timestamps to mvn logs. + MAVEN_OPTS="-Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss ${MAVEN_OPTS}" + # Suppress gobs of "Download from central:" messages + MAVEN_OPTS="-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn ${MAVEN_OPTS}" + MAVEN_LOCAL_REPO="${REPO:-$(pwd)/$(mktemp -d hbase-repo-XXXXX)}" + [[ -d "$MAVEN_LOCAL_REPO" ]] || mkdir -p "$MAVEN_LOCAL_REPO" + MAVEN_SETTINGS_FILE="${MAVEN_LOCAL_REPO}/tmp-settings.xml" + MVN=("${MVN[@]}" --settings "${MAVEN_SETTINGS_FILE}") + export MVN MAVEN_OPTS MAVEN_SETTINGS_FILE MAVEN_LOCAL_REPO + export ASF_USERNAME ASF_PASSWORD + # reference passwords from env rather than storing in the settings.xml file. + cat <<'EOF' > "$MAVEN_SETTINGS_FILE" + + + /${env.MAVEN_LOCAL_REPO} + + apache.snapshots.https${env.ASF_USERNAME} + ${env.ASF_PASSWORD} + apache.releases.https${env.ASF_USERNAME} + ${env.ASF_PASSWORD} + + + + + true + + + ${env.GPG_KEY} + + + + +EOF +} + +# clone of the repo, deleting anything that exists in the working directory named after the project. +# optionally with auth details for pushing. +function git_clone_overwrite { + local asf_repo + if [ -z "${PROJECT}" ] || [ "${PROJECT}" != "${PROJECT#/}" ]; then + error "Project name must be defined and not start with a '/'. PROJECT='${PROJECT}'" + fi + rm -rf "${PROJECT}" + + if [[ -z "${GIT_REPO}" ]]; then + asf_repo="gitbox.apache.org/repos/asf/${PROJECT}.git" + echo "[INFO] clone will be of the gitbox repo for ${PROJECT}." + if [ -n "${ASF_USERNAME}" ] && [ -n "${ASF_PASSWORD}" ]; then + # Ugly! + encoded_username=$(python -c "import urllib; print urllib.quote('''$ASF_USERNAME''', '')") + encoded_password=$(python -c "import urllib; print urllib.quote('''$ASF_PASSWORD''', '')") + GIT_REPO="https://$encoded_username:$encoded_password@${asf_repo}" + else + GIT_REPO="https://${asf_repo}" + fi + else + echo "[INFO] clone will be of provided git repo." + fi + # N.B. we use the shared flag because the clone is short lived and if a local repo repo was + # given this will let us refer to objects there directly instead of hardlinks or copying. + # The option is silently ignored for non-local repositories. see the note on git help clone + # for the --shared option for details. + git clone --shared -b "${GIT_BRANCH}" -- "${GIT_REPO}" "${PROJECT}" + # If this was a host local git repo then add in an alternates and remote that will + # work back on the host if the RM needs to do any post-processing steps, i.e. pushing the git tag + # for more info see 'git help remote' and 'git help repository-layout'. + if [ -n "$HOST_GIT_REPO" ]; then + echo "${HOST_GIT_REPO}/objects" >> "${PROJECT}/.git/objects/info/alternates" + (cd "${PROJECT}"; git remote add host "${HOST_GIT_REPO}") + fi +} + +function start_step { + local name=$1 + if [ -z "${name}" ]; then + name="${FUNCNAME[1]}" + fi + echo "$(date -u +'%Y-%m-%dT%H:%M:%SZ') ${name} start" >&2 + get_ctime +} + +function stop_step { + local name=$2 + local start_time=$1 + local stop_time + if [ -z "${name}" ]; then + name="${FUNCNAME[1]}" + fi + stop_time="$(get_ctime)" + echo "$(date -u +'%Y-%m-%dT%H:%M:%SZ') ${name} stop ($((stop_time - start_time)) seconds)" +} + +# Writes report into cwd! +# TODO should have option for maintenance release that include LimitedPrivate in report +function generate_api_report { + local project="$1" + local previous_tag="$2" + local release_tag="$3" + local previous_version + local timing_token + timing_token="$(start_step)" + # Generate api report. + "${project}"/dev-support/checkcompatibility.py --annotation \ + org.apache.yetus.audience.InterfaceAudience.Public \ + "$previous_tag" "$release_tag" + previous_version="$(echo "${previous_tag}" | sed -e 's/rel\///')" + cp "${project}/target/compat-check/report.html" "./api_compare_${previous_version}_to_${release_tag}.html" + stop_step "${timing_token}" +} + +# Look up the Jira name associated with project. +# Currently all the 'hbase-*' projects share the same HBASE jira name. This works because, +# by convention, the HBASE jira "Fix Version" field values have the sub-project name pre-pended, +# as in "hbase-operator-tools-1.0.0". +# TODO: For non-hbase-related projects, enhance this to use Jira API query instead of text lookup. +function get_jira_name { + local project="$1" + local jira_name + case "${project}" in + phoenix-tephra) jira_name="TEPHRA";; + phoenix-omid) jira_name="OMID";; + phoenix*) jira_name="PHOENIX";; + *) jira_name="";; + esac + if [[ -z "$jira_name" ]]; then + error "Sorry, can't determine the Jira name for project $project" + fi + echo "$jira_name" +} + +# Update the CHANGES.md +# DOES NOT DO COMMITS! Caller should do that. +# requires yetus to have a defined home already. +# yetus requires python2 to be on the path. +function update_releasenotes { + local project_dir="$1" + local jira_fix_version="$2" + local jira_project + local timing_token + timing_token="$(start_step)" + jira_project="$(get_jira_name "$(basename "$project_dir")")" + "${YETUS_HOME}/bin/releasedocmaker" -p "${jira_project}" --fileversions -v "${jira_fix_version}" \ + -l --sortorder=newer --skip-credits + pwd + # First clear out the changes written by previous RCs. + if [ -f "${project_dir}/CHANGES.md" ]; then + sed -i -e \ + "/^## Release ${jira_fix_version}/,/^## Release/ {//!d; /^## Release ${jira_fix_version}/d;}" \ + "${project_dir}/CHANGES.md" || true + fi + if [ -f "${project_dir}/RELEASENOTES.md" ]; then + sed -i -e \ + "/^# ${jira_project} ${jira_fix_version} Release Notes/,/^# ${jira_project}/{//!d; /^# ${jira_project} ${jira_fix_version} Release Notes/d;}" \ + "${project_dir}/RELEASENOTES.md" || true + fi + + # The releasedocmaker call above generates RELEASENOTES.X.X.X.md and CHANGELOG.X.X.X.md. + if [ -f "${project_dir}/CHANGES.md" ]; then + # To insert into project's CHANGES.md...need to cut the top off the + # CHANGELOG.X.X.X.md file removing license and then + # insert it after the license comment closing + sed -i -e '/^# .* Changelog$/,$!d' "CHANGELOG.${jira_fix_version}.md" + sed -i -e '1d' "CHANGELOG.${jira_fix_version}.md" + sed -i -e "/^# .* Changelog$/r CHANGELOG.${jira_fix_version}.md" "${project_dir}/CHANGES.md" + else + mv "CHANGELOG.${jira_fix_version}.md" "${project_dir}/CHANGES.md" + fi + if [ -f "${project_dir}/RELEASENOTES.md" ]; then + # Similar for RELEASENOTES but slightly different. + sed -i -e '/-->/,$!d' "RELEASENOTES.${jira_fix_version}.md" + sed -i -e '1d' "RELEASENOTES.${jira_fix_version}.md" + sed -i -e "/^-->$/r RELEASENOTES.${jira_fix_version}.md" \ + "${project_dir}/RELEASENOTES.md" + else + mv "RELEASENOTES.${jira_fix_version}.md" "${project_dir}/RELEASENOTES.md" + fi + stop_step "${timing_token}" +} + +# Make src release. +# Takes as arguments first the project name -- e.g. hbase or hbase-operator-tools +# -- and then the version string. Expects to find checkout adjacent to this script +# named for 'project', the first arg passed.q +# Expects the following three defines in the environment: +# - GPG needs to be defined, with the path to GPG: defaults 'gpg'. +# - GIT_REF which is the tag to create the tgz from: defaults to 'master'. +# For example: +# $ GIT_REF="master" make_src_release hbase-operator-tools 1.0.0 +make_src_release() { + # Tar up the src and sign and hash it. + local project="${1}" + local version="${2}" + local base_name="${project}-${version}" + local timing_token + timing_token="$(start_step)" + rm -rf "${base_name}"-src* + tgz="${base_name}-src.tar.gz" + cd "${project}" || exit + git clean -d -f -x + git archive --format=tar.gz --output="../${tgz}" --prefix="${base_name}/" "${GIT_REF:-master}" + cd .. || exit + $GPG "${GPG_ARGS[@]}" --armor --output "${tgz}.asc" --detach-sig "${tgz}" + $GPG "${GPG_ARGS[@]}" --print-md SHA512 "${tgz}" > "${tgz}.sha512" + stop_step "${timing_token}" +} + +# Make binary release. +# Takes as arguments first the project name -- e.g. hbase or hbase-operator-tools +# -- and then the version string. Expects to find checkout adjacent to this script +# named for 'project', the first arg passed. +# Expects the following three defines in the environment: +# - GPG needs to be defined, with the path to GPG: defaults 'gpg'. +# - GIT_REF which is the tag to create the tgz from: defaults to 'master'. +# - MVN Default is "mvn -B --settings $MAVEN_SETTINGS_FILE". +# For example: +# $ GIT_REF="master" make_src_release hbase-operator-tools 1.0.0 +make_binary_release() { + local project="${1}" + local version="${2}" + local timing_token + timing_token="$(start_step)" + + cd "$project" || exit + if [[ "$project" == "phoenix" ]]; then + rebuild_hbase_2 + local profiles=( $(maven_get_hbase_profiles) ) + elif [[ "$project" == "phoenix-omid" ]]; then + rebuild_hbase_for_omid + local profiles=( "" ) + else + local profiles=( "" ) + fi + + # The Phoenix scripts assume the assembly module is named "${PROJECT}-assembly". Projects that + # keep a different assembly module name (e.g. phoenix-adapters -> phoenix-ddb-assembly) can + # override this via the ASSEMBLY_MODULE environment variable. + local assembly_module="${ASSEMBLY_MODULE:-${PROJECT}-assembly}" + + for profile in "${profiles[@]}"; do + if [ -z $profile ]; then + local hbase_suffix="" + else + local hbase_suffix="-hbase-${profile}" + fi + local base_name="${project}${hbase_suffix}-${version}" + rm -rf ../"${base_name}"-bin* + + git clean -d -f -x + kick_gpg_agent + # Only pass -Dhbase.profile for projects that actually define an hbase profile matrix; + # for single-artifact projects (empty profile) passing an empty value is undesirable. + if [ -z "$profile" ]; then + "${MVN[@]}" clean package -DskipTests -Dcheckstyle.skip=true "${PUBLISH_PROFILES[@]}" + else + "${MVN[@]}" clean package -DskipTests -Dcheckstyle.skip=true "${PUBLISH_PROFILES[@]}" "-Dhbase.profile=${profile}" + fi + + # Check there is a bin gz output. The build may not produce one: e.g. hbase-thirdparty. + local f_bin_prefix="./${assembly_module}/target/${base_name}" + if ls "${f_bin_prefix}"*-bin.tar.gz &>/dev/null; then + cp "${f_bin_prefix}"*-bin.tar.gz .. + cd .. || exit + for i in "${base_name}"*-bin.tar.gz; do + "${GPG}" "${GPG_ARGS[@]}" --armour --output "${i}.asc" --detach-sig "${i}" + "${GPG}" "${GPG_ARGS[@]}" --print-md SHA512 "${i}" > "${i}.sha512" + done + cd "$project" || exit + else + echo "No ${f_bin_prefix}*-bin.tar.gz product; expected?" + fi + done + cd .. || exit + stop_step "${timing_token}" +} + +# "Wake up" the gpg agent so it responds properly to maven-gpg-plugin, and doesn't cause timeout. +# Specifically this is done between invocation of 'mvn site' and 'mvn assembly:single', because +# the 'site' build takes long enough that the gpg-agent does become unresponsive and the following +# 'assembly' build (where gpg signing occurs) experiences timeout, without this "kick". +function kick_gpg_agent { + # All that's needed is to run gpg on a random file + # TODO could we just call gpg-connect-agent /bye + local i + i="$(mktemp)" + echo "This is a test file" > "$i" + "${GPG}" "${GPG_ARGS[@]}" --armour --output "${i}.asc" --detach-sig "${i}" + rm "$i" "$i.asc" +} + +# Do maven command to set version into local pom +function maven_set_version { #input: + local this_version="$1" + echo "${MVN[@]}" versions:set -DnewVersion="$this_version" + "${MVN[@]}" versions:set -DnewVersion="$this_version" | grep -v "no value" # silence logs +} + +# Do maven command to read version from local pom +function maven_get_version { + # shellcheck disable=SC2016 + "${MVN[@]}" -q -N -Dexec.executable="echo" -Dexec.args='${project.version}' exec:exec +} + +function maven_get_hbase_profiles { + # shellcheck disable=SC2016 + "${MVN[@]}" -q -N -Dexec.executable="echo" -Dexec.args='${hbase.profile.list}' exec:exec +} + +function maven_get_hbase_version_from_profile { + # shellcheck disable=SC2016 + local profile="$1" + "${MVN[@]}" -q -N -Dexec.executable="echo" -Dexec.args='${hbase-'"$profile"'.runtime.version}' exec:exec +} + +function maven_get_omid_hbase_version { + # shellcheck disable=SC2016 + "${MVN[@]}" -q -N -Dexec.executable="echo" -Dexec.args='${hbase.version}' exec:exec +} + +# Do maven deploy to snapshot or release artifact repository, with checks. +function maven_deploy { #inputs: + local timing_token + # Invoke with cwd=$PROJECT + local deploy_type="$1" + local mvn_log_file="$2" #secondary log file used later to extract staged_repo_id + if [[ "$deploy_type" != "snapshot" && "$deploy_type" != "release" ]]; then + error "unrecognized deploy type, must be 'snapshot'|'release'" + fi + if [[ -z "$mvn_log_file" ]] || ! touch "$mvn_log_file"; then + error "must provide writable maven log output filepath" + fi + timing_token=$(start_step) + # shellcheck disable=SC2153 + if [[ "$deploy_type" == "snapshot" ]] && ! [[ "$RELEASE_VERSION" =~ -SNAPSHOT$ ]]; then + error "Snapshots must have a version with suffix '-SNAPSHOT'; you gave version '$RELEASE_VERSION'" + elif [[ "$deploy_type" == "release" ]] && [[ "$RELEASE_VERSION" =~ SNAPSHOT ]]; then + error "Non-snapshot release version must not include the word 'SNAPSHOT'; you gave version '$RELEASE_VERSION'" + fi + # Publish ${PROJECT} to Maven repo + # shellcheck disable=SC2154 + echo "Publishing ${PROJECT} checkout at '$GIT_REF' ($git_hash)" + echo "Publish version is $RELEASE_VERSION" + # Coerce the requested version + maven_set_version "$RELEASE_VERSION" + + declare -a mvn_goals=( clean deploy ) + if is_dry_run; then + echo "Dry run: Deployed artifacts are available as ${PROJECT}.deploy in the output directory." + DRY_DEPLOY_DIR="${BASE_DIR}/${PROJECT}.deploy" + dry_deploy="-DaltDeploymentRepository=dryrun::default::file://${DRY_DEPLOY_DIR}" + fi + if [[ "$PROJECT" == "phoenix-omid" ]]; then + rebuild_hbase_for_omid + local variants=( "" ) + elif [[ "$PROJECT" == "phoenix" ]]; then + rebuild_hbase_2 + local profiles=( $(maven_get_hbase_profiles) ) + local variants=() + for i in "${profiles[@]}"; do + variants+=("-Dhbase.profile=$i") + done + else + local variants=( "" ) + fi + rm -f "$mvn_log_file" + for variant in "${variants[@]}"; do + echo "${MVN[@]}" -DskipTests -Dcheckstyle.skip=true ${dry_deploy:+"$dry_deploy"} "${PUBLISH_PROFILES[@]}" ${variant:+"$variant"} \ + "${mvn_goals[@]}" + echo "Logging to ${mvn_log_file}. This will take a while..." + git clean -d -f -x + # Prepare for signing + kick_gpg_agent + # The tortuous redirect in the next command allows mvn's stdout and stderr to go to mvn_log_file, + # while also sending stderr back to the caller. + # shellcheck disable=SC2094 + if ! "${MVN[@]}" -DskipTests -Dcheckstyle.skip=true ${dry_deploy:+"$dry_deploy"} "${PUBLISH_PROFILES[@]}" ${variant:+"$variant"} \ + "${mvn_goals[@]}" 1>> "$mvn_log_file" 2> >( tee -a "$mvn_log_file" >&2 ); then + error "Deploy build failed, for details see log at '$mvn_log_file'." + fi + done + echo "BUILD SUCCESS." + stop_step "${timing_token}" + return 0 +} + +# guess the host os +# * DARWIN +# * LINUX +function get_host_os() { + uname -s | tr '[:lower:]' '[:upper:]' +} + +function rebuild_hbase_2() { + if [[ "yes" == "$HBASE_ALREADY_REBUILT" ]]; then + return 0 + fi + local profiles=( $(maven_get_hbase_profiles) ) + for i in "${profiles[@]}"; do + local hbase_runtime_version=$(maven_get_hbase_version_from_profile "$i") + if [[ $hbase_runtime_version == 2* ]]; then + rebuild_hbase_locally "$hbase_runtime_version" + fi + done + HBASE_ALREADY_REBUILT="yes" +} + +function rebuild_hbase_for_omid() { + if [[ "yes" == "$HBASE_OMID_ALREADY_REBUILT" ]]; then + return 0 + fi + local hbase_runtime_version=$(maven_get_omid_hbase_version) + if [[ $hbase_runtime_version == 2* ]]; then + rebuild_hbase_locally "$hbase_runtime_version" + fi + HBASE_OMID_ALREADY_REBUILT="yes" +} + + +function rebuild_hbase_locally() { + local hbase_version="$1" + MAVEN_SETTINGS_FILE="$MAVEN_SETTINGS_FILE" "$SELF"/rebuild_hbase.sh "$hbase_version" +} diff --git a/dev/create-release/vote.tmpl b/dev/create-release/vote.tmpl new file mode 100644 index 0000000..bbfbbdd --- /dev/null +++ b/dev/create-release/vote.tmpl @@ -0,0 +1,31 @@ +Please vote on this Apache ${PROJECT_TEXT} release candidate, +${PROJECT}-${RELEASE_TAG} + +The VOTE will remain open for at least 72 hours. + +[ ] +1 Release this package as Apache ${PROJECT_TEXT} ${RELEASE_VERSION} +[ ] -1 Do not release this package because ... + +The tag to be voted on is ${RELEASE_TAG}: + + https://github.com/apache/${PROJECT}/tree/${RELEASE_TAG} + +The release files, including signatures, digests, as well as CHANGES.md +and RELEASENOTES.md included in this RC can be found at: + + https://dist.apache.org/repos/dist/dev/phoenix/${RELEASE_TAG}/ + +Maven artifacts are available in a staging repository at: + + https://repository.apache.org/content/repositories/${staged_repo_id}/ + +Artifacts were signed with the ${GPG_KEY} key which can be found in: + + https://dist.apache.org/repos/dist/release/phoenix/KEYS + +To learn more about Apache ${PROJECT_TEXT}, please see + + https://phoenix.apache.org/ + +Thanks, +Your Phoenix Release Manager diff --git a/dev/phoenix-adapters-vote.sh b/dev/phoenix-adapters-vote.sh new file mode 100755 index 0000000..473e299 --- /dev/null +++ b/dev/phoenix-adapters-vote.sh @@ -0,0 +1,190 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e -o pipefail + +usage() { + SCRIPT=$(basename "${BASH_SOURCE[@]}") + + cat << __EOF +phoenix-adapters-vote. A script for standard vote which verifies the following items +1. Checksum of sources and binaries +2. Signature of sources and binaries +3. Rat check +4. Built from source +5. Unit tests + +Usage: ${SCRIPT} -s | --source [-k | --key ] [-f | --keys-file-url ] [-o | --output-dir ] [-D property[=value]] + ${SCRIPT} -h | --help + + -h | --help Show this screen. + -s | --source '' A URL pointing to the release candidate sources and binaries + e.g. https://dist.apache.org/repos/dist/dev/phoenix/phoenix-adapters-RC0/ + -k | --key '' A signature of the public key, e.g. 9AD2AE49 + -f | --keys-file-url '' the URL of the key file, default is + https://downloads.apache.org/phoenix/KEYS + -o | --output-dir '' directory which has the stdout and stderr of each verification target + -D | list of maven properties to set for the mvn invocations, i.e. <-D hbase.profile=2.4 -DskipTests> Defaults to unset +__EOF +} + +MVN_PROPERTIES=() + +while ((${#})); do + case "${1}" in + -h | --help ) + usage; exit 0 ;; + -s | --source ) + SOURCE_URL="${2}"; shift 2 ;; + -k | --key ) + SIGNING_KEY="${2}"; shift 2 ;; + -f | --keys-file-url ) + KEY_FILE_URL="${2}"; shift 2 ;; + -o | --output-dir ) + OUTPUT_DIR="${2}"; shift 2 ;; + -D ) + MVN_PROPERTIES+=("-D ${2}"); shift 2 ;; + * ) + usage >&2; exit 1 ;; + esac +done + +# Source url must be provided +if [ -z "${SOURCE_URL}" ]; then + usage; + exit 1 +fi + +cat << __EOF +Although This tool helps verifying Phoenix Adapters RC build and unit tests, +operator may still consider to verify the following manually +1. Verify the CHANGES.md and RELEASENOTES.md +2. Any on cluster Integration test or performance test +3. Other concerns if any +__EOF + +[[ "${SOURCE_URL}" != */ ]] && SOURCE_URL="${SOURCE_URL}/" +# e.g. .../dev/phoenix/phoenix-adapters-1.0.0RC0/ -> PHOENIX_RC_VERSION=phoenix-adapters-1.0.0RC0 +PHOENIX_RC_VERSION=$(tr "/" "\n" <<< "${SOURCE_URL}" | tail -n2) +# Strip the "phoenix-adapters-" prefix and the trailing "RC" to get the bare version, e.g. 1.0.0 +PHOENIX_VERSION=$(echo "${PHOENIX_RC_VERSION}" | sed -e 's/RC[0-9]*//g' | sed -e 's/phoenix-adapters-//g') +JAVA_VERSION=$(java -version 2>&1 | cut -f3 -d' ' | head -n1 | sed -e 's/"//g') +OUTPUT_DIR="${OUTPUT_DIR:-$(pwd)}" + +if [ ! -d "${OUTPUT_DIR}" ]; then + echo "Output directory ${OUTPUT_DIR} does not exist, please create it before running this script." + exit 1 +fi + +OUTPUT_PATH_PREFIX="${OUTPUT_DIR}"/"${PHOENIX_RC_VERSION}" + +# default value for verification targets, 0 = failed +SIGNATURE_PASSED=0 +CHECKSUM_PASSED=0 +RAT_CHECK_PASSED=0 +BUILD_FROM_SOURCE_PASSED=0 +UNIT_TEST_PASSED=0 + +function download_and_import_keys() { + KEY_FILE_URL="${KEY_FILE_URL:-https://downloads.apache.org/phoenix/KEYS}" + echo "Obtain and import the publisher key(s) from ${KEY_FILE_URL}" + # download the keys file into file KEYS + wget -O KEYS "${KEY_FILE_URL}" + gpg --import KEYS + if [ -n "${SIGNING_KEY}" ]; then + gpg --list-keys "${SIGNING_KEY}" + fi +} + +function download_release_candidate () { + # get all files from release candidate repo + wget -r -np -N -nH --cut-dirs 4 "${SOURCE_URL}" +} + +function verify_signatures() { + rm -f "${OUTPUT_PATH_PREFIX}"_verify_signatures + for file in *.tar.gz; do + gpg --verify "${file}".asc "${file}" 2>&1 | tee -a "${OUTPUT_PATH_PREFIX}"_verify_signatures && SIGNATURE_PASSED=1 || SIGNATURE_PASSED=0 + done +} + +function verify_checksums() { + rm -f "${OUTPUT_PATH_PREFIX}"_verify_checksums + SHA_EXT=$(find . -name "*.sha*" | awk -F '.' '{ print $NF }' | head -n 1) + for file in *.tar.gz; do + gpg --print-md SHA512 "${file}" > "${file}"."${SHA_EXT}".tmp + diff "${file}"."${SHA_EXT}".tmp "${file}"."${SHA_EXT}" 2>&1 | tee -a "${OUTPUT_PATH_PREFIX}"_verify_checksums && CHECKSUM_PASSED=1 || CHECKSUM_PASSED=0 + rm -f "${file}"."${SHA_EXT}".tmp + done +} + +function unzip_from_source() { + tar -zxvf phoenix-adapters-"${PHOENIX_VERSION}"-src.tar.gz + cd phoenix-adapters-"${PHOENIX_VERSION}" +} + +function rat_test() { + rm -f "${OUTPUT_PATH_PREFIX}"_rat_test + mvn clean apache-rat:check "${MVN_PROPERTIES[@]}" 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_rat_test && RAT_CHECK_PASSED=1 +} + +function build_from_source() { + rm -f "${OUTPUT_PATH_PREFIX}"_build_from_source + # Hardcode skipTests for faster build. Testing is covered later. + mvn clean install "${MVN_PROPERTIES[@]}" -DskipTests 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_build_from_source && BUILD_FROM_SOURCE_PASSED=1 +} + +function run_tests() { + rm -f "${OUTPUT_PATH_PREFIX}"_run_tests + (mvn clean package "${MVN_PROPERTIES[@]}" && mvn verify "${MVN_PROPERTIES[@]}") 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_run_tests && UNIT_TEST_PASSED=1 +} + +function execute() { + ${1} || print_when_exit +} + +function print_when_exit() { + cat << __EOF + * Signature: $( ((SIGNATURE_PASSED)) && echo "ok" || echo "failed" ) + * Checksum : $( ((CHECKSUM_PASSED)) && echo "ok" || echo "failed" ) + * Rat check (${JAVA_VERSION}): $( ((RAT_CHECK_PASSED)) && echo "ok" || echo "failed" ) + - mvn clean apache-rat:check ${MVN_PROPERTIES[@]} + * Built from source (${JAVA_VERSION}): $( ((BUILD_FROM_SOURCE_PASSED)) && echo "ok" || echo "failed" ) + - mvn clean install ${MVN_PROPERTIES[@]} -DskipTests + * Unit tests pass (${JAVA_VERSION}): $( ((UNIT_TEST_PASSED)) && echo "ok" || echo "failed" ) + - mvn clean package ${MVN_PROPERTIES[@]} && mvn verify ${MVN_PROPERTIES[@]} +__EOF + if ((CHECKSUM_PASSED)) && ((SIGNATURE_PASSED)) && ((RAT_CHECK_PASSED)) && ((BUILD_FROM_SOURCE_PASSED)) && ((UNIT_TEST_PASSED)) ; then + exit 0 + fi + exit 1 +} + +download_and_import_keys +download_release_candidate +pushd "${PHOENIX_RC_VERSION}" + +execute verify_signatures +execute verify_checksums +execute unzip_from_source +execute rat_test +execute build_from_source +execute run_tests + +popd + +print_when_exit diff --git a/pom.xml b/pom.xml index 32a7b3a..cddab3b 100644 --- a/pom.xml +++ b/pom.xml @@ -39,6 +39,50 @@ phoenix-adapters 1.0.0-SNAPSHOT + Adapters that expose NoSQL database APIs (e.g. Amazon DynamoDB) backed by Apache + Phoenix on Apache HBase. + https://phoenix.apache.org/ + 2024 + + + scm:git:https://gitbox.apache.org/repos/asf/phoenix-adapters.git + scm:git:https://gitbox.apache.org/repos/asf/phoenix-adapters.git + https://gitbox.apache.org/repos/asf?p=phoenix-adapters.git + HEAD + + + + JIRA + https://issues.apache.org/jira/browse/PHOENIX + + + + + User List + user-subscribe@phoenix.apache.org + user-unsubscribe@phoenix.apache.org + user@phoenix.apache.org + https://lists.apache.org/list.html?user@phoenix.apache.org + + + Developer List + dev-subscribe@phoenix.apache.org + dev-unsubscribe@phoenix.apache.org + dev@phoenix.apache.org + https://lists.apache.org/list.html?dev@phoenix.apache.org + + + + + + dev + The Apache Phoenix Team + dev@phoenix.apache.org + Apache Software Foundation + https://www.apache.org + + + UTF-8 1.8 @@ -116,6 +160,48 @@ + + org.apache.rat + apache-rat-plugin + 0.16.1 + + true + + + **/target/** + **/*.iml + .idea/** + .vscode/** + .git/** + .gitignore + .gitattributes + **/*.log + **/*.log.* + **/logs/** + **/dependency-reduced-pom.xml + + **/*.md + **/*.json + **/*.txt + **/*.jpeg + **/*.jpg + **/*.png + **/*.svg + + dev/create-release/vote.tmpl + + CHANGES.md + RELEASENOTES.md + + **/*.properties + docker/conf/** + conf/** + + LICENSE + NOTICE + + + @@ -140,6 +226,62 @@ + + + release + + + + org.apache.maven.plugins + maven-source-plugin + + + attach-sources + package + + jar-no-fork + + + + + + org.apache.maven.plugins + maven-javadoc-plugin + + false + none + + + + attach-javadocs + package + + jar + + + + + + org.apache.rat + apache-rat-plugin + + + rat-check + verify + + check + + + + + + +