6.31.0 sync#2634
Draft
Stolb27 wants to merge 221 commits into
Draft
Conversation
…e second for tests on password.
this statement it is not possible to check that the username doesn't contains the password so this rule is skipped.
Fix error message for unauthorized letters in password and username.
by the password hook so it is useless.
…d is not a system user.
Create LICENSE
introducing pg_init, pg_fini approach
… information to documentation. - Fix regression test with one user that was not removed.
also changing the comment style
All users passwords are historicized in table (credcheck.pg_auth_history)
together with the timestamps of when these passwords were set.
Two settings allow to control the behavior of this feature:
- credcheck.password_reuse_history: number of distinct passwords set
before a password can be reused.
- credcheck.password_reuse_interval: amount of time it takes before a
password can be reused again.
The default value for these settings are 0 which means that all password
reuse policies are disabled.
The password history consists of passwords a user has been assigned in
the past. credcheck can restrict new passwords from being chosen from
this history:
- If an account is restricted on the basis of number of password changes,
a new password cannot be chosen from the password_reuse_history most
recent passwords. For example, minimum number of password changes is
set to 3, a new password cannot be the same as any of the most recent
3 passwords.
- If an account is restricted based on time elapsed, a new password can't
be chosen from those in the history that are newer than the number of
day set to password_reuse_interval. For example, if the password reuse
interval is set to 365, new password must not be among those previously
chosen within the last year.
Thanks to Umair Shahid and Gabi201265 for the feature request.
In 37ab4e12ccb793b0f2e0acc2d04589c3613a02da we updated a lot of modules,
replacing subprocess with gpsubprocess. But the gpsubprocess module in case of
python2 relies on subprocess32 package for managing subprocesses, which is
pretty-much outdated itself (in our project).
So when we try to combine this subprocesses32 module and coverage utility via
hooking it with site-module, we get following error:
File "test/behave/mgmt_utils/steps/gpstart.py", line 89, in impl
preexec_fn=_handle_sigpipe,
RuntimeError: not holding the import lock
To solve this we should revert changes on gpstart.py from mentioned previously
patch, as updating gpsubprocess will also solve the problem, but on the other
hand will bring even more errors.
SCRAM hashing for passwords uses salted hashes that are locally generated on segments, so hashes are not consistent between segments. So inconsistency in rolpassword field in pg_authid catalog table is not an issue and should not be checked by gpcheckcat. Also fix cleanup in the previous behave test. Changes from original commit: GUC password_hash_algorithm is used instead of password_encryption on 6X. (cherry picked from commit fe08e03) Ticket: GG-438
Switch regression testing to ubuntu24 This patch changes plpythonu to plpython3u almost everywhere, except for places where import of greengage internal libraries happens. There we need to address libraries with version of plpython this libraries were build to. Also some tests were update were taken as subprocesses module acts different in python 3 due to difference in strings. With this sql_isolation_testcase.py were updated: now, when creating psql-like output, for length of column we count not a number of characters but a number of bytes needed for representation of result Mapred test were redacted also, as newer version of perl on ubuntu24 gives different output. Co-authored-by: Maksim Michkov <m.michkov@arenadata.io>
debian/lintian-overrides: - replace broad TODO overrides with precise, documented ones per tag and path - add overrides for script-not-executable, script-uses-unversioned-python-in-shebang, unusual-interpreter, incorrect-path-for-interpreter, package-has-unnecessary-activation-of-ldconfig-trigger, package-contains-hardlink, custom-library-search-path debian/rules: - add override_dh_dwz (debugedit DWARF incompatibility) - add override_dh_fixperms: strip executable bit from *.py, *.pm, *.sh without shebang and from *.md files debian/control: - add Priority: optional debian/copyright: - add explicit copyright notice gpAux/Makefile: - replace STABILITY=unstable with DISTRO_CODENAME via lsb_release -sc in changelog generation gpMgmt/doc/gpmapreduce_help: - replace cp1252 curly quotes (0x93, 0x94) with ASCII double quotes Task: GG-339
Reduced default timeout for resgroup tests. Currently, resgroup tests use the default timeout of 360 minutes. If the tests hang and reach this limit, the GitHub workflow hits its maximum timeout and cancels all jobs immediately, preventing logs from being collected (even with `always()` conditions). This change sets the resgroup job timeout to 3 hours, leaving an additional 30 minutes for log collection and uploading results. Task: GG-180
- ci: sync with upstream - ci (build): target greengage-reusable-build to v30 tag CI updates in v30 refactor (build): cache SHA image only when GHCR push did not succeed - add `id: push` to the push step to expose its outcome to subsequent steps - move Save/Cache steps after the push step instead of before - gate Save and Cache on `steps.push.outcome != 'success'` to cover skipped, failed, and cancelled outcomes - ensure the image exists in exactly one place: GHCR on successful push, cache otherwise Task: GG-162
optimizer_enable_table_alias GUC was implemented in 6cc65ad to support printing table aliases in ORCA. However, it was disabled in test cluster to prevent many tests from breaking, so they were run without aliases. Make that GUC defunct. Make the test cluster run with printing aliases. Fix affected tests. Ticket: GG-353
In Python 2, if an Exception contains unicode characters, an attempt to convert it to unicode will result in an error. This patch restores the old behavior. If we log an Exception, we first cast it to the str. After that, if it is a byte string, then decode it to the unicode. If the Exception contains unicode, then catch it and log a fixed message.
Postgres query optimizer set negative costs for plans with non-distributed (ON MASTER) external web tables. Optimizer didn't use proper values from pg_class and instead used default -1 value for tuple count. Make optimizer use default values from pg_class. Fix affected tests. Ticket: GG-352
- add CODEOWNERS for GitHub Actions review automation Task: CI-5648
Introduce Python code coverage analysis for the GreengageDB gpMgmt component. The changes enable automated collection, aggregation, and reporting of test coverage data during behave BDD test execution as well as gpMgmt unit tests which are currently enabled in the CI, helping developers identify untested code paths. Changes include adding coverage to the Dockerfile image, injecting coverage startup code into the sitecustomize.py file, running data processing and report generation. Running coverage analysis is handled by the COVERAGE_PROCESS_START variable, which is set now for CI runs. Reports are saved in the /tmp/coverage-data directory which is expected to be a mounted volume when running docker container. Co-authored-by: Denis Kovalev <d.kovalev@arenadata.io>
Problem description: If some table was inherited from a leaf of a partitioned table, the leaf table wasn't properly redistributed by the gpexpand. Root cause: gpexpand applies 'ALTER TABLE ... EXPAND PARTITION PREPARE' for all partitioned tables at its 1st phase, and applies 'ALTER TABLE ... SET WITH (REORGANIZE=true) DISTRIBUTED ...' for all leaf tables at its 2nd stage. 'ALTER TABLE ... EXPAND PARTITION PREPARE' for all non-leaf tables only changes numsegments of the current policy to expanded cluster size. The check for non-leaf condition is done via 'relhassubclass' flag of 'pg_class' record (refer to 'ATExecExpandPartitionTablePrepare()' function). Partitioned tables in 6x are implemented with the use of table inheritance. When some table was inherited from a leaf table, the 'relhassubclass' flag for the leaf table was set to 'true'. Therefore, it was treated as non-leaf table. If it was hash-distributed, the distribution policy didn't change, only the segment count was updated (while, for all hash-distributed leaves the distribution is expected to be changed to 'randomly' distributed). And, as the 'relhassubclass' flag was true, gpexpand skipped 'ALTER TABLE ... SET WITH (REORGANIZE=true) DISTRIBUTED ...' operation for the leaf table. This operation is supposed to perform proper redistribution of the table's data according to the new segment configuration. As result, after the gpexpand finished its work, the leaf table had updated distribution policy, but the data remained to reside on the old segments. As a consequence, if we tried to insert data from the partitioned table to some other table with the same distribution policy, the planner created a plan without motions, and the data was inserted into the wrong segments, spoiling the other table as well. Fix: Mixing table partitioning and table inheritance is error prone, especially in Greengage 6x, where partitioning uses much of the inheritance logic. In Greengage 7x and in vanilla Postgres the mixing is already forbidden. Therefore, we follow the same way in 6x - this patch forbids table inheritance from a partitioned table.
Add matrix to upload workflow. Matrix needed to upload latest docker image for ubuntu 24.04. Task: CI-5691
Add target_os_version to upload latest docker image for ubuntu 24.04. Task: CI-5691
…les (#438) Inheriting a table from a partitioned table can cause problems - for example, data can be not redistributed in a partitioned table after gpexpand. This patch adds checks into gpcheckcat that allow to detect if such cases of inheritance have already happened. Test are not added as the capability to inherit a table from a partitioned table is forbidden in a separate patch.
- ci (build): bump version to v33 New in CI v33: Fix: drop buildx, redirect apt to Azure mirror when available - Remove unused `docker/setup-buildx-action` step - Resolve `azure.archive.ubuntu.com` IP at build time and inject via `--add-host` for `archive.ubuntu.com` and `security.ubuntu.com` - Fall back to default mirrors if Azure mirror is unreachable - Update README to reflect removed Buildx step and new mirror behaviour Task: CI-5617
ALTER RESOURCE GROUP did not support execution inside transaction blocks because resource group changes are hard to roll back safely: they affect both catalog state and in-memory/shared-memory state, and uncommitted changes must block other resource group modifications. Introduce an opt-in postmaster GUC that lifts the restriction. With it on, several ALTERs run in one transaction and roll back together if any of them fails or if the user chooses to abort. The catalog write happens during the ALTER itself, while the in-memory cgroup and shared memory apply still run at top-level commit. Check callbacks at PRE_COMMIT against the final pg_resgroupcapability state, so ALTERs rolled back by subtransactions or overwritten by later ALTERs are not applied. CREATE and DROP keep the transaction block restriction. Reject moving a query to another resource group while resource group capabilities are being edited by an uncommitted transaction. Co-authored-by: bandetto <v.sarmin@arenadata.io> Co-authored-by: Denis Garsh <d.garsh@arenadata.io>
Package `greengage6` is a rename of `greengage`. Allow `greengage6` to replace `greengage`. Add to `debian/control`: - `Conflicts: greengage` apt removes old package including `/opt/greengagedb/greengage` - `Replaces: greengage` dpkg is allowed to overwrite shared files registered under old package Task: CI-5690
- Add greengage repository for the sigar package installation. - Sigar was added as dependency to greengage package. - Add enable-gpperfmon configflag. - Remove @Skip flag form gpperfmon.feature test. Workflow changes: - greengage-reusable-package.yml workflow used to test greengage package installation. so we need to add greengage repo to the test container for sigar installation. Task: CI-5243
- ci(package): add ubuntu 24.04 to package matrix - ci(package): parametrize 'test_docker' option - upd(deb): add 'python-is-python3' to DEPS for ubuntu 24.04 - ci(package): bump hotfix 'v34' to 'v37' - Use single quotes in apt sources echo to prevent shell expansion - Remove commented-out 'upload-to-release' job Task: CI-5751
```
adam=# CREATE TABLE ao_reloptions_t1 (c1 INT) WITH (appendonly=true, compresstype=zstd);
CREATE TABLE
adam=# SET allow_system_table_mods = on;
SET
adam=# UPDATE pg_class SET reloptions = '{compresstype=zzzz}' WHERE relname = 'ao_reloptions_t1';
UPDATE 1
adam=# DROP TABLE ao_reloptions_t1;
ERROR: unknown compresstype "zzzz"
adam=# CREATE TABLE ao_reloptions_t2 (c1 INT) WITH (appendonly=true, compresstype=zstd);
CREATE TABLE
adam=# UPDATE pg_class SET reloptions = '{appendonly=true,compresstype=quicklz,compresslevel=3}' WHERE relname = 'ao_reloptions_t2';
UPDATE 1
adam=# DROP TABLE ao_reloptions_t2;
ERROR: QuickLZ library is not supported by this build
HINT: Compile with --with-quicklz to use QuickLZ compression.
```
`validate_and_adjust_options()` doesn't respect the bool parameter
`validate` in some cases. This commit fixes it.
(cherry picked from commit d8f394be232e44793e0dc44adbd7e0ff49b5c910)
Changes compared to the original commit:
- add check for compression type before loading shared library
This patch is to add possibility of sending some arbitrary binary data (metadata) from backends to the frontend. In out particular case it could be segment and coordinator. The data is sent using direct libpq messages with M as a message type. Format of the message is pretty simple - just message length and message body. To send messages there is a new API function pq_metadatasend which can be called from an extension. The messages are processed on the frontend (coordinator in our case) side, and collected into a linked list ggMetadataList in the cdbconn.c. The code for message processing is inspired by the notice processing mechanism which works in a similar way. To lower the number of allocations, flexible array is used. To test the new API there is a test extension and a code for retrieving messages implemented as well. This code uses methods PQMetadataWalk, PQgetNextMetadata, PQgetMetadata and PQCleanMetadata. These methods can be called from an extension as well. New metadata API is available only at the backend (no FRONTEND macro defined). ADBDEV-9854
- update CI README
- replace file header comment with canonical workflow path
- add version, target_os, target_os_version, package_dir to matrix
- add ubuntu 24.04 alongside existing ubuntu 22.04 matrix entry
- replace artifact_name parameter with
version, target_os, target_os_version, package_dir
- bump 'upload-pkgs-to-release' action v27 to v38
- replace artifact_name input with
version, target_os, target_os_version, package_dir
- construct cache key internally
- rename packages with OS revision before upload
Ticket: CI-5752
- bump 'package' job v37 to v38
- append target_os + target_os_version to artifact names and cache keys
Ticket: CI-5768
To replace the upstream one This reverts commit 952567f.
Now Greengage use own repo for this purpose
8f9aa03 migrated isolation2 tests to plpython3 and added Greeengage PYTHONPATH to plpython3.python_path GUC. As a result, we got an issue on python2 builds, because of polluting plpython 3 environment with python 2 modules. To fix it, provide needed modules for python3 and restore PYTHONPATH before running cluster management tools from plpython3 functions.
8f9aa03 switches all regression and isolation2 tests to use plpython3. But some tests require additional python modultes. Moreover, part of them require PyGreSQL that depends on Greengage libPQ. And currently, this module is built only for default python interpreter as a part og Greengage build process. 8f9aa03 tries to pass Greengage PYTHONPATH to plpython3. As a result, tests work correctly only on python 3 builds (e.g. Ubuntu 24). To run isolation2 tests in the python 2 env we should provide tests dependencies for python 3 too. The biggest problem is PyGreSQL that depends on Greengage artifacts and customized to be wrapped in the directory of the same name. To solve this problem I suggest: 1. exclude moving PyGreSQL modules to the dedicated directory 2. provide additional python module that implements such virtual module 3. build PyGreSQL wheels on build Dockerfile stage optionally 4. install them on test stage This approach will be usefull to migrate PYTHONPATH to the virtual env later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Here are some reminders before you submit the pull request
make installcheck