Skip to content

Improve testing: Truncate all tables but migrations #128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rh-amarin wants to merge 2 commits into
base: main
Choose a base branch
from
+34 −6
Open

Improve testing: Truncate all tables but migrations #128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
34ee26c
truncate all tables but migrations
rh-amarin Nov 2, 2025
3d20c97
script to test clond and generator
rh-amarin Nov 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions pkg/db/db_session/testcontainer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,12 +153,21 @@ func (f *Testcontainer) ResetDB() {
ctx := context.Background()
g2 := f.New(ctx)

tables := []string{"dinosaurs", "events"}
for _, table := range tables {
if g2.Migrator().HasTable(table) {
if err := g2.Exec(fmt.Sprintf("TRUNCATE TABLE %s CASCADE", table)).Error; err != nil {
glog.Errorf("Error truncating table %s: %s", table, err)
}
// Dynamically retrieve all table names except for the "migrations" table and truncate them
var tableNames []string
err := g2.Raw(`
SELECT tablename
FROM pg_tables
WHERE schemaname = 'public'
AND tablename != 'migrations'
`).Scan(&tableNames).Error
if err != nil {
glog.Errorf("Error retrieving table names for reset: %s", err)
return
}
for _, table := range tableNames {
if err := g2.Exec(fmt.Sprintf("TRUNCATE TABLE %s CASCADE", table)).Error; err != nil {
glog.Errorf("Error truncating table %s: %s", table, err)

@markturansky markturansky Nov 3, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Truncating all tables by query (instead of a hard-coded list of table names) is a good improvement, but I wonder if it's needed at all?

The improved way of resetting between tests is to dump the database entirely and recreate it from a template. The template (template1, part of default postgres) gets all the migrations applied once, and then new DBs are restored from that template.

https://github.com/openshift-online/rh-trex/blob/main/pkg/db/db_session/test.go#L50

I think we might be able to remove table truncation entirely.

@rh-amarin rh-amarin Nov 3, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then, that mechanism is not properly working in the current tests, since it failed for me.

I will revisit this.

When using test containers, it also allows us to run tests in parallel.
Than means, one PostgreSQL instance per test
Or something in between, having a subset of tests sharing the same instance

@markturansky markturansky Nov 3, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test containers are great. Restoring all those postgres instances from template1 is a significant performance improvement, especially when they number of migrations grows over time.

@rh-amarin rh-amarin Nov 3, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is why I mentioned the "something in between"

Here is a somewhat crazy idea
Generate a testcontainer image with a PostgreSQL already migrated
After that, spinning up a new PostgreSQL is the same as creating instances from template1
And then we can use some parallelization for tests

@markturansky markturansky Nov 3, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that sounds good, too.

}
}
}
Expand Down
19 changes: 19 additions & 0 deletions templates/test-generator.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/bin/bash
tmpdir="$(mktemp -d)"
echo "creating app $tmpdir"

go run cmd/trex/main.go clone --name myapp --destination "/tmp/${tmpdir}" --repo-base github.com/openshift-online

cd "/tmp/${tmpdir}"

go run ./scripts/generator.go --kind Pet --fields "name:string,color:string"

make generate

make binary

make test

# disable RYUK for podman
export TESTCONTAINERS_RYUK_DISABLED=true
make test-integration