diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..5b312c9 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,50 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + workflow_dispatch: + +jobs: + test: + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v7 + + - name: Set up Java for the Firestore emulator + uses: actions/setup-java@v5 + with: + distribution: 'temurin' + java-version: '21' + + - name: Set up the Google Cloud SDK + # Ignore the "no authentication found" warning, the emulator does not connect to Datastore. + uses: google-github-actions/setup-gcloud@v3 + with: + install_components: cloud-firestore-emulator + + - name: Add the Firestore emulator to the PATH + run: echo "$(gcloud info --format='value(installation.sdk_root)')/platform/cloud-firestore-emulator" >> "$GITHUB_PATH" + + - name: Set up Ruby + uses: ruby/setup-ruby@v1 + with: + bundler-cache: true + + - name: Run the gem test suite + run: bundle exec rake + + - name: RuboCop + run: bundle exec rubocop + + - name: Set up the example Rails app + uses: ruby/setup-ruby@v1 + with: + bundler-cache: true + working-directory: test/support/datastore_example_rails_app + + - name: Run the example Rails app test suite + working-directory: test/support/datastore_example_rails_app + run: bundle exec rails test diff --git a/.rubocop.yml b/.rubocop.yml index 2894676..a6bcfdf 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -88,7 +88,7 @@ Style/FetchEnvVar: Metrics/ModuleLength: CountComments: false - Max: 150 + Max: 160 Metrics/ClassLength: CountComments: false diff --git a/README.md b/README.md index d82dee1..18fa25a 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ gem compliant with [active_model](https://github.com/rails/rails/tree/master/act conventions and compatible with your Rails 5+ applications. Why would you want to use Google's NoSQL -[Firestore in Datastore mode](https://cloud.google.com/datastore) with Rails? +[Firestore in Datastore mode](https://docs.cloud.google.com/datastore/docs) with Rails? Use it when you want a Rails app backed by a fully managed, massively scalable NoSQL database, without provisioning database servers or manually sharding data. Datastore stores records as @@ -15,6 +15,7 @@ automatically handles scaling and replication, provides highly available and dur supports indexed queries and ACID transactions. [![Gem Version](https://badge.fury.io/rb/activemodel-datastore.svg)](https://badge.fury.io/rb/activemodel-datastore) +[![CI](https://github.com/Agrimatics/activemodel-datastore/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/Agrimatics/activemodel-datastore/actions/workflows/ci.yml) ## Table of contents @@ -71,8 +72,8 @@ file. Active Model Datastore also supports the following environment variables f the JSON must be stored directly in environment variables: ```bash -SERVICE_ACCOUNT_PRIVATE_KEY = -----BEGIN PRIVATE KEY-----\nMIIFfb3...5dmFtABy\n-----END PRIVATE KEY-----\n -SERVICE_ACCOUNT_CLIENT_EMAIL = web-app@app-name.iam.gserviceaccount.com +export SERVICE_ACCOUNT_PRIVATE_KEY='-----BEGIN PRIVATE KEY-----\nMIIFfb3...5dmFtABy\n-----END PRIVATE KEY-----\n' +export SERVICE_ACCOUNT_CLIENT_EMAIL='web-app@app-name.iam.gserviceaccount.com' ``` On Heroku the environment variables can be set under **Settings > Config Vars**. @@ -88,9 +89,6 @@ configure it during application initialization: ActiveModel::Datastore.logger = MyApplication.logger ``` -Retry messages identify the Datastore operation, entity kind, failed-attempt elapsed time, exception, -and retry delay. - There is an example Puma config file [here](https://github.com/Agrimatics/activemodel-datastore/blob/main/test/support/datastore_example_rails_app/config/puma.rb). ## Model Example @@ -406,8 +404,7 @@ There are two types of indexes, *built-in* and *composite*. #### Built-in By default, Datastore automatically predefines an index for each property of each entity kind. -These single property indexes are suitable for simple types of queries. These indexes are free and -do not count against your index limit. +These single property indexes are suitable for simple types of queries. #### Composite Composite indexes include multiple property values per indexed entity. Composite indexes support @@ -451,11 +448,6 @@ cloud_firestore_emulator start --database-mode=datastore-mode --port=8180 Set `DATASTORE_EMULATOR_HOST=localhost:8180` so the Ruby client connects to the emulator. The gem sets this automatically for Rails development and uses port 8181 for Rails tests. -By default, the emulator does not enforce composite indexes. To validate an index configuration, add -`--require-indexes --index-file=./index.yaml` when starting it. See Google's -[Firestore in Datastore mode emulator documentation](https://cloud.google.com/datastore/docs/emulator) -for more information. - ## Example Rails App There is an example Rails 8.1 app in the test directory [here](https://github.com/Agrimatics/activemodel-datastore/tree/main/test/support/datastore_example_rails_app). diff --git a/lib/active_model/datastore.rb b/lib/active_model/datastore.rb index 1afcf29..852f406 100644 --- a/lib/active_model/datastore.rb +++ b/lib/active_model/datastore.rb @@ -414,18 +414,22 @@ def build_query(options = {}) query_options(query, options) end - def retry_on_exception?(max_retry_count = 5, operation: 'unknown', kind: name) + def retry_on_exception?(max_retry_count = 5, operation: nil, kind: name) retries = 0 sleep_time = 0.25 begin - started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) + started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation yield rescue Google::Cloud::Error => e return false if retries >= max_retry_count - elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round - message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \ - "#{e.message.inspect}; retrying in #{sleep_time} s\e[0m" + if operation + elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round + message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \ + "#{e.message.inspect}; retrying in #{sleep_time} s\e[0m" + else + message = "\e[33mRescued exception #{e.message.inspect}, retrying in #{sleep_time}\e[0m" + end ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message) # 0.25, 0.5, 1, 2, and 4 second between retries. sleep sleep_time @@ -435,18 +439,22 @@ def retry_on_exception?(max_retry_count = 5, operation: 'unknown', kind: name) end end - def retry_on_exception(max_retry_count = 5, operation: 'unknown', kind: name) + def retry_on_exception(max_retry_count = 5, operation: nil, kind: name) retries = 0 sleep_time = 0.25 begin - started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) + started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation yield rescue Google::Cloud::Error => e raise e if retries >= max_retry_count - elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round - message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \ - "#{e.message.inspect}; retrying in #{sleep_time} s\e[0m" + if operation + elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round + message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \ + "#{e.message.inspect}; retrying in #{sleep_time} s\e[0m" + else + message = "\e[33mRescued exception #{e.message.inspect}, retrying in #{sleep_time}\e[0m" + end ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message) # 0.25, 0.5, 1, 2, and 4 second between retries. sleep sleep_time diff --git a/test/cases/retry_test.rb b/test/cases/retry_test.rb new file mode 100644 index 0000000..14ad643 --- /dev/null +++ b/test/cases/retry_test.rb @@ -0,0 +1,112 @@ +require 'test_helper' +require 'minitest/mock' + +class ActiveModel::DatastoreRetryTest < Minitest::Test + def teardown + ActiveModel::Datastore.logger = nil + end + + def test_retry_on_exception_question_mark_returns_successful_result_without_retrying + result = MockModel.retry_on_exception? { :success } + + assert_equal :success, result + end + + def test_retry_on_exception_returns_successful_result_without_retrying + result = MockModel.retry_on_exception { :success } + + assert_equal :success, result + end + + def test_retry_on_exception_question_mark_preserves_the_legacy_message + logger = Minitest::Mock.new + message = "\e[33mRescued exception \"retry error\", retrying in 0.25\e[0m" + logger.expect(:warn, nil, [message]) + ActiveModel::Datastore.logger = logger + attempts = 0 + + result = MockModel.retry_on_exception?(1) do + attempts += 1 + raise Google::Cloud::Error, 'retry error' if attempts == 1 + + :retried + end + + assert_equal :retried, result + logger.verify + end + + def test_retry_on_exception_preserves_the_legacy_message + logger = Minitest::Mock.new + message = "\e[33mRescued exception \"retry error\", retrying in 0.25\e[0m" + logger.expect(:warn, nil, [message]) + ActiveModel::Datastore.logger = logger + attempts = 0 + + result = MockModel.retry_on_exception(1) do + attempts += 1 + raise Google::Cloud::Error, 'retry error' if attempts == 1 + + :retried + end + + assert_equal :retried, result + logger.verify + end + + def test_retry_on_exception_question_mark_logs_enhanced_context_for_an_operation + logger = Minitest::Mock.new + message = "\e[33mDatastore transaction failed for Local after 0 ms: \"retry error\"; " \ + "retrying in 0.25 s\e[0m" + logger.expect(:warn, nil, [message]) + ActiveModel::Datastore.logger = logger + attempts = 0 + + result = Process.stub(:clock_gettime, 1.0) do + MockModel.retry_on_exception?(1, operation: 'transaction', kind: 'Local') do + attempts += 1 + raise Google::Cloud::Error, 'retry error' if attempts == 1 + + :retried + end + end + + assert_equal :retried, result + logger.verify + end + + def test_retry_on_exception_logs_enhanced_context_for_an_operation + logger = Minitest::Mock.new + message = "\e[33mDatastore transaction failed for Local after 0 ms: \"retry error\"; " \ + "retrying in 0.25 s\e[0m" + logger.expect(:warn, nil, [message]) + ActiveModel::Datastore.logger = logger + attempts = 0 + + result = Process.stub(:clock_gettime, 1.0) do + MockModel.retry_on_exception(1, operation: 'transaction', kind: 'Local') do + attempts += 1 + raise Google::Cloud::Error, 'retry error' if attempts == 1 + + :retried + end + end + + assert_equal :retried, result + logger.verify + end + + def test_retry_on_exception_question_mark_returns_false_when_retries_are_exhausted + result = MockModel.retry_on_exception?(0) { raise Google::Cloud::Error, 'retry error' } + + refute result + end + + def test_retry_on_exception_raises_when_retries_are_exhausted + error = assert_raises(Google::Cloud::Error) do + MockModel.retry_on_exception(0) { raise Google::Cloud::Error, 'retry error' } + end + + assert_equal 'retry error', error.message + end +end diff --git a/test/support/datastore_example_rails_app/Gemfile b/test/support/datastore_example_rails_app/Gemfile index 22e91a0..ae89d9f 100644 --- a/test/support/datastore_example_rails_app/Gemfile +++ b/test/support/datastore_example_rails_app/Gemfile @@ -19,7 +19,7 @@ gem 'turbolinks', '~> 5' gem 'puma', '>= 6' gem 'rack-timeout' -gem 'activemodel-datastore', path: File.expand_path('../../../..', __FILE__) +gem 'activemodel-datastore', path: '../../..' # Image storage gem 'carrierwave', '~> 3.1' diff --git a/test/support/datastore_example_rails_app/Gemfile.lock b/test/support/datastore_example_rails_app/Gemfile.lock index e0b9190..894dfe5 100644 --- a/test/support/datastore_example_rails_app/Gemfile.lock +++ b/test/support/datastore_example_rails_app/Gemfile.lock @@ -1,5 +1,5 @@ PATH - remote: /Users/bryce/workspace/activemodel-datastore + remote: ../../.. specs: activemodel-datastore (0.9.0) activemodel (>= 5.0.0) diff --git a/test/support/datastore_example_rails_app/start-local-datastore.sh b/test/support/datastore_example_rails_app/start-local-datastore.sh index 0671e32..4fb99e8 100755 --- a/test/support/datastore_example_rails_app/start-local-datastore.sh +++ b/test/support/datastore_example_rails_app/start-local-datastore.sh @@ -1,2 +1,2 @@ #!/bin/bash -cloud_datastore_emulator start --port=8180 tmp/local_datastore \ No newline at end of file +cloud_firestore_emulator start --database-mode=datastore-mode --port=8180