Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: CI

on:
push:
branches: [main]
pull_request:
workflow_dispatch:

jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v7

- name: Set up Java for the Firestore emulator
uses: actions/setup-java@v5
with:
distribution: 'temurin'
java-version: '21'

- name: Set up the Google Cloud SDK
# Ignore the "no authentication found" warning, the emulator does not connect to Datastore.
uses: google-github-actions/setup-gcloud@v3
with:
install_components: cloud-firestore-emulator

- name: Add the Firestore emulator to the PATH
run: echo "$(gcloud info --format='value(installation.sdk_root)')/platform/cloud-firestore-emulator" >> "$GITHUB_PATH"

- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
bundler-cache: true

- name: Run the gem test suite
run: bundle exec rake

- name: RuboCop
run: bundle exec rubocop

- name: Set up the example Rails app
uses: ruby/setup-ruby@v1
with:
bundler-cache: true
working-directory: test/support/datastore_example_rails_app

- name: Run the example Rails app test suite
working-directory: test/support/datastore_example_rails_app
run: bundle exec rails test
2 changes: 1 addition & 1 deletion .rubocop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Style/FetchEnvVar:

Metrics/ModuleLength:
CountComments: false
Max: 150
Max: 160

Metrics/ClassLength:
CountComments: false
Expand Down
18 changes: 5 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ gem compliant with [active_model](https://github.com/rails/rails/tree/master/act
conventions and compatible with your Rails 5+ applications.

Why would you want to use Google's NoSQL
[Firestore in Datastore mode](https://cloud.google.com/datastore) with Rails?
[Firestore in Datastore mode](https://docs.cloud.google.com/datastore/docs) with Rails?

Use it when you want a Rails app backed by a fully managed, massively scalable NoSQL database,
without provisioning database servers or manually sharding data. Datastore stores records as
Expand All @@ -15,6 +15,7 @@ automatically handles scaling and replication, provides highly available and dur
supports indexed queries and ACID transactions.

[![Gem Version](https://badge.fury.io/rb/activemodel-datastore.svg)](https://badge.fury.io/rb/activemodel-datastore)
[![CI](https://github.com/Agrimatics/activemodel-datastore/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/Agrimatics/activemodel-datastore/actions/workflows/ci.yml)

## Table of contents

Expand Down Expand Up @@ -71,8 +72,8 @@ file. Active Model Datastore also supports the following environment variables f
the JSON must be stored directly in environment variables:

```bash
SERVICE_ACCOUNT_PRIVATE_KEY = -----BEGIN PRIVATE KEY-----\nMIIFfb3...5dmFtABy\n-----END PRIVATE KEY-----\n
SERVICE_ACCOUNT_CLIENT_EMAIL = web-app@app-name.iam.gserviceaccount.com
export SERVICE_ACCOUNT_PRIVATE_KEY='-----BEGIN PRIVATE KEY-----\nMIIFfb3...5dmFtABy\n-----END PRIVATE KEY-----\n'
export SERVICE_ACCOUNT_CLIENT_EMAIL='web-app@app-name.iam.gserviceaccount.com'
```

On Heroku the environment variables can be set under **Settings > Config Vars**.
Expand All @@ -88,9 +89,6 @@ configure it during application initialization:
ActiveModel::Datastore.logger = MyApplication.logger
```

Retry messages identify the Datastore operation, entity kind, failed-attempt elapsed time, exception,
and retry delay.

There is an example Puma config file [here](https://github.com/Agrimatics/activemodel-datastore/blob/main/test/support/datastore_example_rails_app/config/puma.rb).

## <a name="model"></a>Model Example
Expand Down Expand Up @@ -406,8 +404,7 @@ There are two types of indexes, *built-in* and *composite*.

#### Built-in
By default, Datastore automatically predefines an index for each property of each entity kind.
These single property indexes are suitable for simple types of queries. These indexes are free and
do not count against your index limit.
These single property indexes are suitable for simple types of queries.

#### Composite
Composite indexes include multiple property values per indexed entity. Composite indexes support
Expand Down Expand Up @@ -451,11 +448,6 @@ cloud_firestore_emulator start --database-mode=datastore-mode --port=8180
Set `DATASTORE_EMULATOR_HOST=localhost:8180` so the Ruby client connects to the emulator. The gem sets
this automatically for Rails development and uses port 8181 for Rails tests.

By default, the emulator does not enforce composite indexes. To validate an index configuration, add
`--require-indexes --index-file=./index.yaml` when starting it. See Google's
[Firestore in Datastore mode emulator documentation](https://cloud.google.com/datastore/docs/emulator)
for more information.

## <a name="rails"></a>Example Rails App

There is an example Rails 8.1 app in the test directory [here](https://github.com/Agrimatics/activemodel-datastore/tree/main/test/support/datastore_example_rails_app).
Expand Down
28 changes: 18 additions & 10 deletions lib/active_model/datastore.rb
Original file line number Diff line number Diff line change
Expand Up @@ -414,18 +414,22 @@ def build_query(options = {})
query_options(query, options)
end

def retry_on_exception?(max_retry_count = 5, operation: 'unknown', kind: name)
def retry_on_exception?(max_retry_count = 5, operation: nil, kind: name)
retries = 0
sleep_time = 0.25
begin
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
yield
rescue Google::Cloud::Error => e
return false if retries >= max_retry_count

elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
"#{e.message.inspect}; retrying in #{sleep_time} s\e[0m"
if operation
elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
"#{e.message.inspect}; retrying in #{sleep_time} s\e[0m"
else
message = "\e[33mRescued exception #{e.message.inspect}, retrying in #{sleep_time}\e[0m"
end
ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
# 0.25, 0.5, 1, 2, and 4 second between retries.
sleep sleep_time
Expand All @@ -435,18 +439,22 @@ def retry_on_exception?(max_retry_count = 5, operation: 'unknown', kind: name)
end
end

def retry_on_exception(max_retry_count = 5, operation: 'unknown', kind: name)
def retry_on_exception(max_retry_count = 5, operation: nil, kind: name)
retries = 0
sleep_time = 0.25
begin
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
yield
rescue Google::Cloud::Error => e
raise e if retries >= max_retry_count

elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
"#{e.message.inspect}; retrying in #{sleep_time} s\e[0m"
if operation
elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
"#{e.message.inspect}; retrying in #{sleep_time} s\e[0m"
else
message = "\e[33mRescued exception #{e.message.inspect}, retrying in #{sleep_time}\e[0m"
end
ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
# 0.25, 0.5, 1, 2, and 4 second between retries.
sleep sleep_time
Expand Down
112 changes: 112 additions & 0 deletions test/cases/retry_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
require 'test_helper'
require 'minitest/mock'

class ActiveModel::DatastoreRetryTest < Minitest::Test
def teardown
ActiveModel::Datastore.logger = nil
end

def test_retry_on_exception_question_mark_returns_successful_result_without_retrying
result = MockModel.retry_on_exception? { :success }

assert_equal :success, result
end

def test_retry_on_exception_returns_successful_result_without_retrying
result = MockModel.retry_on_exception { :success }

assert_equal :success, result
end

def test_retry_on_exception_question_mark_preserves_the_legacy_message
logger = Minitest::Mock.new
message = "\e[33mRescued exception \"retry error\", retrying in 0.25\e[0m"
logger.expect(:warn, nil, [message])
ActiveModel::Datastore.logger = logger
attempts = 0

result = MockModel.retry_on_exception?(1) do
attempts += 1
raise Google::Cloud::Error, 'retry error' if attempts == 1

:retried
end

assert_equal :retried, result
logger.verify
end

def test_retry_on_exception_preserves_the_legacy_message
logger = Minitest::Mock.new
message = "\e[33mRescued exception \"retry error\", retrying in 0.25\e[0m"
logger.expect(:warn, nil, [message])
ActiveModel::Datastore.logger = logger
attempts = 0

result = MockModel.retry_on_exception(1) do
attempts += 1
raise Google::Cloud::Error, 'retry error' if attempts == 1

:retried
end

assert_equal :retried, result
logger.verify
end

def test_retry_on_exception_question_mark_logs_enhanced_context_for_an_operation
logger = Minitest::Mock.new
message = "\e[33mDatastore transaction failed for Local after 0 ms: \"retry error\"; " \
"retrying in 0.25 s\e[0m"
logger.expect(:warn, nil, [message])
ActiveModel::Datastore.logger = logger
attempts = 0

result = Process.stub(:clock_gettime, 1.0) do
MockModel.retry_on_exception?(1, operation: 'transaction', kind: 'Local') do
attempts += 1
raise Google::Cloud::Error, 'retry error' if attempts == 1

:retried
end
end

assert_equal :retried, result
logger.verify
end

def test_retry_on_exception_logs_enhanced_context_for_an_operation
logger = Minitest::Mock.new
message = "\e[33mDatastore transaction failed for Local after 0 ms: \"retry error\"; " \
"retrying in 0.25 s\e[0m"
logger.expect(:warn, nil, [message])
ActiveModel::Datastore.logger = logger
attempts = 0

result = Process.stub(:clock_gettime, 1.0) do
MockModel.retry_on_exception(1, operation: 'transaction', kind: 'Local') do
attempts += 1
raise Google::Cloud::Error, 'retry error' if attempts == 1

:retried
end
end

assert_equal :retried, result
logger.verify
end

def test_retry_on_exception_question_mark_returns_false_when_retries_are_exhausted
result = MockModel.retry_on_exception?(0) { raise Google::Cloud::Error, 'retry error' }

refute result
end

def test_retry_on_exception_raises_when_retries_are_exhausted
error = assert_raises(Google::Cloud::Error) do
MockModel.retry_on_exception(0) { raise Google::Cloud::Error, 'retry error' }
end

assert_equal 'retry error', error.message
end
end
2 changes: 1 addition & 1 deletion test/support/datastore_example_rails_app/Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ gem 'turbolinks', '~> 5'
gem 'puma', '>= 6'
gem 'rack-timeout'

gem 'activemodel-datastore', path: File.expand_path('../../../..', __FILE__)
gem 'activemodel-datastore', path: '../../..'

# Image storage
gem 'carrierwave', '~> 3.1'
Expand Down
2 changes: 1 addition & 1 deletion test/support/datastore_example_rails_app/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
PATH
remote: /Users/bryce/workspace/activemodel-datastore
remote: ../../..
specs:
activemodel-datastore (0.9.0)
activemodel (>= 5.0.0)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
#!/bin/bash
cloud_datastore_emulator start --port=8180 tmp/local_datastore
cloud_firestore_emulator start --database-mode=datastore-mode --port=8180
Loading