Otherwise we cannot handle the same name options for both column and
table (e.g. `:comment`, `:charset`, `:collation`).

5 years ago, I made dumping full table options at #17569, especially to
dump `ENGINE=InnoDB ROW_FORMAT=DYNAMIC` to use utf8mb4 with large key
prefix.

In that time, omitting the default engine `ENGINE=InnoDB` was not useful
since `ROW_FORMAT=DYNAMIC` always remains as long as using utf8mb4 with
large key prefix.

But now, MySQL 5.7.9 has finally changed the default row format to
DYNAMIC, utf8mb4 with large key prefix can be used without dumping the
default engine and the row format explicitly.

So now is a good time to make the default engine is omitted.

Before:

```ruby
create_table "accounts", options: "ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```

After:

```ruby
create_table "accounts", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```

To entirely omit `:options` option to make schema agnostic, I've added
`:charset` and `:collation` table options to exclude `CHARSET` and
`COLLATE` from `:options`.

Fixes #26209.
Closes #29472.

See also #33608, #33853, and #34742.

Co-authored-by: NAnton Topchii <player1@infinitevoid.net>

`joins_values` is already `compact_blank!`ed, so `blank?` check is
entirely redundant.

We've learned that `merge` causes duplicated multiple values easily, so
if we missed to deduplicate the values, it will cause weird behavior
like #38052, #39171.

I've investigated the deduplication for the values, at least that had
existed since Rails 3.0.

https://github.com/rails/rails/commit/bed9179aa1496f6d28891cf515af0d7e515ebbab

Aggregations with group by multiple fields was introduced at Rails 3.1,
but we missed the deduplication for the aggregation result, unlike the
generated SQL.

https://github.com/rails/rails/commit/a5cdf0b9eb860c4370ae5fde231e1b61f71b6b65

While the investigation, I've found that `annotate` is also missed the
deduplication.

I don't suppose this weird behavior is intended for both.

So I'd like to deprecate the duplicated behavior in Rails 6.1, and will
be deduplicated all multiple values in Rails 6.2.

To migrate to Rails 6.2's behavior, use `uniq!(:group)` to deduplicate
group fields.

```ruby
accounts = Account.group(:firm_id)

# duplicated group fields, deprecated.
accounts.merge(accounts.where.not(credit_limit: nil)).sum(:credit_limit)
# => {
#   [1, 1] => 50,
#   [2, 2] => 60
# }

# use `uniq!(:group)` to deduplicate group fields.
accounts.merge(accounts.where.not(credit_limit: nil)).uniq!(:group).sum(:credit_limit)
# => {
#   1 => 50,
#   2 => 60
# }
```

Related to #39328, #39358.

For now, `merge` cannot override non-equality clauses, so non-equality
clauses will easily be duplicated by `merge`.

This deduplicates redundant same clauses in `merge`.

If source/through scope references other tables in where/order, we
should explicitly maintain joins in the scope, otherwise association
queries will fail with referenced unknown column.

Fixes #33525.

`reflection.scope` is not aware of all source scopes if the association
is through association.

It should use `reflection.join_scopes` for that.

Fixes #39376.

Fixes #39088.

Bump an Active Record instance's lock version after updating its counter
cache. This avoids raising an unnecessary ActiveRecord::StaleObjectError
upon subsequent transactions by maintaining parity with the
corresponding database record's lock_version column.

The `index_exists?` method wasn't very specific so when we added the
`if_not_exists` to `add_index` and `if_exists` to `remove_index` there
were a few cases where behavior was unexpected.

For `add_index` if you added a named index and then added a second index
with the same columns, but a different name, that index would not get
added because `index_exists` was looking only at column named and not at
the exact index name. We fixed `add_index` by moving the `index_exists`
check below `add_index_options` and pass `name` directly to
`index_exists` if there is a `if_not_exists` option.

For `remove_index` if you added a named index and then tried to remove
it with a nil column and a explicit name the index would not get removed
because `index_exists` saw a nil column. We fixed this by only doing the
column check in `index_exists` if `column` is present.
Co-authored-by: NJohn Crepezzi <john.crepezzi@gmail.com>

If a has_many :through association isn't found we can suggest similar associations:

```
class Author
  has_many :categorizations, -> { }
  has_many :categories, through: :categorizations
  has_many :categorized_posts, through: :categorizations, source: :post
  has_many :category_post_comments, through: :categories, source: :post_comments

  has_many :nothings, through: :kateggorisatons, class_name: "Category"
end

Author.first.nothings

Could not find the association :kateggorisatons in model Author
Did you mean?  categorizations
               categories
               categorized_posts
               category_post_comments
```

Currently the query values normalization for multiple values is missing
some parts.

Some values are flatten, but others are not flatten.
Some values are compact, but others are not compact.
some values are compact_blank, but others are not compact_blank.

Originally all multiple values should be flatten and compact_blank
before `build_arel`, if it is not ensured which multiple values are
normalized or not, we need to normalize it again carefully and
conservatively in `build_arel`.

To unify the query values normalization for multiple values, ensure the
normalization in `check_if_method_has_arguments!` since all multiple
values should be checked by the method.

Related #39236.

`relation.merge` method sometimes replaces mergee side condition, but
sometimes maintain both conditions unless `relation.rewhere` is used.

It is very hard to predict merging result whether mergee side condition
will be replaced or not.

One existing way is to use `relation.rewhere` for merger side relation,
but it is also hard to predict a relation will be used for `merge` in
advance, except one-time relation for `merge`.

To address that issue, I propose to support merging option `:rewhere`,
to allow mergee side condition to be replaced exactly.

That option will allow non-`rewhere` relation behaves as `rewhere`d
relation.

```ruby
david_and_mary = Author.where(id: david.id..mary.id)

# both conflict conditions exists
david_and_mary.merge(Author.where(id: bob)) # => []

# mergee side condition is replaced by rewhere
david_and_mary.merge(Author.rewhere(id: bob)) # => [bob]

# mergee side condition is replaced by rewhere option
david_and_mary.merge(Author.where(id: bob), rewhere: true) # => [bob]
```

PostgreSQL doesn't support rename column operation in bulk alter table.

Fixes #39322.

Related #32163.

We should not identify an object by dirty value. If do so, accidentally
matches latest object even though it is a stale object.

Co-authored-by: NEugene Kenny <elkenny@gmail.com>

If an association isn't found we can suggest matching associations:

```
Post.all.merge!(includes: :monkeys).find(6)

Association named 'monkeys' was not found on Post; perhaps you misspelled it?
Did you mean?  funky_tags
               comments
               images
               skimmers
```

In aggregation value's type cast, `klass.attribute_types.fetch(...)`
will find no type almost all cases, since any type is already returned
by `column.type_caster` in that case.
And also, the root of join dependency tree is also the model's `klass`,
so the explicit `klass.attribute_types` is redundant originally.

Add support for finding records based on signed ids, which are tamper-proof, verified ids that can be set to expire and scoped with a purpose. This is particularly useful for things like password reset or email verification, where you want the bearer of the signed id to be able to interact with the underlying record, but usually only within a certain time period.

Otherwise `@_defer_touch_attrs` is touched again with old `@_touch_time`
in `before_committed!`.

This also removes the `if @transaction_state&.finalized?` guard which is
harder to understand optimization introduced at #36049. The guard is
faster enough though, but removing that will make attribute access about
2% ~ 4% faster, and will make code base to ease to maintain.

`sync_with_transaction_state` was introduced at #9068 to address memory
bloat when creating lots of AR objects inside a transaction.

I've found #18638 the same design of this to address memory bloat, but
this differs from #18638 in that it will allocate one `WeakMap` object
only when explicit transaction, no extra allocation for implicit
transaction.

Executable script to reproduce memory bloat:

https://gist.github.com/kamipo/36d869fff81cf878658adc26ee38ea1b
https://github.com/rails/rails/issues/15549#issuecomment-46035848

I can see no memory concern with this.
Co-authored-by: NArthur Neves <arthurnn@gmail.com>

#38354 is caused by #36304, to fix invalid joins order for through
associations.

Actually passing Arel joins to `joins` is not officially supported
unlike string joins, and also Arel joins could be easily converted to
string joins by `to_sql`. But I'd not intend to break existing apps
without deprecation cycle, so I've changed to mark only implicit joins
as leading joins, to maintain the original joins order for user supplied
Arel joins.

Fixes #38354.

`count` is too complex feature in Active Record, it is heavily mangling
select values, so it easily hit the ORDER BY with SELECT DISTINCT
limitation.

https://github.com/rails/rails/blob/35bf86fe838d310833adb283712ce4913447eea7/activerecord/lib/active_record/relation/calculations.rb#L133-L140
https://github.com/rails/rails/blob/35bf86fe838d310833adb283712ce4913447eea7/activerecord/lib/active_record/relation/calculations.rb#L253-L264

But at least in the case of group by, select values are always to be
aggregated value and group values, so meaningful order values are
originally limited. So in that case, remaining order values should
be safe as long as meaningful order values are set by people.

Fixes #38936.

- Calling `Blog.where(title: ['foo', 'bar']).where_values_hash` now
  returns an empty hash.
  This is a regression since 72fd0bae .

  `Arel::Node::HomogeousIn` isn't a `EqualityNode`, the `WhereClause`
  didn't had a case for this.

  I decide to not make `HomogeousIn` inherit from `EqualityNode`,
  because there is a comment questioning it for `In` https://github.com/rails/rails/blob/57d926a78a6968b9268f4ec1acde608132a1c920/activerecord/lib/arel/nodes.rb#L31
  Intead I just modified the `WhereClause` case and implemented
  `right` on the node which is needed by `where_value_hash` https://github.com/rails/rails/blob/57d926a78a6968b9268f4ec1acde608132a1c920/activerecord/lib/active_record/relation/where_clause.rb#L59

Follow up of #39292 and #39259.

Follow up to #39264, and fixes demonstrated case in #39290.

If the column has no type caster and the model don't know the attribute,
let's will attempt to lookup cast type from join dependency tree.

For now argument forwarding doesn't allow some keywords like `true` as a
method name.

To bypass the issue, fallback to `define_method` if method names are
Ruby reserved keywords.

https://bugs.ruby-lang.org/issues/16854

```ruby
class Works
  def true(*args)
    puts(*args)
  end
end

Works.new.true 1, 2, 3
# => 1, 2, 3

class WontWork
  def true(...)
    puts(...)
  end
end
```

```
% ruby a.rb
a.rb:12: syntax error, unexpected ..., expecting ')'
  def true(...)
a.rb:13: unexpected ...
a.rb:15: syntax error, unexpected `end', expecting end-of-input
```

MySQL 8.0.3 (and MariaDB 10.5.2) now supports native RENAME COLUMN
syntax.

https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-3.html#mysqld-8-0-3-sql-syntax
https://mariadb.com/kb/en/alter-table/#rename-column

Looks like the CI failure is caused by rename index on a foreign key
constraint.

https://buildkite.com/rails/rails/builds/69099#1b008fd1-1d2f-4c87-934a-8cdfd6e42c67/1022-2096

Follow up to #39255, #39039.

One of the purpose of this was to unify the behavior between the
databases.

Original code was:

```ruby
    type   = result.column_types.fetch(column_alias) do
      type_for(column_name)
    end
```

The code will attempt looking up type from `column_types`, then fallback
to attribute types, so I supposed the code was originally intended to
cast a value by the database types.

But now, most modern clients will already return casted values, and no
longer use `column_types`, except Postgres.
As a result, now most adapter accidentally fallback to attribute types.
Since casted by attribute types sometimes doesn't return numeric values,
I've unified the behavior to use database types consistently in #39039.

But later, I've learned that attribute types have important settings
like time zone aware attributes (#39255), and some existing code relying
on attribute types over database types (#39271).

I've changed all aggregated values are casted by attribute types.

Fixes #39271.

Delegate to scope values (e.g. `offset_value`, `order_values`) is slow.

We can avoid that if records are already loaded.

```ruby
class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post
  has_many :likes
end

class Like < ActiveRecord::Base
  belongs_to :post
end

10.times { Post.create! }
30.times { |i| Comment.create!(post_id: 1 + i % 10) }
150.times { |i |Like.create!(comment_id: 1 + i % 30) }

post = Post.includes(comments: :likes).last

result = Benchmark.measure do
  100.times { post.comments.each { |c| c.likes.first.id } }
end

puts
puts result
```

Before:

```
  0.013542   0.000582   0.014124 (  0.014125)
```

After:

```
  0.000938   0.000017   0.000955 (  0.000952)
```

Fixes #38252.