Co-Authored-By: NBob Whitelock <bob.whitelock1@gmail.com>
Co-Authored-By: NRafael França <rafael@franca.dev>
Co-Authored-By: NKapilSachdev <kapilsachdev03@gmail.com>

The handling for single database applications has always set a schema.rb
or structure.sql files for loading the database schema. When we first
implemented multiple database support we intended to keep this for the
original, default database. Afterall Rails _has_ to connect to something
on boot. In development only one connection is connected on boot since
we don't eager load the app.

Originally we had thought that all applications should be required to
add a `primary` entry in the database configurations file. However,
this hasn't worked in practice and we have some code now that does not
assume there's a primary. The schema dumping/loading code however,
still assumed there was a "primary" in the configurations file.

We want the "default" database in any application to use the original
files even when converted to a multiple database application as this
reduces the need to make changes when implementing this functionality on
an existing application.

The changes here update Rails to ensure that we treat either "primary"
or the first database configuration for an environment as "default".
If there is a "primary" that will be used as the default configuration.
If there is no primary the configuration that is first for an
environment will be used as the default. For schema dump/load this means
that the default configuration (primary or first) will use `schema.rb`
as the filename and other configurations will use
`[CONFIGURATION_NAME]_schema.rb`.

This should also help us finish the pull request to infer migrations
paths since now we can say the first configuration is the default. This
is a natural assumption for application developers.

Followup to #39536

    - adds the option `metadata_table` to a database connection

This reverts commit 9a5bc246, reversing
changes made to 227ff44e.

I realized afterwards that this was written this way for a reason and it
wasn't accidental. The databases loaded in `for_each` are not complete
database configs because these are generated from the database yaml
before the Rails env is loaded. So we actually do need to refetch the
correct database configuration objects after we've loaded the Rails env.

While working on something else I noticed this code was duplicated. It
was returning `name` from `for_each` and then we'd have to lookup the
`db_config` again from the `name` and `env`. We can instead just return
`db_config` from `for_each` and clean up these tasks a little bit.

Before:

```
DEPRECATION WARNING: spec_name is deprecated and will be removed from Rails 6.2 (spec_name accessors are deprecated and will be removed in Rails 6.2, please use name instead.)
```

After:

```
DEPRECATION WARNING: spec_name is deprecated and will be removed from Rails 6.2 (please use name instead)
```

Follow up of #38536.

I have so. many. regrets. about using `spec_name` for database
configurations and now I'm finally putting this mistake to an end.

Back when I started multi-db work I assumed that eventually
`connection_specification_name` (sometimes called `spec_name`) and
`spec_name` for configurations would one day be the same thing. After
2 years I no longer believe they will ever be the same thing.

This PR deprecates `spec_name` on database configurations in favor of
`name`. It's the same behavior, just a better name, or at least a
less confusing name.

`connection_specification_name` refers to the parent class name (ie
ActiveRecord::Base, AnimalsBase, etc) that holds the connection for it's
models. In some places like ConnectionHandler it shortens this to
`spec_name`, hence the major confusion.

Recently I've been working with some new folks on database stuff and
connection management and realize how confusing it was to explain that
`db_config.spec_name` was not `spec_name` and
`connection_specification_name`. Worse than that one is a symbole while
the other is a class name. This was made even more complicated by the
fact that `ActiveRecord::Base` used `primary` as the
`connection_specification_name` until #38190.

After spending 2 years with connection management I don't believe that
we can ever use the symbols from the database configs as a way to
connect the database without the class name being _somewhere_ because
a db_config does not know who it's owner class is until it's been
connected and a model has no idea what db_config belongs to it until
it's connected. The model is the only way to tie a primary/writer config
to a replica/reader config. This could change in the future but I don't
see value in adding a class name to the db_configs before connection or
telling a model what config belongs to it before connection. That would
probably break a lot of application assumptions. If we do ever end up in
that world, we can use name, because tbh `spec_name` and
`connection_specification_name` were always confusing to me.

This updates the database tasks for dumping the Active Record schema cache as
well as clearing the schema cache file, allowing the path to be defined in the
database configuration YAML file.

As before, the value can also be defined in an ENV variable, though this would
not work for a multi-db application. If the value is specified neither in the
DB config, nor in the ENV, then the path will continue to be derived from the
DB config spec_name.

Note that in order to make this change cleaner I also moved a bit of logic
out of a rake task and into the DatabaseTasks class, for symmetry.

We have two rake tasks for the schema cache:

    $ rake db:schema:cache:dump
    $ rake db:schema:cache:clear

The cache:dump task was implemented in DatabaseTasks, but the
cache:clear one was not.

I also added some tests for the behavior that I was changing, since some of
the code paths weren't tested.

Fixes https://github.com/rails/rails/issues/28827.

The steps to reproduce are as follows:

git clone git@github.com:bbuchalter/rails-issue-28827.git
cd rails-issue-28827
bundle install
bin/rails db:create

Observe that we create two databases when invoking db:create: development and test. Now observe what happens when we invoke our drop command while using DATABASE_URL.

DATABASE_URL=sqlite3://$(pwd)/db/database_url.sqlite3 bin/rails db:create

As expected, the development environment now uses the DATABASE_URL. What is unexpected is that the test environment does not.

It's unclear what the expected behavior should be in this case, but the cause of it is this: https://github.com/rails/rails/blob/9f2c74eda07ea5a9e4e624d5575a717714088dbf/activerecord/lib/active_record/tasks/database_tasks.rb#L494

Because of each_local_configuration, there seems to be no way invoke these database rake on only the development environment to ensure DATABASE_URL is respected.

The smallest scope of change I can think to make would be to conditionalize this behavior so it does not get applied when DATABASE_URL is present.

The `connection_config` method returns a `Hash`, but since we're moving
toward a place where we're using `DatabaseConfiguration::DatabaseConfig`
objects everywhere, we're introducing a new method here to replace it
called `connection_db_config`.
Co-authored-by: Neileencodes <eileencodes@gmail.com>

This is the last property that's used directly inside of Rails (in
`DatabaseTasks`) that doesn't have an accessor in `HashConfig`.

Adding this so that we can have a uniform interface for accessing
properties that Rails needs (that descend from
`DatabaseConfigurations::DatabaseConfig`)
Co-authored-by: Neileencodes <eileencodes@gmail.com>

We have these two objects, `ConnectionAdapters::Resolver` and
`DatabaseConfiguratons` that implement a lot of the same logic. One of
them is used for configurations defined in `config/database.yml` and the
other is used when passing raw configurations `String` or `Hash` objects
into methods like `establish_connection`.

Over time these two have diverged a bit. In the interest of less code
complexity, and more consistency for users this commit brings them back
together.

* Remove `Resolver` altogether and replace its primary method with
  `DatabaseConfigurations#resolve`.

* Move `resolve_pool_config` over to the `ConnectionPool` alongside the code
  that uses it.

We want to call establish connection, not `connected_to` with a database
key for truncate. Then we simplified the method by pushing the removed
tables into truncate.

This also removes an instance of unnecessarily calling
`configuration_hash` when we already have a `db_config` to pass around.
Co-authored-by: Neileencodes <eileencodes@gmail.com>

We are moving to using `DatabaseConfig` objects everywhere inside of
Rails instead of passing around Hash objects.

This PR moves to using `DatabaseConfig` objects inside of the individual
database tasks adapters. In the interest of not breaking existing
adapters, we've introduced an upgrade path by which adapters can get a
hold of database configuration objects instead of Hashes by implementing
a method `self.using_database_configurations?`
Co-authored-by: Neileencodes <eileencodes@gmail.com>

Convert all uses of `db_config.configuration_hash[*]` to use methods
defined on an implementation of `DatabaseConfigurations::DatabaseConfig`.

Since we want to get away from accessing properties directly on the
underlying configuration hash, we'll move here to accessing those values
via the implementations on `DatabaseConfig` (or more specifically,
`HashConfig`).

There are still codepaths that are passing around `configuration_hash`,
and follow-on PRs will address those with the goal of using
configuration objects everywhere up until the point we pass a resolved
hash over to the underlying client.
Co-authored-by: Neileencodes <eileencodes@gmail.com>

We're moving in the direction of getting `DatabaseConfig` to be used in
more places instead of `Hash` connections configurations being passed
around directly.

Here we're making all of `DatabaseTasks` and `databases.rake` use
database configuration objects.

A few notes:

* This also contains some tests that were missing for public methods on
  `DatabaseTasks`: `charset_current` & `collation_current`

* There is a deprecation of some arguments in `schema_up_to_date?` that
  are now duplicative/confusing. One of them wasn't used previously
  (`environment`), and now `spec_name` can be pulled directly off of the
  `db_config (DatatabaseConfig)` object.
Co-authored-by: Neileencodes <eileencodes@gmail.com>

Uses of `current_config` and `current_config=` were removed when
implementing multiple datatabase Rails tasks. This means they are no
longer used internally. However, since it is a documented method it may
be used in gems or applications. This change deprecates these methods so
that we can remove them in 6.2.
Co-authored-by: NJohn Crepezzi <john.crepezzi@gmail.com>

This commit fixes a method typo that was introduced in #37199, also also
adds tests for the `DatabaseTasks#current_config` method given it's
current implementation.

While I was in here, I also made it so that if `current_config` is
called with an `env` it won't try to look up the `Rails.env`
potentially resulting in an error for non-Rails application.

IMO this method should be deprecated & removed as it's no longer used in
Rails, is confusing (for example: `test_current_config_read_after_set`),
and is currently dependent on Rails if `env` isn't passed.
Co-authored-by: Neileencodes <eileencodes@gmail.com>

Eventually we'd like to get rid of this class altogether but for now
this PR reduces the surface area by removing methods from the class and
moving classes out into their own files.

* `adapter_method` was moved into database configurations
* `initialize_dup` was removed because it was only used in tests
* Resolver is now it's own class under connection adapters
* ConnectionUrlResolver, only used by the configurations, is in a class
under DatabaseConfigurations
Co-authored-by: NJohn Crepezzi <john.crepezzi@gmail.com>

Previously in some places we used symbol keys, and in some places we used
string keys. That made it pretty confusing to figure out in a particular
place what type of configuration object you were working with.

Now internally, all configuration hashes are keyed by symbols and
converted to such on the way in.

A few exceptions:

- `DatabaseConfigurations#to_h` still returns strings for backward compatibility
- Same for `legacy_hash`
- `default_hash` previously could return strings, but the associated
  comment mentions it returns symbol-key `Hash` and now it always does

Because this is a change in behavior, a few method renames have happened:

- `DatabaseConfig#config` is now `DatabaseConfig#configuration_hash` and returns a symbol-key `Hash`
- `ConnectionSpecification#config` is now `ConnectionSpecification#underlying_configuration_hash` and returns the `Hash` of the underlying `DatabaseConfig`
- `DatabaseConfig#config` was added back, returns `String`-keys for backward compatibility, and is deprecated in favor of the new `configuration_hash`
Co-authored-by: Neileencodes <eileencodes@gmail.com>

Previously, every time we ran tests in parallel we would load the schema
for all N DBs.

Now that we have the ability to sync DBs with the schema file exactly
using metadata and a SHA, we can instead only load the schemas when they
change.

Previously, we used the migration status to determine whether the test
database(s) needed to be reloaded from the schema. This worked in most
cases, but if a schema.rb was modified outside of migrations or if a
migration was rolled back, it would require a manual db:test:prepare.

This commit updates load_schema to record the SHA1 of the loaded schema
file inside of the ar_internal_metadata table. We can then use this SHA
to determine whether we should reload the schema.

This ensures that the test DB stays exactly in sync with the schema
file, including rollbacks which fixes a test marked TODO.

Previously it was the responsibility of the database tasks to translate
the invalid statement from creating a duplicate database into an
ActiveRecord::Tasks::DatabaseAlreadyExists error.

It's actually easier for us to do this detection inside of the adapter,
where we already do a case statement on the return code to translate the
error.

This commit introduces ActiveRecord::DatabaseAlreadyExists, a subclass
of StatementInvalid, and updates both AbstractMysqlAdapter and
PostgresqlAdapter to return this more specific exception in that case.

Because this is a subclass of the old exception, StatementInvalid, it
should be backwards compatible with any code expecting that from
create_database.

This works for both create_database and exectute("CREATE DATABASE")

In #36560 I accidentally re-introduced a bug where ActiveRecord can't be
used without Rails. This returns an empty hash if we're outside the
context of Rails since we can't create the database tasks without
loading and reading the database yaml which is something only Railties
can do.

For multiple databases we attempt to generate the tasks by reading the
database.yml before the Rails application is booted. This means that we
need to strip out ERB since it could be reading Rails configs.

In some cases like https://github.com/rails/rails/issues/36540 the ERB
is too complex and we can't overwrite with the DummyCompilier we used in
https://github.com/rails/rails/pull/35497. For the complex causes we
simply issue a warning that says we couldn't infer the database tasks
from the database.yml.

While working on this I decided to update the code to only load the
database.yml once initially so that we avoid having to issue the same
warning multiple times. Note that this had no performance impact in my
testing and is merely for not having to save the error off somewhere.
Also this feels cleaner.

Note that this will not break running tasks that exist, it will just
mean that tasks for multi-db like `db:create:other_db` will not be
generated. If the database.yml is actually unreadable it will blow up
during normal rake task calls.

Fixes #36540

This PR moves the `schema_migration` to `migration_context` so that we
can access the `schema_migration` per connection.

This does not change behavior of the SchemaMigration if you are using
one database. This also does not change behavior of any public APIs.
`Migrator` is private as is `MigrationContext` so we can change these as
needed.

We now need to pass a `schema_migration` to `Migrator` so that we can
run migrations on the right connection outside the context of a rake
task.

The bugs this fixes were discovered while debugging the issues around
the SchemaCache on initialization with multiple database. It was clear
that `get_all_versions` wouldn't work without these changes outside the
context of a rake task (because in the rake task we establish a
connection and change AR::Base.connection to the db we're running on).

Because the `SchemaCache` relies on the `SchemaMigration` information we
need to make sure we store it per-connection rather than on
ActiveRecord::Base.

[Eileen M. Uchitelle & Aaron Patterson]

Method added in https://github.com/rails/rails/pull/36416

When one database existed already, but not the other,
during setup of missing one, existing database was wiped out.

This change adds the ability to run up/down for a database in a multi-db
environment.

If you have an app with a primary and animals database the following
tasks will be generated:

```
VERSION=123 rake db:migrate:up:primary
VERSION=123 rake db:migrate:up:primary

VERSION=123 rake db:migrate:down:primary
VERSION=123 rake db:migrate:up:animals
```

I didn't generate descriptions with them since we don't generate a
description for a single database application.

In addition to this change I've made it so if your application has
multiple databases Rails will raise if you try to run `up` or `down`
without a namespace. This is because we don't know which DB you want to
run `up` or `down` against unless the app tells us, so it's safer to
just block it and recommend using namespaced versions of up/down
respectively.

The output for the raise looks like:

```
You're using a multiple database application. To use `db:migrate:down`
you must run the namespaced task with a VERSION. Available tasks are
db:migrate:down:primary and db:migrate:down:animals.
```

When I added the rake tasks for multiple databases I accidentally made
it so you couldn't use Active Record without Railties.

Since getting the database configuration depends on being able to load
Railties (there is no other way to read and parse the yaml file and load
it) I've decided that using multiple databases outside of Rails is a
no-op.

I've changed the code here to return if Rails isn't defined. Then I
changed the other tasks to use `ActiveRecord::Tasks::DatabaseTasks.env`
instead of `Rails.env`. The multi-db tasks can keep using `Rails.env`
because they'll only be generated if we're using Rails and not just
Active Record.

* Remove redundant `table_names.empty?`
* Early return in `truncate_tables` since it is already deeply nested
* Move `truncate_tables` out from between `exec_delete` and `exec_update`

Before:

```
   (16.4ms)  TRUNCATE TABLE `author_addresses`
   (20.5ms)  TRUNCATE TABLE `authors`
   (19.4ms)  TRUNCATE TABLE `posts`
```

After:

```
  Truncate Tables (19.5ms)  TRUNCATE TABLE `author_addresses`;
TRUNCATE TABLE `authors`;
TRUNCATE TABLE `posts`
```

This is to easier make `truncate_tables` to bulk statements.

This change adds a new method that loads the YAML for the database
config without parsing the ERB. This may seem odd but bear with me:

When we added the ability to have rake tasks for multiple databases we
started looping through the configurations to collect the namespaces so
we could do `rake db:create:my_second_db`. See #32274.

This caused a problem where if you had `Rails.config.max_threads` set in
your database.yml it will blow up because the environment that defines
`max_threads` isn't loaded during `rake -T`. See #35468.

We tried to fix this by adding the ability to just load the YAML and
ignore ERB all together but that caused a bug in GitHub's YAML loading
where if you used multi-line ERB the YAML was invalid. That led us to
reverting some changes in #33748.

After trying to resolve this a bunch of ways `@tenderlove` came up with
replacing the ERB values so that we don't need to load the environment
but we also can load the YAML.

This change adds a DummyCompiler for ERB that will replace all the
values so we can load the database yaml and create the rake tasks.
Nothing else uses this method so it's "safe".

DO NOT use this method in your application.

Fixes #35468

* Add `ActiveRecord::Base.connection.truncate` for SQLite3 adapter.

SQLite doesn't support `TRUNCATE TABLE`, but SQLite3 adapter can support
`ActiveRecord::Base.connection.truncate` by using `DELETE FROM`.

`DELETE` without `WHERE` uses "The Truncate Optimization",
see https://www.sqlite.org/lang_delete.html.

* Add `rails db:seed:replant` that truncates database tables and loads the seeds

Closes #34765