未验证 提交 66abd260 编写于 作者: M Marin Jankovski

Merge commit 'cb604491' into 12-2-auto-deploy-20190818

......@@ -284,13 +284,16 @@ Introduced in GitLab 11.3. This file lives in `/var/log/gitlab/gitlab-rails/impo
Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for
installations from source.
## `auth.log`
## `auth.log`
Introduced in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/auth.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/auth.log` for
installations from source.
It logs information whenever [Rack Attack] registers an abusive request.
This log records:
- Information whenever [Rack Attack] registers an abusive request.
- Requests over the [Rate Limit] on raw endpoints.
NOTE: **Note:**
From [%12.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/62756), user id and username are available on this log.
......@@ -334,3 +337,4 @@ installations from source.
[repocheck]: repository_checks.md
[Rack Attack]: ../security/rack_attack.md
[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md
......@@ -77,11 +77,12 @@ authentication is not provided. For
those cases where it is not required, this will be mentioned in the documentation
for each individual endpoint. For example, the [`/projects/:id` endpoint](projects.md).
There are three ways to authenticate with the GitLab API:
There are four ways to authenticate with the GitLab API:
1. [OAuth2 tokens](#oauth2-tokens)
1. [Personal access tokens](#personal-access-tokens)
1. [Session cookie](#session-cookie)
1. [GitLab CI job token](#gitlab-ci-job-token-premium) **(PREMIUM)**
For admins who want to authenticate with the API as a specific user, or who want to build applications or scripts that do so, two options are available:
......@@ -151,6 +152,14 @@ The primary user of this authentication method is the web frontend of GitLab its
which can use the API as the authenticated user to get a list of their projects,
for example, without needing to explicitly pass an access token.
### GitLab CI job token **(PREMIUM)**
With a few API endpoints you can use a [GitLab CI job token](../user/project/new_ci_build_permissions_model.md#job-token)
to authenticate with the API:
* [Get job artifacts](jobs.md#get-job-artifacts)
* [Pipeline triggers](pipeline_triggers.md)
### Impersonation tokens
> [Introduced][ce-9099] in GitLab 9.0. Needs admin permissions.
......
doc/ci/quick_start/img/build_log.png

34.4 KB | W: | H:

doc/ci/quick_start/img/build_log.png

135.1 KB | W: | H:

doc/ci/quick_start/img/build_log.png
doc/ci/quick_start/img/build_log.png
doc/ci/quick_start/img/build_log.png
doc/ci/quick_start/img/build_log.png
  • 2-up
  • Swipe
  • Onion skin
......@@ -88,7 +88,7 @@ visit the project you want to make the Runner work for in GitLab:
## Registering a group Runner
Creating a group Runner requires Maintainer permissions for the group. To create a
Creating a group Runner requires Owner permissions for the group. To create a
group Runner visit the group you want to make the Runner work for in GitLab:
1. Go to **Settings > CI/CD** to obtain the token
......@@ -124,9 +124,9 @@ To lock/unlock a Runner:
## Assigning a Runner to another project
If you are Maintainer on a project where a specific Runner is assigned to, and the
If you are an Owner on a project where a specific Runner is assigned to, and the
Runner is not [locked only to that project](#locking-a-specific-runner-from-being-enabled-for-other-projects),
you can enable the Runner also on any other project where you have Maintainer permissions.
you can enable the Runner also on any other project where you have Owner permissions.
To enable/disable a Runner in your project:
......@@ -250,7 +250,7 @@ When you [register a Runner][register], its default behavior is to **only pick**
[tagged jobs](../yaml/README.md#tags).
NOTE: **Note:**
Maintainer [permissions](../../user/permissions.md) are required to change the
Owner [permissions](../../user/permissions.md) are required to change the
Runner settings.
To make a Runner pick untagged jobs:
......
......@@ -20,7 +20,7 @@ A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a w
We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/).
The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository.
The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository.
When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects.
......@@ -511,7 +511,15 @@ To summarize here's the [directory structure of the `git` user home directory](.
ps aux | grep '^git'
```
GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or Nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process).
GitLab has several components to operate. It requires a persistent database
(PostgreSQL) and redis database, and uses Apache httpd or Nginx to proxypass
Unicorn. All these components should run as different system users to GitLab
(e.g., `postgres`, `redis` and `www-data`, instead of `git`).
As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server
running on port `8080` by default). Under the GitLab user there are normally 4
processes: `unicorn_rails master` (1 process), `unicorn_rails worker`
(2 processes), `sidekiq` (1 process).
### Repository access
......@@ -554,12 +562,9 @@ $ /etc/init.d/nginx
Usage: nginx {start|stop|restart|reload|force-reload|status|configtest}
```
Persistent database (one of the following)
Persistent database
```
/etc/init.d/mysqld
Usage: /etc/init.d/mysqld {start|stop|status|restart|condrestart|try-restart|reload|force-reload}
$ /etc/init.d/postgresql
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]
```
......@@ -597,11 +602,6 @@ PostgreSQL
- `/var/log/postgresql/*`
MySQL
- `/var/log/mysql/*`
- `/var/log/mysql.*`
### GitLab specific config files
GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include:
......
# Hash Indexes
Both PostgreSQL and MySQL support hash indexes besides the regular btree
PostgreSQL supports hash indexes besides the regular btree
indexes. Hash indexes however are to be avoided at all costs. While they may
_sometimes_ provide better performance the cost of rehashing can be very high.
More importantly: at least until PostgreSQL 10.0 hash indexes are not
......
......@@ -9,7 +9,7 @@ bundle exec rake setup
```
The `setup` task is an alias for `gitlab:setup`.
This tasks calls `db:reset` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database.
This tasks calls `db:reset` to create the database, and calls `db:seed_fu` to seed the database.
Note: `db:setup` calls `db:seed` but this does nothing.
### Seeding issues for all or a given project
......
......@@ -2,7 +2,7 @@
Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string
requires at least 40 bytes, an additional byte to store the encoding, and
perhaps more space depending on the internals of PostgreSQL and MySQL.
perhaps more space depending on the internals of PostgreSQL.
On the other hand, if one were to store a SHA1 as binary one would only need 20
bytes for the actual SHA1, and 1 or 4 bytes of additional space (again depending
......
......@@ -15,14 +15,11 @@ FROM issues
WHERE title LIKE 'WIP:%';
```
On PostgreSQL the `LIKE` statement is case-sensitive. On MySQL this depends on
the case-sensitivity of the collation, which is usually case-insensitive. To
perform a case-insensitive `LIKE` on PostgreSQL you have to use `ILIKE` instead.
This statement in turn isn't supported on MySQL.
On PostgreSQL the `LIKE` statement is case-sensitive. To perform a case-insensitive
`LIKE` you have to use `ILIKE` instead.
To work around this problem you should write `LIKE` queries using Arel instead
of raw SQL fragments as Arel automatically uses `ILIKE` on PostgreSQL and `LIKE`
on MySQL. This means that instead of this:
To handle this automatically you should use `LIKE` queries using Arel instead
of raw SQL fragments, as Arel automatically uses `ILIKE` on PostgreSQL.
```ruby
Issue.where('title LIKE ?', 'WIP:%')
......@@ -45,7 +42,7 @@ table = Issue.arel_table
Issue.where(table[:title].matches('WIP:%').or(table[:foo].matches('WIP:%')))
```
For PostgreSQL this produces:
On PostgreSQL, this produces:
```sql
SELECT *
......@@ -53,18 +50,10 @@ FROM issues
WHERE (title ILIKE 'WIP:%' OR foo ILIKE 'WIP:%')
```
In turn for MySQL this produces:
```sql
SELECT *
FROM issues
WHERE (title LIKE 'WIP:%' OR foo LIKE 'WIP:%')
```
## LIKE & Indexes
Neither PostgreSQL nor MySQL use any indexes when using `LIKE` / `ILIKE` with a
wildcard at the start. For example, this will not use any indexes:
PostgreSQL won't use any indexes when using `LIKE` / `ILIKE` with a wildcard at
the start. For example, this will not use any indexes:
```sql
SELECT *
......@@ -75,9 +64,8 @@ WHERE title ILIKE '%WIP:%';
Because the value for `ILIKE` starts with a wildcard the database is not able to
use an index as it doesn't know where to start scanning the indexes.
MySQL provides no known solution to this problem. Luckily PostgreSQL _does_
provide a solution: trigram GIN indexes. These indexes can be created as
follows:
Luckily, PostgreSQL _does_ provide a solution: trigram GIN indexes. These
indexes can be created as follows:
```sql
CREATE INDEX [CONCURRENTLY] index_name_here
......
......@@ -15,16 +15,6 @@ manifest themselves within our code. When designing our tests, take time to revi
our test design. We can find some helpful heuristics documented in the Handbook in the
[Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section.
## Run tests against MySQL
By default, tests are only run against PostgreSQL, but you can run them on
demand against MySQL by following one of the following conventions:
| Convention | Valid example |
|:----------------------|:-----------------------------|
| Include `mysql` in your branch name | `enhance-mysql-support` |
| Include `[run mysql]` in your commit message | `Fix MySQL support<br><br>[run mysql]` |
## Test speed
GitLab has a massive test suite that, without [parallelization], can take hours
......
......@@ -39,7 +39,6 @@ slowest test files and try to improve them.
## CI setup
- On CE and EE, the test suite runs both PostgreSQL and MySQL.
- Rails logging to `log/test.log` is disabled by default in CI [for
performance reasons][logging]. To override this setting, provide the
`RAILS_ENABLE_TEST_LOG` environment variable.
......
......@@ -35,8 +35,8 @@ Once a test is in quarantine, there are 3 choices:
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
- `rspec-pg-quarantine` and `rspec-mysql-quarantine` (CE & EE)
- `rspec-pg-quarantine-ee` and `rspec-mysql-quarantine-ee` (EE only)
- `rspec-pg-quarantine` (CE & EE)
- `rspec-pg-quarantine-ee` (EE only)
## Automatic retries and flaky tests detection
......
# Verifying Database Capabilities
Sometimes certain bits of code may only work on a certain database and/or
Sometimes certain bits of code may only work on a certain database
version. While we try to avoid such code as much as possible sometimes it is
necessary to add database (version) specific behaviour.
To facilitate this we have the following methods that you can use:
- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used
- `Gitlab::Database.mysql?`: returns `true` if MySQL is being used
- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used.
You can normally just assume this is the case.
- `Gitlab::Database.version`: returns the PostgreSQL version number as a string
in the format `X.Y.Z`. This method does not work for MySQL
in the format `X.Y.Z`.
This allows you to write code such as:
......
......@@ -7,9 +7,8 @@ downtime.
## Adding Columns
On PostgreSQL you can safely add a new column to an existing table as long as it
does **not** have a default value. For example, this query would not require
downtime:
You can safely add a new column to an existing table as long as it does **not**
have a default value. For example, this query would not require downtime:
```sql
ALTER TABLE projects ADD COLUMN random_value int;
......@@ -27,11 +26,6 @@ This requires updating every single row in the `projects` table so that
indexes in a table. This in turn acquires enough locks on the table for it to
effectively block any other queries.
As of MySQL 5.6 adding a column to a table is still quite an expensive
operation, even when using `ALGORITHM=INPLACE` and `LOCK=NONE`. This means
downtime _may_ be required when modifying large tables as otherwise the
operation could potentially take hours to complete.
Adding a column with a default value _can_ be done without requiring downtime
when using the migration helper method
`Gitlab::Database::MigrationHelpers#add_column_with_default`. This method works
......@@ -311,8 +305,7 @@ migrations](background_migrations.md#cleaning-up).
## Adding Indexes
Adding indexes is an expensive process that blocks INSERT and UPDATE queries for
the duration. When using PostgreSQL one can work around this by using the
`CONCURRENTLY` option:
the duration. You can work around this by using the `CONCURRENTLY` option:
```sql
CREATE INDEX CONCURRENTLY index_name ON projects (column_name);
......@@ -336,17 +329,9 @@ end
Note that `add_concurrent_index` can not be reversed automatically, thus you
need to manually define `up` and `down`.
When running this on PostgreSQL the `CONCURRENTLY` option mentioned above is
used. On MySQL this method produces a regular `CREATE INDEX` query.
MySQL doesn't really have a workaround for this. Supposedly it _can_ create
indexes without the need for downtime but only for variable width columns. The
details on this are a bit sketchy. Since it's better to be safe than sorry one
should assume that adding indexes requires downtime on MySQL.
## Dropping Indexes
Dropping an index does not require downtime on both PostgreSQL and MySQL.
Dropping an index does not require downtime.
## Adding Tables
......@@ -370,7 +355,7 @@ transaction this means this approach would require downtime.
GitLab allows you to work around this by using
`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method
ensures that when PostgreSQL is used no downtime is needed.
ensures that no downtime is needed.
## Removing Foreign Keys
......
......@@ -9,7 +9,7 @@ type: reference
This setting allows you to rate limit the requests to raw endpoints, defaults to `300` requests per minute.
It can be modified in **Admin Area > Network > Performance Optimization**.
For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/controllers/application_controller.rb` will be blocked.
For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/controllers/application_controller.rb` will be blocked. Access to the raw file will be released after 1 minute.
![Rate limits on raw endpoints](img/rate_limits_on_raw_endpoints.png)
......@@ -18,3 +18,5 @@ This limit is:
- Applied independently per project, per commit and per file path.
- Not applied per IP address.
- Active by default. To disable, set the option to `0`.
Requests over the rate limit are logged into `auth.log`.
# Cycle Analytics
> - Introduced prior to GitLab 12.2 at the project level.
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2 at the group level (enabled by feature flag `analytics`).
Cycle Analytics measures the time spent to go from an [idea to production] - also known
as cycle time - for each of your projects. Cycle Analytics displays the median time for an idea to
reach production, along with the time typically spent in each DevOps stage along the way.
Cycle Analytics is useful in order to quickly determine the velocity of a given
project. It points to bottlenecks in the development process, enabling management
to uncover, triage, and identify the root cause of slowdowns in the software development life cycle.
Cycle Analytics is tightly coupled with the [GitLab flow] and
calculates a separate median for each stage.
## Overview
Cycle Analytics is available:
- From GitLab 12.2, at the group level in the analytics workspace at
**Analytics > Cycle Analytics**. **(PREMIUM)**
In the future, multiple groups will be selectable which will effectively make this an
instance-level feature.
NOTE: **Note:**
Requires the [analytics workspace](index.md) to be enabled.
- At the project level via **Project > Cycle Analytics**.
There are seven stages that are tracked as part of the Cycle Analytics calculations.
- **Issue** (Tracker)
- Time to schedule an issue (by milestone or by adding it to an issue board)
- **Plan** (Board)
- Time to first commit
- **Code** (IDE)
- Time to create a merge request
- **Test** (CI)
- Time it takes GitLab CI/CD to test your code
- **Review** (Merge Request/MR)
- Time spent on code review
- **Staging** (Continuous Deployment)
- Time between merging and deploying to production
- **Production** (Total)
- Total lifecycle time; i.e. the velocity of the project or team
## How the data is measured
Cycle Analytics records cycle time and data based on the project issues with the
exception of the staging and production stages, where only data deployed to
production are measured.
Specifically, if your CI is not set up and you have not defined a `production`
or `production/*` [environment], then you will not have any data for those stages.
Each stage of Cycle Analytics is further described in the table below.
| **Stage** | **Description** |
| --------- | --------------- |
| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list](../project/issue_board.md#creating-a-new-list) created for it. |
| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. |
| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the issue closing pattern is not present in the merge request description, the MR is not considered to the measurement time of the stage. |
| Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. |
| Review | Measures the median time taken to review the merge request that has closing issue pattern, between its creation and until it's merged. |
| Staging | Measures the median time between merging the merge request with closing issue pattern until the very first deployment to production. It's tracked by the [environment] set to `production` or matching `production/*` (case-sensitive, `Production` won't work) in your GitLab CI configuration. If there isn't a production environment, this is not tracked. |
| Production| The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. |
---
How this works, behind the scenes:
1. Issues and merge requests are grouped together in pairs, such that for each
`<issue, merge request>` pair, the merge request has the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically)
for the corresponding issue. All other issues and merge requests are **not**
considered.
1. Then the `<issue, merge request>` pairs are filtered out by last XX days (specified
by the UI - default is 90 days). So it prohibits these pairs from being considered.
1. For the remaining `<issue, merge request>` pairs, we check the information that
we need for the stages, like issue creation date, merge request merge time,
etc.
To sum up, anything that doesn't follow [GitLab flow] will not be tracked and the
Cycle Analytics dashboard will not present any data for:
- merge requests that do not close an issue.
- issues not labeled with a label present in the Issue Board or for issues not assigned a milestone.
- staging and production stages, if the project has no `production` or `production/*`
environment.
## Example workflow
Below is a simple fictional workflow of a single cycle that happens in a
single day passing through all seven stages. Note that if a stage does not have
a start and a stop mark, it is not measured and hence not calculated in the median
time. It is assumed that milestones are created and CI for testing and setting
environments is configured.
1. Issue is created at 09:00 (start of **Issue** stage).
1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of
**Plan** stage).
1. Start working on the issue, create a branch locally and make one commit at
12:00.
1. Make a second commit to the branch which mentions the issue number at 12.30
(stop of **Plan** stage / start of **Code** stage).
1. Push branch and create a merge request that contains the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically)
in its description at 14:00 (stop of **Code** stage / start of **Test** and
**Review** stages).
1. The CI starts running your scripts defined in [`.gitlab-ci.yml`][yml] and
takes 5min (stop of **Test** stage).
1. Review merge request, ensure that everything is OK and merge the merge
request at 19:00. (stop of **Review** stage / start of **Staging** stage).
1. Now that the merge request is merged, a deployment to the `production`
environment starts and finishes at 19:30 (stop of **Staging** stage).
1. The cycle completes and the sum of the median times of the previous stages
is recorded to the **Production** stage. That is the time between creating an
issue and deploying its relevant merge request to production.
From the above example you can conclude the time it took each stage to complete
as long as their total time:
- **Issue**: 2h (11:00 - 09:00)
- **Plan**: 1h (12:00 - 11:00)
- **Code**: 2h (14:00 - 12:00)
- **Test**: 5min
- **Review**: 5h (19:00 - 14:00)
- **Staging**: 30min (19:30 - 19:00)
- **Production**: Since this stage measures the sum of median time off all
previous stages, we cannot calculate it if we don't know the status of the
stages before. In case this is the very first cycle that is run in the project,
then the **Production** time is 10h 30min (19:30 - 09:00)
A few notes:
- In the above example we demonstrated that it doesn't matter if your first
commit doesn't mention the issue number, you can do this later in any commit
of the branch you are working on.
- You can see that the **Test** stage is not calculated to the overall time of
the cycle since it is included in the **Review** process (every MR should be
tested).
- The example above was just **one cycle** of the seven stages. Add multiple
cycles, calculate their median time and the result is what the dashboard of
Cycle Analytics is showing.
## Permissions
The current permissions on the Project Cycle Analytics dashboard are:
- Public projects - anyone can access
- Internal projects - any authenticated user can access
- Private projects - any member Guest and above can access
You can [read more about permissions][permissions] in general.
NOTE: **Note:**
As of GitLab 12.2, the project-level page is deprecated. You should access
project-level Cycle Analytics from **Analytics > Cycle Analytics** in the top
navigation bar. We will ensure that the same project-level functionality is available
to CE users in the new analytics space.
For Cycle Analytics functionality introduced in GitLab 12.2 and later:
- Users must have Reporter access or above.
- Features are available only on
[Premium or Silver tiers](https://about.gitlab.com/pricing/) and above.
## More resources
Learn more about Cycle Analytics in the following resources:
- [Cycle Analytics feature page](https://about.gitlab.com/features/cycle-analytics/)
- [Cycle Analytics feature preview](https://about.gitlab.com/2016/09/16/feature-preview-introducing-cycle-analytics/)
- [Cycle Analytics feature highlight](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/)
[ce-5986]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5986
[ce-20975]: https://gitlab.com/gitlab-org/gitlab-ce/issues/20975
[environment]: ../../ci/yaml/README.md#environment
[GitLab flow]: ../../workflow/gitlab_flow.md
[idea to production]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab
[permissions]: ../permissions.md
[yml]: ../../ci/yaml/README.md
# Analytics workspace
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) in GitLab 12.2 (enabled using `analytics` feature flag).
The Analytics workspace will make it possible to aggregate analytics across
GitLab, so that users can view information across multiple projects and groups
in one place.
To access the centralized analytics workspace:
1. Ensure it's enabled. Requires a GitLab administrator to enable it with the `analytics` feature
flag.
1. Once enabled, click on **Analytics** from the top navigation bar.
## Available analytics
From the centralized analytics workspace, the following analytics are available:
- [Cycle Analytics](cycle_analytics.md).
NOTE: **Note:**
Project-level Cycle Analytics are still available at a project's **Project > Cycle Analytics**.
......@@ -112,57 +112,6 @@ Below are the shared Runners settings.
The full contents of our `config.toml` are:
**DigitalOcean**
```toml
concurrent = X
check_interval = 1
metrics_server = "X"
sentry_dsn = "X"
[[runners]]
name = "docker-auto-scale"
request_concurrency = X
url = "https://gitlab.com/"
token = "SHARED_RUNNER_TOKEN"
executor = "docker+machine"
environment = [
"DOCKER_DRIVER=overlay2"
]
limit = X
[runners.docker]
image = "ruby:2.5"
privileged = true
[runners.machine]
IdleCount = 20
IdleTime = 1800
OffPeakPeriods = ["* * * * * sat,sun *"]
OffPeakTimezone = "UTC"
OffPeakIdleCount = 5
OffPeakIdleTime = 1800
MaxBuilds = 1
MachineName = "srm-%s"
MachineDriver = "digitalocean"
MachineOptions = [
"digitalocean-image=X",
"digitalocean-ssh-user=core",
"digitalocean-region=nyc1",
"digitalocean-size=s-2vcpu-2gb",
"digitalocean-private-networking",
"digitalocean-tags=shared_runners,gitlab_com",
"engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR",
"digitalocean-access-token=DIGITAL_OCEAN_ACCESS_TOKEN",
]
[runners.cache]
Type = "s3"
BucketName = "runner"
Insecure = true
Shared = true
ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER"
AccessKey = "ACCESS_KEY"
SecretKey = "ACCESS_SECRET_KEY"
```
**Google Cloud Platform**
```toml
......@@ -178,20 +127,25 @@ sentry_dsn = "X"
token = "SHARED_RUNNER_TOKEN"
executor = "docker+machine"
environment = [
"DOCKER_DRIVER=overlay2"
"DOCKER_DRIVER=overlay2",
"DOCKER_TLS_CERTDIR="
]
limit = X
[runners.docker]
image = "ruby:2.5"
privileged = true
volumes = [
"/certs/client",
"/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP.
]
[runners.machine]
IdleCount = 20
IdleTime = 1800
IdleCount = 50
IdleTime = 3600
OffPeakPeriods = ["* * * * * sat,sun *"]
OffPeakTimezone = "UTC"
OffPeakIdleCount = 5
OffPeakIdleTime = 1800
MaxBuilds = 1
OffPeakIdleCount = 15
OffPeakIdleTime = 3600
MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused.
MachineName = "srm-%s"
MachineDriver = "google"
MachineOptions = [
......@@ -202,17 +156,18 @@ sentry_dsn = "X"
"google-tags=gitlab-com,srm",
"google-use-internal-ip",
"google-zone=us-east1-d",
"engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/issues/3214#note_82892928
"google-machine-image=PROJECT/global/images/IMAGE",
"engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR"
"engine-opt=ipv6", # This will create IPv6 interfaces in the containers.
"engine-opt=fixed-cidr-v6=fc00::/7",
"google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600
]
[runners.cache]
Type = "s3"
BucketName = "runner"
Insecure = true
Type = "gcs"
Shared = true
ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER"
AccessKey = "ACCESS_KEY"
SecretKey = "ACCESS_SECRET_KEY"
[runners.cache.gcs]
CredentialsFile = "/path/to/file"
BucketName = "bucket-name"
```
## Sidekiq
......
......@@ -59,15 +59,14 @@ Once [Single sign-on](index.md) has been configured, we can:
### Azure
First, double check the [Single sign-on](index.md) configuration for your group and ensure that **Name identifier value** (NameID) points to `user.objectid` or another unique identifier. This will match the `extern_uid` used on GitLab.
The SAML application that was created during [Single sign-on](index.md) setup now needs to be set up for SCIM.
![Name identifier value mapping](img/scim_name_identifier_mapping.png)
1. Check the configuration for your GitLab SAML app and ensure that **Name identifier value** (NameID) points to `user.objectid` or another unique identifier. This will match the `extern_uid` used on GitLab.
#### Set up admin credentials
![Name identifier value mapping](img/scim_name_identifier_mapping.png)
Next, configure your GitLab application in Azure by following the
[Provisioning users and groups to applications that support SCIM](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim)
section in Azure's SCIM setup documentation.
1. Set up automatic provisioning and administrative credentials by following the
[Provisioning users and groups to applications that support SCIM](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim) section in Azure's SCIM setup documentation.
During this configuration, note the following:
......@@ -97,6 +96,7 @@ You can then test the connection by clicking on **Test Connection**. If the conn
NOTE: **Note:** If you used a unique identifier **other than** `objectId`, be sure to map it instead to both `id` and `externalId`.
1. Below the mapping list click on **Show advanced options > Edit attribute list for AppName**.
1. Leave the `id` as the primary and only required field.
NOTE: **Note:**
......@@ -129,8 +129,7 @@ When testing the connection, you may encounter an error: **You appear to have en
When checking the Audit Logs for the Provisioning, you can sometimes see the
error `Namespace can't be blank, Name can't be blank, and User can't be blank.`
This is likely caused because not all required fields (such as first name and
last name) are present for all users being mapped.
This is likely caused because not all required fields (such as first name and last name) are present for all users being mapped.
As a workaround, try an alternate mapping:
......
......@@ -162,7 +162,7 @@ to learn more.
### Cycle Analytics permissions
Find the current permissions on the Cycle Analytics dashboard on
the [documentation on Cycle Analytics permissions](project/cycle_analytics.md#permissions).
the [documentation on Cycle Analytics permissions](analytics/cycle_analytics.md#permissions).
### Issue Board permissions
......@@ -237,13 +237,16 @@ To learn more, read through the documentation on
## Guest User
Create a user and assign to a project with a role as `Guest` user, this user
will be considered as guest user by GitLab and will not take up the license.
There is no specific `Guest` role for newly created users. If this user will
be assigned a higher role to any of the projects and groups then this user will
take a license seat. If a user creates a project this user becomes a maintainer,
therefore, takes up a license seat as well, in order to prevent this you have
to go and edit user profile and mark the user as External.
When a user is given `Guest` permissions on a project and/or group, and holds no
higher permission level on any other project or group on the instance, the user
is considered a guest user by GitLab and will not consume a license seat.
There is no other specific "guest" designation for newly created users.
If the user is assigned a higher role on any projects or groups, the user will
take a license seat. If a user creates a project, the user becomes a `Maintainer`
on the project, resulting in the use of a license seat. To prevent a guest user
from creating projects, you can edit the user profile to mark the user as
[External](#external-users-permissions).
## External users permissions
......
# Cycle Analytics
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) at a group level in [GitLab Premium and Silver](https://about.gitlab.com/pricing/) 12.2 (enabled by feature flag `analytics`).
Cycle Analytics measures the time spent to go from an [idea to production] - also known
as cycle time - for each of your projects. Cycle Analytics displays the median time for an idea to
reach production, along with the time typically spent in each DevOps stage along the way.
Cycle Analytics is useful in order to quickly determine the velocity of a given
project. It points to bottlenecks in the development process, enabling management
to uncover, triage, and root-cause slowdowns in the software development life cycle.
Cycle Analytics is tightly coupled with the [GitLab flow] and
calculates a separate median for each stage.
## Overview
Cycle Analytics are available at a:
- Group level from the top navigation bar **Analytics > Cycle Analytics**. **(PREMIUM)**
In the future, multiple groups will be selectable which will effectively make this an
instance-level feature.
- Project level from a project's **Project > Cycle Analytics**.
![Cycle Analytics landing page](img/cycle_analytics_landing_page.png)
There are seven stages that are tracked as part of the Cycle Analytics calculations.
- **Issue** (Tracker)
- Time to schedule an issue (by milestone or by adding it to an issue board)
- **Plan** (Board)
- Time to first commit
- **Code** (IDE)
- Time to create a merge request
- **Test** (CI)
- Time it takes GitLab CI/CD to test your code
- **Review** (Merge Request/MR)
- Time spent on code review
- **Staging** (Continuous Deployment)
- Time between merging and deploying to production
- **Production** (Total)
- Total lifecycle time; i.e. the velocity of the project or team
## How the data is measured
Cycle Analytics records cycle time and data based on the project issues with the
exception of the staging and production stages, where only data deployed to
production are measured.
Specifically, if your CI is not set up and you have not defined a `production`
or `production/*` [environment], then you will not have any data for those stages.
Below you can see in more detail what the various stages of Cycle Analytics mean.
| **Stage** | **Description** |
| --------- | --------------- |
| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list][board] created for it. |
| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. |
| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern] to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the issue closing pattern is not present in the merge request description, the MR is not considered to the measurement time of the stage. |
| Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. |
| Review | Measures the median time taken to review the merge request that has closing issue pattern, between its creation and until it's merged. |
| Staging | Measures the median time between merging the merge request with closing issue pattern until the very first deployment to production. It's tracked by the [environment] set to `production` or matching `production/*` (case-sensitive, `Production` won't work) in your GitLab CI configuration. If there isn't a production environment, this is not tracked. |
| Production| The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. |
---
redirect_to: '../analytics/cycle_analytics.md'
---
Here's a little explanation of how this works behind the scenes:
1. Issues and merge requests are grouped together in pairs, such that for each
`<issue, merge request>` pair, the merge request has the [issue closing pattern]
for the corresponding issue. All other issues and merge requests are **not**
considered.
1. Then the `<issue, merge request>` pairs are filtered out by last XX days (specified
by the UI - default is 90 days). So it prohibits these pairs from being considered.
1. For the remaining `<issue, merge request>` pairs, we check the information that
we need for the stages, like issue creation date, merge request merge time,
etc.
To sum up, anything that doesn't follow the [GitLab flow] won't be tracked at all.
So, the Cycle Analytics dashboard won't present any data:
- For merge requests that do not close an issue.
- For issues not labeled with a label present in the Issue Board or for issues not assigned a milestone.
- For staging and production stages, if the project has no `production` or `production/*`
environment.
## Example workflow
Below is a simple fictional workflow of a single cycle that happens in a
single day passing through all seven stages. Note that if a stage does not have
a start and a stop mark, it is not measured and hence not calculated in the median
time. It is assumed that milestones are created and CI for testing and setting
environments is configured.
1. Issue is created at 09:00 (start of **Issue** stage).
1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of
**Plan** stage).
1. Start working on the issue, create a branch locally and make one commit at
12:00.
1. Make a second commit to the branch which mentions the issue number at 12.30
(stop of **Plan** stage / start of **Code** stage).
1. Push branch and create a merge request that contains the [issue closing pattern]
in its description at 14:00 (stop of **Code** stage / start of **Test** and
**Review** stages).
1. The CI starts running your scripts defined in [`.gitlab-ci.yml`][yml] and
takes 5min (stop of **Test** stage).
1. Review merge request, ensure that everything is OK and merge the merge
request at 19:00. (stop of **Review** stage / start of **Staging** stage).
1. Now that the merge request is merged, a deployment to the `production`
environment starts and finishes at 19:30 (stop of **Staging** stage).
1. The cycle completes and the sum of the median times of the previous stages
is recorded to the **Production** stage. That is the time between creating an
issue and deploying its relevant merge request to production.
From the above example you can conclude the time it took each stage to complete
as long as their total time:
- **Issue**: 2h (11:00 - 09:00)
- **Plan**: 1h (12:00 - 11:00)
- **Code**: 2h (14:00 - 12:00)
- **Test**: 5min
- **Review**: 5h (19:00 - 14:00)
- **Staging**: 30min (19:30 - 19:00)
- **Production**: Since this stage measures the sum of median time off all
previous stages, we cannot calculate it if we don't know the status of the
stages before. In case this is the very first cycle that is run in the project,
then the **Production** time is 10h 30min (19:30 - 09:00)
A few notes:
- In the above example we demonstrated that it doesn't matter if your first
commit doesn't mention the issue number, you can do this later in any commit
of the branch you are working on.
- You can see that the **Test** stage is not calculated to the overall time of
the cycle since it is included in the **Review** process (every MR should be
tested).
- The example above was just **one cycle** of the seven stages. Add multiple
cycles, calculate their median time and the result is what the dashboard of
Cycle Analytics is showing.
## Permissions
The current permissions on the Project Cycle Analytics dashboard are:
- Public projects - anyone can access
- Internal projects - any authenticated user can access
- Private projects - any member Guest and above can access
You can [read more about permissions][permissions] in general.
NOTE: **Note:**
As of GitLab 12.2, the project-level page is deprecated. You should access
project-level Cycle Analytics from **Analytics > Cycle Analytics** in the top
navigation bar. We will ensure that the same project-level functionality is available
to CE users in the new analytics space.
For Cycle Analytics functionality introduced in GitLab 12.2 and later:
- Users must have Reporter access or above.
- Features are available only on
[Premium or Silver tiers](https://about.gitlab.com/pricing/) and above.
## More resources
Learn more about Cycle Analytics in the following resources:
- [Cycle Analytics feature page](https://about.gitlab.com/features/cycle-analytics/)
- [Cycle Analytics feature preview](https://about.gitlab.com/2016/09/16/feature-preview-introducing-cycle-analytics/)
- [Cycle Analytics feature highlight](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/)
[board]: issue_board.md#creating-a-new-list
[ce-5986]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5986
[ce-20975]: https://gitlab.com/gitlab-org/gitlab-ce/issues/20975
[environment]: ../../ci/yaml/README.md#environment
[GitLab flow]: ../../workflow/gitlab_flow.md
[idea to production]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab
[issue closing pattern]: issues/managing_issues.md#closing-issues-automatically
[permissions]: ../permissions.md
[yml]: ../../ci/yaml/README.md
This document was moved to [another location](../analytics/cycle_analytics.md)
......@@ -93,7 +93,7 @@ even if the status you are changing to is the same.
After saving the configuration, your GitLab project will be able to interact
with all Jira projects in your Jira instance and you'll see the Jira link on the GitLab project pages that takes you to the appropriate Jira project.
![Jira service page](img/jira_service_page.png)
![Jira service page](img/jira_service_page_v12_2.png)
## Jira issues
......
......@@ -2,9 +2,9 @@
type: reference, concepts
---
# Cross-project merge request dependencies **(PREMIUM)**
# Cross-project Merge Request dependencies **(PREMIUM)**
> Introduced in GitLab Premium 12.2
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9688) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2.
Cross-project merge request dependencies allows a required order of merging
between merge requests in different projects to be expressed. If a
......@@ -24,11 +24,11 @@ merge requests in the same project cannot depend on each other.
## Use cases
- Ensure changes to a library are merged before changes to a project that
imports the library
imports the library.
- Prevent a documentation-only merge request from being merged before the merge request
implementing the feature to be documented
implementing the feature to be documented.
- Require an merge request updating a permissions matrix to be merged before merging an
merge request from someone who hasn't yet been granted permissions
merge request from someone who hasn't yet been granted permissions.
It is common for a single logical change to span several merge requests, spread
out across multiple projects, and the order in which they are merged can be
......@@ -60,33 +60,33 @@ new merge request in `awesome-project` (or by editing it, if it already exists).
The dependency needs to be configured on the **dependent** merge
request. There is a "Cross-project dependencies" section in the form:
![Cross-project dependencies form control](img/cross-project-dependencies-edit.png)
![Cross-project dependencies form control](img/cross_project_dependencies_edit_v12_2.png)
Anyone who can edit a merge request can change the list of dependencies.
New dependencies can be added by reference, or by URL. To remove a dependency,
press the "X" by its reference.
press the **X** by its reference.
As dependencies are specified across projects, it's possible that someone else
has added a dependency for a merge request in a project you don't have access to.
These are shown as a simple count:
![Cross-project dependencies form control with inaccessible merge requests](img/cross-project-dependencies-edit-inaccessible.png)
![Cross-project dependencies form control with inaccessible merge requests](img/cross_project_dependencies_edit_inaccessible_v12_2.png)
If necessary, you can remove all the dependencies like this by pressing the "X",
just as you would for a single, visible dependency.
If necessary, you can remove all the dependencies like this by pressing the
**X**, just as you would for a single, visible dependency.
Once you're finished, press the "Save changes" button to submit the request, or
"Cancel" to return without making any changes.
Once you're finished, press the **Save changes** button to submit the request,
or **Cancel** to return without making any changes.
The list of configured dependencies, and the status of each one, is shown in the
merge request widget:
![Cross-project dependencies in merge request widget](img/cross-project-dependencies-view.png)
![Cross-project dependencies in merge request widget](img/cross_project_dependencies_view_v12_2.png)
Until all dependencies have, themselves, been merged, the "Merge"
Until all dependencies have, themselves, been merged, the **Merge**
button will be disabled for the dependent merge request. In
particular, note that **closed** merge request still prevent their
particular, note that **closed merge requests** still prevent their
dependents from being merged - it is impossible to automatically
determine whether the dependency expressed by a closed merge request
has been satisfied in some other way or not.
......
......@@ -40,18 +40,20 @@ discussions, and descriptions:
| `/label ~label1 ~label2` | Add label(s). Label names can also start without ~ but mixed syntax is not supported. | ✓ | ✓ |
| `/unlabel ~label1 ~label2` | Remove all or specific label(s)| ✓ | ✓ |
| `/relabel ~label1 ~label2` | Replace existing label(s) with those specified | ✓ | ✓ |
| `/copy_metadata <#issue | !merge_request>` | Copy labels and milestone from other issue or merge request in the project | ✓ | ✓ |
| `/copy_metadata <#issue>` | Copy labels and milestone from another issue in the project | ✓ | ✓ |
| `/copy_metadata <!merge_request>` | Copy labels and milestone from another merge request in the project | ✓ | ✓ |
| `/estimate <1w 3d 2h 14m>` | Set time estimate | ✓ | ✓ |
| `/remove_estimate` | Remove time estimate | ✓ | ✓ |
| `/spend <time(1h 30m | -1h 5m)> <date(YYYY-MM-DD)>` | Add or subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ |
| `/spend <time(1h 30m)> <date(YYYY-MM-DD)>` | Add spent time; optionally, specify the date that time was spent on | ✓ | ✓ |
| `/spend <time(-1h 5m)> <date(YYYY-MM-DD)>` | Subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ |
| `/remove_time_spent` | Remove time spent | ✓ | ✓ |
| `/lock` | Lock the thread | ✓ | ✓ |
| `/unlock` | Unlock the thread | ✓ | ✓ |
| `/due <in 2 days | this Friday | December 31st>`| Set due date | ✓ | |
| `/due <date>` | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | ✓ | |
| `/remove_due_date` | Remove due date | ✓ | |
| `/weight <0 | 1 | 2 | ...>` | Set weight **(STARTER)** | ✓ | |
| `/weight <value>` | Set weight. Valid options for `<value>` include `0`, `1`, `2`, etc. **(STARTER)** | ✓ | |
| `/clear_weight` | Clears weight **(STARTER)** | ✓ | |
| `/epic <&epic | group&epic | Epic URL>` | Add to epic **(ULTIMATE)** | ✓ | |
| `/epic <epic>` | Add to epic `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. **(ULTIMATE)** | ✓ | |
| `/remove_epic` | Removes from epic **(ULTIMATE)** | ✓ | |
| `/promote` | Promote issue to epic **(ULTIMATE)** | ✓ | |
| `/confidential` | Make confidential | ✓ | |
......@@ -110,9 +112,9 @@ The following quick actions are applicable for epics threads and description:
| `/label ~label1 ~label2` | Add label(s) |
| `/unlabel ~label1 ~label2` | Remove all or specific label(s) |
| `/relabel ~label1 ~label2` | Replace existing label(s) with those specified |
| `/child_epic <&epic | group&epic | Epic URL>` | Adds child epic to epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) |
| `/remove_child_epic <&epic | group&epic | Epic URL>` | Removes child epic from epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) |
| `/parent_epic <&epic | group&epic | Epic URL>` | Sets parent epic to epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) |
| `/child_epic <epic>` | Adds child epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([Introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) **(ULTIMATE)**|
| `/remove_child_epic <epic>` | Removes child epic from `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([Introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) **(ULTIMATE)** |
| `/parent_epic <epic>` | Sets parent epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) **(ULTIMATE)** |
| `/remove_parent_epic` | Removes parent epic from epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) |
<!-- ## Troubleshooting
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册