Minikube has a set of built in addons that can be used enabled, disabled, and opened inside of the local k8s environment. Below is an example of this functionality for the `heapster` addon:
```shell
$ minikube addons list
- registry: disabled
...
...
@@ -26,6 +27,7 @@ Waiting, endpoint for service is not ready yet...
Minikube supports caching non-minikube images using the `minikube cache` command. Images can be added to the cache by running `minikube cache add <img>`, and deleted by running `minikube cache delete <img>`.
Minikube has a "configurator" feature that allows users to configure the Kubernetes components with arbitrary values.
To use this feature, you can use the `--extra-config` flag on the `minikube start` command.
This flag is repeated, so you can pass it several times with several different values to set multiple options.
### Kubeadm bootstrapper
## Kubeadm bootstrapper
The kubeadm bootstrapper can be configured by the `--extra-config` flag on the `minikube start` command. It takes a string of the form `component.key=value` where `component` is one of the strings
***New contributors** ([contributors.md](https://github.com/kubernetes/minikube/blob/master/CONTRIBUTING.md)): Process for new contributors, CLA instructions
***Roadmap** ([roadmap.md](roadmap.md)): The roadmap for future minikube development
## New Features and Dependencies
***Adding a dependency** ([adding_a_dependency.md](adding_a_dependency.md)): How to add or update vendored code
***Adding a new addon** ([adding_an_addon.md](adding_an_addon.md)): How to add a new addon to minikube for `minikube addons`
...
...
@@ -12,6 +13,7 @@
***Adding a new driver** ([adding_driver.md](adding_driver.md)): How to add a new driver to minikube for `minikube create --vm-driver=<driver>`
## Building and Releasing
***Build Guide** ([build_guide.md](build_guide.md)): How to build minikube from source
***ISO Build Guide** ([minikube_iso.md](minikube_iso.md)): How to build and hack on the ISO image that minikube uses
Major releases always get a new ISO. Minor bugfixes may or may not require it: check for changes in the `deploy/iso` folder.
Major releases always get a new ISO. Minor bugfixes may or may not require it: check for changes in the `deploy/iso` folder.
Note: you can build the ISO using the `hack/jenkins/build_iso.sh` script locally.
* navigate to the minikube ISO jenkins job
* Ensure that you are logged in (top right)
* Click "▶️ Build with Parameters" (left)
* For `ISO_VERSION`, type in the intended release version (same as the minikube binary's version)
* For `ISO_BUCKET`, type in `minikube/iso`
* Click *Build*
* navigate to the minikube ISO jenkins job
* Ensure that you are logged in (top right)
* Click "▶️ Build with Parameters" (left)
* For `ISO_VERSION`, type in the intended release version (same as the minikube binary's version)
* For `ISO_BUCKET`, type in `minikube/iso`
* Click *Build*
The build will take roughly 50 minutes.
...
...
@@ -47,7 +47,7 @@ env BUILD_IN_DOCKER=y make cross checksum
Once submitted, HEAD will use the new ISO. Please pay attention to test failures, as this is our integration test across platforms. If there are known acceptable failures, please add a PR comment linking to the appropriate issue.
## Update Release Notes
## Update Release Notes
Run the following script to update the release notes:
...
...
@@ -59,11 +59,11 @@ Merge the output into CHANGELOG.md. See [PR#3175](https://github.com/kubernetes/
## Tag the Release
NOTE: Confirm that all release-related PR's have been submitted before doing this step.
NOTE: Confirm that all release-related PR's have been submitted before doing this step.
Do this in a direct clone of the upstream kubernetes/minikube repository (not your fork!):
```
```shell
version=<new version number>
git fetch
git checkout master
...
...
@@ -76,12 +76,12 @@ git push origin v$version
This step uses the git tag to publish new binaries to GCS and create a github release:
* navigate to the minikube "Release" jenkins job
* Ensure that you are logged in (top right)
* Click "▶️ Build with Parameters" (left)
*`VERSION_MAJOR`, `VERSION_MINOR`, and `VERSION_BUILD` should reflect the values in your Makefile
* For `ISO_SHA256`, run: `gsutil cat gs://minikube/iso/minikube-v<version>.iso.sha256`
* Click *Build*
* navigate to the minikube "Release" jenkins job
* Ensure that you are logged in (top right)
* Click "▶️ Build with Parameters" (left)
*`VERSION_MAJOR`, `VERSION_MINOR`, and `VERSION_BUILD` should reflect the values in your Makefile
* For `ISO_SHA256`, run: `gsutil cat gs://minikube/iso/minikube-v<version>.iso.sha256`
* Click *Build*
## Check releases.json
...
...
@@ -95,8 +95,8 @@ These are downstream packages that are being maintained by others and how to upg
| Package Manager | URL | TODO |
| --- | --- | --- |
| Arch Linux AUR | https://aur.archlinux.org/packages/minikube/ | "Flag as package out-of-date"
| Brew Cask | https://github.com/Homebrew/homebrew-cask/blob/master/Casks/minikube.rb | The release job creates a new PR in [Homebrew/homebrew-cask](https://github.com/Homebrew/homebrew-cask) with an updated version and SHA256, double check that it's created.
| Arch Linux AUR | <https://aur.archlinux.org/packages/minikube/> | "Flag as package out-of-date"
| Brew Cask | <https://github.com/Homebrew/homebrew-cask/blob/master/Casks/minikube.rb> | The release job creates a new PR in [Homebrew/homebrew-cask](https://github.com/Homebrew/homebrew-cask) with an updated version and SHA256, double check that it's created.
## Verification
...
...
@@ -104,7 +104,7 @@ Verify release checksums by running`make check-release`
## Update docs
If there are major changes, please send a PR to update https://kubernetes.io/docs/setup/minikube/
If there are major changes, please send a PR to update <https://kubernetes.io/docs/setup/minikube/>
To debug issues with minikube (not *Kubernetes* but **minikube** itself), you can use the `-v` flag to see debug level info. The specified values for `-v` will do the following (the values are all encompassing in that higher values will give you all lower value outputs as well):
*`--v=0` will output **INFO** level logs
*`--v=1` will output **WARNING** level logs
*`--v=2` will output **ERROR** level logs
...
...
@@ -11,6 +13,5 @@ Example:
If you need to access additional tools for debugging, minikube also includes the [CoreOS toolbox](https://github.com/coreos/toolbox)
You can ssh into the toolbox and access these additional commands using:
NOTE: Ubuntu users on a release older than 18.04, or anyone experiencing [#3206: Error creating new host: dial tcp: missing address.](https://github.com/kubernetes/minikube/issues/3206) you will need to build your own driver until [#3689](https://github.com/kubernetes/minikube/issues/3689) is resolved. Building this binary will require [Go v1.11](https://golang.org/dl/) or newer to be installed.
NOTE: Ubuntu users on a release older than 18.04, or anyone experiencing [#3206: Error creating new host: dial tcp: missing address.](https://github.com/kubernetes/minikube/issues/3206) you will need to build your own driver until [#3689](https://github.com/kubernetes/minikube/issues/3689) is resolved. Building this binary will require [Go v1.11](https://golang.org/dl/) or newer to be installed.
```shell
sudo apt install libvirt-dev
...
...
@@ -90,18 +89,17 @@ and run minikube as usual:
minikube start
```
#### Hyperkit driver
## Hyperkit driver
The Hyperkit driver will eventually replace the existing xhyve driver.
It is built from the minikube source tree, and uses [moby/hyperkit](http://github.com/moby/hyperkit) as a Go library.
To install the hyperkit driver via brew:
```shell
brew install docker-machine-driver-hyperkit
# docker-machine-driver-hyperkit need root owner and uid
# docker-machine-driver-hyperkit need root owner and uid
Hyper-v users may need to create a new external network switch as described [here](https://docs.docker.com/machine/drivers/hyper-v/). This step may prevent a problem in which `minikube start` hangs indefinitely, unable to ssh into the minikube virtual machine. In this add, add the `--hyperv-virtual-switch=switch-name` argument to the `minikube start` command.
The VMware unified driver will eventually replace the existing vmwarefusion driver.
The new unified driver supports both VMware Fusion (on macOS) and VMware Workstation (on Linux and Windows)
To install the vmware unified driver, head over at https://github.com/machine-drivers/docker-machine-driver-vmware/releases and download the release for your operating system.
To install the vmware unified driver, head over at <https://github.com/machine-drivers/docker-machine-driver-vmware/releases> and download the release for your operating system.
minikube supports passing environment variables instead of flags for every value listed in `minikube config list`. This is done by passing an environment variable with the prefix `MINIKUBE_`.
minikube supports passing environment variables instead of flags for every value listed in `minikube config list`. This is done by passing an environment variable with the prefix `MINIKUBE_`.
For example the `minikube start --iso-url="$ISO_URL"` flag can also be set by setting the `MINIKUBE_ISO_URL="$ISO_URL"` environment variable.
`minikube mount /path/to/dir/to/mount:/vm-mount-path` is the recommended way to mount directories into minikube so that they can be used in your local Kubernetes cluster. The command works on all supported platforms. Below is an example workflow for using `minikube mount`:
minikube requires access to the internet via HTTP, HTTPS, and DNS protocols. If a HTTP proxy is required to access the internet, you may need to pass the proxy connection information to both minikube and Docker using environment variables:
...
...
@@ -17,7 +17,7 @@ One important note: If NO_PROXY is required by non-Kubernetes applications, such
Minikube allows users to configure the docker engine's `--insecure-registry` flag. You can use the `--insecure-registry` flag on the
`minikube start` command to enable insecure communication between the docker engine and registries listening to requests from the CIDR range.
...
...
@@ -8,7 +8,9 @@ with TLS certificates. Because the default service cluster IP is known to be ava
deployed inside the cluster by creating the cluster with `minikube start --insecure-registry "10.0.0.0/24"`.
## Private Container Registries
**GCR/ECR/Docker**: Minikube has an addon, `registry-creds` which maps credentials into Minikube to support pulling from Google Container Registry (GCR), Amazon's EC2 Container Registry (ECR), and Private Docker registries. You will need to run `minikube addons configure registry-creds` and `minikube addons enable registry-creds` to get up and running. An example of this is below:
```shell
$ minikube addons configure registry-creds
Do you want to enable AWS Elastic Container Registry? [y/n]: n
The minikube VM is exposed to the host system via a host-only IP address, that can be obtained with the `minikube ip` command.
Any services of type `NodePort` can be accessed over that IP address, on the NodePort.
...
...
@@ -11,52 +11,52 @@ We also have a shortcut for fetching the minikube IP and a service's `NodePort`:
`minikube service --url $SERVICE`
### LoadBalancer emulation (`minikube tunnel`)
## LoadBalancer emulation (`minikube tunnel`)
Services of type `LoadBalancer` can be exposed via the `minikube tunnel` command.
Services of type `LoadBalancer` can be exposed via the `minikube tunnel` command.
````shell
minikube tunnel
````
````
Will output:
```
```text
out/minikube tunnel
Password: *****
Status:
Status:
machine: minikube
pid: 59088
route: 10.96.0.0/12 -> 192.168.99.101
minikube: Running
services: []
errors:
errors:
minikube: no errors
router: no errors
loadbalancer emulator: no errors
````
````
Tunnel might ask you for password for creating and deleting network routes.
# Cleaning up orphaned routes
## Cleaning up orphaned routes
If the `minikube tunnel` shuts down in an unclean way, it might leave a network route around.
This case the ~/.minikube/tunnels.json file will contain an entry for that tunnel.
To cleanup orphaned routes, run:
````
This case the ~/.minikube/tunnels.json file will contain an entry for that tunnel.
To cleanup orphaned routes, run:
````shell
minikube tunnel --cleanup
````
# (Advanced) Running tunnel as root to avoid entering password multiple times
## (Advanced) Running tunnel as root to avoid entering password multiple times
`minikube tunnel` runs as a separate daemon, creates a network route on the host to the service CIDR of the cluster using the cluster's IP address as a gateway.
Adding a route requires root privileges for the user, and thus there are differences in how to run `minikube tunnel` depending on the OS.
`minikube tunnel` runs as a separate daemon, creates a network route on the host to the service CIDR of the cluster using the cluster's IP address as a gateway.
Adding a route requires root privileges for the user, and thus there are differences in how to run `minikube tunnel` depending on the OS.
Recommended way to use on Linux with KVM2 driver and MacOSX with Hyperkit driver:
Recommended way to use on Linux with KVM2 driver and MacOSX with Hyperkit driver:
`sudo -E minikube tunnel`
Using VirtualBox on Windows, Mac and Linux _both_ `minikube start` and `minikube tunnel` needs to be started from the same Administrator user session otherwise [VBoxManage can't recognize the created VM](https://forums.virtualbox.org/viewtopic.php?f=6&t=81551).
Using VirtualBox on Windows, Mac and Linux _both_ `minikube start` and `minikube tunnel` needs to be started from the same Administrator user session otherwise [VBoxManage can't recognize the created VM](https://forums.virtualbox.org/viewtopic.php?f=6&t=81551).
Minikube `kube-apiserver` can be configured to support OpenID Connect Authentication.
Read more about OpenID Connect Authentication for Kubernetes here: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
Read more about OpenID Connect Authentication for Kubernetes here: <https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens>
## Configuring the API Server
...
...
@@ -21,7 +20,7 @@ minikube start \
## Configuring kubectl
You can use the kubectl `oidc` authenticator to create a kubeconfig as shown in the Kubernetes docs: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#option-1-oidc-authenticator
You can use the kubectl `oidc` authenticator to create a kubeconfig as shown in the Kubernetes docs: <https://kubernetes.io/docs/reference/access-authn-authz/authentication/#option-1-oidc-authenticator>
`minikube start` already creates a kubeconfig that includes a `cluster`, in order to use it with your `oidc` authenticator kubeconfig, you can run:
Minikube supports [PersistentVolumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) of type `hostPath` out of the box. These PersistentVolumes are mapped to a directory inside the running Minikube instance (usually a VM, unless you use `--vm-driver=none`). For more information on how this works, read the Dynamic Provisioning section below.
### A note on mounts, persistence, and Minikube hosts
## A note on mounts, persistence, and Minikube hosts
Minikube is configured to persist files stored under the following directories, which are made in the Minikube VM (or on your localhost if running on bare metal). You may lose data from other directories on reboots.
When using a single VM of Kubernetes it's really handy to reuse the Docker daemon inside the VM; as this means you don't have to build on your host machine and push the image into a docker registry - you can just build inside the same docker daemon as minikube which speeds up local experiments.
...
...
@@ -7,7 +7,9 @@ To be able to work with the docker daemon on your mac/linux host use the docker-
```shell
eval$(minikube docker-env)
```
You should now be able to use docker on the command line on your host mac/linux machine talking to the docker daemon inside the minikube VM:
The routing table can be queried with `route print -4`
### Handling unclean shutdowns
### Handling unclean shutdowns
Unclean shutdowns of the tunnel process can result in partially executed cleanup process, leaving network routes in the routing table.
We will keep track of the routes created by each tunnel in a centralized location in the main minikube config directory.
This list serves as a registry for tunnels containing information about
Unclean shutdowns of the tunnel process can result in partially executed cleanup process, leaving network routes in the routing table.
We will keep track of the routes created by each tunnel in a centralized location in the main minikube config directory.
This list serves as a registry for tunnels containing information about
- machine profile
- process ID
- machine profile
- process ID
- and the route that was created
The cleanup command cleans the routes from both the routing table and the registry for tunnels that are not running:
```
The cleanup command cleans the routes from both the routing table and the registry for tunnels that are not running:
```shell
minikube tunnel --cleanup
```
Updating the tunnel registry and the routing table is an atomic transaction:
Updating the tunnel registry and the routing table is an atomic transaction:
- create route in the routing table + create registry entry if both are successful, otherwise rollback
- delete route in the routing table + remove registry entry if both are successful, otherwise rollback
- create route in the routing table + create registry entry if both are successful, otherwise rollback
- delete route in the routing table + remove registry entry if both are successful, otherwise rollback
*Note*: because we don't support currently real multi cluster setup (due to overlapping CIDRs), the handling of running/not-running processes is not strictly required however it is forward looking.
*Note*: because we don't support currently real multi cluster setup (due to overlapping CIDRs), the handling of running/not-running processes is not strictly required however it is forward looking.
### Handling routing table conflicts
### Handling routing table conflicts
A routing table conflict happens when a destination CIDR of the route required by the tunnel overlaps with an existing route.
Minikube tunnel will warn the user if this happens and should not create the rule.
A routing table conflict happens when a destination CIDR of the route required by the tunnel overlaps with an existing route.
Minikube tunnel will warn the user if this happens and should not create the rule.
There should not be any automated removal of conflicting routes.
*Note*: If the user removes the minikube config directory, this might leave conflicting rules in the network routing table that will have to be cleaned up manually.
*Note*: If the user removes the minikube config directory, this might leave conflicting rules in the network routing table that will have to be cleaned up manually.
## Load Balancer Controller
...
...
@@ -138,7 +138,7 @@ sleep
Note that the Minikube ClusterIP can change over time (during system reboots) and this loop should also handle reconciliation of those changes.
## Handling multiple clusters
## Handling multiple clusters
Multiple clusters are currently not supported due to our inability to specify ServiceCIDR.
Multiple clusters are currently not supported due to our inability to specify ServiceCIDR.
This causes conflicting routes having the same destination CIDR.
@@ -14,7 +14,7 @@ The `none` driver supports releases of Debian, Ubuntu, and Fedora that are less
Most continuous integration environments are already running inside a VM, and may not supported nested virtualization. The `none` driver was designed for this use case. Here is an example, that runs minikube from a non-root user, and ensures that the latest stable kubectl is installed:
