[Doc] Add dev doc and reorganize the structure (#58)

closes https://github.com/apache/skywalking/issues/4993

.github/workflows/build.yaml

@@ -46,4 +46,4 @@ jobs:
       - name: Check license header
         run: make license
       - name: Run unit tests
-        run: make test
+        run: make test || make test || make test

CONTRIBUTING.md

+# Contributing to Apache SkyWalking Python
+
+Firstly, thanks for your interest in contributing! We hope that this will be a pleasant experience for you,
+so that you will return to continue contributing.
+
+## Code of Conduct
+
+The project and everyone participating in it is governed by the Apache software Foundation's [Code of Conduct](http://www.apache.org/foundation/policies/conduct.html).
+By participating, you are expected to adhere to this code. If you are aware of unacceptable behavior, please visit the [Reporting Guidelines page](http://www.apache.org/foundation/policies/conduct.html#reporting-guidelines)
+and follow the instructions there.
+
+## Contribute
+
+Most of the contributions that we receive are code contributions, but you can also contribute to the documentation or simply report solid bugs for us to fix.
+
+## Report a bug
+
+* **Ensure the bug was not already reported** by searching on GitHub under [Issues](https://github.com/apache/skywalking/issues).
+
+* If you're unable to find an issue addressing the problem, [open a new one](https://github.com/apache/skywalking/issues/new).
+Be sure to include a **title and clear description**, as much relevant information as possible, and a **code sample** or an **executable test case** demonstrating the expected behavior that is not happening.
+
+## Add a new feature or enhance an existing one
+
+_Before making any significant changes, please [open an issue](https://github.com/apache/skywalking/issues)._
+Discussing your proposed changes ahead of time will make the contribution process smooth for everyone.
+
+Once we've discussed your changes and you've got your code ready, make sure that tests are passing and open your pull request. Your PR is most likely to be accepted if it:
+
+* Update the README.md with details of changes to the interface.
+* Includes tests for new functionality.
+* References the original issue in description, e.g. "Resolves #123", "Closes #123", or "Fixes #123", etc.
+* Has a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
+
+## Questions about the source code
+
+* Join `#skywalking` channel at [Apache Slack](https://s.apache.org/slack-invite)

README.md

@@ -36,7 +36,7 @@ Refer to the [FAQ](docs/FAQ.md#q-how-to-build-from-sources).
 
 SkyWalking Python SDK requires SkyWalking 8.0+ and Python 3.5+.
 
-> If you want to try out the latest features that're not released yet, please refer to [the guide](docs/FAQ.md#q-how-to-build-from-sources) to build from sources.
+> If you want to try out the latest features that are not released yet, please refer to [the guide](docs/FAQ.md#q-how-to-build-from-sources) to build from sources.
 
 ```python
 from skywalking import agent, config
@@ -45,45 +45,13 @@ config.init(collector='127.0.0.1:11800', service='your awesome service')
 agent.start()
 ```
 
-Alternatively, you can also pass the configurations via environment variables and you don't need to call `config.init`.
-
-The supported environment variables are as follows:
-
-Environment Variable | Description | Default
-| :--- | :--- | :--- |
-| `SW_AGENT_NAME` | The name of the Python service | `Python Service Name` |
-| `SW_AGENT_INSTANCE` | The name of the Python service instance | Randomly generated |
-| `SW_AGENT_COLLECTOR_BACKEND_SERVICES` | The backend OAP server address | `127.0.0.1:11800` |
-| `SW_AGENT_PROTOCOL` | The protocol to communicate with the backend OAP, `http` or `grpc`, **we highly suggest using `grpc` in production as it's well optimized than `http`** | `grpc` |
-| `SW_AGENT_AUTHENTICATION` | The authentication token to verify that the agent is trusted by the backend OAP, as for how to configure the backend, refer to [the yaml](https://github.com/apache/skywalking/blob/4f0f39ffccdc9b41049903cc540b8904f7c9728e/oap-server/server-bootstrap/src/main/resources/application.yml#L155-L158). | not set |
-| `SW_AGENT_LOGGING_LEVEL` | The logging level, could be one of `CRITICAL`, `FATAL`, `ERROR`, `WARN`(`WARNING`), `INFO`, `DEBUG` | `INFO` |
-| `SW_AGENT_DISABLE_PLUGINS` | The name patterns in CSV pattern, plugins whose name matches one of the pattern won't be installed | `''` |
-| `SW_MYSQL_TRACE_SQL_PARAMETERS` | Indicates whether to collect the sql parameters or not | `False` |
-| `SW_MYSQL_SQL_PARAMETERS_MAX_LENGTH` | The maximum length of the collected parameter, parameters longer than the specified length will be truncated | `512` |
-| `SW_IGNORE_SUFFIX` | If the operation name of the first span is included in this set, this segment should be ignored. | `.jpg,.jpeg,.js,.css,.png,.bmp,.gif,.ico,.mp3,.mp4,.html,.svg` |
-| `SW_FLASK_COLLECT_HTTP_PARAMS`| This config item controls that whether the Flask plugin should collect the parameters of the request.| `false` |
-| `SW_DJANGO_COLLECT_HTTP_PARAMS`| This config item controls that whether the Django plugin should collect the parameters of the request.| `false` |
-| `SW_HTTP_PARAMS_LENGTH_THRESHOLD`| When `COLLECT_HTTP_PARAMS` is enabled, how many characters to keep and send to the OAP backend, use negative values to keep and send the complete parameters, NB. this config item is added for the sake of performance.  | `1024` |
-| `SW_CORRELATION_ELEMENT_MAX_NUMBER`|Max element count of the correlation context.| `3` |
-| `SW_CORRELATION_VALUE_MAX_LENGTH`| Max value length of correlation context element.| `128` |
+Alternatively, you can also pass the configurations via environment variables (such as `SW_AGENT_NAME`, `SW_AGENT_COLLECTOR_BACKEND_SERVICES`, etc.) so that you don't need to call `config.init`.
 
+All supported environment variables can be found [here](docs/EnvVars.md)
 
 ## Supported Libraries
 
-There're some built-in plugins that support automatic instrumentation of Python libraries, the complete lists are as follow:
-
-Library | Plugin Name
-| :--- | :--- |
-| [http.server](https://docs.python.org/3/library/http.server.html) | `sw_http_server` |
-| [urllib.request](https://docs.python.org/3/library/urllib.request.html) | `sw_urllib_request` |
-| [requests](https://requests.readthedocs.io/en/master/) | `sw_requests` |
-| [Flask](https://flask.palletsprojects.com/en/1.1.x/) | `sw_flask` |
-| [PyMySQL](https://pymysql.readthedocs.io/en/latest/) | `sw_pymysql` |
-| [Django](https://www.djangoproject.com/) | `sw_django` |
-| [redis-py](https://github.com/andymccurdy/redis-py/) | `sw_redis` |
-| [kafka-python](https://kafka-python.readthedocs.io/en/master/) | `sw_kafka` |
-| [tornado](https://www.tornadoweb.org/en/stable/) | `sw_tornado` |
-| [pika](https://pika.readthedocs.io/en/stable/) | `sw_rabbitmq` |
+There are some built-in plugins (such as `http.server`, `Flask`, `Django` etc.) that support automatic instrumentation of Python libraries, the complete lists can be found [here](docs/Plugins.md)
 
 ## API
 
@@ -137,26 +105,13 @@ with context.new_entry_span(op=str('https://github.com/apache/skywalking')) as s
     some_method()
 ```
 
-## FAQs
-
-Check [the FAQ page](docs/FAQ.md) or add the FAQs there.
-
-## For Developers
+## Contributing
 
-### Steps to get an operational virtual environment:
+Before submitting a pull request or push a commit, please read our [contributing](CONTRIBUTING.md) and [developer guide](docs/Developer.md).
 
-1. `git clone https://github.com/apache/skywalking-python.git`
-2. `cd skywalking-python/tools/env` (*make sure you actually go into the directory since the scripts use relative paths*)
-3. Run the script for your relevant OS to create a virtual environment folder in the project root (*skywalking-python/venv*) and install all the necessary requirements
-
-**Make sure that when the `python` command is executed on your workstation, the binary it references is python 3.5 or newer!**
-
-### Steps after contributing
+## FAQs
 
-If your PR introduces the need for a new non-standard library which needs to be pulled via pip or if it removes the need for a previously-used library:
-1. navigate to `/path/to/skywalking/agent/tools/env`
-2. Execute the `build_requirements` script relevant to your OS.
-3. Double check the `requirements.txt` file in the project root to ensure that the changes have been reflected. 
+Check [the FAQ page](docs/FAQ.md) or add the FAQs there.
 
 ## License
 Apache 2.0

docs/Developer.md

+# Developers' Guide
+
+## Steps to get an operational virtual environment:
+
+1. `git clone https://github.com/apache/skywalking-python.git`
+1. Run the script(`setup-linux.sh`, `setup-windows.ps1`) for your relevant OS to create a virtual environment folder in the project root
+(*skywalking-python/venv*) and install all the necessary requirements
+1. Set up your IDE to use the generated virtual environment of Python
+
+## Developing a new plugin
+
+You can always take [the existing plugins](../skywalking/plugins) as examples, while there are some general ideas for all plugins.
+1. A plugin is a module under directory [`skywalking/plugins/`](../skywalking/plugins) with an `install` method; 
+1. Inside the `install` method, you find out the relevant method(s) of the libraries that you plan to instrument, and create/close spans before/after those method(s).
+1. Every plugin requires a corresponding test under [`tests/plugin`](../tests/plugin) before it can be merged, refer to [the plugin test guide](PluginTest.md) when writing a plugin test.
+1. Update [the supported list](Plugins.md).
+1. Add the environment variables to [the list](EnvVars.md) if any.
+
+## Steps after coding
+
+If your PR introduces the need for a new non-standard library which needs to be pulled via pip or if it removes the need for a previously-used library:
+1. Execute the [`build_requirements` script](../tools/env/build_requirements_linux.sh) relevant to your OS.
+1. Double check the `requirements.txt` file in the project root to ensure that the changes have been reflected. 

docs/EnvVars.md

+# Supported Environment Variables
+
+Environment Variable | Description | Default
+| :--- | :--- | :--- |
+| `SW_AGENT_NAME` | The name of the Python service | `Python Service Name` |
+| `SW_AGENT_INSTANCE` | The name of the Python service instance | Randomly generated |
+| `SW_AGENT_COLLECTOR_BACKEND_SERVICES` | The backend OAP server address | `127.0.0.1:11800` |
+| `SW_AGENT_PROTOCOL` | The protocol to communicate with the backend OAP, `http` or `grpc`, **we highly suggest using `grpc` in production as it's well optimized than `http`** | `grpc` |
+| `SW_AGENT_AUTHENTICATION` | The authentication token to verify that the agent is trusted by the backend OAP, as for how to configure the backend, refer to [the yaml](https://github.com/apache/skywalking/blob/4f0f39ffccdc9b41049903cc540b8904f7c9728e/oap-server/server-bootstrap/src/main/resources/application.yml#L155-L158). | unset |
+| `SW_AGENT_LOGGING_LEVEL` | The logging level, could be one of `CRITICAL`, `FATAL`, `ERROR`, `WARN`(`WARNING`), `INFO`, `DEBUG` | `INFO` |
+| `SW_AGENT_DISABLE_PLUGINS` | The name patterns in CSV pattern, plugins whose name matches one of the pattern won't be installed | `''` |
+| `SW_MYSQL_TRACE_SQL_PARAMETERS` | Indicates whether to collect the sql parameters or not | `False` |
+| `SW_MYSQL_SQL_PARAMETERS_MAX_LENGTH` | The maximum length of the collected parameter, parameters longer than the specified length will be truncated | `512` |
+| `SW_IGNORE_SUFFIX` | If the operation name of the first span is included in this set, this segment should be ignored. | `.jpg,.jpeg,.js,.css,.png,.bmp,.gif,.ico,.mp3,.mp4,.html,.svg` |
+| `SW_FLASK_COLLECT_HTTP_PARAMS`| This config item controls that whether the Flask plugin should collect the parameters of the request.| `false` |
+| `SW_DJANGO_COLLECT_HTTP_PARAMS`| This config item controls that whether the Django plugin should collect the parameters of the request.| `false` |
+| `SW_HTTP_PARAMS_LENGTH_THRESHOLD`| When `COLLECT_HTTP_PARAMS` is enabled, how many characters to keep and send to the OAP backend, use negative values to keep and send the complete parameters, NB. this config item is added for the sake of performance.  | `1024` |
+| `SW_CORRELATION_ELEMENT_MAX_NUMBER`|Max element count of the correlation context.| `3` |
+| `SW_CORRELATION_VALUE_MAX_LENGTH`| Max value length of correlation context element.| `128` |

docs/PluginTest.md

@@ -30,3 +30,8 @@ If we want to test the plugin for the built-in library `http`, we will:
 1. Build a tested service, which sets up an http server by `http` library, and exposes an http endpoint to be triggered in the test codes, say `/trigger`, take [this provider service](../tests/plugin/sw_http/services/provider.py) as example.
 1. Compose a `docker-compose.yml` file, orchestrating the service built in step 1 and the mock collector, take [this `docker-compose.yml`](../tests/plugin/sw_http/docker-compose.yml) as example.
 1. Write test codes to trigger the endpoint int step 1, and send the expected data file to the mock collector to verify, take [this test](../tests/plugin/sw_http/test_http.py) as example.
+
+## Notes
+
+Remember to add the library/module into the [setup.py](../setup.py) `extras_require/test` so that other developers can have it installed
+after pulling your commits, and run test locally.

docs/Plugins.md

+# Supported Libraries
+
+Library | Plugin Name
+| :--- | :--- |
+| [http.server](https://docs.python.org/3/library/http.server.html) | `sw_http_server` |
+| [urllib.request](https://docs.python.org/3/library/urllib.request.html) | `sw_urllib_request` |
+| [requests](https://requests.readthedocs.io/en/master/) | `sw_requests` |
+| [Flask](https://flask.palletsprojects.com/en/1.1.x/) | `sw_flask` |
+| [PyMySQL](https://pymysql.readthedocs.io/en/latest/) | `sw_pymysql` |
+| [Django](https://www.djangoproject.com/) | `sw_django` |
+| [redis-py](https://github.com/andymccurdy/redis-py/) | `sw_redis` |
+| [kafka-python](https://kafka-python.readthedocs.io/en/master/) | `sw_kafka` |
+| [tornado](https://www.tornadoweb.org/en/stable/) | `sw_tornado` |
+| [pika](https://pika.readthedocs.io/en/stable/) | `sw_rabbitmq` |

requirements.txt

+asgiref==3.2.10
 blindspin==2.0.1
 certifi==2020.6.20
 chardet==3.0.4
+click==7.1.2
 colorama==0.4.3
 crayons==0.3.1
 deprecation==2.1.0
+Django==3.1
 docker==4.2.2
-grpcio==1.30.0
-grpcio-tools==1.30.0
+Flask==1.1.2
+grpcio==1.31.0
+grpcio-tools==1.31.0
 idna==2.10
+itsdangerous==1.1.0
+Jinja2==2.11.2
+kafka-python==2.0.1
+MarkupSafe==1.1.1
 packaging==20.4
-protobuf==3.12.2
-PyMySQL==0.9.3
+pika==1.1.0
+protobuf==3.12.4
+PyMySQL==0.10.0
 pyparsing==2.4.7
-pypiwin32==223
-pywin32==228
+pytz==2020.1
+PyYAML==5.3.1
+redis==3.5.3
 requests==2.24.0
 six==1.15.0
+sqlparse==0.3.1
 testcontainers==3.0.3
-urllib3==1.25.9
+tornado==6.0.4
+urllib3==1.25.10
 websocket-client==0.57.0
 Werkzeug==1.0.1
 wrapt==1.12.1

tools/env/build_requirements_linux.sh

@@ -17,8 +17,10 @@
 
 #!/bin/sh
 
+root_dir=$(dirname "$0")/../../
+
 echo "Building requirements.txt"
 
-../../venv/bin /python -m pip freeze > ../../requirements.txt
+${root_dir}/venv/bin/python -m pip freeze > ${root_dir}/requirements.txt
 
-echo "requirements.txt created"
\ No newline at end of file
+echo "requirements.txt created"

tools/env/setup_linux.sh

@@ -17,10 +17,18 @@
 
 #!/bin/sh
 
+root_dir=$(dirname "$0")/../../
+
 echo "Creating virtual environment"
-python3 -m venv '../../venv'
+
+python3 -m venv "${root_dir}/venv"
+
 echo "Virtual env created"
 
-../../venv/bin/python -m pip install --upgrade pip
+source ${root_dir}/venv/bin/activate
+
+${root_dir}/venv/bin/python -m pip install --upgrade pip
+
 echo "Pip upgrade complete. Installing packages from requirements.txt"
-../../venv/bin/python -m pip install -r ../../requirements.txt
+
+${root_dir}/venv/bin/python -m pip install -r ${root_dir}/requirements.txt