Skip to content

N
Nightingale

jobily / Nightingale

通知 1
Star 0
Fork 0
N
Nightingale

体验新版 GitCode，发现更多精彩内容 >>

未验证 提交 d45ea025 编写于 作者: Y yubo 提交者： GitHub
浏览文件
操作

Rdb (#479) 

* use collector interface

* mysql can work fine

* add basecollector

* add prober & monapi.plugins

* enable mysql plugins work

* rename collector -> manager

* add white list access check for rdb

* add cache module for authConfig & session

* rollback n9e_rdb_3.3.0.sql

* add sql ddl document

* add white_list, pwd, login access control

* add email code for login & reset password

* use sessionUsername instead of cookieUsername

* remove cookie name and data from session

* rename userName to username

* add remote_addr with session connection

* add get user by sid with cache

* enable cookie life time could be zero

* go mod tidy

* Rdb with session & monapi with telegraf (#456)

* use collector interface

* mysql can work fine

* add basecollector

* add prober & monapi.plugins

* enable mysql plugins work

* rename collector -> manager

* add white list access check for rdb

* add cache module for authConfig & session

* rollback n9e_rdb_3.3.0.sql

* add sql ddl document

* add white_list, pwd, login access control

* add email code for login & reset password

* use sessionUsername instead of cookieUsername

* remove cookie name and data from session

* rename userName to username

* add remote_addr with session connection

* add get user by sid with cache

* enable cookie life time could be zero

* go mod tidy

* add plugins config for prober

* add prober plugin expression parse

* update transfer default config for m3

* Rdb (#458)

* bugfix: session gc

* use flag for pwdMustInclude

* change user login function

* delete invite token after use

* bugfix: login response

* add sessionStart middle ware

* add auth module

* add i18n for rdb

* add i18n.zh for rdb.auth

* add mon plugins(redis, mongodb)

* update config

* add sub struct into definitions

* clean up sid cache after session destory

* bugfix: get user return nil when not found

* update i18n

* bugfix: ignore cache nologin user

* add user for callback output

* add password change api

* update default configfile & sql patch

* merge mon http middleware from rdb

* remove sso logout, sso already supporte one time auth
上级 bad43090
展开全部 隐藏空白更改
内联 并排
Showing with 8288 addition and 1469 deletion
.gitignore
浏览文件 @ d45ea025
......@@ -31,6 +31,7 @@ _test
/build
/dist
/etc/*.local.yml
/etc/plugins/*.local.yml
/etc/log/log.test.json
/data*
/tarball
......
docs/account-access.md 0 → 100644
浏览文件 @ d45ea025
## 登陆相关
#### 来源地址限制
IP地址的获取顺序
- http header "X-Forwarded-For"
- http header "X-Real-Ip"
- http request RemoteAddr
nginx 代理配置客户端地址
```
# https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
```
etc/address.yml
浏览文件 @ d45ea025
......@@ -42,5 +42,11 @@ judge:
addresses:
- 127.0.0.1
prober:
http: 0.0.0.0:8023
rpc: 0.0.0.0:8024
addresses:
- 127.0.0.1
agent:
http: 0.0.0.0:2080
etc/ams.yml
浏览文件 @ d45ea025
......@@ -6,7 +6,7 @@ logger:
http:
mode: release
cookieDomain: ""
cookieName: ecmc-user
cookieName: ecmc-sid
tokens:
- ams-builtin-token
etc/dict.json
浏览文件 @ d45ea025
......@@ -57,6 +57,101 @@
"node is managed by other system": "租户正在被系统系统使用",
"resources not found by %s": "通过 %s 没有找到资源",
"cannot delete root user": "root用户不能删除",
"user not found": "用户未找到"
"user not found": "用户未找到",
"Databases": "数据库",
"if the list is empty, then metrics are gathered from all database tables": "如果列表为空，则收集所有数据库表",
"Process List": "进程列表",
"gather thread state counts from INFORMATION_SCHEMA.PROCESSLIST": "从 INFORMATION_SCHEMA.PROCESSLIST 收集线程状态信息",
"User Statistics": "User Statistics",
"gather user statistics from INFORMATION_SCHEMA.USER_STATISTICS": "从 INFORMATION_SCHEMA.USER_STATISTICS 收集用户状态信息",
"Auto Increment": "Auto Increment",
"gather auto_increment columns and max values from information schema": "采集 auto_increment 和 max values 信息",
"Innodb Metrics": "Innodb Metrics",
"gather metrics from INFORMATION_SCHEMA.INNODB_METRICS": "采集 INFORMATION_SCHEMA.INNODB_METRICS 信息",
"Slave Status": "Slave Status",
"gather metrics from SHOW SLAVE STATUS command output": "采集 metrics from SHOW SLAVE STATUS command output",
"Binary Logs": "Binary Logs",
"gather metrics from SHOW BINARY LOGS command output": "采集 metrics from SHOW BINARY LOGS command output",
"Table IO Waits": "Table IO Waits",
"gather metrics from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMARY_BY_TABLE": "采集 from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMARY_BY_TABLE",
"Table Lock Waits": "Table Lock Waits",
"gather metrics from PERFORMANCE_SCHEMA.TABLE_LOCK_WAITS": "采集 from PERFORMANCE_SCHEMA.TABLE_LOCK_WAITS",
"Index IO Waits": "Index IO Waits",
"gather metrics from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMARY_BY_INDEX_USAGE": "采集 from PERFORMANCE_SCHEMA.TABLE_IO_WAITS_SUMMARY_BY_INDEX_USAGE",
"Event Waits": "Event Waits",
"gather metrics from PERFORMANCE_SCHEMA.EVENT_WAITS": "采集 from PERFORMANCE_SCHEMA.EVENT_WAITS",
"Tables": "Tables",
"gather metrics from INFORMATION_SCHEMA.TABLES for databases provided above list": "采集 from INFORMATION_SCHEMA.TABLES for databases provided above list",
"File Events Stats": "File Events Stats",
"gather metrics from PERFORMANCE_SCHEMA.FILE_SUMMARY_BY_EVENT_NAME": "采集 from PERFORMANCE_SCHEMA.FILE_SUMMARY_BY_EVENT_NAME",
"Perf Events Statements": "Perf Events Statements",
"gather metrics from PERFORMANCE_SCHEMA.EVENTS_STATEMENTS_SUMMARY_BY_DIGEST": "采集 from PERFORMANCE_SCHEMA.EVENTS_STATEMENTS_SUMMARY_BY_DIGEST",
"Interval Slow": "Interval Slow",
"Repositories": "Repositories",
"List of repositories to monitor": "List of repositories to monitor",
"Access token": "Access token",
"Github API access token. Unauthenticated requests are limited to 60 per hour": "Github API access token. Unauthenticated requests are limited to 60 per hour",
"Enterprise base url": "Enterprise base url",
"Github API enterprise url. Github Enterprise accounts must specify their base url": "Github API enterprise url. Github Enterprise accounts must specify their base url",
"HTTP timeout": "HTTP timeout",
"Timeout for HTTP requests": "Timeout for HTTP requests",
"Unable to get captcha": "无法获得验证码",
"Invalid captcha answer": "错误的验证码",
"Username %s is invalid": "用户名 %s 不符合规范",
"Username %s too long > 64": "用户名 %s 太长(64)",
"Unable to get login arguments": "无法获得登陆参数",
"Deny Access from %s with whitelist control": "来自 %s 的访问被白名单规则拒绝",
"Invalid login type %s": "不支持的登陆类型 %s",
"Unable to get type, sms-code | email-code": "无法获得验证码类型",
"Unable to get code arg": "无法获得验证码类型",
"sms/email sender is disabled": "无法发送 短信/邮件 验证码",
"Invalid code type %s": "不支持的验证码类型 %s",
"Cannot find the user by %s": "无法用 %s 找到相关用户",
"Unable to get password": "无法获取密码",
"Invalid code": "不符合规范的验证码",
"The code is incorrect": "无效的验证码",
"The code has expired": "失效的验证码",
"Invalid arguments %s": "不合法的参数 %s",
"Login fail, check your username and password": "登陆失败，请检查用户名/密码",
"User dose not exist": "用户不存在",
"Username %s already exists": "用户名 %s 已存在",
"Upper char": "大写字母",
"Lower char": "小写字母",
"Number": "数字",
"Special char": "特殊字符",
"Must include %s": "必须包含 %s",
"Invalid Password, %s": "密码不符合规范, %s",
"character: %s not supported": "不支持的字符 %s",
"Incorrect login/password %s times, you still have %s chances": "登陆失败%d次，你还有%d次机会",
"The limited sessions %d": "会话数量限制，最多%d个会话",
"Password has been expired": "密码已过期，请重置密码",
"User is inactive": "用户已禁用",
"User is locked": "用户已锁定",
"User is frozen": "用户已休眠",
"User is writen off": "用户已注销",
"Minimum password length %d": "密码最小长度 %d",
"Password too short (min:%d) %s": "密码太短 (最小 %d) %s",
"%s format error":"%s 所填内容不符合规范",
"%s %s format error":"%s %s 所填内容不符合规范",
"username too long (max:%d)": "用户名太长 (最长:%d)",
"dispname too long (max:%d)": "昵称太长 (最长:%d)",
"email %s or phone %s is exists": "邮箱 %s 或者 手机号 %s 已存在",
"Password is not set": "密码未设置",
"Incorrect old password": "密码错误",
"The password is the same as the old password": "密码与历史密码重复",
"phone": "手机号",
"email": "邮箱",
"username": "用户名",
"dispname": "昵称",
"Temporary user has expired": "临时账户,已过有效期",
"Invalid user status %d": "异常的用户状态 %d",
"Password expired, please change the password in time": "密码过期，请及时修改密码",
"First Login, please change the password in time": "初始登陆，请及时修改密码",
"EOF": ""
}
}
\ No newline at end of file
}
etc/identity.yml
浏览文件 @ d45ea025
......@@ -3,7 +3,7 @@ ip:
specify: ""
shell: ifconfig `route|grep '^default'|awk '{print $NF}'`|grep inet|awk '{print $2}'|head -n 1|awk -F':' '{print $NF}'
# MON、JOB的客户端拿来做本机标识
# MON、JOB, judge, prober 的客户端拿来做本机标识
ident:
specify: ""
shell: ifconfig `route|grep '^default'|awk '{print $NF}'`|grep inet|awk '{print $2}'|head -n 1|awk -F':' '{print $NF}'
\ No newline at end of file
shell: ifconfig `route|grep '^default'|awk '{print $NF}'`|grep inet|awk '{print $2}'|head -n 1|awk -F':' '{print $NF}'
etc/job.yml
浏览文件 @ d45ea025
......@@ -6,7 +6,7 @@ logger:
http:
mode: release
cookieDomain: ""
cookieName: ecmc-user
cookieName: ecmc-sid
output:
# database | remote
......
etc/login-code-email.tpl
浏览文件 @ d45ea025
您好，您的登录验证码为 {{.Code}} 。
您好，您的验证码为 {{.Code}} 。
etc/login-code-sms.tpl
浏览文件 @ d45ea025
您好，您的登录验证码为 {{.Code}} 。
您好，您的验证码为 {{.Code}} 。
etc/monapi.yml
浏览文件 @ d45ea025
......@@ -25,6 +25,8 @@ redis:
# conn: 500
# read: 3000
# write: 3000
i18n:
lang: zh
notify:
p1: ["voice", "sms", "mail", "im"]
......@@ -36,3 +38,8 @@ link:
stra: http://n9e.com/mon/strategy/%v
event: http://n9e.com/mon/history/his/%v
claim: http://n9e.com/mon/history/cur/%v
http:
mode: release
cookieDomain: ""
cookieName: ecmc-sid
etc/plugins/mysql.yml 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
etc/prober.yml 0 → 100644
浏览文件 @ d45ea025
region: default
workerProcesses: 5
logger:
dir: logs/prober
level: DEBUG
keepHours: 24
pluginsConfig: etc/plugins
ignoreConfig: false
etc/rdb.yml
浏览文件 @ d45ea025
......@@ -5,8 +5,15 @@ logger:
http:
mode: release
cookieDomain: ""
cookieName: ecmc-user
session:
cookieName: ecmc-sid
domain: ""
httpOnly: true
gcInterval: 60
cookieLifetime: 86400 # 单位秒，0: 与浏览器相同
i18n:
lang: zh
sso:
enable: false
......@@ -23,7 +30,13 @@ sso:
coverAttributes: false
stateExpiresIn: 300
captcha: false
auth:
captcha: false
extraMode:
enable: false # enable whiteList, login retry lock, userControl, ...
whiteList: false
frozenDays: 90 # frozen time (day)
writenOffDays: 365 # writenOff time (day)
tokens:
- rdb-builtin-token
......
go.mod
浏览文件 @ d45ea025
......@@ -3,28 +3,28 @@ module github.com/didi/nightingale
go 1.12
require (
github.com/Shopify/sarama v1.19.0
github.com/Shopify/sarama v1.27.1
github.com/cespare/xxhash v1.1.0
github.com/codegangsta/negroni v1.0.0
github.com/coreos/go-oidc v2.2.1+incompatible
github.com/dgryski/go-tsz v0.0.0-20180227144327-03b7d791f4fe
github.com/eapache/go-resiliency v1.2.0 // indirect
github.com/garyburd/redigo v1.6.2
github.com/gin-contrib/pprof v1.3.0
github.com/gin-gonic/gin v1.6.3
github.com/go-sql-driver/mysql v1.5.0
github.com/google/go-github/v32 v32.1.0
github.com/google/uuid v1.1.2-0.20190416172445-c2e93f3ae59f
github.com/gorilla/mux v1.7.3
github.com/hashicorp/golang-lru v0.5.4
github.com/hpcloud/tail v1.0.0
github.com/influxdata/influxdb v1.8.0
github.com/influxdata/telegraf v1.16.2
github.com/m3db/m3 v0.15.17
github.com/mattn/go-isatty v0.0.12
github.com/mattn/go-sqlite3 v1.14.0 // indirect
github.com/mojocn/base64Captcha v1.3.1
github.com/open-falcon/rrdlite v0.0.0-20200214140804-bf5829f786ad
github.com/pquerna/cachecontrol v0.0.0-20200819021114-67c6ae64274f // indirect
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0 // indirect
github.com/robfig/go-cache v0.0.0-20130306151617-9fc39e0dbf62 // indirect
github.com/shirou/gopsutil v3.20.11+incompatible
github.com/spaolacci/murmur3 v1.1.0
......@@ -35,10 +35,8 @@ require (
github.com/ugorji/go/codec v1.1.7
github.com/unrolled/render v1.0.3
go.uber.org/automaxprocs v1.3.0 // indirect
golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de // indirect
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
golang.org/x/text v0.3.3
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.0.1 // indirect
google.golang.org/protobuf v1.25.0 // indirect
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df
......
go.sum
浏览文件 @ d45ea025
此差异已折叠。
sql/README.md 0 → 100644
浏览文件 @ d45ea025
## sql 的维护
- n9e_{module}.sql 完整的sql
- n9e_{module}-path.sql 增量的sql
## sql 的版本发布
在使用 git tag 之前，将特定版本的增量文件固化下来
```
module=rdb
version=v3.3.3
cat n9e_${module}-patch.sql > upgrade/n9e_${module}-${version}.sql
echo > n9e_{module}-patch.sql
# 然后提交更改后，再打上版本的tag
git add .
git commit -a -m "${version} release"
git tag ${version}
git push
```
sql/n9e_hbs.sql
浏览文件 @ d45ea025
......@@ -10,7 +10,8 @@ create table `instance` (
`identity` varchar(255) not null,
`rpc_port` varchar(16) not null,
`http_port` varchar(16) not null,
`remark` text,
`region` varchar(32) not null,
`remark` text,
`ts` int unsigned not null,
primary key (`id`),
key(`module`,`identity`,`rpc_port`,`http_port`)
......@@ -26,4 +27,4 @@ create table `detector` (
`ts` int unsigned not null,
primary key (`id`),
key(`ip`,`port`)
) engine=innodb default charset=utf8;
\ No newline at end of file
) engine=innodb default charset=utf8;
sql/n9e_mon-patch.sql 0 → 100644
浏览文件 @ d45ea025
set names utf8;
use n9e_mon;
CREATE TABLE `collect_rule` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
`nid` bigint(20) unsigned NOT NULL DEFAULT '0' COMMENT 'nid',
`step` int(11) NOT NULL DEFAULT '0' COMMENT 'step',
`timeout` int(11) NOT NULL DEFAULT '0' COMMENT 'total timeout',
`collect_type` varchar(64) NOT NULL DEFAULT '' COMMENT 'collector name',
`name` varchar(255) NOT NULL DEFAULT '' COMMENT 'name',
`region` varchar(32) NOT NULL DEFAULT 'default' COMMENT 'region',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`data` blob NULL COMMENT 'data',
`tags` varchar(512) NOT NULL DEFAULT '' COMMENT 'tags',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`created` datetime NOT NULL COMMENT 'created',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
KEY `idx_collect_type` (`collect_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT 'api collect';
sql/n9e_mon.sql
浏览文件 @ d45ea025
......@@ -11,7 +11,7 @@ create table `maskconf` (
`metric` varchar(255) not null,
`tags` varchar(255) not null default '',
`cause` varchar(255) not null default '',
`user` varchar(32) not null default 'operate user',
`user` varchar(64) not null default 'operate user',
`btime` bigint not null default 0 comment 'begin time',
`etime` bigint not null default 0 comment 'end time',
primary key (`id`),
......@@ -176,7 +176,7 @@ CREATE TABLE `stra_log` (
`sid` bigint(20) NOT NULL DEFAULT '0' COMMENT 'collect id',
`action` varchar(255) NOT NULL DEFAULT '' COMMENT '动作 update, delete',
`body` text COMMENT '修改之前采集的内容',
`creator` varchar(255) NOT NULL DEFAULT '' COMMENT 'creator',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`created` timestamp NOT NULL DEFAULT '1971-01-01 00:00:00' COMMENT 'created',
PRIMARY KEY (`id`),
KEY `idx_sid` (`sid`)
......@@ -192,9 +192,9 @@ CREATE TABLE `port_collect` (
`step` int(11) NOT NULL DEFAULT '0' COMMENT '采集周期',
`timeout` int(11) NOT NULL DEFAULT '0' COMMENT 'connect time',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`creator` varchar(255) NOT NULL DEFAULT '' COMMENT 'creator',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`created` datetime NOT NULL COMMENT 'created',
`last_updator` varchar(128) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'last_updated',
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
......@@ -211,9 +211,9 @@ CREATE TABLE `proc_collect` (
`target` varchar(255) NOT NULL DEFAULT '' COMMENT '采集对象',
`step` int(11) NOT NULL DEFAULT '0' COMMENT '采集周期',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`creator` varchar(255) NOT NULL DEFAULT '' COMMENT 'creator',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`created` datetime NOT NULL COMMENT 'created',
`last_updator` varchar(128) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
......@@ -237,9 +237,9 @@ CREATE TABLE `log_collect` (
`unit` varchar(64) NOT NULL DEFAULT '' COMMENT 'unit',
`zero_fill` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'zero fill',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`creator` varchar(255) NOT NULL DEFAULT '' COMMENT 'creator',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`created` datetime NOT NULL COMMENT 'created',
`last_updator` varchar(128) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
......@@ -257,9 +257,9 @@ CREATE TABLE `plugin_collect` (
`stdin` text NOT NULL COMMENT 'stdin',
`env` text NOT NULL COMMENT 'env',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`creator` varchar(255) NOT NULL DEFAULT '' COMMENT 'creator',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`created` datetime NOT NULL COMMENT 'created',
`last_updator` varchar(128) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
......@@ -286,9 +286,9 @@ CREATE TABLE `api_collect` (
`unexpected_string` varchar(255) NOT NULL DEFAULT '' COMMENT 'unexpected_string',
`region` varchar(32) NOT NULL DEFAULT 'default' COMMENT 'region',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`creator` varchar(255) NOT NULL DEFAULT '' COMMENT 'creator',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`created` datetime NOT NULL COMMENT 'created',
`last_updator` varchar(128) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
......@@ -309,9 +309,9 @@ CREATE TABLE `snmp_collect` (
`step` int(11) NOT NULL DEFAULT '0' COMMENT 'step',
`timeout` int(11) NOT NULL DEFAULT '0' COMMENT 'total timeout',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`creator` varchar(255) NOT NULL DEFAULT '' COMMENT 'creator',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`created` datetime NOT NULL COMMENT 'created',
`last_updator` varchar(128) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
......@@ -319,6 +319,26 @@ CREATE TABLE `snmp_collect` (
KEY `idx_collect_type` (`collect_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT 'api collect';
CREATE TABLE `collect_rule` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
`nid` bigint(20) unsigned NOT NULL DEFAULT '0' COMMENT 'nid',
`step` int(11) NOT NULL DEFAULT '0' COMMENT 'step',
`timeout` int(11) NOT NULL DEFAULT '0' COMMENT 'total timeout',
`collect_type` varchar(64) NOT NULL DEFAULT '' COMMENT 'collector name',
`name` varchar(255) NOT NULL DEFAULT '' COMMENT 'name',
`region` varchar(32) NOT NULL DEFAULT 'default' COMMENT 'region',
`comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'comment',
`data` blob NULL COMMENT 'data',
`tags` varchar(512) NOT NULL DEFAULT '' COMMENT 'tags',
`creator` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
`last_updator` varchar(64) NOT NULL DEFAULT '' COMMENT 'last_updator',
`created` datetime NOT NULL COMMENT 'created',
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_nid` (`nid`),
KEY `idx_collect_type` (`collect_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT 'api collect';
CREATE TABLE `aggr_calc` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
`nid` bigint(20) unsigned NOT NULL DEFAULT '0' COMMENT 'nid',
......@@ -389,12 +409,3 @@ CREATE TABLE `collect_hist` (
PRIMARY KEY (`id`),
KEY `idx_cid` (`cid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT 'hist';
CREATE TABLE `api_collect_sid` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
`sid` bigint(20) NOT NULL DEFAULT '0' COMMENT 'stra id',
`cid` bigint(20) NOT NULL DEFAULT '0' COMMENT 'collect id',
PRIMARY KEY (`id`),
KEY (`sid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
sql/n9e_rdb-patch.sql 0 → 100644
浏览文件 @ d45ea025
set names utf8;
use n9e_rdb;
CREATE TABLE `white_list` (
`id` bigint unsigned not null AUTO_INCREMENT,
`start_ip` varchar(32) DEFAULT '0' NOT NULL,
`end_ip` varchar(32) DEFAULT '0' NOT NULL,
`start_ip_int` bigint DEFAULT '0' NOT NULL,
`end_ip_int` bigint DEFAULT '0' NOT NULL,
`start_time` bigint DEFAULT '0' NOT NULL,
`end_time` bigint DEFAULT '0' NOT NULL,
`created_at` bigint DEFAULT '0' NOT NULL,
`updated_at` bigint DEFAULT '0' NOT NULL,
`creator` varchar(64) DEFAULT '' NOT NULL,
`updater` varchar(64) DEFAULT '' NOT NULL,
PRIMARY KEY (`id`),
KEY (`start_ip_int`, `end_ip_int`),
KEY (`start_time`, `end_time`),
KEY (`created_at`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8;
CREATE TABLE `session` (
`sid` char(128) NOT NULL,
`username` varchar(64) DEFAULT '',
`remote_addr` varchar(32) DEFAULT '',
`created_at` integer unsigned DEFAULT '0',
`updated_at` integer unsigned DEFAULT '0' NOT NULL,
PRIMARY KEY (`sid`),
KEY (`username`),
KEY (`updated_at`)
) ENGINE = InnoDB DEFAULT CHARACTER SET = utf8;
alter table user add `login_err_num` int unsigned not null default 0 after leader_name;
alter table user add `active_begin` bigint not null default 0 after login_err_num;
alter table user add `active_end` bigint not null default 0 after active_begin;
alter table user add `locked_at` bigint not null default 0 after active_end;
alter table user add `updated_at` bigint not null default 0 after locked_at;
alter table user add `pwd_updated_at` bigint not null default 0 after updated_at;
alter table user add `logged_at` bigint not null default 0 after pwd_updated_at;
alter table user add `passwords` varchar(512) not null default '' after password;
alter table login_log add `err` varchar(128) not null default '' after loginout;
sql/n9e_rdb.sql
浏览文件 @ d45ea025
......@@ -6,23 +6,31 @@ use n9e_rdb;
CREATE TABLE `user`
(
`id` int unsigned not null AUTO_INCREMENT,
`uuid` varchar(128) not null comment 'use in cookie',
`username` varchar(64) not null comment 'login name, cannot rename',
`password` varchar(128) not null default '',
`dispname` varchar(32) not null default '' comment 'display name, chinese name',
`phone` varchar(16) not null default '',
`email` varchar(64) not null default '',
`im` varchar(64) not null default '',
`portrait` varchar(2048) not null default '',
`intro` varchar(2048) not null default '',
`organization` varchar(255) not null default '',
`typ` tinyint(1) not null default 0 comment '0: long-term account; 1: temporary account',
`status` tinyint(1) not null default 0 comment '0: active; 1: inactive 2: disable',
`is_root` tinyint(1) not null,
`leader_id` int unsigned not null default 0,
`leader_name` varchar(32) not null default '',
`create_at` timestamp not null default CURRENT_TIMESTAMP,
`id` int unsigned not null AUTO_INCREMENT,
`uuid` varchar(128) not null comment 'use in cookie',
`username` varchar(64) not null comment 'login name, cannot rename',
`password` varchar(128) not null default '',
`passwords` varchar(512) not null default '',
`dispname` varchar(32) not null default '' comment 'display name, chinese name',
`phone` varchar(16) not null default '',
`email` varchar(64) not null default '',
`im` varchar(64) not null default '',
`portrait` varchar(2048) not null default '',
`intro` varchar(2048) not null default '',
`organization` varchar(255) not null default '',
`typ` tinyint(1) not null default 0 comment '0: long-term account; 1: temporary account',
`status` tinyint(1) not null default 0 comment '0: active, 1: inactive, 2: locked, 3: frozen, 5: writen-off',
`is_root` tinyint(1) not null,
`leader_id` int unsigned not null default 0,
`leader_name` varchar(32) not null default '',
`login_err_num` int unsigned not null default 0,
`active_begin` bigint not null default 0,
`active_end` bigint not null default 0,
`locked_at` bigint not null default 0,
`updated_at` bigint not null default 0,
`pwd_updated_at` bigint not null default 0,
`logged_at` bigint not null default 0,
`create_at` timestamp not null default CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY (`username`),
UNIQUE KEY (`uuid`)
......@@ -269,6 +277,7 @@ CREATE TABLE `login_log`
`client` varchar(128) not null comment 'client ip',
`clock` bigint not null comment 'login timestamp',
`loginout` char(3) not null comment 'in or out',
`err` varchar(128) not null comment 'err msg',
PRIMARY KEY (`id`),
KEY (`username`),
KEY (`clock`)
......@@ -315,3 +324,32 @@ CREATE TABLE `captcha` (
KEY (`captcha_id`, `answer`),
KEY (`created_at`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8;
CREATE TABLE `white_list` (
`id` bigint unsigned not null AUTO_INCREMENT,
`start_ip` varchar(32) DEFAULT '0' NOT NULL,
`end_ip` varchar(32) DEFAULT '0' NOT NULL,
`start_ip_int` bigint DEFAULT '0' NOT NULL,
`end_ip_int` bigint DEFAULT '0' NOT NULL,
`start_time` bigint DEFAULT '0' NOT NULL,
`end_time` bigint DEFAULT '0' NOT NULL,
`created_at` bigint DEFAULT '0' NOT NULL,
`updated_at` bigint DEFAULT '0' NOT NULL,
`creator` varchar(64) DEFAULT '' NOT NULL,
`updater` varchar(64) DEFAULT '' NOT NULL,
PRIMARY KEY (`id`),
KEY (`start_ip_int`, `end_ip_int`),
KEY (`start_time`, `end_time`),
KEY (`created_at`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8;
CREATE TABLE `session` (
`sid` char(128) NOT NULL,
`username` varchar(64) DEFAULT '',
`remote_addr` varchar(32) DEFAULT '',
`created_at` integer unsigned DEFAULT '0',
`updated_at` integer unsigned DEFAULT '0' NOT NULL,
PRIMARY KEY (`sid`),
KEY (`username`),
KEY (`updated_at`)
) ENGINE = InnoDB DEFAULT CHARACTER SET = utf8;
src/common/dataobj/metric.go
浏览文件 @ d45ea025
......@@ -30,9 +30,9 @@ type MetricValue struct {
Step int64 `json:"step"`
ValueUntyped interface{} `json:"value"`
Value float64 `json:"-"`
CounterType string `json:"counterType"`
Tags string `json:"tags"`
TagsMap map[string]string `json:"tagsMap"` //保留2种格式，方便后端组件使用
CounterType string `json:"counterType"` // GAUGE | COUNTER | SUBTRACT | DERIVE
Tags string `json:"tags"` // a=1,b=2,c=3
TagsMap map[string]string `json:"tagsMap"` // {"a":1, "b"=2, "c="3} 保留2种格式，方便后端组件使用
Extra string `json:"extra"`
}
......@@ -69,7 +69,7 @@ func (m *MetricValue) CheckValidity(now int64) (err error) {
}
if m.Nid == "" && m.Endpoint == "" {
err = fmt.Errorf("nid or endpoint should not be empty")
err = fmt.Errorf("nid and endpoint should not be empty")
return
}
......@@ -133,7 +133,7 @@ func (m *MetricValue) CheckValidity(now int64) (err error) {
k = filterString(k)
v = filterString(v)
if len(k) == 0 || len(v) == 0 {
err = fmt.Errorf("tag key and value should not be empty")
err = fmt.Errorf("tag key and value should not be empty key:%s value:%s", k, v)
return
}
......
src/common/dataobj/query_item.go
浏览文件 @ d45ea025
......@@ -110,4 +110,5 @@ type IndexByFullTagsResp struct {
Tags []string `json:"tags"`
Step int `json:"step"`
DsType string `json:"dstype"`
Count int `json:"count"`
}
src/models/configs.go
浏览文件 @ d45ea025
package models
import (
"encoding/json"
"fmt"
"log"
"os"
"strings"
"time"
"github.com/toolkits/pkg/runner"
......@@ -88,3 +90,103 @@ func ConfigsGets(ckeys []string) (map[string]string, error) {
return kvmap, nil
}
type AuthConfig struct {
MaxNumErr int `json:"maxNumErr"`
MaxSessionNumber int64 `json:"maxSessionNumber"`
MaxConnIdelTime int64 `json:"maxConnIdelTime" description:"minute"`
LockTime int64 `json:"lockTime" description:"minute"`
PwdHistorySize int `json:"pwdHistorySize"`
PwdMinLenght int `json:"pwdMinLenght"`
PwdExpiresIn int64 `json:"pwdExpiresIn" description:"month"`
PwdMustInclude []string `json:"pwdMustInclude" description:"upper,lower,number,specChar"`
PwdMustIncludeFlag int `json:"pwdMustIncludeFlag"`
}
func (p AuthConfig) PwdRules() []string {
s := []string{}
if p.PwdMinLenght > 0 {
s = append(s, _s("Minimum password length %d", p.PwdMinLenght))
}
if rule := p.MustInclude(); rule != "" {
s = append(s, rule)
}
return s
}
func (p AuthConfig) MustInclude() string {
s := []string{}
if p.PwdMustIncludeFlag&PWD_INCLUDE_UPPER > 0 {
s = append(s, _s("Upper char"))
}
if p.PwdMustIncludeFlag&PWD_INCLUDE_LOWER > 0 {
s = append(s, _s("Lower char"))
}
if p.PwdMustIncludeFlag&PWD_INCLUDE_NUMBER > 0 {
s = append(s, _s("Number"))
}
if p.PwdMustIncludeFlag&PWD_INCLUDE_SPEC_CHAR > 0 {
s = append(s, _s("Special char"))
}
if len(s) > 0 {
return _s("Must include %s", strings.Join(s, ","))
}
return ""
}
const (
PWD_INCLUDE_UPPER = 1 << iota
PWD_INCLUDE_LOWER
PWD_INCLUDE_NUMBER
PWD_INCLUDE_SPEC_CHAR
)
func (p *AuthConfig) Validate() error {
for _, v := range p.PwdMustInclude {
switch v {
case "upper":
p.PwdMustIncludeFlag |= PWD_INCLUDE_UPPER
case "lower":
p.PwdMustIncludeFlag |= PWD_INCLUDE_LOWER
case "number":
p.PwdMustIncludeFlag |= PWD_INCLUDE_NUMBER
case "specChar":
p.PwdMustIncludeFlag |= PWD_INCLUDE_SPEC_CHAR
default:
return fmt.Errorf("invalid pwd flags, must be 'upper', 'lower', 'number', 'specChar'")
}
}
return nil
}
var DefaultAuthConfig = AuthConfig{
MaxConnIdelTime: 30,
PwdMustInclude: []string{},
}
func AuthConfigGet() (*AuthConfig, error) {
buf, err := ConfigsGet("auth.config")
if err != nil {
return &DefaultAuthConfig, nil
}
c := &AuthConfig{}
if err := json.Unmarshal([]byte(buf), c); err != nil {
return &DefaultAuthConfig, nil
}
return c, nil
}
func AuthConfigSet(config *AuthConfig) error {
if err := config.Validate(); err != nil {
return err
}
buf, err := json.Marshal(config)
if err != nil {
return err
}
return ConfigsSet("auth.config", string(buf))
}
src/models/init.go
浏览文件 @ d45ea025
package models
import (
"fmt"
"time"
"github.com/didi/nightingale/src/toolkits/i18n"
"github.com/toolkits/pkg/cache"
)
func init() {
cache.InitMemoryCache(time.Hour)
}
func _e(format string, a ...interface{}) error {
return fmt.Errorf(i18n.Sprintf(format, a...))
}
func _s(format string, a ...interface{}) string {
return i18n.Sprintf(format, a...)
}
src/models/invite.go
浏览文件 @ d45ea025
......@@ -35,3 +35,8 @@ func InviteNew(token, creator string) error {
_, err := DB["rdb"].Insert(obj)
return err
}
func (i *Invite) Del() error {
_, err := DB["rdb"].Where("token=?", i.Token).Delete(i)
return err
}
src/models/login_log.go
浏览文件 @ d45ea025
package models
import "time"
import (
"fmt"
"time"
)
type LoginLog struct {
Id int64 `json:"id"`
......@@ -8,18 +11,20 @@ type LoginLog struct {
Client string `json:"client"`
Clock int64 `json:"clock"`
Loginout string `json:"loginout"`
Err string `json:"err"`
}
func LoginLogNew(username, client, inout string) error {
func LoginLogNew(username, client, inout string, err error) error {
now := time.Now().Unix()
obj := LoginLog{
Username: username,
Client: client,
Clock: now,
Loginout: inout,
Err: fmt.Sprintf("%v", err),
}
_, err := DB["rdb"].Insert(obj)
return err
_, e := DB["rdb"].Insert(obj)
return e
}
func LoginLogTotal(username string, btime, etime int64) (int64, error) {
......
src/models/mon_collect.go
浏览文件 @ d45ea025
......@@ -185,12 +185,6 @@ type LogCollect struct {
ParseSucc bool `xorm:"-" json:"-"`
}
type ApiCollectSid struct {
Id int64 `json:"id"`
Cid int64 `json:"cid"`
Sid int64 `json:"sid"`
}
type ApiCollect struct {
Id int64 `json:"id"`
Nid int64 `json:"nid"`
......@@ -531,58 +525,6 @@ func (a *ApiCollect) Update() error {
return err
}
func DeleteApiCollect(id int64) error {
session := DB["mon"].NewSession()
defer session.Close()
_, err := session.Where("id = ?", id).Delete(new(ApiCollect))
if err != nil {
session.Rollback()
return err
}
var relCidSid ApiCollectSid
has, err := session.Where("cid = ?", id).Get(&relCidSid)
if err != nil {
session.Rollback()
return err
}
if has {
err = StraDel(relCidSid.Sid)
if err != nil {
session.Rollback()
return err
}
}
return session.Commit()
}
func GetSidByCid(cid int64) (int64, error) {
var cidSid ApiCollectSid
_, err := DB["mon"].Where("cid = ?", cid).Get(&cidSid)
return cidSid.Sid, err
}
func (a *ApiCollectSid) Add() error {
session := DB["mon"].NewSession()
defer session.Close()
_, err := session.Where("cid = ?", a.Cid).Delete(new(ApiCollectSid))
if err != nil {
session.Rollback()
return err
}
_, err = session.Insert(a)
if err != nil {
session.Rollback()
return err
}
return session.Commit()
}
func CreateCollect(collectType, creator string, collect interface{}) error {
session := DB["mon"].NewSession()
defer session.Close()
......@@ -611,126 +553,6 @@ func CreateCollect(collectType, creator string, collect interface{}) error {
return session.Commit()
}
func GetCollectByNid(collectType string, nids []int64) ([]interface{}, error) {
var res []interface{}
switch collectType {
case "port":
collects := []PortCollect{}
err := DB["mon"].In("nid", nids).Find(&collects)
for _, c := range collects {
res = append(res, c)
}
return res, err
case "proc":
collects := []ProcCollect{}
err := DB["mon"].In("nid", nids).Find(&collects)
for _, c := range collects {
res = append(res, c)
}
return res, err
case "log":
collects := []LogCollect{}
err := DB["mon"].In("nid", nids).Find(&collects)
for _, c := range collects {
c.Decode()
res = append(res, c)
}
return res, err
case "plugin":
collects := []PluginCollect{}
err := DB["mon"].In("nid", nids).Find(&collects)
for _, c := range collects {
res = append(res, c)
}
return res, err
default:
return nil, fmt.Errorf("采集类型不合法")
}
}
func GetCollectById(collectType string, cid int64) (interface{}, error) {
switch collectType {
case "port":
collect := new(PortCollect)
has, err := DB["mon"].Where("id = ?", cid).Get(collect)
if !has {
return nil, err
}
return collect, err
case "proc":
collect := new(ProcCollect)
has, err := DB["mon"].Where("id = ?", cid).Get(collect)
if !has {
return nil, err
}
return collect, err
case "log":
collect := new(LogCollect)
has, err := DB["mon"].Where("id = ?", cid).Get(collect)
if !has {
return nil, err
}
collect.Decode()
return collect, err
case "plugin":
collect := new(PluginCollect)
has, err := DB["mon"].Where("id = ?", cid).Get(collect)
if !has {
return nil, err
}
return collect, err
default:
return nil, fmt.Errorf("采集类型不合法")
}
return nil, nil
}
func GetCollectByNameAndNid(collectType string, name string, nid int64) (interface{}, error) {
switch collectType {
case "port":
collect := new(PortCollect)
has, err := DB["mon"].Where("name = ? and nid = ?", name, nid).Get(collect)
if !has {
return nil, err
}
return collect, err
case "proc":
collect := new(ProcCollect)
has, err := DB["mon"].Where("name = ? and nid = ?", name, nid).Get(collect)
if !has {
return nil, err
}
return collect, err
case "log":
collect := new(LogCollect)
has, err := DB["mon"].Where("name = ? and nid = ?", name, nid).Get(collect)
if !has {
return nil, err
}
collect.Decode()
return collect, err
case "plugin":
collect := new(PluginCollect)
has, err := DB["mon"].Where("name = ? and nid = ?", name, nid).Get(collect)
if !has {
return nil, err
}
return collect, err
default:
return nil, fmt.Errorf("采集类型不合法")
}
return nil, nil
}
func DeleteCollectById(collectType, creator string, cid int64) error {
session := DB["mon"].NewSession()
defer session.Close()
......
src/models/mon_collect_rule.go 0 → 100644
浏览文件 @ d45ea025
package models
import (
"encoding/json"
"fmt"
"time"
"github.com/didi/nightingale/src/common/dataobj"
)
const (
defaultStep = 10
)
type CollectRule struct {
Id int64 `json:"id"`
Nid int64 `json:"nid"`
Step int `json:"step" description:"interval"`
Timeout int `json:"timeout"`
CollectType string `json:"collect_type" description:"plugin name"`
Name string `json:"name"`
Region string `json:"region"`
Comment string `json:"comment"`
Data json.RawMessage `json:"data"`
Tags string `json:"tags" description:"k1=v1,k2=v2,k3=v3,..."`
Creator string `json:"creator" description:"just for output"`
LastUpdator string `xorm:"last_updator" json:"last_updator" description:"just for output"`
Created time.Time `xorm:"updated" json:"created" description:"just for output"`
LastUpdated time.Time `xorm:"updated" json:"last_updated" description:"just for output"`
}
type validator interface {
Validate() error
}
func (p *CollectRule) Validate(v ...interface{}) error {
if p.Name == "" {
return fmt.Errorf("invalid collectRule.name")
}
if p.Step == 0 {
p.Step = defaultStep
}
if _, err := dataobj.SplitTagsString(p.Tags); err != nil {
return err
}
if len(v) > 0 && v[0] != nil {
if err := json.Unmarshal(p.Data, v[0]); err != nil {
return err
}
if o, ok := v[0].(validator); ok {
if err := o.Validate(); err != nil {
return err
}
}
}
return nil
}
func GetCollectRules() ([]*CollectRule, error) {
rules := []*CollectRule{}
err := DB["mon"].Find(&rules)
return rules, err
}
func (p *CollectRule) Update() error {
session := DB["mon"].NewSession()
defer session.Close()
err := session.Begin()
if err != nil {
return err
}
if _, err = session.Id(p.Id).AllCols().Update(p); err != nil {
session.Rollback()
return err
}
b, err := json.Marshal(p)
if err != nil {
session.Rollback()
return err
}
if err := saveHist(p.Id, p.CollectType, "update", p.Creator, string(b), session); err != nil {
session.Rollback()
return err
}
if err = session.Commit(); err != nil {
return err
}
return err
}
func DeleteCollectRule(sid int64) error {
_, err := DB["mon"].Where("id=?", sid).Delete(new(CollectRule))
return err
}
src/models/mon_hbs.go
浏览文件 @ d45ea025
......@@ -10,6 +10,7 @@ type Instance struct {
HTTPPort string `json:"http_port" xorm:"http_port"`
TS int64 `json:"ts" xorm:"ts"`
Remark string `json:"remark"`
Region string `json:"region"`
Active bool `xorm:"-" json:"active"`
}
......
src/models/session.go 0 → 100644
浏览文件 @ d45ea025
package models
import (
"fmt"
"time"
"github.com/toolkits/pkg/cache"
"github.com/toolkits/pkg/logger"
)
type Session struct {
Sid string `json:"sid"`
Username string `json:"username"`
RemoteAddr string `json:"remote_addr"`
CreatedAt int64 `json:"created_at"`
UpdatedAt int64 `json:"updated_at"`
}
func SessionAll() (int64, error) {
return DB["rdb"].Count(new(Session))
}
func SessionUserAll(username string) (int64, error) {
return DB["rdb"].Where("username=?", username).Count(new(Session))
}
func SessionGet(sid string) (*Session, error) {
var obj Session
has, err := DB["rdb"].Where("sid=?", sid).Get(&obj)
if err != nil {
return nil, fmt.Errorf("get session err %s", err)
}
if !has {
return nil, fmt.Errorf("not found")
}
return &obj, nil
}
func SessionInsert(in *Session) error {
_, err := DB["rdb"].Insert(in)
return err
}
func SessionDel(sid string) error {
_, err := DB["rdb"].Where("sid=?", sid).Delete(new(Session))
return err
}
func SessionUpdate(in *Session) error {
_, err := DB["rdb"].Where("sid=?", in.Sid).AllCols().Update(in)
return err
}
func SessionCleanup(ts int64) error {
n, err := DB["rdb"].Where("updated_at<?", ts).Delete(new(Session))
logger.Debugf("delete before updated_at %d session %d", ts, n)
return err
}
func SessionCleanupByCreatedAt(ts int64) error {
n, err := DB["rdb"].Where("created_at<?", ts).Delete(new(Session))
logger.Debugf("delete before created_at %d session %d", ts, n)
return err
}
func (s *Session) Update(cols ...string) error {
_, err := DB["rdb"].Where("id=?", s.Sid).Cols(cols...).Update(s)
return err
}
// SessionGetWithCache will update session.UpdatedAt
func SessionGetWithCache(sid string) (*Session, error) {
if sid == "" {
return nil, fmt.Errorf("unable to get sid")
}
sess := &Session{}
if err := cache.Get("sid."+sid, &sess); err == nil {
return sess, nil
}
var err error
if sess, err = SessionGet(sid); err != nil {
return nil, fmt.Errorf("session not found")
}
// update session
sess.UpdatedAt = time.Now().Unix()
sess.Update("updated_at")
if sess.Username != "" {
cache.Set("sid."+sid, sess, time.Second*30)
}
return sess, nil
}
func SessionGetUserWithCache(sid string) (*User, error) {
s, err := SessionGetWithCache(sid)
if err != nil {
return nil, err
}
if s.Username == "" {
return nil, fmt.Errorf("user not found")
}
return UserMustGet("username=?", s.Username)
}
func SessionCacheDelete(sid string) {
cache.Delete("sid." + sid)
}
src/models/tools.go
浏览文件 @ d45ea025
......@@ -10,9 +10,12 @@ import (
// CryptoPass crypto password use salt
func CryptoPass(raw string) (string, error) {
if raw == "" {
return "", _e("Password is not set")
}
salt, err := ConfigsGet("salt")
if err != nil {
return "", fmt.Errorf("query salt from mysql fail: %v", err)
return "", _e("query salt from mysql fail: %v", err)
}
return str.MD5(salt + "<-*Uk30^96eY*->" + raw), nil
......
src/models/user.go
浏览文件 @ d45ea025
......@@ -21,17 +21,30 @@ import (
const (
LOGIN_T_SMS = "sms-code"
LOGIN_T_EMAIL = "email-code"
LOGIN_T_RST = "rst-code"
LOGIN_T_PWD = "password"
LOGIN_T_LDAP = "ldap"
LOGIN_T_RST = "rst-code"
LOGIN_T_LOGIN = "login-code"
LOGIN_EXPIRES_IN = 300
)
const (
USER_S_ACTIVE = iota
USER_S_INACTIVE
USER_S_LOCKED
USER_S_FROZEN
USER_S_WRITEN_OFF
)
const (
USER_T_NATIVE = iota
USER_T_TEMP
)
type User struct {
Id int64 `json:"id"`
UUID string `json:"uuid" xorm:"'uuid'"`
Username string `json:"username"`
Password string `json:"-"`
Passwords string `json:"-"`
Dispname string `json:"dispname"`
Phone string `json:"phone"`
Email string `json:"email"`
......@@ -39,14 +52,63 @@ type User struct {
Portrait string `json:"portrait"`
Intro string `json:"intro"`
Organization string `json:"organization"`
Typ int `json:"typ"`
Status int `json:"status"`
Type int `json:"type" xorm:"'typ'" description:"0: long-term account; 1: temporary account"`
Status int `json:"status" description:"0: active, 1: inactive, 2: locked, 3: frozen, 4: writen-off"`
IsRoot int `json:"is_root"`
LeaderId int64 `json:"leader_id"`
LeaderName string `json:"leader_name"`
LoginErrNum int `json:"login_err_num"`
ActiveBegin int64 `json:"active_begin" description:"for temporary account"`
ActiveEnd int64 `json:"active_end" description:"for temporary account"`
LockedAt int64 `json:"locked_at" description:"locked time"`
UpdatedAt int64 `json:"updated_at" description:"user info change time"`
PwdUpdatedAt int64 `json:"pwd_updated_at" description:"password change time"`
LoggedAt int64 `json:"logged_at" description:"last logged time"`
CreateAt time.Time `json:"create_at" xorm:"<-"`
}
func (u *User) Validate() error {
u.Username = strings.TrimSpace(u.Username)
if u.Username == "" {
return _e("username is blank")
}
if str.Dangerous(u.Username) {
return _e("%s %s format error", _s("username"), u.Username)
}
if str.Dangerous(u.Dispname) {
return _e("%s %s format error", _s("dispname"), u.Dispname)
}
if u.Phone != "" && !str.IsPhone(u.Phone) {
return _e("%s %s format error", _s("phone"), u.Phone)
}
if u.Email != "" && !str.IsMail(u.Email) {
return _e("%s %s format error", _s("email"), u.Email)
}
if len(u.Username) > 32 {
return _e("username too long (max:%d)", 32)
}
if len(u.Dispname) > 32 {
return _e("dispname too long (max:%d)", 32)
}
if strings.ContainsAny(u.Im, "%'") {
return _e("%s %s format error", "im", u.Im)
}
cnt, _ := DB["rdb"].Where("((email <> '' and email=?) or (phone <> '' and phone=?)) and username=?",
u.Email, u.Phone, u.Username).Count(u)
if cnt > 0 {
return _e("email %s or phone %s is exists", u.Email, u.Phone)
}
return nil
}
func (u *User) CopyLdapAttr(sr *ldap.SearchResult) {
attrs := config.Config.LDAP.Attributes
if attrs.Dispname != "" {
......@@ -95,59 +157,59 @@ func InitRooter() {
log.Println("user root init done")
}
func LdapLogin(user, pass string) (*User, error) {
sr, err := ldapReq(user, pass)
func LdapLogin(username, pass string) (*User, error) {
sr, err := ldapReq(username, pass)
if err != nil {
return nil, err
}
var u User
has, err := DB["rdb"].Where("username=?", user).Get(&u)
var user User
has, err := DB["rdb"].Where("username=?", username).Get(&user)
if err != nil {
return nil, err
}
u.CopyLdapAttr(sr)
user.CopyLdapAttr(sr)
if has {
if config.Config.LDAP.CoverAttributes {
_, err := DB["rdb"].Where("id=?", u.Id).Update(u)
return &u, err
_, err := DB["rdb"].Where("id=?", user.Id).Update(user)
return &user, err
} else {
return &u, err
return &user, err
}
}
u.Username = user
u.Password = "******"
u.UUID = GenUUIDForUser(user)
_, err = DB["rdb"].Insert(u)
return &u, nil
user.Username = username
user.Password = "******"
user.UUID = GenUUIDForUser(username)
_, err = DB["rdb"].Insert(user)
return &user, nil
}
func PassLogin(user, pass string) (*User, error) {
var u User
has, err := DB["rdb"].Where("username=?", user).Get(&u)
func PassLogin(username, pass string) (*User, error) {
var user User
has, err := DB["rdb"].Where("username=?", username).Get(&user)
if err != nil {
return nil, err
return nil, _e("Login fail, check your username and password")
}
if !has {
logger.Infof("password auth fail, no such user: %s", user)
return nil, fmt.Errorf("login fail, check your username and password")
logger.Infof("password auth fail, no such user: %s", username)
return nil, _e("Login fail, check your username and password")
}
loginPass, err := CryptoPass(pass)
if err != nil {
return nil, err
return &user, err
}
if loginPass != u.Password {
logger.Infof("password auth fail, password error, user: %s", user)
return nil, fmt.Errorf("login fail, check your username and password")
if loginPass != user.Password {
logger.Infof("password auth fail, password error, user: %s", username)
return &user, _e("Login fail, check your username and password")
}
return &u, nil
return &user, nil
}
func SmsCodeLogin(phone, code string) (*User, error) {
......@@ -156,15 +218,15 @@ func SmsCodeLogin(phone, code string) (*User, error) {
return nil, fmt.Errorf("phone %s dose not exist", phone)
}
lc, err := LoginCodeGet("username=? and code=? and login_type=?", user.Username, code, LOGIN_T_SMS)
lc, err := LoginCodeGet("username=? and code=? and login_type=?", user.Username, code, LOGIN_T_LOGIN)
if err != nil {
logger.Infof("sms-code auth fail, user: %s", user.Username)
return nil, fmt.Errorf("login fail, check your sms-code")
logger.Debugf("sms-code auth fail, user: %s", user.Username)
return user, _e("The code is incorrect")
}
if time.Now().Unix()-lc.CreatedAt > LOGIN_EXPIRES_IN {
logger.Infof("sms-code auth expired, user: %s", user.Username)
return nil, fmt.Errorf("login fail, the code has expired")
logger.Debugf("sms-code auth expired, user: %s", user.Username)
return user, _e("The code has expired")
}
lc.Del()
......@@ -178,15 +240,15 @@ func EmailCodeLogin(email, code string) (*User, error) {
return nil, fmt.Errorf("email %s dose not exist", email)
}
lc, err := LoginCodeGet("username=? and code=? and login_type=?", user.Username, code, LOGIN_T_EMAIL)
lc, err := LoginCodeGet("username=? and code=? and login_type=?", user.Username, code, LOGIN_T_LOGIN)
if err != nil {
logger.Infof("email-code auth fail, user: %s", user.Username)
return nil, fmt.Errorf("login fail, check your email-code")
logger.Debugf("email-code auth fail, user: %s", user.Username)
return user, _e("The code is incorrect")
}
if time.Now().Unix()-lc.CreatedAt > LOGIN_EXPIRES_IN {
logger.Infof("email-code auth expired, user: %s", user.Username)
return nil, fmt.Errorf("login fail, the code has expired")
logger.Debugf("email-code auth expired, user: %s", user.Username)
return user, _e("The code has expired")
}
lc.Del()
......@@ -208,56 +270,40 @@ func UserGet(where string, args ...interface{}) (*User, error) {
return &obj, nil
}
func (u *User) IsRooter() bool {
return u.IsRoot == 1
}
func (u *User) CheckFields() {
u.Username = strings.TrimSpace(u.Username)
if u.Username == "" {
errors.Bomb("username is blank")
}
if str.Dangerous(u.Username) {
errors.Bomb("username is dangerous")
}
if str.Dangerous(u.Dispname) {
errors.Bomb("dispname is dangerous")
}
if u.Phone != "" && !str.IsPhone(u.Phone) {
errors.Bomb("%s format error", u.Phone)
}
if u.Email != "" && !str.IsMail(u.Email) {
errors.Bomb("%s format error", u.Email)
func UserMustGet(where string, args ...interface{}) (*User, error) {
var obj User
has, err := DB["rdb"].Where(where, args...).Get(&obj)
if err != nil {
return nil, err
}
if len(u.Username) > 32 {
errors.Bomb("username too long")
if !has {
return nil, _e("User dose not exist")
}
if len(u.Dispname) > 32 {
errors.Bomb("dispname too long")
}
return &obj, nil
}
if strings.ContainsAny(u.Im, "%'") {
errors.Bomb("im invalid")
}
func (u *User) IsRooter() bool {
return u.IsRoot == 1
}
func (u *User) Update(cols ...string) error {
u.CheckFields()
if err := u.Validate(); err != nil {
return err
}
_, err := DB["rdb"].Where("id=?", u.Id).Cols(cols...).Update(u)
return err
}
func (u *User) Save() error {
u.CheckFields()
if err := u.Validate(); err != nil {
return err
}
if u.Id > 0 {
return fmt.Errorf("user.id[%d] not equal 0", u.Id)
return _e("user.id[%d] not equal 0", u.Id)
}
if u.UUID == "" {
......@@ -270,9 +316,11 @@ func (u *User) Save() error {
}
if cnt > 0 {
return fmt.Errorf("username already exists")
return _e("Username %s already exists", u.Username)
}
u.UpdatedAt = time.Now().Unix()
_, err = DB["rdb"].Insert(u)
return err
}
......@@ -440,7 +488,7 @@ func (u *User) HasPermGlobal(operation string) (bool, error) {
rids, err := RoleIdsHasOp(operation)
if err != nil {
return false, fmt.Errorf("[CheckPermGlobal] RoleIdsHasOp fail: %v, operation: %s", err, operation)
return false, _e("[CheckPermGlobal] RoleIdsHasOp fail: %v, operation: %s", err, operation)
}
if rids == nil || len(rids) == 0 {
......@@ -449,7 +497,7 @@ func (u *User) HasPermGlobal(operation string) (bool, error) {
has, err := UserHasGlobalRole(u.Id, rids)
if err != nil {
return false, fmt.Errorf("[CheckPermGlobal] UserHasGlobalRole fail: %v, username: %s", err, u.Username)
return false, _e("[CheckPermGlobal] UserHasGlobalRole fail: %v, username: %s", err, u.Username)
}
return has, nil
......@@ -531,7 +579,9 @@ func safeUserIds(ids []int64) ([]int64, error) {
return ret, nil
}
// Deprecated
func UsernameByUUID(uuid string) string {
logger.Warningf("UsernameByUUID is Deprectaed, use UsernameBySid instead of it")
if uuid == "" {
return ""
}
......
src/models/white_list.go 0 → 100644
浏览文件 @ d45ea025
package models
import (
"errors"
"fmt"
"time"
)
type WhiteList struct {
Id int64 `json:"id"`
StartIp string `json:"startIp"`
StartIpInt int64 `json:"-"`
EndIp string `json:"endIp"`
EndIpInt int64 `json:"-"`
StartTime int64 `json:"startTime"`
EndTime int64 `json:"endTime"`
CreatedAt int64 `json:"createdAt"`
UpdatedAt int64 `json:"updateAt"`
Creator string `json:"creator"`
Updater string `json:"updater"`
}
func WhiteListAccess(addr string) error {
ip := parseIPv4(addr)
if ip == 0 {
return fmt.Errorf("invalid remote address %s", addr)
}
now := time.Now().Unix()
count, _ := DB["rdb"].Where("start_ip_int<? and end_ip_int>? and start_time>? and end_time<?", ip, ip, now, now).Count(new(WhiteList))
if count == 0 {
return fmt.Errorf("access deny from %s", addr)
}
return nil
}
const big = 0xFFFFFF
func dtoi(s string) (n int, i int, ok bool) {
n = 0
for i = 0; i < len(s) && '0' <= s[i] && s[i] <= '9'; i++ {
n = n*10 + int(s[i]-'0')
if n >= big {
return big, i, false
}
}
if i == 0 {
return 0, 0, false
}
return n, i, true
}
func parseIPv4(s string) uint32 {
var p [4]uint32
for i := 0; i < 4; i++ {
if len(s) == 0 {
// Missing octets.
return 0
}
if i > 0 {
if s[0] != '.' {
return 0
}
s = s[1:]
}
n, c, ok := dtoi(s)
if !ok || n > 0xFF {
return 0
}
s = s[c:]
p[i] = uint32(n)
}
if len(s) != 0 {
return 0
}
return p[0]<<24 + p[1]<<16 + p[2]<<8 + p[3]
}
func (p *WhiteList) Validate() error {
if p.StartIpInt = int64(parseIPv4(p.StartIp)); p.StartIpInt == 0 {
return fmt.Errorf("invalid start ip %s", p.StartIp)
}
if p.EndIpInt = int64(parseIPv4(p.EndIp)); p.EndIpInt == 0 {
return fmt.Errorf("invalid end ip %s", p.EndIp)
}
return nil
}
func WhiteListTotal(query string) (int64, error) {
if query != "" {
q := "%" + query + "%"
return DB["rdb"].Where("start_ip like ? or end_ip like ?", q, q).Count(new(NodeTrash))
}
return DB["rdb"].Count(new(WhiteList))
}
func WhiteListGets(query string, limit, offset int) ([]WhiteList, error) {
session := DB["rdb"].Desc("id").Limit(limit, offset)
if query != "" {
q := "%" + query + "%"
session = session.Where("start_ip like ? or end_ip like ?", q, q)
}
var objs []WhiteList
err := session.Find(&objs)
return objs, err
}
func WhiteListGet(where string, args ...interface{}) (*WhiteList, error) {
var obj WhiteList
has, err := DB["rdb"].Where(where, args...).Get(&obj)
if err != nil {
return nil, err
}
if !has {
return nil, errors.New("whiteList not found")
}
return &obj, nil
}
func (p *WhiteList) Save() error {
_, err := DB["rdb"].Insert(p)
return err
}
func (p *WhiteList) Update(cols ...string) error {
_, err := DB["rdb"].Where("id=?", p.Id).Cols(cols...).Update(p)
return err
}
func (p *WhiteList) Del() error {
_, err := DB["rdb"].Where("id=?", p.Id).Delete(new(WhiteList))
return err
}
src/modules/ams/http/http_middleware.go
浏览文件 @ d45ea025
......@@ -47,7 +47,7 @@ func shouldBeService() gin.HandlerFunc {
}
func mustUsername(c *gin.Context) string {
username := cookieUsername(c)
username := sessionUsername(c)
if username == "" {
username = headerUsername(c)
}
......@@ -59,8 +59,12 @@ func mustUsername(c *gin.Context) string {
return username
}
func cookieUsername(c *gin.Context) string {
return models.UsernameByUUID(readCookieUser(c))
func sessionUsername(c *gin.Context) string {
sess, err := models.SessionGetWithCache(readSessionId(c))
if err != nil {
return ""
}
return sess.Username
}
func headerUsername(c *gin.Context) string {
......@@ -84,11 +88,10 @@ func headerUsername(c *gin.Context) string {
// ------------
func readCookieUser(c *gin.Context) string {
uuid, err := c.Cookie(config.Config.HTTP.CookieName)
func readSessionId(c *gin.Context) string {
sid, err := c.Cookie(config.Config.HTTP.CookieName)
if err != nil {
return ""
}
return uuid
return sid
}
src/modules/job/http/http_middleware.go
浏览文件 @ d45ea025
......@@ -47,7 +47,7 @@ func shouldBeService() gin.HandlerFunc {
}
func mustUsername(c *gin.Context) string {
username := cookieUsername(c)
username := sessionUsername(c)
if username == "" {
username = headerUsername(c)
}
......@@ -59,8 +59,12 @@ func mustUsername(c *gin.Context) string {
return username
}
func cookieUsername(c *gin.Context) string {
return models.UsernameByUUID(readCookieUser(c))
func sessionUsername(c *gin.Context) string {
sess, err := models.SessionGetWithCache(readSessionId(c))
if err != nil {
return ""
}
return sess.Username
}
func headerUsername(c *gin.Context) string {
......@@ -84,11 +88,10 @@ func headerUsername(c *gin.Context) string {
// ------------
func readCookieUser(c *gin.Context) string {
uuid, err := c.Cookie(config.Config.HTTP.CookieName)
func readSessionId(c *gin.Context) string {
sid, err := c.Cookie(config.Config.HTTP.CookieName)
if err != nil {
return ""
}
return uuid
return sid
}
src/modules/monapi/collector/basecollector.go 0 → 100644
浏览文件 @ d45ea025
package collector
import (
"encoding/json"
"fmt"
"github.com/didi/nightingale/src/models"
"github.com/influxdata/telegraf"
)
type BaseCollector struct {
name string
category Category
newRule func() interface{}
}
func NewBaseCollector(name string, category Category, newRule func() interface{}) *BaseCollector {
return &BaseCollector{
name: name,
category: category,
newRule: newRule,
}
}
type telegrafPlugin interface {
TelegrafInput() (telegraf.Input, error)
}
func (p BaseCollector) Name() string { return p.name }
func (p BaseCollector) Category() Category { return p.category }
func (p BaseCollector) Template() (interface{}, error) { return Template(p.newRule()) }
func (p BaseCollector) TelegrafInput(rule *models.CollectRule) (telegraf.Input, error) {
r2 := p.newRule()
if err := json.Unmarshal(rule.Data, r2); err != nil {
return nil, err
}
plugin, ok := r2.(telegrafPlugin)
if !ok {
return nil, errUnsupported
}
return plugin.TelegrafInput()
}
func (p BaseCollector) Get(id int64) (interface{}, error) {
collect := &models.CollectRule{}
has, err := models.DB["mon"].Where("id = ?", id).Get(collect)
if !has {
return nil, err
}
return collect, err
}
func (p BaseCollector) Gets(nids []int64) (ret []interface{}, err error) {
collects := []models.CollectRule{}
err = models.DB["mon"].Where("collect_type=?", p.name).In("nid", nids).Find(&collects)
for _, c := range collects {
ret = append(ret, c)
}
return ret, err
}
func (p BaseCollector) GetByNameAndNid(name string, nid int64) (interface{}, error) {
collect := &models.CollectRule{}
has, err := models.DB["mon"].Where("collect_type = ? and name = ? and nid = ?", p.name, name, nid).Get(collect)
if !has {
return nil, err
}
return collect, err
}
func (p BaseCollector) Create(data []byte, username string) error {
collect := &models.CollectRule{CollectType: p.name}
rule := p.newRule()
if err := json.Unmarshal(data, collect); err != nil {
return fmt.Errorf("unmarshal body %s err:%v", string(data), err)
}
if err := collect.Validate(rule); err != nil {
return err
}
can, err := models.UsernameCandoNodeOp(username, "mon_collect_create", collect.Nid)
if err != nil {
return err
}
if !can {
return fmt.Errorf("permission deny")
}
collect.Creator = username
collect.LastUpdator = username
old, err := p.GetByNameAndNid(collect.Name, collect.Nid)
if err != nil {
return err
}
if old != nil {
return fmt.Errorf("同节点下策略名称 %s 已存在", collect.Name)
}
return models.CreateCollect(p.name, username, collect)
}
func (p BaseCollector) Update(data []byte, username string) error {
collect := &models.CollectRule{}
rule := p.newRule()
if err := json.Unmarshal(data, collect); err != nil {
return fmt.Errorf("unmarshal body %s err:%v", string(data), err)
}
if err := collect.Validate(rule); err != nil {
return err
}
can, err := models.UsernameCandoNodeOp(username, "mon_collect_modify", collect.Nid)
if err != nil {
return err
}
if !can {
return fmt.Errorf("permission deny")
}
//校验采集是否存在
obj, err := p.Get(collect.Id) //id找不到的情况
if err != nil {
return fmt.Errorf("采集不存在 type:%s id:%d", p.name, collect.Id)
}
tmpId := obj.(*models.CollectRule).Id
if tmpId == 0 {
return fmt.Errorf("采集不存在 type:%s id:%d", p.name, collect.Id)
}
collect.Creator = username
collect.LastUpdator = username
old, err := p.GetByNameAndNid(collect.Name, collect.Nid)
if err != nil {
return err
}
if old != nil && tmpId != old.(*models.CollectRule).Id {
return fmt.Errorf("同节点下策略名称 %s 已存在", collect.Name)
}
return collect.Update()
}
func (p BaseCollector) Delete(id int64, username string) error {
tmp, err := p.Get(id) //id找不到的情况
if err != nil {
return fmt.Errorf("采集不存在 type:%s id:%d", p.name, id)
}
nid := tmp.(*models.CollectRule).Nid
can, err := models.UsernameCandoNodeOp(username, "mon_collect_delete", int64(nid))
if err != nil {
return fmt.Errorf("models.UsernameCandoNodeOp error %s", err)
}
if !can {
return fmt.Errorf("permission deny")
}
return models.DeleteCollectRule(id)
}
src/modules/monapi/collector/collector.go 0 → 100644
浏览文件 @ d45ea025
package collector
import (
"errors"
"fmt"
"github.com/didi/nightingale/src/models"
"github.com/didi/nightingale/src/toolkits/i18n"
"github.com/influxdata/telegraf"
"github.com/toolkits/pkg/logger"
)
var (
collectors = map[string]Collector{}
remoteCollectors = []string{}
localCollectors = []string{}
errUnsupported = errors.New("unsupported")
)
type Category string
const (
RemoteCategory Category = "remote" // used for prober
LocalCategory Category = "local" // used for agent
)
type Collector interface {
Name() string
Category() Category
Get(id int64) (interface{}, error)
Gets(nids []int64) ([]interface{}, error)
GetByNameAndNid(name string, nid int64) (interface{}, error)
Create(data []byte, username string) error
Update(data []byte, username string) error
Delete(id int64, username string) error
Template() (interface{}, error)
TelegrafInput(*models.CollectRule) (telegraf.Input, error)
}
func CollectorRegister(c Collector) error {
name := c.Name()
if _, ok := collectors[name]; ok {
return fmt.Errorf("collector %s exists", name)
}
collectors[name] = c
if c.Category() == RemoteCategory {
remoteCollectors = append(remoteCollectors, name)
}
if c.Category() == LocalCategory {
localCollectors = append(localCollectors, name)
}
return nil
}
func GetCollector(name string) (Collector, error) {
if c, ok := collectors[name]; !ok {
return nil, fmt.Errorf("collector %s does not exist", name)
} else {
return c, nil
}
}
func GetRemoteCollectors() []string {
return remoteCollectors
}
func GetLocalCollectors() []string {
return localCollectors
}
func _s(format string, a ...interface{}) string {
logger.Debugf(` "%s": "%s",`, format, format)
return i18n.Sprintf(format, a...)
}
src/modules/monapi/collector/template.go 0 → 100644
浏览文件 @ d45ea025
package collector
import (
"encoding/json"
"fmt"
"reflect"
"strings"
"sync"
"unicode"
)
var fieldCache sync.Map // map[reflect.Type]structFields
type Field struct {
skip bool `json:"-"`
// definitions map[string][]Field `json:"-"`
Name string `json:"name,omitempty"`
Label string `json:"label,omitempty"`
Example string `json:"example,omitempty"`
Description string `json:"description,omitempty"`
Required bool `json:"required,omitempty"`
Items *Field `json:"items,omitempty" description:"arrays's items"`
Type string `json:"type,omitempty" description:"struct,boolean,integer,folat,string,array"`
Ref string `json:"$ref,omitempty" description:"name of the struct ref"`
Fields []Field `json:"fields,omitempty" description:"fields of struct type"`
Definitions map[string][]Field `json:"definitions,omitempty"`
}
func (p Field) String() string {
return prettify(p)
}
// cachedTypeContent is like typeFields but uses a cache to avoid repeated work.
func cachedTypeContent(t reflect.Type) Field {
if f, ok := fieldCache.Load(t); ok {
return f.(Field)
}
f, _ := fieldCache.LoadOrStore(t, typeContent(t))
return f.(Field)
}
func typeContent(t reflect.Type) Field {
definitions := map[string][]Field{t.String(): nil}
ret := Field{
// definitions: map[string][]Field{
// t.String(): nil,
// },
}
for i := 0; i < t.NumField(); i++ {
sf := t.Field(i)
isUnexported := sf.PkgPath != ""
if sf.Anonymous {
panic("unsupported anonymous field")
} else if isUnexported {
// Ignore unexported non-embedded fields.
continue
}
field := getTagOpt(sf)
if field.skip {
continue
}
ft := sf.Type
fieldType(ft, &field, definitions)
// Record found field and index sequence.
if field.Name != "" || !sf.Anonymous || ft.Kind() != reflect.Struct {
ret.Fields = append(ret.Fields, field)
continue
}
panic("unsupported anonymous, struct field")
}
definitions[t.String()] = ret.Fields
ret.Definitions = definitions
return ret
}
// tagOptions is the string following a comma in a struct field's "json"
// tag, or the empty string. It does not include the leading comma.
type tagOptions string
// parseTag splits a struct field's json tag into its name and
// comma-separated options.
func parseTag(tag string) (string, tagOptions) {
if idx := strings.Index(tag, ","); idx != -1 {
return tag[:idx], tagOptions(tag[idx+1:])
}
return tag, tagOptions("")
}
// Contains reports whether a comma-separated list of options
// contains a particular substr flag. substr must be surrounded by a
// string boundary or commas.
func (o tagOptions) Contains(optionName string) bool {
if len(o) == 0 {
return false
}
s := string(o)
for s != "" {
var next string
i := strings.Index(s, ",")
if i >= 0 {
s, next = s[:i], s[i+1:]
}
if s == optionName {
return true
}
s = next
}
return false
}
func getTagOpt(sf reflect.StructField) (opt Field) {
if sf.Anonymous {
return
}
tag := sf.Tag.Get("json")
if tag == "-" {
opt.skip = true
return
}
name, opts := parseTag(tag)
if opts.Contains("required") {
opt.Required = true
}
opt.Name = name
opt.Label = _s(sf.Tag.Get("label"))
opt.Example = sf.Tag.Get("example")
opt.Description = _s(sf.Tag.Get("description"))
return
}
func isValidTag(s string) bool {
if s == "" {
return false
}
for _, c := range s {
switch {
case strings.ContainsRune("!#$%&()*+-./:<=>?@[]^_{|}~ ", c):
case !unicode.IsLetter(c) && !unicode.IsDigit(c):
return false
}
}
return true
}
func panicType(ft reflect.Type, args ...interface{}) {
msg := fmt.Sprintf("type field %s %s", ft.PkgPath(), ft.Name())
if len(args) > 0 {
panic(fmt.Sprint(args...) + " " + msg)
}
panic(msg)
}
func Template(v interface{}) (interface{}, error) {
rv := reflect.Indirect(reflect.ValueOf(v))
if rv.Kind() != reflect.Struct {
return nil, fmt.Errorf("invalid argument, must be a struct")
}
content := cachedTypeContent(rv.Type())
return content, nil
}
func prettify(in interface{}) string {
b, _ := json.MarshalIndent(in, "", " ")
return string(b)
}
func fieldType(t reflect.Type, in *Field, definitions map[string][]Field) {
if t.Name() == "" && t.Kind() == reflect.Ptr {
// Follow pointer.
t = t.Elem()
}
switch t.Kind() {
case reflect.Int, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint32, reflect.Uint64:
in.Type = "integer"
case reflect.Float32, reflect.Float64:
in.Type = "float"
case reflect.Bool:
in.Type = "boolean"
case reflect.String:
in.Type = "string"
case reflect.Struct:
name := t.String()
if _, ok := definitions[name]; !ok {
f := cachedTypeContent(t)
for k, v := range f.Definitions {
definitions[k] = v
}
}
in.Ref = t.String()
case reflect.Slice, reflect.Array:
t2 := t.Elem()
if t2.Kind() == reflect.Ptr {
t2 = t2.Elem()
}
if k := t2.Kind(); k == reflect.Int || k == reflect.Int32 || k == reflect.Int64 ||
k == reflect.Uint || k == reflect.Uint32 || k == reflect.Uint64 ||
k == reflect.Float32 || k == reflect.Float64 ||
k == reflect.Bool || k == reflect.String || k == reflect.Struct {
in.Type = "array"
in.Items = &Field{}
fieldType(t2, in.Items, definitions)
} else {
panic(fmt.Sprintf("unspport type %s items %s", t.String(), t2.String()))
}
default:
panic(fmt.Sprintf("unspport type %s", t.String()))
// in.Type = "string"
}
}
src/modules/monapi/config/const.go
浏览文件 @ d45ea025
......@@ -3,6 +3,7 @@ package config
const Version = 1
const JudgesReplicas = 500
const ProbersReplicas = 500
const DetectorReplicas = 500
const (
......
src/modules/monapi/config/yaml.go
浏览文件 @ d45ea025
......@@ -94,7 +94,9 @@ type loggerSection struct {
}
type httpSection struct {
Listen string `yaml:"listen"`
Mode string `yaml:"mode"`
CookieName string `yaml:"cookieName"`
CookieDomain string `yaml:"cookieDomain"`
}
type proxySection struct {
......
src/modules/monapi/http/http_middlewre.go → src/modules/monapi/http/http_middleware.go
浏览文件 @ d45ea025
......@@ -5,7 +5,6 @@ import (
"github.com/didi/nightingale/src/models"
"github.com/didi/nightingale/src/modules/monapi/config"
"github.com/didi/nightingale/src/modules/monapi/tools"
"github.com/gin-gonic/gin"
"github.com/toolkits/pkg/errors"
......@@ -15,7 +14,7 @@ import (
// GetCookieUser 从cookie中获取username
func GetCookieUser() gin.HandlerFunc {
return func(c *gin.Context) {
username := cookieUser(c)
username := sessionUsername(c)
if username == "" {
username = headerUser(c)
}
......@@ -29,15 +28,6 @@ func GetCookieUser() gin.HandlerFunc {
}
}
func cookieUser(c *gin.Context) string {
uuid, err := c.Cookie("ecmc-user")
if err != nil {
return ""
}
return tools.UsernameByUUID(uuid)
}
func headerUser(c *gin.Context) string {
token := c.GetHeader("X-User-Token")
if token == "" {
......@@ -88,3 +78,19 @@ func getUserByToken(token string) (user *models.User, err error) {
return
}
func sessionUsername(c *gin.Context) string {
sess, err := models.SessionGetWithCache(readSessionId(c))
if err != nil {
return ""
}
return sess.Username
}
func readSessionId(c *gin.Context) string {
sid, err := c.Cookie(config.Get().HTTP.CookieName)
if err != nil {
return ""
}
return sid
}
src/modules/monapi/http/http_server.go
浏览文件 @ d45ea025
......@@ -2,9 +2,11 @@ package http
import (
"context"
"log"
"fmt"
"net/http"
_ "net/http/pprof"
"os"
"strings"
"time"
"github.com/didi/nightingale/src/common/address"
......@@ -31,7 +33,7 @@ func Start() {
loggerMid := middleware.LoggerWithConfig(middleware.LoggerConfig{SkipPaths: skipPaths})
recoveryMid := middleware.Recovery()
if c.Logger.Level != "DEBUG" {
if strings.ToLower(c.HTTP.Mode) == "release" {
gin.SetMode(gin.ReleaseMode)
middleware.DisableConsoleColor()
} else {
......@@ -47,9 +49,10 @@ func Start() {
srv.Handler = r
go func() {
log.Println("starting http server, listening on:", srv.Addr)
fmt.Println("http.listening:", srv.Addr)
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("listening %s occur error: %s\n", srv.Addr, err)
fmt.Printf("listening %s occur error: %s\n", srv.Addr, err)
os.Exit(3)
}
}()
}
......@@ -59,14 +62,15 @@ func Shutdown() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Fatalln("cannot shutdown http server:", err)
fmt.Println("cannot shutdown http server:", err)
os.Exit(2)
}
// catching ctx.Done(). timeout of 5 seconds.
select {
case <-ctx.Done():
log.Println("shutdown http server timeout of 5 seconds.")
fmt.Println("shutdown http server timeout of 5 seconds.")
default:
log.Println("http server stopped")
fmt.Println("http server stopped")
}
}
src/modules/monapi/http/router.go
浏览文件 @ d45ea025
......@@ -83,20 +83,41 @@ func Config(r *gin.Engine) {
event.POST("/cur/claim", eventCurClaim)
}
// TODO: merge to collect-rule
collect := r.Group("/api/mon/collect").Use(GetCookieUser())
{
collect.POST("", collectPost)
collect.GET("/list", collectsGet)
collect.GET("", collectGet)
collect.PUT("", collectPut)
collect.DELETE("", collectsDel)
collect.POST("/check", regExpCheck)
collect.POST("", collectRulePost) // create a collect rule
collect.GET("/list", collectRulesGet) // get collect rules
collect.GET("", collectRuleGet) // get collect rule by type & id
collect.PUT("", collectRulePut) // update collect rule by type & id
collect.DELETE("", collectsRuleDel) // delete collect rules by type & ids
collect.POST("/check", regExpCheck) // check collect rule
}
// TODO: merge to collect-rules, used by agent
collects := r.Group("/api/mon/collects")
{
collects.GET("/:endpoint", collectGetByEndpoint)
collects.GET("", collectsGet)
collects.GET("/:endpoint", collectRulesGetByLocalEndpoint) // get collect rules by endpoint, for agent
collects.GET("", collectRulesGet) // get collect rules
}
collectRules := r.Group("/api/mon/collect-rules").Use(GetCookieUser())
{
collectRules.POST("", collectRulePost) // create a collect rule
collectRules.GET("/list", collectRulesGet) // get collect rules
collectRules.GET("", collectRuleGet) // get collect rule by type & id
collectRules.PUT("", collectRulePut) // update collect rule by type & id
collectRules.DELETE("", collectsRuleDel) // delete collect rules by type & ids
collectRules.POST("/check", regExpCheck) // check collect rule
collectRules.GET("/types", collectRuleTypesGet) // get collect types, category: local|remote
collectRules.GET("/types/:type/template", collectRuleTemplateGet) // get collect teplate by type
}
collectRulesAnonymous := r.Group("/api/mon/collect-rules")
{
collectRulesAnonymous.GET("/endpoints/:endpoint/remote", collectRulesGetByRemoteEndpoint) // for prober
collectRulesAnonymous.GET("/endpoints/:endpoint/local", collectRulesGetByLocalEndpoint) // for agent
}
stra := r.Group("/api/mon/stra").Use(GetCookieUser())
......@@ -151,6 +172,15 @@ func Config(r *gin.Engine) {
indexProxy.POST("/counter/detail", indexReq)
}
/*
v1 := r.Group("/v1/mon")
{
v1.POST("/report-detector-heartbeat", detectorHeartBeat)
v1.GET("/detectors", detectorInstanceGets)
v1.GET("/rules", collectRulesGet)
}
*/
if config.Get().Logger.Level == "DEBUG" {
pprof.Register(r, "/api/monapi/debug/pprof")
}
......
src/modules/monapi/http/router_collect.go
浏览文件 @ d45ea025
......@@ -6,8 +6,7 @@ import (
"regexp"
"strings"
"github.com/didi/nightingale/src/models"
"github.com/didi/nightingale/src/modules/monapi/config"
"github.com/didi/nightingale/src/modules/monapi/collector"
"github.com/didi/nightingale/src/modules/monapi/scache"
"github.com/gin-gonic/gin"
......@@ -16,201 +15,44 @@ import (
)
type CollectRecv struct {
Type string `json:"type"`
Data interface{} `json:"data"`
Type string `json:"type"`
Data json.RawMessage `json:"data"`
}
//此处实现需要重构
func collectPost(c *gin.Context) {
creator := loginUsername(c)
func collectRulePost(c *gin.Context) {
var recv []CollectRecv
errors.Dangerous(c.ShouldBind(&recv))
creator := loginUsername(c)
for _, obj := range recv {
switch obj.Type {
case "port":
collect := new(models.PortCollect)
b, err := json.Marshal(obj.Data)
if err != nil {
bomb("marshal body %s err:%v", obj, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_create", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Creator = creator
collect.LastUpdator = creator
nid := collect.Nid
name := collect.Name
old, err := models.GetCollectByNameAndNid(obj.Type, name, nid)
errors.Dangerous(err)
if old != nil {
bomb("same stra name %s in node", name)
}
errors.Dangerous(models.CreateCollect(obj.Type, creator, collect))
case "proc":
collect := new(models.ProcCollect)
b, err := json.Marshal(obj.Data)
if err != nil {
bomb("marshal body %s err:%v", obj, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_create", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Creator = creator
collect.LastUpdator = creator
nid := collect.Nid
name := collect.Name
old, err := models.GetCollectByNameAndNid(obj.Type, name, nid)
errors.Dangerous(err)
if old != nil {
bomb("same stra name %s in node", name)
}
errors.Dangerous(models.CreateCollect(obj.Type, creator, collect))
case "log":
collect := new(models.LogCollect)
b, err := json.Marshal(obj.Data)
if err != nil {
bomb("marshal body %s err:%v", obj, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_create", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Encode()
collect.Creator = creator
collect.LastUpdator = creator
nid := collect.Nid
name := collect.Name
old, err := models.GetCollectByNameAndNid(obj.Type, name, nid)
errors.Dangerous(err)
if old != nil {
bomb("same stra name %s in node", name)
}
errors.Dangerous(models.CreateCollect(obj.Type, creator, collect))
case "plugin":
collect := new(models.PluginCollect)
b, err := json.Marshal(obj.Data)
if err != nil {
bomb("marshal body %s err:%v", obj, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_create", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Creator = creator
collect.LastUpdator = creator
nid := collect.Nid
name := collect.Name
old, err := models.GetCollectByNameAndNid(obj.Type, name, nid)
errors.Dangerous(err)
if old != nil {
bomb("same stra name %s in node", name)
}
errors.Dangerous(models.CreateCollect(obj.Type, creator, collect))
case "api":
collect := new(models.ApiCollect)
b, err := json.Marshal(obj.Data)
if err != nil {
bomb("marshal body %s err:%v", obj, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_create", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Encode()
collect.Creator = creator
collect.LastUpdator = creator
nid := collect.Nid
name := collect.Name
old, err := models.GetCollectByNameAndNid(obj.Type, name, nid)
errors.Dangerous(err)
if old != nil {
bomb("same stra name %s in node", name)
}
errors.Dangerous(models.CreateCollect(obj.Type, creator, collect))
cl, err := collector.GetCollector(obj.Type)
errors.Dangerous(err)
default:
bomb("collect type not support")
if err := cl.Create([]byte(obj.Data), creator); err != nil {
errors.Bomb("%s add rule err %s", obj.Type, err)
}
}
renderData(c, "ok", nil)
}
func collectGetByEndpoint(c *gin.Context) {
func collectRulesGetByLocalEndpoint(c *gin.Context) {
collect := scache.CollectCache.GetBy(urlParamStr(c, "endpoint"))
renderData(c, collect, nil)
}
func collectGet(c *gin.Context) {
func collectRuleGet(c *gin.Context) {
t := mustQueryStr(c, "type")
nid := mustQueryInt64(c, "id")
collect, err := models.GetCollectById(t, nid)
id := mustQueryInt64(c, "id")
cl, err := collector.GetCollector(t)
errors.Dangerous(err)
renderData(c, collect, nil)
ret, err := cl.Get(id)
renderData(c, ret, err)
}
func collectsGet(c *gin.Context) {
func collectRulesGet(c *gin.Context) {
nid := queryInt64(c, "nid", -1)
tp := queryStr(c, "type", "")
var resp []interface{}
......@@ -222,275 +64,36 @@ func collectsGet(c *gin.Context) {
types = []string{tp}
}
if nid == -1 && tp != "" { //没有nid参数，tp不为空
if tp == "api" {
collects, err := models.GetApiCollects()
errors.Dangerous(err)
for _, c := range collects {
c.Decode()
resp = append(resp, c)
}
nids := []int64{nid}
for _, t := range types {
cl, err := collector.GetCollector(t)
if err != nil {
logger.Warning(t, err)
continue
}
} else {
nids := []int64{nid}
for _, t := range types {
collects, err := models.GetCollectByNid(t, nids)
if err != nil {
logger.Warning(t, err)
continue
}
resp = append(resp, collects...)
ret, err := cl.Gets(nids)
if err != nil {
logger.Warning(t, err)
continue
}
resp = append(resp, ret...)
}
renderData(c, resp, nil)
}
func collectPut(c *gin.Context) {
creator := loginUsername(c)
func collectRulePut(c *gin.Context) {
var recv CollectRecv
errors.Dangerous(c.ShouldBind(&recv))
switch recv.Type {
case "port":
collect := new(models.PortCollect)
b, err := json.Marshal(recv.Data)
if err != nil {
bomb("marshal body %s err:%v", recv, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_modify", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
nid := collect.Nid
name := collect.Name
//校验采集是否存在
obj, err := models.GetCollectById(recv.Type, collect.Id) //id找不到的情况
if err != nil {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
tmpId := obj.(*models.PortCollect).Id
if tmpId == 0 {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
collect.Creator = creator
collect.LastUpdator = creator
old, err := models.GetCollectByNameAndNid(recv.Type, name, nid)
errors.Dangerous(err)
if old != nil && tmpId != old.(*models.PortCollect).Id {
bomb("same stra name %s in node", name)
}
errors.Dangerous(collect.Update())
renderData(c, "ok", nil)
return
case "proc":
collect := new(models.ProcCollect)
b, err := json.Marshal(recv.Data)
if err != nil {
bomb("marshal body %s err:%v", recv, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_modify", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
nid := collect.Nid
name := collect.Name
//校验采集是否存在
obj, err := models.GetCollectById(recv.Type, collect.Id) //id找不到的情况
if err != nil {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
tmpId := obj.(*models.ProcCollect).Id
if tmpId == 0 {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
can, err = models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_modify", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Creator = creator
collect.LastUpdator = creator
old, err := models.GetCollectByNameAndNid(recv.Type, name, nid)
errors.Dangerous(err)
if old != nil && tmpId != old.(*models.ProcCollect).Id {
bomb("same stra name %s in node", name)
}
errors.Dangerous(collect.Update())
renderData(c, "ok", nil)
return
case "log":
collect := new(models.LogCollect)
b, err := json.Marshal(recv.Data)
if err != nil {
bomb("marshal body %s err:%v", recv, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_modify", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Encode()
nid := collect.Nid
name := collect.Name
//校验采集是否存在
obj, err := models.GetCollectById(recv.Type, collect.Id) //id找不到的情况
if err != nil {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
tmpId := obj.(*models.LogCollect).Id
if tmpId == 0 {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
collect.Creator = creator
collect.LastUpdator = creator
old, err := models.GetCollectByNameAndNid(recv.Type, name, nid)
errors.Dangerous(err)
if old != nil && tmpId != old.(*models.LogCollect).Id {
bomb("same stra name %s in node", name)
}
errors.Dangerous(collect.Update())
renderData(c, "ok", nil)
return
case "plugin":
collect := new(models.PluginCollect)
b, err := json.Marshal(recv.Data)
if err != nil {
bomb("marshal body %s err:%v", recv, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_modify", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
nid := collect.Nid
name := collect.Name
//校验采集是否存在
obj, err := models.GetCollectById(recv.Type, collect.Id) //id找不到的情况
if err != nil {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
tmpId := obj.(*models.PluginCollect).Id
if tmpId == 0 {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
collect.Creator = creator
collect.LastUpdator = creator
old, err := models.GetCollectByNameAndNid(recv.Type, name, nid)
errors.Dangerous(err)
if old != nil && tmpId != old.(*models.PluginCollect).Id {
bomb("same stra name %s in node", name)
}
errors.Dangerous(collect.Update())
renderData(c, "ok", nil)
return
case "api":
collect := new(models.ApiCollect)
b, err := json.Marshal(recv.Data)
if err != nil {
bomb("marshal body %s err:%v", recv, err)
}
err = json.Unmarshal(b, collect)
if err != nil {
bomb("unmarshal body %s err:%v", string(b), err)
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_modify", collect.Nid)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
collect.Encode()
nid := collect.Nid
name := collect.Name
//校验采集是否存在
obj, err := models.GetCollectById(recv.Type, collect.Id) //id找不到的情况
if err != nil {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
tmpId := obj.(*models.ApiCollect).Id
if tmpId == 0 {
bomb("采集不存在 type:%s id:%d", recv.Type, collect.Id)
}
collect.Creator = creator
collect.LastUpdator = creator
old, err := models.GetCollectByNameAndNid(recv.Type, name, nid)
errors.Dangerous(err)
if old != nil && tmpId != old.(*models.ApiCollect).Id {
bomb("same stra name %s in node", name)
}
errors.Dangerous(collect.Update())
renderData(c, "ok", nil)
return
cl, err := collector.GetCollector(recv.Type)
errors.Dangerous(err)
default:
bomb("采集类型不合法")
creator := loginUsername(c)
if err := cl.Update([]byte(recv.Data), creator); err != nil {
errors.Bomb("%s update rule err %s", recv.Type, err)
}
renderData(c, "ok", nil)
}
......@@ -499,89 +102,44 @@ type CollectsDelRev struct {
Ids []int64 `json:"ids"`
}
func collectsDel(c *gin.Context) {
username := loginUsername(c)
func collectsRuleDel(c *gin.Context) {
var recv []CollectsDelRev
errors.Dangerous(c.ShouldBind(&recv))
for _, obj := range recv {
username := loginUsername(c)
for _, obj := range recv {
for i := 0; i < len(obj.Ids); i++ {
tmp, err := models.GetCollectById(obj.Type, obj.Ids[i]) //id找不到的情况
if err != nil {
bomb("采集不存在 type:%s id:%d", obj.Type, obj.Ids[i])
}
if tmp == nil {
bomb("采集不存在 type:%s id:%d", obj.Type, obj.Ids[i])
}
var nid int64
switch obj.Type {
case "log":
nid = tmp.(*models.LogCollect).Nid
case "proc":
nid = tmp.(*models.ProcCollect).Nid
case "port":
nid = tmp.(*models.PortCollect).Nid
case "plugin":
nid = tmp.(*models.PluginCollect).Nid
}
can, err := models.UsernameCandoNodeOp(loginUsername(c), "mon_collect_delete", int64(nid))
cl, err := collector.GetCollector(obj.Type)
errors.Dangerous(err)
if !can {
bomb("permission deny")
}
}
for i := 0; i < len(obj.Ids); i++ {
switch obj.Type {
case "api":
err := models.DeleteApiCollect(obj.Ids[i])
errors.Dangerous(err)
default:
err := models.DeleteCollectById(obj.Type, username, obj.Ids[i])
if err := cl.Delete(obj.Ids[i], username); err != nil {
errors.Dangerous(err)
}
}
}
renderData(c, "ok", nil)
}
type ApiStraRev struct {
Sid int64 `json:"sid"`
Cid int64 `json:"cid"`
}
type ApiStraRes struct {
Has bool `json:"has"`
Sid int64 `json:"sid"`
}
func ApiStraGet(c *gin.Context) {
cid := mustQueryInt64(c, "cid")
sid, err := models.GetSidByCid(cid)
var res ApiStraRes
errors.Dangerous(err)
if sid != 0 {
res.Has = true
res.Sid = sid
func collectRuleTypesGet(c *gin.Context) {
category := mustQueryStr(c, "category")
switch category {
case "remote":
renderData(c, collector.GetRemoteCollectors(), nil)
case "local":
renderData(c, collector.GetLocalCollectors(), nil)
default:
renderData(c, nil, nil)
}
renderData(c, res, nil)
}
func ApiStraPost(c *gin.Context) {
recv := new(models.ApiCollectSid)
errors.Dangerous(c.ShouldBind(&recv))
errors.Dangerous(recv.Add())
renderData(c, "ok", nil)
return
}
func collectRuleTemplateGet(c *gin.Context) {
t := urlParamStr(c, "type")
collector, err := collector.GetCollector(t)
errors.Dangerous(err)
func ApiRegionGet(c *gin.Context) {
renderData(c, config.Get().Region, nil)
tpl, err := collector.Template()
renderData(c, tpl, err)
}
type RegExpCheckDto struct {
......@@ -788,3 +346,9 @@ func genSubErrMsg(sign string) string {
func genIllegalCharErrMsg() string {
return fmt.Sprintf(`正则匹配成功。但是tag的key或者value包含非法字符:[:,/=\r\n\t], 请重新调整`)
}
func collectRulesGetByRemoteEndpoint(c *gin.Context) {
rules := scache.CollectRuleCache.GetBy(urlParamStr(c, "endpoint"))
renderData(c, rules, nil)
}
src/modules/monapi/http/router_funcs.go
浏览文件 @ d45ea025
......@@ -142,22 +142,13 @@ func renderData(c *gin.Context, data interface{}, err error) {
renderMessage(c, err.Error())
}
func cookieUsername(c *gin.Context) string {
uuid, err := c.Cookie("ecmc-user")
if err != nil {
return ""
}
return models.UsernameByUUID(uuid)
}
func loginUsername(c *gin.Context) string {
username1, has := c.Get("username")
if has {
return username1.(string)
}
username2 := cookieUsername(c)
username2 := sessionUsername(c)
if username2 == "" {
bomb("unauthorized")
}
......
src/modules/monapi/http/router_stra.go
浏览文件 @ d45ea025
......@@ -129,7 +129,7 @@ func effectiveStrasGet(c *gin.Context) {
if queryInt(c, "all", 0) == 1 {
stras = scache.StraCache.GetAll()
} else if instance != "" {
node, err := scache.ActiveNode.GetNodeBy(instance)
node, err := scache.ActiveJudgeNode.GetNodeBy(instance)
errors.Dangerous(err)
stras = scache.StraCache.GetByNode(node)
......
src/modules/monapi/monapi.go
浏览文件 @ d45ea025
......@@ -18,6 +18,7 @@ import (
"github.com/didi/nightingale/src/modules/monapi/scache"
"github.com/didi/nightingale/src/toolkits/i18n"
_ "github.com/didi/nightingale/src/modules/monapi/plugins/all"
_ "github.com/go-sql-driver/mysql"
"github.com/toolkits/pkg/cache"
......
src/modules/monapi/plugins/all/all.go 0 → 100644
浏览文件 @ d45ea025
package all
import (
// remote
_ "github.com/didi/nightingale/src/modules/monapi/plugins/api"
_ "github.com/didi/nightingale/src/modules/monapi/plugins/github"
_ "github.com/didi/nightingale/src/modules/monapi/plugins/mysql"
// _ "github.com/didi/nightingale/src/modules/monapi/plugins/prometheus"
_ "github.com/didi/nightingale/src/modules/monapi/plugins/redis"
// local
_ "github.com/didi/nightingale/src/modules/monapi/plugins/log"
_ "github.com/didi/nightingale/src/modules/monapi/plugins/plugin"
_ "github.com/didi/nightingale/src/modules/monapi/plugins/port"
_ "github.com/didi/nightingale/src/modules/monapi/plugins/proc"
)
src/modules/monapi/plugins/api/api.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/github/README.md 0 → 100644
浏览文件 @ d45ea025
# GitHub Input Plugin
Gather repository information from [GitHub][] hosted repositories.
**Note:** Telegraf also contains the [webhook][] input which can be used as an
alternative method for collecting repository information.
### Configuration
```toml
[[inputs.github]]
## List of repositories to monitor
repositories = [
"influxdata/telegraf",
"influxdata/influxdb"
]
## Github API access token. Unauthenticated requests are limited to 60 per hour.
# access_token = ""
## Github API enterprise url. Github Enterprise accounts must specify their base url.
# enterprise_base_url = ""
## Timeout for HTTP requests.
# http_timeout = "5s"
```
### Metrics
- github_repository
- tags:
- name - The repository name
- owner - The owner of the repository
- language - The primary language of the repository
- license - The license set for the repository
- fields:
- forks (int)
- open_issues (int)
- networks (int)
- size (int)
- subscribers (int)
- stars (int)
- watchers (int)
When the [internal][] input is enabled:
+ internal_github
- tags:
- access_token - An obfuscated reference to the configured access token or "Unauthenticated"
- fields:
- limit - How many requests you are limited to (per hour)
- remaining - How many requests you have remaining (per hour)
- blocks - How many requests have been blocked due to rate limit
### Example Output
```
github_repository,language=Go,license=MIT\ License,name=telegraf,owner=influxdata forks=2679i,networks=2679i,open_issues=794i,size=23263i,stars=7091i,subscribers=316i,watchers=7091i 1563901372000000000
internal_github,access_token=Unauthenticated rate_limit_remaining=59i,rate_limit_limit=60i,rate_limit_blocks=0i 1552653551000000000
```
[GitHub]: https://www.github.com
[internal]: /plugins/inputs/internal
[webhook]: /plugins/inputs/webhooks/github
src/modules/monapi/plugins/github/github.go 0 → 100644
浏览文件 @ d45ea025
package github
import (
"fmt"
"time"
"github.com/didi/nightingale/src/modules/monapi/collector"
"github.com/influxdata/telegraf"
)
func init() {
collector.CollectorRegister(NewGitHubCollector()) // for monapi
}
type GitHubCollector struct {
*collector.BaseCollector
}
func NewGitHubCollector() *GitHubCollector {
return &GitHubCollector{BaseCollector: collector.NewBaseCollector(
"github",
collector.RemoteCategory,
func() interface{} { return &GitHubRule{} },
)}
}
type GitHubRule struct {
Repositories []string `label:"Repositories" json:"repositories" description:"List of repositories to monitor"`
AccessToken string `label:"Access token" json:"access_token" description:"Github API access token. Unauthenticated requests are limited to 60 per hour"`
EnterpriseBaseURL string `label:"Enterprise base url" json:"enterprise_base_url" description:"Github API enterprise url. Github Enterprise accounts must specify their base url"`
HTTPTimeout int `label:"HTTP timeout" json:"http_timeout" description:"Timeout for HTTP requests"`
}
func (p *GitHubRule) Validate() error {
if len(p.Repositories) == 0 || p.Repositories[0] == "" {
return fmt.Errorf("github.rule.repositories must be set")
}
if p.HTTPTimeout == 0 {
p.HTTPTimeout = 5
}
return nil
}
func (p *GitHubRule) TelegrafInput() (telegraf.Input, error) {
if err := p.Validate(); err != nil {
return nil, err
}
return &GitHub{
Repositories: p.Repositories,
AccessToken: p.AccessToken,
EnterpriseBaseURL: p.EnterpriseBaseURL,
HTTPTimeout: time.Second * time.Duration(p.HTTPTimeout),
}, nil
}
src/modules/monapi/plugins/github/github.telegraf.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/github/github_test.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/log/log.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/mongodb/mongodb.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/mysql/mysql.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/plugin/plugin.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/port/port.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/proc/proc.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/plugins/redis/redis.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/scache/collectrule.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/scache/init.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/scache/judge.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/monapi/scache/stra_cache.go
浏览文件 @ d45ea025
......@@ -46,7 +46,7 @@ func (s *StraCacheMap) GetAll() []*models.Stra {
data := []*models.Stra{}
for node, stras := range s.Data {
instance, exists := ActiveNode.GetInstanceBy(node)
instance, exists := ActiveJudgeNode.GetInstanceBy(node)
if !exists {
continue
}
......
src/modules/prober/cache/cache.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/cache/collect_rule.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/cache/history.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/cache/plugins_config.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/config/config.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/core/clients.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/core/common.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/core/push.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/expr/expr.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/http/http_server.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/http/router.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/http/router_funcs.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/manager/heap.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/manager/manager.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/manager/metric.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/manager/rule_entity.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/prober/prober.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/auth/auth.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/auth/authenticator.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/cache/cache.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/cache/config.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/cache/session.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/config/yaml.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/http/router.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/http/router_auth.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/http/router_hbs.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/http/router_self.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/http/router_user.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/rdb.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/session/session.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/session/session_db.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/session/session_mem.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/session/session_options.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/session/session_test.go 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
src/modules/rdb/ssoc/sso.go
浏览文件 @ d45ea025
此差异已折叠。
src/modules/transfer/backend/m3db/.gitignore 0 → 100644
浏览文件 @ d45ea025
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或