Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Pinoxchio
apollo
提交
fc4ad19b
A
apollo
项目概览
Pinoxchio
/
apollo
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
A
apollo
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
fc4ad19b
编写于
2月 04, 2019
作者:
A
Aaron Xiao
提交者:
Jiangtao Hu
2月 04, 2019
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Data: Migrate warehouse to off-board repo.
上级
7b40ff1d
变更
12
隐藏空白更改
内联
并排
Showing
12 changed file
with
0 addition
and
1094 deletion
+0
-1094
modules/data/warehouse/README.md
modules/data/warehouse/README.md
+0
-46
modules/data/warehouse/tools/import_record.py
modules/data/warehouse/tools/import_record.py
+0
-61
modules/data/warehouse/tools/mongo_util.py
modules/data/warehouse/tools/mongo_util.py
+0
-82
modules/data/warehouse/tools/parse_record.py
modules/data/warehouse/tools/parse_record.py
+0
-176
modules/data/warehouse/web_server/add_pythonpath.py
modules/data/warehouse/web_server/add_pythonpath.py
+0
-21
modules/data/warehouse/web_server/display_util.py
modules/data/warehouse/web_server/display_util.py
+0
-113
modules/data/warehouse/web_server/main.py
modules/data/warehouse/web_server/main.py
+0
-158
modules/data/warehouse/web_server/records_util.py
modules/data/warehouse/web_server/records_util.py
+0
-44
modules/data/warehouse/web_server/static/js/gmap_util.js
modules/data/warehouse/web_server/static/js/gmap_util.js
+0
-58
modules/data/warehouse/web_server/templates/base.tpl
modules/data/warehouse/web_server/templates/base.tpl
+0
-52
modules/data/warehouse/web_server/templates/record.tpl
modules/data/warehouse/web_server/templates/record.tpl
+0
-190
modules/data/warehouse/web_server/templates/records.tpl
modules/data/warehouse/web_server/templates/records.tpl
+0
-93
未找到文件。
modules/data/warehouse/README.md
已删除
100644 → 0
浏览文件 @
7b40ff1d
# Data Warehouse
We choose document based data warehousing, which performs well on composite
search criterias.
We defined the document as a proto, saying apollo::data::Record. It's easy to
convert proto to Json/Bson and vice versa.
## Setup MongoDB
For local testing, simply bring up a docker container:
```
bash
docker run
--name
mongo
-p
27017:27017
-d
mongo
```
Advanced users should setup credentials and pass flags to mongo_util.py:
```
bash
python <tool>
\
--mongo_host
=
x.x.x.x
\
--mongo_port
=
xxxx
\
--mongo_db_name
=
apollo
\
--mongo_user
=
xxxxxx
\
--mongo_pass
=
xxxxxx
\
<other arguments>
```
## Import Record
```
bash
python tools/import_record.py <record_file>
```
## Serve Data
```
bash
python web_server/main.py
\
--mongo_host
=
x.x.x.x
\
--mongo_port
=
x
\
--mongo_db_name
=
apollo
\
--mongo_user
=
xxxxxx
\
--mongo_pass
=
xxxxxx
\
--collection
=
xxxxxx
\
--gmap_api_key
=
xxxxxx
```
modules/data/warehouse/tools/import_record.py
已删除
100755 → 0
浏览文件 @
7b40ff1d
#!/usr/bin/env python
# -*- coding: UTF-8-*-
###############################################################################
# Copyright 2018 The Apollo Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
"""
Import Apollo record into a MongoDB.
Use as command tool: import_record.py <record>
Use as util lib: RecordImporter.Import(<record>)
"""
import
os
import
sys
import
gflags
import
glog
from
mongo_util
import
Mongo
from
parse_record
import
RecordParser
gflags
.
DEFINE_string
(
'mongo_collection_name'
,
'records'
,
'MongoDB collection name.'
)
class
RecordImporter
(
object
):
"""Import Apollo record files."""
@
staticmethod
def
Import
(
record_file
):
"""Import one record."""
parser
=
RecordParser
(
record_file
)
if
not
parser
.
ParseMeta
():
glog
.
error
(
'Fail to parse record {}'
.
format
(
record_file
))
return
parser
.
ParseMessages
()
doc
=
Mongo
.
pb_to_doc
(
parser
.
record
)
collection
=
Mongo
.
collection
(
gflags
.
FLAGS
.
mongo_collection_name
)
collection
.
replace_one
({
'path'
:
parser
.
record
.
path
},
doc
,
upsert
=
True
)
glog
.
info
(
'Imported record {}'
.
format
(
record_file
))
if
__name__
==
'__main__'
:
gflags
.
FLAGS
(
sys
.
argv
)
if
len
(
sys
.
argv
)
>
0
:
RecordImporter
.
Import
(
sys
.
argv
[
-
1
])
modules/data/warehouse/tools/mongo_util.py
已删除
100755 → 0
浏览文件 @
7b40ff1d
#!/usr/bin/env python
# -*- coding: UTF-8-*-
###############################################################################
# Copyright 2017 The Apollo Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
"""
MongoDB util.
Requirements: pymongo 3.x
"""
import
os
import
sys
import
gflags
import
google.protobuf.json_format
as
json_format
import
pymongo
gflags
.
DEFINE_string
(
'mongo_host'
,
'127.0.0.1'
,
'MongoDB host ip.'
)
gflags
.
DEFINE_integer
(
'mongo_port'
,
27017
,
'MongoDB port.'
)
gflags
.
DEFINE_string
(
'mongo_db_name'
,
'apollo'
,
'MongoDB db name.'
)
gflags
.
DEFINE_string
(
'mongo_user'
,
None
,
'MongoDB user (optional).'
)
gflags
.
DEFINE_string
(
'mongo_pass'
,
None
,
'MongoDB password (optional).'
)
class
Mongo
(
object
):
"""MongoDB util"""
@
staticmethod
def
db
():
"""Connect to MongoDB instance."""
# Try to read config from environ, and the flags.
G
=
gflags
.
FLAGS
host
=
os
.
environ
.
get
(
'MONGO_HOST'
,
G
.
mongo_host
)
port
=
int
(
os
.
environ
.
get
(
'MONGO_PORT'
,
G
.
mongo_port
))
user
=
os
.
environ
.
get
(
'MONGO_USER'
,
G
.
mongo_user
)
passwd
=
os
.
environ
.
get
(
'MONGO_PASS'
,
G
.
mongo_pass
)
client
=
pymongo
.
MongoClient
(
host
,
port
)
db
=
client
[
G
.
mongo_db_name
]
if
user
and
passwd
:
db
.
authenticate
(
user
,
passwd
)
return
db
@
staticmethod
def
collection
(
collection_name
):
"""
Get collection handler. To use it, please refer
https://api.mongodb.com/python/current/api/pymongo/collection.html
"""
return
Mongo
.
db
()[
collection_name
]
@
staticmethod
def
pb_to_doc
(
pb
):
"""Convert proto to mongo document."""
including_default_value_fields
=
False
preserving_proto_field_name
=
True
return
json_format
.
MessageToDict
(
pb
,
including_default_value_fields
,
preserving_proto_field_name
)
@
staticmethod
def
doc_to_pb
(
doc
,
pb
):
"""Convert mongo document to proto."""
ignore_unknown_fields
=
True
return
json_format
.
ParseDict
(
doc
,
pb
,
ignore_unknown_fields
)
if
__name__
==
'__main__'
:
gflags
.
FLAGS
(
sys
.
argv
)
print
Mongo
.
db
().
collection_names
()
modules/data/warehouse/tools/parse_record.py
已删除
100755 → 0
浏览文件 @
7b40ff1d
#!/usr/bin/env python
# -*- coding: UTF-8-*-
###############################################################################
# Copyright 2018 The Apollo Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
"""
Parse Cyber record into apollo.data.Record.
Use as command tool: parse_record.py <record>
Use as util lib: RecordParser.Parse(<record>)
"""
import
math
import
os
import
sys
import
gflags
import
glog
import
utm
from
cyber.proto.record_pb2
import
Header
from
cyber_py.record
import
RecordReader
from
modules.canbus.proto.chassis_pb2
import
Chassis
from
modules.data.proto.record_pb2
import
Record
from
modules.localization.proto.localization_pb2
import
LocalizationEstimate
gflags
.
DEFINE_float
(
'pos_sample_min_duration'
,
2
,
'In seconds.'
)
gflags
.
DEFINE_float
(
'pos_sample_min_distance'
,
3
,
'In meters.'
)
gflags
.
DEFINE_integer
(
'utm_zone_id'
,
10
,
'UTM zone id.'
)
gflags
.
DEFINE_string
(
'utm_zone_letter'
,
'S'
,
'UTM zone letter.'
)
kChassisChannel
=
'/apollo/canbus/chassis'
kDriveEventChannel
=
'/apollo/drive_event'
kHMIStatusChannel
=
'/apollo/hmi/status'
kLocalizationChannel
=
'/apollo/localization/pose'
def
utm_distance_meters
(
pos0
,
pos1
):
"""Return distance of pos0 and pos1 in meters."""
return
math
.
sqrt
((
pos0
.
x
-
pos1
.
x
)
**
2
+
(
pos0
.
y
-
pos1
.
y
)
**
2
+
(
pos0
.
z
-
pos1
.
z
)
**
2
)
class
RecordParser
(
object
):
"""Wrapper of a Cyber record."""
@
staticmethod
def
Parse
(
record_file
):
"""Simple interface to parse a cyber record."""
parser
=
RecordParser
(
record_file
)
if
not
parser
.
ParseMeta
():
return
None
parser
.
ParseMessages
()
return
parser
.
record
def
__init__
(
self
,
record_file
):
"""Init input reader and output record."""
record_file
=
os
.
path
.
abspath
(
record_file
)
self
.
record
=
Record
(
path
=
record_file
,
dir
=
os
.
path
.
dirname
(
record_file
))
self
.
_reader
=
RecordReader
(
record_file
)
# State during processing messages.
self
.
_current_driving_mode
=
None
self
.
_last_position
=
None
# To sample driving path.
self
.
_last_position_sampled
=
None
self
.
_last_position_sampled_time
=
None
def
ParseMeta
(
self
):
"""
Parse meta info which doesn't need to scan the record.
Currently we parse the record ID, header and channel list here.
"""
self
.
record
.
header
.
ParseFromString
(
self
.
_reader
.
get_headerstring
())
for
chan
in
self
.
_reader
.
get_channellist
():
self
.
record
.
channels
[
chan
]
=
self
.
_reader
.
get_messagenumber
(
chan
)
if
len
(
self
.
record
.
channels
)
==
0
:
glog
.
error
(
'No message found in record'
)
return
False
return
True
def
ParseMessages
(
self
):
"""Process all messages."""
for
channel
,
msg
,
_type
,
timestamp
in
self
.
_reader
.
read_messages
():
if
channel
==
kHMIStatusChannel
:
self
.
ProcessHMIStatus
(
msg
)
elif
channel
==
kDriveEventChannel
:
self
.
ProcessDriveEvent
(
msg
)
elif
channel
==
kChassisChannel
:
self
.
ProcessChassis
(
msg
)
elif
channel
==
kLocalizationChannel
:
self
.
ProcessLocalization
(
msg
)
def
ProcessHMIStatus
(
self
,
msg
):
"""Save HMIStatus."""
# Keep the first message and assume it doesn't change in one recording.
if
not
self
.
record
.
HasField
(
'hmi_status'
):
self
.
record
.
hmi_status
.
ParseFromString
(
msg
)
def
ProcessDriveEvent
(
self
,
msg
):
"""Save DriveEvents."""
self
.
record
.
drive_events
.
add
().
ParseFromString
(
msg
)
def
ProcessChassis
(
self
,
msg
):
"""Process Chassis, save disengagements."""
chassis
=
Chassis
()
chassis
.
ParseFromString
(
msg
)
timestamp
=
chassis
.
header
.
timestamp_sec
if
self
.
_current_driving_mode
==
chassis
.
driving_mode
:
# DrivingMode doesn't change.
return
# Save disengagement.
if
(
self
.
_current_driving_mode
==
Chassis
.
COMPLETE_AUTO_DRIVE
and
chassis
.
driving_mode
==
Chassis
.
EMERGENCY_MODE
):
glog
.
info
(
'Disengagement found at'
,
timestamp
)
disengagement
=
self
.
record
.
disengagements
.
add
(
time
=
timestamp
)
if
self
.
_last_position
is
not
None
:
lat
,
lon
=
utm
.
to_latlon
(
self
.
_last_position
.
x
,
self
.
_last_position
.
y
,
gflags
.
FLAGS
.
utm_zone_id
,
gflags
.
FLAGS
.
utm_zone_letter
)
disengagement
.
location
.
lat
=
lat
disengagement
.
location
.
lon
=
lon
# Update DrivingMode.
self
.
_current_driving_mode
=
chassis
.
driving_mode
def
ProcessLocalization
(
self
,
msg
):
"""Process Localization, stat mileages and save driving path."""
localization
=
LocalizationEstimate
()
localization
.
ParseFromString
(
msg
)
timestamp
=
localization
.
header
.
timestamp_sec
cur_pos
=
localization
.
pose
.
position
# Stat mileages.
if
(
self
.
_last_position
is
not
None
and
self
.
_current_driving_mode
is
not
None
):
driving_mode
=
Chassis
.
DrivingMode
.
Name
(
self
.
_current_driving_mode
)
meters
=
utm_distance_meters
(
self
.
_last_position
,
cur_pos
)
if
driving_mode
in
self
.
record
.
stat
.
mileages
:
self
.
record
.
stat
.
mileages
[
driving_mode
]
+=
meters
else
:
self
.
record
.
stat
.
mileages
[
driving_mode
]
=
meters
# Sample driving path.
G
=
gflags
.
FLAGS
if
(
self
.
_last_position_sampled
is
None
or
(
timestamp
-
self
.
_last_position_sampled_time
>
G
.
pos_sample_min_duration
and
utm_distance_meters
(
self
.
_last_position_sampled
,
cur_pos
)
>
G
.
pos_sample_min_distance
)):
self
.
_last_position_sampled
=
cur_pos
self
.
_last_position_sampled_time
=
timestamp
lat
,
lon
=
utm
.
to_latlon
(
cur_pos
.
x
,
cur_pos
.
y
,
G
.
utm_zone_id
,
G
.
utm_zone_letter
)
self
.
record
.
stat
.
driving_path
.
add
(
lat
=
lat
,
lon
=
lon
)
# Update position.
self
.
_last_position
=
cur_pos
if
__name__
==
'__main__'
:
gflags
.
FLAGS
(
sys
.
argv
)
if
len
(
sys
.
argv
)
>
0
:
print
RecordParser
.
Parse
(
sys
.
argv
[
-
1
])
modules/data/warehouse/web_server/add_pythonpath.py
已删除
100644 → 0
浏览文件 @
7b40ff1d
#!/usr/bin/env python
###############################################################################
# Copyright 2017 The Apollo Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
"""Add required paths to PYTHONPATH."""
import
os
import
sys
sys
.
path
.
append
(
os
.
path
.
join
(
os
.
path
.
dirname
(
__file__
),
'../tools'
))
modules/data/warehouse/web_server/display_util.py
已删除
100755 → 0
浏览文件 @
7b40ff1d
#!/usr/bin/env python
# -*- coding: UTF-8-*-
###############################################################################
# Copyright 2017 The Apollo Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
"""Utils for displaying."""
import
datetime
import
math
import
pytz
import
sys
import
gflags
from
modules.common.proto.drive_event_pb2
import
DriveEvent
gflags
.
DEFINE_string
(
'timezone'
,
'America/Los_Angeles'
,
'Timezone.'
)
def
timestamp_to_time
(
timestamp
):
"""Convert Unix epoch timestamp to readable time."""
dt
=
datetime
.
datetime
.
fromtimestamp
(
timestamp
,
pytz
.
utc
)
local_tz
=
pytz
.
timezone
(
gflags
.
FLAGS
.
timezone
)
return
dt
.
astimezone
(
local_tz
).
strftime
(
'%Y-%m-%d %H:%M:%S'
)
def
timestamp_ns_to_time
(
timestamp
):
"""Convert Unix epoch timestamp to readable time."""
return
timestamp_to_time
(
timestamp
/
1e9
)
def
draw_path_on_gmap
(
driving_path
,
canvas_id
):
"""Draw driving path on Google map."""
if
not
driving_path
:
return
''
# Get center and zoom.
min_lat
,
max_lat
=
sys
.
float_info
.
max
,
-
sys
.
float_info
.
max
min_lng
,
max_lng
=
sys
.
float_info
.
max
,
-
sys
.
float_info
.
max
for
point
in
driving_path
:
if
point
.
lat
>
max_lat
:
max_lat
=
point
.
lat
if
point
.
lat
<
min_lat
:
min_lat
=
point
.
lat
if
point
.
lon
>
max_lng
:
max_lng
=
point
.
lon
if
point
.
lon
<
min_lng
:
min_lng
=
point
.
lon
center_lat
=
(
min_lat
+
max_lat
)
/
2.0
center_lng
=
(
min_lng
+
max_lng
)
/
2.0
zoom
=
int
(
math
.
log
(
1.28
/
(
max_lat
-
min_lat
+
0.001
))
/
math
.
log
(
2.0
))
+
8
result
=
'var gmap = LoadGoogleMap("{}", {}, {}, {});
\n
'
.
format
(
canvas_id
,
center_lat
,
center_lng
,
zoom
)
latlng_list
=
[
'[{},{}]'
.
format
(
point
.
lat
,
point
.
lon
)
for
point
in
driving_path
]
result
+=
'var latlng_list = [{}];
\n
'
.
format
(
','
.
join
(
latlng_list
))
result
+=
'DrawPolyline(gmap, latlng_list, "blue", 2);
\n
'
start
,
end
=
driving_path
[
0
],
driving_path
[
-
1
]
result
+=
'DrawCircle(gmap, {}, {}, 20, "green");
\n
'
.
format
(
start
.
lat
,
start
.
lon
)
result
+=
'DrawCircle(gmap, {}, {}, 20, "red");
\n
'
.
format
(
end
.
lat
,
end
.
lon
)
return
result
def
draw_disengagements_on_gmap
(
record
):
"""Draw disengagements on Google map."""
result
=
''
for
dis
in
record
.
disengagements
:
info
=
'disengage at %.1fs'
%
(
dis
.
time
-
record
.
header
.
begin_time
/
1e9
)
result
+=
'DrawInfoWindow(gmap, {}, {}, "{}");
\n
'
.
format
(
dis
.
location
.
lat
,
dis
.
location
.
lon
,
info
)
return
result
def
readable_data_size
(
num
):
"""Print data size in readable format."""
for
unit
in
[
'B'
,
'KB'
,
'MB'
,
'GB'
,
'TB'
,
'PB'
,
'EB'
,
'ZB'
]:
if
num
<
1024.0
:
return
'%.2f %s'
%
(
num
,
unit
)
num
/=
1024.0
return
"%.2f %s"
%
(
num
,
'YB'
)
def
drive_event_type_name
(
event_type
):
"""Convert DriveEvent type to name."""
return
DriveEvent
.
Type
.
Name
(
event_type
)
# To be registered into jinja2 templates.
utils
=
{
'draw_disengagements_on_gmap'
:
draw_disengagements_on_gmap
,
'draw_path_on_gmap'
:
draw_path_on_gmap
,
'readable_data_size'
:
readable_data_size
,
'timestamp_to_time'
:
timestamp_to_time
,
'timestamp_ns_to_time'
:
timestamp_ns_to_time
,
'drive_event_type_name'
:
drive_event_type_name
,
}
modules/data/warehouse/web_server/main.py
已删除
100755 → 0
浏览文件 @
7b40ff1d
#!/usr/bin/env python
###############################################################################
# Copyright 2017 The Apollo Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
"""Serve data imported in MongoDB."""
import
collections
import
datetime
import
os
import
pickle
import
sys
import
flask
import
gflags
import
gunicorn.app.base
import
pymongo
import
add_pythonpath
import
display_util
import
records_util
from
modules.data.proto.record_pb2
import
Record
from
mongo_util
import
Mongo
gflags
.
DEFINE_string
(
'host'
,
'0.0.0.0'
,
'Web host IP.'
)
gflags
.
DEFINE_integer
(
'port'
,
8000
,
'Web host port.'
)
gflags
.
DEFINE_integer
(
'workers'
,
5
,
'Web host workers.'
)
gflags
.
DEFINE_boolean
(
'debug'
,
False
,
'Enable debug mode.'
)
gflags
.
DEFINE_integer
(
'page_size'
,
20
,
'Search results per page.'
)
gflags
.
DEFINE_string
(
'mongo_collection_name'
,
'records'
,
'MongoDB collection name.'
)
app
=
flask
.
Flask
(
__name__
)
app
.
secret_key
=
str
(
datetime
.
datetime
.
now
())
app
.
jinja_env
.
filters
.
update
(
display_util
.
utils
)
@
app
.
route
(
'/'
)
@
app
.
route
(
'/tasks/<int:page_idx>'
)
def
tasks_hdl
(
page_idx
=
1
):
"""Handler of the task list page."""
G
=
gflags
.
FLAGS
mongo_col
=
Mongo
.
collection
(
G
.
mongo_collection_name
)
task_dirs
=
{
doc
[
'dir'
]
for
doc
in
mongo_col
.
find
({},
{
'dir'
:
1
})}
page_count
=
(
len
(
task_dirs
)
+
G
.
page_size
-
1
)
/
G
.
page_size
if
page_idx
>
page_count
:
flask
.
flash
(
'Page index out of bound'
)
return
flask
.
render_template
(
'base.tpl'
)
offset
=
G
.
page_size
*
(
page_idx
-
1
)
task_dirs
=
sorted
(
list
(
task_dirs
),
reverse
=
True
)
query
=
{
'dir'
:
{
'$in'
:
task_dirs
[
offset
:
offset
+
G
.
page_size
]}}
kFields
=
{
'dir'
:
1
,
'header.begin_time'
:
1
,
'header.end_time'
:
1
,
'header.size'
:
1
,
'hmi_status.current_mode'
:
1
,
'hmi_status.current_map'
:
1
,
'hmi_status.current_vehicle'
:
1
,
'disengagements'
:
1
,
'drive_events'
:
1
,
'stat.mileages'
:
1
,
}
task_records
=
collections
.
defaultdict
(
list
)
for
doc
in
mongo_col
.
find
(
query
,
kFields
):
task_records
[
doc
[
'dir'
]].
append
(
Mongo
.
doc_to_pb
(
doc
,
Record
()))
tasks
=
[
records_util
.
CombineRecords
(
records
)
for
records
in
task_records
.
values
()]
tasks
.
sort
(
key
=
lambda
task
:
task
.
dir
,
reverse
=
True
)
return
flask
.
render_template
(
'records.tpl'
,
page_count
=
page_count
,
current_page
=
page_idx
,
records
=
tasks
,
is_tasks
=
True
)
@
app
.
route
(
'/task/<path:task_path>'
)
def
task_hdl
(
task_path
):
"""Handler of the task detail page."""
mongo_col
=
Mongo
.
collection
(
gflags
.
FLAGS
.
mongo_collection_name
)
docs
=
mongo_col
.
find
({
'dir'
:
os
.
path
.
join
(
'/'
,
task_path
)})
records
=
[
Mongo
.
doc_to_pb
(
doc
,
Record
())
for
doc
in
docs
]
task
=
records_util
.
CombineRecords
(
records
)
return
flask
.
render_template
(
'record.tpl'
,
record
=
task
,
sub_records
=
records
)
@
app
.
route
(
'/records'
)
@
app
.
route
(
'/records/<int:page_idx>'
)
def
records_hdl
(
page_idx
=
1
):
"""Handler of the record list page."""
G
=
gflags
.
FLAGS
kFields
=
{
'path'
:
1
,
'header.begin_time'
:
1
,
'header.end_time'
:
1
,
'header.size'
:
1
,
'hmi_status.current_mode'
:
1
,
'hmi_status.current_map'
:
1
,
'hmi_status.current_vehicle'
:
1
,
'disengagements'
:
1
,
'drive_events'
:
1
,
'stat.mileages'
:
1
,
}
kSort
=
[(
'header.begin_time'
,
pymongo
.
DESCENDING
)]
docs
=
Mongo
.
collection
(
G
.
mongo_collection_name
).
find
({},
kFields
)
page_count
=
(
docs
.
count
()
+
G
.
page_size
-
1
)
/
G
.
page_size
offset
=
G
.
page_size
*
(
page_idx
-
1
)
records
=
[
Mongo
.
doc_to_pb
(
doc
,
Record
())
for
doc
in
docs
.
sort
(
kSort
).
skip
(
offset
).
limit
(
G
.
page_size
)]
return
flask
.
render_template
(
'records.tpl'
,
page_count
=
page_count
,
current_page
=
page_idx
,
records
=
records
)
@
app
.
route
(
'/record/<path:record_path>'
)
def
record_hdl
(
record_path
):
"""Handler of the record detail page."""
mongo_col
=
Mongo
.
collection
(
gflags
.
FLAGS
.
mongo_collection_name
)
doc
=
mongo_col
.
find_one
({
'path'
:
os
.
path
.
join
(
'/'
,
record_path
)})
record
=
Mongo
.
doc_to_pb
(
doc
,
Record
())
return
flask
.
render_template
(
'record.tpl'
,
record
=
record
)
class
FlaskApp
(
gunicorn
.
app
.
base
.
BaseApplication
):
"""A wrapper to run flask app."""
def
__init__
(
self
,
flask_app
):
flask_app
.
debug
=
gflags
.
FLAGS
.
debug
self
.
application
=
flask_app
super
(
FlaskApp
,
self
).
__init__
()
def
load_config
(
self
):
"""Load config."""
G
=
gflags
.
FLAGS
self
.
cfg
.
set
(
'bind'
,
'{}:{}'
.
format
(
G
.
host
,
G
.
port
))
self
.
cfg
.
set
(
'workers'
,
G
.
workers
)
self
.
cfg
.
set
(
'proc_name'
,
'ApolloData'
)
def
load
(
self
):
"""Load app."""
return
self
.
application
if
__name__
==
'__main__'
:
gflags
.
FLAGS
(
sys
.
argv
)
if
gflags
.
FLAGS
.
debug
:
app
.
run
(
gflags
.
FLAGS
.
host
,
gflags
.
FLAGS
.
port
,
gflags
.
FLAGS
.
debug
)
else
:
FlaskApp
(
app
).
run
()
modules/data/warehouse/web_server/records_util.py
已删除
100644 → 0
浏览文件 @
7b40ff1d
#!/usr/bin/env python
# -*- coding: UTF-8-*-
###############################################################################
# Copyright 2018 The Apollo Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################
"""Records utils."""
from
modules.data.proto.record_pb2
import
Record
def
CombineRecords
(
records
):
"""Combine multiple records info to one."""
records
.
sort
(
key
=
lambda
record
:
record
.
header
.
begin_time
)
virtual_record
=
Record
(
path
=
records
[
0
].
dir
,
dir
=
records
[
0
].
dir
)
virtual_record
.
header
.
begin_time
=
records
[
0
].
header
.
begin_time
virtual_record
.
header
.
end_time
=
records
[
-
1
].
header
.
end_time
for
record
in
records
:
virtual_record
.
header
.
size
+=
record
.
header
.
size
channels
=
virtual_record
.
channels
for
channel
,
count
in
record
.
channels
.
iteritems
():
channels
[
channel
]
=
(
channels
.
get
(
channel
)
or
0
)
+
count
if
record
.
hmi_status
.
current_mode
:
virtual_record
.
hmi_status
.
CopyFrom
(
record
.
hmi_status
)
virtual_record
.
disengagements
.
extend
(
record
.
disengagements
)
virtual_record
.
drive_events
.
extend
(
record
.
drive_events
)
mileages
=
virtual_record
.
stat
.
mileages
for
driving_mode
,
miles
in
record
.
stat
.
mileages
.
iteritems
():
mileages
[
driving_mode
]
=
(
mileages
.
get
(
driving_mode
)
or
0
)
+
miles
virtual_record
.
stat
.
driving_path
.
extend
(
record
.
stat
.
driving_path
)
return
virtual_record
modules/data/warehouse/web_server/static/js/gmap_util.js
已删除
100644 → 0
浏览文件 @
7b40ff1d
/******************************************************************************
* Copyright 2017 The Apollo Authors. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*****************************************************************************/
function
LoadGoogleMap
(
canvas_id
,
center_lat
,
center_lng
,
zoom
)
{
var
options
=
{
center
:
new
google
.
maps
.
LatLng
(
center_lat
,
center_lng
),
zoom
:
zoom
,
mapTypeId
:
google
.
maps
.
MapTypeId
.
ROADMAP
};
return
new
google
.
maps
.
Map
(
document
.
getElementById
(
canvas_id
),
options
);
}
function
DrawPolyline
(
map_obj
,
latlng_list
,
color
,
weight
)
{
var
coords
=
latlng_list
.
map
(
function
(
latlng
)
{
return
new
google
.
maps
.
LatLng
(
latlng
[
0
],
latlng
[
1
]);
});
new
google
.
maps
.
Polyline
({
strokeColor
:
color
,
strokeWeight
:
weight
,
path
:
coords
,
clickable
:
false
,
map
:
map_obj
});
}
function
DrawCircle
(
map_obj
,
lat
,
lng
,
radius
,
color
)
{
new
google
.
maps
.
Circle
({
strokeColor
:
color
,
fillOpacity
:
0
,
map
:
map_obj
,
center
:
new
google
.
maps
.
LatLng
(
lat
,
lng
),
radius
:
radius
});
}
function
DrawInfoWindow
(
map_obj
,
lat
,
lng
,
info
)
{
var
infowindow
=
new
google
.
maps
.
InfoWindow
({
content
:
info
});
var
marker
=
new
google
.
maps
.
Marker
({
position
:
{
lat
:
lat
,
lng
:
lng
},
map
:
map_obj
,
title
:
'
Info
'
});
marker
.
addListener
(
'
click
'
,
function
()
{
infowindow
.
open
(
map_obj
,
marker
);});
}
modules/data/warehouse/web_server/templates/base.tpl
已删除
100644 → 0
浏览文件 @
7b40ff1d
{
#
#
Copyright
2017
The
Apollo
Authors
.
All
Rights
Reserved
.
#
#
Licensed
under
the
Apache
License
,
Version
2.0
(
the
"License"
);
#
you
may
not
use
this
file
except
in
compliance
with
the
License
.
#
You
may
obtain
a
copy
of
the
License
at
#
#
http
://
www
.
apache
.
org
/
licenses
/
LICENSE
-
2.0
#
#
Unless
required
by
applicable
law
or
agreed
to
in
writing
,
software
#
distributed
under
the
License
is
distributed
on
an
"AS IS"
BASIS
,
#
WITHOUT
WARRANTIES
OR
CONDITIONS
OF
ANY
KIND
,
either
express
or
implied
.
#
See
the
License
for
the
specific
language
governing
permissions
and
#
limitations
under
the
License
.
#
}
<!DOCTYPE html>
<html>
<head>
<meta
charset=
"utf-8"
>
<meta
name=
"viewport"
content=
"width=device-width, initial-scale=1"
>
<link
rel=
"stylesheet"
href=
"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
>
<script
src=
"https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"
></script>
{% block head %} {% endblock %}
</head>
<body>
<div
class=
"container"
>
<nav
class=
"navbar navbar-default"
>
<div
class=
"container-fluid"
>
<div
class=
"navbar-header"
><a
class=
"navbar-brand"
href=
"/"
>
Apollo Data
</a></div>
<ul
class=
"nav navbar-nav"
>
<li><a
href=
"{{ url_for('tasks_hdl') }}"
>
Tasks
</a></li>
<li><a
href=
"{{ url_for('records_hdl') }}"
>
Records
</a></li>
</ul>
</div>
</nav>
{% for msg in get_flashed_messages() %}
<div
class=
"alert alert-warning"
>
<button
type=
"button"
class=
"close"
data-dismiss=
"alert"
>
×
</button>
{{ msg }}
</div>
{% endfor %}
{% block body %}
{% endblock %}
</div>
<script
src=
"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"
></script>
</body>
</html>
modules/data/warehouse/web_server/templates/record.tpl
已删除
100644 → 0
浏览文件 @
7b40ff1d
{
#
#
Copyright
2018
The
Apollo
Authors
.
All
Rights
Reserved
.
#
#
Licensed
under
the
Apache
License
,
Version
2.0
(
the
"License"
);
#
you
may
not
use
this
file
except
in
compliance
with
the
License
.
#
You
may
obtain
a
copy
of
the
License
at
#
#
http
://
www
.
apache
.
org
/
licenses
/
LICENSE
-
2.0
#
#
Unless
required
by
applicable
law
or
agreed
to
in
writing
,
software
#
distributed
under
the
License
is
distributed
on
an
"AS IS"
BASIS
,
#
WITHOUT
WARRANTIES
OR
CONDITIONS
OF
ANY
KIND
,
either
express
or
implied
.
#
See
the
License
for
the
specific
language
governing
permissions
and
#
limitations
under
the
License
.
#
}
{% extends "base.tpl" %}
{% block head %}
<title>
Apollo Data - Record - {{ record.path }}
</title>
<style>
.green
{
color
:
green
;
}
.red
{
color
:
red
;
}
.text_center
{
text
-
align
:
center
;
}
</style>
{% endblock %}
{% block body %}
<div
class=
"panel panel-default"
>
<div
class=
"panel-heading"
>
Information
</div>
<div
class=
"panel-body"
>
<table
class=
"table table-striped"
>
<thead>
<tr>
<th>
Mode
</th>
<th>
Map
</th>
<th>
Vehicle
</th>
<th>
Begin Time
</th>
<th>
Duration
</th>
<th>
Size
</th>
<th>
Mileage (Auto/Total)
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
{{ record.hmi_status.current_mode }}
</td>
<td>
{{ record.hmi_status.current_map }}
</td>
<td>
{{ record.hmi_status.current_vehicle }}
</td>
<td>
{{ record.header.begin_time | timestamp_ns_to_time }}
</td>
<td>
{{ ((record.header.end_time - record.header.begin_time) / 1000000000.0) | round(1) }} s
</td>
<td>
{{ record.header.size | readable_data_size }}
</td>
<td>
{{ record.stat.mileages['COMPLETE_AUTO_DRIVE'] | int }} / {{ record.stat.mileages.values() | sum | int }} m
</td>
</tr>
</tbody>
</table>
{
#
Draw
map
path
.
#
}
{% if record.stat.driving_path %}
<script
type=
"text/javascript"
src=
"http://maps.google.com/maps/api/js?sensor=false&key=AIzaSyC2THXHPs0lkchGfcUOHTm-aVujoBHh2Sc"
></script>
<script
type=
"text/javascript"
src=
"{{ url_for('static', filename='js/gmap_util.js') }}"
></script>
<div
style=
"width:100%; height:350px;"
>
<div
id=
"gmap_canvas"
style=
"width: 100%; height: 100%;"
></div>
<script>
{{
record
.
stat
.
driving_path
|
draw_path_on_gmap
(
'
gmap_canvas
'
)
}}
{{
record
|
draw_disengagements_on_gmap
}}
</script>
</div>
<table
class=
"table text_center"
>
<tbody>
<tr>
<td><span
class=
"glyphicon glyphicon-record green"
></span>
Start Point
</td>
<td><span
class=
"glyphicon glyphicon-record red"
></span>
Stop Point
</td>
<td><span
class=
"glyphicon glyphicon-map-marker red"
></span>
Disengagement
</td>
</tr>
</tbody>
</table>
{% endif %}
</div>
</div>
{% if sub_records %}
<div
class=
"panel panel-default"
>
<div
class=
"panel-heading"
>
Records
</div>
<div
class=
"panel-body"
>
<table
class=
"table table-striped"
>
<thead>
<tr>
<th>
Begin Time
</th>
<th>
Duration
</th>
<th>
Size
</th>
<th>
Path
</th>
<th></th>
</tr>
</thead>
<tbody>
{% for record in sub_records %}
<tr>
<td>
{{ record.header.begin_time | timestamp_ns_to_time }}
</td>
<td>
{{ ((record.header.end_time - record.header.begin_time) / 1000000000.0) | round(1) }} s
</td>
<td>
{{ record.header.size | readable_data_size }}
</td>
<td>
{{ record.path }}
</td>
<td><a
href=
"{{ url_for('record_hdl', record_path=record.path[1:]) }}"
>
View
</a></td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endif %}
{% if record.disengagements %}
<div
class=
"panel panel-default"
>
<div
class=
"panel-heading"
>
Disengagements
</div>
<div
class=
"panel-body"
>
<table
class=
"table table-striped"
>
<thead>
<tr>
<th>
Time
</th>
<th>
Description
</th>
<th>
Offset
</th>
</tr>
</thead>
<tbody>
{% for dis in record.disengagements %}
<tr>
<td>
{{ dis.time | timestamp_to_time }}
</td>
<td>
{{ dis.desc }}
</td>
<td>
{{ (dis.time - record.header.begin_time / 1000000000.0) | round(1) }} s
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endif %}
{% if record.drive_events %}
<div
class=
"panel panel-default"
>
<div
class=
"panel-heading"
>
Drive Events
</div>
<div
class=
"panel-body"
>
<table
class=
"table table-striped"
>
<thead>
<tr>
<th>
Time
</th>
<th>
Type
</th>
<th>
Description
</th>
<th>
Offset
</th>
</tr>
</thead>
<tbody>
{% for event in record.drive_events %}
<tr>
<td>
{{ event.header.timestamp_sec | timestamp_to_time }}
</td>
<td>
{% for type in event.type %} {{ type | drive_event_type_name }} {% endfor %}
</td>
<td>
{{ event.event }}
</td>
<td>
{{ (event.header.timestamp_sec - record.header.begin_time / 1000000000.0) | round(1) }} s
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endif %}
<div
class=
"panel panel-default"
>
<div
class=
"panel-heading"
>
Channels
</div>
<div
class=
"panel-body"
>
<table
class=
"table table-striped"
>
<thead>
<tr>
<th>
Name
</th>
<th>
Messages
</th>
</tr>
</thead>
<tbody>
{% for channel, msg_count in record.channels.iteritems() %}
<tr>
<td>
{{ channel }}
</td>
<td>
{{ msg_count }}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endblock %}
modules/data/warehouse/web_server/templates/records.tpl
已删除
100644 → 0
浏览文件 @
7b40ff1d
{
#
#
Copyright
2018
The
Apollo
Authors
.
All
Rights
Reserved
.
#
#
Licensed
under
the
Apache
License
,
Version
2.0
(
the
"License"
);
#
you
may
not
use
this
file
except
in
compliance
with
the
License
.
#
You
may
obtain
a
copy
of
the
License
at
#
#
http
://
www
.
apache
.
org
/
licenses
/
LICENSE
-
2.0
#
#
Unless
required
by
applicable
law
or
agreed
to
in
writing
,
software
#
distributed
under
the
License
is
distributed
on
an
"AS IS"
BASIS
,
#
WITHOUT
WARRANTIES
OR
CONDITIONS
OF
ANY
KIND
,
either
express
or
implied
.
#
See
the
License
for
the
specific
language
governing
permissions
and
#
limitations
under
the
License
.
#
}
{% extends "base.tpl" %}
{% block head %}
<title>
Apollo Data
</title>
<style>
.topic_options
{
min-width
:
500px
;
font-size
:
16px
;
}
.page_button
{
width
:
40px
;
}
</style>
{% endblock %}
{% block body %}
<div
class=
"panel panel-default"
>
<div
class=
"panel-heading"
>
{% if is_tasks %} Tasks {% else %} Records {% endif %}
</div>
<div
class=
"panel-body"
>
<table
class=
"table table-striped"
>
<thead>
<tr>
<th>
Mode
</th>
<th>
Map
</th>
<th>
Vehicle
</th>
<th>
Begin Time
</th>
<th>
Duration
</th>
<th>
Size
</th>
<th>
Issues
</th>
<th>
Mileage (Auto/Total)
</th>
<th></th>
</tr>
</thead>
<tbody>
{% for record in records %}
<tr>
<td>
{{ record.hmi_status.current_mode }}
</td>
<td>
{{ record.hmi_status.current_map }}
</td>
<td>
{{ record.hmi_status.current_vehicle }}
</td>
<td>
{{ record.header.begin_time | timestamp_ns_to_time }}
</td>
<td>
{{ ((record.header.end_time - record.header.begin_time) / 1000000000.0) | round(1) }} s
</td>
<td>
{{ record.header.size | readable_data_size }}
</td>
<td>
{{ (record.disengagements | length) + (record.drive_events | length) }}
</td>
<td>
{{ record.stat.mileages['COMPLETE_AUTO_DRIVE'] | int }} / {{ record.stat.mileages.values() | sum | int }} m
</td>
<td><a
target=
"_blank"
{%
if
is_tasks
%}
href=
"{{ url_for('task_hdl', task_path=record.path[1:]) }}"
{%
else
%}
href=
"{{ url_for('record_hdl', record_path=record.path[1:]) }}"
{%
endif
%}
>
View
</a></td>
</tr>
{% endfor %}
</tbody>
</table>
{
#
Pagination
#
}
{% for page_idx in range(1, page_count + 1) %}
<a
class=
"btn btn-default page_button"
role=
"button"
{%
if
page_idx
!=
current_page
%}
{%
if
is_tasks
%}
href=
"{{ url_for('tasks_hdl', page_idx=page_idx) }}"
{%
else
%}
href=
"{{ url_for('records_hdl', page_idx=page_idx) }}"
{%
endif
%}
{%
else
%}
disabled
{%
endif
%}
>
{{ page_idx }}
</a>
{% endfor %}
</div>
</div>
{% endblock %}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录