提交 7bd4b423 编写于 作者: A aodongbiao

添加rom、ram分析工具

Signed-off-by: Naodongbiao <aodongbiao@huawei.com>
上级 81e80bc5
[toc]
# Rom_analyzer.py
## 功能介绍
基于BUILD.gn、bundle.json、编译产物system_module_info.json、out/{product_name}/packages/phone目录下的编译产物,分析各子系统及部件的rom大小。
结果以json与xls格式进行存储,其中,json格式是必输出的,xls格式需要-e参数控制。
## 使用说明
前置条件:
1. 获取整个rom_ram_analyzer目录
1. 对系统进行编译
1. linux平台
1. python3.8及以后
1. 安装requirements
```txt
xlwt==1.3.0
```
命令介绍:
1. `-h``--help`命令查看帮助
```shell
> python3 rom_analyzer.py -h
usage: rom_analyzer.py [-h] [-v] -p PROJECT_PATH -j MODULE_INFO_JSON -n PRODUCT_NAME -d PRODUCT_DIR [-o OUTPUT_FILE] [-e EXCEL]
analyze rom size of component.
optional arguments:
-h, --help show this help message and exit
-v, -version show program\'s version number and exit
-p PROJECT_PATH, --project_path PROJECT_PATH
root path of openharmony. eg: -p ~/openharmony
-j MODULE_INFO_JSON, --module_info_json MODULE_INFO_JSON
path of out/{product_name}/packages/phone/system_module_info.json
-n PRODUCT_NAME, --product_name PRODUCT_NAME
product name. eg: -n rk3568
-d PRODUCT_DIR, --product_dir PRODUCT_DIR
subdirectories of out/{product_name}/packages/phone to be counted.eg: -d system -d vendor
-o OUTPUT_FILE, --output_file OUTPUT_FILE
basename of output file, default: rom_analysis_result. eg: demo/rom_analysis_result
-e EXCEL, --excel EXCEL
if output result as excel, default: False. eg: -e True
```
1. 使用示例
```shell
python3 rom_analyzer.py -p ~/nomodify_oh/ -j ../system_module_info.json -n rk3568 -d system -d vendor -d updater -o demo/demo -e True
# oh:rootpath of openharmony
# rk3568: product_name, same as out/{product_name}
# demo/demo: path of output file, where the second 'demo' is the basename of output file
# -e True:output result in excel format additionally
```
## 输出格式说明(json)
```json
{
子系统名: {
"size": 整个子系统输出文件的总大小,
"file_count": 整个子系统产生的文件数,
输出文件名: 本文件的大小,
...
},
...
}
```
# ram_analyzer.py
## 功能介绍
基于out/{product_name}/packages/phone下所有cfg文件、out/{product_name}/packages/phone/system/profile下所有xml文件,分析各进程及对应部件的ram占用(默认取Pss)
结果以json与xls格式存储,其中,json格式是必输出的,xls格式需要-e参数控制。
## 使用说明
前置条件:
1. 获取整个rom_ram_analyzer目录
2. hdc可用
2. 设备已连接
3. python3.8及以后
4. 安装requirements
```txt
xlwt==1.3.0
```
5. 准备好相关数据:
1. out/{product_name}/packages/phone下所有cfg文件,并将其放置于同一个目录中(ps:同名文件仅保存一份即可)
1. out/{product_name}/packages/phone/system/profile下所有xml文件
6. 运行rom_analyzer.py产生的json结果一份(即-o参数对应的文件,默认rom_analysis_result.json)
命令介绍:
1. 使用`-h``--help`查看帮助
```shell
> python .\ram_analyzer.py -h
usage: ram_analyzer.py [-h] [-v] -x XML_PATH -c CFG_PATH [-j ROM_RESULT] -n DEVICE_NUM [-o OUTPUT_FILENAME] [-e EXCEL]
analyze ram size of component
optional arguments:
-h, --help show this help message and exit
-v, -version show program\'s version number and exit
-x XML_PATH, --xml_path XML_PATH
path of xml file. eg: -x ~/openharmony/out/rk3568/packages/phone/system/profile
-c CFG_PATH, --cfg_path CFG_PATH
path of cfg files. eg: -c ./cfgs/
-j ROM_RESULT, --rom_result ROM_RESULT
json file produced by rom_analyzer_v1.0.py, default: ./rom_analysis_result.json.eg: -j ./demo/rom_analysis_result.json
-n DEVICE_NUM, --device_num DEVICE_NUM
device number to be collect hidumper info. eg: -n 7001005458323933328a01fce16d3800
-o OUTPUT_FILENAME, --output_filename OUTPUT_FILENAME
base name of output file, default: rom_analysis_result. eg: -o ram_analysis_result
-e EXCEL, --excel EXCEL
if output result as excel, default: False. eg: -e True
```
2. 使用示例:
```shell
python .\ram_analyzer.py -x .\profile\ -c .\init\ -n 7001005458323933328a01fce16d3800 -j .\rom_analysis_result.json -o /demo/demo -e True
# demo/demo: path of output file, where the second 'demo' is the basename of output file
# -e True:output result in excel format additionally
```
## 输出格式说明(json)
```json
{
进程名:{
"size": 本进程占用内存的大小,
部件名: {
elf文件名: elf文件大小
...
}
...
},
...
}
```
\ No newline at end of file
import sys
import typing
import os
from pathlib import Path
from typing import *
class BasicTool:
@classmethod
def find_all_files(cls, folder: str, real_path: bool = True, apply_abs: bool = True, de_duplicate: bool = True,
p_filter: typing.Callable = lambda x: True) -> list:
filepath_list = set()
for root, _, file_names in os.walk(folder):
filepath_list.update(
[os.path.abspath(os.path.realpath(
os.path.join(root, f) if real_path else os.path.join(root, f))) if apply_abs else os.path.relpath(
os.path.realpath(os.path.join(root, f) if real_path else os.path.join(root, f))) for f in file_names
if p_filter(os.path.join(root, f))])
if de_duplicate:
filepath_list = set(filepath_list)
filepath_list = sorted(filepath_list, key=str.lower)
return filepath_list
@classmethod
def get_abs_path(cls, path: str) -> str:
return os.path.abspath(os.path.expanduser(path))
if __name__ == '__main__':
# print(BasicTool.get_abs_path("~/git/.."))
for i in BasicTool.find_all_files(".", apply_abs=False):
print(i)
\ No newline at end of file
import os
import json
class GnCommonTool:
"""
处理BUILD.gn文件的通用方法
"""
@classmethod
def is_gn_variable(cls, target: str, has_quote: bool = True):
"""
判断target是否是gn中的变量:
规则:如果是有引号的模式,则没有引号均认为是变量,有引号的情况下,如有是"$xxx"的模式,则认为xxx是变量;如果是无引号模式,则只要$开头就认为是变量
b = "xxx"
c = b
c = "${b}"
"$p"
"""
target = target.strip()
if not has_quote:
return target.startswith("$")
if target.startswith('"') and target.endswith('"'):
target = target.strip('"')
if target.startswith("${") and target.endswith("}"):
return True
elif target.startswith("$"):
return True
return False
else:
return True
# 给__find_variables_in_gn用的,减少io
__var_val_mem_dict = dict()
@classmethod
def find_variables_in_gn(cls, var_name_tuple: tuple, path: str, stop_tail: str = "home") -> tuple:
"""
同时查找多个gn变量的值
var_name_tuple:变量名的tuple,变量名应是未经过处理后的,如:
xxx
"${xxx}"
"$xxx"
"""
if os.path.isfile(path):
path = os.path.split(path)[0]
var_val_dict = dict()
not_found_count = len(var_name_tuple)
for var in var_name_tuple:
val = GnCommonTool.__var_val_mem_dict.get(var)
if val is not None:
not_found_count -= 1
var_val_dict[var] = val
while not path.endswith(stop_tail) and not_found_count != 0:
for v in var_name_tuple:
cmd = r"grep -Ern '^( *){} *= *\".*?\"' --include=*.gn* {}| grep -Ev '\$' | head -n 1 | grep -E '\".*\"' -wo".format(
v.strip('"').lstrip("${").rstrip('}'), path)
output = os.popen(cmd).read().strip().strip('"')
if len(output) != 0:
not_found_count -= 1
var_val_dict[v] = output
GnCommonTool.__var_val_mem_dict[v] = output
path = os.path.split(path)[0]
return tuple(var_val_dict.values())
@classmethod
def __find_part_subsystem_from_bundle(cls, gnpath: str, stop_tail: str = "home") -> tuple:
"""
根据BUILD.gn的全路径,一层层往上面查找bundle.json文件,
并从bundle.json中查找part_name和subsystem
"""
filename = "bundle.json"
part_name = None
subsystem_name = None
if stop_tail not in gnpath:
return part_name, subsystem_name
if os.path.isfile(gnpath):
gnpath = os.path.split(gnpath)[0]
while not gnpath.endswith(stop_tail):
bundle_path = os.path.join(gnpath, filename)
if not os.path.isfile(bundle_path): # 如果该文件不在该目录下
gnpath = os.path.split(gnpath)[0]
continue
with open(bundle_path, 'r', encoding='utf-8') as f:
content = json.load(f)
try:
part_name = content["component"]["name"]
subsystem_name = content["component"]["subsystem"]
except KeyError:
...
finally:
break
part_name = None if (part_name is not None and len(
part_name) == 0) else part_name
subsystem_name = None if (subsystem_name is not None and len(
subsystem_name) == 0) else subsystem_name
return part_name, subsystem_name
@classmethod
def find_part_subsystem(cls, gn_file: str, project_path: str) -> tuple:
"""
查找gn_file对应的part_name和subsystem
如果在gn中找不到,就到bundle.json中去找
"""
part_name = None
subsystem_name = None
part_var_flag = False # 标识这个变量从gn中取出的原始值是不是变量
subsystem_var_flag = False
var_list = list()
part_name_pattern = r"part_name *=\s*\S*"
subsystem_pattern = r"subsystem_name *=\s*\S*"
meta_grep_pattern = "grep -E '{}' {} | head -n 1"
part_cmd = meta_grep_pattern.format(part_name_pattern, gn_file)
subsystem_cmd = meta_grep_pattern.format(subsystem_pattern, gn_file)
part = os.popen(part_cmd).read().strip()
if len(part) != 0:
part = part.split('=')[-1].strip()
if GnCommonTool.is_gn_variable(part):
part_var_flag = True
var_list.append(part)
else:
part_name = part.strip('"')
if len(part_name) == 0:
part_name = None
subsystem = os.popen(subsystem_cmd).read().strip()
if len(subsystem) != 0: # 这里是只是看有没有grep到关键字
subsystem = subsystem.split('=')[-1].strip()
if GnCommonTool.is_gn_variable(subsystem):
subsystem_var_flag = True
var_list.append(subsystem)
else:
subsystem_name = subsystem.strip('"')
if len(subsystem_name) == 0:
subsystem_name = None
if part_var_flag and subsystem_var_flag:
part_name, subsystem_name = GnCommonTool.find_variables_in_gn(
tuple(var_list), gn_file, project_path)
elif part_var_flag:
t = GnCommonTool.find_variables_in_gn(
tuple(var_list), gn_file, project_path)[0]
part_name = t if t is not None and len(t) != 0 else part_name
elif subsystem_var_flag:
t = GnCommonTool.find_variables_in_gn(
tuple(var_list), gn_file, project_path)[0]
subsystem_name = t if t is not None and len(
t) != 0 else subsystem_name
if part_name is not None and subsystem_name is not None:
return part_name, subsystem_name
# 如果有一个没有找到,就要一层层去找bundle.json文件
t_part_name, t_subsystem_name = cls.__find_part_subsystem_from_bundle(
gn_file, stop_tail=project_path)
if t_part_name is not None:
part_name = t_part_name
if t_subsystem_name is not None:
subsystem_name = t_subsystem_name
return part_name, subsystem_name
import xlwt
from xlwt import Worksheet
import typing
from typing import Optional
from collections.abc import Iterable
class SimpleExcelWriter:
def __init__(self, default_sheet_name: str = "sheet1"):
self.__book = xlwt.Workbook(encoding='utf-8', style_compression=0)
self.__sheet_dict = {
default_sheet_name: self.__book.add_sheet(
sheetname=default_sheet_name, cell_overwrite_ok=True)
}
self.__sheet_pos = {
default_sheet_name: (0, 0) # 记录各个sheet页已经写到什么位置了,当前值为还没有写的
}
self.__default_sheet_name = default_sheet_name
# 表头样式
self.__head_style = xlwt.XFStyle()
# 内容样式
self.__content_style = xlwt.XFStyle()
# 字体
font = xlwt.Font()
font.bold = True
# 居中对齐
alignment = xlwt.Alignment()
alignment.horz = xlwt.Alignment.HORZ_CENTER # 水平方向
alignment.vert = xlwt.Alignment.VERT_CENTER # 垂直方向
# 设置背景颜色
pattern = xlwt.Pattern()
pattern.pattern = xlwt.Pattern.SOLID_PATTERN
pattern.pattern_fore_colour = 22 # 背景颜色
self.__head_style.font = font
self.__head_style.alignment = alignment
self.__head_style.pattern = pattern
self.__content_style.alignment = alignment
def __increment_y(self, sheet_name: str, value: int = 1) -> int:
if sheet_name in self.__sheet_pos.keys():
x, y = self.__sheet_pos.get(sheet_name)
y = y + value
self.__sheet_pos[sheet_name] = (x, y)
return y
def __increment_x(self, sheet_name: str, value: int = 1) -> int:
if sheet_name in self.__sheet_pos.keys():
x, y = self.__sheet_pos.get(sheet_name)
x = x + value
self.__sheet_pos[sheet_name] = (x, 0)
return x
def append_line(self, content: list, sheet_name: str = None):
sheet_name = self.__default_sheet_name if sheet_name is None else sheet_name
if sheet_name not in self.__sheet_dict.keys():
print("error: sheet name '{}' not exist".format(sheet_name))
return
sheet: Worksheet = self.__sheet_dict.get(sheet_name)
x, y = self.__sheet_pos.get(sheet_name)
for ele in content:
sheet.write(x, y, ele, style=self.__content_style)
y = self.__increment_y(sheet_name)
self.__increment_x(sheet_name)
def write_merge(self, x0: int, y0: int, x1: int, y1: int, content: typing.Any,
sheet_name: str = None):
sheet_name = self.__default_sheet_name if sheet_name is None else sheet_name
if sheet_name not in self.__sheet_dict.keys():
print("error: sheet name '{}' not exist".format(sheet_name))
return
sheet: Worksheet = self.__sheet_dict.get(sheet_name)
sheet.write_merge(x0, x1, y0, y1, content, style=self.__content_style)
def set_sheet_header(self, headers: Iterable, sheet_name: str = None):
"""
给sheet页设置表头
"""
sheet_name = self.__default_sheet_name if sheet_name is None else sheet_name
if sheet_name not in self.__sheet_dict.keys():
print("error: sheet name '{}' not exist".format(sheet_name))
return
x, y = self.__sheet_pos.get(sheet_name)
if x != 0 or y != 0:
print(
"error: pos of sheet '{}' is not (0,0). set_sheet_header must before write".format(sheet_name))
return
sheet: Worksheet = self.__sheet_dict.get(sheet_name)
for h in headers:
sheet.write(x, y, h, self.__head_style)
y = self.__increment_y(sheet_name)
self.__increment_x(sheet_name)
def add_sheet(self, sheet_name: str, cell_overwrite_ok=True) -> Optional[xlwt.Worksheet]:
if sheet_name in self.__sheet_dict.keys():
print("error: sheet name '{}' has exist".format(sheet_name))
return
self.__sheet_dict[sheet_name] = self.__book.add_sheet(
sheetname=sheet_name, cell_overwrite_ok=cell_overwrite_ok)
self.__sheet_pos[sheet_name] = (0, 0)
return self.__sheet_dict.get(sheet_name)
def save(self, file_name: str):
self.__book.save(file_name)
if __name__ == '__main__':
writer = SimpleExcelWriter(default_sheet_name="first")
writer.add_sheet("second")
writer.add_sheet("third")
writer.set_sheet_header(["h", "m", "n"])
writer.append_line([1, 2, 3])
writer.append_line([2, 3, 4], "second")
writer.append_line([3, 4, 5], "third")
writer.append_line([3, 2, 1])
writer.save("demo.xls")
此差异已折叠。
import argparse
import json
import os
import sys
import typing
from copy import deepcopy
from typing import *
from packages.basic_tool import BasicTool
from packages.gn_common_tool import GnCommonTool
from packages.simple_excel_writer import SimpleExcelWriter
debug = bool(sys.gettrace())
class RomAnalyzer:
@classmethod
def __collect_product_info(cls, system_module_info_json: Text,
project_path: Text) -> Dict[Text, Dict[Text, Text]]:
"""
根据system_module_info.json生成target字典
"""
with open(system_module_info_json, 'r', encoding='utf-8') as f:
product_list = json.loads(f.read())
project_path = BasicTool.get_abs_path(project_path)
product_info_dict: Dict[Text, Dict[Text, Text]] = dict()
for unit in product_list:
dest: List = unit.get("dest")
if dest is None:
print("warning: keyword 'dest' not found in {}".format(system_module_info_json))
continue
label: Text = unit.get("label")
gn_path = component_name = subsystem_name = None
if label:
gn_path = os.path.join(project_path, label.split(':')[0].lstrip('/'), "BUILD.gn")
component_name, subsystem_name = GnCommonTool.find_part_subsystem(gn_path, project_path)
else:
print("warning: keyword 'label' not found in {}".format(unit))
for target in dest:
product_info_dict[target] = {
"component_name": component_name,
"subsystem_name": subsystem_name,
"gn_path": gn_path,
}
return product_info_dict
@classmethod
def __save_result_as_excel(cls, result_dict: dict, output_name: str):
header = ["subsystem_name", "component_name", "output_file", "size(Byte)"]
tmp_dict = deepcopy(result_dict)
excel_writer = SimpleExcelWriter("rom")
excel_writer.set_sheet_header(headers=header)
subsystem_start_row = 1
subsystem_end_row = 0
subsystem_col = 0
component_start_row = 1
component_end_row = 0
component_col = 1
for subsystem_name in tmp_dict.keys():
subsystem_dict = tmp_dict.get(subsystem_name)
subsystem_size = subsystem_dict.get("size")
subsystem_file_count = subsystem_dict.get("file_count")
del subsystem_dict["file_count"]
del subsystem_dict["size"]
subsystem_end_row += subsystem_file_count
for component_name in subsystem_dict.keys():
component_dict: Dict[str, int] = subsystem_dict.get(component_name)
component_size = component_dict.get("size")
component_file_count = component_dict.get("file_count")
del component_dict["file_count"]
del component_dict["size"]
component_end_row += component_file_count
for file_name, size in component_dict.items():
excel_writer.append_line(
[subsystem_name, component_name, file_name, size])
excel_writer.write_merge(component_start_row, component_col, component_end_row, component_col,
component_name)
component_start_row = component_end_row + 1
excel_writer.write_merge(subsystem_start_row, subsystem_col, subsystem_end_row, subsystem_col,
subsystem_name)
subsystem_start_row = subsystem_end_row + 1
excel_writer.save(output_name + ".xls")
@classmethod
def __put(cls, unit: typing.Dict[Text, Any], result_dict: typing.Dict[Text, Dict]):
"""
{
subsystem_name:{
component_name: {
file_name: file_size
}
}
}
"""
component_name = "others" if unit.get("component_name") is None else unit.get("component_name")
subsystem_name = "others" if unit.get("subsystem_name") is None else unit.get("subsystem_name")
size = unit.get("size")
relative_filepath = unit.get("relative_filepath")
if result_dict.get(subsystem_name) is None:
result_dict[subsystem_name] = dict()
result_dict[subsystem_name]["size"] = 0
result_dict[subsystem_name]["file_count"] = 0
if result_dict.get(subsystem_name).get(component_name) is None:
result_dict[subsystem_name][component_name] = dict()
result_dict[subsystem_name][component_name]["size"] = 0
result_dict[subsystem_name][component_name]["file_count"] = 0
result_dict[subsystem_name]["size"] += size
result_dict[subsystem_name]["file_count"] += 1
result_dict[subsystem_name][component_name]["size"] += size
result_dict[subsystem_name][component_name]["file_count"] += 1
result_dict[subsystem_name][component_name][relative_filepath] = size
@classmethod
def analysis(cls, system_module_info_json: Text, product_dirs: List[str],
project_path: Text, product_name: Text, output_file: Text, output_execel: bool):
"""
system_module_info_json: json文件
product_dirs:要处理的产物的路径列表如["vendor", "system/"]
project_path: 项目根路径
product_name: eg,rk3568
output_file: basename of output file
"""
project_path = BasicTool.get_abs_path(project_path)
phone_dir = os.path.join(project_path, "out", product_name, "packages", "phone")
product_dirs = [os.path.join(phone_dir, d) for d in product_dirs]
product_info_dict = cls.__collect_product_info(system_module_info_json, project_path) # 所有产物信息
result_dict: Dict[Text:Dict] = dict()
for d in product_dirs:
file_list: List[Text] = BasicTool.find_all_files(d)
for f in file_list:
size = os.path.getsize(f)
relative_filepath = f.replace(phone_dir, "").lstrip(os.sep)
unit: Dict[Text, Any] = product_info_dict.get(relative_filepath)
if unit is None:
unit = {
"relative_filepath": relative_filepath,
}
unit["size"] = size
unit["relative_filepath"] = relative_filepath
cls.__put(unit, result_dict)
output_dir, _ = os.path.split(output_file)
if len(output_dir) != 0:
os.makedirs(output_dir, exist_ok=True)
with open(output_file + ".json", 'w', encoding='utf-8') as f:
f.write(json.dumps(result_dict, indent=4))
if output_execel:
cls.__save_result_as_excel(result_dict, output_file)
def get_args():
VERSION = 2.0
parser = argparse.ArgumentParser(
description=f"analyze rom size of component.\n")
parser.add_argument("-v", "-version", action="version",
version=f"version {VERSION}")
parser.add_argument("-p", "--project_path", type=str, required=True,
help="root path of openharmony. eg: -p ~/openharmony")
parser.add_argument("-j", "--module_info_json", required=True, type=str,
help="path of out/{product_name}/packages/phone/system_module_info.json")
parser.add_argument("-n", "--product_name", required=True, type=str, help="product name. eg: -n rk3568")
parser.add_argument("-d", "--product_dir", required=True, action="append",
help="subdirectories of out/{product_name}/packages/phone to be counted."
"eg: -d system -d vendor")
parser.add_argument("-o", "--output_file", type=str, default="rom_analysis_result",
help="basename of output file, default: rom_analysis_result. eg: demo/rom_analysis_result")
parser.add_argument("-e", "--excel", type=bool, default=False,
help="if output result as excel, default: False. eg: -e True")
args = parser.parse_args()
return args
if __name__ == '__main__':
args = get_args()
module_info_json = args.module_info_json
project_path = args.project_path
product_name = args.product_name
product_dirs = args.product_dir
output_file = args.output_file
output_excel = args.excel
RomAnalyzer.analysis(module_info_json, product_dirs, project_path, product_name, output_file, output_excel)
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册