Skip to content

O
openEuler-Advisor

openeuler / openEuler-Advisor

通知 32
Star 4
Fork 4
O
openEuler-Advisor

体验新版 GitCode，发现更多精彩内容 >>

提交 4a082cd0 编写于 作者: Y Yiru Wang Mac
浏览文件
操作

modify search database logic based on new database structure

上级 e8ba7df5
展开全部 隐藏空白更改
内联 并排
Showing with 1094 addition and 1424 deletion
packageship/README.md
浏览文件 @ 4a082cd0
# pkgmnt
#### 介绍
pkgmnt希望提供软件包依赖，生命周期，补丁查询等功能。
1.软件包依赖：方便社区人员在新引入、软件包更新和删除的时候能方便的了解软件的影响范围。
2.生命周期管理：跟踪upstream软件包发布状态，方便维护人员了解当前软件状态，及时升级到合理的版本。
3.补丁查询：方便社区人员了解openEuler软件包的补丁情况，方便的提取补丁内容（待规划）
#### 软件架构
系统采用flask-restful开发，使用SQLAlchemy ORM查询框架，同时支持mysql和sqlite数据库，通过配置文件的
形式进行更改
#### 安装教程
1. 安装系统的依赖包
pip install -r requirements.txt
2. 执行打包命令，打包命令行工具,其中（pkgship）为命令行的名称，可以随意更改
2.1 打包生成 .spec打包文件
pyinstaller -F -n pkgship cli.py
2.2 修改 .spec打包文件，将hiddenimports中加入如下配置
hiddenimports=['pkg_resources.py2_warn']
2.3 生成二进制命令文件
pyinstaller pkgship.spec
2.4 二进制命令文件拷贝至可运行目录
cp dist/pkgship /usr/local/bin
3. 系统的部署
3.1 安装uwsgi服务器
pip install uwsgi
3.2 修改服务的配置文件
cd /etc/pkgship/
vi package.ini
备注： 配置文件中可以支撑sqlite数据库和mysql数据库，可根据相应配置进行修改
如果需要调整 查询和修改相关端口，请同步更改 mange.ini 和selfpkg.ini 中的配置
切记（manage.py为拥有写入权限，selfpkg为拥有查询权限）
3.3 启动系统服务
单独启动manage服务： pkgshipd start manage
单独启动selfpkg服务： pkgshipd start selfpkg
同时启动manage和selfpkg服务： pkgshipd start
3.4 停止系统服务
停止manage服务： pkgshipd stop manage
停止selfpkg服务： pkgshipd stop selfpkg
同时停止manage和selfpkg服务： pkgshipd stop
#### 使用说明
1. 命令行使用
pkgship --help
2. restful接口使用
参考接口设计文档中的接口定义，进行相关接口调用
#### 参与贡献
1. Fork 本仓库
2. 新建 Feat_xxx 分支
3. 提交代码
4. 新建 Pull Request
#### 会议记录
1. 2020.5.18：https://etherpad.openeuler.org/p/aHIX4005bTY1OHtOd_Zc
[English](./README-en.md) | 简体中文
# pkgship
## 介绍
pkgship是一款管理OS软件包依赖关系，提供依赖和被依赖关系的完整图谱查询工具，pkgship提供软件包依赖，生命周期，补丁查询等功能。
1. 软件包依赖：方便社区人员在新引入、软件包更新和删除的时候能方便的了解软件的影响范围。
2. 生命周期管理：跟踪upstream软件包发布状态，方便维护人员了解当前软件状态，及时升级到合理的版本。
3. 补丁查询：方便社区人员了解openEuler软件包的补丁情况，方便的提取补丁内容
### 软件架构
系统采用flask-restful开发，使用SQLAlchemy ORM查询框架，同时支持mysql和sqlite数据库，通过配置文件的形式进行更改
安装教程
---
#### 方法一: 可以使用dnf挂载pkgship软件在所在repo源，直接下载安装pkgship及其依赖
```
dnf install pkgship(版本号)
```
#### 方法二: 可以直接下载pkgship的rpm包后安装软件包
```
rpm -ivh pkgship.rpm
```
或者
```
dnf install pkgship-(版本号)
```
系统配置
---
系统的默认配置文件存放在 /etc/pkgship/packge.ini，请根据实际情况进行配置更改
```
vim /etc/pkgship/package.ini
```
创建初始化数据库的yaml配置文件:
conf.yaml 文件默认存放在 /etc/pkgship/ 路径下，pkgship会通过该配置读取要建立的数据库名称以及需要导入的sqlite文件。conf.yaml 示例如下：
```
- dbname:openEuler-20.03-LTS
src_db_file:
- /etc/pkgship/src.sqlite
bin_db_file:
- /etc/pkgship/bin.sqlite
status:enable
priority:1
```
如需更改存放路径，请更改package.ini下的 init_conf_path 选项
服务启动和停止
---
pkgship使用uWSGI web服务器
```
pkgshipd start
pkgshipd stop
```
使用说明
---
#### 1. 数据库初始化
```
pkgship init
```
#### 2. 单包查询
查询源码包(sourceName)在所有数据库中的信息
```
pkgship single sourceName
```
查询当前包(sourceName)在指定数据库(dbName)中的信息
```
pkgship single sourceName -db dbName
```
#### 3. 查询所有包
查询所有数据库下包含的所有包的信息
```
pkgship list
```
查询指定数据库(dbName)下的所有包的信息
```
pkgship list -db dbName
```
#### 4. 安装依赖查询
查询二进制包(binaryName)的安装依赖，按照默认优先级查询数据库
```
pkgship installdep binaryName
```
在指定数据库(dbName)下查询二进制包(binaryName)的所有安装依赖
按照先后顺序指定数据库查询的优先级
```
pkgship installdep binaryName -dbs dbName1 dbName2...
```
#### 5. 编译依赖查询
查询源码包(sourceName)的所有编译依赖，按照默认优先级查询数据库
```
pkgship builddep sourceName
```
在指定数据库(dbName)下查询源码包(sourceName)的所有安装依赖
按照先后顺序指定数据库查询的优先级
```
pkgship builddep sourceName -dbs dbName1 dbName2...
```
#### 6. 自编译自安装依赖查询
查询二进制包(binaryName)的安装和编译依赖，按照默认优先级查询数据库
```
pkgship selfbuild binaryName
```
查询源码包(sourceName )的安装和编译依赖，按照默认优先级查询数据库
```
pkgship selfbuild sourceName -t source
```
其他参数:
-dbs 指定数据库优先级.
```
示例:pkgship selfbuild binaryName -dbs dbName1 dbName2
```
-s 是否查询自编译依赖
默认为0不查询自编译依赖，可以指定0或1(表示查询自编译)
```
查询自编译示例:pkgship selfbuild sourceName -t source -s 1
```
-w 是否查询对应包的子包.默认为0，不查询对应子包，可以指定 0或1(表示查询对应子包)
```
查询子包示例:pkgship selfbuild binaryName -w 1
```
#### 7. 被依赖查询
查询源码包(sourceName)在某数据库(dbName)中被哪些包所依赖
查询结果默认不包含对应二进制包的子包
```
pkgship bedepend sourceName -db dbName
```
使查询结果包含二进制包的子包 加入参数 -w
```
pkgship bedepend sourceName -db dbName -w 1
```
#### 8. 修改包信息记录
变更数据库中(dbName)源码包(sourceName)的maintainer为Newmaintainer
```
pkgship updatepkg sourceName db dbName -m Newmaintainer
```
变更数据库中(dbName)源码包(sourceName)的maintainlevel为Newmaintainlevel，值在1～4之间
```
pkgship updatepkg sourceName db dbName -l Newmaintainlevel
```
同时变更数据库中(dbName)源码包(sourceName)的maintainer 为Newmaintainer和变更maintainlevel为Newmaintainlevel
```
pkgship updatepkg sourceName db dbName -m Newmaintainer -l Newmaintainlevel
```
#### 9. 删除数据库
删除指定数据库(dbName)
```
pkgship rm db dbName
```
参与贡献
---
我们非常欢迎新贡献者加入到项目中来，也非常高兴能为新加入贡献者提供指导和帮助。在您贡献代码前，需要先签署[CLA](https://openeuler.org/en/cla.html)。
1. Fork 本仓库
2. 新建 Feat_xxx 分支
3. 提交代码
4. 新建 Pull Request
### 会议记录
1. 2020.5.18：https://etherpad.openeuler.org/p/aHIX4005bTY1OHtOd_Zc
packageship/packageship/application/apps/package/function/be_depend.py
浏览文件 @ 4a082cd0
......@@ -70,68 +70,54 @@ class BeDepend():
]
self.source_name_set.add(self.source_name)
self.package_bedepend(
[src_obj.id], data_base, package_type='src')
[self.source_name], data_base, package_type='src')
return self.result_dict
def package_bedepend(self, pkg_id_list, data_base, package_type):
def package_bedepend(self, pkg_name_list, data_base, package_type):
"""
Description: Query the dependent function
Args:
pkg_id_list:source or binary packages id
pkg_name_list:source or binary packages name
data_base: database
package_type: package type
Returns:
Raises:
SQLAlchemyError: Database connection exception
"""
search_set = set(pkg_id_list)
id_in = literal_column('id').in_(search_set)
search_set = set(pkg_name_list)
# package_type
if package_type == 'src':
sql_str = text("""
SELECT b1.name AS search_bin_name,
b1.version AS search_bin_version,
src.NAME AS source_name,
b2.name AS bin_name,
b2.id AS bin_id,
s1.name AS bebuild_src_name,
s1.id AS bebuild_src_id,
s2.name AS install_depend_src_name,
s2.id AS install_depend_src_id
FROM
( SELECT id,NAME FROM src_pack WHERE {} ) src
LEFT JOIN bin_pack b1 ON b1.srcIDkey = src.id
LEFT JOIN pack_provides ON pack_provides.binIDkey = b1.id
LEFT JOIN pack_requires ON pack_requires.depProIDkey = pack_provides.id
LEFT JOIN src_pack s1 ON s1.id = pack_requires.srcIDkey
LEFT JOIN bin_pack b2 ON b2.id = pack_requires.binIDkey
LEFT JOIN src_pack s2 ON s2.id = b2.srcIDkey;""".format(id_in))
name_in = literal_column('src_name').in_(search_set)
if package_type == 'bin':
sql_str = text("""
SELECT b1.name AS search_bin_name,
b1.version AS search_bin_version,
s3.NAME AS source_name,
b2.name AS bin_name,
b2.id AS bin_id,
s1.name AS bebuild_src_name,
s1.id AS bebuild_src_id,
s2.name AS install_depend_src_name,
s2.id AS install_depend_src_id
FROM
(SELECT id,NAME,version,srcIDkey FROM bin_pack WHERE {} ) b1
LEFT JOIN src_pack s3 ON s3.id = b1.srcIDkey
LEFT JOIN pack_provides ON pack_provides.binIDkey = b1.id
LEFT JOIN pack_requires ON pack_requires.depProIDkey = pack_provides.id
LEFT JOIN src_pack s1 ON s1.id = pack_requires.srcIDkey
LEFT JOIN bin_pack b2 ON b2.id = pack_requires.binIDkey
LEFT JOIN src_pack s2 ON s2.id = b2.srcIDkey;
""".format(id_in))
name_in = literal_column('name').in_(search_set)
sql_str = text("""
SELECT b1.name AS search_bin_name,
b1.version AS search_bin_version,
b1.src_name AS source_name,
b2.name AS bin_name,
s1.name AS bebuild_src_name,
b2.src_name AS install_depend_src_name
FROM ( SELECT pkgKey,src_name,name,version FROM bin_pack WHERE {} ) b1
LEFT JOIN bin_provides ON bin_provides.pkgKey = b1.pkgKey
LEFT JOIN bin_requires br ON br.name = bin_provides.name
LEFT JOIN src_requires sr ON sr.name = bin_provides.name
LEFT JOIN src_pack s1 ON s1.pkgKey = sr.pkgKey
LEFT JOIN bin_pack b2 ON b2.pkgKey = br.pkgKey;""".format(name_in))
try:
result = data_base.session.execute(
sql_str, {
'id_{}'.format(i): v for i, v in enumerate(
search_set, 1)}).fetchall()
if package_type == 'src':
result = data_base.session.execute(
sql_str, {
'src_name_{}'.format(i): v for i, v in enumerate(
search_set, 1)}).fetchall()
if package_type == 'bin':
result = data_base.session.execute(
sql_str, {
'name_{}'.format(i): v for i, v in enumerate(
search_set, 1)}).fetchall()
except SQLAlchemyError as sql_err:
current_app.logger.error(sql_err)
return ResponseCode.response_json(ResponseCode.CONNECT_DB_ERROR)
......@@ -139,8 +125,8 @@ class BeDepend():
if result is None:
return
# Source and binary packages that were found to be dependent
source_id_list = []
bin_id_list = []
source_name_list = []
bin_name_list = []
for obj in result:
if obj.source_name is None:
source_name = 'NOT FOUND'
......@@ -160,7 +146,7 @@ class BeDepend():
if obj.bebuild_src_name not in self.source_name_set:
self.source_name_set.add(obj.bebuild_src_name)
source_id_list.append(obj.bebuild_src_id)
source_name_list.append(obj.bebuild_src_name)
if obj.bin_name:
# Determine if the bin package has been checked
......@@ -176,20 +162,34 @@ class BeDepend():
if obj.bin_name not in self.bin_name_set:
self.bin_name_set.add(obj.bin_name)
bin_id_list.append(obj.bin_id)
bin_name_list.append(obj.bin_name)
# withsubpack=1
# With_sub_pack=1
if self.with_sub_pack == "1":
if obj.install_depend_src_name not in self.source_name_set:
self.source_name_set.add(
obj.install_depend_src_name)
source_id_list.append(obj.install_depend_src_id)
source_name_list.append(
obj.install_depend_src_name)
if len(source_id_list) != 0:
# Sqlite older versions default to a single query with a maximum of 999
# parameters
if 0 < len(source_name_list) < 999:
self.package_bedepend(
source_id_list, data_base, package_type="src")
if len(bin_id_list) != 0:
self.package_bedepend(bin_id_list, data_base, package_type="bin")
source_name_list, data_base, package_type="src")
elif len(source_name_list) >= 999:
count = len(source_name_list) // 999
for i in range(count + 1):
self.package_bedepend(
source_name_list[999 * i:999 * (i + 1)], data_base, package_type="src")
if 0 < len(bin_name_list) < 999:
self.package_bedepend(bin_name_list, data_base, package_type="bin")
elif len(bin_name_list) >= 999:
count = len(bin_name_list) // 999
for i in range(count + 1):
self.package_bedepend(
bin_name_list[999 * i:999 * (i + 1)], data_base, package_type="bin")
def make_dicts(self, key, source_name, version, parent_node, be_type):
"""
......
packageship/packageship/application/apps/package/function/build_depend.py
浏览文件 @ 4a082cd0
......@@ -80,9 +80,9 @@ class BuildDepend():
res_status, build_list = self.search_db.get_build_depend(pkg_list)
if not build_list:
return res_status if res_status == \
ResponseCode.DIS_CONNECTION_DB else \
return res_status if res_status == ResponseCode.DIS_CONNECTION_DB else \
ResponseCode.PACK_NAME_NOT_FOUND
# create root node and get next search list
search_list = self._create_node_and_get_search_list(build_list, pkg_list)
......@@ -153,7 +153,7 @@ class BuildDepend():
self.result_dict[obj.bin_name] = [
obj.source_name,
obj.version,
obj.db_name,
self.search_db.binary_search_database_for_first_time(obj.bin_name),
[
[obj.search_name, 'build']
]
......@@ -191,19 +191,21 @@ class BuildDepend():
return
# generate data content
search_name_set = set()
for obj in bin_info_lis:
search_name_set.add(obj.search_name)
if obj.search_name not in self.source_dict:
self.source_dict[obj.search_name] = [obj.db_name, obj.search_version]
if not obj.bin_name:
continue
# for first loop, init the source_dict
if not self.source_dict:
for src_name in pkg_name_li:
self.source_dict[src_name] = [obj.db_name, obj.search_version]
if obj.bin_name not in self.result_dict:
self.result_dict[obj.bin_name] = [
obj.source_name if obj.source_name else None,
obj.version if obj.version else None,
obj.db_name if obj.db_name else "NOT_FOUND",
self.search_db.binary_search_database_for_first_time(obj.bin_name),
[
[obj.search_name, "build"]
]
......@@ -216,11 +218,14 @@ class BuildDepend():
if obj.source_name and \
obj.source_name not in self.source_dict and \
obj.source_name not in self.history_dicts:
self.source_dict[obj.source_name] = [obj.db_name,
obj.version]
obj.source_name not in self.history_dicts:
next_src_set.add(obj.source_name)
not_found_pkg = set(pkg_name_li) - search_name_set
for pkg_name in not_found_pkg:
if pkg_name not in self.source_dict:
self.source_dict[pkg_name] = ['NOT FOUND', 'NOT FOUND']
not_found_pkg.clear()
self.self_build(next_src_set)
return
packageship/packageship/application/apps/package/function/packages.py
浏览文件 @ 4a082cd0
......@@ -5,19 +5,26 @@ functions: get_packages, buildep_packages, sub_packages, get_single_package,
update_single_package, update_maintaniner_info
"""
from flask import current_app
from flask import jsonify
from sqlalchemy.exc import SQLAlchemyError
from packageship.application.apps.package.function.constants import ResponseCode
from packageship.application.apps.package.function.searchdb import db_priority
from packageship.libs.dbutils import DBHelper
from packageship.application.models.package import src_pack
from packageship.application.models.package import pack_provides
from packageship.application.models.package import maintenance_info
from packageship.application.models.package import pack_requires
from packageship.application.models.package import src_requires
from packageship.application.models.package import bin_pack
from packageship.application.models.package import maintenance_info
from packageship.application.models.package import bin_requires
from packageship.application.models.package import bin_provides
from packageship.libs.exception import Error
def get_packages(dbname):
"""
Description: Get all packages info
Get all packages info in search databases
Args:
dbname: Database name
Returns:
......@@ -28,89 +35,146 @@ def get_packages(dbname):
"""
with DBHelper(db_name=dbname) as db_name:
src_pack_queryset = db_name.session.query(src_pack).all()
resp_list = []
if src_pack_queryset is None:
return None
resp_list = list()
for src_pack_obj in src_pack_queryset:
package = {}
package = dict()
package["sourceName"] = src_pack_obj.name
package["version"] = src_pack_obj.version
package["license"] = src_pack_obj.license
package["maintainer"] = src_pack_obj.Maintaniner
package["maintainlevel"] = src_pack_obj.MaintainLevel
package["sourceURL"] = src_pack_obj.sourceURL
package["maintainlevel"] = src_pack_obj.MaintainLevel
package["downloadURL"] = src_pack_obj.downloadURL
package["license"] = src_pack_obj.rpm_license
package["sourceURL"] = src_pack_obj.url
package["rpm_packager"] = src_pack_obj.rpm_packager
package["maintaniner"] = src_pack_obj.maintaniner
package["maintainlevel"] = src_pack_obj.maintainlevel
package["dbname"] = dbname
resp_list.append(package)
return resp_list
def buildep_packages(dbname, src_pack_id):
def get_all_packages(db_name):
"""
all packages info
Args:
db_name: database name
Returns:
response code: response status code
"""
dbpreority = db_priority()
if dbpreority is None:
return jsonify(
ResponseCode.response_json(ResponseCode.FILE_NOT_FOUND)
)
if not db_name:
response = []
for dbname in dbpreority:
query_result = get_packages(dbname)
if query_result is None:
return None
for item in query_result:
if item is None:
query_result.remove(item)
response.append(item)
return jsonify(
ResponseCode.response_json(ResponseCode.SUCCESS, response)
)
if db_name not in dbpreority:
return jsonify(
ResponseCode.response_json(ResponseCode.DB_NAME_ERROR)
)
response = get_packages(db_name)
if not response:
return jsonify(
ResponseCode.response_json(ResponseCode.PACK_NAME_NOT_FOUND)
)
return jsonify(
ResponseCode.response_json(ResponseCode.SUCCESS, response)
)
def buildep_packages(dbname, src_pack_pkgkey):
"""
Description: Query package layer 1 compilation dependency
Query package layer 1 compilation dependency
Args:
dbname: databases name
src_pack_id: The ID of the source package
src_pack_pkgkey: The ID of the source package
Returns:
buildDep Compile dependencies of source packages
Raises:
AttributeError: Object does not have this property
"""
with DBHelper(db_name=dbname) as db_name:
b_pack_requires_set = db_name.session.query(
pack_requires).filter_by(srcIDkey=src_pack_id).all()
b_dep_proid_keys = [
dep_proid_obj.depProIDkey for dep_proid_obj in b_pack_requires_set]
b_pack_pro_set = db_name.session.query(pack_provides).filter(
pack_provides.id.in_(b_dep_proid_keys)).all()
b_bin_pack_ids = [
bin_pack_obj.binIDkey for bin_pack_obj in b_pack_pro_set]
# srcpack's pkgkey to src_requires find pkgkey
s_pack_requires_set = db_name.session.query(
src_requires).filter_by(pkgKey=src_pack_pkgkey).all()
# src_requires pkykey to find the name of the dependent component
s_pack_requires_names = [
s_pack_requires_obj.name for s_pack_requires_obj in s_pack_requires_set]
# Find pkgkey in bin_provides by the name of the dependent component
b_pack_provides_set = db_name.session.query(bin_provides).filter(
bin_provides.name.in_(s_pack_requires_names)).all()
b_pack_provides_pkg_list = [
b_pack_provides_obj.pkgKey for b_pack_provides_obj in b_pack_provides_set]
# Go to bin_pack to find the name by pkgkey of bin_provides
b_bin_pack_set = db_name.session.query(bin_pack).filter(
bin_pack.id.in_(b_bin_pack_ids)).all()
builddep = [bin_pack_obj.name for bin_pack_obj in b_bin_pack_set]
bin_pack.pkgKey.in_(b_pack_provides_pkg_list)).all()
builddep = [b_bin_pack_obj.name for b_bin_pack_obj in b_bin_pack_set]
return builddep
def sub_packages(dbname, src_pack_id):
def sub_packages(dbname, sourcename):
"""
Description: Query package layer 1 installation dependency
Query package layer 1 installation dependency
Args:
dbname: databases name
src_pack_id: srcpackage id
src_pack_pkgkey: srcpackage id
Returns:
subpack Source package to binary package, then find the installation dependencies
of the binary package
of the binary package
Raises:
AttributeError: Object does not have this property
"""
with DBHelper(db_name=dbname) as db_name:
subpack = {}
subpack = dict()
# The name of src_pack finds the sub-package bin_pack query set
i_bin_pack_set = db_name.session.query(
bin_pack).filter_by(srcIDkey=src_pack_id).all()
i_bin_pack_ids = [
bin_pack_obj.id for bin_pack_obj in i_bin_pack_set]
for i_bin_pack_id in i_bin_pack_ids:
i_bin_pack_name = db_name.session.query(
bin_pack).filter_by(id=i_bin_pack_id).first().name
i_pack_req_set = db_name.session.query(
pack_requires).filter_by(binIDkey=i_bin_pack_id).all()
i_dep_proid_keys = [
dep_proid_obj.depProIDkey for dep_proid_obj in i_pack_req_set]
i_dep_proid_keys = list(set(i_dep_proid_keys))
i_pack_provides_set = db_name.session.query(pack_provides).filter(
pack_provides.id.in_(i_dep_proid_keys)).all()
i_bin_pack_ids = [
bin_pack_obj.binIDkey for bin_pack_obj in i_pack_provides_set]
bin_pack).filter_by(src_name=sourcename).all()
if i_bin_pack_set is None:
return subpack
# Find the objects of each sub-package
for b_bin_pack_obj in i_bin_pack_set:
i_bin_pack_name = b_bin_pack_obj.name
i_bin_pack_pkgkey = b_bin_pack_obj.pkgKey
# Find the names of the components required to install bin_requires
# dependencies
i_bin_requires_set = db_name.session.query(
bin_requires).filter_by(pkgKey=i_bin_pack_pkgkey).all()
i_bin_requires_names = [
b_bin_requires_obj.name for b_bin_requires_obj in i_bin_requires_set]
# Find pkykey in bin_provides by the name of the dependent
# component
i_bin_provides_set = db_name.session.query(bin_provides).filter(
bin_provides.name.in_(i_bin_requires_names))
i_bin_provides_pkg_list = [
i_bin_provides_obj.pkgKey for i_bin_provides_obj in i_bin_provides_set]
# Find the name in bin_pack by pkgkey
i_bin_pack_set = db_name.session.query(bin_pack).filter(
bin_pack.id.in_(i_bin_pack_ids)).all()
bin_pack.pkgKey.in_(i_bin_provides_pkg_list))
i_bin_pack_names = [
bin_pack_obj.name for bin_pack_obj in i_bin_pack_set]
in_bin_pack_obj.name for in_bin_pack_obj in i_bin_pack_set]
subpack[i_bin_pack_name] = i_bin_pack_names
return subpack
def get_single_package(dbname, sourcename):
"""
Description: Get all packages info
Get single packages info
Args：
dbname: Database name
sourcename: Source package name
......@@ -120,89 +184,119 @@ def get_single_package(dbname, sourcename):
AttributeError: Object does not have this property
"""
with DBHelper(db_name=dbname) as db_name:
package = {}
package = dict()
src_pack_obj = db_name.session.query(src_pack).filter_by(
name=sourcename).first()
if src_pack_obj is None:
return None
package["sourceName"] = src_pack_obj.name
package["version"] = src_pack_obj.version
package["license"] = src_pack_obj.license
package["maintainer"] = src_pack_obj.Maintaniner
package["maintainlevel"] = src_pack_obj.MaintainLevel
package["sourceURL"] = src_pack_obj.sourceURL
package["downloadURL"] = src_pack_obj.downloadURL
package["license"] = src_pack_obj.rpm_license
package["sourceURL"] = src_pack_obj.url
package["rpm_packager"] = src_pack_obj.rpm_packager
package["maintaniner"] = src_pack_obj.maintaniner
package["maintainlevel"] = src_pack_obj.maintainlevel
package["dbname"] = dbname
src_pack_id = src_pack_obj.id
builddep = buildep_packages(dbname, src_pack_id)
subpack = sub_packages(dbname, src_pack_id)
src_pack_pkgkey = src_pack_obj.pkgKey
builddep = buildep_packages(dbname, src_pack_pkgkey)
subpack = sub_packages(dbname, sourcename)
package['buildDep'] = builddep
package['subpack'] = subpack
return package
def update_single_package(
def get_single(dbnames, sourcename):
"""
get single package
Args:
dbname: database name
sourcename: source name
"""
response_data = None
dbpreority = db_priority()
if db_priority is None:
response_data = ResponseCode.FILE_NOT_FOUND
if not dbnames:
response = []
for db_names in dbpreority:
query_result = get_single_package(db_names, sourcename)
response.append(query_result)
for key in response:
if key is None:
response.remove(key)
if not response:
return jsonify(
ResponseCode.response_json(ResponseCode.PACK_NAME_NOT_FOUND)
)
return jsonify(
ResponseCode.response_json(ResponseCode.SUCCESS, response)
)
# Database queries data and catches exceptions
if dbnames not in dbpreority:
return jsonify(
ResponseCode.response_json(ResponseCode.DB_NAME_ERROR)
)
response = get_single_package(dbnames, sourcename)
if response is None:
response_data = ResponseCode.PACK_NAME_NOT_FOUND
if response_data is not None:
return jsonify(ResponseCode.response_json(response_data))
return jsonify(
ResponseCode.response_json(ResponseCode.SUCCESS, [response])
)
def _update_package_info(
package_name,
dbname,
maintainer,
maintain_level):
"""
Description: change single package management
Args：
change single package management
Args:
package_name: package name
dbname: Database name
maintainer: maintainer info
maintain_level: maintain_level info
Returns：
Returns:
message success or failed
Raises:
AttributeError: Object does not have this property
TypeError: Abnormal error
"""
with DBHelper(db_name=dbname) as db_name:
update_obj = db_name.session.query(
src_pack).filter_by(name=package_name).first()
update_obj.Maintaniner = maintainer
update_obj.MaintainLevel = maintain_level
db_name.session.commit()
def update_maintaniner_info(package_name,
dbname,
maintaniner,
maintainlevel):
"""
Description: update separately maintaniner info
Args：
package_name: package name
dbname: Database name
maintainer: maintainer info
maintain_level: maintain_level info
Returns：
message success or failed
Raises:
AttributeError: Object does not have this property
Error: Abnormal error
SQLAlchemyError: Exception of type
Error: Abnormal error
"""
with DBHelper(db_name=dbname) as db_name:
src_pack_obj = db_name.session.query(src_pack).filter_by(
name=package_name).first()
name = src_pack_obj.name
version = src_pack_obj.version
with DBHelper(db_name='maintenance.information') as dbs_name:
try:
try:
result_data = True
with DBHelper(db_name=dbname) as data_name:
update_obj = data_name.session.query(
src_pack).filter_by(name=package_name).first()
if update_obj is None:
return False
update_obj.maintaniner = maintainer
update_obj.maintainlevel = maintain_level
data_name.session.commit()
name = update_obj.name
version = update_obj.version
with DBHelper(db_name='maintenance.information') as dbs_name:
information_obj = dbs_name.session.query(maintenance_info).filter_by(
name=package_name, version=version).first()
if information_obj is None:
information = maintenance_info(
name=name,
version=version,
maintaniner=maintaniner,
maintainlevel=maintainlevel)
maintaniner=maintainer,
maintainlevel=maintain_level)
dbs_name.session.add(information)
dbs_name.session.commit()
else:
information_obj.maintaniner = maintaniner
information_obj.maintainlevel = maintainlevel
information_obj.maintaniner = maintainer
information_obj.maintainlevel = maintain_level
dbs_name.session.commit()
except (AttributeError, Error) as attri_error:
current_app.logger.error(attri_error)
return
return result_data
except (AttributeError, SQLAlchemyError, Error) as attri_error:
current_app.logger.error(attri_error)
raise attri_error
packageship/packageship/application/initsystem/datamerge.py 已删除 100644 → 0
浏览文件 @ e8ba7df5
#!/usr/bin/python3
"""
Description: Integration of multiple sqlite file data, including reading
sqlite database and inserting data
Class: MergeData
"""
from sqlalchemy.exc import SQLAlchemyError
from packageship.application.models.temporarydb import src_package
from packageship.application.models.temporarydb import src_requires
from packageship.application.models.temporarydb import bin_package
from packageship.application.models.temporarydb import bin_requiresment
from packageship.application.models.temporarydb import bin_provides
from packageship.application.models.package import maintenance_info
from packageship.libs.dbutils import DBHelper
from packageship.libs.log import Log
LOGGER = Log(__name__)
class MergeData():
"""
Description: Load data from sqlite database
Attributes:
db_file: Database file
db_type: Connected database type
datum_database: Base database name
"""
def __init__(self, db_file):
"""
Description: Class instance initialization
Args:
db_file: Database file
"""
self.db_file = db_file
self.db_type = 'sqlite:///'
self.datum_database = 'maintenance.information'
self.src_requires_dicts = dict()
self.src_package_datas = []
self.bin_provides_dicts = dict()
self.bin_package_datas = []
self.mainter_infos = dict()
self.bin_requires_dicts = dict()
@staticmethod
def __columns(cursor):
"""
Description: functional description:Returns all the column names
queried by the current cursor
Args:
cursor: Cursor
Returns:
The first columns
Raises:
"""
return [col[0] for col in cursor.description]
def get_package_data(self):
"""
Description: get binary package or source package data
Args:
Returns:
All source package data queried
Raises:
SQLAlchemyError: An error occurred while executing the sql statement
"""
try:
with DBHelper(db_name=self.db_file, db_type=self.db_type, import_database=True) \
as database:
src_packages_data = database.session.execute(
"select pkgKey,name,version,rpm_license,url,rpm_sourcerpm from packages")
columns = MergeData.__columns(
src_packages_data.cursor)
return [dict(zip(columns, row)) for row in src_packages_data.fetchall()]
except SQLAlchemyError as sql_error:
LOGGER.logger.error(sql_error)
return None
def get_requires_data(self):
"""
Description: get dependent package data of binary package or source package
Args:
Returns:
All dependent data queried
Raises:
SQLAlchemyError: An error occurred while executing the sql statement
"""
try:
with DBHelper(db_name=self.db_file, db_type=self.db_type, import_database=True) \
as database:
requires = database.session.execute(
"select pkgKey,name from requires")
columns = MergeData.__columns(requires.cursor)
return [dict(zip(columns, row)) for row in requires.fetchall()]
except SQLAlchemyError as sql_error:
LOGGER.logger.error(sql_error)
return None
def get_provides(self):
"""
Description: get the dependency package provided by the binary package
Args:
Returns:
Query the component data provided by all binary packages
Raises:
SQLAlchemyError: An error occurred while executing the sql statement
"""
try:
with DBHelper(db_name=self.db_file, db_type=self.db_type, import_database=True) \
as database:
requires = database.session.execute(
"select pkgKey,name from provides")
columns = MergeData.__columns(requires.cursor)
return [dict(zip(columns, row)) for row in requires.fetchall()]
except SQLAlchemyError as sql_error:
LOGGER.logger.error(sql_error)
return None
def get_maintenance_info(self):
"""
Description: Obtain the information of the maintainer
Args:
Returns:
Maintainer related information
Raises:
SQLAlchemyError: An error occurred while executing the sql statement
"""
try:
if not hasattr(self, 'mainter_infos'):
self.mainter_infos = dict()
with DBHelper(db_name=self.datum_database) as database:
for info in database.session.query(maintenance_info).all():
if info.name not in self.mainter_infos.keys():
self.mainter_infos[info.name] = []
self.mainter_infos[info.name].append({
'version': info.version,
'maintaniner': info.maintaniner
})
except SQLAlchemyError as sql_error:
LOGGER.logger.error(sql_error)
def src_file_merge(self, src_package_key, db_file):
"""
Description: Source code related data integration
Args:
src_package_key: The relevant key value of the source package
db_file: Database file
Returns:
Key value after successful data combination
(0, False) or (src_package_key, True)
Raises:
SQLAlchemyError: An error occurred while executing the sql statement
"""
self.get_maintenance_info()
self.__compose_src_package()
self.__compose_src_rquires()
# Combination of relationships between source packages and dependent packages
src_requires_data = []
for src_package_item in self.src_package_datas:
src_package_key += 1
requires = self.src_requires_dicts.get(
src_package_item.get('pkgKey'))
if requires:
for src_requires_item in requires:
src_requires_item['pkgKey'] = src_package_key
src_requires_data.append(src_requires_item)
src_package_item['pkgKey'] = src_package_key
try:
with DBHelper(db_name=db_file, db_type=self.db_type) as data_base:
data_base.batch_add(self.src_package_datas, src_package)
data_base.batch_add(src_requires_data, src_requires)
except SQLAlchemyError as sql_error:
LOGGER.logger.error(sql_error)
return (0, False)
else:
return (src_package_key, True)
def __compose_src_package(self):
"""
Description: Combine source package data
Args:
Returns:
Raises:
"""
if getattr(self, 'src_package_datas', None) is None:
self.src_package_datas = []
for src_package_item in self.get_package_data():
src_package_name = src_package_item.get('name')
if src_package_name:
# Find the maintainer information of the current data
maintenance_infos = self.mainter_infos.get(src_package_name)
maintenance = []
version = src_package_item.get('version')
if self.mainter_infos.get(src_package_name):
for maintenance_item in maintenance_infos:
if maintenance_item.get('version') == version:
maintenance.append(maintenance_item)
self.src_package_datas.append(
{
"name": src_package_item.get('name'),
"version": version,
"rpm_license": src_package_item.get('rpm_license'),
"url": src_package_item.get('url'),
"pkgKey": src_package_item.get('pkgKey'),
'maintaniner':
maintenance[0].get('maintaniner') if maintenance and len(
maintenance) > 0 else None
}
)
def __compose_src_rquires(self):
"""
Description: Combine source package dependent package data
Args:
Returns:
Raises:
"""
if getattr(self, 'src_requires_dicts', None) is None:
self.src_requires_dicts = dict()
for src_requires_item in self.get_requires_data():
pkg_key = src_requires_item.get('pkgKey')
if pkg_key:
if pkg_key not in self.src_requires_dicts.keys():
self.src_requires_dicts[pkg_key] = []
self.src_requires_dicts[pkg_key].append(
{
'name': src_requires_item.get('name'),
'pkgKey': pkg_key
}
)
def __compose_bin_package(self):
"""
Description: Combine binary package data
Args:
Returns:
Raises:
AttributeError
"""
if getattr(self, 'bin_package_datas', None) is None:
self.bin_package_datas = []
for bin_package_item in self.get_package_data():
try:
src_package_name = bin_package_item.get('rpm_sourcerpm').split(
'-' + bin_package_item.get('version'))[0]
except AttributeError as exception_msg:
src_package_name = None
LOGGER.logger.warning(exception_msg)
else:
self.bin_package_datas.append(
{
"name": bin_package_item.get('name'),
"version": bin_package_item.get('version'),
"license": bin_package_item.get('rpm_license'),
"sourceURL": bin_package_item.get('url'),
"src_pack_name": src_package_name,
"pkgKey": bin_package_item.get('pkgKey')
}
)
def __compose_bin_requires(self):
"""
Description: Combining binary dependent package data
Args:
Returns:
Raises:
"""
if getattr(self, 'bin_requires_dicts', None) is None:
self.bin_requires_dicts = dict()
for bin_requires_item in self.get_requires_data():
pkg_key = bin_requires_item.get('pkgKey')
if pkg_key:
if pkg_key not in self.bin_requires_dicts.keys():
self.bin_requires_dicts[pkg_key] = []
self.bin_requires_dicts[pkg_key].append({
'name': bin_requires_item.get('name'),
'pkgKey': 0
})
def __compose_bin_provides(self):
"""
Description: Combine binary package data
Args:
Returns:
Raises:
"""
if getattr(self, 'bin_provides_dicts', None) is None:
self.bin_provides_dicts = dict()
for bin_provides_item in self.get_provides():
pkg_key = bin_provides_item.get('pkgKey')
if pkg_key:
if pkg_key not in self.bin_provides_dicts.keys():
self.bin_provides_dicts[pkg_key] = []
self.bin_provides_dicts[pkg_key].append({
'name': bin_provides_item.get('name'),
'pkgKey': 0
})
def bin_file_merge(self, bin_package_key, db_file):
"""
Description: Binary package related data integration
Args:
bin_package_key: Primary key of binary package
db_file: Database file
Returns:
Key value after successful data combination
(0, False) or (bin_package_key, True)
Raises:
SQLAlchemyError: An error occurred while executing the sql statement
"""
self.__compose_bin_package()
# binary package dependent package integration
self.__compose_bin_requires()
self.__compose_bin_provides()
# integrate the id data of the binary package
bin_requires_datas = []
bin_provides_datas = []
for bin_package_item in self.bin_package_datas:
bin_package_key += 1
# dependent packages
requires = self.bin_requires_dicts.get(
bin_package_item.get('pkgKey'))
if requires:
for bin_requires_item in requires:
bin_requires_item['pkgKey'] = bin_package_key
bin_requires_datas.append(bin_requires_item)
provides = self.bin_provides_dicts.get(
bin_package_item.get('pkgKey'))
if provides:
for bin_provides_item in provides:
bin_provides_item['pkgKey'] = bin_package_key
bin_provides_datas.append(bin_provides_item)
bin_package_item['pkgKey'] = bin_package_key
# save binary package related data
try:
with DBHelper(db_name=db_file, db_type=self.db_type) as data_base:
data_base.batch_add(self.bin_package_datas, bin_package)
data_base.batch_add(bin_requires_datas, bin_requiresment)
data_base.batch_add(bin_provides_datas, bin_provides)
except SQLAlchemyError as sql_error:
LOGGER.logger.error(sql_error)
return (0, False)
else:
return (bin_package_key, True)
packageship/packageship/application/models/package.py
浏览文件 @ 4a082cd0
......@@ -2,33 +2,45 @@
"""
Description: Database entity model mapping
"""
from sqlalchemy import Column, Integer, String, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, String
from packageship.libs.dbutils.sqlalchemy_helper import DBHelper
class src_pack(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
Description: functional description:Source package model
Source package model
"""
__tablename__ = 'src_pack'
id = Column(Integer, primary_key=True)
name = Column(String(500), nullable=True)
version = Column(String(200), nullable=True)
license = Column(String(500), nullable=True)
sourceURL = Column(String(200), nullable=True)
downloadURL = Column(String(200), nullable=True)
Maintaniner = Column(String(50), nullable=True)
MaintainLevel = Column(String(20), nullable=True)
pkgKey = Column(Integer, primary_key=True)
pkgId = Column(String(500), nullable=True)
name = Column(String(200), nullable=True)
arch = Column(String(200), nullable=True)
version = Column(String(500), nullable=True)
epoch = Column(String(200), nullable=True)
release = Column(String(500), nullable=True)
summary = Column(String(500), nullable=True)
description = Column(String(500), nullable=True)
url = Column(String(500), nullable=True)
time_file = Column(Integer, nullable=True)
time_build = Column(Integer, nullable=True)
rpm_license = Column(String(500), nullable=True)
rpm_vendor = Column(String(500), nullable=True)
rpm_group = Column(String(500), nullable=True)
rpm_buildhost = Column(String(500), nullable=True)
rpm_sourcerpm = Column(String(500), nullable=True)
rpm_header_start = Column(Integer, nullable=True)
rpm_header_end = Column(Integer, nullable=True)
rpm_packager = Column(String(500), nullable=True)
size_package = Column(Integer, nullable=True)
size_installed = Column(Integer, nullable=True)
size_archive = Column(Integer, nullable=True)
location_href = Column(String(500), nullable=True)
location_base = Column(String(500), nullable=True)
checksum_type = Column(String(500), nullable=True)
maintaniner = Column(String(100), nullable=True)
maintainlevel = Column(String(100), nullable=True)
class bin_pack(DBHelper.BASE): # pylint: disable=C0103,R0903
......@@ -37,48 +49,81 @@ class bin_pack(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
__tablename__ = 'bin_pack'
id = Column(Integer, primary_key=True)
pkgKey = Column(Integer, primary_key=True)
pkgId = Column(String(500), nullable=True)
name = Column(String(500), nullable=True)
version = Column(String(200), nullable=True)
srcIDkey = Column(Integer, ForeignKey('src_pack.id'))
src_pack = relationship('src_pack', backref="bin_pack")
class pack_requires(DBHelper.BASE): # pylint: disable=C0103,R0903
arch = Column(String(500), nullable=True)
version = Column(String(500), nullable=True)
epoch = Column(String(500), nullable=True)
release = Column(String(500), nullable=True)
summary = Column(String(500), nullable=True)
description = Column(String(500), nullable=True)
url = Column(String(500), nullable=True)
time_file = Column(Integer, nullable=True)
time_build = Column(Integer, nullable=True)
rpm_license = Column(String(500), nullable=True)
rpm_vendor = Column(String(500), nullable=True)
rpm_group = Column(String(500), nullable=True)
rpm_buildhost = Column(String(500), nullable=True)
rpm_sourcerpm = Column(String(500), nullable=True)
rpm_header_start = Column(Integer, nullable=True)
rpm_header_end = Column(Integer, nullable=True)
rpm_packager = Column(String(500), nullable=True)
size_package = Column(Integer, nullable=True)
size_installed = Column(Integer, nullable=True)
size_archive = Column(Integer, nullable=True)
location_href = Column(String(500), nullable=True)
location_base = Column(String(500), nullable=True)
checksum_type = Column(String(500), nullable=True)
src_name = Column(String(500), nullable=True)
class bin_requires(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
functional description:
Binary package dependent package entity model
"""
__tablename__ = 'pack_requires'
__tablename__ = 'bin_requires'
id = Column(Integer, primary_key=True)
name = Column(String(200), nullable=True)
flags = Column(String(200), nullable=True)
epoch = Column(String(200), nullable=True)
version = Column(String(500), nullable=True)
release = Column(String(200), nullable=True)
pkgKey = Column(Integer, nullable=True)
pre = Column(String(20), nullable=True)
name = Column(String(500), nullable=True)
# depProIDkey = Column(Integer, ForeignKey(
# 'pack_provides.id'), nullable=True)
depProIDkey = Column(Integer)
srcIDkey = Column(Integer, ForeignKey('src_pack.id'), nullable=True)
class src_requires(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
Source entity package dependent package entity model
"""
__tablename__ = 'src_requires'
binIDkey = Column(Integer, ForeignKey('bin_pack.id'), nullable=True)
id = Column(Integer, primary_key=True)
name = Column(String(200), nullable=True)
flags = Column(String(200), nullable=True)
epoch = Column(String(200), nullable=True)
version = Column(String(500), nullable=True)
release = Column(String(200), nullable=True)
pkgKey = Column(Integer, nullable=True)
pre = Column(String(20), nullable=True)
class pack_provides(DBHelper.BASE): # pylint: disable=C0103,R0903
class bin_provides(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
functional description:
Component entity model provided by binary package
"""
__tablename__ = 'pack_provides'
__tablename__ = 'bin_provides'
id = Column(Integer, primary_key=True)
name = Column(String(500), nullable=True)
binIDkey = Column(Integer, ForeignKey('bin_pack.id'))
name = Column(String(200), nullable=True)
flags = Column(String(200), nullable=True)
epoch = Column(String(200), nullable=True)
version = Column(String(500), nullable=True)
release = Column(String(200), nullable=True)
pkgKey = Column(Integer, nullable=True)
class maintenance_info(DBHelper.BASE): # pylint: disable=C0103,R0903
......
packageship/packageship/application/models/temporarydb.py 已删除 100644 → 0
浏览文件 @ e8ba7df5
#!/usr/bin/python3
"""
Description: Database entity model mapping
"""
from sqlalchemy import Column, Integer, String
from packageship.libs.dbutils.sqlalchemy_helper import DBHelper
class src_package(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
Description: Temporary source package model
"""
__tablename__ = 'src_package'
pkgKey = Column(Integer, primary_key=True)
name = Column(String(500), nullable=True)
version = Column(String(200), nullable=True)
rpm_license = Column(String(500), nullable=True)
url = Column(String(200), nullable=True)
maintaniner = Column(String(100), nullable=True)
class bin_package(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
Description: Temporary binary package model
"""
__tablename__ = 'bin_package'
pkgKey = Column(Integer, primary_key=True)
name = Column(String(500), nullable=True)
version = Column(String(200), nullable=True)
rpm_license = Column(String(500), nullable=True)
url = Column(String(500), nullable=True)
rpm_sourcerpm = Column(String(500), nullable=True)
src_pack_name = Column(String(200), nullable=True)
class src_requires(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
Description: Temporary source package depends on package model
"""
__tablename__ = 'src_requires'
id = Column(Integer, primary_key=True)
pkgKey = Column(Integer)
name = Column(String(500), nullable=True)
class bin_requiresment(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
Description: Dependency package model for temporary binary packages
"""
__tablename__ = 'bin_requiresment'
id = Column(Integer, primary_key=True)
pkgKey = Column(Integer)
name = Column(String(500), nullable=True)
class bin_provides(DBHelper.BASE): # pylint: disable=C0103,R0903
"""
Description: Provided package model for temporary binary packages
"""
__tablename__ = 'bin_provides'
id = Column(Integer, primary_key=True)
pkgKey = Column(Integer)
name = Column(String(500), nullable=True)
packageship/setup.py
浏览文件 @ 4a082cd0
......@@ -30,9 +30,7 @@ setup(
'packageship.application.apps.package.function.searchdb',
'packageship.application.apps.package.function.self_depend',
'packageship.application.initsystem.data_import',
'packageship.application.initsystem.datamerge',
'packageship.application.models.package',
'packageship.application.models.temporarydb',
'packageship.application.settings',
'packageship.libs.__init__',
'packageship.libs.configutils.readconfig',
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或