Skip to content

T
treevalue

OpenDILab开源决策智能平台 / treevalue
9 个月 前同步成功

通知 3
Star 213
Fork 3
T
treevalue

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

未验证 提交 df273bad 编写于 作者: HansBug's avatar HansBug 😆 提交者： GitHub
1 浏览文件
操作

Merge pull request #71 from opendilab/dev/constraint 

dev(hansbug): add constraint feature
上级 7acf1304 b3d7ee2a
展开全部 隐藏空白更改
内联 并排
Showing with 1974 addition and 77 deletion
.github/workflows/badge.yml
浏览文件 @ df273bad
......@@ -14,9 +14,9 @@ jobs:
- '3.8'
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Download cloc
......
.github/workflows/doc.yml
浏览文件 @ df273bad
......@@ -25,9 +25,9 @@ jobs:
- 18080:8080
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
......
.github/workflows/release.yml
浏览文件 @ df273bad
......@@ -12,17 +12,17 @@ jobs:
strategy:
matrix:
os:
- 'ubuntu-18.04'
- 'ubuntu-20.04'
python-version:
- '3.8'
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
fetch-depth: 20
- name: Set up python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Set up python dependences
......@@ -58,9 +58,9 @@ jobs:
fail-fast: false
matrix:
os:
- 'ubuntu-18.04'
- 'ubuntu-20.04'
- 'windows-2019'
- 'macos-10.15'
- 'macos-12'
python:
- '3.7'
- '3.8'
......@@ -73,11 +73,11 @@ jobs:
- x86
- AMD64
exclude:
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: arm64
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: x86
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: AMD64
- os: windows-2019
architecture: x86_64
......@@ -85,18 +85,18 @@ jobs:
architecture: arm64
- os: windows-2019
architecture: aarch64
- os: macos-10.15
- os: macos-12
architecture: aarch64
- os: macos-10.15
- os: macos-12
architecture: x86
- os: macos-10.15
- os: macos-12
architecture: AMD64
- python: '3.7'
architecture: arm64
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
fetch-depth: 20
......@@ -125,16 +125,16 @@ jobs:
# the publishing can only be processed on linux system
wheel_publish:
name: Publish the wheels to pypi
runs-on: ubuntu-18.04
runs-on: ubuntu-20.04
needs:
- wheel_build
strategy:
fail-fast: false
matrix:
os:
- 'ubuntu-18.04'
- 'ubuntu-20.04'
- 'windows-2019'
- 'macos-10.15'
- 'macos-12'
python:
- '3.7'
- '3.8'
......@@ -147,11 +147,11 @@ jobs:
- x86
- AMD64
exclude:
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: arm64
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: x86
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: AMD64
- os: windows-2019
architecture: x86_64
......@@ -159,11 +159,11 @@ jobs:
architecture: arm64
- os: windows-2019
architecture: aarch64
- os: macos-10.15
- os: macos-12
architecture: aarch64
- os: macos-10.15
- os: macos-12
architecture: x86
- os: macos-10.15
- os: macos-12
architecture: AMD64
- python: '3.7'
architecture: arm64
......
.github/workflows/release_test.yml
浏览文件 @ df273bad
......@@ -10,17 +10,17 @@ jobs:
fail-fast: false
matrix:
os:
- 'ubuntu-18.04'
- 'ubuntu-20.04'
python-version:
- '3.8'
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
fetch-depth: 20
- name: Set up python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Set up python dependences
......@@ -48,9 +48,9 @@ jobs:
fail-fast: false
matrix:
os:
- 'ubuntu-18.04'
- 'ubuntu-20.04'
- 'windows-2019'
- 'macos-10.15'
- 'macos-12'
python:
- '3.7'
- '3.8'
......@@ -63,11 +63,11 @@ jobs:
- x86
- AMD64
exclude:
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: arm64
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: x86
- os: ubuntu-18.04
- os: ubuntu-20.04
architecture: AMD64
- os: windows-2019
architecture: x86_64
......@@ -75,18 +75,18 @@ jobs:
architecture: arm64
- os: windows-2019
architecture: aarch64
- os: macos-10.15
- os: macos-12
architecture: aarch64
- os: macos-10.15
- os: macos-12
architecture: x86
- os: macos-10.15
- os: macos-12
architecture: AMD64
- python: '3.7'
architecture: arm64
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
fetch-depth: 20
......
.github/workflows/run.yml
浏览文件 @ df273bad
......@@ -13,7 +13,7 @@ jobs:
fail-fast: false
matrix:
os:
- 'ubuntu-18.04'
- 'ubuntu-20.04'
python-version:
- '3.7'
- '3.8'
......@@ -22,7 +22,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
fetch-depth: 20
- name: Set up system dependences on linux
......@@ -33,7 +33,7 @@ jobs:
sudo apt-get install -y libxml2-dev libxslt-dev python-dev # need by pypy3
dot -V
- name: Set up python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
......
.github/workflows/test.yml
浏览文件 @ df273bad
......@@ -17,14 +17,20 @@ jobs:
fail-fast: false
matrix:
os:
- 'ubuntu-18.04'
- 'ubuntu-20.04'
- 'windows-2019' # need to be fixed, see: https://github.com/opendilab/treevalue/issues/41
- 'macos-10.15'
- 'macos-12'
python-version:
- '3.7'
- '3.8'
- '3.9'
- '3.10'
- '3.11'
exclude:
- os: 'windows-2019'
python-version: '3.11'
- os: 'macos-12'
python-version: '3.11'
steps:
- name: Get system version for Linux
......@@ -59,7 +65,7 @@ jobs:
run: |
echo "IS_PYPY=1" >> $GITHUB_ENV
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
fetch-depth: 20
- name: Set up system dependences on linux
......@@ -82,7 +88,7 @@ jobs:
brew install tree cloc wget curl make zip graphviz
dot -V
- name: Set up python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
......@@ -110,7 +116,7 @@ jobs:
run: |
make clean build unittest
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./coverage.xml
......
Makefile
浏览文件 @ df273bad
.PHONY: docs test unittest build clean benchmark zip
PYTHON := $(shell which python)
NO_DEBUG ?=
NO_DOCSTRING ?=
NO_DEBUG_CMD := $(if ${NO_DOCSTRING},-OO,$(if ${NO_DEBUG},-O,))
PYTHON := $(shell which python) ${NO_DEBUG_CMD}
DOC_DIR := ./docs
DIST_DIR := ./dist
......@@ -39,13 +42,14 @@ clean:
rm -rf $(shell find ${SRC_DIR} -name '*.so') \
$(shell ls $(addsuffix .c, $(basename ${CYTHON_FILES})) \
$(addsuffix .cpp, $(basename ${CYTHON_FILES})) \
$(addsuffix .h, $(basename ${CYTHON_FILES})) \
2> /dev/null)
rm -rf ${DIST_DIR} ${WHEELHOUSE_DIR}
test: unittest benchmark
unittest:
pytest "${RANGE_TEST_DIR}" \
$(PYTHON) -m pytest "${RANGE_TEST_DIR}" \
-sv -m unittest \
$(shell for type in ${COV_TYPES}; do echo "--cov-report=$$type"; done) \
--cov="${RANGE_SRC_DIR}" \
......@@ -53,7 +57,7 @@ unittest:
$(if ${WORKERS},-n ${WORKERS},)
benchmark:
pytest "${RANGE_TEST_DIR}" \
$(PYTHON) -m pytest "${RANGE_TEST_DIR}" \
-sv -m benchmark \
--benchmark-columns=min,max,mean,median,IQR,ops,rounds,iterations \
--benchmark-disable-gc \
......@@ -61,7 +65,7 @@ benchmark:
$(if ${WORKERS},-n ${WORKERS},)
compare:
pytest "${RANGE_BENCH_DIR}" \
$(PYTHON) -m pytest "${RANGE_BENCH_DIR}" \
-sv -m benchmark \
--benchmark-columns=min,max,mean,median,IQR,ops,rounds,iterations \
--benchmark-disable-gc \
......
test/tree/tree/base.py
浏览文件 @ df273bad
......@@ -4,8 +4,10 @@ from typing import Type
import pytest
from treevalue import raw, TreeValue, delayed
from test.tree.tree.test_constraint import GreaterThanConstraint
from treevalue import raw, TreeValue, delayed, ValidationError
from treevalue.tree.common import create_storage
from treevalue.tree.tree.constraint import cleaf
try:
_ = reversed({}.keys())
......@@ -41,6 +43,27 @@ class _Container:
def get_treevalue_test(treevalue_class: Type[TreeValue]):
# noinspection DuplicatedCode,PyMethodMayBeStatic
def get_demo_constraint_tree():
return treevalue_class({
'a': delayed(lambda x, y: x * (y + 1), 3, 6),
'b': delayed(lambda x: TreeValue({
'x': f'f-{x * x!r}',
'y': x * 1.1,
}), x=7)
}, constraint=[
object,
{
'a': [int, GreaterThanConstraint(3)],
'b': {
'x': [cleaf(), str, None],
'y': float,
},
'c': None,
}
])
# noinspection PyMethodMayBeStatic
@pytest.mark.unittest
class _TestClass:
def test_tree_value_init(self):
......@@ -521,4 +544,189 @@ def get_treevalue_test(treevalue_class: Type[TreeValue]):
('d', {'x': 2, 'y': 3}),
]
def test_validation(self):
t1 = treevalue_class({
'a': delayed(lambda x, y: x * (y + 1), 3, 6),
'b': delayed(lambda x: TreeValue({
'x': f'f-{x * x!r}',
'y': x * 1.1,
'z': None,
}), x=7)
}, constraint=[
object,
{
'a': [int, GreaterThanConstraint(3)],
'b': {
'x': [cleaf(), str, None],
'y': float,
},
'c': None,
}
])
t1.validate()
t2 = treevalue_class({
'a': delayed(lambda x, y: x * (y + 1), 3, 6),
'b': delayed(lambda x: TreeValue({
'x': f'f-{x * x!r}',
'y': x * 1,
}), x=7)
}, constraint=[
object,
{
'a': [int, GreaterThanConstraint(3)],
'b': {
'x': [cleaf(), str, None],
'y': float,
},
'c': None,
}
])
with pytest.raises(ValidationError) as ei:
t2.validate()
err = ei.value
self_, reterr, retpath, retcons = err.args
assert self_ == treevalue_class({'a': 21, 'b': {'x': 'f-49', 'y': 7}})
assert isinstance(reterr, TypeError)
assert retpath == ('b', 'y')
assert retcons == float
line1, *_ = str(err).splitlines(keepends=False)
assert line1 == "Validation failed on <TypeConstraint <class 'float'>> at position ('b', 'y')"
t3 = treevalue_class({
'a': delayed(lambda x, y: x * (y + 1), 3, 6),
'b': delayed(lambda x: TreeValue({
'x': f'f-{x * x!r}',
'y': x * 1,
}), x=7)
})
t3 = treevalue_class(t3, constraint=[
object,
{
'a': [int, GreaterThanConstraint(3)],
'b': {
'x': [cleaf(), str, None],
'y': float,
},
'c': None,
}
])
with pytest.raises(ValidationError) as ei:
t3.validate()
err = ei.value
self_, reterr, retpath, retcons = err.args
assert self_ == treevalue_class({'a': 21, 'b': {'x': 'f-49', 'y': 7}})
assert isinstance(reterr, TypeError)
assert retpath == ('b', 'y')
assert retcons == float
line1, *_ = str(err).splitlines(keepends=False)
assert line1 == "Validation failed on <TypeConstraint <class 'float'>> at position ('b', 'y')"
def test_constraint_get(self, ):
t1 = get_demo_constraint_tree()
assert t1.constraint.equiv([
object, {
'a': [int, GreaterThanConstraint(3)],
'b': {'x': [cleaf(), str], 'y': float}
}
])
assert t1.a == 21
t1b = t1.b
assert t1b.x == 'f-49'
assert t1b.y == pytest.approx(7.7)
assert t1b.constraint.equiv([object, {'x': [cleaf(), str], 'y': float}])
t1b = t1.get('b')
assert t1b.x == 'f-49'
assert t1b.y == pytest.approx(7.7)
assert t1b.constraint.equiv([object, {'x': [cleaf(), str], 'y': float}])
t1b = t1['b']
assert t1b.x == 'f-49'
assert t1b.y == pytest.approx(7.7)
assert t1b.constraint.equiv([object, {'x': [cleaf(), str], 'y': float}])
# noinspection PyTypeChecker
def test_constraint_pop(self):
t1 = get_demo_constraint_tree()
assert t1.constraint.equiv([
object, {
'a': [int, GreaterThanConstraint(3)],
'b': {'x': [cleaf(), str], 'y': float}
}
])
assert t1.pop('a') == 21
assert 'a' not in t1
t1b = t1.pop('b')
assert 'b' not in t1
assert t1b.x == 'f-49'
assert t1b.y == pytest.approx(7.7)
assert t1b.constraint.equiv([object, {'x': [cleaf(), str], 'y': float}])
def test_constraint_popitem(self):
t1 = get_demo_constraint_tree()
a_found, b_found = False, False
while t1:
key, value = t1.popitem()
if key == 'a':
assert not a_found, f'Duplicate key {"a"!r} found.'
assert value == 21
a_found = True
elif key == 'b':
assert not b_found, f'Duplicate key {"b"!r} found.'
assert value.x == 'f-49'
assert value.y == pytest.approx(7.7)
assert value.constraint.equiv([object, {'x': [cleaf(), str], 'y': float}])
b_found = True
else:
pytest.fail(f'Unexpected key {key!r} found.')
assert a_found and b_found, f'Key {"a"!r} or {"b"!r} not found in {t1!r}.'
def test_with_constraints(self):
t1 = get_demo_constraint_tree()
t2 = t1.with_constraints(GreaterThanConstraint(10))
assert t2.constraint.equiv([
GreaterThanConstraint(10),
object,
{
'a': [int, GreaterThanConstraint(3)],
'b': {'x': [cleaf(), str], 'y': float}
}
])
t3 = t1.with_constraints([GreaterThanConstraint(10), int], clear=True)
assert t3.constraint.equiv([int, GreaterThanConstraint(10)])
def test_pickle_constraints(self):
t1 = get_demo_constraint_tree()
assert t1.a == 21
assert t1.b.x == 'f-49'
assert t1.b.y == pytest.approx(7.7)
assert t1.constraint.equiv([
object, {
'a': [int, GreaterThanConstraint(3)],
'b': {'x': [cleaf(), str], 'y': float}
}
])
binary = pickle.dumps(t1)
newt1 = pickle.loads(binary)
assert newt1.a == 21
assert newt1.b.x == 'f-49'
assert newt1.b.y == pytest.approx(7.7)
assert newt1.constraint.equiv([
object, {
'a': [int, GreaterThanConstraint(3)],
'b': {'x': [cleaf(), str], 'y': float}
}
])
assert newt1 == t1
assert newt1.constraint == t1.constraint
return _TestClass
test/tree/tree/test_constraint.py 0 → 100644
浏览文件 @ df273bad
此差异已折叠。
test/tree/tree/test_tree_benchmark.py
浏览文件 @ df273bad
from functools import lru_cache
from operator import __eq__
import pytest
from treevalue import TreeValue, raw
from treevalue import TreeValue, raw, to_constraint
from .test_constraint import GreaterThanConstraint
_TREE_DATA = {'a': 1, 'b': 2, 'c': raw({'x': 3, 'y': 4}), 'd': {'x': 3, 'y': 4}}
_TREE = TreeValue(_TREE_DATA)
_TREE_CONSTRAINT = [GreaterThanConstraint(0), {'a': int, 'b': int, 'c': dict, 'd': {'x': int, 'y': int}}]
_TREE_DATA_2 = {'e': 3, 'f': 'klsjdfgklsdf', 'g': raw({'x': 3, 'y': 4}), 'c': {'x': 3, 'y': 4}}
_TREE_2 = TreeValue(_TREE_DATA_2)
......@@ -17,27 +20,49 @@ _TREE_DATA_4 = {'a': 1, 'b': 2, 'd': {'x': 3, 'y': 4}}
_TREE_4 = TreeValue(_TREE_DATA_4)
@pytest.mark.benchmark(group='treevalue_class')
# need to warm up when first run this
# because some features (e.g. child tree's constraint) will use cache
@pytest.mark.benchmark(group='treevalue_class', warmup=True, min_rounds=20)
class TestTreeValueBenchmark:
@lru_cache()
def __setup_tree(self):
return TreeValue(_TREE_DATA)
@lru_cache()
def __setup_constraint_tree(self):
return TreeValue(_TREE_DATA, constraint=_TREE_CONSTRAINT)
@pytest.mark.parametrize('data', [_TREE_DATA])
def test_init(self, benchmark, data):
result = benchmark(TreeValue, data)
assert result == _TREE
@pytest.mark.parametrize('data, constraint', [(_TREE_DATA, _TREE_CONSTRAINT),
(_TREE_DATA, to_constraint(_TREE_CONSTRAINT))])
def test_init_constraint(self, benchmark, data, constraint):
result = benchmark(TreeValue, data, constraint)
assert result == _TREE
assert result.constraint == constraint
def test_detach(self, benchmark):
benchmark(TreeValue._detach, self.__setup_tree())
@pytest.mark.parametrize('args', [('a',), ('a', 1), ('f', 1)])
@pytest.mark.parametrize('args', [('a',), ('a', 1), ('d',), ('f', 1)])
def test_get(self, benchmark, args):
benchmark(TreeValue.get, self.__setup_tree(), *args)
@pytest.mark.parametrize('key', ['a'])
@pytest.mark.parametrize('args', [('a',), ('a', 1), ('d',), ('f', 1)])
def test_get_constraint(self, benchmark, args):
benchmark(TreeValue.get, self.__setup_constraint_tree(), *args)
@pytest.mark.parametrize('key', ['a', 'd'])
def test_getattr(self, benchmark, key):
benchmark(getattr, self.__setup_tree(), key)
@pytest.mark.parametrize('key', ['a', 'd'])
def test_getattr_constraint(self, benchmark, key):
benchmark(getattr, self.__setup_constraint_tree(), key)
@pytest.mark.parametrize('key, data', [('a', 1), ('e', 233)])
def test_setattr(self, benchmark, key, data):
benchmark(setattr, self.__setup_tree(), key, data)
......
treevalue/tree/tree/__init__.py
浏览文件 @ df273bad
from .constraint import to_constraint, Constraint, NodeConstraint, ValueConstraint, cleaf, vval, vcheck, nval, ncheck
from .flatten import flatten, unflatten, flatten_values, flatten_keys
from .functional import mapping, filter_, mask, reduce_
from .graph import graphics
from .io import loads, load, dumps, dump
from .service import jsonify, clone, typetrans, walk
from .structural import subside, union, rise
from .tree import TreeValue, delayed
from .tree import TreeValue, delayed, ValidationError
treevalue/tree/tree/constraint.pxd 0 → 100644
浏览文件 @ df273bad
# distutils:language=c++
# cython:language_level=3
import cython
from libcpp cimport bool
cdef class _WrappedConstraintException(Exception):
pass
cdef class Constraint:
cpdef void _validate_node(self, object instance) except*
cpdef void _validate_value(self, object instance) except*
cpdef object _features(self)
cpdef bool _contains(self, Constraint other)
cpdef Constraint _transaction(self, str key)
cdef bool _feature_match(self, Constraint other)
cdef bool _contains_check(self, Constraint other)
cdef tuple _native_validate(self, object instance, type type_, list path)
cpdef tuple check(self, object instance)
cpdef bool equiv(self, object other)
@cython.final
cdef class EmptyConstraint(Constraint):
pass
cdef EmptyConstraint _EMPTY_CONSTRAINT
cdef Constraint _r_parse_cons(object obj)
cpdef Constraint to_constraint(object obj)
cdef class ValueConstraint(Constraint):
pass
cdef class NodeConstraint(Constraint):
pass
@cython.final
cdef class TypeConstraint(ValueConstraint):
cdef readonly type type_
cdef str _c_func_fullname(object f)
cdef class ValueFuncConstraint(ValueConstraint):
cdef readonly object func
cdef readonly str name
@cython.final
cdef class ValueValidateConstraint(ValueFuncConstraint):
pass
@cython.final
cdef class ValueCheckConstraint(ValueFuncConstraint):
pass
cpdef ValueValidateConstraint vval(object func, object name= *)
cpdef ValueCheckConstraint vcheck(object func, object name= *)
@cython.final
cdef class LeafConstraint(Constraint):
pass
cpdef LeafConstraint cleaf()
cdef class NodeFuncConstraint(NodeConstraint):
cdef readonly object func
cdef readonly str name
@cython.final
cdef class NodeValidateConstraint(NodeFuncConstraint):
pass
@cython.final
cdef class NodeCheckConstraint(NodeFuncConstraint):
pass
cpdef NodeValidateConstraint nval(object func, object name= *)
cpdef NodeCheckConstraint ncheck(object func, object name= *)
cdef class TreeConstraint(Constraint):
cdef readonly dict _constraints
cdef Constraint _s_tree_merge(list constraints)
cdef Constraint _s_tree(TreeConstraint constraint)
cdef class CompositeConstraint(Constraint):
cdef readonly tuple _constraints
cdef void _rec_composite_iter(Constraint constraint, list lst)
cdef list _r_composite_iter(Constraint constraint)
cdef Constraint _s_generic_merge(list constraints)
cdef Constraint _s_composite(CompositeConstraint constraint)
cdef Constraint _s_simplify(Constraint constraint)
cpdef Constraint transact(object cons, str key)
treevalue/tree/tree/constraint.pyx 0 → 100644
浏览文件 @ df273bad
# distutils:language=c++
# cython:language_level=3
import os
import cython
from libcpp cimport bool
from .tree cimport TreeValue
from ..common.storage cimport TreeStorage, _c_undelay_data
cdef class _WrappedConstraintException(Exception):
pass
cdef class Constraint:
cpdef void _validate_node(self, object instance) except*:
raise NotImplementedError # pragma: no cover
cpdef void _validate_value(self, object instance) except*:
raise NotImplementedError # pragma: no cover
cpdef object _features(self):
raise NotImplementedError # pragma: no cover
cpdef bool _contains(self, Constraint other):
raise NotImplementedError # pragma: no cover
cpdef Constraint _transaction(self, str key):
raise NotImplementedError # pragma: no cover
@cython.final
cdef inline bool _feature_match(self, Constraint other):
return type(self) == type(other) and self._features() == other._features()
@cython.final
cdef inline bool _contains_check(self, Constraint other):
return isinstance(other, EmptyConstraint) or self._feature_match(other) or self._contains(other)
@cython.final
cdef inline tuple _native_validate(self, object instance, type type_, list path):
cdef dict raw
cdef str key
cdef object value
cdef Constraint subcons
cdef bool retval
cdef tuple retpath
cdef Constraint retcons
cdef Exception reterr
if isinstance(instance, TreeStorage):
try:
self._validate_node(type_(instance))
except _WrappedConstraintException as err:
reterr, retcons = err.args
return False, tuple(path), retcons, reterr
except Exception as err:
return False, tuple(path), self, err
raw = instance.detach()
for key, value in raw.items():
value = _c_undelay_data(raw, key, value)
path.append(key)
subcons = self._transaction(key)
retval, retpath, retcons, reterr = subcons._native_validate(raw[key], type_, path)
path.pop()
if not retval:
return retval, retpath, retcons, reterr
else:
try:
self._validate_value(instance)
except _WrappedConstraintException as err:
reterr, retcons = err.args
return False, tuple(path), retcons, reterr
except Exception as err:
return False, tuple(path), self, err
return True, None, None, None
cpdef tuple check(self, object instance):
cdef list path = []
if isinstance(instance, TreeValue):
return self._native_validate(instance._detach(), type(instance), path)
else:
return self._native_validate(instance, TreeValue, path)
def validate(self, object instance):
cdef bool retval
cdef tuple retpath
cdef Constraint retcons
cdef Exception reterr
retval, retpath, retcons, reterr = self.check(instance)
if not retval:
raise reterr
def __eq__(self, other):
return self._feature_match(to_constraint(other))
def __hash__(self):
return hash(self._features())
cpdef bool equiv(self, object other):
cdef Constraint c = to_constraint(other)
return self._contains_check(c) and c._contains_check(self)
def __ge__(self, other):
cdef Constraint c = to_constraint(other)
return self._contains_check(c)
def __gt__(self, other):
cdef Constraint c = to_constraint(other)
return self._contains_check(c) and not c._contains_check(self)
def __le__(self, other):
cdef Constraint c = to_constraint(other)
return c._contains_check(self)
def __lt__(self, other):
cdef Constraint c = to_constraint(other)
return c._contains_check(self) and not self._contains_check(c)
def __repr__(self):
return f'<{type(self).__name__} {self._features()!r}>'
@cython.final
cdef class EmptyConstraint(Constraint):
cpdef inline void _validate_node(self, object instance) except*:
pass
cpdef inline void _validate_value(self, object instance) except*:
pass
cpdef inline object _features(self):
return ()
cpdef inline bool _contains(self, Constraint other):
return isinstance(other, EmptyConstraint)
cpdef inline Constraint _transaction(self, str key):
return self
def __bool__(self):
return False
def __repr__(self):
return f'<{type(self).__name__}>'
_EMPTY_CONSTRAINT = EmptyConstraint()
cdef inline Constraint _r_parse_cons(object obj):
if isinstance(obj, Constraint):
return obj
elif obj is None:
return _EMPTY_CONSTRAINT
elif isinstance(obj, type):
return TypeConstraint(obj)
elif isinstance(obj, (list, tuple)):
return CompositeConstraint([_r_parse_cons(c) for c in obj])
elif isinstance(obj, dict):
return TreeConstraint({key: _r_parse_cons(value) for key, value in obj.items()})
else:
raise TypeError(f'Invalid constraint - {obj!r}.')
cpdef inline Constraint to_constraint(object obj):
if obj is None:
return _EMPTY_CONSTRAINT
else:
return _s_simplify(_r_parse_cons(obj))
cdef class ValueConstraint(Constraint):
cpdef void _validate_node(self, object instance) except*:
pass
cpdef Constraint _transaction(self, str key):
return self
cdef class NodeConstraint(Constraint):
cpdef void _validate_value(self, object instance) except*:
raise TypeError(f'TreeValue node expected, but value {instance!r} found.')
cpdef Constraint _transaction(self, str key):
return _EMPTY_CONSTRAINT
cdef class TypeConstraint(ValueConstraint):
def __cinit__(self, type type_):
self.type_ = type_
cpdef void _validate_value(self, object instance) except*:
if not isinstance(instance, self.type_):
raise TypeError(f'Invalid type, {self.type_!r} expected but {type(instance)!r} found - {instance!r}.')
cpdef object _features(self):
return self.type_
cpdef bool _contains(self, Constraint other):
return isinstance(other, TypeConstraint) and issubclass(self.type_, other.type_)
def __reduce__(self):
return TypeConstraint, (self.type_,)
cdef inline str _c_func_fullname(object f):
cdef str fname = f.__name__
cdef str mname = getattr(f, '__module__', '')
return f'{mname}.{fname}' if mname else fname
cdef class ValueFuncConstraint(ValueConstraint):
def __cinit__(self, object func, str name):
self.func = func
self.name = name
cpdef object _features(self):
return self.name, self.func
cpdef bool _contains(self, Constraint other):
return type(self) == type(other) and self.func == other.func
def __repr__(self):
return f'<{type(self).__name__} {self.name}>'
def __reduce__(self):
return type(self), (self.func, self.name)
@cython.final
cdef class ValueValidateConstraint(ValueFuncConstraint):
cpdef inline void _validate_value(self, object instance) except*:
self.func(instance)
@cython.final
cdef class ValueCheckConstraint(ValueFuncConstraint):
cpdef inline void _validate_value(self, object instance) except*:
assert self.func(instance), f'Check of {self.name!r} failed.'
cpdef inline ValueValidateConstraint vval(object func, object name=None):
return ValueValidateConstraint(func, str(name or _c_func_fullname(func)))
cpdef inline ValueCheckConstraint vcheck(object func, object name=None):
return ValueCheckConstraint(func, str(name or _c_func_fullname(func)))
@cython.final
cdef class LeafConstraint(Constraint):
cpdef inline void _validate_node(self, object instance) except*:
raise TypeError(f'TreeValue leaf expected, but node found:{os.linesep}{instance!r}.')
cpdef inline void _validate_value(self, object instance) except*:
pass
cpdef inline object _features(self):
return None
cpdef inline bool _contains(self, Constraint other):
return isinstance(other, LeafConstraint)
cpdef inline Constraint _transaction(self, str key):
return _EMPTY_CONSTRAINT # pragma: no cover
def __repr__(self):
return f'<{type(self).__name__}>'
cpdef inline LeafConstraint cleaf():
return LeafConstraint()
cdef class NodeFuncConstraint(NodeConstraint):
def __cinit__(self, object func, str name):
self.func = func
self.name = name
cpdef object _features(self):
return self.name, self.func
cpdef bool _contains(self, Constraint other):
return type(self) == type(other) and self.func == other.func
def __repr__(self):
return f'<{type(self).__name__} {self.name}>'
def __reduce__(self):
return type(self), (self.func, self.name)
@cython.final
cdef class NodeValidateConstraint(NodeFuncConstraint):
cpdef inline void _validate_node(self, object instance) except*:
self.func(instance)
@cython.final
cdef class NodeCheckConstraint(NodeFuncConstraint):
cpdef inline void _validate_node(self, object instance) except*:
assert self.func(instance), f'Check of {self.name!r} failed.'
cpdef inline NodeValidateConstraint nval(object func, object name=None):
return NodeValidateConstraint(func, str(name or _c_func_fullname(func)))
cpdef inline NodeCheckConstraint ncheck(object func, object name=None):
return NodeCheckConstraint(func, str(name or _c_func_fullname(func)))
cdef class TreeConstraint(Constraint):
def __cinit__(self, dict constraints, bool need_sort=True):
if need_sort:
self._constraints = {key: constraints[key] for key in sorted(constraints.keys())}
else:
self._constraints = dict(constraints)
cpdef void _validate_node(self, object instance) except*:
pass
cpdef void _validate_value(self, object instance) except*:
raise TypeError(f'TreeValue node expected, but value {instance!r} found.')
cpdef object _features(self):
cdef str key
cdef list ft = []
for key in self._constraints:
ft.append((key, self._constraints[key]))
return tuple(ft)
cpdef bool _contains(self, Constraint other):
cdef str key
cdef Constraint _s_cons, _o_cons
cdef list _f_keys
if isinstance(other, TreeConstraint):
_f_keys = sorted(set(self._constraints.keys()) | set(other._constraints.keys()))
for key in _f_keys:
_s_cons = self._transaction(key)
_o_cons = other._transaction(key)
if not (_s_cons >= _o_cons):
return False
return True
else:
return False
cpdef Constraint _transaction(self, str key):
if key in self._constraints:
return self._constraints[key]
else:
return _EMPTY_CONSTRAINT
def __reduce__(self):
return TreeConstraint, (self._constraints, False)
cdef inline Constraint _s_tree_merge(list constraints):
cdef dict cmap = {}
cdef str key
cdef Constraint cons
cdef TreeConstraint tcons
for tcons in constraints:
for key, cons in tcons._constraints.items():
if key not in cmap:
cmap[key] = [cons]
else:
cmap[key].append(cons)
cdef list clist
cdef dict fmap = {}
for key, clist in cmap.items():
cons = _s_generic_merge(clist)
if cons:
fmap[key] = cons
if fmap:
return TreeConstraint(fmap)
else:
return _EMPTY_CONSTRAINT
cdef inline Constraint _s_tree(TreeConstraint constraint):
cdef dict dcons = {}
cdef list keys = sorted(constraint._constraints.keys())
cdef str key
cdef Constraint cons, pcons
for key in keys:
pcons = _s_simplify(constraint._constraints[key])
if not isinstance(pcons, EmptyConstraint):
dcons[key] = pcons
if dcons:
return TreeConstraint(dcons)
else:
return _EMPTY_CONSTRAINT
cdef class CompositeConstraint(Constraint):
def __cinit__(self, list constraints, bool need_sort=True):
if need_sort:
self._constraints = tuple(sorted(constraints, key=lambda x: repr(x._features())))
else:
self._constraints = tuple(constraints)
cpdef void _validate_node(self, object instance) except*:
cdef Constraint cons
for cons in self._constraints:
try:
cons._validate_node(instance)
except _WrappedConstraintException:
raise
except Exception as err:
raise _WrappedConstraintException(err, cons)
cpdef void _validate_value(self, object instance) except*:
cdef Constraint cons
for cons in self._constraints:
try:
cons._validate_value(instance)
except _WrappedConstraintException:
raise
except Exception as err:
raise _WrappedConstraintException(err, cons)
cpdef object _features(self):
return self._constraints
cpdef bool _contains(self, Constraint other):
cdef Constraint cons, pcons
cdef bool found_contains
if isinstance(other, CompositeConstraint):
for pcons in other._constraints:
found_contains = False
for cons in self._constraints:
if cons >= pcons:
found_contains = True
break
if not found_contains:
return False
return True
else:
for cons in self._constraints:
if cons >= other:
return True
return False
cpdef Constraint _transaction(self, str key):
return CompositeConstraint([c._transaction(key) for c in self._constraints])
def __reduce__(self):
return CompositeConstraint, (list(self._constraints), False)
cdef inline void _rec_composite_iter(Constraint constraint, list lst):
cdef Constraint cons
if isinstance(constraint, CompositeConstraint):
for cons in constraint._constraints:
_rec_composite_iter(cons, lst)
else:
lst.append(constraint)
cdef inline list _r_composite_iter(Constraint constraint):
cdef list lst = []
_rec_composite_iter(constraint, lst)
return lst
cdef inline Constraint _s_generic_merge(list constraints):
cdef Constraint gcons, cons, pcons
cdef list sins = []
cdef list trees = []
for gcons in constraints:
for cons in _r_composite_iter(gcons):
pcons = _s_simplify(cons)
if isinstance(pcons, TreeConstraint):
trees.append(pcons)
elif not isinstance(pcons, EmptyConstraint):
sins.append(pcons)
cdef Constraint tree = _s_tree_merge(trees)
if not isinstance(tree, EmptyConstraint):
sins.append(tree)
if not sins:
return _EMPTY_CONSTRAINT
cdef int i, j
cdef set _child_ids = set()
cdef int n = len(sins)
for i in range(n):
if i in _child_ids:
continue
for j in range(n):
if i == j or j in _child_ids:
continue
if sins[i] >= sins[j]:
_child_ids.add(j)
cdef list finals = []
for i in range(n):
if i not in _child_ids:
finals.append(sins[i])
assert finals, 'Finals should not be empty, but it\'s empty actually.'
if len(finals) == 1:
return finals[0]
else:
return CompositeConstraint(finals)
cdef inline Constraint _s_composite(CompositeConstraint constraint):
return _s_generic_merge(list(constraint._constraints))
cdef inline Constraint _s_simplify(Constraint constraint):
if isinstance(constraint, CompositeConstraint):
return _s_composite(constraint)
elif isinstance(constraint, TreeConstraint):
return _s_tree(constraint)
else:
return constraint
cpdef inline Constraint transact(object cons, str key):
cdef Constraint constraint
if isinstance(cons, EmptyConstraint):
return _EMPTY_CONSTRAINT
else:
constraint = to_constraint(cons)
# noinspection PyProtectedMember
return _s_simplify(constraint._transaction(key))
treevalue/tree/tree/tree.pxd
浏览文件 @ df273bad
......@@ -3,24 +3,38 @@
from libcpp cimport bool
from .constraint cimport Constraint
from ..common.delay cimport DelayedProxy
from ..common.storage cimport TreeStorage
cdef class _CObject:
pass
cdef class _SimplifiedConstraintProxy:
cdef readonly Constraint cons
cdef Constraint _c_get_constraint(object cons)
cdef class ValidationError(Exception):
cdef readonly TreeValue _object
cdef readonly Exception _error
cdef readonly tuple _path
cdef readonly Constraint _cons
cdef class TreeValue:
cdef readonly TreeStorage _st
cdef readonly Constraint constraint
cdef readonly type _type
cdef readonly dict _child_constraints
cpdef TreeStorage _detach(self)
cdef object _unraw(self, object obj)
cdef object _unraw(self, object obj, str key)
cdef object _raw(self, object obj)
cpdef _attr_extern(self, str key)
cpdef _getitem_extern(self, object key)
cpdef _setitem_extern(self, object key, object value)
cpdef _delitem_extern(self, object key)
cdef void _update(self, object d, dict kwargs) except *
cdef void _update(self, object d, dict kwargs) except*
cpdef public get(self, str key, object default= *)
cpdef public pop(self, str key, object default= *)
cpdef public popitem(self)
......@@ -31,6 +45,8 @@ cdef class TreeValue:
cpdef public treevalue_values values(self)
cpdef public treevalue_items items(self)
cpdef void validate(self) except*
cdef str _prefix_fix(object text, object prefix)
cdef str _title_repr(TreeStorage st, object type_)
cdef object _build_tree(TreeStorage st, object type_, str prefix, dict id_pool, tuple path)
......@@ -44,11 +60,19 @@ cdef class treevalue_keys(_CObject):
cdef class treevalue_values(_CObject):
cdef readonly TreeStorage _st
cdef readonly type _type
cdef readonly Constraint _constraint
cdef readonly dict _child_constraints
cdef _SimplifiedConstraintProxy _transact(self, str key)
# noinspection PyPep8Naming
cdef class treevalue_items(_CObject):
cdef readonly TreeStorage _st
cdef readonly type _type
cdef readonly Constraint _constraint
cdef readonly dict _child_constraints
cdef _SimplifiedConstraintProxy _transact(self, str key)
cdef class DetachedDelayedProxy(DelayedProxy):
cdef DelayedProxy proxy
......
treevalue/tree/tree/tree.pyx
浏览文件 @ df273bad
......@@ -8,6 +8,7 @@ from operator import itemgetter
import cython
from hbutils.design import SingletonMark
from .constraint cimport Constraint, to_constraint, transact, _EMPTY_CONSTRAINT
from ..common.delay cimport undelay, _c_delayed_partial, DelayedProxy
from ..common.storage cimport TreeStorage, create_storage, _c_undelay_data
from ...utils import format_tree
......@@ -47,6 +48,28 @@ cdef inline TreeStorage _dict_unpack(dict d):
_DEFAULT_STORAGE = create_storage({})
cdef class _SimplifiedConstraintProxy:
def __cinit__(self, Constraint cons):
self.cons = cons
cdef inline Constraint _c_get_constraint(object cons):
if isinstance(cons, _SimplifiedConstraintProxy):
return cons.cons
else:
return to_constraint(cons)
cdef class ValidationError(Exception):
def __init__(self, TreeValue obj, Exception error, tuple path, Constraint cons):
Exception.__init__(self, obj, error, path, cons)
self._object = obj
self._error = error
self._path = path
self._cons = cons
def __str__(self):
return f"Validation failed on {self._cons!r} at position {self._path!r}{os.linesep}" \
f"{type(self._error).__name__}: {self._error}"
cdef class TreeValue:
r"""
Overview:
......@@ -56,12 +79,14 @@ cdef class TreeValue:
The `TreeValue` class is a light-weight framework just for DIY.
"""
def __cinit__(self, object data):
def __cinit__(self, object data, object constraint=None):
self._st = _DEFAULT_STORAGE
self.constraint = _EMPTY_CONSTRAINT
self._type = type(self)
self._child_constraints = {}
@cython.binding(True)
def __init__(self, object data):
def __init__(self, object data, object constraint=None):
"""
Constructor of :class:`TreeValue`.
......@@ -91,10 +116,17 @@ cdef class TreeValue:
"""
if isinstance(data, TreeStorage):
self._st = data
self.constraint = _c_get_constraint(constraint)
elif isinstance(data, TreeValue):
self._st = data._detach()
if constraint is None:
self.constraint = data.constraint
self._child_constraints = data._child_constraints
else:
self.constraint = _c_get_constraint(constraint)
elif isinstance(data, dict):
self._st = _dict_unpack(data)
self.constraint = _c_get_constraint(constraint)
else:
raise TypeError(
"Unknown initialization type for tree value - {type}.".format(
......@@ -117,9 +149,15 @@ cdef class TreeValue:
"""
return self._st
cdef inline object _unraw(self, object obj):
cdef inline object _unraw(self, object obj, str key):
cdef _SimplifiedConstraintProxy child_constraint
if isinstance(obj, TreeStorage):
return self._type(obj)
if key in self._child_constraints:
child_constraint = self._child_constraints[key]
else:
child_constraint = _SimplifiedConstraintProxy(transact(self.constraint, key))
self._child_constraints[key] = child_constraint
return self._type(obj, constraint=child_constraint)
else:
return obj
......@@ -156,7 +194,7 @@ cdef class TreeValue:
>>> t.get('f', 123)
123
"""
return self._unraw(self._st.get_or_default(key, default))
return self._unraw(self._st.get_or_default(key, default), key)
@cython.binding(True)
cpdef pop(self, str key, object default=_GET_NO_DEFAULT):
......@@ -196,7 +234,7 @@ cdef class TreeValue:
else:
value = self._st.pop_or_default(key, default)
return self._unraw(value)
return self._unraw(value, key)
@cython.binding(True)
cpdef popitem(self):
......@@ -229,7 +267,7 @@ cdef class TreeValue:
cdef object v
try:
k, v = self._st.popitem()
return k, self._unraw(v)
return k, self._unraw(v, k)
except KeyError:
raise KeyError(f'popitem(): {self._type.__name__} is empty.')
......@@ -307,9 +345,9 @@ cdef class TreeValue:
├── 'ff' --> None
└── 'g' --> 1
"""
return self._unraw(self._st.setdefault(key, self._raw(default)))
return self._unraw(self._st.setdefault(key, self._raw(default)), key)
cdef inline void _update(self, object d, dict kwargs) except *:
cdef inline void _update(self, object d, dict kwargs) except*:
cdef object dt
if d is None:
dt = {}
......@@ -421,7 +459,7 @@ cdef class TreeValue:
# new order: self._st, __dict__, self._attr_extern
# this may cause problem when pickle.loads, so __getnewargs_ex__ and __cinit__ is necessary
if self._st.contains(item):
return self._unraw(self._st.get(item))
return self._unraw(self._st.get(item), item)
else:
try:
return object.__getattribute__(self, item)
......@@ -528,7 +566,7 @@ cdef class TreeValue:
3
"""
if isinstance(key, str):
return self._unraw(self._st.get(key))
return self._unraw(self._st.get(key), key)
else:
return self._getitem_extern(_item_unwrap(key))
......@@ -754,7 +792,7 @@ cdef class TreeValue:
return False
@cython.binding(True)
def __setstate__(self, TreeStorage state):
def __setstate__(self, tuple state):
"""
Deserialize operation, can support `pickle.loads`.
......@@ -769,7 +807,7 @@ cdef class TreeValue:
>>> bin_ = pickle.dumps(t) # dump it to binary
>>> pickle.loads(bin_) # TreeValue({'a': 1, 'b': 2, 'x': {'c': 3}})
"""
self._st = state
self._st, self.constraint = state
@cython.binding(True)
def __getstate__(self):
......@@ -786,7 +824,7 @@ cdef class TreeValue:
>>> bin_ = pickle.dumps(t) # dump it to binary
>>> pickle.loads(bin_) # TreeValue({'a': 1, 'b': 2, 'x': {'c': 3}})
"""
return self._st
return self._st, self.constraint
@cython.binding(True)
cpdef treevalue_keys keys(self):
......@@ -875,6 +913,25 @@ cdef class TreeValue:
"""
return treevalue_items(self, self._st)
@cython.binding(True)
cpdef void validate(self) except*:
cdef bool retval
cdef tuple retpath
cdef Constraint retcons
cdef Exception reterr
if __debug__:
retval, retpath, retcons, reterr = self.constraint.check(self)
if not retval:
raise ValidationError(self, reterr, retpath, retcons)
@cython.binding(True)
def with_constraints(self, object constraint, bool clear=False):
if clear:
return self._type(self._st, to_constraint(constraint))
else:
return self._type(self._st, to_constraint([constraint, self.constraint]))
cdef str _prefix_fix(object text, object prefix):
cdef list lines = []
cdef int i
......@@ -952,6 +1009,8 @@ cdef class treevalue_values(_CObject, Sized, Container, Reversible):
def __cinit__(self, TreeValue tv, TreeStorage storage):
self._st = storage
self._type = type(tv)
self._constraint = tv.constraint
self._child_constraints = {}
def __len__(self):
return self._st.size()
......@@ -963,10 +1022,19 @@ cdef class treevalue_values(_CObject, Sized, Container, Reversible):
return False
cdef inline _SimplifiedConstraintProxy _transact(self, str key):
cdef _SimplifiedConstraintProxy cons
if key in self._child_constraints:
return self._child_constraints[key]
else:
cons = _SimplifiedConstraintProxy(transact(self._constraint, key))
self._child_constraints[key] = cons
return cons
def _iter(self):
for v in self._st.iter_values():
for k, v in self._st.iter_items():
if isinstance(v, TreeStorage):
yield self._type(v)
yield self._type(v, self._transact(k))
else:
yield v
......@@ -974,9 +1042,9 @@ cdef class treevalue_values(_CObject, Sized, Container, Reversible):
return self._iter()
def _rev_iter(self):
for v in self._st.iter_rev_values():
for k, v in self._st.iter_rev_items():
if isinstance(v, TreeStorage):
yield self._type(v)
yield self._type(v, self._transact(k))
else:
yield v
......@@ -994,6 +1062,8 @@ cdef class treevalue_items(_CObject, Sized, Container, Reversible):
def __cinit__(self, TreeValue tv, TreeStorage storage):
self._st = storage
self._type = type(tv)
self._constraint = tv.constraint
self._child_constraints = {}
def __len__(self):
return self._st.size()
......@@ -1005,10 +1075,19 @@ cdef class treevalue_items(_CObject, Sized, Container, Reversible):
return False
cdef inline _SimplifiedConstraintProxy _transact(self, str key):
cdef _SimplifiedConstraintProxy cons
if key in self._child_constraints:
return self._child_constraints[key]
else:
cons = _SimplifiedConstraintProxy(transact(self._constraint, key))
self._child_constraints[key] = cons
return cons
def _iter(self):
for k, v in self._st.iter_items():
if isinstance(v, TreeStorage):
yield k, self._type(v)
yield k, self._type(v, self._transact(k))
else:
yield k, v
......@@ -1018,7 +1097,7 @@ cdef class treevalue_items(_CObject, Sized, Container, Reversible):
def _rev_iter(self):
for k, v in self._st.iter_rev_items():
if isinstance(v, TreeStorage):
yield k, self._type(v)
yield k, self._type(v, self._transact(k))
else:
yield k, v
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或