utils¶
-
download_file
(save_dir, filename, url, md5=None)[source]¶ Download the file from the url to specified directory. Check md5 value when the file is exists, if the md5 value is the same as the existed file, just use the older file, if not, will download the file from the url.
- Parameters
save_dir (string) – The specified directory saving the file.
filename (string) – The specified filename saving the file.
url (string) – The url downling the file.
md5 (string, optional) – The md5 value that checking the version downloaded.
-
download_check
(task)[source]¶ Check the resource statuc in the specified task.
- Parameters
task (string) – The name of specified task.
-
cut_chinese_sent
(para)[source]¶ Cut the Chinese sentences more precisely, reference to “https://blog.csdn.net/blmoistawinde/article/details/82379256”.
-
class
TermTreeNode
(sid: str, term: str, base: str, node_type: str = 'term', term_type: Optional[str] = None, hyper: Optional[str] = None, level: Optional[int] = None, alias: Optional[List[str]] = None, alias_ext: Optional[List[str]] = None, sub_type: Optional[List[str]] = None, sub_term: Optional[List[str]] = None, data: Optional[Dict[str, Any]] = None)[source]¶ Bases:
object
Defination of term node. All members are protected, to keep rigorism of data struct.
- Parameters
sid (str) – term id of node.
term (str) – term, common name of this term.
base (str) –
cb
indicates concept base,eb
indicates entity base.term_type (Optional[str], optional) – type of this term, constructs hirechical of
term
node. Defaults to None.hyper (Optional[str], optional) – parent type of a
type
node. Defaults to None.node_type (str, optional) – type statement of node,
type
orterm
. Defaults to “term”.alias (Optional[List[str]], optional) – alias of this term. Defaults to None.
alias_ext (Optional[List[str]], optional) – extended alias of this term, CANNOT be used in matching. Defaults to None.
sub_type (Optional[List[str]], optional) – grouped by some term. Defaults to None.
sub_term (Optional[List[str]], optional) – some lower term. Defaults to None.
data (Optional[Dict[str, Any]], optional) – to sore full imformation of a term. Defaults to None.
-
class
TermTree
[source]¶ Bases:
object
TermTree class.
-
add_term
(term: Optional[str] = None, base: Optional[str] = None, term_type: Optional[str] = None, sub_type: Optional[List[str]] = None, sub_term: Optional[List[str]] = None, alias: Optional[List[str]] = None, alias_ext: Optional[List[str]] = None, data: Optional[Dict[str, Any]] = None)[source]¶ Add a term into TermTree.
- Parameters
term (str) – common name of name.
base (str) – term is concept or entity.
term_type (str) – term type of this term
sub_type (Optional[List[str]], optional) – sub type of this term, must exists in TermTree. Defaults to None.
sub_terms (Optional[List[str]], optional) – sub terms of this term. Defaults to None.
alias (Optional[List[str]], optional) – alias of this term. Defaults to None.
alias_ext (Optional[List[str]], optional) – . Defaults to None.
data (Optional[Dict[str, Any]], optional) – [description]. Defaults to None.
-
find_term
(term: str, term_type: Optional[str] = None) → Tuple[bool, Optional[List[str]]][source]¶ Find a term in Term Tree. If term not exists, return None. If
term_type
is not None, will find term with this type.- Parameters
term (str) – term to look up.
term_type (Optional[str], optional) – find term in this term_type. Defaults to None.
- Returns
[description]
- Return type
Union[None, List[str]]
-
build_from_dir
(term_schema_path, term_data_path, linking=True)[source]¶ Build TermTree from a directory which should contain type schema and term data.
- Parameters
dir ([type]) – [description]
-
classmethod
from_dir
(term_schema_path, term_data_path, linking) → paddlenlp.taskflow.utils.TermTree[source]¶ Build TermTree from a directory which should contain type schema and term data.
- Parameters
source_dir ([type]) – [description]
- Returns
[description]
- Return type
-
-
levenstein_distance
(s1: str, s2: str) → int[source]¶ Calculate minimal Levenstein distance between s1 and s2.
- Parameters
s1 (str) – string
s2 (str) – string
- Returns
the minimal distance.
- Return type
int
-
class
BurkhardKellerNode
(word: str)[source]¶ Bases:
object
Node implementatation for BK-Tree. A BK-Tree node stores the information of current word, and its approximate words calculated by levenstein distance.
- Parameters
word (str) – word of current node.
-
class
BurkhardKellerTree
[source]¶ Bases:
object
Implementataion of BK-Tree