mirror of
https://github.com/moses-smt/mosesdecoder.git
synced 2024-09-11 11:25:40 +03:00
I wanted to properly parse links on https://dumps.wikimedia.org/mirrors.html when page copied as text My proposed changes does the job. Basically I had to change by replacing the + at end of line 5 with *(\/)? The pipe symbol could lead to crashes why I broke up line 5 to three lines. I suggest not using the pipe (|) after reading various posts. |
||
---|---|---|
.. | ||
mosestokenizer | ||
basic-protected-patterns | ||
deescape-special-chars-PTB.perl | ||
deescape-special-chars.perl | ||
delete-long-words.perl | ||
detokenizer.perl | ||
escape-special-chars.perl | ||
lowercase.perl | ||
normalize-punctuation.perl | ||
pre_tokenize_cleaning.py | ||
pre-tok-clean.perl | ||
pre-tokenizer.perl | ||
remove-non-printing-char.perl | ||
replace-unicode-punctuation.perl | ||
tokenizer_PTB.perl | ||
tokenizer.perl |