Set of functions used to preprocess french text characters.
External dependencies :
pip install pandas
Configure tabular data display in this notebook :
Character set normalization for french¶
The config object from frenchtext.core defines the directory where the character normalization tables are located :
!ls {chardatadir}
0. Unicode characters properties¶
charname("🙂")
charcategory("🙂")
charsubcategory("🙂")
charblock("🙂")
blockfamily('Emoticons')
1. Explore french dataset characters¶
French datasets often contain several thousands distinct Unicode characters.
We need to reduce the number of distinct characters fed to our natural language processing applications, for three reasons :
- chars considered by the user as visually equivalent will often produce a different application behavior : this is a huge problem for the user experience
- with so many chars, the designer of the NLP application will not be able to reason about all possible combinations : this could harm the explainability of the system
- this huge number of distinct characters brings a significant amount complexity the NLP models will have to deal with
1.1 Characters frequency in french datasets¶
dfcharstats = pd.read_csv(chardatadir / "charsetstats_raw.csv", sep=";")
dfcharstats
1.2 Characters stats in Wikipedia dataset¶
- 35.6 billion chars
charsCountWikipedia = dfcharstats["CountWikipedia"].sum()
charsCountWikipedia
- 13 502 distinct Unicode chars
distinctCharsWikipedia = len(dfcharstats[dfcharstats["CountWikipedia"]>0])
distinctCharsWikipedia
- Only 1316 chars more frequent than 1 in 100 million
frequentCharsWikipedia = len(dfcharstats[dfcharstats["CountWikipedia"]>356])
frequentCharsWikipedia
- Frequent chars represent 9.7 % of all distinct Unicode chars
pctFreqCharsWikipedia = frequentCharsWikipedia/distinctCharsWikipedia*100
pctFreqCharsWikipedia
- 99.9987 % of Wikipedia chars would be preserved if we only kept the frequent chars
pctPreservedCharsWikipedia = (1-dfcharstats[dfcharstats["CountWikipedia"]<=356]["CountWikipedia"].sum()/dfcharstats["CountWikipedia"].sum())*100
pctPreservedCharsWikipedia
1.3 Characters stats in Business dataset¶
- 27.5 billion chars
charsCountBusiness = dfcharstats["CountBusiness"].sum()
charsCountBusiness
- 3 763 distinct Unicode chars
distinctCharsBusiness = len(dfcharstats[dfcharstats["CountBusiness"]>0])
distinctCharsBusiness
- Only 531 chars more frequent than 1 in 100 million
frequentCharsBusiness = len(dfcharstats[dfcharstats["CountBusiness"]>275])
frequentCharsBusiness
- Frequent chars represent 14.1 % of all distinct Unicode chars
pctFreqCharsBusiness = frequentCharsBusiness/distinctCharsBusiness*100
pctFreqCharsBusiness
- 99.9996 % of Business chars would be preserved if we only kept the frequent chars
pctPreservedCharsBusiness = (1-dfcharstats[dfcharstats["CountBusiness"]<=275]["CountBusiness"].sum()/dfcharstats["CountBusiness"].sum())*100
pctPreservedCharsBusiness
- 99.985 % of Wikipedia chars would be preserved if we only kept the frequent Business chars
pctPreservedBizCharsInWikipedia = (1-dfcharstats[dfcharstats["CountBusiness"]<=275]["CountWikipedia"].sum()/dfcharstats["CountWikipedia"].sum())*100
pctPreservedBizCharsInWikipedia
1.4 Character stats after Unicode normalization¶
After applying the normalization process defined below in this notebook, here are the remaining chars :
dfcharsnorm = pd.read_csv(chardatadir / "charset-fr.csv", sep=";")
dfcharsnorm
Stats for the character families after normalization¶
The table below shows the number of chars in each category (after normalization) per 100 million characters :
dfblocks = dfcharsnorm.groupby(by=["Category","SubCategory"]).agg({"Char":["count","sum"],"CountBusiness":"sum"})
dfblocks["CountBusiness"] = (dfblocks["CountBusiness"] / charsCountBusiness * 100000000).astype(int)
dfblocks
2. Characters normalization pipeline¶
After a detailed study of all the frequent chars, the goal is to design a noramization pipeline which can retain as much information as possible while greatly reducing the number of dinstinct chars.
We saw before that it is possible to preserve 99.9996% of the original chars while keeping only 500 distinct chars. By being clever and replacing equivalent chars, we can divide this number by 2 and still retain the same amount of information.
It may then be useful to limit the number of distinct characters after normalization to 255 distinct characters :
- if needed, french text chars can then be encoded with a single byte
- the list of supported chars can be memorized by NLP application developers and users
The normalization pipeline applies the following 14 steps, which are explained and illustrated in the sections below.
- Fix encoding errors
- fix windows1252 text read as iso8859-1
- fix utf8 text read as windows1252
- fix windows1252 text read as utf8
- merge Unicode combining chars
- ignore control chars
- Remove display attributes
- replace latin letter symbols
- replace latin letter ligatures
- replace latin number symbols
- Normalize visually equivalent chars
- replace equivalent chars
- replace cyrillic and greek chars looking like latin letters
- Encode infrequent chars while losing a little bit of information
- replace infrequent latin letters with diacritics
- replace infrequent chars from other scripts
- replace infrequent symbols
- ignore remaining chars with no glyph
2.1 Frequent encoding errors : windows1252 read as iso8859-1¶
dfencodingwin1252 = pd.read_csv(chardatadir / "windows1252-iso8859-errors.csv", sep=";")
dfencodingwin1252.head(10)
print(f"{len(dfencodingwin1252)} frequent encoding errors seen in french datasets : a character encoded as windows1252 was incorrectly decoded as iso8859-1")
Columns :
- Code/Char : incorrectly decoded control char seen in french text
- DecodedCode/DecodedChare : properly decoded char which should replace the original control char
2.2 Frequent encoding errors : utf8 read as windows1252¶
dfencodingutf8 = pd.read_csv(chardatadir / "utf8-windows1252-errors.csv", sep=";")
dfencodingutf8.head(10)
print(f"{len(dfencodingutf8)} very unlikely substrings produced when text encoded with UTF-8 is decoded by mistake as iso8859-1 or windows1252")
Columns :
- ErrorSubstring : unlikely substring of length 2 or 3 characters produced when UTF-8 text is decoded by mistake as windows1252
- DecodedCode/DecodedChar : properly decoded char which should be used to replace the unlikley substring
2.3 Frequent encoding errors : windows1252 read as utf8¶
dfencodingwin1252utf8 = pd.read_csv(chardatadir / "windows1252-utf8-errors.csv", sep=";")
dfencodingwin1252utf8.head()
print(f"{len(dfencodingwin1252utf8)} char very unlikely in french text produced when text encoded with iso8859-1 or windows1252 is decoded by mistake as UTF-8")
Columns :
- Char : unlikely char produced when text encoded with iso8859-1 or windows1252 is decoded by mistake as UTF-8
- DecodedCodes/DecodedChars : properly decoded substring which should be used to replace the unlikley char
2.4 Unicode combining chars¶
dfcombiningchars = pd.read_csv(chardatadir / "combiningdiacritics.csv", sep=";")
dfcombiningchars.head()
print(f"{len(dfcombiningchars['Char'].unique())} combining chars {list(dfcombiningchars['Diacritic'].unique())} should be recombined with {len(dfcombiningchars)} base latin characters to produce standard latin characters with diacritics")
Columns :
- BaseChar : latin char encountered first in the string, which will be modified by the combining char immediately following it
- Code/Char : combining char immediately following BaseChar, which should be combined with it to produce CombinedChar
- Diacritic : type of accent / diacritic applied by the combining char
- CombinedChar : latin char with diacritic produced by the combination of BaseChar and the combining Char following it
2.5 Control chars¶
dfcontrolchars = pd.read_csv(chardatadir / "controlchars.csv", sep=";")
dfcontrolchars.loc[0,"Char"] = chr(0) # chr(0) can't be saved in CSV file
dfcontrolchars
print(f"{len(dfcontrolchars)} control chars seen in french datasets, which can't be displayed and should be ignored")
Columns :
- Code : Unicode code point for the character
- Char : control character
- CharName : name of the character in the Python Unicode database
2.6 Latin letter symbols¶
dflatinsymbols = pd.read_csv(chardatadir / "latinsymbols.csv", sep=";")
dflatinsymbols.head(10)
dflatinsymbols[230:240]
print(f"{len(dflatinsymbols)} Unicode symbols which represent latin letters with a specific layout like {list(dflatinsymbols['Layout'].unique())}")
Columns :
- Code/Char/CharName : Unicode symbol representing a latin letter with a specific layout
- NormString : normalized string using only very frequent chars
- Layout : info about the specific layout applied to the latin char
2.7 Latin letters ligatures / Latin letters diacritics¶
dflatinletters = pd.read_csv(chardatadir / "latinletters.csv", sep=";")
dflatinletters[89:99]
print(f"{len(dflatinletters)} chars representing latin letters, {len(dflatinletters[dflatinletters['IsUpper']])} upper case and {len(dflatinletters[dflatinletters['IsLower']])} lower case, {len(dflatinletters[dflatinletters['IsDiacritic']])} with diacritics like {list(dflatinletters[dflatinletters['IsDiacritic']]['Diacritics'].unique())[0:20]}, {len(dflatinletters[dflatinletters['IsLigature']])} representing multiple letters in ligature")
Columns :
- Code/Char/CharName : Unicode character representing one or more latin letters
- LetterName : name of the latin letter (without case and diacritics qualifiers)
- IsUpper/UpperChar and IsLower/LowerChar : upper case or lower case equivalent chars
- IsDiacritic => BaseChar : equivalent char without any diacritic (accents ...), Diacritics : description of all diacritics applied to the char
- IsLigature => MultiChars : if the char represents multiple latin letters in a single ligature, string representing the equivalent list of letters
- Block/Category/SubCategory : Unicode classification for each char
2.8 Latin numbers and number symbols¶
dflatinnumbers = pd.read_csv(chardatadir / "latinnumbers.csv", sep=";")
dflatinnumbers[30:40]
dflatinnumbers[200:210]
print(f"{len(dflatinnumbers)} chars representing latin digits, some with specific layouts like {list(dflatinnumbers['Layout'].unique())[1:]}")
Columns :
- Code/Char/CharName : Unicode char representing on or more latin digits
- NormString : normalized string representing the equivalent number, plus punctuation if needed
- Layout : info about the specific layout applied to the latin digits
2.9 Variations on frequent chars to normalize¶
dfnormchars = pd.read_csv(chardatadir / "normalizedchars.csv", sep=";")
dfnormchars.head()
dfnormchars[49:59]
dfnormchars[25:35]
print(f"{len(dfnormchars)} alternative chars which are sometimes used as equivalent visual representations for {len(dfnormchars['NormChar'].unique())} other very frequent chars")
Columns :
- Code/Char/CharName : alternative Unicode char often used as a visual equivalent of a more frequent char
- NormCode/NormChar/NormCharName : more frequent char which should be used to normalize text
2.10 Cyrillic and greek chars looking like latin letters¶
dfcgnormchars = pd.read_csv(chardatadir / "cyrillic-greek-chars.csv", sep=";")
dfcgnormchars[5:15]
print(f"{len(dfcgnormchars)} cyrillic and greek chars used as equivalent visual representations for {len(dfcgnormchars['NormChar'].unique())} latin letters")
NOTE : this standardization step is optional.
Even if it sounds strange, all cyrillic and greek letters in the table above are most often used as equivalent of latin letters in the french datasets.
Columns :
- Code/Char/CharName : cyrillic or greek char often used as a visual equivalent of a latin letter
- NormCode/NormChar/NormCharName : more frequent char which should be used to normalize text
2.11 Replace infrequent latin letters with diacritics¶
supportedchars = dfcharsnorm["Char"].values[1:]
' '.join(supportedchars)
latinlettersnodiacritics = {}
for rowidx,row in dflatinletters.iterrows():
if row["IsDiacritic"]:
latinlettersnodiacritics[row["Char"]] = row["BaseChar"]
for idx,letter in enumerate(latinlettersnodiacritics):
if not letter in supportedchars:
print(f"{letter} => {latinlettersnodiacritics[letter]}")
if idx >= 60 : break
2.12 Replace infrequent chars from other scripts¶
def replaceotherscripts(charset, chariterator):
for char in chariterator:
if char in charset:
yield char
else:
family = blockfamily(charblock(char))
if not family in ("Symbols","Ignore"):
resStr = chr(65532) + str(ord(char)) + '_'
for outchar in resStr:
yield outchar
else:
yield char
''.join(replaceotherscripts(supportedchars,"Guānhuà (官话/官話)"))
All characters from non latin scripts are preserved by encoding them with the following sequence :
[object replacement character] + unicode char number + [underscore]
This is necessary to preserve the distinct entity names in unsupported scripts, and enables decoding with full fidelity at a later stage of the pipeline.
2.13 Replace infrequent symbols¶
def replacesymbols(charset, chariterator):
for index,char in enumerate(chariterator):
if char in charset:
yield char
else:
family = blockfamily(charblock(char))
if family == "Symbols":
resStr ='$' + charname(char).replace(' ','') + '_'
for outchar in resStr:
yield outchar
else:
yield char
''.join(replacesymbols(supportedchars,"😀😈😎🙈🙌"))
All unsupported symbols are preserved by encoding them with the following sequence :
[dollar] + unicode char name + [underscore]
This enables a NLP pipeline to add english words to its vocabulary if some symbols are used frequently in the context of a sentiment analysis task.
2.14 Ignore remaining chars with no glyph¶
unicodefamilies[unicodefamilies["CharFamily"]=="Ignore"]
3. Text normalization¶
3.1 Normalization functions¶
We need to apply several replacement functions in a row, each replacement function building on the replacements already applied by the previous ones.
We can't simply use replace statements on immutable strings to do this : we would need to allocate new strings for each replacement at each level, and this would put a high load on the garbage collector.
A better solution is to implement our normalization function as a chain of iterators on chars.
Examples :
def ignorechars(chariterator, charset):
for char in chariterator:
if not char in charset:
yield char
def replacechars1to1(chariterator, chardict):
for char in chariterator:
if char in chardict:
yield chardict[char]
else:
yield char
def replacechars1toN(chariterator, chardict):
for char in chariterator:
if char in chardict:
for outchar in chardict[char]:
yield outchar
else:
yield char
To match several chars in an iterator, we have to build a hierarchical dictionary structure.
For example, if we want to implement the following replacements :
ABC => 1
ABD => 2
AC => 3
BC => 4We build the following dictionary structure :
A : { B : { C : 1
D : 2
C : 3 }
B : { C : 4 }The normalization functions are then chained in a chars replacement pipeline.
3.2 Normalization class with change tracking¶
%time norm = TextNormalizer()
norm
teststring = chr(127995)+"① l`"+chr(156)+"uv"+chr(127)+"re est¨ "+chr(147)+"belle"+chr(148)+"¸ à ½ € énième ‰ "+chr(133)+" ⁽🇪ffic🇦ce⁾ ！"
teststring
result = norm(teststring)
result
print(result.describeChanges())
result.output[0:12]
result.input[result.mapOutputIndexToInput(0):result.mapOutputIndexToInput(12)]
result.output[3:10]
result.input[result.mapOutputIndexToInput(3):result.mapOutputIndexToInput(10)]
%timeit -n100 norm(teststring)
3.3 Normalization pipeline stats¶
The statistics below count the number of chars normalized for 1 million chars in 4 distinct parts of the french datasets : business websites, forums, news, wikipedia.
The first line of the table below shows that :
- in 1 million chars extracted from forum pages (raw users input), 41.8 chars will be encoding errors (windows1252 read as iso8859-1)
- in 1 million chars extracted from business websites (curated content), only 0.5 chars will be encoding errors
normstats = pd.read_csv(chardatadir / "stats" / "normalization.total.stats.csv")
normstats[["Transform","FreqBusiness","FreqForum","FreqPresse","FreqWikipedia"]]
Most frequent chars replaced from equivalent characters :
replacestats = pd.read_csv(chardatadir / "stats" / "normalization.layer8.stats.csv")
replacestats[["Char","CharName","FreqBusiness","FreqForum","FreqPresse","FreqWikipedia"]].head(20)
Frequency of characters from other scripts :
scriptsstats = pd.read_csv(chardatadir / "stats" / "normalization.layer11.stats.csv")
scriptsstats[["CharFamily","FreqBusiness","FreqForum","FreqPresse","FreqWikipedia"]]
Detailed stats for the 14 layers of the normalization pipeline :
layersstats = pd.read_csv(chardatadir / "stats" / "normalization.stats.csv")
layer=8
layersstats[layersstats["Layer"]==layer][["Layer","Input","CharName","Output","CountBusiness","CountForum","CountPresse","CountWikipedia"]].head(15)