Discussion
Yesterday I studied how Papyrix splits words up further into chunks which should get hyphenated. So with all that extra complexity swept away how does the core logic here, the Liang Hyphenation Algorithm, work?
1st up a helper validates it has a subword, populates initial byte+offset arrays, iterates over the codepoints validating them & populating those arrays reencoding into UTF-8, validates the lengths, appends a final entry, fills a 3rd array with -1s, & builds a 4th inverse array.
1/?
For each subsequent char it checks the "levels" array of the current state & iterates over them if any. For each level it carefully looks up the appropriate offset in the scores array skipping if the offset's too low, & updating the max score for that subsequent char.
During the reencoding odd scores are potential splitpoints, even scores aren't. So the state machine can turn on & off potential splitpoints.
As for how it traverses this state machine...
3/4!
To traverse to a new state Papyrix validates its current one, iterates over the state's children looking for the appropriate transition, once found decodes (via helpers) a relative pointer & the state at that offset if valid.
States in Papyrix's encoding have a "stride" which determines how many bytes it reads to get those relative pointers.
And to decode states (also in the toplevel Liang Hyphenation routine) it validates the offset, reads & bit-unpacks a header byte, ...
4/4.5!
... if the header byte's most significant bit was set reads a 2byte offset to the levels array & bit-unpacks its length, validates the 2 transitions arrays, & populates this data into a state object.
From a computer science perspective this Liang Hyphenation algorithm has fairly poor performance, but since words are never that long this shouldn't ever be an issue.
In porting this to merephone I'll need to store those state machines on the SD card, otherwise I don't have near enough ROM!
5/5!
Following that Papyrix's implementation of Liang Hyphenation carefully populates (via another helper) the initial state of a state machine propagating any errors, zeroes out a scores array, iterates over the characters, & iterates over (via yet another helper) the scores to reencode valid ones into possible linebreak indices. Taking into account configured minimum prefix/suffix lengths.
For each char it iterates over all subsequent chars following that state machine.
2/4?