Turn text into tokens using the Token Marcher Algorithm. Capable of creating over 8,200 tokens from 1,000 lines of C# code in under 2 ms on a 3.2 GHz quad core CPU. Doesn’t use regex, supports custom patterns, and small enough to fit on a business card.
No comments:
Post a Comment