Skip to main content

ISourceTokenizer

  • Namespace: PPDS.Dataverse.Sql.Intellisense
  • Assembly: PPDS.Dataverse
  • Kind: Interface

Summary​

Produces source tokens for syntax highlighting. Language-agnostic interface — implementations provide language-specific tokenization.

Members​

Methods​

Tokenize​

IReadOnlyList<SourceToken> Tokenize(string text)

Tokenizes the given text into source tokens for syntax highlighting. Must not throw — invalid input should produce Error tokens.

  • Param text: The source text to tokenize.

  • Returns: An ordered list of tokens covering the entire input text.