Split text

Split text into an array of strings based on separator or number of tokens

This transformation enables you to convert text into an array of chunks based on either a separator or the number of tokens.

If you use the separator method, you can provide a separator which will be used to split the text. For example, if you have text like "hello,my,name,is" and a separator "," it will turn it into ["hello", "my", "name", "is"].

If you use the tokens method, you can specify the maximum number of tokens a chunk can have. LLMs process text using tokens, which are common sequences of characters found in text. Using this method can assist with making sure that a single prompt doesn't exceed the maximum number of tokens.

Screenshot of the split text transformation in the Relevance AI Notebook

Screenshot of the split text transformation in the Notebook

Common errors

  • No separator or number of tokens specified

Schema

Inputs

NameDescriptionTypeRequired
TextThe text to be splitStringโœ…
Splitting methodThe method by which to split the text into chunksEnumโœ…
SeparatorThe pattern that will be used to split by when found in the textString๏ผŸif method is separator
Number of TokensThe maximum number of tokens to allow in each chunk before splittingNumber๏ผŸif method is tokens

Outputs

NameDescriptionType
ChunksThe chunks of text that the original text was split intoArray