Split text
Split text into an array of strings based on separator or number of tokens
This transformation enables you to convert text into an array of chunks based on either a separator or the number of tokens.
If you use the separator method, you can provide a separator which will be used to split the text. For example, if you have text like "hello,my,name,is" and a separator "," it will turn it into ["hello", "my", "name", "is"].
If you use the tokens method, you can specify the maximum number of tokens a chunk can have. LLMs process text using tokens, which are common sequences of characters found in text. Using this method can assist with making sure that a single prompt doesn't exceed the maximum number of tokens.
Screenshot of the split text transformation in the Notebook
Common errors
- No separator or number of tokens specified
Schema
Inputs
| Name | Description | Type | Required |
|---|---|---|---|
| Text | The text to be split | String | β |
| Splitting method | The method by which to split the text into chunks | Enum | β |
| Separator | The pattern that will be used to split by when found in the text | String | οΌif method is separator |
| Number of Tokens | The maximum number of tokens to allow in each chunk before splitting | Number | οΌif method is tokens |
Outputs
| Name | Description | Type |
|---|---|---|
| Chunks | The chunks of text that the original text was split into | Array |
Updated 4 months ago