latest--SentenceTransformersTokenTextSplitter

shared
Type object
Schema URL https://catalog.lintel.tools/schemas/schemastore/eidolon-resource/_shared/latest--SentenceTransformersTokenTextSplitter.json
Parent schema eidolon-resource
Type: object

Properties

implementation const: "SentenceTransformersTokenTextSplitter" required
Constant: "SentenceTransformersTokenTextSplitter"
chunk_size integer

Maximum size of chunks to return

Default: 4000
chunk_overlap integer
Default: 50
keep_separator boolean

Whether to keep the separator in the chunks

Default: false
strip_whitespace boolean

If True, strips whitespace from the start and end of every document

Default: true
model string

Model name

Default: "sentence-transformers/all-mpnet-base-v2"
tokens_per_chunk integer | null

Number of tokens per chunk

Default: null