INNER CODE UNIT · Swift

tokenize

ZachNagengast/similarity-search-kit · Sources/SimilaritySearchKit/Core/Embeddings/EmbeddingProtocols.swift:70

    func tokenize(text: String) -> [String]
    func detokenize(tokens: [String]) -> String
}

View source record →