indexers.main

indexers.main

This module provides functionality for creating a vector index from a text file. It defines the VectorStore class, which is used to model and create vector databases from CSV text files using a vectoriser object.

This class interacts with the Vectoriser class from the vectorisers submodule, expecting that any vector model used to generate embeddings used in the VectorStore objects is an instance of one of these classes, most notably that each vectoriser object should have a transform method.

Key Features: - Batch processing of input files to handle large datasets. - Support for CSV file format (additional formats may be added in future updates). - Integration with a custom embedder for generating vector embeddings. - Logging for tracking progress and handling errors during processing.

Dependencies: - polars: For handling data in tabular format and saving it as a Parquet file. - tqdm: For displaying progress bars during batch processing. - numpy: for vector cosine similarity calculations - A custom file iterator (iter_csv) for reading input files in batches.

Usage: This module is intended to be used with the Vectoriers mdodule and the the servers module from ClassifAI, to created scalable, modular, searchable vector databases from your own text data.

Classes

Name	Description
VectorStore	A class to model and create ‘VectorStore’ objects for building and searching vector databases from CSV text files.

VectorStore

indexers.main.VectorStore(
    file_name,
    data_type,
    vectoriser,
    batch_size=8,
    meta_data=None,
    output_dir=None,
    overwrite=False,
    hooks=None,
)

A class to model and create ‘VectorStore’ objects for building and searching vector databases from CSV text files.

Attributes

Name	Type	Description
file_name	str	the original file with the knowledgebase to build the vector store
data_type	str	the data type of the original file (curently only csv supported)
vectoriser	object	A Vectoriser object from the corresponding ClassifAI Pacakge module
batch_size	int	the batch size to pass to the vectoriser when embedding
meta_data	dict	key-value pairs of metadata to extract from the input file and their correpsonding types
output_dir	str	the path to the output directory where the VectorStore will be saved
vectors	np.array	a numpy array of vectors for the vector DB
vector_shape	int	the dimension of the vectors
num_vectors	int	how many vectors are in the vector store
vectoriser_class	str	the type of vectoriser used to create embeddings
hooks	dict	A dictionary of user-defined hooks for preprocessing and postprocessing.

Methods

Name	Description
embed	Converts text into vector embeddings using the vectoriser and returns a VectorStoreEmbedOutput dataframe with columns ‘id’, ‘text’, and ‘embedding’.
from_filespace	Creates a VectorStore instance from stored metadata and Parquet files.
reverse_search	Reverse searches the vector store using a VectorStoreReverseSearchInput object
search	Searches the vector store using queries from a VectorStoreSearchInput object and returns

embed

indexers.main.VectorStore.embed(query)

Converts text into vector embeddings using the vectoriser and returns a VectorStoreEmbedOutput dataframe with columns ‘id’, ‘text’, and ‘embedding’.

Parameters

Name	Type	Description	Default
query	VectorStoreEmbedInput	The VectorStoreEmbedInput object containing the strings to be embedded and their ids.	required

Returns

Name	Type	Description
VectorStoreEmbedOutput	VectorStoreEmbedOutput	The output object containing the embeddings along with their corresponding ids and texts.

Raises

Name	Type	Description
	DataValidationError	Raised if invalid arguments are passed.
	HookError	Raised if user-defined hooks fail.
	ClassifaiError	Raised if embedding operation fails.

from_filespace

indexers.main.VectorStore.from_filespace(folder_path, vectoriser, hooks=None)

Creates a VectorStore instance from stored metadata and Parquet files. This method reads the metadata and vectors from the specified folder, validates the contents, and initializes a VectorStore object with the loaded data. It checks that the metadata contains the required keys, that the Parquet file exists and is not empty, and that the vectoriser class matches the one used to create the vectors. If any checks fail, it raises a ValueError with an appropriate message. This method is useful for loading previously created vector stores without needing to reprocess the original text data.

Parameters

Name	Type	Description	Default
folder_path	str	The folder path containing the metadata and Parquet files.	required
vectoriser	object	The vectoriser object used to transform text into vector embeddings.	required
hooks	dict	[optional] A dictionary of user-defined hooks for preprocessing and postprocessing. Defaults to None.	None

Returns

Name	Type	Description
	VectorStore	An instance of the VectorStore class.

Raises

Name	Type	Description
	DataValidationError	If input arguments are invalid or if there are issues with the metadata or Parquet files.
	ConfigurationError	If there are configuration issues, such as vectoriser mismatches.
	IndexBuildError	If there are failures during loading or parsing the files.

reverse_search

indexers.main.VectorStore.reverse_search(
    query,
    max_n_results=100,
    partial_match=False,
)

Reverse searches the vector store using a VectorStoreReverseSearchInput object and returns matched results in VectorStoreReverseSearchOutput object. If using partial matching, matches if document label starts with query label.

Parameters

Name	Type	Description	Default
query	VectorStoreReverseSearchInput	A VectorStoreReverseSearchInput object containing the text query or list of queries to search for with ids.	required
max_n_results	int	[optional] Number of top results to return for each query, set to -1 to return all results. Default 100.	100
partial_match	bool	[optional] Set the search behaviour to use join_where to match query checks that document id startsWith query. Default False	False

Returns

Name	Type	Description
result_df	VectorStoreReverseSearchOutput	A VectorStoreReverseSearchOutput object containing reverse search results with columns for query ID, query text, document ID, document text and any associated metadata columns.

Raises

Name	Type	Description
	DataValidationError	Raised if invalid arguments are passed.
	HookError	Raised if user-defined hooks fail.
	ClassifaiError	Raised if reverse search operation fails.

search

indexers.main.VectorStore.search(query, n_results=10, batch_size=8)

Searches the vector store using queries from a VectorStoreSearchInput object and returns ranked results in VectorStoreSearchOutput object. In batches, converts users text queries into vector embeddings, computes cosine similarity with stored document vectors, and retrieves the top results.

Parameters

Name	Type	Description	Default
query	VectorStoreSearchInput	A VectoreStoreSearchInput object containing the text query or list of queries to search for with ids.	required
n_results	int	[optional] Number of top results to return for each query. Default 10.	10
batch_size	int	[optional] The batch size for processing queries. Default 8.	8

Returns

Name	Type	Description
result_df	VectorStoreSearchOutput	A VectorStoreSearchOutput object containing search results with columns for query ID, query text, document ID, document text, rank, score, and any associated metadata columns.

Raises

Name	Type	Description
	DataValidationError	Raised if invalid arguments are passed.
	ConfigurationError	Raised if the vector store is not initialized.
	HookError	Raised if user-defined hooks fail.
	VectorisationError	Raised if embedding queries fails.