evaluation package

evaluation package#

Submodules#

evaluation.MMD module#

class evaluation.MMD.MMD(real_cells: ndarray)[source]#

Bases: object

Maximum Mean Discrepancy (MMD) class for computing distribution similarity between real and generated samples using Gaussian kernels.

__init__(real_cells: ndarray)[source]#

Initialize the MMD class with scale and weight parameters based on the median nearest neighbor distance among real cells.

Parameters:

real_cells (np.ndarray) – A NumPy array representing real cell data (cells x features).

squaredDistance(X: ndarray | Tensor, Y: ndarray | Tensor) → Tensor[source]#

Compute pairwise squared Euclidean distances between rows of X and Y.

Parameters:

  • X (np.ndarray or tf.Tensor) – Input array of shape (n, d).

  • Y (np.ndarray or tf.Tensor) – Input array of shape (m, d).

Returns:

A tensor of shape (n, m) representing squared distances.

Return type:

tf.Tensor

gaussian_kernel(a: ndarray | Tensor, b: ndarray | Tensor) → Tensor[source]#

Compute the multi-scale Gaussian kernel between two datasets.

Parameters:

  • a (np.ndarray or tf.Tensor) – Input array of shape (n, d).

  • b (np.ndarray or tf.Tensor) – Input array of shape (m, d).

Returns:

A tensor of shape (n, m) representing the Gaussian kernel matrix.

Return type:

tf.Tensor

compute(a: ndarray | Tensor, b: ndarray | Tensor) → Tensor[source]#

Compute the Maximum Mean Discrepancy (MMD) between two samples.

Parameters:

  • a (np.ndarray or tf.Tensor) – First sample of shape (n, d).

  • b (np.ndarray or tf.Tensor) – Second sample of shape (m, d).

Returns:

The MMD score between the two distributions.

Return type:

tf.Tensor

evaluation.data_quality module#

evaluation.data_quality.read_datasets(cfg: ConfigParser) → Tuple[ndarray, ndarray][source]#

Load real and simulated (fake) gene expression datasets.

Parameters:

cfg (ConfigParser) – Parser for config file containing program params.

Returns:

A tuple containing: - real_cells.X : NumPy array of real gene expression data - fake_cells.X : NumPy array of simulated gene expression data, truncated to match real data row count

Return type:

Tuple[numpy.ndarray, numpy.ndarray]

evaluation.data_quality.plot_tSNE(real: ndarray, fake: ndarray, output_dir: str) → Tuple[ndarray, ndarray][source]#

Perform t-SNE embedding on real and fake cell data and save a scatter plot.

Parameters:

  • real (np.ndarray) – A NumPy array of real cell data with shape (n_real_cells, n_features).

  • fake (np.ndarray) – A NumPy array of fake/generated cell data with shape (n_fake_cells, n_features).

  • output_dir (str) – Path to the directory where the t-SNE plot image will be saved. If empty, the plot is not saved.

Returns:

A tuple containing the 2D t-SNE embeddings of the real and fake data, in the form (real_embedding, fake_embedding).

Return type:

Tuple[np.ndarray, np.ndarray]

evaluation.data_quality.compute_distances(real_cells: ndarray, fake_cells: ndarray, axis: int = 0) → Tuple[float, float][source]#

Compute Euclidean and Cosine distances between the mean expression profiles of real and fake cells.

Parameters:

  • real_cells (np.ndarray) – A NumPy array representing real cell data (cells x features).

  • fake_cells (np.ndarray) – A NumPy array representing fake cell data (cells × features).

  • axis (int, optional) – Axis along which to compute the mean expression (default is 0, meaning across cells), by default 0

Returns:

A tuple containing: - Euclidean distance between the mean expression profiles. - Cosine distance between the mean expression profiles.

Return type:

Tuple[float, float]

evaluation.data_quality.compute_RF_AUROC(real_cells: ndarray, fake_cells: ndarray, output_dir: str, n_components: int = 50) → float[source]#

Compute the AUROC and plot the ROC curve of a Random Forest classifier distinguishing real from fake cells.

Parameters:

  • real_cells (np.ndarray) – A NumPy array representing real cell data (cells x features).

  • fake_cells (np.ndarray) – A NumPy array representing fake/generated cell data (cells x features).

  • output_dir (str) – Path to the directory where the ROC curve plot will be saved. The plot is saved as ‘RF.png’.

  • n_components (int, optional) – Number of principal components to retain during PCA, by default 50

Returns:

The area under the ROC curve (AUROC) for the classifier.

Return type:

float

evaluation.data_quality.evaluate(cfg: ConfigParser) → None[source]#

Assess the data quality of the simulated dataset.

  • T-SNE plots of real vs simulated cells (jointly embedded and plotted).

  • Euclidean and Cosine distances between the centroids of real cells and simulated cells.

    The centroid cell was obtained by calculating the mean along the gene axis (across all simulated or real cells).

  • Area under the receiver operating characteristic curve of a random forest in distinguishing real cells from simulated ones.

    We perform dimensionality reduction using PCA to extract the top 50 PCs of each cell as the input features to the RF model. The RF model is composed of 1000 trees and the Gini impurity was used to measure the quality of a split.

  • Maximum mean discrepancy (MMD) to estimate the proximity of high-dimensional distributions of real and simulated cells without creating centroids.

  • Mean integration local inverse Simpson’s Index (miLISI).

    iLISI captures the effective number of datatypes (real or simulated) to which datapoints of its local neighborhood belong.

As a “control” (and to enable calibration of these scores), we also calculated these metrics using two halves of the reference test set.

Parameters:

cfg (ConfigParser) – Parser for config file containing program params.

evaluation.grn_inference module#

evaluation.grn_inference.get_imposed_grn(cfg: ConfigParser) → Tuple[DataFrame, Set[str], Set[str], List[str], List[str], int][source]#

Load and process the imposed (ground-truth) GRN from a causal graph file.

This function reads the GRouNdGAN GRN and maps numeric gene indices to gene names and constructs a structured representation of the GRN.

Parameters:

cfg (ConfigParser) – Configuration object providing the necessary file paths: - “Data” -> “train”: Path to real cells .h5ad file for gene name mapping. - “Data” -> “causal graph”: Path to the pickle file containing the ground truth GRN. - “GRN Benchmarking” -> “ground truth save path”: Optional output path for saving the ground truth GRN in csv format.

Returns:

  • imposed_grn (pd.DataFrame) – DataFrame with a single column “res” containing string-formatted edges (“TF -> target”).

  • imposed_TFs (set) – Set of transcription factors (TFs) involved in the imposed GRN.

  • imposed_targets (set) – Set of target genes in the imposed GRN.

  • imposed_edges (list of str) – List of edges as strings in the format “TF -> target”.

  • TFs (list of str) – List of unique transcription factors (from imposed_TFs), useful for downstream filtering.

  • possible_edges_no (int) – Total number of possible TF-target combinations based on imposed GRN (|TFs| * |targets|).

evaluation.grn_inference.get_fake_grn(cfg: ConfigParser, imposed_TFs: Iterable[str], imposed_targets: Iterable[str], TFs: Iterable[str], possible_edges_no: int) → DataFrame[source]#

Construct a benchmark gene regulatory network (GRN) DataFrame using inferred and imposed edges.

This function loads an inferred GRN from a file, filters it based on imposed TFs and targets, and augments it with all possible unreported edges (assigned zero importance). This ensures that all theoretically possible edges are included for evaluation purposes (e.g., computing recall).

Parameters:

  • cfg (ConfigParser) – Configuration object providing the path to the inferred GRN file via the section “GRN Benchmarking” -> “grn to benchmark”.

  • imposed_TFs (Iterable[str]) – List of TFs that are considered “imposed” in the simulation and should be included as sources in the GRN.

  • imposed_targets (Iterable[str]) – List of target genes that are expected to be regulated in the benchmark.

  • TFs (Iterable[str]) – List of all possible TFs used to filter the GRN (must be valid source nodes).

  • possible_edges_no (int) – Total number of possible edges in the benchmark GRN (used for validation/debugging).

Returns:

DataFrame representing the benchmark GRN. Columns include: - “TF”: Source transcription factor - “target”: Target gene - “importance”: Importance score (0 for non-inferred edges) - “res”: String representation of the edge (“TF -> target”)

Return type:

pd.DataFrame

evaluation.grn_inference.compute_precision_at_k(fake_grn: DataFrame, imposed_grn: DataFrame, TFs: List[str] | Set[str], numTFs_pergene: int, save_path: str) → None[source]#

Compute precision at k for inferred GRN against the imposed ground truth. It plots the precision and baseline curves and optionally saves the figure.

Parameters:

  • fake_grn (pd.DataFrame) – Inferred GRN with columns “TF”, “target”, and “res” (formatted as “TF -> target”).

  • imposed_grn (pd.DataFrame) – Ground truth GRN with a column “res” representing edges in “TF -> target” format.

  • TFs (Union[List[str], Set[str]]) – List or set of valid transcription factors. Used to exclude predictions where a TF is treated as a target.

  • numTFs_pergene (int) – Maximum number of top TFs to evaluate per gene.

  • save_path (str) – Directory path where the precision-at-k plot should be saved. If empty, the plot is not saved.

Return type:

None

evaluation.grn_inference.compute_PR(fake_grn: DataFrame, imposed_grn: DataFrame, imposed_edges: List[str], imposed_targets: Set[str], imposed_TFs: Set[str], save_path: str | None = '') → None[source]#

Compute and plot the Precision-Recall (PR) curve for a predicted gene regulatory network (GRN) against a ground-truth GRN.

Parameters:

  • fake_grn (pd.DataFrame) – Inferred GRN DataFrame with a column ‘res’ representing edges in the format “TF -> target”.

  • imposed_grn (pd.DataFrame) – Ground-truth GRN DataFrame with a column ‘res’ representing edges in the same format.

  • imposed_edges (List[str]) – List of all ground-truth edges as strings (“TF -> target”).

  • imposed_targets (Set[str]) – Set of ground-truth target genes.

  • imposed_TFs (Set[str]) – Set of ground-truth transcription factors.

  • save_path (Optional[str]) – Directory path to save the precision-recall plot as “PR_curve.png”. If None or empty, the plot is not saved.

Return type:

None

evaluation.grn_inference.evaluate(cfg: ConfigParser) → None[source]#

Evaluate the reconstructed GRN against the ground truth GRN

Parameters:

cfg (ConfigParser) – Parser for config file containing program params.

Return type:

None

evaluation.lisi module#

evaluation.lisi.compute_lisi(X: array, metadata: DataFrame, label_colnames: Iterable[str], perplexity: float = 30) → ndarray[source]#

Compute the Local Inverse Simpson Index (LISI) for each label column in the metadata.

LISI measures the diversity of labels in the local neighborhood of each sample, based on distances in the feature space X. It is commonly used to evaluate dataset mixing (e.g., integration of multiple batches or experimental conditions).

A LISI score close to:

  • 1 indicates homogeneity (neighbors mostly from one label category),

  • N (number of categories) indicates high diversity (neighbors from all categories).

Parameters:

  • X (np.ndarray) – Data matrix of shape (n_samples, n_features), representing the embedding space (e.g., PCA, UMAP).

  • metadata (pd.DataFrame) – DataFrame containing categorical metadata for each sample (e.g., batch or cell type).

  • label_colnames (Iterable[str]) – List of column names in metadata for which to compute LISI.

  • perplexity (float, optional) – Perplexity parameter influencing the neighborhood size, by default 30.

Returns:

A matrix of shape (n_samples, len(label_colnames)) with LISI scores per label.

Return type:

np.ndarray

evaluation.lisi.compute_simpson(distances: ndarray, indices: ndarray, labels: Categorical, n_categories: int, perplexity: float, tol: float = 1e-05) → ndarray[source]#

Compute the Simpson’s diversity index for each sample using a locally scaled probability distribution.

This function computes the effective number of categories in each sample’s neighborhood, based on the similarity-weighted label distribution. It is internally used by compute_lisi.

Parameters:

  • distances (np.ndarray) – Array of shape (n_neighbors, n_samples) with distances to each sample’s neighbors.

  • indices (np.ndarray) – Array of shape (n_neighbors, n_samples) with indices of nearest neighbors for each sample.

  • labels (pd.Categorical) – Categorical labels for all samples, aligned with the rows of X.

  • n_categories (int) – Number of unique categories in the label.

  • perplexity (float) – Perplexity value, used to set the target entropy for neighborhood probability distribution.

  • tol (float, optional) – Tolerance for the entropy convergence, by default 1e-5.

Returns:

Array of shape (n_samples,) with the Simpson’s index per sample. Values closer to 1 indicate less diversity, higher values indicate more.

Return type:

np.ndarray

Module contents#

Contents