Metadata-Version: 2.4
Name: CLTS_mtx
Version: 0.1.1
Summary: Fast single-cell and single-nucleus RNA/ATAC-seq normalization model
Author-email: Your Name <songjian.lu@uth.tmc.edu>
Project-URL: Homepage, https://github.com/songjian2022/CLTS_mtx
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: scanpy>=1.8.0
Dynamic: license-file

# CLTS_mtx
By directly processing sparse matrix files (.mtx), optimizing core algorithms, and expanding capabilities to snATAC-seq data, **CLTS_mtx**, an improved model of **CLTS** we developed recently for scRNA-seq normalization (Lu S. et al., Nat. Commun., 2025; PMID:39893178), significantly speeds up the normalization of large-scale snRNA-seq and snATAC-seq datasets without scaling issues.


Widely used normalization techniques, such as CP10K for snRNA-seq and TF-IDF for snATAC-seq, introduce severe scaling artifacts. Specifically, they over-amplify expression and chromatin accessibility profiles in cells with lower total counts while systematically suppressing them in cells with higher counts. **Fundamentally, applying inconsistent adjustment ratios across cells within the same sample and sequencing run (having the same sequencing depth) distorts underlying biological signals.** These artifacts confound comparisons both across distinct cell types and within the same cell type across experimental conditions.

This problem is particularly acute in single-cell multiome (**snRNA-seq + snATAC-seq**) data integration. For instance, within L6 IT neurons from a representative SEA-AD multiome sample, standard normalization exposes a stark discordance between transcriptomic and epigenetic scaling factors. When cells are ordered by their CP10K scaling factors, the corresponding TF-IDF scaling factors for matched chromatin profiles vary stochastically, exhibiting extreme fluctuations, such as a 30-fold spike between adjacent cells followed by a 40-fold drop. Consequently, CP10K and TF-IDF artificially decouple the transcriptomic and epigenetic landscapes of matched individual cells, severely compromising the computational accuracy of multiome data integration.

<!-- <img width="952" height="499" alt="RNA_ATAC_matched" src="https://github.com/user-attachments/assets/2fb3b0f3-ccf7-4cb9-b45c-4db77641aef7" /> -->
<img width="90%" alt="RNA_ATAC_matched" src="https://github.com/user-attachments/assets/2fb3b0f3-ccf7-4cb9-b45c-4db77641aef7" />

**CLTS_mtx** operates on the same core principle as the original **CLTS** framework. Because all cells within a given sample share the same underlying sequencing depth, variations in sequencing depth across samples should cause the mean total counts (or peak counts) of matched cell types to scale proportionally. Indeed, across numerous snRNA-seq and snATAC-seq datasets, we observed a strong linear correlation between cell-type count means across samples. The primary objective of **CLTS_mtx** is therefore to align the cell-type count distributions between a target sample and a baseline reference along the **y = x** line. Crucially, CLTS_mtx applies a uniform adjustment ratio to all cellular profiles within a single sample, thereby eliminating cell-level scaling artifacts.

<!-- <img width="90%" alt="CLTS_idea" src="https://github.com/user-attachments/assets/611d0ce0-95ab-498d-8f47-544920862bd5" /> -->
<img width="1111" height="638" alt="CLTS_idea" src="https://github.com/user-attachments/assets/611d0ce0-95ab-498d-8f47-544920862bd5" />

Comparative evaluation of **CLTS_mtx** and TPM  (the pseudobulk equivalent of CP10K) on the same dataset demonstrated that **CLTS_mtx** superiorly preserves underlying biological signal. Nuclear-encoded mitochondrial genes (NEMGs) encode the essential subunits for the five oxidative phosphorylation (OXPHOS) complexes driving 90–95% of neuronal ATP generation. Given that 75–80% of the brain’s energy budget fuels neuron-specific signaling processes, such as synaptic transmission and membrane potential maintenance, NEMGs are biologically expected to exhibit higher expression in neurons relative to non-neuronal cells. CLTS-normalized profiles faithfully recapitulated this expected biology, maintaining significantly higher mean expression of NEMGs in neurons compared to non-neurons. In contrast, conventional TPM normalization completely obscured this signal, failing to detect the expected neuronal enrichment of NEMGs.

<!-- <img width="2938" height="600" alt="CLTS_vs_TPM" src="https://github.com/user-attachments/assets/cf5ea7c5-1979-4aa9-a515-75404419e5a7" /> -->
<img width="90%" alt="CLTS_vs_TPM" src="https://github.com/user-attachments/assets/cf5ea7c5-1979-4aa9-a515-75404419e5a7" />


## How to install/start ReDeconv
### By PyPI with conda (conda is optional, but recommended)
`conda create -n CLTS_mtx python=3.8`<br>
`conda activate CLTS_mtx`<br>
`pip install CLTS_mtx`

then in your Python script or interactive environment, import the package

`import CLTS_mtx as CLTS`

## Input Data Requirements for CLTS_mtx 
### 1. matrix.mtx
This file contains the raw-count gene expression/chromatin accessibility profiles for the dataset. It follows the standard Matrix Market sparse coordinate format:

* **The Header:** The file begins with a header line (e.g., %%MatrixMarket matrix coordinate integer general). Lines starting with % are treated as comments. Note that while some formats include a second comment line, it is optional and may be absent in your file.
* **The Metadata Line:** The first line that does not start with % is the metadata line. It contains three space-separated integers:
  * **Total Genes (e.g., 219070):** Matches the total number of rows in genes.tsv.
  * **Total Cells (e.g., 130418):** Matches the total number of rows in the barcodes file.
  * **Total Non-zero Entries (e.g., 195243553):** Represents the total number of data rows following the metadata line.
 
<!-- <img width="1061" height="446" alt="mtx_format" src="https://github.com/user-attachments/assets/92fbf282-ee6b-41c0-80c7-c8948d5d08fb" /> -->
<img width="90%" alt="mtx_format" src="https://github.com/user-attachments/assets/92fbf282-ee6b-41c0-80c7-c8948d5d08fb" />

    
* **The Data Triplets:** Each subsequent row contains three values: [**Gene Index**] [**Cell Index**] [**Raw Count**].
  * **1-Based Indexing:** The indices for genes and cells start at **1**, not 0. For example, a triplet **"64 1 2"** indicates that the gene listed on **line 64** of **genes.tsv** has a raw count of 2 in the cell listed on **line 1** of the **barcodes.tsv** (**barcodes_Sample_CellType.tsv**) file.
  * **Integrity Check:** The data is considered invalid if any gene index exceeds the total row count of **genes.tsv** or if any cell index exceeds the row count of the **barcodes.tsv** (**barcodes_Sample_CellType.tsv**) file.

**Note: CLTS-mtx** utilize the total number of non-zero entries (the third number in the metadata line) to track and display data processing progress. Please ensure the input data is formatted correctly, as CLTS_mtx does not perform automated data validation.

### 2. barcodes_Sample_CellType.tsv
This file provides metadata for each cell and must contain at least three tab-separated columns. It is typically generated by appending sample and cell-type/cluster information to the original barcodes.tsv file.

* **Column 1 (Cell ID):** The unique barcode or cell identifier.
* **Column 2 (Sample Name):** The identifier for the sequencing sample.
* **Column 3 (Cell-Type/Cluster):** The assigned cell type identity or Seurat cluster ID.

This file may contain additional columns. **CLTS_mtx** needs you to manually specify the column indices for **Cell ID**, **Sample Name**, and **Cell Type**.

**Note on Indexing:** Unlike the **.mtx file**, column selection in this configuration uses **0-based indexing** (e.g., the first column is index 0, the second is index 1, etc.).

To ensure successful data integration, the **row order** of this file must exactly match the **cell index** in the **matrix.mtx** file. Any mismatch will result in a misalignment of cell identities and expression profiles.

<!-- <img width="657" height="463" alt="barcodes_cell_type" src="https://github.com/user-attachments/assets/4b118c5b-e1a6-4432-816f-83a783770ccf" /> -->
<img width="60%"  alt="barcodes_cell_type" src="https://github.com/user-attachments/assets/4b118c5b-e1a6-4432-816f-83a783770ccf" />



## Tutorials
We provide a demo script **CLTS_mtx_User_RNA_demo.py** and demo data [**Demo_data_4_CLTS_mtx.zip**](https://zenodo.org/records/21708359/files/Demo_data_4_CLTS_mtx.zip?download=1) to demonstrate snRNA-seq normalization using **CLTS_mtx**.

**Setup & Execution:**
1. Create a main working directory (e.g., **CLTS_mtx_test/**) containing two subfolders: **demo_data_RNA/** and **Results_4_normalization_CLTS_mtx/**.

2. Save **CLTS_mtx_User_RNA_demo.p**y in **CLTS_mtx_test**/ and extract (unzip) the contents of **Demo_data_4_CLTS_mtx.zip** into **demo_data_RNA/**.

3. Run **CLTS_mtx_User_RNA_demo.py**.

**Pipeline Performance:**

The execution consists of 4 steps, with Steps 1 and 4 requiring the most compute time (~1.3 min and ~5.15 min, respectively, for the demo data of 61,472 cells across 18 samples on a standard PC). Users can modify this script or call its 4 underlying core functions to process custom datasets.
<br><br><br><br>


------------------- Below is a detailed breakdown of the specific processes executed in each step ------------------- 

### Step-1: Find the size means of all cell types and cell counts of all cell types, which will be used for normalization. The time for this step is usually long when the number of cells is large

**Inputs:------------------**
* **Meta data file:** [`fn_bar`] barcodes_Sample_CellType.tsv in the demo data.
* **Expression or accessibility data file:** [`fn_mtx`] matrix.mtx in the demo data.
* **Lower bound of cell#:** [`L_cell_count_Low_bound`] If a cell type has more cells, then the size mean of the cell type should be more reliable. So, for each sample, if the number of cells in a cell type is less than the lower bound, then the program would not compute the size mean for the cell type and set its value as **"nan"**. The size mean of the cell type in the sample would not be used by the linear regression model to decide the parameters for normalization. (Note: the expression/accessibility profiles of cells of the cell type with cell number less than the threshold in the sample can still be normalized using parameters decided by the size means of other cell types.)
* **Cell ID index:** [`L_Cell_ID_Column_Index`] The 0-based column index in your metadata file containing unique **Cell IDs**.
* **Sample name index:** [`L_Sample_Column_Index`] The 0-based column index in your metadata file specifying the **Sample Name** for each cell.
* **Cell type index:** [`L_Cell_Type_Column_Index`] The 0-based column index in your metadata file specifying the annotated **Cell Type**.

**outputs:------------------**
* **Size mean file:** [`fn_ctyp_mean`] Matrix or table listing the mean library/accessibility size for each (Sample, Cell Type) combination. Cell types in samples falling below **L_cell_count_Low_bound** will display **"nan"** here and are excluded from linear regression modeling. 
* **Cell type cell count file:** [`fn_ctyp_count`] Summary matrix showing the number of captured cells per cell type per sample.
* **Cell transcriptome size file:** [`fn_cell_transcriptome_size`] Single-cell level transcriptome/accessibility sizes for all cells.

### Step-2: Check if transcriptome size means of cell types in different samples have strong linear correlation -- Heatmap
**Inputs:------------------**
* **Size mean file:** [`fn_ctyp_mean`] Size mean summary table generated in Step 1.

**outputs:------------------**
* **Heatmap plot file:** [`fn_heatmap`] Visual figure displaying heatmap of Pearson correlation coefficients of cell type transcriptome/accessibility size means for all pairs of samples
* **Heatmap plot matrix file:** [`fn_heatmap_matrix`] Numeric matrix where both rows and columns represent sample IDs, and cell values contain the pairwise Pearson correlation values.


### Step-3: Check if transcriptome size means of cell types in different samples have strong linear correlation -- Point-plot
**Inputs:------------------**
* **Cell type cell count file:** [`fn_ctyp_count`] Cell count summary table generated in Step 1.
* **Size mean file:** [`fn_ctyp_mean`] Size mean summary table generated in Step 1.
* **Index of the baseline sample:** [`L_baseline`] 0-based index specifying which sample serves as the reference baseline.
* **Lower bound for correlation coefficients:** [`L_Pearson_LB`] Pearson correlation threshold (`r`). Only samples with `r > threshold` relative to the baseline sample will be included in the point-plots.
* **Number of figures per row:** [`L_figureNo_eachRow`] Grid layout parameter specifying how many subplot panels to display per row in the output figure.
* **Draw fit line:** [`L_fit_line_noShift`]  **1**-- to (**0**-- not to)  draw the fit line (passing through the origin) on each point-plot.

**outputs:------------------**
* **Points plot file:** [`fn_point`] The file will be used to save the point-plot of cell type transcriptome size means of the baseline sample and all other samples with Pearson correlation coefficients larger than the lower bound threshold.
* **Extra information file:** [`fn_extra_info`] This file will be used to save some extra information that can help you to choose baseline sample and the lower bound for the correlation coefficients. In the file, for the given lower bound for the correlation coefficients and each sample used as baseline, it gave the information about which samples will be merged and the number of cells in each cell type such that the merged data will have. For example, if we choose sample index **1 – ‘Sample-17’** as baseline and 0.9 as correlation coefficient lower-bound, then we will merge **12** samples together. In the merged file, it will have **1537 ASC1 cells**. If we input **1** as baseline and 0.9 as correlation coefficient lower-bound in the Step-4, then we will obtain a new normalized snRNA-seq/snATAC-seq data submatrix file, which merge cells from 12 samples.

<!-- <img width="1883" height="552" alt="Extra_info" src="https://github.com/user-attachments/assets/d49e07f4-7160-44f6-bd03-b29fe7245d51" /> -->
<img width="90%" alt="Extra_info" src="https://github.com/user-attachments/assets/d49e07f4-7160-44f6-bd03-b29fe7245d51" />

  
### Step-4: Get data submatrix and/or perform the snRNA-seq/snATAC-seq data normalization. This step is time consuming if the data set has a large number of cells
**Inputs:------------------**
* **Meta data file:** [`fn_bar`] barcodes_Sample_CellType.tsv.
* **Expression or accessibility data file:** [fn_mtx] matrix.mtx.
* **Size mean file:** [`fn_ctyp_mean`] Size mean summary table generated in Step 1.
* **Cell transcriptome size file:** [`fn_cell_transcriptome_size`] Single-cell level transcriptome/accessibility size talble generaged in Step 1.
* **Index of the baseline sample:** [`L_baseline`]  0-based index specifying which sample serves as the reference baseline.
* **Lower bound for correlation coefficients:** [`L_Pearson_LB`] Pearson correlation threshold (`r`). Only samples with `r > threshold` relative to the baseline sample will be included in datasubset for normalization.
* **Cell ID index:** [`L_Cell_ID_Column_Index`] The 0-based column index in your metadata file containing unique **Cell IDs**.
* **Sample name index:** [`L_Sample_Column_Index`] The 0-based column index in your metadata file specifying the **Sample Name** for each cell.

**outputs:------------------**
* **A temporary data file:** [`fn_mtx_temp`] Intermediate working file (default: `scRNA_seq_temp_file.tsv`) used to store sub-sampled cell expression/accessibility data when normalizing a cell subset.
* **Normalized snRNA-seq/snARAC-seq data file:** [`fn_mtx_CLTS`] Output matrix containing the final CLTS-normalized single-cell expression or accessibility data.


**Note:** The output normalized matrix shares the standard Matrix Market (.mtx) format with the input **matrix.mtx**, with two key differences: (**a**) Raw integer counts are replaced with normalized floating-point numbers. (**b**) If you perform normalization on a subset of cells, the total number of non-zero entry rows following the metadata header line will be smaller than the total entry count declared in the **3rd number** of the .mtx metadata header line.
