INNER CODE UNIT · Python

convert_linear_to_moe

Leeroo-AI/mergoo · mergoo/compose_layers.py:7

def convert_linear_to_moe(
    name: str,
    config: dict,
    layer_idx: int,
    in_features: int,
    out_features: int,
    bias: bool = True,
):
    """Converts nn.Linear to MoeLayer
    Args:
        name (str): Layer Name
        config (dict): Composer config
        layer_idx (int): Transformer block id.
        in_features (int): Input features of Default nn.Linear layer.
        out_features (int): Output features of Default nn.Linear layer.
        bias (bool, optional): Defaults to True.
    """
    if (layer_idx in config.router_layers_index) and (name in config.router_layers):

View source record →