Random Factorized Synthesizer

• Random Factorized Synthesizer is a neural module that applies successive low-rank projections to condense, process, and re-expand features, enabling efficient attention mechanisms.
• It is exemplified by DC-AC modules that use dual condensation paths and sigmoid gating to achieve competitive results in skin lesion classification and ImageNet benchmarks.
• The architecture strategically balances representational capacity with TinyML constraints by leveraging structured factorization, reduced computational overhead, and optimized design choices.

A Random Factorized Synthesizer is not explicitly defined or referenced as a distinct module in the surveyed research; however, the term plausibly refers to a class of architectural blocks within neural attention models that employ random (or structured, possibly low-rank) factorization techniques to accelerate self-attention. In the context of efficient self-attention and neural condensation for TinyML and edge computing, the closest concrete instantiations are the Double-Condensing Attention Condenser (DC-AC) modules, which utilize factorized projection paths and highly condensed intermediate representations to achieve expressive yet compact attention mechanisms. These approaches are implemented and analyzed in recent work on efficient neural backbones such as AttendNeXt and skin cancer screening networks (Tai et al., 2023, Wong et al., 2022).

1. Definition and General Concept

"Random Factorized Synthesizer" as an Editor's term can be applied to network modules that replace dense projection or attention steps with one or more factorization layers—either random or structured—aiming to reduce computational and storage burden by decomposing high-dimensional feature maps into lower-dimensional embeddings before applying attention or synthesis. In these methods, input features are first projected into a lower-dimensional latent space via learned or random matrices, which can be seen as a structured or random factorization. Attention or gating then operates in this compact space, followed by re-expansion to the original feature dimension.

The salient characteristics are:

• Application of two or more successive low-rank (often 1×1 convolutional) projections to reduce channel dimension.
• Attentional mechanisms implemented on these reduced representations.
• Re-expansion to original dimension, synthesizing selective, context-dependent outputs.

This factorized condensation and synthesis forms the core of the DC-AC approach (Wong et al., 2022). No direct evidence is found for genuinely random (i.e., non-learned) projections in the reviewed modules—the projections are learned during training.

2. Architectural Realizations: Double-Condensing Attention Condensers

The Double-Condensing Attention Condenser (DC-AC) module exemplifies the factorized synthesizer template (Tai et al., 2023, Wong et al., 2022). The module replaces monolithic dense attention with a pipeline of low-rank projections and pointwise operations. A representative DC-AC block consists of two parallel condenser paths, each comprising:

• Condense: 1×1 conv projection, channel reduction $C \to C_1 = C / r$
• Embed: 3×3 conv operating in the condensed space
• Expand: 1×1 conv, channel increase $C_1 \to C$

The outputs of each condenser are summed and gated via a sigmoid, generating a highly selective spatial–channel attention mask:

$$Y = Conv_{out}\left( ReLU\left( F \odot \sigma (\phi_1(F) + \phi_2(F) ) \right) \right)$$

where $F$ is the input feature tensor and $\phi_i$ the condense-embed-expand paths.

These structures can be viewed as a deterministic variant of a "factorized synthesizer", with all factorization layers being learned and optimized via backpropagation (Tai et al., 2023).

3. Mathematical Formulation and Data Flow

The DC-AC module data flow is:

1. Input $F \in \mathbb{R}^{B \times C \times H \times W}$
2. For each condenser $i$, perform:
   • $U_i = \text{Conv1x1}(F)$ (channel reduction)
   • $V_i = \text{Conv3x3}(\text{ReLU}(U_i))$ (context integration)
   • $E_i = \text{Conv1x1}(\text{ReLU}(V_i))$ (re-expansion)
3. Fuse outputs: $S = E_1 + E_2$
4. Gate: $M = \sigma(S)$ (sigmoid activation)
5. Attention mask: $F_{att} = F \odot M$
6. Output: $Y = \text{Conv3x3}(\text{ReLU}(F_{att}))$

The factorization occurs via dimensionality reduction in both parallel condensers before expanding—thus, two levels of synthesis are imposed before forming the output.

4. Practical Applications and Empirical Results

The DC-AC, serving as a representative factorized synthesizer, was embedded in TinyML-optimized deep neural networks for both medical image classification and general computer vision tasks. In the ISIC 2020 skin lesion classification challenge, the DC-AC backbone achieved:

• 1.6 million parameters
• 0.325 GFLOPs per inference
• Public AUROC: 0.9045, Private AUROC: 0.8865 (Tai et al., 2023) This outperformed larger MobileViT-S models and Cancer-Net SCa variants, with significantly lower computational budgets.

For ImageNet classification, AttendNeXt with DC-AC modules achieved 75.8% top-1 accuracy at a >10× speedup over FB-Net C and 1.37× smaller size than MobileNetV3-L (Wong et al., 2022).

5. Design Strategies and Optimization Constraints

Key architectural choices for these synthesizers include:

• Selection of condensation ratios (reduction factors $r$, latent dims $d_1, d_2$) for minimal parameterization while preserving representational capacity.
• Use of multiple columns (multi-branch) architecture to enable receptive field diversity.
• Avoidance of strided pointwise convolutions and reliance on anti-aliased downsampling for information preservation (Wong et al., 2022).
• Standard normalization (BatchNorm) and gating (sigmoid) after projection layers.

Design exploration leveraged machine-driven synthesis to ensure models met or exceeded specific accuracy and size constraints (e.g., ≥75.8% top-1 ImageNet), and produced backbones with sub-10MB footprint and <300 MFLOPs per 224×224 image.

6. Comparison to Other Attention Factorization Strategies

In contrast to DC-AC, $A^2$-Nets implement "double attention" blocks, which first aggregate global features via second-order pooling and then adaptively redistribute them to all locations. The $A^2$ block employs a similar philosophy—factorizing global context aggregation and redistribution—but with a distinct two-step attention process (Chen et al., 2018). Double attention blocks reduce complexity from $O(N^2)$ to $O(Nm)$ (for $N$ positions, $m$ condensed descriptors), providing further evidence of the impact of attention factorization and synthesis on network efficiency.

7. Relevance for TinyML and Edge Deployment

The double-condensing architecture enables:

• Memory footprint: ≤6.4 MB (32-bit), ≤1.6 MB post quantization (8-bit)
• Efficient 8-bit quantization: <1% accuracy degradation
• FLOPs per inference and parameter counts compatible with sub-100 MHz MCUs

Magnitude-based pruning of the 1×1 convolutions in DC-AC blocks further reduces size without loss of AUROC, supporting pervasive deployment scenarios (Tai et al., 2023).

---

In summary, the Random Factorized Synthesizer epitomizes a class of efficient attention modules built upon serial low-rank projections and synthesis of attended features, as realized in the double-condensing attention condensers integral to modern TinyML and efficient vision architectures (Tai et al., 2023, Wong et al., 2022, Chen et al., 2018).