🏆 GWO Benchmark: The Architect's Arena 🏆

An official leaderboard for efficient neural network operations.

Rank	Model	Author	Score ▾Overall efficiency score. Higher is better!	Test AccAccuracy on CIFAR-10	Ω_proxyOperational Complexity. Lower is better.	C_DDescriptive Complexity	C_P (M)Parametric Complexity (in Millions)	Latency (ms)Inference time per batch	Tier
1	StandardConv	Official Baseline	990.14	69.31%	6.00	6	0.000	0.50	B
2	DeformableConv	Official Baseline	771.40	69.45%	8.00	8	0.003	1.63	C
3	DepthwiseConv	Official Baseline	681.67	61.35%	8.00	8	0.000	0.53	C

---

Official Submission Guidelines

To ensure that all submissions to "The Architect's Arena" are fair, consistent, and reproducible, all participants must adhere to the following official benchmarking rules.

1. Core Environment

Your submission will be evaluated based on its performance under this specific, standardized environment.

• Dataset: CIFAR-10
  • All models will be trained on the official CIFAR-10 training set and evaluated on the official test set. Standard data transformations (normalization, random horizontal flip, random crop) provided by the default CIFAR10Dataset class in the benchmark package are applied.
• Evaluator: gwo_benchmark.Evaluator
  • You must use the default Evaluator provided within the package. Custom evaluators are not permitted for official leaderboard submissions to maintain a level playing field in the training and evaluation loop.
• Training Configuration:
  • The train_config dictionary passed to the Evaluator must exactly match the following parameters:

    train_config = {
        "epochs": 30,
        "lr": 0.001,
        "momentum": 0.9,
        "batch_size": 64
    }
    

  • The optimizer used by the default Evaluator is Stochastic Gradient Descent (SGD). No learning rate schedulers, weight decay modifications, or alternative optimizers (e.g., Adam) are allowed in the official track.

2. Model Implementation (GWOModule)

Your model must adhere to the GWOModule interface to allow for automated complexity calculation and evaluation.

• Inheritance: Your model must inherit from gwo_benchmark.base.GWOModule.
• Descriptive Complexity (C_D): You must define the C_D class variable as a static integer. This value should be calculated honestly, based on the Official Primitive Dictionary provided in the README. Your reasoning for the chosen C_D value should be included as comments in your code or in your submission description.
• Parametric Complexity (C_P): You must correctly implement the get_parametric_complexity_modules() method. This method should return a list of all nn.Modules whose parameters contribute to the dynamic generation of the operation (e.g., offset prediction networks), not the primary weights of the operation itself (like the convolutional kernel).

3. Submission Process

1. Prepare Your Code: Ensure your GWOModule implementation is clean, commented, and ready for review.
2. Run Locally (Optional but Recommended): Run the benchmark on your local machine to verify that your model trains correctly and to get a preliminary idea of its performance.
3. Submit via Google Form: Link to your Google Form
   • You will be asked to provide:
     • Your Name / Pseudonym.
     • A link to a GitHub Gist or Repository containing your GWOModule code.
     • The calculated Descriptive Complexity (C_D) and a brief justification for it.
4. Review Process:
   • All submissions will be manually reviewed by the maintainers to ensure they comply with these guidelines. We check for correctness, fairness, and honesty in the C_D calculation.
   • Approved submissions will be run on our official benchmark hardware to ensure consistent Latency measurements.
   • Once verified, your result will be added to the live leaderboard. Please allow for some time for this manual process.

By standardizing these conditions, we can confidently compare the architectural efficiency of diverse neural network operations. We look forward to seeing your innovative designs in the Arena.

---