Natural Grad

Natural Grad#

HyperCausal Natural Gradient (State-Space)#

Precondition estimated gradients using an empirical Fisher-like metric derived from the covariance of state branches. Operates in state geometry and maps back to parameter space via random projection (simple, effective proxy).

Interface:

  • initialize(params) -> state

  • step_params(model, params, context) -> (new_params, state)

Context:

  • context[“info”][“branches”]: (K x D) states from model.forward(…)

  • gradient estimated via context[‘grads’] or a grad_estimator

class qmlhc.optim.numpy_optim.natural_grad.HCNaturalGrad(lr=0.005, fisher_damp=0.001, cg_iters=8, clip=None, grad_estimator=None, seed=12345)[source]#

Bases: object

Natural-gradient preconditioning using state covariance.

initialize(params)[source]#
Return type:

Dict[str, Any]

step_params(model, params, context)[source]#
Return type:

Tuple[Dict[str, Any], Dict[str, Any]]

Contents