This is a well-structured VM with instructions analogous to x86 architecture, but with a larger set of registers. There are about 30 opcodes. The program executed by the VM is just under 400 instructions long and is some kind of monstrous crypto algorithm of the author's design (Genaytyk calls it a "high cipher" in his crackmes.de description).
UPDATE: artif has solved this crackme too! see his keygen written in pure ASM in the crackme solutions download link