The "out of memory" error is caused because the game is too big to be loaded into memory. There have been reports of large fangames that worked in windows 7/8 not working in windows 10, sadly there's pretty much nothing that can be done about it aside from editing the game to remove some assets.
