Preheat the oven to 350 F. In a medium bowl, stir together the cream cheese, mayonnaise, green onions, pepper and nutmeg. Mix in the Swiss cheese. Spread the mixture in a glass casserole dish and sprinkle the almonds on top. Bake for 7 minutes, stir, then bake 8 minutes more. Serve with crackers.