Custom rules in email marketing refer to a set of predefined guidelines and criteria that marketers use to tailor their email campaigns. These rules can be applied to segment audiences, trigger specific actions, or personalize content based on the recipient's behavior and preferences.