Currently, critical peak load caused by residential customers has attracted utility companies and policymakers to pay more attention to residential demand response (RDR) programs. In typical RDR programs, residential customers react to the price or incentive-based signals, but the actions can fall behind flexible market situations. For those residential customers equipped with smart meters, they may contribute more DR loads if they can participate in DR events in a proactive way. In this paper, we propose a comprehensive market framework in which residential customers can provide proactive RDR actions in a day-Ahead market (DAM). We model and evaluate the interactions between generation companies (GenCos), retailers, residential customers, and the independent system operator (ISO) via an agent-based modeling and simulation (ABMS) approach. The simulation framework contains two main procedures-The bottom-up modeling procedure and the reinforcement learning (RL) procedure. The bottom-up modeling procedure models the residential load profiles separately by household types to capture the RDR potential differences in advance so that residential customers may rationally provide automatic DR actions. Retailers and GenCos optimize their bidding strategies via the RL procedure. The modified optimization approach in this procedure can prevent the training results from falling into local optimum solutions. The ISO clears the DAM to maximize social welfare via Karush-Kuhn-Tucker (KKT) conditions. Based on realistic residential data in China, the proposed models and methods are verified and compared in a large multi-scenario test case with 30,000 residential households. Results show that proactive RDR programs and interactions between market entities may yield significant benefits for both the supply and demand sides. The models and methods in this paper may be used by utility companies, electricity retailers, market operators, and policymakers to evaluate the consequences of a proactive RDR and the interactions among multi-entities.