Filed Pursuant to Rule 424(b)(3)
                                                     Registration No. 333-112598



PROSPECTUS SUPPLEMENT NO. 1
TO PROSPECTUS DATED APRIL 21, 2004




                         CASUAL MALE RETAIL GROUP, INC.

                        $100,000,000 Principal Amount of
                   5% Convertible Subordinated Notes Due 2024
                                       and
          Shares of Common Stock Issuable Upon Conversion of the Notes
                                -----------------

      This prospectus supplement relates to the resale by the selling
securityholders (including their transferees, pledgees or donees or their
successors) of up to $100 million aggregate principal amount of 5% Convertible
Subordinated Notes due 2024 issued by Casual Male Retail Group, Inc. and the
shares of common stock issuable upon conversion of the notes.

      This prospectus supplement should be read in conjunction with the
prospectus dated April 21, 2004, which is to be delivered with this prospectus
supplement, and this prospectus supplement is qualified by reference to the
prospectus, except to the extent that the information in this prospectus
supplement supercedes the information contained in the prospectus. This
prospectus supplement is not complete without, and may not be delivered or
utilized except in connection with, the prospectus, including any amendments or
supplements to it.

      Investing in the notes or the common stock offered by the prospectus
involves risks that are described in the "Risk Factors" section beginning on
page 8 of the prospectus.

                                -----------------

      Neither the Securities and Exchange Commission, any state securities
commission nor any other regulatory body has approved or disapproved of these
securities or determined if this prospectus supplement is truthful or complete.
Any representation to the contrary is a criminal offense.

                                -----------------

      The date of this prospectus supplement is June 14, 2004.

      The information in the table appearing under the heading "Selling
Securityholders" beginning on page 46 of the prospectus is amended by adding the
information below with respect to persons not previously listed in the
prospectus, and by superceding the information with respect to persons listed in
the prospectus that are listed below.

The following table sets forth certain information as of June 2, 2004, except where otherwise noted, concerning the principal amount of notes beneficially owned by each selling securityholder and the number of shares of common stock that may be offered from time to time by each selling securityholder under this prospectus. The information is based on information provided by or on behalf of the selling securityholders. The number of shares of common stock issuable upon conversion of the notes shown in the table below assumes conversion of the full amount of notes held by each holder at an initial conversion price of $10.65 per share. This conversion price is subject to adjustments in certain circumstances. Because the selling securityholders may offer all or some portion of the notes or the common stock issuable upon conversion of the notes, we have assumed for purposes of the table below that the named selling securityholders will sell all of the notes or convert all of the notes and sell all of the common stock issuable upon conversion of the notes offered by this prospectus. In addition, the selling securityholders identified below may have sold, transferred or otherwise disposed of all or a portion of their notes since the date on which they provided the information regarding their notes in transactions exempt from the registration requirements of the Securities Act. Information about the selling securityholders may change over time. Any changed information given to us by the selling securityholders will be set forth in prospectus supplements if and when necessary. 2

Other Shares of Common Stock Beneficially Aggregate Shares of Owned Before Principal Common Stock the Offering Amount of Issuable and Assumed Percentage of Notes Percentage of Upon to be Owned Common Stock Beneficially Notes Conversion Following the Outstanding Owned and Outstanding of the Notes Offering **** Name* Offered ** *** - ----------------------------------- ----------- ------------- ------------ ------------- ------------- Argent Classic Convertible Arbitrage Fund (Bermuda) Limited(1) 3,880,000 3.9% 364,332 None 1.0% Barclays Global Investors Diversified Alpha Plus Funds(2) 264,000 - 24,790 None - Dodeca Fund, L.P.(3) 580,000 - 54,462 None - Forest Global Convertible Fund, Ltd., Class A-5(4) 730,000 - 68,547 None - Forest Multi-Strategy Master Fund SPC, on behalf of its Multi-Strategy Segregated Portfolio(5) 779,000 - 73,148 None - - ----------------------- (1) Henry Cox and Allan Marshall are the controlling shareholders of Argent Classic Convertible Arbitrage Fund (Bermuda) Limited. Argent International Management Company, LLC is the investment advisor to Argent Classic Convertible Arbitrage Fund (Bermuda) Limited with respect to the securities listed on the table. Nathanial Braun and Robert Richardson exercise voting and investment power over such securities on behalf of Argent International Management Company, LLC. (2) Forest Investment Management is the investment advisor to Barclays Global Investors Diversified Alpha Plus Funds with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (3) Convert Co., LLC is the General Partner of Dodeca Fund, L.P. Thomas Y. Roy, President and Chief Financial Officer of Inflective Asset Management LLC, exercises voting and dispositive power over the securities on behalf of Convert Co., LLC. (4) Forest Investment Management is the investment advisor to Forest Global Convertible Fund, Ltd., Class A-5 with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (5) Forest Investment Management is the investment advisor to Forest Multi-Strategy Master Fund SPC, on behalf of its Multi-Strategy Segregated Portfolio with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. 3

Other Shares of Common Stock Beneficially Aggregate Shares of Owned Before Principal Common Stock the Offering Amount of Issuable and Assumed Percentage of Notes Percentage of Upon to be Owned Common Stock Beneficially Notes Conversion Following the Outstanding Owned and Outstanding of the Notes Offering **** Name* Offered ** *** - ----------------------------------- ----------- ------------- ------------ ------------- ------------- HFR CA Global Opportunity Master Trust(6) 203,000 - 19,062 None - HFR RVA Select Performance Master Trust(7) 203,000 - 19,062 None - Inflective Convertible Opportunity Fund I, L.P.(8) 20,000 - 1,878 None - LLT LTD(9) 112,000 - 10,519 None - Lyxor/Forest Fund Limited(10) 488,000 - 45,823 None - Relay 11 Holdings Co.(11) 80,000 - 7,512 None - Sphinx Convertible Arbitrage SPC(12) 176,000 - 16,526 None - - ---------------------- (6) Forest Investment Management is the investment advisor to HFR RVA Select Performance Master Trust with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (7) Forest Investment Management is the investment advisor to HFR CA Global Opportunity Master Trust with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (8) Inflective Portfolio Management, LLC is the General Partner of Inflective Convertible Opportunity Fund I, L.P. Thomas J. Roy, President and Chief Financial Officer of Inflective Asset Management LLC exercises voting and dispositive power over the securites on behalf of Inflective Portfolio Management, LLC. (9) Forest Investment Management is the investment advisor to LLT LTD with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (10) Forest Investment Management is the investment advisor to Lyxor/Forest Fund Limited with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (11) Forest Investment Management is the investment advisor to Relay 11 Holdings Co. with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (12) Forest Investment Management is the investment advisor to Sphinx Convertible Arbitrage SPC with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. 4

Other Shares of Common Stock Beneficially Aggregate Shares of Owned Before Principal Common Stock the Offering Amount of Issuable and Assumed Percentage of Notes Percentage of Upon to be Owned Common Stock Beneficially Notes Conversion Following the Outstanding Owned and Outstanding of the Notes Offering **** Name* Offered ** *** - ----------------------------------- ----------- ------------- ------------ ------------- ------------- Xavex Convertible Arbitrage 4 Fund(13) 33,000 - 3,099 None - Zurich Institutional Benchmarks Master Fund, LTD.(14) 292,000 - 27,419 None - Any other holder of Notes or future transferee, pledgee, donee or successor of any holder***** 66,000 - 6,197 - - - ---------------------- (13) Forest Investment Management is the investment advisor to Xavex Convertible Arbitrage 4 Fund with respect to the securities listed on the table. Forest Investment Management is wholly owned by Forest Partners II, the sole General Partner of which is Michael A. Boyd Inc. Michael A. Boyd is the sole owner of Michael A. Boyd Inc. (14) TQA Investors, L.L.C. is the investment advisor for Zurich Institutional Benchmarks Master Fund, LTD. with respect to 150,000 of principal amount of the securities listed on the table. Robert Butman, Managing Member of TQA Investors, L.L.C., exercises voting and investment power over such securities. However, Zazove Associates, LLC is the investment advisor to Zurich Institutional Benchmarks Master Fund, LTD. with respect to 2,050,000 of principal amount of the securities listed on the table. Gene T. Prelti exercises investment and voting control on behalf of Zazove Associates, LLC. ***** Assumes that any other holders of Notes, or any future transferees, pledgees, donees or successors of or from any such other holders of Notes, do not beneficially own any common stock other than the common stock issuable upon conversion of the Notes at the initial conversion rate. 5