UNITED STATES
                     SECURITIES AND EXCHANGE COMMISSION
                           Washington, D.C.  20549
                        ______________________________

                                   FORM 8-K
                               CURRENT REPORT


                   PURSUANT TO SECTION 13 OR 15(d) OF THE
                      SECURITIES EXCHANGE ACT OF 1934


      Date of Report (Date of earliest event reported):  May 20, 2004


                                   0-15898
                          (Commission File Number)

                       ______________________________

                     CASUAL MALE RETAIL GROUP, INC.
        (Exact name of registrant as specified in its charter)

      Delaware                                             04-2623104
(State of Incorporation)                                 (IRS Employer
                                                      Identification Number)


              555 Turnpike Street, Canton, Massachusetts 02021
            (Address of registrant's principal executive office)


                              (781) 828-9300
                     (Registrant's telephone number)
                      ______________________________



















ITEM 7.  Financial Statements, Pro Forma Financial Information and Exhibits.

	(c) Exhibits
	    Exhibit No.	Description
          -----------   ------------
	     99.1		Press Release announcing Casual Male Retail Group,
	      		Inc.'s First Quarter Fiscal 2004 Results and CMRG's
				announcement of the sale of its interest in Ecko
				retail stores.

ITEM 12.  Results of Operations and Financial Condition.

On May 20, 2004, Casual Male Retail Group, Inc. (the "Company") issued
a press release announcing operating results for the first quarter of fiscal
2004, which ended on May 1, 2004.  In addition, the Company announced that on
May 19, 2004, the Company signed a sale agreement with Ecko.Complex, LLC to
sell its 50.5% joint venture interest in the Ecko Unltd. (r) outlet and full-
price stores to Ecko. Complex, LLC.

The press release contains certain non-GAAP measures which the Company believes
is important for investors to help gain a better understanding of the Company.
The release includes a reconciliation of such non-GAAP measures.

A copy of the press release is attached hereto as Exhibit 99.1 and is
incorporated by reference herein.






































                                  SIGNATURES

Pursuant to the requirements of the Securities Exchange Act of 1934,
the registrant has duly caused this report to be signed on its behalf by the
undersigned, thereunto duly authorized.

                                              CASUAL MALE RETAIL GROUP, INC.

                                              By: /s/ Dennis R. Hernreich
						  ---------------------------
                                              Name:  Dennis R. Hernreich
                                              Title: Executive Vice President
                                                     and Chief Financial Officer


Date:  May 20, 2004



For Information, Contact:

Jeff Unger, Investor Relations
(561) 514-0115

Andrew Bard, Weber Shandwick
(212) 445-8368


         CASUAL MALE RETAIL GROUP INC. ANNOUNCES FIRST QUARTER
                       FISCAL 2004 RESULTS

     ~ CMRG sells its interest in the Ecko Unltd.(r)  retail stores ~

CANTON, MA  (May 20, 2004)  -- Casual Male Retail Group, Inc. (NASDAQ/NMS:
"CMRG"), retail brand operator of Casual Male Big & Tall, and the exclusive
retailer of the Comfort Zone by George Foreman(tm), GF Sport by George
Foreman(tm) and Signature Collection by George Foreman(tm), today announced
its operating results for the first quarter of fiscal 2004 of a net loss of
$5.1 million, or $0.15 per share, compared to a net loss of $2.8 million, or
$0.08 per share, for the first quarter of 2003, as more fully described
below.

For the first quarter of fiscal 2004, the Company reported a loss from
continuing operations of $4.1 million, or $0.12 per share, compared to a loss
from continuing operations of $2.6 million, or $0.07 per share.  Assuming a
normalized tax rate of 37% for fiscal 2004, the adjusted loss from continuing
operations for the first quarter of 2004 was $0.07 per share as compared to an
adjusted loss from continuing operations of $0.05 per share last year.  Casual
Male Big & Tall sales for the first quarter of fiscal 2004 were approximately
$78.1 million compared to the prior year's first quarter of $72.8 million.
Casual Male Big & Tall comparable store sales for the first quarter of fiscal
2004 increased 9.2%.  The Company's loss widened during the quarter primarily
due to the approximately $3.4 million of marketing expenses spent on the
national advertising launch of its George Foreman line of clothing.

Also during the first quarter of 2004, the Company recorded a loss from
discontinued operations of approximately $1.0 million, or $0.03 per share,
related to the acceleration of plans to close and the closing of several
Levi's(r)/Dockers(r) outlet stores.  The Company anticipates that at the end of
2004, the Levi's(r)/Dockers(r) business will generate sufficient operating
income, together with previously established reserves to fund the planned
closings of approximately 25-30 Levi's(r)/Dockers(r) stores.

"The first quarter of 2004 represents a very significant milestone as comparable
store sales achieved levels which have never been accomplished at Casual Male in
the past decade.  Key improvements in Casual Male's merchandise assortments as
part of our overall key merchandising initiatives, as well as the successful
national launch of the Comfort Zone by George Foreman(tm) line of clothing
produced a dramatic turnaround in Casual Male's sales trends," stated
David A. Levin, President and Chief Executive Officer of CMRG.  We are
aggressively expanding all three George Foreman lines of clothing for
Fall 2004 and we are confident that Casual Male comparable store sales will
continue to perform at strong single digit levels for the balance of 2004."

"The Company's incremental marketing expenditure in the first quarter represents
a critical strategic investment for the Company's long term goal of increasing
its market share and brand awareness in the men's big and tall market and is not
a departure from the Company's keen focus on cost controls.  Due to the
investment in the Casual Male and George Foreman brands made in the first
quarter, the Company's incremental increase in SG&A of 17% in the first quarter
over last year includes approximately 70% of this year's anticipated increase in
marketing expenditures and while the Company anticipates an overall 5-7%
increase in SG&A for 2004, the first quarter results of an adjusted loss from
continuing operations of $0.07 per share versus an adjusted loss from continuing
operations of $0.05 per share a year ago reflects this marketing investment,"
added Dennis R. Hernreich, Executive Vice President and COO/CFO of CMRG.  "In
addition, Casual Male's gross margin for 2004 is expected to improve from the
prior year by between 100-150 basis points for the year in spite of the 80 basis
point drop in the first quarter.  Gross margins are strong in most of Casual
Male's merchandise assortments, particularly the George Foreman lines, and only
markdowns taken in young men's and active wear seasonal categories resulted in
lower gross margins during the first quarter."

The Company also announced the sale of its 50.5% joint venture interest in the
Ecko Unltd.(r) outlet and full price retail stores, to its 49.5% partner
Ecko.Complex, LLC.  In connection with the sale, the Company shall receive a
cash payment totaling $8 million, an amount which approximates twice its
investment in the joint venture.  The sale agreement was entered into on May 19,
2004 and is expected to close prior to the end of the second quarter of fiscal
2004.  "The sale of our interests in the Ecko Unltd.(r) retail stores will allow
the Company to devote its resources towards its primary Casual Male business.
We will continue to offer the Ecko Unltd.(r), the largest urban lifestyle brand
In the country, merchandise assortments in our Casual Male Big & Tall stores
as a staple of our young men's lifestyle," stated Mr. Levin.

Operating Results by Business Segment
- -------------------------------------
The following table reflects operating results by segment for the Company's
first quarter of fiscal 2004 and for the prior first quarter of fiscal 2003.
These segment results have been prepared on a basis consistent with the
presentation in the Company's recent Form 10-K and Form 10-Q filings.

For the three months ended:    May 1, 2004                May 3, 2003
(in millions)
                               Other                         Other
                    Casual    Branded             Casual    Branded
                     Male     Apparel Combined     Male     Apparel   Combined
                   Business  Business  Company    Business  Business  Company
                   --------  -------- --------    --------  --------  ---------

Sales              $  78.1  $  25.7  $  103.8    $  72.8    $  19.5     $ 92.3
Gross margin, net
 of occupancy
 costs                31.8      5.5      37.3       30.2        3.9       34.1
Gross margin rate     40.7%    21.5%     36.0%      41.5%      20.0%      36.9%
Selling, general and
Administrative        30.9      6.1      37.0       26.3        5.6       31.9
Depreciation and
Amortization           1.9      0.7       2.6        1.5        0.5        2.0
                   --------  -------- --------    --------  --------  ---------
Operating income
 (loss)             $ (1.0) $  (1.3)   $ (2.3)    $  2.4     $ (2.2)    $  0.2
                   ========  ======== ========    ========   =======   ========

In addition to the financial measures prepared in accordance with generally
accepted accounting principles (GAAP), our above discussion refers to a
normalized tax rate, which is a non-GAAP measure. Normalized tax basis reflects
a 37% effective tax rate on the pre-tax loss. The Company believes that the
inclusion of such non-GAAP measure helps investors to gain a better
understanding of the Company's performance, especially when comparing such
results to previous periods or forecasts. However, the non-GAAP financial
measures included in this press release are not meant to be considered superior
to or as a substitute for results of operations prepared in accordance with
GAAP.  The following table shows the reconciliation of the loss from continuing
operations on a GAAP basis effected for the adjustment for normalized taxes:

For the three months ended:         May 1, 2004               May 3, 2003
(in millions, except         -------------------          ---------------------
for loss per share)                       Loss                         Loss
                                           Per                          Per
                                          Share                        Share
                             -------------------          ---------------------
Loss from continuing
 operations, as reported     $  (4.1)   $(0.12)           $  (2.6)    $(0.07)
Income tax benefit,
 assuming normalized tax
 rate of 37%                    (1.5)      --                (1.0)       --
                             --------   -------           --------    -------
Adjusted loss from
 continuing operations,
 after normalized tax
 benefit                     $  (2.6)   $(0.07)          $   (1.6)    $(0.05)
                             ========   =======          =========    =======
Weighted average shares
 outstanding
 -basic and diluted             35.1                         35.8


CMRG will conduct a conference call to discuss its first quarter earnings
results today at 11:00 a.m. Eastern Time at www.casualmale.com/investor. The
call will be archived online within one hour after its completion. Participating
in the call will be Seymour Holtzman, Chairman, David Levin, President and Chief
Executive Officer and Dennis Hernreich, Executive Vice President, Chief
Operating Officer and Chief Financial Officer.

Commencing in the second quarter of 2004, the Company will provide monthly
Casual Male sales and comparable store sales results, and investor conference
calls when necessary, so to keep the investor community more promptly informed
of the Company's progress.

CMRG, the largest retailer of big and tall men's apparel, operates 487 Casual
Male Big & Tall stores, the Casual Male e-commerce site, Casual Male catalog
business, 52 Levi's(r)/Dockers(r) Outlet by Designs stores and 26 Ecko
Unltd.(r) outlet and retail stores, pending consummation of the sale discussed
above, located throughout the United States and Puerto Rico. The Company is
headquartered in Canton, Massachusetts and its common stock is listed on the
Nasdaq National Market under the symbol "CMRG."

The discussion of forward-looking information requires management of the Company
to make certain estimates and assumptions regarding the Company's strategic
direction and the effect of such plans on the Company's financial results. The
Company's actual results and the implementation of its plans and operations may
differ materially from forward-looking statements made by the Company. The
Company encourages readers of forward-looking information concerning the Company
to refer to its prior filings with the Securities and Exchange Commission that
set forth certain risks and uncertainties that may have an impact on future
results and direction of the Company. The Company does not report on its
progress during a quarter until after the quarter has been completed and its
results have been appropriately disclosed.

                              (Tables to follow)



                        CASUAL MALE RETAIL GROUP, INC.
                     CONSOLIDATED STATEMENTS OF OPERATIONS
                     (In thousands, except per share data)
                                 (Unaudited)


                                                 For the three months ended
                                              May 1, 2004          May 3, 2003


Sales                                        $  103,812             $  92,343

Cost of goods sold including occupancy           66,465                58,256
                                             ----------             ---------
Gross profit                                     37,347                34,087

Expenses:
   Selling, general and administrative           37,071                31,884
   Depreciation and amortization                  2,569                 1,985
                                             ----------             ---------
Total expenses                                   39,640                33,869
                                             ----------             ---------
Operating income (loss)                          (2,293)                  218

Interest expense, net                             2,179                 2,885
                                             ----------             ---------
Loss from continuing operations
 before minority interest and income taxes       (4,472)               (2,667)

Minority interest                                  (347)                  (72)

Income taxes                                         -                      -
                                              ----------            -----------
Loss from continuing operations               $  (4,125)             $  (2,595)

Loss from discontinued operations                  (969)                  (160)
                                              ----------            -----------
Net loss                                      $  (5,094)             $  (2,755)
                                              ==========            ===========

Net loss per share - basic and diluted
  Loss from continuing operations                ($0.12)                ($0.07)
  Loss from discontinued operations              ($0.03)                ($0.00)
  Net loss                                       ($0.15)                ($0.08)

Weighted-average number of common shares
Outstanding- Basic and diluted                   35,126                  35,754


                       CASUAL MALE RETAIL GROUP, INC.
                        CONSOLIDATED BALANCE SHEETS
                      May 1, 2004 and January 31, 2004
                              (In thousands)

                                                 May 1,           January 31,
                                                  2004                 2004
ASSETS                                        -----------         -----------
                                              (unaudited)

Cash and investments                           $  2,181            $   4,179
Inventories                                     109,822               98,673
Other current assets                             15,565               10,831
Property and equipment, net                      69,933               68,345
Goodwill and other intangibles                   81,281               81,306
Other assets                                      9,565                9,408
                                              ---------            ----------
 Total assets                                 $ 288,347            $ 272,742
                                              =========            ==========
LIABILITIES AND STOCKHOLDERS' EQUITY

Accounts payable, accrued expenses
 and other liabilities                        $  66,553            $  59,155
Notes payable                                    15,797                3,623
Restructuring reserves                            2,945                2,945
Long-term debt, net of current portion          122,190              122,374
Minority interest                                 4,451                3,804
Stockholders' equity                             76,411               80,841
                                              ---------            ---------
 Total liabilities and stockholders' equity   $ 288,347            $ 272,742
                                              =========            =========